Why should people feel sorry about past racism?

The argument for affirmative action is that it is to amend for the terrible racism that Blacks endured in America's past. A lot of Blacks do feel that they should receive retribution for what happened to their forefathers. This is something that feels weird to me. It's like saying that because my great great grandfather enslaved your great great grandfather, I now will be punished for what he did centuries earlier. Plus, the US is made up of millions of immigrants who had no part in the enslavement of Blacks. Yet, they will have to pay for it as well. It is just an unsettling feeling that I have. Maybe, I am the only one who feels this way. I hope my great great grandchildren do not have to pay for the mistakes that I made.