Tim Wise, a white, Jewish, anti-racist, has a great analogy for what you just said. It's like living in a house with a bad smell. You may not be responsible for causing that smell, but you live in that house and someone has to clean up the mess, since it will only worsen over time and make it horrible for everyone in the house.