I understand exactly what you are concerned about, but I have to agree with those who say "yes," you're worrying too much - that way lies madness. As people have already articulated above, it would be difficult to see almost ANY part of the world as "innocent" of some kind of injustice, human pride, general wrongdoing.

On the other hand, your attitude does speak to some of the concerns we have now with works of literature that contain or display attitudes that we now find unpalatable. Or when we know something unpleasant about the author - I, for example, have difficulty reading William Mayne now.