Don't Be an Ass
The Shape of Days: Part of being an adult is knowing how not to be an ass
What are we, four years old again? People, this isn't that complicated an idea: Don't be an ass. If somebody asks you not to take their picture, say you're sorry and move along. Don't insist that "photography is not a crime." Of course it's not, and everybody knows it, and dammit, you know perfectly well that's not the issue. The issue is that when people are made uncomfortable by something you're doing, you should stop. That's called being courteous. It's called being respectful. It's called not being an ass.
(A bit of follow-up here: Hammering in my head and discussion here: Hassled for photography?.)