The fact that I don't have any kind of business relationship with them and am not seeking them out for the purpose of further bothering?
I don't understand why people care about hurting others by using "gay" in that way, but those who are upset about this issue are branded overly PC and much worse--even when they have a personal stake in it. I tend to let it go (except when there is a debate ABOUT it, such as here) precisely because of the negative reaction to people who disagree with it. However, if I had even more of a personal stake in it-- if I had a child who was developmentally disabled, say--you bet your ass I'd be coming down hard on people who upset me that way, and I feel that I'd have every right to be hurt. For some reason, other people don't. I don't understand that.
Suppose somebody is walking down the street with a basket. Somebody bumps into them, spilling the contents. "What the fuck, man? Are you blind?"
Yes, language changes. But we're not there yet.
Disability is bad. It causes its sufferers to have difficulty in their everyday lives, bars them from a variety of experiences, hinders their ambitions, and (I guess this is the part that makes you angry?) causes inconvenience and discomfort to others.
Nope, but I expect that's the way it'll go. Good or bad? Dunno. Language is emergent behavior, and the collective species is a bit of a dick.