Not to split hairs here, but gay people and black people might not see with you on this one eye to eye. Not sure anyone has ever been lynched to cries of "retard," and I'm positive that "retard bashing" is not a thing. I don't think the word retard belongs in the same category as "nigger" or "fag." Frankly implying such a thing is more offensive than the words themselves.
I disagree. Plenty of people with mental handicaps have been beaten and murdered for it. Just a few sources. It doesn't get reported as much as violence against gays (a group I fall into) and blacks, (cynical moment: probably because it's not as 'sexy' for news media), but it's definitely there, and it always has been - all throughout history. Don't forget the Nazi practice throughout their reign of executing those with mental handicaps and birth defects (Nazi Germany wasn't anywhere near or the last nation to do this, but it's probably the most well known). It does belong in the same category, I think.