This is why I will not apply to fellowships aimed at anybody other than the general population for grad school unless I'm really, really desperate. It is not unheard of for a lot of non-white/Asian/Indian people and a handful of women (it's pretty much unheard of in the biological sciences since we're now 50 percent - actually now more than 50 percent, might need to kick in a little for men - of PhD recipients per year. Now we just need to achieve parity at the faculty level) to get this sort of harassment and accusation that they supposedly only got in because they were non-white/Asian/Indian or female.
I think this is not a racist reaction but rather a hidden cost of having those programs to begin with. When we use different acceptance criteria for women or minorities, it comes with a racist/sexist assumption that they need extra help or special treatment. In a real sense that undermines the legitimate work and contributions they make later. It's tough to have it both ways because if there are different admission criteria for minority groups, then it is correct to say that some got in only because they were non-white/Asian/Indian or female. If there are not different admission criteria, that distinction no longer exists. I have heard the arguments for affirmative action too, however, and they are good. I don't know the best way to fix it though.