Disclaimer: That is a phrase that would NEVER come out of my mouth except to criticize it.
It makes me crazy. It's true, not everything is racist, but by saying that you basically out yourself as being one of those people who think that there's too much focus on racism/sexism/homophobia.
And, you're probably also one of those people who is "tired" of hearing about racism/sexism/homophobia. In other words, The Worst.
To be clear, I think you can disagree with a characterization of something as racist/sexist/homophobic. But I think you should do it with some substance.