Why do people say you are racist if you think Obama is awful?

I think his policies should do the speaking. I think it is clear that he has the worst monetary policy in the history of our country. Whenever I try to debate people about the validity of his policies, more often than not somebody says I wouldn't be saying he was awful if he was white. What are your experiences with this?