I hate someone who's gay but not because he's gay, how do I say it?

There's guy who's gay and is overusing his gayness to play around with me and others (who's guys too) and sometime we pick on him and he just flirt or get close with us to bug us out. I entirely hate him for that. He act really bitchy, think he own it and think it's absolutely fine to attack guys. How do I say that I hate him but not because he's gay and act extremely fag? He DID change my point of view to gays...in a bad way. Sorry.