Upsets them? I doubt it. Men don’t usually respond to things that emotionally.
I’d say that if you want to be taken seriously, anyone (regardless of their sex) ought to use terminology that the other person doesn’t consider to be derogatory. That’s all.