he would probably be genuinely confused that you thought that way.

Yeah, but that's the thing -- it doesn't stop it from being offensive. I mean, if you were talking with a guy and he called someone an "old broad," wouldn't you at the very least cringe?