The “who makes the call?” from Adam D’s post also gets complicated, because while the obvious answer is “the group affected”, you sometimes (often?) get contradictory answers from different members of the same group when things are fuzzy.
(I mean, sure, there’s times when you’re so far over the line that you’re obviously offensive. That doesn’t seem like what we’re talking about.)