(Which I am, btw.)
As a referee, I have to say, honestly, I can't remember if I've ever given a card for something said to ME. I've been known to card players getting too verbally aggressive with each other, spectators, coaches, etc., but I think I'm a pretty thick skinned ref. I always think twice and try to see that there will always be someone who doesn't like my call, specifically those who it is against, and unless there is swearing or deep, deep insults to my family or something (Past the "Your mother!" level, of course. :p ) then I'll just wave it off. I dunno, maybe I'm too lenient. :p
As a player, though, I've seen, and been on the carding end of refs who won't hear you have two words about a call, which I think is really unfair. I know I really won't influence a ref's call with my complaining, but in the moment, I think the ref should understand a little anger directed towards them, especially in the matter of a very close 50-50 call, or a game changing one.