It's a problem that's been around for just shy of 40 years now, and will probably be around for at least 40 more. Most commonly, the disputes that arise are due to:
- Different ideas about what a certain rule or ability does
- A mistake on the player's part that makes them want to strong-arm the GM into a "do-over"
- An idea or plan that a player has fallen in love with, and cannot accept has failed to produce results
My favorite means of dealing with this behavior is to nip it in the bud by laying out a set of ground rules at the start of a campaign. Lots of GMs like to prepare a campaign packet beforehand, letting the players know some back story for the game, listing important factions and NPCs, as well as specific materials and books they recommend. All of this information doesn't need to take up more than a page or two, and can serve as the spoonful of sugar for the medicine that is a final page of campaign policies
The term policy has a very firm connotation. It is most likely that your players have all encountered policies in their work or school lives, and so they know that such things are taken seriously by whoever has set them. Subtle things like this can, over time, make all the difference.
A way to set the tone with a new group of players is to (politely) insist that they roll their stats in front of you, explaining that it simply a matter of personal policy in how you like to run your games and no particular reflection on your trust in them. It sends the message that you have a handle on your game, and are not afraid to induce a little squirming to maintain that handle. If you have the luxury of a longtime gaming pal in your otherwise new group, make sure that the new players see her rolling stats in front of you as well, so they get the message that there will be no favoritism.
One policy I've found to work very well is an edict against rules-lawyering. We're only human and we will make mistakes when it comes to a strict interpretation of the rules. These mistakes might, if left uncorrected, even have a seriously negative impact upon the player character(s) they are affecting.
This changes nothing.
Most of us only have a limited amount of time during any given week to come together and indulge in our little slice of escapism, and if that time is eaten up by one person rifling through books and heatedly debating their nuances, it is a disservice to everyone else at the table. Don't allow it, don't engage in it. Simply do your best to become familiar with the rules (so you make the fewest mistakes possible), and explain to your players that any concerns should be made a note of, and discussed with you at the end of the game, preferably in private.
If and when you realize you've made a mistake (it happens to the best of us!), be sure to smilingly concede the point. This is not a competition, and even a hint of non-ironic pouting might instill an "us versus him" mentality that you really want to avoid. You shouldn't ever retcon something that has happened in game, but you can make it up to the player by offering a little extra XP, a hero point, or maybe even a new magical item that's just their size (this last should ideally be worked into the story as smoothly as possible).
If the mistake resulted in the actual loss of a character, even that isn't an unfixable situation. Even the best fantasy stories of all time have their contrivances and miraculously improbable resurrections! Simply whip up an excuse to save them, and then give them a shot at coming back in dramatic fashion, maybe even somehow carrying new plot information and a nice chunk of free XP they didn't have before.
What about generally cantankerous behavior? Some otherwise great people can turn into real passive-aggressive ogres around the game table, muttering about this and glaring after that, or getting all worked up over something else. Don't put up with it! It's important that you, as the GM, enjoy the game as much as anyone else. My remedy for this has two parts -
First, consider the teachings of Pavlov. Conditioned behavior is a real thing, and can result in all sorts of positive changes. If a player is getting snarky and unpleasant, consider making their character's life subtly more difficult. The key word here is subtly. As tempting and hilarious as it may seem to drop a house on them, this will only exacerbate the bad behavior. However, if you do small things when they get out of line, it will eventually have an effect. Declare them to have failed a Will save that they actually just BARELY would have made, as though the DC had been a single point higher for them. Perhaps they get a slightly lower ratio of useful loot than the rest of the party, or perhaps they miss a trap they otherwise might have found. Skill checks and saves are the best way to go about this without them catching directly on.
The other side of that is to reward them when they are being pleasant to game with. Maybe an NPC is a little nicer to them, they make a save they otherwise would have just BARELY failed, or the treasure the party finds is just a tiny bit (10% or so) richer than they would otherwise have expected. This works wonders if done properly.
Secondly, a discrete one on one conversation. Don't stop the game to tell them to knock it off, because the satisfaction you'll feel will come at the cost of embarrassment and resentment on behalf of the offending party, and could possibly escalate into a shouting match that splinters the group. Simply take them aside at some point, ideally after calling a ten minute break (a good idea for any game), and let them know that this isn't okay. You're doing your best, and you get enough harassment from your boss/teacher/customers that you don't need it here, in a game you're trying to facilitate for their enjoyment. Be sure to remain relaxed during this conversation, and if they get defensive just let them get it out of their system. If they have a specific complaint, address it, sure. But let them know that this isn't the way to go about bringing it to your attention, or making their voice heard. Remind them of the policies outlined at the start of the campaign, and if they can't get a handle on it, it may be time to amicably part ways.
Of course that isn't something anybody wants to happen.
Remember - you aren't there to be a smug tyrant god that lords your power over your friends for four hours a week, but neither are you there to be a punching bag upon which they can take out frustrations that they simply can't vent on other, more serious authority figures. Do your best to keep the PCs the stars of the show, but don't be a doormat. Be even handed in the choices and calls you make, and stand by those rulings even if the player doesn't agree. Above all, don't let your group lose sight of the fact that it is just a game, and that anger over a game is behavior befitting of children.