I think it's more of a case of the awarding of the Prize being somewhat more dubious in the case of the Literature and, particularly, the Peace ones. Usually its awarding for the Physics, Chemistry, Physiology and Medicine, and, to a lesser degree, Economics ones will be uncontroversial amongst the peer group for the respective field, and the merit of the recipient unquestioned and broadly praised. This is certainly different for the other two - particularly the Peace Prize which is awarded by a political group and can thus be slanted accordingly, Obama's receipt being case in point.
As for Literature, the original will of Alfred Nobel specified the requirement for an ideology or rather "idealism" to be present in the author's work. This was interpreted quite strictly at first with people like Joyce, F. Scott Fitzgerald and other such novelists being excluded. The whole history has a level of snubbing and counter-snubbing to it. Furthermore, there was always a level of national and idealogical pride/prejudice that played up like Sweden historically avoiding honouring Russia (Tolstoy, Chekhov) due their mutual antipathy or in the aftermath of Wars avoiding awarding to formerly combatant countries. There's been a bizarre tendency to disqualify some from consideration on the basis of translation or technical errors while at the other end of the spectrum there's been a persistent level of idealogical bias in deciding who gets the award. Both Historically and to the present day, the Nobel Prize for Literature tends to be about who isn't there rather then who is.
I mean, on only has to look at things like this to understand the justification for the reputation it has:
Nobel judge and permanent secretary Horace Engdahl told the Associated Press in an interview. wrote: "The US is too isolated, too insular. They don't translate enough and don't really participate in the big dialogue of literature. That ignorance is restraining."
"You can't get away from the fact that Europe still is the centre of the literary world, not the United States,"
“People understand me so poorly that they don't even understand my complaint about them not understanding me.”
~ Soren Kierkegaard