Hey, we found R'lyeh! Horrifying city of impossible geometry, presented as scholarship on scholarship on reports? It fits, doesn't it?
House of Leaves did what H. P. Lovecraft continually tried to do: it made a horror out of the unknown and impossible. I know Lovecraft has his share of crazed fans*, and I won't insult them by saying Lovecraft's efforts were a universal failure on his part; I merely mean they did not work for me. Lovecraft relied heavily on the exotic mystery of his ancient lore and his Elder Gods from before the dawn of time to unsettle his readers, and perhaps in the 1920's this was effective, but this is an age when we know what the Ancient Egyptians and the Babylonians and the Mayans meant by their mystical symbols, and it turns out to be nothing more ominous than the guest list for the high priest's barbecue and his sales receipts for the hot dogs and buns. We have year-round stations on Antarctica and probes on Mars, and we have a pretty good general idea of what to expect from Out There. To achieve a similar goal as Lovecraft's for a world in which the main suspense centers on how low human nature will stoop House of Leaves has to create a new uncharted space. Unlike Lovecraft's, this isn't an uncharted space in a region we know to be uncharted; it's an uncharted space that we know to be charted. The book doesn't rely on the setting or players being in any way exotic to us; quite the opposite. Somehow I can't imagine reciting Red Line stops would save your sanity in that house.
The book uses something of a distillation of Lovecraft's approach, I think, because it cuts out the suspension of disbelief. Flying shapeless monsters that defy the laws of physics are scary, yes, but that's because they're *monsters*, and they can *rip our guts out*. Explicitly describing them as breaking the rules of what is possible and describable makes them a little scarier than your more traditional dragons, because it moves them out of myth into a world that *ought* to be describable, but the badness of an encounter with a shoggoth is not dependent on the shoggoth's indescribability but on its great capacity and eagerness to do you and the world harm. 'Tis better to have lived and gone insane than not to have lived at all, no? House of Leaves dispenses with that by taking out the "rip your guts out" step. It's just a little more wall space in the beginning. There is nothing scary about that ...except that when you think about it, it makes a lie of everything. There are no limitations, no specifications, on that "everything", not "everything that we know about the surface of Mars or the properties of spider bites"; just everything. When we read about this, we are aware it's just a book, but House of Leaves makes this less comforting by adding more (not fewer) layers of abstraction between us and the core events. What I present to you here is analysis of what was presented to me as analysis of what was presented to them as analysis of what was presented to that presenter in turn as analysis of [...]** what was presented to that first level of analyzers as a recording of an event. I believe all of them but me are just fiction, but that's what nearly every layer has thought so far. I seem to be no different than they are in my response to the text I'm given. Navidson could be having us all on. So there you have it: us. We're a group, a fictional group, but where you draw the line between real and unreal, where you say, "This has to be fiction, and that has to be fact", is pretty arbitrary. There's me -- I'm real. (I swear!) Then there's the person who presented it to me. I'll call him the author, and I'll assume he's real. It's not completely impossible -- although I don't believe it happens to be true -- that this person is not the author, and that he presented it to me after being given it by another person, who had a pretty wacky mind. Of course, there's no reason that there couldn't have existed somebody else who gave it to him, and so the not-wacky guy passed it on to the not-author who passed it on to me. It's all within the realm of possibility down to the Navidson family purchasing the house and measuring the walls. Then with the smallest deviation in the inches of the outer wall, it's suddenly impossible, only by that point it's too late and you've stumbled into a world where geometry doesn't work and where sleep never comes and where you find yourself writing rambling posts in a forum about electronic representations of pen and paper representations of real live stick figures. It's true, goddamnit.
*Much like xkcd
**How many times does this reiterate?
***Which is why you should never do that, ever.