This is my idea on how the protagonist would be able to 'hack' the simulation. There are normal, run of the mill, paradoxes
that humans cannot or have not solved; lets assume that the computer can solve these paradoxes. When the protagonists discovers a paradox that the computer cannot solve (perhaps he cannot solve it either), any attempts to express* the paradox result in the world glitching. As the protagonist discovers new paradoxes and experiments with different ways to express these paradoxes he can better predict what glitches will result. Eventually, he gets so good at this that when he wants something to happen, he knows exactly how to express a paradox who's corresponding glitch is what he desired. Despite this technically
not being hacking, it should be close enough for your story. Also, you could use questions that cannot be answered instead of paradoxes that cannot be resolved.
*Whether thinking about the paradox qualifies as an expression of that paradox is up to you to decide. Personally, I would have it so that the protagonist can think of the paradox without anything changing and have the world glitch whenever he talks about it or writes it down, because that implies that the minds of the people in the simulation exist independently of the simulation itself.