Well Theon, both in the books and the show, is clearly a complex person. He's not exactly an asshole, but he's clearly not as nice as most of the Stark children. He has some good characteristics, and some flaws. Mostly, he's very vain, and quite naive.
The background is that Balon Greyjoy rebelled against the kingdom, sometime after Robert Baratheon took the throne. This rebellion was quickly squashed (they were hopelessly outnumbered) and Theon was given to the Starks as their ward. But ward really is a nice word for 'hostage' here. Though Ned, being the honourable guy that he was, tried to raise Theon to the best of his abilities. But Theon is just old enough to remember this. So it must have been a traumatic experience for him, ending up at the enemy.
So he's thorn between loyalty to his old family, who he still dimly remembers, and loyalty to the Starks, who were enemies at first, but ended up showing him love and kindness (though they never really accepted him. At least not all of them. Ned probably did, as did the children. But Catelyn never does).
And then he and Robb think up this plan for the Greyjoys to join Robb's rebellion. Obviously Theon is overjoyed with this. It's his chance to support both his old and his new family at the same time. He's pretty much expecting a hero's welcome at home. The long lost son returning home. But not just that, returning home with a plan that allows the Greyjoy's to gain significant influence and power.
The rejection of his father, both of him personally as a son, and of his plan to ally with Robb, is completely unexpected, and really shatters his confidence. He's desperate for recognition.
Also, like I said, he's naive. He clearly did not expect that conquering Winterfell would involve, you know, killing people.
It's one of those irregular verbs, isn't it? I have an independent mind, you are an eccentric, he is round the twist
- Bernard Woolley in Yes, Prime Minister