I believe they would combine and
>What if you threw a pokeball at a pokeball?
I was under the impression they could only capture pokemon. Not humans, not cars, not houses, not even other pokeballs.*
>What if you released pokemon from two balls at once into an area only large enough to hold one?
Given how it appears when one is released, it may be that a pokeball does a quick scan of the area to make sure the matter will be able to occupy the space. If it cannot, it probably cancels the procedure or moves the field elsewhere. Or, I'm going to assume such safety precautions were put in to place after "The Experiment" that left two 8 year old boys scarred for life, trying to make sense of the inside out bits of what was once their squirtles.
*My impression comes from half-watching a few dozen episodes. I may be mistaken. But, if that was the case, that'd be the easiest way to get Pikachu from Ash.. Team Rocket could just shove Ash, Brock, and Misty in a couple of pokeballs and then capture Pikachu. Problem solved, right? So.. yeah, that's where I get my assumption.