Whoa! Harsh! I had pretty much the opposite of Dark567's experience at Wisconsin. I dislike applied math and physics, too, which is why I didn't take any when majored in computer engineering. And I, too, would have hated doing single board computer design and signal analysis, so I've avoided jobs involving anything related to either of those things.
This was a few years ago, so perhaps things have changed, but pretty much no math was required for computer engineering while I was there, and none of the required courses were math intensive, either. EE required ECE 320 (E-mag II) and some sort differential equations course, but Comp E. just required the basic calculus sequence (which was required for every quantitative discipline anyway) and E-mag I (which was just a re-hash of stuff that was in the required introductory physics course, which was just AP physics).
Which advanced courses require busy work? I specialized in VLSI / architecture, so this might be different for other areas, but the core courses (551, 551, 552, 553, 554, 555, and 556 – Verilog design and synthesis, computer architecture, DFT, microprocessor design lab, VLSI design / layout, and EDA/CAD, for you non-Wisconsinites) had no busy work. The projects and exams were weighted heavily enough that you didn't even have to do the homework to get an A. Maybe you disliked the projects? I thought doing stuff like trying to make the smallest / fastest / lowest power multiplier, or putting together a P6-esque Sega emulator in an FPGA was a lot of fun, but YMMV. If you skipped class, each course was maybe 10-40 hours of work on a project (except for 554, which was more like 100-200 hours), plus 4 hours of exams. I think that’s not too bad compared to a lot of other majors, where you often to have show up for discussions, or there are weekly quizzes or homework assignments.
I also disagree with the assertion that computer engineering is so specialized that you don't have time for other things. At least at Wisconsin, if there’s some requirement you want to skip, you can probably talk somebody into letting you work around it. Personally, I was a huge slacker, and didn't even have the minimum GPA requirement to make it into the program, but the Dean let me in conditionally. I didn't want to take the boring required discrete math course, so I got a prof to sign off on letting me use one of my combinatorics courses towards that requirement (and I hear it's standard to allow that substitution now). There was an annoying rule that you had to get someone to sign off on it if you wanted to take more than 21 (or is it 23?) credits, but like most rules, it was just a formality, and it wasn’t too hard to get the right signature. I think it's easy to get stuck feeling like the system is some giant bureaucracy where no one cares, but most professors do actually care about their students. There are some notable exceptions, but those will exist everywhere, and you can avoid them if you ask around.
It’s funny that you mention being interested in econ and philosophy, since I had the same interests in undergrad. I would have easily had enough time to finish my additional major in econ if I hadn’t gotten an additional major in yet another area, and taken a bunch of random fun non-major courses, or if I’d stayed for a seventh semester. It’s true that there’s more required coursework for engineering than for other majors, but most majors at Wisconsin only require something like 40 credits, and most of my friends graduated with 150+ credits, so you could easily rack up three or four majors, including econ and philosophy, if you avoid taking non-required classes, but where’s the fun in that?
On the original topic, for reasons which seem pretty silly to me, it seems like a computer engineering degree gives you a lot more flexibility in terms of your career. Companies like MS and Google are perfectly willing to interview CEs for software positions, and companies like Intel and Lucent are happy to interview CEs for hardware positions, but it's and it's a little tougher to get a non-embedded software position with a pure EE degree, and it's really hard* to get a hardware position with a CS/SE degree.
*with the obvious exceptions, like specializing in computer architecture in a CS department, and then going after a computer architecture position.