If you're going for cheap, you might look for old Palm devices - like Palm III series and earlier. These all had the old cradle connector and ran on Alkaline batteries - so despite being old hardware you needn't worry much about battery viability... Then you would just want to transfer notes off the device as soon as possible, to manage the risk of battery drainage. My assumption that this will be a good, cheap solution is based on the fact that this was one of the few Palm cradle connectors that actually stuck around for a while, up to the peak of Palm's popularity - they introduced a new connector with the m500 line that was supposed to be universal for all their devices and make it easier for people to design add-on hardware, but I don't think it really caught on as an add-on system the way the original cradle connector did... The downside is that you'll need a serial port in order to be able to interface with the thing - so a USB-to-serial adapter may be necessary if you go this route.
The infrared keyboards like ash.gti mentioned would also work - these have the advantage of being hardware-compatible with almost any device you might want to connect with, then it's just a matter of getting the right software. An infrared keyboard may be the only viable option if you want to use a device that offers more data security than a RAM-only Palm would (for instance, flash memory, like on newer Palms or various WinCE devices...)
If you get yourself the right kind of keyboard, a solution like this will be substantially more compact than even a netbook would be... Though if you can't find a real good deal on this gear, I'd just go for the netbook.