Server Spec Fun time!

[phaerus]

So, as most any self-respecting nerd will tell you, when a problem comes along there are two courses of action: buy simply the cheap parts that will remedy your situation, or go balls to the wall and use it as an excuse to upgrade everything.

I fall into the latter category of nerd, and though I'm not completely certain I'd like to buy a new motherboard/cpu/ram, it's always enjoyable to look and have the information handy.

So here it goes! This will be a file server. Chassis, hard drives, and PSU are already in place. We'll just be upgrading the CPU, Motherboard, and RAM. It will most likely run Debian Linux. Speed isn't as important as bandwidth and multi-threaded ability.

The following categories define needs and wants of the system in order of importance:
NEED
1> plays well with linux
2> fat buses. This means at least 3 PCIe 4x or higher slots
3> at least 1 PCIe 8x slot
4> multi-cored. 2 core minimum, 4 cores preferable, speed of each core not important

WANT
1> integrated video
2> SATA / SAS controllers on board
3> Low power
4> Integrated gbit LAN

Again, not sure I'll actually be upgrading this all quite yet, but if anyone would like to contribute to the parts extravaganza, I look forward to seeing you recommendations!

[Carnildo]

Do you really need the multiple cores? I've got a fileserver running on an Atom 230 (reportedly equivalent to a 900Mhz Celeron), and it can write to the software RAID-6 array at 100Mbps while using less than 15% of the CPU. If you're doing something like encrypting the data streams, I could see the need, but simply pushing data over the network doesn't take much.

[phaerus]

Do you really need the multiple cores? I've got a fileserver running on an Atom 230 (reportedly equivalent to a 900Mhz Celeron), and it can write to the software RAID-6 array at 100Mbps while using less than 15% of the CPU. If you're doing something like encrypting the data streams, I could see the need, but simply pushing data over the network doesn't take much.

It's not so much a speed issue as it is that I'm tired of waiting for a CPU scheduler or a single core to catch up to whatever it is I'm doing. The server will often be piping a video stream across the network while doing a fair bit of compressing/decompressing into and out of archives.

So yea, I don't need a fast processor at all, I just want something that's at least 2 cores (preferably 4) to keep everything running smoothly.

[phillipsjk]

So, as most any self-respecting nerd will tell you, when a problem comes along there are two courses of action: buy simply the cheap parts that will remedy your situation, or go balls to the wall and use it as an excuse to upgrade everything.

I fall into the latter category of nerd, and though I'm not completely certain I'd like to buy a new motherboard/cpu/ram, it's always enjoyable to look and have the information handy.

I was the opposite: slowly working up to "newer" parts as it becomes clear the old parts aren't working. I have also been working on my "fileserver" for years. I find configuration (FreeBSD) is the hard part. Made harder by trying to run a heterogeneous network with GNU/Linux and Windows clients. The BSD NFS uses a different dialect than the GNU/Linux NFS for example. (And windows doesn't understand NFS at all)

I fall into the latter category of nerd, and though I'm not completely certain I'd like to buy a new motherboard/cpu/ram, it's always enjoyable to look and have the information handy.

So here it goes! This will be a file server. Chassis, hard drives, and PSU are already in place. We'll just be upgrading the CPU, Motherboard, and RAM. It will most likely run Debian Linux. Speed isn't as important as bandwidth and multi-threaded ability.

The following categories define needs and wants of the system in order of importance:
NEED
1> plays well with linux
2> fat buses. This means at least 3 PCIe 4x or higher slots
3> at least 1 PCIe 8x slot
4> multi-cored. 2 core minimum, 4 cores preferable, speed of each core not important

WANT
1> integrated video
2> SATA / SAS controllers on board
3> Low power
4> Integrated gbit LAN

Again, not sure I'll actually be upgrading this all quite yet, but if anyone would like to contribute to the parts extravaganza, I look forward to seeing you recommendations!

Your "NEED"s (2,3,4 in particular) conflict with your "WANT"s (3 in particular). Before a Memory upgrade and actually installing a DVD burner for backups (I expect current power consumption to be similar), my server drew 52 Watts under load (one drive), 37 Watts idle (may be lower with extra power management feature enabled now: need to retest). The multi-core CPUs alone will draw about 65Watts, with another 20Watts for the board.

If you insist on a new board, try to get one that can Wake On LAN from any frame directed to it, rather than a specific magic packet. You may have to use suspend rather than soft-off mode to wake within 2 minutes, preventing a TCP timeout (assuming the first packet isn't lost). I just had a thought: a lot of the boot time on my machine is the memory test: with ECC memory you don't need to test the memory on start-up.

My machine: (Base is a Compaq Deskpro I got for $20, no drive, 16MB RAM sticks that were replaced)

• PII 350Mhz
• 256MB PC100 RAM (PC133 doesn't work, upgradeable to 384MB (not worth it))
• 80GB 160GB hard disk attached to Promise (Ultra100TX2) PDC20268
  The apparently defective 80GB drive was actually faster than the replacement 160GB drive
• Intel 82558 Pro/100 Ethernet NIC with WOL
• Integrated sound: ESS Audiodrive ES1869F
• USB (Intel 82371AB/EB (PIIX4) USB controller)
• Video: ATI Rage Pro Turbo AGP (2 or 4 MB, looks upgradeable but odd-ball)
• BIOS controlled case lock. I've been too scared to actually use it.

I am able to saturate the network connection with NFS, FTP and SAMBA. You are going for a Gigabit connection that may be harder to saturate without a RAID or fast SATA drive (Need 100 MB/second). I did find my computer was slow for a few things though:

• "% dd if=/dev/ad6 | bzip2 > 10GbXPimg.bz2" (on /dev/ad4) 13 hours
• Backing up client via ssh was server-CPU bound. Can't find a good measurement in my notes, but it was on the order of 1MB/sec (10% of the Net bandwidth)
• Doing MD5 hashes of large files takes a while as well, but not so slow I wrote down the throughput

So in summary: I don't think you need new hardware, but you didn't tell us your existing CPU

Yes, compression and decompression takes a lot of CPU time. Enough that you need a faster CPU? It is hard to say.
I suspect the real reason you want to upgrade is the Gigabit ethernet. The (32 bit 33Mhz= 133MB/s ) PCI bus does not have enough bandwidth to fill a Gigabit pipe AND pull from the drive at the same time. A 66MHz or 64 bit PCI bus would work though.

Edit: It's GNU/Linux, not just "Linux."

[phaerus]

amazing words

I'm trying to shy away from PCI-X as it's a dying spec and pretty pricey.

I realize also that a bunch of PCI E options conflict with the "low power" aspect as something like an atom board won't have robust expansion options. I meant to say that I'd like something lower power, but only after the "need" categories have been met. Thus if there is no PCIe then it doesn't matter how low or high power something is.

As for the current server... It's an old Gigabyte board with a couple of cut traces (it still works, but it gets finicky with integrated peripherals now) with an Athlon 64, 3700+ running Gentoo Linux. the storage subsystem consists of a primary boot drive, a 4x 1.5TB RAID 5 array and a 5x 500GB RAID 5 array (both in mdadm). I have a gigabit network backbone and the server needs to be able to read from disks, and pipe out 2gbps (two gigabit adapters, or at least full duplex on one link). I also regularly run various file compression or checksum jobs on the server, and sadly anything single core gets bogged down with this (I like a responsive system)

Currently everything is made available through the network via SMB and rsync's are done through SSH. Anything remote to the network is done via SFTP



I'm not very familiar with Wake On LAN, it sounds useful, but I haven't the faintest idea how to use it.


The primary upgrade reason for me is bus. I have a lot of hard drives that will easily saturate the PCI bus, will easily saturate a PCI-X bus, and really only will shine on a PCIe 4x or wider bus. The second upgrade reason is that a single core of anything (even with hyperthreading) will hiccup when doing a video stream through the network while doing something CPU intensive locally.

[phillipsjk]

Eh, I spent yesterday reading RFC's so I could wish for Automatic Wake-on-lan Of Sleeping Computers. Sending a magic packet is inconvenient if you just want the sleeping computer to "wake up" when you need it. Some adapters can wake the computer based on arbitrary frame data in the first 128 octects (layers 2, 3 +)

I didn't realize how simple (and protocol independent) the WOL packet was until I read the Wikepedia entry. In practice, it is not used for servers much (makes them less reliable). In a home environment it might make sense if you can get the power management to behave (and not interfere with your work too much). I think proper ACPI support in GNU/Linux and BSD (and windows) is (was) a sticking point as well.

Edit: forgot to mention: yeah, "unreliable operation"= "not worth your time". I choose the Deskpro because the CPU fan was 80mm (mounted on the case). That was one of my most important criteria. In my experience 40mm (486) fans last 2 years, 80mm (power supply) fans last 10 years. Other sizes are in between.
Edit2: I have no reason to doubt rumors that ball-bearing fans last longer than fans using sleeve bearings.
Edit3: It's GNU/Linux, not just "Linux."

[Carnildo]

amazing words

I'm trying to shy away from PCI-X as it's a dying spec and pretty pricey.

It's "dying" in the sense that you'll have trouble finding parts in a decade, but there's still an incredible amount of hardware out there that uses it. And it's only pricy if you buy new: at various times, I've purchased four-port Ethernet cards for $20, and a eight-port SATA RAID controller for $150. You might need to lurk on Ebay for a few months to find that sort of deal, but they're out there.

For comparison purposes, my storage server specs:
Atom 230
512MB DDR533 RAM
5x 1TB hard drives in RAID-6
A 4GB CompactFlash card as the boot drive.
Onboard everything.

It takes about a minute and a half to boot from off to login, but the slow part is spinning up the hard drives (one every five seconds) and waiting for the RAID card to detect them. I use a 100Mbit backbone for my network (when I replaced my router/switch six months ago, I checked to see if I was doing anything that needed gigabit speeds, and found that I wasn't). This lets me do things like use 64-bit PCI-X cards in 32-bit PCI slots without them being the bottleneck.

In terms of power usage, it draws about 68 watts idle, 70 watts under full load (I haven't figured out if I can spin down the hard drives yet).

As for the current server... It's an old Gigabyte board with a couple of cut traces (it still works, but it gets finicky with integrated peripherals now) with an Athlon 64, 3700+ running Gentoo Linux. the storage subsystem consists of a primary boot drive, a 4x 1.5TB RAID 5 array and a 5x 500GB RAID 5 array (both in mdadm). I have a gigabit network backbone and the server needs to be able to read from disks, and pipe out 2gbps (two gigabit adapters, or at least full duplex on one link). I also regularly run various file compression or checksum jobs on the server, and sadly anything single core gets bogged down with this (I like a responsive system)

The primary upgrade reason for me is bus. I have a lot of hard drives that will easily saturate the PCI bus, will easily saturate a PCI-X bus, and really only will shine on a PCIe 4x or wider bus. The second upgrade reason is that a single core of anything (even with hyperthreading) will hiccup when doing a video stream through the network while doing something CPU intensive locally.

One option for a low-power PCIe system would be to try to find an Atom 330 with a PCIe x16 slot: it's got the two cores you want (although I suspect it's disk access rather than CPU limitations that's causing the hiccups). CPU power might be borderline, though: my Atom 230 averages 10% of the CPU to move files at 100Mbps, so if it scales linearly, it'll take the full power of one core to use your gigabit link.

[phaerus]

amazing words

I'm trying to shy away from PCI-X as it's a dying spec and pretty pricey.

It's "dying" in the sense that you'll have trouble finding parts in a decade, but there's still an incredible amount of hardware out there that uses it. And it's only pricy if you buy new: at various times, I've purchased four-port Ethernet cards for $20, and a eight-port SATA RAID controller for $150. You might need to lurk on Ebay for a few months to find that sort of deal, but they're out there.

I've had so many bad experiences with used hardware... it seems the magic smoke is so much more prone to leaving when I buy from Ebay etc. I'll buy used from people I know where I can be reasonably certain of the history, but overall I don't mind paying for new hardware (even if it's outdated) in exchange for lower risk of abuse from others. But thanks, I didn't realize that PCI-X could be had so cheaply, I'll definitely try to keep my eyes open.

For comparison purposes, my storage server specs:
Atom 230
512MB DDR533 RAM
5x 1TB hard drives in RAID-6
A 4GB CompactFlash card as the boot drive.
Onboard everything.

It takes about a minute and a half to boot from off to login, but the slow part is spinning up the hard drives (one every five seconds) and waiting for the RAID card to detect them. I use a 100Mbit backbone for my network (when I replaced my router/switch six months ago, I checked to see if I was doing anything that needed gigabit speeds, and found that I wasn't). This lets me do things like use 64-bit PCI-X cards in 32-bit PCI slots without them being the bottleneck.

In terms of power usage, it draws about 68 watts idle, 70 watts under full load (I haven't figured out if I can spin down the hard drives yet).

Thanks for the advice. What raid card are you using? Staggered spinups and idled spindowns I'd think would go hand in hand... but failing that, (assuming you're using Linux) you should be able to use hdparm to specify an idle time spindown on the drives. It *shouldn't* interfere with mdadm, just set all the drive parameters the same. I think the syntax is something like

Code: Select all

hdparm -S n /dev/sda


where n is the timeout interval in multiples of 5 seconds

[phillipsjk]

In terms of power usage, it draws about 68 watts idle, 70 watts under full load (I haven't figured out if I can spin down the hard drives yet).

I didn't consider the staggered spin-up RAIDs need. To phaerus: Waiting 90 30 seconds for a page fault may be... awkward.

By "full load", I mean stressing the memory and CPU with prime95. If your CPU is able to go into even a minimal power-saving mode the difference should be more than 2W. Now that I installed the DVD drive, "Full-load" is going to have to include burning a backup disk

One option for a low-power PCIe system would be to try to find an Atom 330 with a PCIe x16 slot: it's got the two cores you want (although I suspect it's disk access rather than CPU limitations that's causing the hiccups). CPU power might be borderline, though: my Atom 230 averages 10% of the CPU to move files at 100Mbps, so if it scales linearly, it'll take the full power of one core to use your gigabit link.

Top under both BSD and GNU/Linux tells me approximately how long the CPU is sitting waiting for the disk to respond:

Code: Select all

Cpu(s): 46.8%us,  9.3%sy,  0.0%ni, 43.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
        User,    System,  Niced, Idle, Waiting for I/O, Hw IRQ,Soft Int., time "stolen" by hypervisor


What does somebody have to do the get a mono-spaced font around here? Edit: nevermind. the edit window uses a proportional font *face palm*

Before looking it up I thought 'wa' referred to hardware interrupts. Looks like it pertains to DMA transfers (where the CPU must wait).
If you are moving data between the disk and net at the same time it may be tricky to figure out where the bottle-neck is (well, not really: look at the throughput). On my desktop, I sometimes see the "wa" field over 10% when doing disk-intensive operations.

Edit: I was going to mention: some DVDs have a data rate high enough to loose information if played across a 100Mbit switch (without transcoding that trashes the quality anyway (I was playing with vlc one day)) Edit: retested with another computer: the drop-outs were due to the target machine, not the network. I think 10Mbps may be enough for DVD video (not tested yet).
Edit: It's GNU/Linux, not just "Linux."

[Carnildo]

Thanks for the advice. What raid card are you using?

The 3Ware 9500S-8. It's one of the two cards I've found (the other being the 9500S-12) that supports all of the following:
* SATA
* 32-bit 5-volt PCI
* 8 or more drives
* Staggered spinup
* Arrays larger than 2TB

Staggered spinups and idled spindowns I'd think would go hand in hand... but failing that, (assuming you're using Linux) you should be able to use hdparm to specify an idle time spindown on the drives. It *shouldn't* interfere with mdadm, just set all the drive parameters the same. I think the syntax is something like

Code: Select all

hdparm -S n /dev/sda


where n is the timeout interval in multiples of 5 seconds

Yes, that should work, but it doesn't. I think the RAID card is getting in the way.

[phillipsjk]

From reading psykx's plans in this thread, I had an epiphany.

If you put /var/log on a SSD (I consider a CF card a SSD), your odds of being able to spin down this disks increase dramatically. At least in FreeBSD, Hard-drive spindowns are logged. You want them logged to make sure you are not burning through too many power-on cycles (you have about 10,000 to play with, IIRC).

I think the Swap space may provide similar problems. You do not want to put a swap partition on a CF card, though some of the more expensive SSDs may have good enough wear-leveling to handle it.

[psykx]

glad I could help!


What exactly do you use your server for? It'll help in choosing parts.

[Meteorswarm]

From reading psykx's plans in this thread, I had an epiphany.

If you put /var/log on a SSD (I consider a CF card a SSD), your odds of being able to spin down this disks increase dramatically. At least in FreeBSD, Hard-drive spindowns are logged. You want them logged to make sure you are not burning through too many power-on cycles (you have about 10,000 to play with, IIRC).

I think the Swap space may provide similar problems. You do not want to put a swap partition on a CF card, though some of the more expensive SSDs may have good enough wear-leveling to handle it.

Other options would be to just buy more ram, and put /var/log on tmpfs, and never swap. This has the downside that if your system goes down, you might lose the logs, however.

Is there a way to put just hard drive spin-down into a log?

[phillipsjk]

glad I could help!


What exactly do you use your server for? It'll help in choosing parts.

It was originally supposed to be a "File and everything else" server, but I decided I need to focus on:

• Storing user files (/home) of clients and disk images of various client computers
• Tested backups so if the disk dies, I don't lose all data in the house.
• Print server for both an inkjet and Laser printer.

For the first task, the heterogeneous network made things difficult. The "family" computer is still running Win98, but I think I am about ready to replace it with Gnu/Linux and Wine. The problem is that Wine installs Windows Programs in ~/.wine : wasting (server) disk space. It also appears to use a Win98 security model, but that can be avoided with Wine bottles.

For the second task, I was originally going to try to use the DVD drive on a client machine. I realized this would create an awkward "chicken and egg" problem upon restoration. I pulled the DVD drive from the "family" computer and replaced it with a CD-R/RW. When I finally created the test back-up DVDs, I discovered that they were corrupted near the end (edge) of the disk. I suspect local vibration, but it could be DVD rot already. Just yesterday I installed ~2.5 inches of semi-rigid packing foam under the machine to test this (need to burn a test DVD). Another concern is that "dump" does not support media verification. I found a solution using forward error correction, but I would have to code a Command-line version of the tool.

For the printers, support for PCL or Postscript is good. "Dumb" printers will be hit-and miss. The current machine is too slow to be faster with a PCL over PostScript, AFAIK. I currently have a printer switch box set up, but am not sure how that works with lpr. I may just get a printer with a USB interface for one of them.

The network is 100Mbps, so a single disk can easily saturate the link. If I upgrade the network to 1GBps, I would seriously consider a RAID. That failed disk is back in the box after the manufacturer utility gave it a clean bill of health (for total of 2 disks). Yet another reason I don't want to use it without verified backups

Would be nice:

• Answering machine/fax (external modem using RS-232)
• "Neutral" server for hosting (original) quake games.
• VOIP Gateway running Asterix
• possibly Host the IPv6 portion of my website in a "Jail"

Again the current machine may be under-powered. Once it is working, I can start on a replacement box, if needed (I've had the current hardware mainly finalized for ~4 years).

Disk layout:

• / (root/boot) 512MB (up from 128MB)
• Swap 1 GB (on each disk, IIRC)
• /var 6 GB (from 5 (includes symlink from /tmp meant to be able to hold a DVD image))
• /usr 6GB (from 5)
• /home 40 GB (from 30, expected to be expandable if needed)
• /srv 60 GB (from 36 GB, expected to be expandable if needed, stores partition images, information for "servers" not stored in /var)

Note: This is not large enough to backup my brother and sister's computers (160 GB drives). I will have to consider something larger than DVDs if I want to try that.

Meteorswarm: most your suggestions would involve new hardware. It is not economical to upgrade the RAM. APM instead of ACPI does not log (but is not recommended). I suspect logging can be disabled, but don't want to. I have also been told that you can log one spin-down event without waking the disk (gets saved on spin-up).

Another issue is I don't think Chron is power-saving aware. You would have to do a powersave-aware version of chron or something if you want to spin down the disks effectively (read: do chron jobs before spinning the drives down, but machine is otherwise idle). It would be similar to anachron (designed for computers intermittently powered on), possibly with some tasks canceled if nothing interesting is happening. Would have to check if anachron can do that with very little modification (wake-up events at specific times might be useful).

[psykx]

whats your budget?

[phillipsjk]

I have not added up the receipts yet, and am scared to find out. Probably at least $50 a year, but less than $200 a year for a total of between $200-$800. I am aware of cheap NAS boxes for less than that. I keep telling myself my server can do more like burn DVD backups, print functions, modem support.

I tried to buy a $200 printer last week, but the box instructed me to return it to the store because the cartridges are defective by design. (They are region-coded like DVDs).

Please read before opening. Opening this package or using the patented cartridge included with this product confirms your acceptance of the following license agreement. The patented Return Program cartridge sold with this product is provided subject to the restriction that it be used only once. Following this initial use, you agree to return the empty cartridge only to Lexmark for remanufacturing and recycling. Lexmark provides a prepaid return label in every replacement cartridge package. If you don't accept these terms, return this unopened package to your point of purchase.