Unconventional ramdisk setups

[deathinlonging]

So I've been thinking about building a computer for a few days, and as a result I came up with a few design ideas... and then some crazy ones I probably could never do myself, but figured I'd post.

In the process of putting together a rough idea of a computer I hit on ram disks, and decided I liked them. A lot. They're really, really fast; they sound like just the sort of thing that you'd want to run a virtual machine on. Unfortunately they have a few downsides, like the fact that one large enough to run a VM on would use up a significant portion of your available ram and that the software for emulating a hard drive in ram also consumes ram, and processor resources. There are a few hardware ram drives out there, but the newest and best one I know of only takes DDR2 ram and is accessed through a single SATA II port.

There's no reason I couldn't just use a software ram disk; in a few months there will probably be non-server 16gb ram sticks that run at a decent speed. However, my curiosity has been aroused. Surely there exists a better way. Perhaps a person could build their own sort of hardware ram disk. Buy a small, cheap barebones system. Install a processor capable of using DDR3 1066 at full speed, with two memory channels, and add two 8gb sticks of DDR3 1066. Put in a cheap HDD, and install a lightweight linux distribution on it. Install ram disk software, and allocate ram to the ram disk, leaving just enough for the OS and the ram disk software. Unfortunately, I then hit a wall. How do you make an internal hard drive accessible to another computer? Surely there is a way.

Making an internal hard drive accessible to another system is critical to the idea of homemade external ram disks. Unfortunately, I've so far been unable to find a way. I'm considering two possibilities. If there are others I'm missing, please speak up.
One: Some way to output over SATA ports. I don't know if this is possible, with the default SATA drivers or with custom software. If it isn't, oh well. If it is, we're still limited to the 6gb/s of SATA III, which is slower than the speeds a ram disk is capable of. If the system is implemented this way, then the best set up would be multiple barebones computer ram drives hooked up to a hardware RAID controller configured for RAID 0. The card would be connected to the system that needed to access the ram disks.
Two: Some way to make a hard drive accessible over PCIe. I know that it's possible to access an emulated hard drive over PCIe; RAID controller cards do it all the time, including those in the PCIe SSDs. However, I don't know if there is software out there that can make a hard drive accessible over PCIe, or if there isn't, how hard it would be to write some (borrowing as much as possible from other sources). If the system was implemented this way, the best (and most expensive) solution would be to build a single larger computer to act as a ram drive, with a processor with four channels. Install four sticks of 8gb ram for a desktop processor and motherboard, or if the processor is an xeon or some other server processor that caps out at DDR3 1066, four sticks of 16gb (Kingston makes 16gb DDR3 1066 ECC Registered ram).

I don't know if these things are possible, but I'm interested in learning. I could be missing something obvious; I am by no means an expert at any of this. However, I figured I'd post this idea and see if others had any useful or interesting input.

EDIT: Some reference numbers from various places (Wikipedia, IBM Redbooks, et cetera)
16 lane PCIe, each way
v1.x: 4 GB/s (40 GT/s) | I *think* that's 1024x4 MB, but the standards are rather hard to be sure about. However, after comparing the Wikipedia numbers to the Redbook numbers for the original PCIe, I'm fairly sure it is indeed 1024 rather than 1000. On the other hand, I'm not even sure of how the Redbooks were handling the numbers, so I'm going to think on this for a bit. The Wikipedia numbers don't actually match the Redbook numbers for the same thing... Agh. Ah, Wikipedia is assuming Gigabytes as 1000 Megabytes. The Redbooks, on the other hand, aren't assuming any sort of connection with reality or any standard I've heard of. E.g.: PCIe, Lane width x1, Throughput (duplex, bits): 5 Gbps | Throughput (duplex, bytes): 400 MBps. Of course, this could all be my fault and I'm having a brain failure, but I don't think so. Someone help me out here.
v2.x: 8 GB/s (80 GT/s)
v3.0: 16 GB/s (128 GT/s) |
Ethernet:
GbE: 1 Gigabit per second, each way, or 125 MB (Megabytes)
10 GbE: 10 Gigabits per second, each way, or 1250 MB (Megabytes) per second
SATA:
1.5 Gigabits per second, 1500 Megabits per second, 187.5 Megabytes per second
3 Gigabits per second, 375 Megabytes per second
6 Gigabits per second, 750 Megabytes per second

[Link]

I was just about to call this close to impossible, but then I found this article. It looks like there are some people working on using PCIe as an interconnection method for clustering, but I still doubt it's easy to get anything like that up-and-running at home for a reasonable price.

An easier and much less unorthodox method would be to simply use an Ethernet connection. From what I understand, 10 Gbit Ethernet is quite readily available, and 40/100 Gbit is the current state-of-the-art. There should be plenty of resources on interconnecting Linux boxen over Ethernet floating around the web. From there on, AFAIK, you can just use NFS to export a RAM filesystem.

[deathinlonging]

Aha! The idea of Ethernet over PCIe sounds like it could be a lot of fun. The fact that people are indeed working on using PCIe as a computer-to-computer communication method makes me very happy. However, for my purposes (and the purposes of anyone else who wants to try this), 10 gigabit Ethernet looks like the way to go.

Strangely, a few hours after I posted this I ended up reading about the Ethernet physical layers and the standard itself. I saw that a standard for 40 Gigabit Ethernet existed, but didn't realize that unlike the 100 Gigabit Ethernet, it was actually feasible to acquire and use. Your post made me stop and thing "Wait, can I buy 40 Gigabit Ethernet hardware?", and after checking Newegg, the answer is yes. There are cheaper single-port 40 Gigabit NICs (although 100 Gigabit are harder to find, and if I built a system with them it would cost nearly as much as all the other components. There are cheaper devices, but they're of questionable effectiveness... like the 40 gigabit card that uses an 8x PCIe 2.1 interface). I then started to check for RAID controller cards that can handle ethernet, until it occurred to me: A mid-range processor and two sticks of ram should be able to cap the 10 Gigabit Ethernet. Two of those would cap even 16x PCIe 3.0, which would make having a RAID controller basically pointless. I'd be better off with a single box and a single 10 Gigabit line, or software RAID (which would be suboptimal at best). Better yet, with two 10 Gigabit ports, I could then set up the ram disks with LVM, which is both capable of striping them but also perfect for backing up and imaging the ram disks.

This is all AWESOME, especially because it looks like it's not only a fun idea, but something I could actually implement. However, I'm a bit confused. Am I misunderstanding something, or is a 10 Gigabit Ethernet NIC that has a PCIe 2.0 x8 interface not actually going to be able to make full use of the Ethernet?

EDIT: Gah. I was a little suspicious of whether it was possible to do LVM on top of NFS drives. After poking around on wikipedia, and noticing that NFS works at the file level, I was pretty sure it didn't. Then I googled around some... yea, you can't do LVM on top of NFS drives. It'd still be the fastest storage device in the world, but its capabilities outside of pure storage would be limited. Is there any way to make a drive block-level accessible over Ethernet?
EDIT Again: GAH! So it looks like what I want is a SAN. This is getting messy quickly. Oh well, just means more fun.

[GeorgeH]

This looks like a fun project, but keep in mind that latency is a huge factor in how fast storage (be it cache, RAM, or whatever) is in the real world. I'd be surprised if you could do significantly better than a SSD using ethernet, regardless of bandwidth.

[deathinlonging]

I hadn't even thought about latency.. but after poking around, it looks like 10 Gigabit Ethernet latency is way below that of SATA, or even below that of the internal functions on an SSD, so that shouldn't be a bottleneck. Now, because I keep mixing up Gigabit and Gigabyte, especially when written as Gb and GB (usually only catching it when I pull up a document that actually writes it out and I notice what at first appear to be inconsistencies), I'm going to go put some reference numbers up on my original post. Also, this way I can hopefully pull my number of open tabs back below 40.

[mosc]

ethernet has WAAAY too much overhead to do what you want to do. SAN devices these days typically run fiber and use larger protocols than exchanging packets. Build yourself a SAN. It's somewhat expensive but not totally unreasonable. A couple of decent fiber controllers, some linux work, and you should be in business.

Then again, if you can put an array of disks in your actual computer rather than housing them separately, that would be better in most regards. SAN's are best used when you have disk shared by many servers. One of the main advantages of this is AVOIDING Ethernet.

[Kromix]

What about using USB3.0 Flash RAID


http://itexpertvoice.com/home/usb-flash ... windows-7/

[mosc]

USB3.0 is not a protocol that lends itself well to high throughput. Also, most motherboards are only using 1x PCIe lane, which cuts down the maximum bandwidth. I would imagine a single good SSD hooked up on a SATA port would be faster overall than anything hooked up to a USB 3.0 port.

[Pinky's Brain]

I have thought about this ... there are basically two options, Infiniband and SAS with SCST. You can buy controllers for both at pretty reasonable prices.

The problem is getting the latency down low enough to beat a good SSD on read performance to make it worth it, SSDs are getting pretty good ... SAS+SCST is the easiest, but I have my doubts about the latency achievable. Ideally you'd use Infiniband and write your own low latency network filesystem drivers for windows and linux

[Kromix]

USB3.0 is not a protocol that lends itself well to high throughput. Also, most motherboards are only using 1x PCIe lane, which cuts down the maximum bandwidth. I would imagine a single good SSD hooked up on a SATA port would be faster overall than anything hooked up to a USB 3.0 port.

my MoBo's USB 3.0 are directly controlled on the MoBo, no PCIe lane used. , 2xEtron EJ168

[PhoenixEnigma]

USB3.0 is not a protocol that lends itself well to high throughput. Also, most motherboards are only using 1x PCIe lane, which cuts down the maximum bandwidth. I would imagine a single good SSD hooked up on a SATA port would be faster overall than anything hooked up to a USB 3.0 port.

my MoBo's USB 3.0 are directly controlled on the MoBo, no PCIe lane used. , 2xEtron EJ168

No, they're still using PCI-E lanes to connect to the host system - 1x PCI-E 2.0 for the pair of them, actually, according to the product sheet (PDF). There's other ways to connect to a PCI-E controller than though a pluggable slot, you know.

Right now, only AMD has native USB3.0 on anything, and that is only some of the higher end Hudson chipsets.

That aside, your intended workload is going to come in to play when determining if latency matters.

Also, don't discredit PCI-E based SSDs - something like a RevoDrive 3 X2 will provide more sequential performance than you could fit over a single 10GbE link, likely substantially better latency,(being local), plus it's more practical, though at this point I'm not sure that's a concern