Grid Computers.

[NewKat]

Just wondered what everybody thought...

[iw]

The most I've ever really heard about grid computing are some talks students gave when I worked at Argonne Labs. They consisted mostly of buzzwords, talking about how exciting grid computing is, and explaining what grid computing was without explaining what grid computing is ("grids are scalable, fault tolerant," and so on).

So I have no idea what it is. My impression of grid computing is cluster computing on steroids. To me it seems like one of those computing "revolutions" that no one can actually describe well. I'm reminded of XML and .NET.

[Ended]

They consisted mostly of buzzwords

Like the first sentence of this.

My uni has what I think would be termed a 'grid computing' structure, where it's possible for people who need a lot of power to harness all the different terminals scattered over the uni site. It sounds pretty cool.

But yeah, I'm unsure if I understand the terminology correctly.

[NewKat]

Here is a quote from Catalyst;

"Some of today's large-scale scientific activities - modelling climate change, Earth observation, ... - involve handling millions of bytes of data very rapidly. such activities would not be possible without a new typw of computering. It is not possible for one single institution to store and analyse all this data, so scientists share computer storage and processing power around the world at hundreds of differnet locations. This is called grid computing. much of it is based on vast numbers of PCs linked to process data in parallel with other tasks. In the past, individual supercomputers might have been used but these could only deal with single, although very large, tasks."

Also, see GripPP for the UK's computing grid for handling particle physics data (http://www.gridpp.ac.uk)and CERN wich goes live this year (http://lcg.web.cern.ch/LCG/overview.html) and will produce 15petabytes of data a year. (1 petabyte = 10^15 bytes.)[/url]

[QuantumTroll]

I've heard people say that Grid Computing is the next step for the internet. I believe this is balderdash.

The basic idea behind Grids is that you can basically get as much computing power as you need without having to deal with individual supercomputing centers. Processing power becomes a shared resource among all the members of the Grid network. The word "grid" draws an analogy between power distribution and computation.

Unless everyone stops using desktops and laptops and moves to dumb terminals, Grids will be used for science and business only.

Not all science and business applications are appropriate for running on a Grid. What makes a program suitable? Generally, security must be a low priority, because putting data on the wire to god-knows-where obviously carries risk. For scheduling reasons, it helps if you know how long the program will run. Unless the data is already being distributed (like the CERN Grid and others), your input data and/or results might be in transit longer than the computation would take on a local machine.

All these gripes aside, I think Grid Computing is a tremendously exciting concept. Right now I work for IBM with free access to some BlueGene/L machines, and they're just sitting idle most of the time. It breaks my heart because I know there are researchers and grad students with projects on hold because of insufficient computer power. It would be totally swank if all this computer were hooked up so other people could use it when IBMers don't.

[aoanla]

All these gripes aside, I think Grid Computing is a tremendously exciting concept. Right now I work for IBM with free access to some BlueGene/L machines, and they're just sitting idle most of the time. It breaks my heart because I know there are researchers and grad students with projects on hold because of insufficient computer power. It would be totally swank if all this computer were hooked up so other people could use it when IBMers don't.

Ahh, you should just stick Condor on them then... ¬¬

(I work in Grid Computing, and indeed look after one of the nodes for GridPP.)

[QuantumTroll]

All these gripes aside, I think Grid Computing is a tremendously exciting concept. Right now I work for IBM with free access to some BlueGene/L machines, and they're just sitting idle most of the time. It breaks my heart because I know there are researchers and grad students with projects on hold because of insufficient computer power. It would be totally swank if all this computer were hooked up so other people could use it when IBMers don't.

Ahh, you should just stick Condor on them then... ¬¬

(I work in Grid Computing, and indeed look after one of the nodes for GridPP.)

Heh, I just read your entry on the intro thread. I'm a Computer Scientist pretending to be a Physicist (but failing at the moment), and you're a Physicist turned Computer Scientist. Not surprising our paths have crossed...

Sticking Condor on BlueGene is an excellent idea. One of my coworkers was at a Condor convention of some sort at U Wisconsin just a few weeks ago. I think the goal is to eventually do what you said. Given the unusual nature of BG/L and BG/P, this may not be as easy as just slapping the code in position and typing "make"...

As a physicist looking after a node for GridPP, what does your job entail? Updating the software when new stuff comes out? Swapping out bad nodes? I'm interested in some of these practical aspects of Grids. I've got no idea what it's like working with one.

[zenten]

Isn't Google running all their stuff on grid computers? Although I'm not sure that this is the correct term, since it's basically just a bunch of datacentre's located all over the place, with cheap commercially available computers running in banks within them.

[aoanla]

Ahh, you should just stick Condor on them then... ¬¬

(I work in Grid Computing, and indeed look after one of the nodes for GridPP.)

Heh, I just read your entry on the intro thread. I'm a Computer Scientist pretending to be a Physicist (but failing at the moment), and you're a Physicist turned Computer Scientist. Not surprising our paths have crossed...

If life was a movie, this would be where we either turn out to be archnemeses or form a dynamic duo to fight crime.

Sticking Condor on BlueGene is an excellent idea. One of my coworkers was at a Condor convention of some sort at U Wisconsin just a few weeks ago. I think the goal is to eventually do what you said. Given the unusual nature of BG/L and BG/P, this may not be as easy as just slapping the code in position and typing "make"...

As a physicist looking after a node for GridPP, what does your job entail? Updating the software when new stuff comes out? Swapping out bad nodes? I'm interested in some of these practical aspects of Grids. I've got no idea what it's like working with one.

Updating the software (especially for the grid middleware itself, which is frequently updated, often in ways which unintentionally break stuff) is certainly a major part of it, as is dealing with all the usual sysadmin problems with things breaking and needing maintained. Here at Edinburgh, we're actually in the process of upgrading our presence, so there's also a certain amount of planning to be done (and political stuff which thankfully I have little to do with) as to what we upgrade to.
Perhaps the use of "node" was sloppy as well - Edinburgh, like any of the regional sites for GridPP/EGEE/WLHC (grid people like acronyms and initialisms) provides a number of systems to the grid; a number of "front-end nodes" which provide interfaces for job submission, storage and data-transfer and monitoring/accounting, and pools of compute nodes and storage nodes which actually do the computing and storage here. It's more important to keep the front-end nodes happy, as the compute nodes especially can afford to have a couple die without anyone really noticing elsewhere.

[QuantumTroll]

If life was a movie, this would be where we either turn out to be archnemeses or form a dynamic duo to fight crime.

Heh. Foreshadowing?

Updating the software (especially for the grid middleware itself, which is frequently updated, often in ways which unintentionally break stuff) is certainly a major part of it, as is dealing with all the usual sysadmin problems with things breaking and needing maintained. Here at Edinburgh, we're actually in the process of upgrading our presence, so there's also a certain amount of planning to be done (and political stuff which thankfully I have little to do with) as to what we upgrade to.
Perhaps the use of "node" was sloppy as well - Edinburgh, like any of the regional sites for GridPP/EGEE/WLHC (grid people like acronyms and initialisms) provides a number of systems to the grid; a number of "front-end nodes" which provide interfaces for job submission, storage and data-transfer and monitoring/accounting, and pools of compute nodes and storage nodes which actually do the computing and storage here. It's more important to keep the front-end nodes happy, as the compute nodes especially can afford to have a couple die without anyone really noticing elsewhere.

Aww, it sounds awfully typical for computer work. Pretty much what I expected, but I was hoping for a VR-type interface to the machines, programs manifesting themselves as glowing cubes, which you feed to a virtual maw of computational power, with warning claxons and strobing orange lights announcing the failure of a node. Meh, some other time, perhaps...

I imagine you have some pretty nice connection speeds at work, though.

[wac]

Aww, it sounds awfully typical for computer work. Pretty much what I expected, but I was hoping for a VR-type interface to the machines, programs manifesting themselves as glowing cubes, which you feed to a virtual maw of computational power, with warning claxons and strobing orange lights announcing the failure of a node. Meh, some other time, perhaps...

I imagine you have some pretty nice connection speeds at work, though. :)

If node failure requires claxons and orange lights, odds are high you're probably doing something wrong. If you have an 11 year MTBF and ~8k machines you have to anticipate a couple component failures every day. And if you're thinking defensively, you have to presume that your single point of failure will be where things break. Fault-tolerance is everything.

[adlaiff6]

Aww, it sounds awfully typical for computer work. Pretty much what I expected, but I was hoping for a VR-type interface to the machines, programs manifesting themselves as glowing cubes, which you feed to a virtual maw of computational power, with warning claxons and strobing orange lights announcing the failure of a node. Meh, some other time, perhaps...

And you can walk around the towers that make up the Gibson too!

[kaiden11]

The way new chip architecture trends are going, I think "grid computing," from a programming perspective, is going to become incredibly important. Considering how parallel architectures are developed, marketed, and ultimately used as they are today, many programmers I've come into contact with have no concept of threading, parallelization, or even how their respective operating systems handle their various tasks. I think we're going to start seeing many more grid computing concepts make their way into the common programming vernacular, hopefully alongside tools or languages that will make the arduous and tedious task a little more friendly (and hopefully more foolproof).