Large databases, then and now

[pozorvlak]

Back in 1998, I had a job at JET, the Joint European Torus, one of the world centres of research into nuclear fusion. The complex sprawled over most of the former RAF base at Culham, but the heart was the Machine: a huge torus of metal, machined to precisions of thousandths of an inch, which during experiments contained a few milligrammes of heavy

Comments

[totherme]

I wonder if elvum can give us similar estimates for the LHC? ;)

I didn't know dreamstothesky had an LJ - I like the photos in his latest entry...

[pozorvlak]

He doesn't post to it much, but he has one, yes. And they are great pictures - I'd forgotten about that post!

The LHC won't come on-line until November, so I doubt elvum would have too much to say about it (or did he work on it when he was there?), but I'd be very interested to hear about data-processing at CERN... incidentally, I see that the LHC was calibrated using a distrbuted computing system, LHC@Home...

[totherme]

Yeah, a small but perfectly formed bit of it is of elvum's design. A bit of one of the detectors, I think. A blue bit.

Apparently one of the interesting things about it was figuring out a way to make a magnetic field take one value in one place, and a value ten thousand times greater in a place 50cm away from the first...

[elvum]

I've never seen a description of the LHC data-processing architecture written for the non-physics expert, but this document may be of interest - it details the processing and storage requirements anticipated for the smallest of the LHC experiments (LHCb) in the first year of LHC collisions.

[elvum]

Big. To take one of the four main experiments as an example, LHCb has an event size of ~35kB and an event rate of 40MHz, which implies a raw data rate of 1.3TB per second. To avoid the ludicrous impracticality of storing this much data, only 1 in every 20,000 events is actually stored - a series of steps of increasing sophistication and computational complexity analyses each event in turn and discards those thought unlikely to contain "interesting" things (a rare kind of decay, for example). So that leaves the problem of storing a mere 70MB/s - running the accelerator for about 1/3 of each year, that works out as a mere 700TB per annum.

Incidentally, pozorvlak may wish to look at CASTOR for details of a somewhat more modern hierarchical storage system. :-)

[elvum]

(Yes this does mean that the smallest of the four LHC experiments will store about as much data in three hours and forty minutes as JET did in twenty years. That's progress for you...)

[totherme]

It'll be interesting to see how long it takes for storage to get cheap enough that they start turning off those filters...