Your bi-annual current affairs post

[hirez]

I haven't looked too hard at the online meejah hoopla over the Megaupload business, but my instant reaction was 'Oh fuck it's Kimble again' followed by 'They really were taking the piss, weren't they

Comments

[steer]

Heh... I don't think you've quite grasped the scale of the clickhosts here. This isn't a batshit cargo cult -- these things are coming to be a dominant traffic source -- no joke. rapidshare, megaupload and the like are actually getting up there with torrents for their traffic share. (Torrent traffic is up a a number but down as a share is down because everything is moving to streaming video right now).

Almost by stealth these things are a massive part of the network right now.

Sure, they know it's mostly warez, pr0n etc... (Although these days why anyone would get their pr0n like that is confusing). Weird though how it's almost in "stealth" that they got there -- I mean everyone who's a geek knows about torrents, magnets, streaming as sources of traffic. Probably if you know a bit you know about CDNs and the like. But most people are only vaguely aware of this ecosystem and it's huge. I'm supposed to know this stuff and I can't name more than three without looking.

[hirez]

Oh, I'm not surprised at the scale. It's the functional equivalent of the Big Yellow hangars that live at the edge of every medium-sized town.

(Hell, we were driving up through northen Wisconsin, which is empty apart from pine trees and lakes, the other year. I'd not seen a building that wasn't straight out of the Rural Ruin community here for a half hour, yet there was a clearing with an access-road and a self-storage facility at the end of it. Barking.)

The last time I looked, it transpired that you could buy mongo-fuck-off boxes of disk that were torrent proxy/accelerators for ISPs. It is all about Somewhere For Your Stuff.

(Business plan: buy up one of those Big Yellow places. Fill it with cheap disks. Profit!)

[steer]

Oh yeah -- just found the reference I was looking for -- rapidshare generates more traffic than youtube. I think that's bloody amazing really.

The last time I looked, it transpired that you could buy mongo-fuck-off boxes of disk that were torrent proxy/accelerators for ISPs.Not sure what you're talking of here -- ISPs can jiggle around with where torrents get stuff from but I don't think it's much used in the wild -- going through standardisation stuff right now. ALTO/P4P -- the idea is the ISP says "don't get your torrent from that client there, get it from this one here, he's dead fast" and you do so, and hence ISP saves money from international transit costs and client gets faster traffic. However, ISPs do not (AFAIK) do this yet

[hirez]

It was a while ago, when bittorrent was going to destroy the internet. It may have been vapourware, or a bright idea that turned out to be a waste of money when bandwidth got cheaper somewhere else.

In the other window I am reading up about what's new with QoS. I've not paid attention to that since 1998. I am assuming that the likes of Virgin traffic-shape like bastards.

CDNs are a bit different, since you've to optimise yr site a little to reference the C that's being Ded by the N. (Yes I know there are various magic Nginx proxy/cache tricks too)

Tangentially, there was a quite head-warping article referenced in BLDGBLOG.. Ha! Here we are: http://www.information-age.com/channels/comms-and-networking/company-analysis/1660458/mining-dark-fibre.thtml (Making HFT go more better by finding redundant fibre runs and exploiting the speed of light. Yes I am going to use this in a story.)

[steer]

It may have been vapourware, or a bright idea that turned out to be a waste of money when bandwidth got cheaper somewhere else.There are many many cool things proposed to help bittorrent... locality is the usual suggestion -- bittorrent does not destroy internet if traffic remains within ISP network and you connect next door rather than to new zealand. Best research I ever saw was some guys who hacked the DHT controlling the swarm to block some requests and send in others and ended up optimising it by stealth. Some users worked out something weird was going on "hey, how come all my peers are local not from miles away" but they weren't properly spotted until it was cached

[hirez]

Oh. Do you have doc or weblog about this sort of thing? I are out of touch these days.

[steer]

I really ought to start doing such things but I patch together my knowledge as I come across it. Problem is that even when you research the interwebs full time it's still a fast moving target.