Avoiding pitfalls in designing distributed systems
We had a talk entitled "Avoiding pitfalls in designing distributed systems" this Thursday at work. The speaker, Felipe Cabrera, is one of the most impressive speakers I've heard. The topic was very abstract and I was hoping that it would not turn out to be high level abstract BS. But I was glad I attended. He would bring in examples from projects he had worked on and mistakes he had seen made all the time. And he's worked on implementing TCP/IP in BSD Unix, consulted for Xerox PARC, been in research at IBM, architect at Microsoft, VP at Amazon....
I really liked a few points from his presentation. He spoke of making systems open - open to external change through clean interfaces. Innovation, he said, is more likely to happen outside the system than inside. Another aspect we touched upon was adoption. Adoption can be because of reasons that may be non-technical. The Xerox PARC network protocol stack was much simpler and yet sophisticated than TCP(or UDP) over IP yet the latter is more common because DARPA sponsored its inclusion in Unix and AT&T gave away the Unix source to the varsities and later Microsoft too adopted TCP(UDP)/IP. Adoption should be a critical factor in developing technologies. He explained the importance of decentralization using a very interesting example of the earliest network file systems when client-side caching was first designed to poll the server for changes. Obviously, the server got hosed. And then he compared this with a recent system in a company he had worked at which had similar issues. Another very impressive point he made was about generalizations. Someone asked him about over-generalization and he spent a few minutes talking about early disks - drums and then cylinders and how the read/write heads would work on multiple cylinders simultaneously in the later ones and how layers of abstraction allowed defraggers to be written easily. What sounds like over-generalization today would be a bare necessity tomorrow. The point is how much are you willing to re-invest in making it ready for tomorrow.
The entire talk was interspersed with very interesting and relevant examples and a lot of good humour. I wish we had had teachers like that!
All Things Distributed has a little something about some of the more interesting work happening on complex distributed systems in Amazon. It sounds bloody interesting. I wish I were qualified enough to try my hand at that.
I really liked a few points from his presentation. He spoke of making systems open - open to external change through clean interfaces. Innovation, he said, is more likely to happen outside the system than inside. Another aspect we touched upon was adoption. Adoption can be because of reasons that may be non-technical. The Xerox PARC network protocol stack was much simpler and yet sophisticated than TCP(or UDP) over IP yet the latter is more common because DARPA sponsored its inclusion in Unix and AT&T gave away the Unix source to the varsities and later Microsoft too adopted TCP(UDP)/IP. Adoption should be a critical factor in developing technologies. He explained the importance of decentralization using a very interesting example of the earliest network file systems when client-side caching was first designed to poll the server for changes. Obviously, the server got hosed. And then he compared this with a recent system in a company he had worked at which had similar issues. Another very impressive point he made was about generalizations. Someone asked him about over-generalization and he spent a few minutes talking about early disks - drums and then cylinders and how the read/write heads would work on multiple cylinders simultaneously in the later ones and how layers of abstraction allowed defraggers to be written easily. What sounds like over-generalization today would be a bare necessity tomorrow. The point is how much are you willing to re-invest in making it ready for tomorrow.
The entire talk was interspersed with very interesting and relevant examples and a lot of good humour. I wish we had had teachers like that!
All Things Distributed has a little something about some of the more interesting work happening on complex distributed systems in Amazon. It sounds bloody interesting. I wish I were qualified enough to try my hand at that.