Work Breakages
Today the power went out at the co-location facility where my work store all our production servers. This shouldn't be a problem as there is a seamless transition to diesel powered generators and UPSes on all the boxes. Only the generators didn't start and the UPSes run out after 20 minutes. This caused some degree of panic...
Cue me and a bunch of other guys attempting get a patched up version of the site running in our office for the people who pay us a serious chunk of change for access to us. The complexity of the site's configuration and the amount of data needed to run the site meant that we couldn't get it done before power was restored (almost three hours after the servers went done) and our jobs changed to checking they were booting correctly.
Today was the first time I had to stay late to a fix a problem, and hopefully I won't have to do it again in the near future.
Cue me and a bunch of other guys attempting get a patched up version of the site running in our office for the people who pay us a serious chunk of change for access to us. The complexity of the site's configuration and the amount of data needed to run the site meant that we couldn't get it done before power was restored (almost three hours after the servers went done) and our jobs changed to checking they were booting correctly.
Today was the first time I had to stay late to a fix a problem, and hopefully I won't have to do it again in the near future.