My job
I don't post a lot anymore--I seem to be turning in to an anti-social hermit that all he does is work and play computer games. However, I am enjoying my job. Tech support and limited programming for a small web design company in St. Paul. Specifically dealing with automated remote backups, which we offer for a small monthly fee. It's a growing part of our business with several new clients added each week. We've got approximately 7TB of RAID connected to a few-year-old Dell server running Ubuntu and the backup software.
This company currently consists of four people. The husband-and-wife owners (husband: infrastructure, marketing and some programming; wife: billing, marketing, HR, and design) the main (currently only) web programmer, and myself (support for the automated backup and some limited programming). The owners went out of town this morning, for their yearly vacation. So of course, the backup server crashed this evening. After I'd been out of the apartment all day until 9:45 doing work and reading questions for HS quiz bowl. Megan and I had a Conversation, and when we finished I went to check my email to find that the server had been down for about 20 minutes. I called the other person who was in town and we decided that he could get to the data center faster than I could (as I'm limited to a bus that at this hour only runs hourly). We did get it working--eventually. Following is the email I sent out to everyone, with some redaction for paranoid security and privacy/politeness's sake:
bkup.reliablesites.com - fixed!
For now, at least as far as I can tell.
What seems to have happened:
Adapter X, array X, drive XX failed, or encountered some sort of error (it is now rebuilding)
This caused a kernel panic (I am annoyed/suspicious that single failed drives cause kernel panics)
John Doe rebooted the box, the arrays resolved their journaled transactions and the OS eventually came up.
However! OBS did not start picking up new connections. Also, I was unable to reach the web console for system management.
We were both confused by this. The box was running, I could ssh to it. It wasn't a firewall issue; I could ping remote servers, and obviously could ssh to the machine. The XXXX logs indicated that obs was indeed listening on ports XXXX and XXXX, as it should. The brainstorm came when I thought to check the iptables rules. There weren't any. I don't know why. Somehow, the kernel panic and rebooting the box caused it to forget that it should forward packets on 80 and 443 to XXXX and XXXX respectively. Fortunately, a quick google search turned up the correct iptables commands to forward port 80 to 8080 (a common task on LAMP servers), which I then modified for our particular needs. The server seems to be happy now, accepting quite a few connections. I can also connect to the web admin console. I'm going to keep an eye on it for a bit, then go to bed. I will be in the office tomorrow morning...at some point.
--end of quoted email
The paranoid suspicious part of me suspects the boss of setting up some automated cron script to cause problems that I only vaguely understand and don't exactly know how to fix as soon as they leave to test my adaptiveness. I really don't know anything about iptables, used to control network traffic. I knew we used them rather than the firewall in this particular case. And I had to learn -real- fast. So I did. Kind of. Hopefully everything is working and I can go to sleep.
I feel pretty good about the whole thing, though. Dealing with this sort of thing is why they hired me. I don't feel bad about not working at maximum efficiency when I'm in the office when I might wind up having to deal withs something like this at any time.
This company currently consists of four people. The husband-and-wife owners (husband: infrastructure, marketing and some programming; wife: billing, marketing, HR, and design) the main (currently only) web programmer, and myself (support for the automated backup and some limited programming). The owners went out of town this morning, for their yearly vacation. So of course, the backup server crashed this evening. After I'd been out of the apartment all day until 9:45 doing work and reading questions for HS quiz bowl. Megan and I had a Conversation, and when we finished I went to check my email to find that the server had been down for about 20 minutes. I called the other person who was in town and we decided that he could get to the data center faster than I could (as I'm limited to a bus that at this hour only runs hourly). We did get it working--eventually. Following is the email I sent out to everyone, with some redaction for paranoid security and privacy/politeness's sake:
bkup.reliablesites.com - fixed!
For now, at least as far as I can tell.
What seems to have happened:
Adapter X, array X, drive XX failed, or encountered some sort of error (it is now rebuilding)
This caused a kernel panic (I am annoyed/suspicious that single failed drives cause kernel panics)
John Doe rebooted the box, the arrays resolved their journaled transactions and the OS eventually came up.
However! OBS did not start picking up new connections. Also, I was unable to reach the web console for system management.
We were both confused by this. The box was running, I could ssh to it. It wasn't a firewall issue; I could ping remote servers, and obviously could ssh to the machine. The XXXX logs indicated that obs was indeed listening on ports XXXX and XXXX, as it should. The brainstorm came when I thought to check the iptables rules. There weren't any. I don't know why. Somehow, the kernel panic and rebooting the box caused it to forget that it should forward packets on 80 and 443 to XXXX and XXXX respectively. Fortunately, a quick google search turned up the correct iptables commands to forward port 80 to 8080 (a common task on LAMP servers), which I then modified for our particular needs. The server seems to be happy now, accepting quite a few connections. I can also connect to the web admin console. I'm going to keep an eye on it for a bit, then go to bed. I will be in the office tomorrow morning...at some point.
--end of quoted email
The paranoid suspicious part of me suspects the boss of setting up some automated cron script to cause problems that I only vaguely understand and don't exactly know how to fix as soon as they leave to test my adaptiveness. I really don't know anything about iptables, used to control network traffic. I knew we used them rather than the firewall in this particular case. And I had to learn -real- fast. So I did. Kind of. Hopefully everything is working and I can go to sleep.
I feel pretty good about the whole thing, though. Dealing with this sort of thing is why they hired me. I don't feel bad about not working at maximum efficiency when I'm in the office when I might wind up having to deal withs something like this at any time.