Sleep? What's That?
Since 21 April 2007, I've been diving headlong into trying to fix a major issue with my statistics collections and processing at my job. It's been one of the most trying things I've ever done. I knew what the problem was (NFS file system we need for processing intermittently going unreachable, leading hosts to crash hard when their process tables became overwhelmed), but trying to get the right person to own up to the issue and fix it was darned near impossible.
Going into "working" hours on Wednesday, I'd put in 58 hours on the problem. That's not a typo. You can do that math and determine that since discovering the problem late Saturday night, I had worked all day Sunday, Monday, and about half of Tuesday. I took about 4 hours for a doctor visit on Tuesday. The rest of the time was sleep, or lack thereof. I was so punch-drunk at one point that I was literally falling asleep between keystrokes. That's not a joke. I'd be typing a command and wake up and find a screen full of "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee" or other such crap.
I finally thought I'd fixed the problem on Wednesday. After staying up for a meeting w/ my boss via phone, I went to bed (around 1600). I woke up to watch Idol Gives Back at 2000 (8 PM) and went to bed around 2300 (11 PM), thinking that tomorrow morning I would awaken and everything would be back to normal.
Wrong.
Pager went off and declared that once more, the processing hadn't completed. I discovered that yet again, one of my hosts did not restart crond on reboot and thus hadn't done the necessary work to allow the other jobs to run to completion. In addition, as I discovered, I'd apparently fallen asleep during keystrokes and put a bunch of stuff and nonsense into a startup file. Joy. I fixed both of those and collapsed, literally, on the bed for the next five hours.
When I awoke, the processing was finally finishing. Yay.
Maybe tomorrow will be better.
Right now, I'm too tired to care.
Going into "working" hours on Wednesday, I'd put in 58 hours on the problem. That's not a typo. You can do that math and determine that since discovering the problem late Saturday night, I had worked all day Sunday, Monday, and about half of Tuesday. I took about 4 hours for a doctor visit on Tuesday. The rest of the time was sleep, or lack thereof. I was so punch-drunk at one point that I was literally falling asleep between keystrokes. That's not a joke. I'd be typing a command and wake up and find a screen full of "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee" or other such crap.
I finally thought I'd fixed the problem on Wednesday. After staying up for a meeting w/ my boss via phone, I went to bed (around 1600). I woke up to watch Idol Gives Back at 2000 (8 PM) and went to bed around 2300 (11 PM), thinking that tomorrow morning I would awaken and everything would be back to normal.
Wrong.
Pager went off and declared that once more, the processing hadn't completed. I discovered that yet again, one of my hosts did not restart crond on reboot and thus hadn't done the necessary work to allow the other jobs to run to completion. In addition, as I discovered, I'd apparently fallen asleep during keystrokes and put a bunch of stuff and nonsense into a startup file. Joy. I fixed both of those and collapsed, literally, on the bed for the next five hours.
When I awoke, the processing was finally finishing. Yay.
Maybe tomorrow will be better.
Right now, I'm too tired to care.