recipe for disaster

[_disdain_]

Well, maybe not disaster, but certainly a bit of a headache:

1. a server script is set to run every day at 8pm
2. each time the script runs, it creates a plaintext log file, which is recycled daily
3. one of the script runs hangs, keeping the log file open
4. subsequent runs of the script add to the log file, unable to delete it
5. the log file grows to 3.6GB, consuming the available RAM on the server
6. the server starts paging to disk, while trying to keep the enormous log file open, so goes into a loop of disk thrashing which pegs the CPU usage at 100%

I just got off the phone after dealing with this wonderful little set of circumstances. This is a very simple script which has been running for over a year without issue, but still I think job #1 tomorrow will be to put some error checking into it.