Must Control Fist Of Death
My job is doing some major hardware and personnel reconfiguration (read: consolidating into fewer buildings), and every time monitoring and stats host (like the ones I run) get moved, they need to be added back to the control port rules list (because their IPs change). The CP process owner has mostly been good about adding the hosts. That's no problem.
But after adding the hosts, the new rules have to be pushed out to the production complex. This takes two steps: Loading the rules file, then restarting the CP rules process. I used to do this job, and I got bit several times by the "load first, then restart" sequence; if you do them in reverse order or one without the other, the new rules never take effect. I finally wrote a web page with the exact commands on it so I could cut and paste and never screw it up again.
Then I changed jobs and was no longer allowed to update these hosts on my own. Enter new CP process owner, who I know well from previous assignments together. Good guy, understands the importance of host complex stats and is quick to make accessing those stats easier on everybody.
That said, after having recently stood up 2 new hosts in their new locations, I asked for the rules file to be updated and pushed to production. The files got updated, but I discovered after starting my stats jobs on my new hosts that they were locked out of hosts they needed to access. I asked whether the push was complete.
"Oh, it runs by cron. It'll be done sometime tomorrow morning during the normal maint window."
O.K., great.
That was Tuesday. On Friday, I was still having trouble accessing hosts from the new stats collectors, and I had a bad host that had to come out, so I needed those hosts in production ASAP. The CP process owner wasn't in, so for snicks and grins, I looked at his automated scripts to push out the new rules list to get a refresher on the commands needed to push out new rules...
...and it only contained "restart CP_rules". No mention of "load CP_rules_config" anywhere.
#sigh#
So, guess who got to work late yesterday to load CP_rules_config on a few thousand processes last night so that when the cron job ran in the morning, the rules would finally be implemented correctly?
I need a vacation.
But after adding the hosts, the new rules have to be pushed out to the production complex. This takes two steps: Loading the rules file, then restarting the CP rules process. I used to do this job, and I got bit several times by the "load first, then restart" sequence; if you do them in reverse order or one without the other, the new rules never take effect. I finally wrote a web page with the exact commands on it so I could cut and paste and never screw it up again.
Then I changed jobs and was no longer allowed to update these hosts on my own. Enter new CP process owner, who I know well from previous assignments together. Good guy, understands the importance of host complex stats and is quick to make accessing those stats easier on everybody.
That said, after having recently stood up 2 new hosts in their new locations, I asked for the rules file to be updated and pushed to production. The files got updated, but I discovered after starting my stats jobs on my new hosts that they were locked out of hosts they needed to access. I asked whether the push was complete.
"Oh, it runs by cron. It'll be done sometime tomorrow morning during the normal maint window."
O.K., great.
That was Tuesday. On Friday, I was still having trouble accessing hosts from the new stats collectors, and I had a bad host that had to come out, so I needed those hosts in production ASAP. The CP process owner wasn't in, so for snicks and grins, I looked at his automated scripts to push out the new rules list to get a refresher on the commands needed to push out new rules...
...and it only contained "restart CP_rules". No mention of "load CP_rules_config" anywhere.
#sigh#
So, guess who got to work late yesterday to load CP_rules_config on a few thousand processes last night so that when the cron job ran in the morning, the rules would finally be implemented correctly?
I need a vacation.