The next thing to realize is that this platform never really had a chance of making any money for the organization. They do a little accounting trick (read: lying) which I'll talk about in another post that makes it seem like they've had huge wins, but really this is just many times more expensive than our previous operational model.
The deal is that we pretend the whole team is doing something or other, and we stay within budget because the organization can't afford to spend infinite money on this social fiction. However, the budget for our database costs was being drastically overrun. I'm not sure what the original estimate was, but I think it was intended to cost something like 200K for a year of operations, but we were now close to a million dollars.
Some quick facts:
We use Snowflake as our database, which charges you based on the size of the computer you use to run your queries.
You only pay for computers while they're on.
We probably run a few thousand queries per week, mostly developers experimenting with little tweaks for PowerBI reports that no one reads, and on average they take about 2 seconds to run.
The computers are set to idle for 10 minutes after every query.
I noticed this about a month into joining the team, and suggested we uh... don't have the computers run for like two orders of magnitude longer than they need to for every query. I literally can't remember what was said, there was some Agile bullshit about doing a discovery piece, then it just never happened.
IV. Just Doing The Fucking Thing
Anyway, months later, they finally give me a card that says "Discovery: Optimise Costs". Now I have to optimize costs so that I have something to say at the next standup, and fortunately I know just the thing! I'll test my hypothesis that this is all a sick joke, and I'm going to push the button that I secretly think should obviously have been pushed.
We've got a new guy on another team who seems excellent, so I ask management if I can give him admin credentials since we need competent people. They say no. I flick him some lower-level database credentials that I technically wasn't told not to do since they aren't admin credentials, and he sanity-checks that it would save money. At 4PM on the last day of the week, I ping a chat full of good engineers and no managers to make sure I'm not about to nuke everything, then just do it.
I return to work the following Monday. I suspected that this would save a bunch of money, and guess what, our projected bill dropped from a million to half a million dollars, and everyone is losing their fucking minds.
My team has spun this as a huge cost saving, when really we just applied a fire extinguisher to the pile of money that we had set alight.
Other teams are attacking my team, insisting that it can't be a coincidence that the one new guy joined exactly as we did this, and how was it possible we didn't know how to generate that kind of saving without his help? They are saying this because it makes them seem higher status and their teams only produce money in the land where you lie all day, but it is a fair question.
While my managers are very happy, they quietly suggest it may be unwise to roll out the changes to all the computers (I only did a few to be safe) because it would oversaturate the department to hear about us all day. And invite unwelcome questions. The subtext is that if we do this all slowly enough, it might seem like it took a lot of effort instead of just clicking buttons that I said had to be clicked almost a year ago.
I am asked to write some PowerPoints, which include phrases like "a careful statistical analysis of user usage patterns indicated an opportunity to more effectively allocate resources", implying that nothing was wrong, we just needed to collect more data before deciding not to let the expensive machines idle all day.
Every day, I dread someone asking me to explain what the change was, because I will have to fucking yeet some managers I like under a bus, but they can't resist talking about the change non-stop because it is the closest some of them will ever get to impacting the bottom line. And many of them are actually decent managers, it's just that this whole department, like many departments, is some sort of weird political PsyOp to get executives promoted. It's cosplaying as a real business and the board thinks the costume is convincing.
VI. The Aftermaths and Takeaways
By identifying a handful of good engineers and going totally rogue, we outperformed the entire department pretty effortlessly. The competent people are there, just made totally impotent by the organization, and I'm still convinced that this place is probably better than the median organization.
I ask management for a 30K raise after saving 500K and my message is still unread. I suspect I will eventually receive either nothing or 5K.
I have even more meetings now because everyone wants to talk about how we saved the money. I had to make a PowerPoint. Kill me.
I would have been better off not doing anything. Let that be a lesson to you. Do you hear me? I applied myself for five minutes against my own better judgement, had the greatest success of my career, and have immediately been punished for it. Learn from my mistakes, I beg of you.
PS: While this is doing immense traffic, any similar stories being sent to ludicity.hackernews@gmail.com would be greatly appreciated so that I feel less alone in this madhouse.
II. The Budget
The next thing to realize is that this platform never really had a chance of making any money for the organization. They do a little accounting trick (read: lying) which I'll talk about in another post that makes it seem like they've had huge wins, but really this is just many times more expensive than our previous operational model.
The deal is that we pretend the whole team is doing something or other, and we stay within budget because the organization can't afford to spend infinite money on this social fiction. However, the budget for our database costs was being drastically overrun. I'm not sure what the original estimate was, but I think it was intended to cost something like 200K for a year of operations, but we were now close to a million dollars.
Some quick facts:
I noticed this about a month into joining the team, and suggested we uh... don't have the computers run for like two orders of magnitude longer than they need to for every query. I literally can't remember what was said, there was some Agile bullshit about doing a discovery piece, then it just never happened.
IV. Just Doing The Fucking Thing
Anyway, months later, they finally give me a card that says "Discovery: Optimise Costs". Now I have to optimize costs so that I have something to say at the next standup, and fortunately I know just the thing! I'll test my hypothesis that this is all a sick joke, and I'm going to push the button that I secretly think should obviously have been pushed.
We've got a new guy on another team who seems excellent, so I ask management if I can give him admin credentials since we need competent people. They say no. I flick him some lower-level database credentials that I technically wasn't told not to do since they aren't admin credentials, and he sanity-checks that it would save money. At 4PM on the last day of the week, I ping a chat full of good engineers and no managers to make sure I'm not about to nuke everything, then just do it.
Reply
V. Chaos Reigns
I return to work the following Monday. I suspected that this would save a bunch of money, and guess what, our projected bill dropped from a million to half a million dollars, and everyone is losing their fucking minds.
My team has spun this as a huge cost saving, when really we just applied a fire extinguisher to the pile of money that we had set alight.
Other teams are attacking my team, insisting that it can't be a coincidence that the one new guy joined exactly as we did this, and how was it possible we didn't know how to generate that kind of saving without his help? They are saying this because it makes them seem higher status and their teams only produce money in the land where you lie all day, but it is a fair question.
While my managers are very happy, they quietly suggest it may be unwise to roll out the changes to all the computers (I only did a few to be safe) because it would oversaturate the department to hear about us all day. And invite unwelcome questions. The subtext is that if we do this all slowly enough, it might seem like it took a lot of effort instead of just clicking buttons that I said had to be clicked almost a year ago.
I am asked to write some PowerPoints, which include phrases like "a careful statistical analysis of user usage patterns indicated an opportunity to more effectively allocate resources", implying that nothing was wrong, we just needed to collect more data before deciding not to let the expensive machines idle all day.
Every day, I dread someone asking me to explain what the change was, because I will have to fucking yeet some managers I like under a bus, but they can't resist talking about the change non-stop because it is the closest some of them will ever get to impacting the bottom line. And many of them are actually decent managers, it's just that this whole department, like many departments, is some sort of weird political PsyOp to get executives promoted. It's cosplaying as a real business and the board thinks the costume is convincing.
VI. The Aftermaths and Takeaways
By identifying a handful of good engineers and going totally rogue, we outperformed the entire department pretty effortlessly. The competent people are there, just made totally impotent by the organization, and I'm still convinced that this place is probably better than the median organization.
I ask management for a 30K raise after saving 500K and my message is still unread. I suspect I will eventually receive either nothing or 5K.
I have even more meetings now because everyone wants to talk about how we saved the money. I had to make a PowerPoint. Kill me.
I would have been better off not doing anything. Let that be a lesson to you. Do you hear me? I applied myself for five minutes against my own better judgement, had the greatest success of my career, and have immediately been punished for it. Learn from my mistakes, I beg of you.
PS: While this is doing immense traffic, any similar stories being sent to ludicity.hackernews@gmail.com would be greatly appreciated so that I feel less alone in this madhouse.
Reply
Leave a comment