No Matter Where You Go...

[midiamin]

eng: I have another change to push to production.
me: This fixes the problem you were working on earlier?
eng: Yeah.
me: You checked it into the current release branch?
eng: Yeah.
me: And you tested it?
eng: Yeah.
me: Why didn't this problem show up when you tested the earlier change?
eng: Oh, because I pushed the change into the

Comments

[midiamin]

And, really, I like your statement about trust and I think it
applies out a lot more generally as well.

Indeed. Six months ago, this rant would've been all about IT. Some of
the circumstances would be different, but the sentiment would be the
same. It's quite possible my mechanics have the same gripe about me,
and so on.

Yet, for an awful lot of them, people who do not have that skillset
and knowledge seem to believe that it is "just " and
anyone could do it -- that they know enough about it that they can decide
on their own (again, without the subject matter skills and knowledge) to
change it.

That's the one that I think taarna, fruitylips,
wambold and I run into the most when we're doing release
or release-like things. The other big problem I run into is people
who have done the job before but aren't doing it now. In my
experience, when doing IT for a bunch of engineers most of whom used to
be IT themselves, a bunch of them will understand that, while they have
the skills to do my job, they aren't currently doing it and therefore
don't see the whole picture, and a few of them won't. The latter can be
difficult to please, especially when the thing they're requesting seems
like a very simple thing to implement. Often it is, but there are other
more important issues blocking it, and I really hate explaining my entire
workload and dependency tree every few days to someone I don't report to.
Trust me to do my job as a whole, or trust that my boss will Do The
Right Thing in response to your complaint.

Also, one thing you've not addressed here but seems relevant:
It appears that the connotation of "process" in the USA has come
to include "slow and in the way". Sort of universally, not just in
engineering.

Every process is slow and cumbersome when it doesn't let you do what you
want to do Right Now.

[midiamin]

...especially when the thing they're requesting seems like a very simple thing to implement. Often it is, but there are other more important issues blocking it...

Something I forgot here is a philosophy that's been in the back of my head for the last several years but finally found its way into words yesterday: It's not a question of what I can implement. It's a question of what I can afford to support

I have any number of quick solutions to problems that exist in my domain, but if implementing a quick fix to make a small number of people happy and supporting that quick fix means I don't have time to implement the thing that will make a larger number of people happy, I'm not going to do it. It's that support cost (most usually paid in the currency of time) that many people seem oblivious to.

Release engineering is hard. Let's go shopping!

[felinecanine]

Every process is slow and cumbersome when it doesn't let you do what you want to do Right Now.

The concept of "acceptable risk" from security seems relevant here. There's generally an inverse relationship between the speed and thoroughness of the process. The latter is directly related to the quality of the product, and that's where the acceptable risk comes in. Your process has to be at least through enough to reach the desired level of risk.

I think this is one part where the release engineer needs a fair amount of experience to get things right. That person has to understand how the process affects overall risk. Another part which I've so far neglected to mention is the efficiency of the process (i.e. thoroughness vs. speed).

On another loosely related topic, I find it interesting to compare release practices between hardware (eg. ASIC) engineering and software engineering.
In hardware engineering, the level of acceptable risk is far lower than
that for software engineering because of the enormous cost of a show-stopper bug. While the chip is under development, interim releases follow a relatively quick validation procedure to ensure that things aren't totally
broken, which will impede progress. But as the project nears completion (i.e. tapeout), procedures are strictly followed and any last minute changes
go through several stages of review. Since everyone understands the cost involved, there isn't much cheating that happens. No one wants to be responsible for a mistake that results in a six-figure or more cost.