I admit it; I’m a fan of auto racing. I love how the teams work together, with practiced precision, to give their driver every opportunity to use her wits and skills to compete against equally formidable opponents. Engineering. Mechanics. Teamwork. Racing against the clock. Outwitting opponents on the track and in the pits.
My favorite racing is Formula 1TM. Open wheeled cars mere inches apart at over 200 miles per hour is no place for the faint of heart. The slightest mistake both on or off the track can mean disaster or worse.
Have new methodologies like DevOps rendered change management obsolete?
In the world of IT service management (ITSM), the stakes have generally been much more pedestrian. But as the need for speed increases in the businesses we serve, we could probably learn a thing or two from the elite crews of automotive racing.
If Formula 1 Adopted IT Change Management
I’ve been involved with IT change management, in one form or another, for most of my professional career. I’ve been a change manager and attended more CAB meetings than I care to count. I’ve heard every objection and seen every trick in the book to avoid CAB. I may have even contributed a few myself back in the day. But as IT systems have gotten increasingly complex, the negative impact of badly managed changes have grown exponentially, sometimes with catastrophic results.
Unfortunately, the response has often been a variation on the theme ”if a little is good, a lot is better.” More rigid change policies evolved into often heavy-handed threats of disciplinary action to be taken against change management refuseniks. Along the way, change management amassed unilateral powers wielded in the form of CAB with singular authority to approve or deny changes.
What was the scope of these change juggernauts? The more we realized that any change could impact critical IT services, all changes fell under CAB review.
Let’s play a quick word association game. I say “change management,” and you say, “slow and bureaucratic.” I say “CAB,” you say, ”marathon meeting so painful, I’d rather have a root canal.”
Wow, you’re good at this.
One more time. I say “RFC,” and you say “Really!? (I have to) Face CAB?”
Rockin’ CAB, Old Skool
If you go way back to the early days of computing—pre-smartphone, pre-PC, heck, pre-FAX machine—you find a fairly monolithic mainframe computing environment, where highly skilled developers were intimately familiar with every aspect of the hardware over which their programs commanded control. The architecture was well understood, and pretty much everything was under the control of developers. If changes didn’t work as expected, developers were often able to fix the error on the spot, and users would be none the wiser.
As time went on, help desks emerged as the point of contact for users who experienced problems. Often, the first the help desk became aware of changes was when angry users called to complain. One of the bigger challenges was that of communication. The logical (though culturally radical idea at the time) solution was for development and operations to get together for weekly change meetings.
Development lifecycles were long—often six months, a year, or more—so getting together once a week to review upcoming changes was no big deal.
The basic concept of a weekly change meeting continues to this day in many organizations. It’s the foundation for many effective change management programs. The good news is it can do much to prevent unnecessary business impact. Getting smart people together to discuss change implementations can help an organization avoid a lot of problems.
From a capability maturity standpoint, that’s both good and bad. Good, in that it’s improving service delivery and avoiding negative business impact. Bad, in that it can only do so much to ensure desired business outcomes are achieved. Bad too, that staff form a view of change management that it’s little more than a quality inspection point at the end of the development cycle, a last check before implementing.
If Old Skool Change Management Ran Formula 1
Imagine, if you will, a world in which car racing adopted traditional old skool change management to manage changes. I think you’d agree that the slightest mistake in racing would be catastrophic, even life threatening. It’s unthinkable to allow that level of risk without proper change management oversight, right?
Given the level of risk, let’s establish a change policy that states all changes must come to CAB before implementation. Unauthorized changes will not be tolerated, and crew members who perform unauthorized changes will be immediately relieved of their duties and suspended, no exceptions. This isn’t a game, folks.
Next, we need a change manager. It must be someone who is fairly knowledgeable, but without undue ties with the individual disciplines represented on the CAB. Let’s make the crew chief the change manager.
CAB will be comprised of the heads of all the race disciplines—race strategy, fuel, tires, engine, suspension, chassis, driver welfare. These are the best of the best, a finer team of professionals cannot be found.
We then need to establish CAB meeting frequency. Because of the nature of the race, it seems to make sense that changes would be reviewed on the same frequency as pit stops. That way, changes can be implemented in a systematic and orderly way during the normal pit. Think of it like a change window. All changes will be thoroughly reviewed and flawlessly executed, each pit a well vetted release. This is starting to sound like a really great way to improve performance; pity those teams who just implement changes as needed.
The crew has been fully briefed—no unauthorized changes. All changes must be reviewed by CAB and approved by the change manager. As soon as the car gets out on the track, the driver is making unauthorized changes right and left.
The crew chief asks what he thinks he’s doing out there, making unauthorized changes. “Driving the car,” is the family-friendly version of his response. “We’re getting unauthorized changes to throttle position sensor (aka accelerator), multiple unauthorized gear changes, and untold numbers of unauthorized braking.”
I think you see where I’m going. Can you imagine a Formula 1 race team operating like the typical change management program? If you follow the logic that all changes have risk, with significant likelihood of negative impact, then you’d have to agree that race teams would have to review all changes.
But that’s exactly how many organizations’ change management capability runs. And, just like you reading this, everyone knows just how silly it is. And yet it’s more common than you may think.
If race cars ran at a few miles an hour, then you could possibly review all changes in a CAB-like fashion. But, in racing, like in business, speed is of the essence.
The reality is, this kind of change management simply doesn’t support the rate of change required by many businesses today. They just don’t have time for unnecessary drag, for time wasted in bureaucracy and delays. Change management has to match the rate of change required by the business it serves. Change must be managed, risk must be evaluated, and outcomes must be assured. But, really, does every change need to be formally reviewed in a CAB-like meeting?
Change Management at the Pace of Business
Ever heard the phrase built for speed? Change management, in business as well as automotive racing, must be built to meet the needs of the business. Not the other way around. Let’s take a look at some of the “unauthorized” changes in my story above.
Gear changes. The transmission in a race car is designed to optimize engine power and torque, adjusting, as needed to speed and track conditions. Put another way, the race driver has been delegated the authority to change gears as he sees fit to optimize performance.
Drivers have a deep understanding of the machines they drive, and it would be unthinkable to suggest that they require permission to shift gears. If they don’t know when and how to change gears, they don’t belong in the driver’s seat.
Tire changes. Tires are a critical component of competitive racing. They wear quickly under the intense punishment of competitive racing. Tires are changed at pit stops as a matter of course. If you’ve ever watched a Formula 1 pit crew, it’s a thing to behold, every move carefully choreographed and executed with precision. While the decision as to when to change tires and what compound to use is made by the crew experts, the process used to remove and install them is practiced again and again until pit crews can perform them rapidly and repeatedly.
In change management terms, standard changes are established for changing tires. Standard changes are worked out over time to ensure maximum performance and accuracy. Once approved, standard changes are executed as needed in daily operations.
Chassis Adjustments. Adjustments to the chassis are somewhat more complex and require considerable understanding of the driving conditions, aerodynamics, and automotive engineering. In short, adjustments are not to be undertaken without proper consideration.
However, the ability to quickly achieve the desired results, the process and guidelines for chassis adjustments have been distilled down to their essential elements and are executed in a consistent manner—what’s known in change management terms as a change model.
In practice, change models serve as templates, of sorts, for ensuring that similar activities are performed in a consistent and predictable manner. Change management has the opportunity to review and influence the model, but CAB input is generally not required for changes.
DevOps and Change Management
I hope you’ll agree that reviewing every change is an old model, whose time has passed. But have new methodologies like DevOps rendered change management obsolete?
I frequently talk about change management in my writing and speaking. I hear from a lot of people who are struggling with these and a lot of other change related issues. A lot of organizations are still struggling with getting a handle on basic change management. Others see DevOps as a way to accelerate implementation of business-required changes and put an end to traditional change management with its infernal CAB.
Change-related risk must be managed. That’s an IT fundamental. Do no harm to the business. But somewhere along the way, many organizations got the notion that to do that, every change had to be reviewed.
Even a passing knowledge of the work of W. Edwards Deming points out the limitation of a last-check change management. In Deming’s words, “Cease reliance on mass inspection to achieve quality. Eliminate the need of inspection on a mass basis by building quality into the product in the first place.”
Notice that he didn’t say to cease managing quality. He said that quality had to be built in in the first place. I contend that an inspection-before-implementation change management has limited value and represents the most basic form of change management. Rather than stepping up inspection at this last stage—all changes must come to CAB—change management is matured by becoming concerned about building quality into the overall development process. To mature, change management must shift its focus away from individual changes and begin looking at the overall value chain, especially ensuring that desired outcomes are being achieved in the process.
Greg Sanker is an ITSM blogger, speaker, and practitioner with decades of global IT experience with organizations ranging from Fortune 10 tech giants to the public sector. After leaving the corporate life, Greg headed the service management office at a state agency, where he led the adoption of a basic change management program. Read Greg’s blogs about excellence in ITSM.