I favor project management and process improvement using lean and agile methods. Visual management – simple ways for everyone to see “what should be happening” and “what is actually happening” so that anyone can contribute where most required. Agile methods like Scrum are crossing over into mainstream practice, but along the way businesses are having a hard time letting go of their existing waterfall practices and roles. The waterfall-agile hybrids can be ugly.
The imperative for certainty
Business leaders are under constant pressure to drive out uncertainty. You're obliged to predict what's going to happen: forecast sales, forecast for production, forecast earnings, generate confidence inside and out. There is a kind of Emperor's New Clothes haze over the whole exercise, where people are expected to give dependable answers even when the matter is demonstrably unpredictable.
Leaders in lean thinking identified this as one of the big problems in American car manufacturing. It took a long time to build a car, so to have stock on hand dealers were obliged to try to forecast what customers would want - models, colors, optional features and so on. Inevitably they would be wrong, and end up saddled with cars on the lot that didn't exactly meet the needs of any buyer. Customers were obliged to wait a long time for a car to be built to their specification or to buy a car that was available right away, but wasn't really what they wanted. Dealers would end up selling off stock at a discount, and bargain-conscious consumers learned the patterns and would wait for the next sale period. There's speculation that this cycle of overproduction and deep discounting (and others like it) are an underlying cause of slumps and peaks in the economy.
Lean and agile: gunfight or knife fight?
Lean helped this in a number of ways. It accepts that we can't estimate with any accuracy what customers are going to want to buy six months out. It handles this by reducing the need for forecasts that nobody can make. (The Emperor gets to put some clothes on.) The overall lead time for a vehicle is made shorter by eliminating waste and other actions. A platform approach is used for major components as a basis for multiple products. Breakthroughs in reducing the time to switch tooling make it economically viable to manage high-mix low-volume production - instead of having to make a lot of the same thing in a batch because of the overhead of changing the line over to make something else. Inventories of unwanted goods don't build up. The customer specifies and gets exactly what they want, in a short timeframe.
- LEAN therefore is an adaptation to the inherent unpredictability of consumer demand in a high-mix business. Its strategy is to minimize the need for forecasting the mix by having a short enough production cycle that the business can respond directly to demand.
- AGILE is an adaptation to the inherent unpredictability of developing something where customer needs are emergent, and the solution is also emergent. Its strategy is to drive out risk by ruthless prioritization and eliminate non-value adding process work that aims for certainty but fails to provide it (such as elaborate long-term schedules).
Lean is not in quite the same space as agile - in lean the customer needs are typically constrained to choosing from available options, and the method of production is well understood and repeated. Lean applies to a problem that is complicated, not complex. A complicated problem may have a lot of parts but its solution is essentially knowable. A complex problem contains sufficient unknowns that the solution is not fully predictable.
During the software crisis of the 90s a lot of energy was spent on trying to shift software development from a complex problem to a complicated one. Comparisons were made to construction, where a "blue book" of work types could be used for parametric estimating (so many hours to paint so many square feet).
Agile instead says that trying to solve a complex problem with a complicated method is like bringing a knife to a gunfight.
Instead of attempting to nail the jelly to the wall, embrace the jelly. Adapt your working method to gradually expose emergent needs and to finding emergent solutions. Decide what the most valuable thing to do is, and do it first. Proceed incrementally and iteratively. Provide predictions, but only for the short term, recognizing that beyond a few weeks the uncertainty multiplies. Collect data frequently and use it empirically for short term goal setting and to manage longer-term expectations, without overcommitting.
The ugly hybrids
Unfortunately, if you've made your career in the Emperor's New Clothes paradigm of providing commitments even when you know the forecast is unreliable, being asked to explicitly acknowledge and dance with uncertainty can feel a lot as though someone is copping out. If Anna can't tell Paul exactly what features will be delivered in the release eight months from now - the one just before that important trade show - Anna must be incompetent, or taking Paul for a ride. In my experience, if you put enough pressure on someone to give you an answer, finally she'll give you an answer whether she believes it or not, just to make you go away. The problem of how to deal with you when the reality doesn't turn out that way is a future problem. Hence the Emperor's New Clothes: We have a schedule; nobody on the team believes it; but everyone acts as though they do.
What we get as a result is an organization that has (or at least is attempting) an agile development process (in the following case Scrum) but without an agile business process. What might this look like?
Silver bullet:
Senior managers may have formed the impression that agile is a silver bullet. If a development team isn't meeting expectations of X times former performance, it may be labelled as a "troubled" team. A "stronger" Scrum Master may be recruited to knock the team into shape. Unfortunately, like Deming before her, she is likely to find that most issues are not individual, but systematic. She will have a grace period of a few months to try to influence systematic issues, before time runs out along with her credibility. Meanwhile, the team will focus on the only thing that seems to matter to management - their velocity, not the business value created nor the quality of the work. Definition of ready and definition of done are compromised while trying to cram more stories into each sprint; technical debt mounts and user delight declines.
Insistence on long-term plans:
Senior managers may have absorbed that agile will deliver more frequent increments of business value, but not understood the underlying concepts and separation of roles that allow people to take responsibility in a sustainable way. Product Owners start to be pressurized to explain progress, and under that pressure they begin to unduly influence the estimation processes. Scrum Masters' job descriptions start to resemble traditional project managers and require them to estimate tasks and make commitments for what the team "should" be able to do. Both the Product Owner and the Scrum Master may get drawn into satisfying senior managers' thirst for detailed long-term plans by developing documents that are pure placebos - but which take time and effort away from refining the backlog and servicing the team. The developers shrug and carry on, ignoring the Emperor to the best of their ability.
Waterfall-like requirements phase:
Product Owners who don't have good access to the customer base or don't much enjoy "the gemba" may retain the habit of collecting a mass of requirements and then feeding them to the team for months on end. In Lean, the gemba is "the real place", the place where the work happens. To understand and solve problems, you get out of the office or meeting room and go to the gemba. You don't look at the gemba from afar or have it described to you by intermediaries. It's not necessary to collect new needs at the same cadence as the development sprints, but unless the Product Owner adapts her process for listening to customers to also be agile, she isn't learning much from each release apart from the problems that are big enough to be logged as faults.
Sales resisting Product Owner access to customers:
Another attribute of a gemba-averse Product Owner can be that the sales account managers are not accustomed to the Product Owner having frequent direct contact with users. They begin to see Product Owner visits to customers as a threat, fearing that the Product Owner will exhaust the customer's attention span, disrupt a delicate sales growth process, or be an unfortunate reminder to the customer of previous requests that are not yet fulfilled. All of this tends to even further distance the Product Owner from the people she needs to understand the most. This is not to say that the Product Owner is the only person who can collect needs from customers. A business functioning in a VUCA world (volatile, uncertain, complex, ambiguous) needs as many people as possible acting as sensors of the emerging future. Still, the Product Owner is the person who has to aggregate and prioritize all of that sensory input and can hardly do it if she can barely get access to sense something for herself.
Business targets not adapted by learning:
If the company is small enough that the founders are still around and were from a technical background, they may have enough muscle memory of being a developer and manage to stay away from distorting the development process too much. Unfortunately the other parts of the business have no such luxury. The release plan may be grounded in the reality of the development velocity, but the customer account penetration, sales targets or promotional plans may be more aspirational or driven by survival imperatives. If there is a lot of juice in the value proposition bringing in revenue, or the company has other profitable product lines, there may be enough leeway to survive until the situation stabilizes. If not, quickly The Four Steps to the Epiphany play out (sales survives a couple of uncomfortable board meetings and then is first under the bus, later followed by marketing, later by the CEO), as the business is unable to respond in an agile way to adapt business plans.
In conclusion:
If some of this sounds adversarial, that's because it is. To prevent that kind of dysfunction, companies need to have a management sponsor of agile methods that has enough influence and credibility to ensure the senior management understands what they are embracing and why, and the behaviors and attitudes that are required from them to make it work.
The book Patterns of Agile Practice Adoption calls the symptoms when business value is not being delivered "smells" and goes into more detail about how to remove the smells that may be having the worst effects.
Some agile-waterfall hybrids are developed to deal with the issues of processes that have very different risks and cadences - for example aligning hardware and software development, as in this example.
However, when legacy waterfall thinking is inappropriately blended with an agile approach, the results are often ugly and suboptimal.