Organizations, of all types, are jumping on the “business agility” or “Agile transformation” bandwagon. Business agility is the ability to compete and thrive in volatile business or mission environments by responding quickly to changing market or mission circumstances. Agile transformations involve the application of Agile and Lean principles to business or mission processes and decision making. Organizations seek to emulate the efficiency gains and flexibility achieved through Agile software development, but in a business or mission context. Business (or organizational) agility is the goal of Agile transformations.
The value proposition for Agile transformation is compelling. According to proponents of Agile transformation initiatives, organizations stand to become more competitive and thrive in increasingly volatile markets and mission domains by achieving:
- Cost savings through greater efficiency
- Faster responses to changing market or mission conditions
- Better business performance or mission execution
Also driving growth in the number of Agile transformation initiatives is a growing appreciation for the need to align enterprise information technology (IT) investment, governance, and operations with Agile’s iterative and incremental delivery of software system capabilities.
These incentives have fueled a multi-billion-dollar Agile transformation services industry. According to a report published by Allied Market Research in August 2019, “… the U.S. enterprise Agile transformation services market was pegged at $4.91 billion in 2018 and is expected to hit $18.19 billion in 2026 …”.
Despite all of the money spent (or, perhaps, partly because of it), many Agile transformation initiatives fail. We can only estimate how many, but according to Jeff Sutherland, co-creator of the Scrum framework, 47% of Agile transformations do.
The Organizational Culture Challenge
Most Agile transformation initiatives failures are tied to organizational culture. The 14th Annual State of Agile Report, published in 2020, supports this assertion. Four out of the top five reasons, cited by survey participants, for failed Agile scaling initiatives are directly tied to organizational culture:
For organizations that develop software systems, or sponsor their development, scaling of Agile software development and system operations is a crucial part of Agile transformation. Cultural issues that frustrate their success also negatively affect business or organizational agility. The first and third challenges, marked in red in Figure 2 above, explicitly call out organizational culture. The second and fourth marked challenges point to lack of commitment and participation from organizational leadership and management. Organizational leadership is responsible for setting the tone for the rest of the organization, so a lack of support from them translates into a lack of support for Agile transformation across the organization.
Addressing the Organizational Culture Challenge
As common as the “culture” challenge is, it is hard to tackle because it is so vague. What are the reasons for “resistance to change” and how do they manifest themselves? What is “enough” leadership and management participation and support and what do they entail? Beyond performing the mechanics of a chosen Agile approach (e.g., Scrum, Kanban, SAFe), when do we know the organization has achieved an “Agile culture”?
Organizational culture influences behavior. The inverse is also true: changing behavior changes culture. Merely exhorting people to “think Agile” and “adopt Lean thinking” is a waste of time because they likely do not know how to do so within the context of the organization and their jobs. On the other hand, engaging in a mechanical implementation of an Agile approach, without understanding its values and principles, guarantees that existing organizational cultural biases will corrupt the approach.
Organizational culture change starts with laying out a clear and compelling vision for the organization. The organization’s chosen Agile approach becomes the means by which the vision is implemented. An Agile transformation effort entails some degree of uncertainty and risk. It will likely be the first time the organization embarks on such an effort. Mistakes will be made and lessons will be learned. This is why closely aligning to the chosen Agile approach and understanding its values and principles is so important.
The Agile approach serves as the foundation upon which organization-specific practices eventually emerge. While the Agile approach likely requires some degree of tailoring to ensure it aligns to the constraints and needs of the organization, such tailoring must be informed by a deep understanding of the implications of each change. It is best to explore ways of changing existing organizational processes to align to the Agile approach before tailoring the approach to fit the organization’s processes.
As people learn how to do their jobs, in alignment with the Agile approach, through training, coaching, and actual experience, their understanding of the approach’s values and principles take root and increasingly shift organizational culture towards a shared, agile-enabled vision.
Four Organizational Characteristics Most Likely to Derail Agile Transformations
Before embarking on an Agile transformation initiative, organizational leadership would be wise take time to reflect on how well positioned they and their organizations are to achieve a successful Agile transformation. While it is true that the only way to know for sure is to try, an honest self-assessment enables leaders to approach Agile transformation with a clear-eyed understanding of what cultural obstacles they are likely to encounter. Leaders may come to the conclusion that their organization is not ready for the transformation. In our experience, the organizations least likely to achieve business agility through Agile transformation share the following four cultural characteristics:
- They emphasize process execution over actual progress
- They emphasize control to the detriment of speed and flexibility
- They discourage experimentation and punish failure
- They possess a consistent record of failure
1. Emphasizing Process Execution over Actual Progress
Many organizations operate under the assumption that following plans and procedures ensures good outcomes. While no organization can operate without them, many organizations lose sight of the purpose of plans and procedures: to accomplish business or mission goals and objectives. When this happens, the following cultural dysfunctions arise:
- People are rewarded for following plans and procedures, rather than for achieving desired outcomes
- Failure is accepted so long as procedures were followed
- Project status reporting tracks process activity completion rather than progress towards desired outcomes
One of the 12 Principles of Agile Software Development is, “Working software is the primary measure of progress.” The only way to validate the value of any software capability is by getting it into the hands of customers and end users. Only they can determine whether expected business or mission outcomes were achieved through use of the software.
Similarly, the effectiveness of business plans and processes should be validated against business or mission outcomes. Doing so involves quantifying expected outcomes and continually measuring their attainment against relevant metrics. Failure to achieve desired outcomes should prompt process changes or the formulation of new processes. This empiricism is a critical characteristic of Agile and Lean.
2. Emphasizing Control to the Detriment of Speed and Flexibility
The point of carrying out an Agile transformation is to achieve business or organizational agility. Agile Cheetah defines agility as the combination of speed, control, and flexibility:
The elements of this “Agility Triad” are mutually dependent constraints. In other words, more or less of any one constraint influences the other two. For example:
- Accelerating the pace of work requires some degree of control and flexibility
- Overemphasizing control slows progress and decreases flexibility
- Underemphasizing control can turn flexibility into chaos, robbing teams of both planning and process stability, thereby slowing progress
Many organizations are guilty of the second bullet. Overemphasizing control to the detriment of speed and flexibility is emblematic of fear-based organizational cultures. In such cultures, Taylorism prevails. Organizational structure tends to be hierarchical with a strong command-and-control management approach. There is often an unwillingness to delegate decision making down to the most responsible levels. In the most extreme cases, position, status, and tenure trump expertise, knowledge, and skills.
Rather than accepting that some amount of risk is an inevitable part of solving tough problems, fearful organizations avoid taking risks. Fear of failure produces cultures where failure to act is deemed preferable to acting and failing. When failure happens, finding blame takes precedence over finding solutions. Additional control mechanisms are put in place to ensure it does not happen again.
Unfortunately, such mechanisms often feed a vicious cycle where people lose some autonomy and latitude to act, resulting in a loss of effectiveness, which breeds further failure, which then prompts further controls. Left unchecked, this cycle can whittle down organizational effectiveness to the point where the organization loses the ability to reach previous levels of performance.
Organizations with strong hierarchical and command-and-control cultures struggle through Agile transformation more than those with more egalitarian and permissive cultures. Leaders should ask themselves; how much authority are they willing to delegate to achieve greater flexibility and speed? What changes need to happen to enable their people to assume greater ownership and responsibility for their work and resulting outcomes?
3. Discouraging Experimentation and Punishing Failure
Organizations that overemphasize control also tend to discourage experimentation and punish failure. Rather than exploring different solution options, they rush to settle on one approach. Time that could have been spent investigating options is spent improving a solution that was likely suboptimal from the start. Fear of ambiguity and risk leads to the formulation of unrealistic plans (In search for certainty, they conjure fantasies). Premature commitment to plans takes away the flexibility needed to adjust to changing circumstances and new information. Closed off to alternatives and fearing the consequences of admitting failure, the organization continues to sink resources into failing approaches.
As mentioned earlier, Agile and Lean are based on empiricism. In an empirical process, information gained through experimentation and observation informs and validates plans, decisions, and actions. Planning is adaptive as opposed to predictive. In other words, Agile plans change based on current circumstances and new information and insights.
Agile plans skew towards shorter planning horizons measured in weeks to months rather than months to years. When problems or mistakes happen, Agile teams re-plan and correct course. Short planning horizons afford the flexibility to make new plans and pivot to new solution approaches.
Like experiments, plans and decisions are treated as a hypothesis. When a hypothesis proves to be wrong, another one is formulated and tried. The faster teams cycle through these learning cycles, the faster they arrive at solutions that provide real business or mission value, cost effectively, and in time to be valuable.
4. Possessing a Consistent Record of Failure
Agile is not a silver bullet. Agile, on its own, does not solve problems, it makes them plain for everyone to see so they may take action. Transforming organizations from traditional project management structures to Agile requires courage, vision, and competence. Organizations with long and consistent track records of underperformance and failure are most likely to fail at implementing Agile transformations.
Underperforming organizations may actually be missing core competencies that must be developed or acquired prior to embarking on any type of organizational transformation. Such organizations are more likely to corrupt their chosen Agile approach to accommodate their deficiencies.
Conclusion
While most organizations would benefit greatly from attaining business agility, many do not possess cultures amenable to Agile transformation. However, cultural change results from changing behavior. Organizational leaders maximize the odds of successful Agile transformations by acknowledging organizational culture problems and addressing them before and during Agile transformation initiatives.