Category: Agile Transformation

Stop Managing Software Projects

Manage Software Products Instead

If you are managing software development efforts as projects, you are doing it wrong.  Instead of managing projects, organizations that sponsor and develop software solutions need to shift towards managing solutions as products.  A product management approach to software development aligns with modern software development and with Agile’s inherent product management bias.

Project Management vs. Product Management

In a nutshell, project management differs from product management in that the former is primarily concerned with managing people, schedules, and tasks while the latter focuses on managing the product and how it is created.

Table 1: Product Management vs. Project Management

Traditional project management is misaligned with modern software development for two reasons:

  1. An overemphasis on managing the triple constraints of the “Iron triangle” to the detriment of technical execution, product quality, and value delivery
  2. An overemphasis on control to the detriment of speed and flexibility

The “Iron Triangle”

Traditional program management focuses on enforcing the triple constraints of the Iron Triangle:  cost, schedule, and scope (the amount of work done). The objective is to deliver expected value on time and on budget.  Unfortunately, fixing the three constraints leaves quality as the only variable, opening the door to sacrificing quality when things don’t go as planned.  Because no one can predict the future, most traditional projects fail to deliver on at least one of these constraints.  In other words, “I can get it to you cheap, fast, or good.  Pick any two.”

Figure 1: The Iron Triangle

Traditional project managers manage against detailed plans that attempt to fix all three constraints for the life of the project.  They base plans on estimates of how long it will take a certain number of people, with specific skills, to accomplish a set amount of work at a certain cost.  As problems arise, managers work to keep the project as close to the original plan as possible.  If project reality strays too far from the plan, they develop a new project plan or scrap the project altogether.  Replanning is a risky and expensive proposition, so project managers work mightily to avoid it.

Fixing cost and schedule begins by fixing scope.  The more detailed project requirements are upfront, the more precise the cost and schedule estimates become.  However, these calculations are not necessarily more accurate because they are based on estimates and on requirements that will likely change the bigger and more complex project scope becomes.

How Enforcing the Iron Triangle Affects Software Development

The problem with applying traditional project management to software development is that development of non-trivial and/or large-scale software solutions is too complex for linear predictive planning.  As software solutions grow and evolve, their complexity increases, sometimes at a seemingly exponential rate.  Since we can develop and modify software solutions in infinite ways, they are often significantly more complex than physical systems of comparable cost.

Enforcement of the Iron Triangle in software development projects often leads to the following problems:

  • Narrowing requirements and technical approaches too early – Predictive project planning typically forces teams to define what they will build and how before they really understand the business/mission and technology domains they will work in.  This often leads to developing things that customers and end users do not need or that do not adequately address their needs.
  • Tendency to overengineer solutions – Specifying technical scope upfront often leads to overengineering.  Defining and developing features that few, if any, customers or end users will actually use is quite common.  Also, overengineering can lead to creating complex features or functionality where simpler versions could have satisfied requirements.  All those rarely-used or overly complex features cost money to develop and maintain.  They rob time and budget from developing and releasing features customers and end users truly value.
  • Difficulty managing risks and taking advantage of opportunities – In the zeal to keep projects “on plan”, planners often identify emerging risks too late to keep them from becoming issues or forgo opportunities to improve the product and how it is developed.
  • Forgoing opportunities to validate solutions – Projects based on predictive planning, such as Waterfall projects, often afford only one chance to validate what teams develop.  Typically, validation of the solution occurs towards the end of the project, when the time to address problems is most limited.
  • Delaying delivery of business or mission value – With only one chance to deliver / release, sponsors, stakeholders, customers, and end users must wait until the end of the project to gain access to solution features.

Agile “Breaks” the Iron Triangle

Instead of attempting to fix all three constraints of the Iron Triangle, Agile fixes cost and schedule while allowing scope to “float”. In other words, Agile efforts are managed primarily by managing scope.

Under an Agile approach, there are no surprises. We have a certain amount of budget and time to spend developing software capabilities. We continuously verify (test) the solution as we build it to ensure we are building things right and demonstrate and release functionality often to ensure we built the right things. Since we build and deliver solutions iteratively and incrementally, customers and end users get valuable business or mission capabilities sooner.

Figure 2: Waterfall, Agilefall, and Agile Release Frequencies

For as long as we have enough budget and schedule, we have the option to continue developing functionality or stop after realizing enough value. We also have the option to pivot to a different solution approach or cancel the solution altogether. When budget and schedule run out, we can choose to invest more on the solution based on demonstrated functionality and customer and end-user feedback.

Figure 3: Pivot or Persevere

Overemphasizing Control

Enforcing the constraints of the Iron Triangle causes project management to implement heavyweight management and reporting processes.  These processes are attempts to lower risk by increasing control.  Overemphasizing control slows software development and decreases the flexibility necessary to tackle risks and challenges.  On the other hand, underemphasizing control can turn flexibility into chaos, robbing software development teams of both planning and process stability, thereby slowing progress.  An optimal balance between control, flexibility, and speed requires empowering technical contributors to plan their own work based on sponsor, stakeholder, customer, and end user needs and enterprise technical constraints.  The main reason for this is the inherent complexity of software systems.

High-performing teams are empowered teams.  They take on or participate in many duties traditionally left to project management:  Devising work plans, decomposing work and deliverables, monitoring task and reporting progress, and meeting commitments.

Software Development’s Inherent Product Management Bias

Software is never finished.  We’ve moved away from delivering discrete, standalone software package products (think Windows 95) to continual delivery of new software functionality and bug fixes via updates distributed over the internet.  Even in organizational IT and enterprise system integration efforts, development never truly finishes.  If any solution or product is to survive, it must adapt to meet changing customer needs while maintaining quality and performance.

Also, software is infinitely changeable.  Since software is abstract, we can add and modify it at will with much less effort and expense than physical solutions or products.  Thus, we don’t see the same level of resistance to updating software solutions as we see with physical products.  The very nature of software solutions makes them likely to continue changing and evolving long after initial development.

Thus, managing the development and evolution of software solutions as projects that start and stop does not make sense.  All software solutions go through the stages of the Product Lifecycle.

The Product Lifecycle

Figure 4: The Product Lifecycle

The Product Lifecycle includes the phases all products go through during their useful lives: from when they are first conceived, to when they are retired from the market or their mission domains.  This cycle focuses on the performance of the product in its intended market or mission domain.  All products, including software solutions, commercial or otherwise:

  1. Start out as an idea developed into an initial version of a releasable product
  2. Are introduced / released to their intended consumers or user bases
  3. See their market or user base expand, if successful.  This phase tends to drive the most product changes that keep the product relevant and/or profitable
  4. See an eventual stabilization or flattening in customer use, sales, or revenue growth as the product’s market or mission domain matures
  5. See their market or mission domain decline as it becomes saturated or customers move on to new solutions
  6. Are retired when they cease to be profitable, effective, or relevant

Projects Incur Wasteful Overhead

Traditional project management’s focus on projects is wasteful.  Starting and stopping projects incurs significant overhead.  Examples include:

  • Contracting activities (and acquisition activities, for government projects) – Finding contractors / vendors, putting them on contract, monitoring performance, “Color of Money” concerns, etc.
  • Ramping up new employees or vendors to be productive – Accesses, equipment, software tools, training (formal and on-the-job), etc.
  • Ramping down employees or vendors as projects end – Removing accesses, returning equipment, managing software licenses, etc.
  • Not having the flexibility to easily terminate failing projects or projects considered no longer necessary or desirable – It is much cheaper and easier to cancel a project after spending $10 million than after spending $100 million

Agile’s Inherent Product Management Bias

So, if we should manage software development as product development efforts, rather than as projects, what does that look like?  It looks like Agile software development.  Agile software development in general, and Scrum specifically, have an inherent product management bias.  Table 2 below shows how Scrum fulfills the product management duties listed in Table 1:

Table 2: How Scrum Fulfills Product Management Duties

Conclusion

In my experience, the single biggest reason why so many Agile software development efforts fail to achieve the benefits of Agile is the conflict between traditional project management thinking and Agile’s inherent product management approach to software development.  This is not just my opinion, it is a growing realization Lean-Agile circles[i].

Today, business and mission requirements change too quickly for slow-moving predictive planning and management approaches.  To meet the challenges brought about by the quickening pace of technology, we must leverage flexibility and control to achieve speed while maintaining quality.  This requires a shift from a project management to a product management mindset.

[i] https://www.scaledagileframework.com/lean-portfolio-management/



Why Integrate Agile, DevOps, & Microservices?

Increase development and organizational agility, and better serve stakeholders, by integrating Agile, DevOps, and microservices approaches.

Well-run organizations invest in information systems to achieve specific and measurable goals and objectives.  They determine the return on information technology (IT) investments by how much they lower cost, increase productivity, lower risk, and increase quality.  This expectation of business or mission value achieved for money spent carries over to how organizations conceive, develop, field, and maintain information systems.  Valuable information systems allow users to leverage technology in ways that improve how they work to accomplish organizational goals and objectives.

Improving this alignment between business or mission needs and information systems development has been the holy grail of analysts, engineers, and managers since the dawn of the Information Age.  Unfortunately, past engineering and management approaches actually widened the communication gap between business / mission stakeholders and software developers.  This state of affairs made information systems:

  • More expensive to develop and operate;
  • Take longer to develop than expected;
  • Less likely to generate expected value after fielding; and
  • More likely to be cancelled before initial fielding.

Over the last 20 years, innovations in how we develop and field software systems, of all kinds, are breaking down barriers to collaboration between business / mission stakeholders and development teams.  Arguably, the three most important innovations are Agile software development, DevOps, and Microservices Architecture.

Here, I discuss how the three approaches can work together to address the aforementioned shortfalls. We aim to foster tighter collaboration between business / mission stakeholders and development teams as we improve development and organizational agility.

The Agility Triad

Today, mature software development organizations can maximize development and organizational agility by integrating Agile software development, DevOps and microservice architectural practices.  Agile Cheetah defines agility as:

Agility = Speed + Control + Flexibility

We call this definition “The Agility Triad.”  The Agility Triad’s elements apply to organizational, process, and technological agility.  The three elements are mutually dependent and reinforcing constraints defined as follows:

Figure 1: Agility Triad Elements Definitions

DevOps and Microservices Contribute to Agility

DevOps and microservice architectures provide the means of implementing the Agility Triad’s elements.  DevOps serves as the Agile implementation of the Software Development Lifecycle (SDLC).  Microservices help align software deliverables directly to the business / mission domain and requirements.  They also help structure software development teams, their work, and what they deploy to directly address business or mission needs.

DevOps as the Agile SDLC

Agile projects perform the same SDLC activities as Waterfall projects.  The difference is that software teams execute the Agile SDLC as a continuous flow of activity rather than as a linear, one-time progression through time-bound phases.

Figure 2: Waterfall vs. Agile SDLC / DevOps

Agile projects are feature-based.  In the context of Agile, features are cohesive and complete slices of system functionality capable of performing useful work from the perspective of customers and end users (human or automated). 

Throughout development, Agile teams define, develop, test, deploy, and release features.  Rather than waiting until the end of the project for a finished system, project stakeholders, customers, and end users gain access to released features throughout the project.  These features, individually and collectively, are valuable business / mission capabilities ready for use upon release.

Every feature goes through its own mini–Agile SDLC iteration.  Mature Agile teams leverage automation to continually test and integrate features into complete increments of releasable functionality.  Newly deployed and released increments work with all previous increments.  The system grows and evolves iteratively and incrementally.

At the project level, the Agile SDLC is essentially a “swirl” or “flow” of activity occurring concurrently across multiple features and increments.  The most mature Agile teams release so quickly and often that their releases approach a continuous flow.

DevOps is the implementation of this flow-based approach to software development.  DevOps is not an Agile concept, nor did it start with Agile.  However, it aligns perfectly with Agile because it reinterprets the SDLC in ways that support Agile’s iterative and incremental value delivery and team-based culture. It also aligns with Lean’s emphasis on achieving flow and promoting continuous improvement.

Ways DevOps Enables Agility

I now explain how DevOps enables the three elements of the Agility Triad.

Figure 3: The Agility Triad

DevOps Contribution to Project Speed

DevOps accelerates software delivery by leveraging modern software development techniques, tools, and technologies and by eliminating wasteful handoffs between development teams and operations staff.  This culture of automation and continuous improvement facilitates faster iterations through the SDLC for every feature developed.

DevOps automation goes beyond automating the testing, packaging, and deployment of source code.  Automated building and configuration of host environments is also a critical facet of DevOps.  This capability is termed, “Infrastructure as Code (IaC).”

IaC allows for the repeatable provisioning and configuration of host environments via execution of code.  Rather than incurring the time and cost of manually setting up environments, infrastructure automation platforms provision and configure virtual environments by executing scripts and implementing configuration files. 

DevOps Contribution to Controlling Projects

Faster and more frequent iterations through the SDLC support greater control of project scope, budget, and schedule through increased empiricism.  In other words, Agile teams and project management monitor progress, make decisions, and act based on lessons learned from previous releases.  The faster and more frequently teams iterate through the SDLC, the faster and more frequently they receive feedback that informs upcoming work plans (scope).

Project management can make decisions about the state of the project based, not on estimated levels of completion but, on information gained through actual deployment, release, and use of functionality.  This arms managers with the information they need to decide whether to continue the current course, update project plans, or terminate the project.  All along, teams deliver complete increments of functionality capable of providing value immediately.

DevOps Contribution to Project Flexibility

DevOps contributions to software project flexibility include:

  • Faster and more frequent iterations through the SDLC that allow experimentation with different approaches and quickly pivoting away from failing or undesirable approaches.
  • The ability to separate software deployments in production from releases, which enables:
    • Testing in Production – Teams can deploy and test new software features in production before making them available to users.  This allows teams to experiment with different versions of the same functionality and see which performs best.
    • Releasing on Demand – Rather than deploying and releasing on a fixed schedule, teams can deploy functionality at any time and make it available to users when they are ready to use it.

Monolithic vs. Microservices Architectures

Traditional software projects delivered solutions all at once.  In other words, software teams deployed and released all application components as one unit.  Changes to an application often required complete redeployment.  In microservices parlance, we refer to applications architected this way as monoliths.[1]

While architects and developers typically split monoliths into packages, modules, or components, interdependencies between those components make it difficult to develop, test, and deliver them separately.  In systems engineering terms, these system components are “tightly coupled.”

A microservices architecture, on the other hand, builds applications as sets of independently deployable services that represent key entities of a particular business or mission domain.  For example, an inventory system may include services that handle transactions associated with customers, invoices, warehouses, payment, etc.  All of the information / data associated with each of these entities belongs to its respective service.  A service needing information from outside the confines of its own entity (e.g., the warehouse service needs a customer’s address) must get it from the corresponding service.

Three-Tiered (Web) Applications

Since the 1990’s, the predominant architectural model for monolith applications has been the three-tiered (web) application.  This distributed architecture grouped together components based on where they fit within each of the following tiers:

Figure 4: Three-Tiered (Web) App Monolith

  1. Front End:  Implements the application’s user interface(s) or externally-facing interface(s)/API(s)
  2. Middle Tier:  Performs the bulk of the application’s business logic
  3. Datastore:  Manages application data and state

Microservices Cut Across Three-Tiered Architectures

In contrast, since microservices are end-to-end slices of business or mission functionality, each microservice includes just enough of the three tiers to perform its job. 

Figure 5: Each Microservice Includes Components from Each Tier

Individual microservices are not coupled to the internal workings of other services.  Thus, microservices are independent or “loosely coupled.” As long as the interfaces they expose do not change, internal changes to individual microservices will not ripple across the application or system.  Teams can deploy and release microservices independently, allowing applications to grow and evolve iteratively and incrementally. 

Figure 6: Microservice Composition

Features Map to Microservices

As stated earlier, Agile value delivery is feature-based.  This differs from the component-based delivery of functionality performed in traditional software projects.

In traditional software projects, management organizes teams to develop system components which they later integrate into complete solutions.  Until that integration happens, nothing of business or mission value exists.  Complete and cohesive functionality / features only exist after system integration.

System integration occurs infrequently in traditional software projects. This increases project risk.  Validation of an application’s actual business / mission value typically occurs towards the end of the project, when budget and schedule are scarce.

As end-to-end slices of business or mission functionality, microservices help development teams develop, test, and deliver applications, iteratively and incrementally, as features, from the start.  A feature may be satisfied by one microservice or through orchestration of multiple microservices.

Ways Microservices Enable Agility

Agile approaches, such as Scrum, help teams organize and manage their work as increments of functionality (features / user stories), developed within short timeframes (timeboxes), as part of frequent releases.  DevOps provides the automation and coordination required to continually and sustainably deliver features agilely.  Microservices are as close as we have come to directly implementing, deploying, and releasing requested features as features.

Microservices Contribution to Project Speed

Microservices accelerate software applications development and fielding by:

  • Breaking down application scope into smaller and more manageable units of work – A concept of Lean is that smaller batches of work increase flow.  Microservices make it easier for development teams to manage their work through Agile and DevOps practices.
  • Allowing customers and end users access to valuable features earlier – Rather than waiting until full system deployment to validate and use a solution, customers and end users gain access to useful functionality throughout development.  Lessons learned from released functionality help inform ongoing development.
  • Allowing development teams to work in parallel – Development teams can work on different microservices in parallel with little worry about inter-service dependencies beyond the interfaces the services expose.
  • Facilitating mixing and matching of technologies and technical approaches – Microservice independence also applies to the technology stacks and platforms used to implement them.  Teams can develop microservices in whatever technologies they are most comfortable with and still achieve service integration and interoperability.  Since microservices communicate via networks, they do not need to run on the same platform or under the same operating system.  This technology independence lowers the amount of learning and coordination required across development teams.
  • Lowering the risk associated with fielding new functionality – Adding or replacing individual microservices is much easier and less risky than redeploying monolith applications.  This flexibility lowers outage times.

Microservices Contribution to Controlling Projects

Microservices help teams and project management control projects by:

  • Facilitating Development Around Features – Agile teams control projects by managing scope.  Breaking up system scope into features helps teams manage their work.  Microservices support the iterative and incremental development and release of features.  This strategy affords teams multiple opportunities throughout development to evaluate progress and update project plans based on actual use.
  • Lowering Inter-Team Dependencies – Agile software development teams are cross-functional.  Members possess the skills necessary to develop complete, coherent, and deployable features across short timeboxes.  This organization around features, rather than components, aligns with the tenet of building microservices that include all of the components necessary to perform as independently deployable services.  Organizing teams along the lines of the microservices they develop lowers dependencies across teams and associated integration risks. 

Figure 7: Tier-Based / Component Teams vs. Microservices / Feature Teams

Microservices Contribution to Project Flexibility

Spreading application scope across individual microservices simplifies changes and promotes experimentation.  Teams can change their piece of the application without fear of affecting the rest. Thus, there is less resistance to trying out different approaches and settling on the most optimal.  For example, we can deploy different versions of the same microservice with different user interfaces in production, selectively release them to users, and evaluate them.  This technique is known as A/B testing.

Summary of DevOps and Microservices Contributions to Agility

I summarize the contributions DevOps and Microservice Architecture can make to agility below:

Figure 8: DevOps and Microservices Contributions to Agility

Conclusion

Proper integration of Agile, DevOps, and microservice architectural approaches can narrow the gulf between business or mission stakeholders and those who deliver software solutions.  We do this by improving development and organizational agility.  Under this construct:

  • Agile planning and execution are supported by DevOps processes
  • DevOps processes accelerate development, testing, packaging, and deployment of microservice-based applications
  • Resulting applications are better aligned with Agile delivery and business / mission needs

The decision to adopt any of the three approaches is not be taken lightly.  There is no one-size-fits-all approach to making them work and few organizations make all three work well.  It may also be the case that any of three approaches are not right for your organization or specific project.

However, these approaches are firmly in the canon of modern software development practice.  It is worthwhile investigating how your organization can improve how it practices these approaches or begin adopting them.

[1] Sam Newman provides a comprehensive explanation of the ideas covered in this section in his book:  Monolith to Microservices, 2020, O’Reilly, Chp 1



Why So Many Developers are Fed Up with Agile (Pt. 3)

Top-Down Driven Agile

No one likes being told how to do their job, especially by people less knowledgeable about the work than themselves.  One of the defining characteristics of knowledge workers is that they know more about the work they do than their supervisors.

Knowledge workers amass a wealth of education, training, and experience in their specialties.  Their success depends on how well they apply those intellectual resources to their work.  Knowledge workers do not merely produce tangible or quantifiable products and services. They help employers or clients achieve business or mission outcomes.  Thus, they usually enjoy much greater latitude in determining how they work than workers in traditional service or manufacturing sectors.  Software development teams are made up of knowledge workers.

Unfortunately, traditional project management culture is still highly influenced by Taylorist notions regarding work organization and worker management.  Too many organizations reflexively assume that management should determine how employees work.  This predisposition runs directly counter to the Agile principle of:

Build[ing] projects around motivated individuals, [giving] them the environment and support they need, and trust[ing] them to get the job done.

http://agilemanifesto.org/principles.html

Organizations possessing traditional project management cultures often adopt Agile in ways that largely ignore the perspectives and input of those most affected by the change: software development teams.

This article is the third of a four-part series about why so many developers are fed up with Agile and why so many organizations fail to achieve Agile’s much-touted benefits.  The common thread across the series is that what often passes for Agile is anything but Agile.

The other topics covered in the series are:

  • The Never-Ending Agile Grind – Developers often feel overwhelmed by the constant pressure to produce under tight deadlines.
  • Greater Visibility & Oversight with Less Control – The “radical transparency” of Agile approaches, such as Scrum and Kanban, is often exploited by management as an opportunity to micromanage development teams.
  • Changing Expectations for Developers – Developers are moving away from functioning as “order takers”, focused almost exclusively on coding, to collaborating with product owners/managers, sponsors and stakeholders, and customers and end users to define, verify, and validate what they develop.  Many technical contributors find this level of direct collaboration difficult to manage alongside their growing list of technical responsibilities.

A Reaction Against Traditional Project Management

Agile software development started as a reaction against the decades-old, heavyweight, project management approach to software development commonly referred to as Waterfall.  It began as a collection of “lightweight” software development approaches implemented by a few like-minded, software industry thought leaders.

Realizing their approaches shared a common philosophy, they gathered at a ski resort in Utah in February 2001.  There they drafted “The Agile Manifesto for Software Development”.  Adoption of the Manifesto and its supporting 12 Principles spread quickly across the software industry.  That formulation of what constitutes software development agility helped popularize Agile and shape the ongoing evolution and refinement of Agile approaches. 

Agile was a bottom-up movement started by software developers.  As stated in the first sentence of the Agile Manifesto:

We are uncovering better ways of developing software by doing it and helping others do it.

http://agilemanifesto.org/

Viewed through the lens of management practice at the time, what these early Agilists advocated was disruptive; almost subversive.  They carved out space within which they could work without the distractions or frustrations of managerial micromanagement and organizational politics. They broke down organizational silos and included customers and end users in defining, verifying, and validating software solutions.  While Agile helped shift software and IT management culture towards becoming more cooperative and participatory, progress has been spotty.

Few organizations fundamentally changed their traditional management cultures to align with Agile.  While many digital native companies established during the first two decades of the 21st century internalized Agile and Lean principles from the start, many (if not most) organizations did not.  Traditional project management biases still color how most organizations view Agile.  They perceive Agile as a new way of developing software rather than a new way of delivering value through software.  For those organizations, Agile is something developers do while the rest of the organization functions as before.

This fundamental mismatch between Agile’s bottom-up culture and top-down traditional project management culture is the primary reason so many Agile implementations go wrong.  It is also why those with the most at stake in adopting Agile often have the least say in its implementation.

Aligning Organizations to Agile Delivery

Many organizational leaders are unaware of the need to align their organizations with Agile’s iterative and incremental mode of value delivery.  In Agile, value is whatever solution sponsors, stakeholders, customers, and end users (i.e., the Stakeholder Community) care about or makes them happy.  This definition equates value with real business or mission value rather than the cost incurred in developing functionality.  Agile teams help organizations achieve predefined and measurable outcomes by delivering valuable software capabilities.

Agile teams determine what to build through negotiation and collaboration.  The Stakeholder Community is actively engaged with development teams in defining requirements and verifying and validating solution increments throughout development.  This way of working affords plenty of opportunities to inspect and adapt emerging solutions as they grow and evolve.  Frequent releases provide the most important validation of all: customer and end user acceptance of solutions and achievement of desired outcomes.  What teams learn along the way informs future development.

Organizations with strong traditional project management cultures are not organized to participate in this form of value delivery and empiricism.  They are organized to outsource the definition and development of software solutions to development teams after a period of requirements definition.  They then wait until the solution is finished to validate its value. 

Outsourcing Agile Adoption

When organizational leadership sees Agile as a process for developers to follow, instead of a culture change for the entire organization, they are more likely to “outsource” Agile adoption to Agile trainers, coaches, and consultants.

Many organizations spend large sums training developers on the mechanics of an Agile approach but fail to internalize the values and principles behind the approach.  The result are development teams that perform the practices of the chosen Agile approach without understanding how those practices foster agility.  Meanwhile, management continues to operate under Waterfall assumptions and expectations.  Organizations end up with crops of newly Agile-certified employees but no plans to apply what they learned towards achieving greater agility.

Many developers feel they do not get enough value from Agile trainers and coaches.  Some trainers and coaches lack enough understanding of the day-to-day work of software developers or the organizational environment to be effective.  Others are Ill-equipped to help organizations move beyond executing team-level Agile practices and establish Agile-based organizational cultures.

Another manifestation of organizational leadership outsourcing Agile adoption is hiring consultants to develop “Agile strategies.”  Rather than leveraging consultants to help them develop Agile strategy, organizational leadership pays consultants to develop their Agile strategy with little organizational input or understanding of the organizational culture.  When this happens, the resulting strategies tend to be long on theory and short on actionable objectives for the organization to pursue. 

“Hybrid Agile” (Scrumfall, Agilefall)

Many organizations struggling to transform from traditional project management to Agile decide to go for a “Hybrid Agile” approach.  “Hybrid Agile” is not a recognized Agile software development approach. It is an attempt to gain the benefits of Agile while avoiding internalizing and applying Agile values and principles. Rather than reevaluating existing processes to achieve greater agility, these organizations contort Agile approaches to fit within existing organizational culture. This leads to a tailoring of Agile misinformed by traditional project management biases.

Development teams working in “Hybrid Agile” projects experience the worst of both Waterfall and Agile.  Teams are expected to:

  • “Plan” their work while being denied control over the amount of work they commit to
  • “Self-organize” while having no say over team structure, tools, and processes
  • Provide “estimates” that are treated as deadlines
  • Document what they work in “Agile” productivity tools to keep management informed
  • Develop what they are told to according to an “approved” design while expected to accommodate customer and end-user change requests

Proposed Solution

Developing valuable software solutions and products takes more than writing code.  Regardless of the software development approach followed, all projects are accountable to their target Stakeholder Communities and must provide business or mission value, cost effectively, and in time to be valuable.  Management and development teams have to support each other to be successful.

Bottom-up Agile adoption typically fails to foster organizational agility beyond development teams.  Top-down Agile adoption tends to lack in-depth understanding of the realities of software development and buy-in from development teams.  A “meet-in-the-middle” approach accounts for the strengths and weaknesses of the other two approaches.  The meet-in-the-middle approach entails:

Prepare Everyone to Work Agilely

While Agile is inherently bottom-up, organizational transformations required to achieve agility are not.  Organizational leadership and management must play an active role in leading the transformation.  However, to do so in ways that do not crush agility, they must adopt a “Servant Leadership” model.  This starts with a recognition that the primary role of management is to empower those who deliver value to the customer and to create work environments and customer relationships in which value providers can succeed and flourish.

A big piece of leading the change involves training, coaching, and mentoring employees, at all levels and roles, to understand Agile values and principles and how to apply them to their work.  This may require different types of training/mentoring or individual study for different roles throughout the organization.  For example, coaching and mentoring for organizational leadership and management could be geared towards:

  • Understanding the differences between traditional project management and Agile
  • Learning and internalizing the tenets of Agile Servant Leadership
  • Aligning organizations to Agile value delivery in ways that align with Agile values and principles

I stress the need for coaching and mentoring.  Setting people loose after a two-day Agile certification class and telling them to “be Agile” guarantees failure.  Most people outside of software development have not worked in Agile environments. Thus, they do not understand how an organization rooted in Agile values and principles would impact how they work.  Organizational agility is not about everyone organizing themselves as a Scrum team. Like any new skill, Agile thinking and practice must be honed to gain the most benefit.

Hire Knowledgeable and Experienced Trainers & Coaches

This point is obvious but warrants explanation: hire knowledgeable and experienced trainers and coaches.  They need experience in more than just teaching classes and holding Agile certifications.  The more experience trainers and coaches possess in developing software solutions or leading their development, the better. 

I temper this with the caveat that the best trainers and coaches possess subject matter expertise and instruction, facilitation, and communication skills specific to their role.  The most technical people are often not the best fit for teaching and coaching Agile.  What are needed are generalists who possess a mix of theoretical knowledge, relevant industry or mission experience, exposure to the work of software development, and strong interpersonal and communication skills.

Hire, Educate, Train, & Mentor Developers for Leadership Roles

Employers in technical fields tend to select out “soft skills” while overemphasizing technical skills.  They do this at their peril.  Agile depends on the development of self-organizing “high-performance” teams.

Agile teams develop leaders.  Hiring well-rounded people, who work well in teams, and developing their technical and leadership skills ensures development teams have the skills they need to self-organize and to build productive relationships with their Stakeholder Communities.

Conclusion

Reconciling Agile’s inherent bottom-up perspective to organizational change with the need for transformative leadership from above is difficult.  There are no one-size-fits-all approaches for doing so.  Successful implementation of any Agile approach requires deep understanding of Agile, the chosen Agile approach, the organizational environment, the Stakeholder Community, and the day-to-day realities of development teams.  It requires discernment and a willingness to make tough decisions.

Software developers are on the front lines of Agile transformations because they create solutions that enable business or mission success.  They have a huge stake in the direction Agile adoptions take because they are directly affected by the outcome.  Thus, they need and deserve a significant voice in their implementation.

In the next and final article of this series, I’ll discuss difficulties software developers face managing Agile and DevOps-driven changes to their role.



Business Agility Depends on Culture

While most organizations would benefit greatly from attaining business agility, many do not possess cultures amenable to Agile transformation.

Organizations, of all types, are jumping on the “business agility” or “Agile transformation” bandwagon. Business agility is the ability to compete and thrive in volatile business or mission environments by responding quickly to changing market or mission circumstances.  Agile transformations involve the application of Agile and Lean principles to business or mission processes and decision making.  Organizations seek to emulate the efficiency gains and flexibility achieved through Agile software development, but in a business or mission context. Business (or organizational) agility is the goal of Agile transformations.

The value proposition for Agile transformation is compelling.  According to proponents of Agile transformation initiatives, organizations stand to become more competitive and thrive in increasingly volatile markets and mission domains by achieving:

  • Cost savings through greater efficiency
  • Faster responses to changing market or mission conditions
  • Better business performance or mission execution

Also driving growth in the number of Agile transformation initiatives is a growing appreciation for the need to align enterprise information technology (IT) investment, governance, and operations with Agile’s iterative and incremental delivery of software system capabilities.

Figure 1: Agile Transformation Taxonomy

These incentives have fueled a multi-billion-dollar Agile transformation services industry.  According to a report published by Allied Market Research in August 2019, “… the U.S. enterprise Agile transformation services market was pegged at $4.91 billion in 2018 and is expected to hit $18.19 billion in 2026 …”.

Despite all of the money spent (or, perhaps, partly because of it), many Agile transformation initiatives fail.  We can only estimate how many, but according to Jeff Sutherland, co-creator of the Scrum framework, 47% of Agile transformations do.

The Organizational Culture Challenge

Most Agile transformation initiatives failures are tied to organizational culture.  The 14th Annual State of Agile Report, published in 2020, supports this assertion.  Four out of the top five reasons, cited by survey participants, for failed Agile scaling initiatives are directly tied to organizational culture:

Figure 2: Challenges to Scaling Agile

For organizations that develop software systems, or sponsor their development, scaling of Agile software development and system operations is a crucial part of Agile transformation.  Cultural issues that frustrate their success also negatively affect business or organizational agility.  The first and third challenges, marked in red in Figure 2 above, explicitly call out organizational culture.  The second and fourth marked challenges point to lack of commitment and participation from organizational leadership and management.  Organizational leadership is responsible for setting the tone for the rest of the organization, so a lack of support from them translates into a lack of support for Agile transformation across the organization.

Addressing the Organizational Culture Challenge

As common as the “culture” challenge is, it is hard to tackle because it is so vague.  What are the reasons for “resistance to change” and how do they manifest themselves?  What is “enough” leadership and management participation and support and what do they entail?  Beyond performing the mechanics of a chosen Agile approach (e.g., Scrum, Kanban, SAFe), when do we know the organization has achieved an “Agile culture”?

Organizational culture influences behavior.  The inverse is also true: changing behavior changes culture.  Merely exhorting people to “think Agile” and “adopt Lean thinking” is a waste of time because they likely do not know how to do so within the context of the organization and their jobs.  On the other hand, engaging in a mechanical implementation of an Agile approach, without understanding its values and principles, guarantees that existing organizational cultural biases will corrupt the approach.

Organizational culture change starts with laying out a clear and compelling vision for the organization.  The organization’s chosen Agile approach becomes the means by which the vision is implemented.  An Agile transformation effort entails some degree of uncertainty and risk.  It will likely be the first time the organization embarks on such an effort.  Mistakes will be made and lessons will be learned.  This is why closely aligning to the chosen Agile approach and understanding its values and principles is so important.

The Agile approach serves as the foundation upon which organization-specific practices eventually emerge.  While the Agile approach likely requires some degree of tailoring to ensure it aligns to the constraints and needs of the organization, such tailoring must be informed by a deep understanding of the implications of each change.  It is best to explore ways of changing existing organizational processes to align to the Agile approach before tailoring the approach to fit the organization’s processes.

As people learn how to do their jobs, in alignment with the Agile approach, through training, coaching, and actual experience, their understanding of the approach’s values and principles take root and increasingly shift organizational culture towards a shared, agile-enabled vision.

Four Organizational Characteristics Most Likely to Derail Agile Transformations

Before embarking on an Agile transformation initiative, organizational leadership would be wise take time to reflect on how well positioned they and their organizations are to achieve a successful Agile transformation.  While it is true that the only way to know for sure is to try, an honest self-assessment enables leaders to approach Agile transformation with a clear-eyed understanding of what cultural obstacles they are likely to encounter. Leaders may come to the conclusion that their organization is not ready for the transformation.  In our experience, the organizations least likely to achieve business agility through Agile transformation share the following four cultural characteristics:

  1. They emphasize process execution over actual progress
  2. They emphasize control to the detriment of speed and flexibility
  3. They discourage experimentation and punish failure
  4. They possess a consistent record of failure

1. Emphasizing Process Execution over Actual Progress

Many organizations operate under the assumption that following plans and procedures ensures good outcomes.  While no organization can operate without them, many organizations lose sight of the purpose of plans and procedures: to accomplish business or mission goals and objectives.  When this happens, the following cultural dysfunctions arise:

  • People are rewarded for following plans and procedures, rather than for achieving desired outcomes
  • Failure is accepted so long as procedures were followed
  • Project status reporting tracks process activity completion rather than progress towards desired outcomes

One of the 12 Principles of Agile Software Development is, “Working software is the primary measure of progress.”  The only way to validate the value of any software capability is by getting it into the hands of customers and end users.  Only they can determine whether expected business or mission outcomes were achieved through use of the software. 

Similarly, the effectiveness of business plans and processes should be validated against business or mission outcomes.  Doing so involves quantifying expected outcomes and continually measuring their attainment against relevant metrics.  Failure to achieve desired outcomes should prompt process changes or the formulation of new processes.  This empiricism is a critical characteristic of Agile and Lean.

2. Emphasizing Control to the Detriment of Speed and Flexibility

The point of carrying out an Agile transformation is to achieve business or organizational agility.  Agile Cheetah defines agility as the combination of speed, control, and flexibility:

Agility = Speed + Control + Flexibility

The elements of this “Agility Triad” are mutually dependent constraints.  In other words, more or less of any one constraint influences the other two.  For example:

  • Accelerating the pace of work requires some degree of control and flexibility
  • Overemphasizing control slows progress and decreases flexibility
  • Underemphasizing control can turn flexibility into chaos, robbing teams of both planning and process stability, thereby slowing progress

Many organizations are guilty of the second bullet.  Overemphasizing control to the detriment of speed and flexibility is emblematic of fear-based organizational cultures.  In such cultures, Taylorism prevails.  Organizational structure tends to be hierarchical with a strong command-and-control management approach.  There is often an unwillingness to delegate decision making down to the most responsible levels.  In the most extreme cases, position, status, and tenure trump expertise, knowledge, and skills.

Source: geek & poke under CC-BY-3.0 license. No changes made.

Rather than accepting that some amount of risk is an inevitable part of solving tough problems, fearful organizations avoid taking risks.  Fear of failure produces cultures where failure to act is deemed preferable to acting and failing.  When failure happens, finding blame takes precedence over finding solutions.  Additional control mechanisms are put in place to ensure it does not happen again. 

Unfortunately, such mechanisms often feed a vicious cycle where people lose some autonomy and latitude to act, resulting in a loss of effectiveness, which breeds further failure, which then prompts further controls.  Left unchecked, this cycle can whittle down organizational effectiveness to the point where the organization loses the ability to reach previous levels of performance.

Figure 3: Vicious Cycle of Degraded Organizational Performance

Organizations with strong hierarchical and command-and-control cultures struggle through Agile transformation more than those with more egalitarian and permissive cultures.  Leaders should ask themselves; how much authority are they willing to delegate to achieve greater flexibility and speed?  What changes need to happen to enable their people to assume greater ownership and responsibility for their work and resulting outcomes?

3. Discouraging Experimentation and Punishing Failure

Organizations that overemphasize control also tend to discourage experimentation and punish failure.  Rather than exploring different solution options, they rush to settle on one approach.  Time that could have been spent investigating options is spent improving a solution that was likely suboptimal from the start.  Fear of ambiguity and risk leads to the formulation of unrealistic plans (In search for certainty, they conjure fantasies).  Premature commitment to plans takes away the flexibility needed to adjust to changing circumstances and new information.  Closed off to alternatives and fearing the consequences of admitting failure, the organization continues to sink resources into failing approaches.

As mentioned earlier, Agile and Lean are based on empiricism.  In an empirical process, information gained through experimentation and observation informs and validates plans, decisions, and actions.  Planning is adaptive as opposed to predictive.  In other words, Agile plans change based on current circumstances and new information and insights. 

Agile plans skew towards shorter planning horizons measured in weeks to months rather than months to years.  When problems or mistakes happen, Agile teams re-plan and correct course.  Short planning horizons afford the flexibility to make new plans and pivot to new solution approaches.

Like experiments, plans and decisions are treated as a hypothesis.  When a hypothesis proves to be wrong, another one is formulated and tried.  The faster teams cycle through these learning cycles, the faster they arrive at solutions that provide real business or mission value, cost effectively, and in time to be valuable.

4. Possessing a Consistent Record of Failure

Agile is not a silver bullet.  Agile, on its own, does not solve problems, it makes them plain for everyone to see so they may take action.  Transforming organizations from traditional project management structures to Agile requires courage, vision, and competence.  Organizations with long and consistent track records of underperformance and failure are most likely to fail at implementing Agile transformations.

Underperforming organizations may actually be missing core competencies that must be developed or acquired prior to embarking on any type of organizational transformation.  Such organizations are more likely to corrupt their chosen Agile approach to accommodate their deficiencies.

Conclusion

While most organizations would benefit greatly from attaining business agility, many do not possess cultures amenable to Agile transformation.  However, cultural change results from changing behavior.  Organizational leaders maximize the odds of successful Agile transformations by acknowledging organizational culture problems and addressing them before and during Agile transformation initiatives.