In this video, Frank Velazquez explains how Agile accelerates value delivery.
Many organizational leaders think of Agile as a way to speed up software development, and nothing more. They approach Agile with a traditional project management emphasis on improving efficiency. In doing so, they unwittingly corrupt Agile practices to the point of undermining the very agility they are trying to achieve.
Agile software development teams develop more software functionality faster by focusing on diminishing what Lean terms lead time and cycle time. Agile teams diminish lead time and cycle time by:
Minimizing waste from software development processes
Managing technical scope (increasing flow)
Striving for high quality through technical excellence
Increase development and organizational agility, and better serve stakeholders, by integrating Agile, DevOps, and microservices approaches.
Well-run organizations invest in information systems to achieve specific and measurable goals and objectives. They determine the return on information technology (IT) investments by how much they lower cost, increase productivity, lower risk, and increase quality. This expectation of business or mission value achieved for money spent carries over to how organizations conceive, develop, field, and maintain information systems. Valuable information systems allow users to leverage technology in ways that improve how they work to accomplish organizational goals and objectives.
Improving this alignment between business or mission needs and information systems development has been the holy grail of analysts, engineers, and managers since the dawn of the Information Age. Unfortunately, past engineering and management approaches actually widened the communication gap between business / mission stakeholders and software developers. This state of affairs made information systems:
More expensive to develop and operate;
Take longer to develop than expected;
Less likely to generate expected value after fielding; and
More likely to be cancelled before initial fielding.
Over the last 20 years, innovations in how we develop and field software systems, of all kinds, are breaking down barriers to collaboration between business / mission stakeholders and development teams. Arguably, the three most important innovations are Agile software development, DevOps, and Microservices Architecture.
Here, I discuss how the three approaches can work together to address the aforementioned shortfalls. We aim to foster tighter collaboration between business / mission stakeholders and development teams as we improve development and organizational agility.
The Agility Triad
Today, mature software development organizations can maximize development and organizational agility by integrating Agile software development, DevOps and microservice architectural practices. Agile Cheetah defines agility as:
Agility = Speed + Control + Flexibility
We call this definition “The Agility Triad.” The Agility Triad’s elements apply to organizational, process, and technological agility. The three elements are mutually dependent and reinforcing constraints defined as follows:
Figure 1: Agility Triad Elements Definitions
DevOps and Microservices Contribute to Agility
DevOps and microservice architectures provide the means of implementing the Agility Triad’s elements. DevOps serves as the Agile implementation of the Software Development Lifecycle (SDLC). Microservices help align software deliverables directly to the business / mission domain and requirements. They also help structure software development teams, their work, and what they deploy to directly address business or mission needs.
DevOps as the Agile SDLC
Agile projects perform the same SDLC activities as Waterfall projects. The difference is that software teams execute the Agile SDLC as a continuous flow of activity rather than as a linear, one-time progression through time-bound phases.
Throughout development, Agile teams define, develop, test, deploy, and release features. Rather than waiting until the end of the project for a finished system, project stakeholders, customers, and end users gain access to released features throughout the project. These features, individually and collectively, are valuable business / mission capabilities ready for use upon release.
Every feature goes through its own mini–Agile SDLC iteration. Mature Agile teams leverage automation to continually test and integrate features into complete increments of releasable functionality. Newly deployed and released increments work with all previous increments. The system grows and evolves iteratively and incrementally.
At the project level, the Agile SDLC is essentially a “swirl” or “flow” of activity occurring concurrently across multiple features and increments. The most mature Agile teams release so quickly and often that their releases approach a continuous flow.
DevOps is the implementation of this flow-based approach to software development. DevOps is not an Agile concept, nor did it start with Agile. However, it aligns perfectly with Agile because it reinterprets the SDLC in ways that support Agile’s iterative and incremental value delivery and team-based culture. It also aligns with Lean’s emphasis on achieving flow and promoting continuous improvement.
Ways DevOps Enables Agility
I now explain how DevOps enables the three elements of the Agility Triad.
Figure 3: The Agility Triad
DevOps Contribution to Project Speed
DevOps accelerates software delivery by leveraging modern software development techniques, tools, and technologies and by eliminating wasteful handoffs between development teams and operations staff. This culture of automation and continuous improvement facilitates faster iterations through the SDLC for every feature developed.
DevOps automation goes beyond automating the testing, packaging, and deployment of source code. Automated building and configuration of host environments is also a critical facet of DevOps. This capability is termed, “Infrastructure as Code (IaC).”
IaC allows for the repeatable provisioning and configuration of host environments via execution of code. Rather than incurring the time and cost of manually setting up environments, infrastructure automation platforms provision and configure virtual environments by executing scripts and implementing configuration files.
Project management can make decisions about the state of the project based, not on estimated levels of completion but, on information gained through actual deployment, release, and use of functionality. This arms managers with the information they need to decide whether to continue the current course, update project plans, or terminate the project. All along, teams deliver complete increments of functionality capable of providing value immediately.
DevOps Contribution to Project Flexibility
DevOps contributions to software project flexibility include:
Faster and more frequent iterations through the SDLC that allow experimentation with different approaches and quickly pivoting away from failing or undesirable approaches.
The ability to separate software deployments in production from releases, which enables:
Testing in Production – Teams can deploy and test new software features in production before making them available to users. This allows teams to experiment with different versions of the same functionality and see which performs best.
Releasing on Demand – Rather than deploying and releasing on a fixed schedule, teams can deploy functionality at any time and make it available to users when they are ready to use it.
Monolithic vs. Microservices Architectures
Traditional software projects delivered solutions all at once. In other words, software teams deployed and released all application components as one unit. Changes to an application often required complete redeployment. In microservices parlance, we refer to applications architected this way as monoliths.[1]
While architects and developers typically split monoliths into packages, modules, or components, interdependencies between those components make it difficult to develop, test, and deliver them separately. In systems engineering terms, these system components are “tightly coupled.”
A microservices architecture, on the other hand, builds applications as sets of independently deployable services that represent key entities of a particular business or mission domain. For example, an inventory system may include services that handle transactions associated with customers, invoices, warehouses, payment, etc. All of the information / data associated with each of these entities belongs to its respective service. A service needing information from outside the confines of its own entity (e.g., the warehouse service needs a customer’s address) must get it from the corresponding service.
Three-Tiered (Web) Applications
Since the 1990’s, the predominant architectural model for monolith applications has been the three-tiered (web) application. This distributed architecture grouped together components based on where they fit within each of the following tiers:
Figure 4: Three-Tiered (Web) App Monolith
Front End: Implements the application’s user interface(s) or externally-facing interface(s)/API(s)
Middle Tier: Performs the bulk of the application’s business logic
Datastore: Manages application data and state
Microservices Cut Across Three-Tiered Architectures
In contrast, since microservices are end-to-end slices of business or mission functionality, each microservice includes just enough of the three tiers to perform its job.
Figure 5: Each Microservice Includes Components from Each Tier
Individual microservices are not coupled to the internal workings of other services. Thus, microservices are independent or “loosely coupled.” As long as the interfaces they expose do not change, internal changes to individual microservices will not ripple across the application or system. Teams can deploy and release microservices independently, allowing applications to grow and evolve iteratively and incrementally.
Figure 6: Microservice Composition
Features Map to Microservices
As stated earlier, Agile value delivery is feature-based. This differs from the component-based delivery of functionality performed in traditional software projects.
In traditional software projects, management organizes teams to develop system components which they later integrate into complete solutions. Until that integration happens, nothing of business or mission value exists. Complete and cohesive functionality / features only exist after system integration.
System integration occurs infrequently in traditional software projects. This increases project risk. Validation of an application’s actual business / mission value typically occurs towards the end of the project, when budget and schedule are scarce.
As end-to-end slices of business or mission functionality, microservices help development teams develop, test, and deliver applications, iteratively and incrementally, as features, from the start. A feature may be satisfied by one microservice or through orchestration of multiple microservices.
Ways Microservices Enable Agility
Agile approaches, such as Scrum, help teams organize and manage their work as increments of functionality (features / user stories), developed within short timeframes (timeboxes), as part of frequent releases. DevOps provides the automation and coordination required to continually and sustainably deliver features agilely. Microservices are as close as we have come to directly implementing, deploying, and releasing requested features as features.
Microservices Contribution to Project Speed
Microservices accelerate software applications development and fielding by:
Breaking down application scope into smaller and more manageable units of work – A concept of Lean is that smaller batches of work increase flow. Microservices make it easier for development teams to manage their work through Agile and DevOps practices.
Allowing customers and end users access to valuable features earlier – Rather than waiting until full system deployment to validate and use a solution, customers and end users gain access to useful functionality throughout development. Lessons learned from released functionality help inform ongoing development.
Allowing development teams to work in parallel – Development teams can work on different microservices in parallel with little worry about inter-service dependencies beyond the interfaces the services expose.
Facilitating mixing and matching of technologies and technical approaches – Microservice independence also applies to the technology stacks and platforms used to implement them. Teams can develop microservices in whatever technologies they are most comfortable with and still achieve service integration and interoperability. Since microservices communicate via networks, they do not need to run on the same platform or under the same operating system. This technology independence lowers the amount of learning and coordination required across development teams.
Lowering the risk associated with fielding new functionality – Adding or replacing individual microservices is much easier and less risky than redeploying monolith applications. This flexibility lowers outage times.
Microservices Contribution to Controlling Projects
Microservices help teams and project management control projects by:
Facilitating Development Around Features – Agile teams control projects by managing scope. Breaking up system scope into features helps teams manage their work. Microservices support the iterative and incremental development and release of features. This strategy affords teams multiple opportunities throughout development to evaluate progress and update project plans based on actual use.
Lowering Inter-Team Dependencies – Agile software development teams are cross-functional. Members possess the skills necessary to develop complete, coherent, and deployable features across short timeboxes. This organization around features, rather than components, aligns with the tenet of building microservices that include all of the components necessary to perform as independently deployable services. Organizing teams along the lines of the microservices they develop lowers dependencies across teams and associated integration risks.
Figure 7: Tier-Based / Component Teams vs. Microservices / Feature Teams
Microservices Contribution to Project Flexibility
Spreading application scope across individual microservices simplifies changes and promotes experimentation. Teams can change their piece of the application without fear of affecting the rest. Thus, there is less resistance to trying out different approaches and settling on the most optimal. For example, we can deploy different versions of the same microservice with different user interfaces in production, selectively release them to users, and evaluate them. This technique is known as A/B testing.
Summary of DevOps and Microservices Contributions to Agility
I summarize the contributions DevOps and Microservice Architecture can make to agility below:
Figure 8: DevOps and Microservices Contributions to Agility
Conclusion
Proper integration of Agile, DevOps, and microservice architectural approaches can narrow the gulf between business or mission stakeholders and those who deliver software solutions. We do this by improving development and organizational agility. Under this construct:
Agile planning and execution are supported by DevOps processes
DevOps processes accelerate development, testing, packaging, and deployment of microservice-based applications
Resulting applications are better aligned with Agile delivery and business / mission needs
The decision to adopt any of the three approaches is not be taken lightly. There is no one-size-fits-all approach to making them work and few organizations make all three work well. It may also be the case that any of three approaches are not right for your organization or specific project.
However, these approaches are firmly in the canon of modern software development practice. It is worthwhile investigating how your organization can improve how it practices these approaches or begin adopting them.
[1] Sam Newman provides a comprehensive explanation of the ideas covered in this section in his book: Monolith to Microservices, 2020, O’Reilly, Chp 1
Many organizational leaders think of Agile as a way to speed up software development, and nothing more. They approach Agile with a traditional project management emphasis on improving efficiency. In doing so, they unwittingly corrupt Agile practices to the point of undermining the very agility they are trying to achieve. One of the most obvious signs of this are high-pressure work environments where people work harder and longer to complete unrealistic amounts of work against arbitrary deadlines.
People can work at an accelerated pace for only so long before becoming fatigued, physically and mentally. As time wears on and stress builds, morale and quality suffer. The results are inefficient and ineffective development teams, low quality software; and unhappy sponsors, stakeholders, customers, and end users.
Ultimately, these corrupted Agile projects fail, leaving burned out teams and failed projects in their wake.
Speed is One Characteristic of Agility
At Agile Cheetah, we see speed as one of three characteristics of agility: Speed, Control, and Flexibility. We call this formulation “The Agility Triad”.
The Agility Triad’s Three Constraints
The Agility Triad is not a concept of either Agile or Scrum. It is a construct we developed to organize discussions concerning:
What agility means from the perspective of technical projects in general and software development projects specifically
Concepts, techniques, and practices from the fields of Lean process management, organizational behavior, product management, and project management that support or impact agility
The triad’s constraints are mutually dependent. In other words, more or less of any one constraint influences the other two. For example, accelerating the pace of software delivery requires some degree of control and flexibility. Overemphasizing control slows software development and decreases flexibility. Underemphasizing control can turn flexibility into chaos, robbing software development teams of both planning and process stability, thereby slowing progress.
In this article, we focus on the speed constraint of the Agility Triad.
Accelerating the Pace Agilely
From an Agile perspective, developing software solutions faster is not about working harder and faster. Agile teams plan and execute work in ways that make working at a sustainable pace the norm.
Our definition of speed includes two considerations: the timeframe within which we deliver functionality and ensuring that whatever we release is potentially-releasable.
By timeframe, we mean how long does it take to release functionality that addresses customer or end user requirements or needs (i.e., diminishing lead time and cycle time)
By potentially-releasable, we mean that the software functionality is ready for deployment and use, by customers and end users, should a relevant release authority decide to release it
Diminishing Lead Time and Cycle Time
Agile software development teams develop more software functionality faster by focusing on diminishing what Lean terms lead time and cycle time.
Lead Time and Cycle Time
Lead time is the time from when a customer orders a product to when the customer receives it. Cycle time is the time between gaining authorization to produce an order and completion of a deliverable product. When speaking about software development, we replace the word product with functionality, feature, capability, component, etc.
Thinking in terms of lead time and cycle time brings to light ways to improve how we develop software and the software solutions themselves. This work is ongoing for as long as the solution exists. It is a “Continuous Improvement” mindset.
Development teams are not solely responsible for diminishing lead time and cycle times. Since teams depend on their sponsor organizations for support, sponsor organizations must adopt a continuous improvement mindset as well. For example, slow customer order approval and product delivery processes add to lead time. Not having necessary production resources in place adds to cycle time, and by extension, lead time. The more lead time and cycle time look the same, the better.
Agile teams diminish lead time and cycle time by:
Minimizing waste from software development processes
Managing technical scope (increasing flow)
Striving for high quality through technical excellence
Minimizing Waste
The Lean Enterprise Institute defines Lean as a set of management practices that maximize customer value while minimizing waste. Lean takes a continuous improvement approach to eliminating waste. While it is not possible to eliminate all instances of waste, Lean teams take every opportunity to identify and remove wasteful practices. Waste does not always result from unnecessary or deficient process inputs, outputs, and activities. Sometimes waste is too much of what is needed, like excess product inventory for example.
Lean identifies seven sources of waste (TIMWOOD):
Lean’s Seven Sources of Waste
Managing Technical Scope
Treating project scope as variable, while keeping cost and schedule fixed (i.e., “Breaking the Iron Triangle”) accelerates software development and releases. Agile teams define project scope as increments of functionality developed and released throughout a project. Rather than defining the entire solution at the beginning, the solution takes shape over the course of development.
Agile development is iterative and incremental. Agile teams break up project scope into batches of functionality to produce (e.g., features, components). By keeping batches small, teams process each batch faster, thus accelerating the overall throughput of completed work (flow). Each new batch of functionality works with the aggregate of all previously released batches. That aggregate of functionality is the evolving solution. The bonus is that, as long as released functionality provides real business or mission value, users can use the solution as it “emerges” rather than awaiting release of the entire solution.
Quality through Technical Excellence
High-quality software is faster and cheaper to produce and maintain than low quality software. Development teams spend less time fixing and refactoring well architected, properly designed, well-written, thoroughly tested code. Quality software is easier to deploy, configure, and upgrade, shrinking the amount of time development teams spend on operations and maintenance (O&M). The common theme is saving time. Software development teams have more time to spend developing new functionality and improving how they work, which translates into faster and more frequent releases.
Software development teams achieve high technical quality by striving for technical excellence. Like eliminating wasteful process activities, technical excellence is a pursuit rather than an achievement. Technical excellence entails:
Adopting technical practices and techniques that improve software development (e.g., DevOps, Test Driven Development (TDD), Feature Driven Development (FDD))
Committing to ongoing technical professional development
Working effectively as part of cohesive technical teams
Conclusion
While increasing the pace and frequency of software development and delivery are key tenets of Agile’s value proposition, we must do so sustainably. This requires a shift in management thinking from a traditional project management emphasis on managing people and tasks to empowering teams. In an Agile context, empowering teams includes: ensuring they have the right skills; equipping them with the right tools; and giving them enough latitude to define and manage their own work while establishing clear business/mission intent and boundaries. In other words, speed comes from continually improving teams and the solutions they deliver.
In part one of this blog series, I explained what “Agilish” projects are and why we should avoid them. In this article, I discuss the reasons why government Agile initiatives tend to move away from Agile principles and towards becoming primarily Waterfall projects with Agile practices added (Agilish).
As I stated in part one, my perspective comes from years of working Agile projects for U.S. Federal Government clients. While many of the references are specific to that environment, equivalents exist at all levels of government, as well as in many private organizations, worldwide.
Process over Outcomes
Many government organizations possess process-centered cultures where progress is judged more on process execution than on outcomes achieved. The fundamental assumption being that if people correctly follow processes and procedures, desired outcomes will follow.
It is easy to see why. Governments are much more constrained than private organizations by laws, regulations, and policies (LRPs). Government institutions are ultimately accountable to the public, thus they must abide by extensive oversight rules. Documenting and enforcing processes and procedures help to ensure adherence to the law and to promote fairness and equality in government dealings.
With respect to managing software projects, the issue is not so much about having processes and procedures. They are necessary. The issue is about replacing judgement and understanding with highly-detailed and prescriptive processes and procedures. Much of current U.S. Federal acquisitions and contracting LRPs were written with traditional project management in mind. When carried too far, this urge to codify and template everything results in risk adverse acquisitions and contracting cultures hamstrung by their own rules.
This mentality leads to mechanistic Agile implementations that are at odds with Agile. Agencies “tailor” their Agile approaches to fit within their existing rules, processes, and procedures. While there are limits to how much agencies can do to align acquisitions and contracting processes to Agile, government agencies tend to overlook (and often reject) opportunities to do so.
Blaming Waterfall
For many organizations, years (if not decades) of managing
software development projects through traditional project management coincide
with years of project failures.
Frustration over the inability to deliver planned project scope on time,
on budget, and at acceptable levels of quality serves as a powerful motivator
for instituting Agile practices.
While traditional project management is increasingly out of step with modern software development, for many organizations, Waterfall is not the root cause of software project failures. Many organizations, public and private, lack the programmatic, engineering, and governance maturity necessary to effectively manage complex technology implementations, regardless of approach.
Organizations struggling to deliver under traditional project management processes will likely struggle even more using Agile. You see, Agile does not on its own fix problems; it makes them plainly visible for people to take action. Agile is a catalyst for change, not a solution.
When lack of programmatic, engineering, and governance maturity render organizations unable to address problems uncovered by Agile, they often retreat into familiar ways of working more in line with their actual capabilities. This, rather than traditional project management processes and culture, may be the single greatest obstacle to government Agile adoption.
Waterfall-Based Funding Mechanisms
Government builds software systems the way it funds them. For the most part, government agencies have not done enough to align how they fund software systems development with Agile and DevOps. This lack of alignment is a principal factor in turning potential Agile projects Agilish and is most prominently manifested by:
Establishing projects rather than managing services
Adhering to outmoded “color of money” funding allocations between new development, test and evaluation, and O&M support
The Trouble with Projects
The trouble with structuring software system development and funding around projects is that projects begin and end. A lot of coordination, planning, contracts management, and funding allocations happen before project start. That overhead is costly and time consuming, especially in government projects. These activities frustrate the very agility organizations try to achieve by imposing a predictive planning approach on software development.
Traditional projects afford one opportunity to get the solution right. As a result, there is a great deal of pressure to include as many requirements as possible for a wide range of users. This increases project scope and, consequently, increases project complexity, planning overhead, risk, and cost. Organizations also lose flexibility to address new requirements, manage risks, and seize opportunities during development.
All of this predictive planning relies exclusively on estimates against a plan or program baseline. If the plan turns out to be sufficiently wrong, a new plan is formulated (the program is re-baselined). Changes to traditional project plans are highly disruptive, so project managers work mightily to avoid them. In many cases, this effectively prevents projects from acting on empirical reality, until it is too late.
As I described in part one, software systems evolve for as long as they are in operation. We cannot accurately predict what customer needs and priorities will be months or years from now. Requirements are highly perishable, sometimes to the point that software systems are obsolete or irrelevant by the time they are fielded. A funding model that aligns with Agile and DevOps treats software systems as long-standing managed services rather than as serial “one-time” investments.
“Color of Money” Concerns
As stated in part one, Agile and DevOps compress planning, analysis, design, testing, and release activities into short, fast, and frequent iterations or cycles. Under DevOps, software development and operations teams share development and production support duties. Development and maintenance work are included in the ongoing flow of work done by teams. Traditional discrete project phases for these activities, performed by separate teams, no longer happen under Agile and DevOps. Thus, it does not make sense to segregate funding and related planning along the lines of Waterfall project phases.
The US Federal Government and the Department of Defense (DoD) in particular, are coming to this realization. Unfortunately, it will take changes in law and policy to move away from “color of money” funding mechanisms and that moves slowly.
Earned Value Management (EVM) Requirements
Applying Earned Value Management (EVM) to Agile projects imposes a vastly different definition of value on Agile. Agile does not attempt to quantify value in terms of dollars and cents. Instead, the focus is on delivering capabilities that project stakeholders, customers, and end users consider most important to mission accomplishment. Imposing EVM requirements on Agile projects tends to subvert Agile practices to accommodate the needs of EVM.
Vendors Switching to Agile while Government Lags Behind
Agile software development is the new normal in IT. Each succeeding batch of newly minted software development professionals entering software development is increasingly less likely to work on Waterfall projects. This reality is now a forcing function on government agencies to change how they work with vendors to support agility. Some agencies are doing better than others, but organizational cultures do not change overnight.
Conclusion
Recognizing the driving factors behind Agilish projects helps us understand the organizational environments within which we attempt to institute Agile approaches. Such understanding is invaluable to successful Agile adoptions/transformations, specially in the government space. Success depends on leveraging existing LRPs to align processes with Agile. While “textbook” Agile may never be realistic in many government settings, achieving the benefits of Agile requires us to understand the trade offs involved and strive to move towards greater organizational agility whenever possible.
In the third and final article of this series, I’ll cover ways to make Agilish projects more Agile.
