Well-run organizations invest in information systems to achieve specific and measurable goals and objectives. They determine the return on information technology (IT) investments by how much they lower cost, increase productivity, lower risk, and increase quality. This expectation of business or mission value achieved for money spent carries over to how organizations conceive, develop, field, and maintain information systems. Valuable information systems allow users to leverage technology in ways that improve how they work to accomplish organizational goals and objectives.

Improving this alignment between business or mission needs and information systems development has been the holy grail of analysts, engineers, and managers since the dawn of the Information Age. Unfortunately, past engineering and management approaches actually widened the communication gap between business / mission stakeholders and software developers. This state of affairs made information systems:

More expensive to develop and operate;
Take longer to develop than expected;
Less likely to generate expected value after fielding; and
More likely to be cancelled before initial fielding.

Over the last 20 years, innovations in how we develop and field software systems, of all kinds, are breaking down barriers to collaboration between business / mission stakeholders and development teams. Arguably, the three most important innovations are Agile software development, DevOps, and Microservices Architecture.

Here, I discuss how the three approaches can work together to address the aforementioned shortfalls. We aim to foster tighter collaboration between business / mission stakeholders and development teams as we improve development and organizational agility.

The Agility Triad

Today, mature software development organizations can maximize development and organizational agility by integrating Agile software development, DevOps and microservice architectural practices. Agile Cheetah defines agility as:

Agility = Speed + Control + Flexibility

We call this definition “The Agility Triad.” The Agility Triad’s elements apply to organizational, process, and technological agility. The three elements are mutually dependent and reinforcing constraints defined as follows:

DevOps and Microservices Contribute to Agility

DevOps and microservice architectures provide the means of implementing the Agility Triad’s elements. DevOps serves as the Agile implementation of the Software Development Lifecycle (SDLC). Microservices help align software deliverables directly to the business / mission domain and requirements. They also help structure software development teams, their work, and what they deploy to directly address business or mission needs.

DevOps as the Agile SDLC

Agile projects perform the same SDLC activities as Waterfall projects. The difference is that software teams execute the Agile SDLC as a continuous flow of activity rather than as a linear, one-time progression through time-bound phases.

Agile projects are feature-based. In the context of Agile, features are cohesive and complete slices of system functionality capable of performing useful work from the perspective of customers and end users (human or automated).

Throughout development, Agile teams define, develop, test, deploy, and release features. Rather than waiting until the end of the project for a finished system, project stakeholders, customers, and end users gain access to released features throughout the project. These features, individually and collectively, are valuable business / mission capabilities ready for use upon release.

Every feature goes through its own mini–Agile SDLC iteration. Mature Agile teams leverage automation to continually test and integrate features into complete increments of releasable functionality. Newly deployed and released increments work with all previous increments. The system grows and evolves iteratively and incrementally.

At the project level, the Agile SDLC is essentially a “swirl” or “flow” of activity occurring concurrently across multiple features and increments. The most mature Agile teams release so quickly and often that their releases approach a continuous flow.

DevOps is the implementation of this flow-based approach to software development. DevOps is not an Agile concept, nor did it start with Agile. However, it aligns perfectly with Agile because it reinterprets the SDLC in ways that support Agile’s iterative and incremental value delivery and team-based culture. It also aligns with Lean’s emphasis on achieving flow and promoting continuous improvement.

Ways DevOps Enables Agility

I now explain how DevOps enables the three elements of the Agility Triad.

DevOps Contribution to Project Speed

DevOps accelerates software delivery by leveraging modern software development techniques, tools, and technologies and by eliminating wasteful handoffs between development teams and operations staff. This culture of automation and continuous improvement facilitates faster iterations through the SDLC for every feature developed.

DevOps automation goes beyond automating the testing, packaging, and deployment of source code. Automated building and configuration of host environments is also a critical facet of DevOps. This capability is termed, “Infrastructure as Code (IaC).”

IaC allows for the repeatable provisioning and configuration of host environments via execution of code. Rather than incurring the time and cost of manually setting up environments, infrastructure automation platforms provision and configure virtual environments by executing scripts and implementing configuration files.

DevOps Contribution to Controlling Projects

Faster and more frequent iterations through the SDLC support greater control of project scope, budget, and schedule through increased empiricism. In other words, Agile teams and project management monitor progress, make decisions, and act based on lessons learned from previous releases. The faster and more frequently teams iterate through the SDLC, the faster and more frequently they receive feedback that informs upcoming work plans (scope).

Project management can make decisions about the state of the project based, not on estimated levels of completion but, on information gained through actual deployment, release, and use of functionality. This arms managers with the information they need to decide whether to continue the current course, update project plans, or terminate the project. All along, teams deliver complete increments of functionality capable of providing value immediately.

DevOps Contribution to Project Flexibility

DevOps contributions to software project flexibility include:

Faster and more frequent iterations through the SDLC that allow experimentation with different approaches and quickly pivoting away from failing or undesirable approaches.
The ability to separate software deployments in production from releases, which enables:
- Testing in Production – Teams can deploy and test new software features in production before making them available to users. This allows teams to experiment with different versions of the same functionality and see which performs best.
- Releasing on Demand – Rather than deploying and releasing on a fixed schedule, teams can deploy functionality at any time and make it available to users when they are ready to use it.

Monolithic vs. Microservices Architectures

Traditional software projects delivered solutions all at once. In other words, software teams deployed and released all application components as one unit. Changes to an application often required complete redeployment. In microservices parlance, we refer to applications architected this way as monoliths.[1]

While architects and developers typically split monoliths into packages, modules, or components, interdependencies between those components make it difficult to develop, test, and deliver them separately. In systems engineering terms, these system components are “tightly coupled.”

A microservices architecture, on the other hand, builds applications as sets of independently deployable services that represent key entities of a particular business or mission domain. For example, an inventory system may include services that handle transactions associated with customers, invoices, warehouses, payment, etc. All of the information / data associated with each of these entities belongs to its respective service. A service needing information from outside the confines of its own entity (e.g., the warehouse service needs a customer’s address) must get it from the corresponding service.

Three-Tiered (Web) Applications

Since the 1990’s, the predominant architectural model for monolith applications has been the three-tiered (web) application. This distributed architecture grouped together components based on where they fit within each of the following tiers:

Front End: Implements the application’s user interface(s) or externally-facing interface(s)/API(s)
Middle Tier: Performs the bulk of the application’s business logic
Datastore: Manages application data and state

Microservices Cut Across Three-Tiered Architectures

In contrast, since microservices are end-to-end slices of business or mission functionality, each microservice includes just enough of the three tiers to perform its job.

Individual microservices are not coupled to the internal workings of other services. Thus, microservices are independent or “loosely coupled.” As long as the interfaces they expose do not change, internal changes to individual microservices will not ripple across the application or system. Teams can deploy and release microservices independently, allowing applications to grow and evolve iteratively and incrementally.

Features Map to Microservices

As stated earlier, Agile value delivery is feature-based. This differs from the component-based delivery of functionality performed in traditional software projects.

In traditional software projects, management organizes teams to develop system components which they later integrate into complete solutions. Until that integration happens, nothing of business or mission value exists. Complete and cohesive functionality / features only exist after system integration.

System integration occurs infrequently in traditional software projects. This increases project risk. Validation of an application’s actual business / mission value typically occurs towards the end of the project, when budget and schedule are scarce.

As end-to-end slices of business or mission functionality, microservices help development teams develop, test, and deliver applications, iteratively and incrementally, as features, from the start. A feature may be satisfied by one microservice or through orchestration of multiple microservices.

Ways Microservices Enable Agility

Agile approaches, such as Scrum, help teams organize and manage their work as increments of functionality (features / user stories), developed within short timeframes (timeboxes), as part of frequent releases. DevOps provides the automation and coordination required to continually and sustainably deliver features agilely. Microservices are as close as we have come to directly implementing, deploying, and releasing requested features as features.

Microservices Contribution to Project Speed

Microservices accelerate software applications development and fielding by:

Breaking down application scope into smaller and more manageable units of work – A concept of Lean is that smaller batches of work increase flow. Microservices make it easier for development teams to manage their work through Agile and DevOps practices.
Allowing customers and end users access to valuable features earlier – Rather than waiting until full system deployment to validate and use a solution, customers and end users gain access to useful functionality throughout development. Lessons learned from released functionality help inform ongoing development.
Allowing development teams to work in parallel – Development teams can work on different microservices in parallel with little worry about inter-service dependencies beyond the interfaces the services expose.
Facilitating mixing and matching of technologies and technical approaches – Microservice independence also applies to the technology stacks and platforms used to implement them. Teams can develop microservices in whatever technologies they are most comfortable with and still achieve service integration and interoperability. Since microservices communicate via networks, they do not need to run on the same platform or under the same operating system. This technology independence lowers the amount of learning and coordination required across development teams.
Lowering the risk associated with fielding new functionality – Adding or replacing individual microservices is much easier and less risky than redeploying monolith applications. This flexibility lowers outage times.

Microservices Contribution to Controlling Projects

Microservices help teams and project management control projects by:

Facilitating Development Around Features – Agile teams control projects by managing scope. Breaking up system scope into features helps teams manage their work. Microservices support the iterative and incremental development and release of features. This strategy affords teams multiple opportunities throughout development to evaluate progress and update project plans based on actual use.
Lowering Inter-Team Dependencies – Agile software development teams are cross-functional. Members possess the skills necessary to develop complete, coherent, and deployable features across short timeboxes. This organization around features, rather than components, aligns with the tenet of building microservices that include all of the components necessary to perform as independently deployable services. Organizing teams along the lines of the microservices they develop lowers dependencies across teams and associated integration risks.

Microservices Contribution to Project Flexibility

Spreading application scope across individual microservices simplifies changes and promotes experimentation. Teams can change their piece of the application without fear of affecting the rest. Thus, there is less resistance to trying out different approaches and settling on the most optimal. For example, we can deploy different versions of the same microservice with different user interfaces in production, selectively release them to users, and evaluate them. This technique is known as A/B testing.

Summary of DevOps and Microservices Contributions to Agility

I summarize the contributions DevOps and Microservice Architecture can make to agility below:

Conclusion

Proper integration of Agile, DevOps, and microservice architectural approaches can narrow the gulf between business or mission stakeholders and those who deliver software solutions. We do this by improving development and organizational agility. Under this construct:

Agile planning and execution are supported by DevOps processes
DevOps processes accelerate development, testing, packaging, and deployment of microservice-based applications
Resulting applications are better aligned with Agile delivery and business / mission needs

The decision to adopt any of the three approaches is not be taken lightly. There is no one-size-fits-all approach to making them work and few organizations make all three work well. It may also be the case that any of three approaches are not right for your organization or specific project.

However, these approaches are firmly in the canon of modern software development practice. It is worthwhile investigating how your organization can improve how it practices these approaches or begin adopting them.

[1] Sam Newman provides a comprehensive explanation of the ideas covered in this section in his book: Monolith to Microservices, 2020, O’Reilly, Chp 1

Let us know what you think!!!

I am astonished at the speed with which DevOps has gained traction across private industry and government organizations. This is especially surprising to me with respect to the Federal Government, which is often a (very) late adopter of new IT technologies and methodologies. Although, to my knowledge, no definitive studies exist that confirm this, DevOps seems to be gaining acceptance and adoption within the Federal Government at a faster rate than Agile did during the last 16 years.

The general consensus of when DevOps started is 2008, the year Andrew Shafer and Patrick Debois met over a talk on “Agile Infrastructure.” I worked my first Agile project as a developer that year. The Agile Manifesto was published seven years prior and adoption of Agile software development within Federal software development programs was in its infancy. Within private industry, Agile development was not as prominent as it is today, but it was further ahead in acceptance and practice and the momentum was definitely building. Many “Gray Beards” across the Federal Government and contracting industry were convinced that Agile was a fad and that it could not work within a Federal acquisitions environment (Many still believe this today!). As depicted in the chart below (source: “Agile by the Numbers,” Deloitte Insights, 5/5/2017), in 2008, roughly 17% of all major Federal IT projects were described as Agile or iterative. By 2017, that percentage rose to approximately 80%: a 370% increase.

Since the Federal Government does not yet collect information on the DevOps implementations it sponsors, I am making an anecdotal comparison. I started hearing government contractors pitching DevOps to Federal agencies in 2014. As of 2017, most, if not every RFI or RFP for Agile software development services I encounter include requirements for DevOps services. It took Agile roughly 16 years to become a standard practice for Federal software development efforts. Ten years after it was first proposed, a large number of Federal agencies see DevOps as desirable and necessary and are willing to spend significant sums of money to implement it.

I suppose this relatively swift embrace of DevOps by the Federal Government should not surprise me. DevOps is an easy sell because of three main factors. First, Agile laid the groundwork for acceptance of DevOps. The second factor is a realization across the Federal Government that iterative and incremental software development approaches are superior to traditional monolithic (“Big Bang”) development and delivery. Third, the shift to cloud architectures enables and encourages DevOps implementations. Agile and cloud adoption serve as catalysts for DevOps adoption. In addition, DevOps has a compelling value proposition: Increased opportunity for efficiencies, faster and more frequent deployments, and tighter collaboration and alignment between developers, operations, and organizational sponsors.

While DevOps is, indeed, a game changer, for the large-scale, software-intensive, systems-of-systems integration projects typically funded by the Federal Government, it is not enough. Despite DevOps’ core emphasis on developing a collaborative culture between development, operations, and across the organization, there is little guidance on how to do that as part of a DevOps approach. The Scaled Agile Framework (SAFe) fully incorporates DevOps while providing a more holistic approach to organizing multiple Agile teams into programs that live the DevOps culture.

SAFe Approach to DevOps

The Continuous Delivery Pipeline is SAFe’s DevOps approach. This approach tightly integrates with SAFe’s concept of the Agile Release Train (ART). The ART is a cross-functional team/organization of 50 to 125 people that delivers technical capabilities (i.e., software, hardware, firmware, systems). The ART is a “virtual” or matrix organization in the sense that it brings together people who, under traditional software program management practices, are typically siloed apart from each other: Engineers, testers, security, enterprise architects, configuration managers, release managers, program managers, product managers, operations, maintenance, business stakeholders, etc.

The ART aligns very well with the DevOps concept of collaboration. However, collaboration in an ART goes well beyond the work performed by developers and operations. It includes representation from all stakeholders who affect or who are affected by the ART. SAFe also aligns program and portfolio-level concerns with the work performed by Agile teams. DevOps emphasizes the need for collaboration at all levels, but there is next to no guidance on how to foster it.

An ART fulfills the IT capabilities needs of a value stream. A value stream is the chain of processes, systems, and people that that delivers business value (goods or services) to a customer. As an example, consider the steps required for a delivery company to deliver a package. Since value streams operate for as long as customers procure value from them, value streams, and the systems that support them, are long-lived. Just like the flow of value, the flow of IT capabilities is continuous: unencumbered by wasteful handoffs, reviews, queues, delays, and project starts and stops. For a given value stream, developers and operations personnel apply DevOps concepts and practices as part of an ART. This emphasis on eliminating waste and promoting flow aligns with DevOps philosophy and practice.

With respect to SAFe’s Continuous Delivery Pipeline, the ART performs three interlocking and reinforcing activities: Continuous Exploration (CE), Continuous Integration (CI), and Continuous Deployment (CD). The goal of CE is to develop a program-level vision, roadmap, and backlog that the ART fulfills by delivering features. As the name suggests, CI encompasses the activities that develop, integrate, and test prioritized features resulting from CE activities. The CD process deploys validated features into the production environment where they are further tested and prepared for release. SAFe’s CI and CD concepts closely align with the same-named concepts in DevOps. Automation is a critical enabler of CI and CD activities in both DevOps and SAFe. However, SAFe goes further by enabling the creation of portfolio and program-level ecosystems that align with DevOps.

Bottom line: Like Agile, attempts to solve problems and improve performance by applying new technology and practices, while ignoring the cultural and organizational prerequisites that support those approaches, has colored many DevOps implementations. Also like Agile, DevOps is not solely or even primarily about tools, technologies, and technical practices. It is a mindset. Rather than merely exhorting practitioners to change their organizational culture to one that is more collaborative and integrated, SAFe provides a framework that enables the creation of such a culture. SAFe enables and enforces a balance between DevOps technical practices and the cultural and organizational behaviors and structures that enable its success.