Category: DevOps Concepts

Looking for an “Agilish” approach to Agile? (pt. 1)

Don’t! But if you must …

This is the first of a three-part blog series about:

  • The U.S. Federal Government’s practice of implementing hybrid-Agile/”Agilish” software development projects
  • The perils of Agilish projects
  • How we can make Agilish projects more Agile

You are likely reading this because you are looking for a way to make Agile work within the byzantine world of government/public sector Information Technology (IT).  My perspective comes from more than a decade of working with the U.S. Federal Government implementing software systems using Agile.  I have spent a good portion of that time working in “hybrid-Agile”, “Scrumfall”, “Agilefall”, or “Agilish” projects.  My advice: Do your best to avoid Agilish projects.

Reasons to Avoid Agilish Projects

Look, I get it.  You are likely getting conflicting direction from your government agency or client.  They want Agile’s speed and flexibility with traditional project management or “Waterfall” project controls.  They want end users to have greater say in shaping solution features during development, but award firm-fixed-price contracts and require adherence to inflexible program baselines.  Even when agencies use “Agile-friendly” contracting rules, contract language and oversight requirements often impose Waterfall processes and reporting requirements.

The problem is that you can’t have your cake and it too.  There is no free lunch. <Insert another hackneyed phrase here> In trying to be both Agile and Waterfall, most Agilish government projects toss agility aside to accommodate established project management processes.  I now discuss concrete reasons why Agilish projects are a bad idea.

Agilish isn’t a Real Thing

Hybrid Agile or Agilish is not a recognized software development approach.  It is an project management approach to Agile without roots in proven Agile or software engineering practices.  There is no standard framework (like Scrum) or body of knowledge (like SAFe or PMBOK) for Agilish implementations.  Without standards or best practices, you’ll be making up your own Agilish implementation as you go.  The same goes for contract firms that claim expertise in hybrid-Agile.

Predictive vs. Adaptive Planning

In traditional project management, the bulk of project planning happens before and immediately after the start of a project.  This is a predictive planning approach.  Predictive planning assumes the next project will be similar to previous projects.  Thus, it is possible to estimate (predict) the amount of work (scope), necessary resources, timeframes (schedule), and cost (budget) of the next project based on historical data and experience.  This assumption is fine for routine and relatively simple projects but breaks down for more complex, technically challenging, or innovative projects. 

Agile planning is adaptive planning.  In Agile, planning and management of risks and opportunities occur throughout the entire effort.  Decisions are based on lessons learned and insights gained during development.  This allows Agile teams to, “Welcome changing requirements, even late in development.”

The Worst of Both Worlds

Mixing Waterfall with Agile yields the worst of both worlds.  You introduce predictive planning and project controls (e.g., Earned Value Management (EVM)) that sap the Agility out of Agile while lacking detailed program baseline data necessary to measure project progress and apply project controls a la Waterfall.

Table listing the negative results on Waterfall and Agile when mixing the two approaches.
When Waterfall and Agile Mix

Agilish from the Start

In government institutions, the shift away from traditional project management to Agile is rarely, if ever, a bottom up initiative.  Most likely, your agency or government client decreed Agile as its preferred or official software “project management” approach.  Such pronouncements usually mark the beginning of Agilish projects.

Agile, and its most popular implementation, Scrum, are not “project management” approaches at all.  Agile is an umbrella term covering multiple approaches (e.g., Scrum, XP, Kanban, Crystal) to building and delivering complex technical systems.  These approaches share the tenets of the Agile Manifesto and its foundational 12 Principles of Agile, which I paraphrase and distill below:

  • Teams develop and deliver system capabilities iteratively and incrementally within short timeframes
  • Empowered teams manage their own work and work at a sustainable pace
  • The measure of progress is delivery of valuable capabilities rather than alignment to plans
  • Direct and ongoing collaboration between and among technical, organizational, and customer stakeholders largely replaces formal documentation

Agile is a vision for a way of working, not a way of managing work or workers.  Scrum is a lightweight framework of roles, rules, and best practices that embody the tenets of the Agile Manifesto and the principles of Agile.  Characterizing Agile or Scrum, or any other flavor of Agile, as a “project management methodology” virtually guarantees Agile adoption failure.

Software Development Has Changed

Most enterprise software solutions are complex systems that change and evolve for as long as they are operational.  Agile and DevOps enable this ongoing evolution by:

  • Decreasing the time it takes to develop, test, and deploy new software capabilities
  • Increasing the frequency of new software capability deployments
  • Automating Operations and Maintenance (O&M) activities

Agile and DevOps allow organizations to move away from deploying large software releases periodically or infrequently towards continuous flows of software feature releases.  A strong DevOps culture fosters:

  • Close collaboration between developers and systems operations personnel
  • Automation of system deployment, monitoring, alerting, and troubleshooting activities

Agile does away with discrete Waterfall phases for planning, analysis, design, testing, and deployment followed by a long O&M phase.  All of these activities happen in Agile, but occur continually and in rapid cycles, thanks to DevOps.  Information and insights gained during these cycles inform how solutions evolve and how teams improve technical quality and team processes.

Agile and DevOps open the door to a different way of working.  DevOps enables software development teams to deliver increments of valuable functionality sustainably and when needed rather than on a set schedule.   As the speed and frequency of releases increase, scheduled project start and end dates lose their significance.

We move into what some call a “Software Development as a Service (SDaaS)” model.  Instead of standing up new projects, agencies augment available software development teams or form new ones.  The additional software development capacity adds to the existing software development flows.  Agencies have the choice of adding technical contributors as agency hires or contractors.  By treating software development as a service capability, organizations are able to ramp software development capacity up or down depending on mission needs and budget constraints.

Conclusion

Acquisition of software systems by the U.S. Federal Government is transitioning from a traditional project management approach to Agile. Over the last five years, the Federal government has enacted significant statutory and policy changes to align acquisitions and contracting processes with Agile and DevOps. Unfortunately, this transition will take time due to the complexity of Federal acquisitions and contracting and the resulting decades-old predictive planning culture.

In the part two of this series, I’ll talk about the reasons why Federal software systems projects become Agilish and the resulting problems. Part three will cover ways to make Agilish projects more Agile.



At Large Scale, DevOps is not Enough

For the large-scale, software-intensive, systems-of-systems integration projects typically funded by the Federal Government, DevOps alone is not enough. The Scaled Agile Framework (SAFe) fully incorporates DevOps while providing a more holistic approach to organizing multiple Agile teams into programs that live the DevOps culture.

I am astonished at the speed with which DevOps has gained traction across private industry and government organizations. This is especially surprising to me with respect to the Federal Government, which is often a (very) late adopter of new IT technologies and methodologies. Although, to my knowledge, no definitive studies exist that confirm this, DevOps seems to be gaining acceptance and adoption within the Federal Government at a faster rate than Agile did during the last 16 years.

The general consensus of when DevOps started is 2008, the year Andrew Shafer and Patrick Debois met over a talk on “Agile Infrastructure.” I worked my first Agile project as a developer that year. The Agile Manifesto was published seven years prior and adoption of Agile software development within Federal software development programs was in its infancy. Within private industry, Agile development was not as prominent as it is today, but it was further ahead in acceptance and practice and the momentum was definitely building. Many “Gray Beards” across the Federal Government and contracting industry were convinced that Agile was a fad and that it could not work within a Federal acquisitions environment (Many still believe this today!). As depicted in the chart below (source: “Agile by the Numbers,” Deloitte Insights, 5/5/2017), in 2008, roughly 17% of all major Federal IT projects were described as Agile or iterative. By 2017, that percentage rose to approximately 80%: a 370% increase.

Since the Federal Government does not yet collect information on the DevOps implementations it sponsors, I am making an anecdotal comparison. I started hearing government contractors pitching DevOps to Federal agencies in 2014. As of 2017, most, if not every RFI or RFP for Agile software development services I encounter include requirements for DevOps services. It took Agile roughly 16 years to become a standard practice for Federal software development efforts. Ten years after it was first proposed, a large number of Federal agencies see DevOps as desirable and necessary and are willing to spend significant sums of money to implement it.

I suppose this relatively swift embrace of DevOps by the Federal Government should not surprise me. DevOps is an easy sell because of three main factors. First, Agile laid the groundwork for acceptance of DevOps. The second factor is a realization across the Federal Government that iterative and incremental software development approaches are superior to traditional monolithic (“Big Bang”) development and delivery. Third, the shift to cloud architectures enables and encourages DevOps implementations. Agile and cloud adoption serve as catalysts for DevOps adoption. In addition, DevOps has a compelling value proposition: Increased opportunity for efficiencies, faster and more frequent deployments, and tighter collaboration and alignment between developers, operations, and organizational sponsors.

While DevOps is, indeed, a game changer, for the large-scale, software-intensive, systems-of-systems integration projects typically funded by the Federal Government, it is not enough. Despite DevOps’ core emphasis on developing a collaborative culture between development, operations, and across the organization, there is little guidance on how to do that as part of a DevOps approach. The Scaled Agile Framework (SAFe) fully incorporates DevOps while providing a more holistic approach to organizing multiple Agile teams into programs that live the DevOps culture.

SAFe Approach to DevOps

The Continuous Delivery Pipeline is SAFe’s DevOps approach. This approach tightly integrates with SAFe’s concept of the Agile Release Train (ART). The ART is a cross-functional team/organization of 50 to 125 people that delivers technical capabilities (i.e., software, hardware, firmware, systems). The ART is a “virtual” or matrix organization in the sense that it brings together people who, under traditional software program management practices, are typically siloed apart from each other: Engineers, testers, security, enterprise architects, configuration managers, release managers, program managers, product managers, operations, maintenance, business stakeholders, etc.

The ART aligns very well with the DevOps concept of collaboration. However, collaboration in an ART goes well beyond the work performed by developers and operations. It includes representation from all stakeholders who affect or who are affected by the ART. SAFe also aligns program and portfolio-level concerns with the work performed by Agile teams. DevOps emphasizes the need for collaboration at all levels, but there is next to no guidance on how to foster it.

An ART fulfills the IT capabilities needs of a value stream. A value stream is the chain of processes, systems, and people that that delivers business value (goods or services) to a customer. As an example, consider the steps required for a delivery company to deliver a package. Since value streams operate for as long as customers procure value from them, value streams, and the systems that support them, are long-lived. Just like the flow of value, the flow of IT capabilities is continuous: unencumbered by wasteful handoffs, reviews, queues, delays, and project starts and stops. For a given value stream, developers and operations personnel apply DevOps concepts and practices as part of an ART. This emphasis on eliminating waste and promoting flow aligns with DevOps philosophy and practice.

With respect to SAFe’s Continuous Delivery Pipeline, the ART performs three interlocking and reinforcing activities: Continuous Exploration (CE), Continuous Integration (CI), and Continuous Deployment (CD). The goal of CE is to develop a program-level vision, roadmap, and backlog that the ART fulfills by delivering features. As the name suggests, CI encompasses the activities that develop, integrate, and test prioritized features resulting from CE activities. The CD process deploys validated features into the production environment where they are further tested and prepared for release. SAFe’s CI and CD concepts closely align with the same-named concepts in DevOps. Automation is a critical enabler of CI and CD activities in both DevOps and SAFe. However, SAFe goes further by enabling the creation of portfolio and program-level ecosystems that align with DevOps.

Bottom line: Like Agile, attempts to solve problems and improve performance by applying new technology and practices, while ignoring the cultural and organizational prerequisites that support those approaches, has colored many DevOps implementations. Also like Agile, DevOps is not solely or even primarily about tools, technologies, and technical practices. It is a mindset. Rather than merely exhorting practitioners to change their organizational culture to one that is more collaborative and integrated, SAFe provides a framework that enables the creation of such a culture. SAFe enables and enforces a balance between DevOps technical practices and the cultural and organizational behaviors and structures that enable its success.