In part one of this blog series, I explained what “Agilish” projects are and why we should avoid them. In this article, I discuss the reasons why government Agile initiatives tend to move away from Agile principles and towards becoming primarily Waterfall projects with Agile practices added (Agilish).

As I stated in part one, my perspective comes from years of working Agile projects for U.S. Federal Government clients. While many of the references are specific to that environment, equivalents exist at all levels of government, as well as in many private organizations, worldwide.

Process over Outcomes

Many government organizations possess process-centered cultures where progress is judged more on process execution than on outcomes achieved. The fundamental assumption being that if people correctly follow processes and procedures, desired outcomes will follow.

It is easy to see why. Governments are much more constrained than private organizations by laws, regulations, and policies (LRPs). Government institutions are ultimately accountable to the public, thus they must abide by extensive oversight rules. Documenting and enforcing processes and procedures help to ensure adherence to the law and to promote fairness and equality in government dealings.

With respect to managing software projects, the issue is not so much about having processes and procedures. They are necessary. The issue is about replacing judgement and understanding with highly-detailed and prescriptive processes and procedures. Much of current U.S. Federal acquisitions and contracting LRPs were written with traditional project management in mind. When carried too far, this urge to codify and template everything results in risk adverse acquisitions and contracting cultures hamstrung by their own rules.

This mentality leads to mechanistic Agile implementations that are at odds with Agile. Agencies “tailor” their Agile approaches to fit within their existing rules, processes, and procedures. While there are limits to how much agencies can do to align acquisitions and contracting processes to Agile, government agencies tend to overlook (and often reject) opportunities to do so.

Blaming Waterfall

For many organizations, years (if not decades) of managing software development projects through traditional project management coincide with years of project failures.  Frustration over the inability to deliver planned project scope on time, on budget, and at acceptable levels of quality serves as a powerful motivator for instituting Agile practices. 

While traditional project management is increasingly out of step with modern software development, for many organizations, Waterfall is not the root cause of software project failures.  Many organizations, public and private, lack the programmatic, engineering, and governance maturity necessary to effectively manage complex technology implementations, regardless of approach.

Organizations struggling to deliver under traditional project management processes will likely struggle even more using Agile.  You see, Agile does not on its own fix problems; it makes them plainly visible for people to take action.  Agile is a catalyst for change, not a solution.

When lack of programmatic, engineering, and governance maturity render organizations unable to address problems uncovered by Agile, they often retreat into familiar ways of working more in line with their actual capabilities. This, rather than traditional project management processes and culture, may be the single greatest obstacle to government Agile adoption.

Waterfall-Based Funding Mechanisms

Government builds software systems the way it funds them.  For the most part, government agencies have not done enough to align how they fund software systems development with Agile and DevOps. This lack of alignment is a principal factor in turning potential Agile projects Agilish and is most prominently manifested by:

• Establishing projects rather than managing services
• Adhering to outmoded “color of money” funding allocations between new development, test and evaluation, and O&M support

The Trouble with Projects

The trouble with structuring software system development and funding around projects is that projects begin and end.  A lot of coordination, planning, contracts management, and funding allocations happen before project start.  That overhead is costly and time consuming, especially in government projects.  These activities frustrate the very agility organizations try to achieve by imposing a predictive planning approach on software development.

Traditional projects afford one opportunity to get the solution right.  As a result, there is a great deal of pressure to include as many requirements as possible for a wide range of users.  This increases project scope and, consequently, increases project complexity, planning overhead, risk, and cost.  Organizations also lose flexibility to address new requirements, manage risks, and seize opportunities during development.

All of this predictive planning relies exclusively on estimates against a plan or program baseline.  If the plan turns out to be sufficiently wrong, a new plan is formulated (the program is re-baselined).  Changes to traditional project plans are highly disruptive, so project managers work mightily to avoid them.  In many cases, this effectively prevents projects from acting on empirical reality, until it is too late.

As I described in part one, software systems evolve for as long as they are in operation.  We cannot accurately predict what customer needs and priorities will be months or years from now.  Requirements are highly perishable, sometimes to the point that software systems are obsolete or irrelevant by the time they are fielded.  A funding model that aligns with Agile and DevOps treats software systems as long-standing managed services rather than as serial “one-time” investments.

“Color of Money” Concerns

As stated in part one, Agile and DevOps compress planning, analysis, design, testing, and release activities into short, fast, and frequent iterations or cycles.  Under DevOps, software development and operations teams share development and production support duties.  Development and maintenance work are included in the ongoing flow of work done by teams.  Traditional discrete project phases for these activities, performed by separate teams, no longer happen under Agile and DevOps.  Thus, it does not make sense to segregate funding and related planning along the lines of Waterfall project phases.

The US Federal Government and the Department of Defense (DoD) in particular, are coming to this realization.  Unfortunately, it will take changes in law and policy to move away from “color of money” funding mechanisms and that moves slowly.

Earned Value Management (EVM) Requirements

Applying Earned Value Management (EVM) to Agile projects imposes a vastly different definition of value on Agile. Agile does not attempt to quantify value in terms of dollars and cents. Instead, the focus is on delivering capabilities that project stakeholders, customers, and end users consider most important to mission accomplishment.  Imposing EVM requirements on Agile projects tends to subvert Agile practices to accommodate the needs of EVM.

Vendors Switching to Agile while Government Lags Behind

Agile software development is the new normal in IT.  Each succeeding batch of newly minted software development professionals entering software development is increasingly less likely to work on Waterfall projects.  This reality is now a forcing function on government agencies to change how they work with vendors to support agility.  Some agencies are doing better than others, but organizational cultures do not change overnight.

Conclusion

Recognizing the driving factors behind Agilish projects helps us understand the organizational environments within which we attempt to institute Agile approaches. Such understanding is invaluable to successful Agile adoptions/transformations, specially in the government space. Success depends on leveraging existing LRPs to align processes with Agile. While “textbook” Agile may never be realistic in many government settings, achieving the benefits of Agile requires us to understand the trade offs involved and strive to move towards greater organizational agility whenever possible.

In the third and final article of this series, I’ll cover ways to make Agilish projects more Agile.