Putting the Federal Government in the Agile Driver’s Seat

The definition of system features and scope in federal Agile and SAFe development efforts is an inherently governmental responsibility.

Recently, I came across a job posting for a Scaled Agile Framework (SAFe) product owner (PO) position with a federal government contracting firm. As someone interested in implementing SAFe within the Federal Government, I reviewed it. I wanted to get a sense for whether the job description adhered to the role as described in SAFe and, perhaps, glean the agency’s level of commitment to implementing the framework. Indeed, the duties described were the same as those prescribed by SAFe, however, my initial excitement at seeing yet another example of SAFe adoption within the Federal Government quickly turned into disappointment. The agency was handing over the definition of features and overall work scope to a contractor. This is a fundamental mistake rooted in traditional federal acquisition biases.

The Lead System Integrator Model

The Federal Government tends to outsource the overwhelming majority of its information technology (IT) and digital systems work to contracting firms. This includes the full life-cycle development and operation & maintenance (O&M) of digital systems as well as most of the program management of those activities. The Department of Defense (DoD) often designates contracting firms as Lead Systems Integrators (LSIs) to manage the integration of large-scale, complex, multi-year, digital systems-of-systems (SoS) implementations. Section 805 of the FY2006 National Defense Authorization Act defines two types of LSIs:

  • LSIs “with system responsibility” – Prime contractors responsible for the technical coordination and managerial execution of a systems development program at the system level. They do not perform technical implementation work at either the system or sub-system levels.
  • LSIs “without system responsibility” – Contractors who perform acquisition functions closely associated with inherently governmental functions.

Federal agencies turned to the LSI model in the late 1990s to compensate for a lack of in-house, technical and project management expertise caused by reductions in force of the federal acquisition workforce (by more than 50 percent between 1994 and 2005) and the rapidly increasing complexity of system solutions. By the mid-2000s, however, issues concerning LSI conflicts of interest (e.g., shaping integrations to promote LSI products), lack of effective governmental oversight over LSIs, lack of government visibility into sub-contractor activities, and the difficulty of replacing entrenched LSIs led to legislative and policy reforms and less affinity for the LSI model within the federal acquisitions community.

Today, many agencies are increasingly assuming the LSI role and instituting modular contracting policies to narrow the scope of large acquisition programs into smaller, more manageable programs and projects that do not necessitate a large LSI role. However, in many parts of the Federal Government, there is still a bias towards limited government involvement in the day-to-day management of contractor activities and capability definition. Typically, this bias exists because of a lack of expertise and/or resources. However, this bias sometimes bleeds into activities that can/are considered inherently governmental. I posit that the product management work performed by SAFe POs and Product Managers (PMs) is inherently governmental.

The Role of Product Owner in Agile

The PO is a member of an Agile software development team. They represent the “face of the customer” who will be using the functionality developed by the team. They bring to the team a deep understanding of customer needs and the business domain/market environment the solution will serve. The PO is primarily responsible for:

  • Defining solution capabilities and features through the development of Agile epics and user stories and their prioritization within the team’s product and iteration backlogs

Backlogs include technical work that supports the development and implementation of new business/mission capabilities and features.   Such technical work mostly revolves around the implementation of physical and software infrastructure (“enablers” in SAFe).  While POs are not expected to understand the technical details, they must be able to take into account the scope of these enablers and collaborate with technical contributors in prioritizing the work.

  • Planning upcoming sprints/iterations
  • Elaborating/amplifying user stories before an iteration, during iteration planning, and during iterations. They provide additional information necessary to implement stories.
  • Supporting user story acceptance testing by participating in the development of user story acceptance criteria and formally accepting user stories when they meet the criteria
  • Participating in team demos and retrospectives

All of these activities are informed by input from project stakeholders, customers, product management, program/project management, and the Agile team itself.

Product Management in SAFe

As a framework founded on Agile principles and Agile-Scrum practices, SAFe includes the Agile PO role but goes further by including the concept of a product management team function that cuts across every level of the SAFe organizational hierarchy: Portfolio, Large Solution, Program, and Team levels. SAFe POs at the team level perform the same activities as POs of non-SAFe Agile teams but also help formulate and articulate to their teams a program vision and a program roadmap. The program vision and program roadmap inform/guide the development of program features and team user stories. The program vision and program roadmap are themselves shaped by solution and portfolio-level concerns (depending on the SAFe configuration implemented). POs in SAFe are at the most tactical end of a strategic product management regime.

Breaking the Iron Triangle

The most fundamental way Agile and SAFe achieve agility is by breaking the Iron Triangle. Instead of attempting to fix schedule, budget, and scope upfront, we fix schedule (timebox) and budget while letting scope “float.” However, both Agile and SAFe prevent project/work scope from floating too far because POs are empowered to define, prioritize, and approve scope.  Moreover, in SAFe, POs and PMs across the SAFe hierarchy enforce scope discipline by:

  • Communicating with development teams regularly and often
  • Defining and articulating SAFe Agile requirements (epics, capabilities, features, user stories, and enablers)
  • Prioritizing SAFe Agile requirements
  • Actively participating as the acceptance authorities of developed functionality vis-a-vis SAFe Agile requirements
  • Working with teams to plan sprints/iterations, releases, and program increments (PI) – Breaking up scope across timeboxes

Agile and SAFe promote and enforce transparency at all levels. Decisions are made in the open with the input and concurrence of all pertinent stakeholders. In properly run Agile teams, there are no surprises: teams commit to a known set of work; the work is tested as it is developed and integrated and test results are made known; and release candidates are demonstrated to work at the end of each iteration and PI. Issues and impediments that jeopardize delivery are exposed every day through Scrums, automated testing, and frequent integration.

Breaking the Iron Triangle is a big departure from the way federal acquisitions currently works. Federal program management oversight culture and practices must change so that government program managers and project stakeholders can begin to take a much more active role in the definition of functional requirements alongside Agile teams. This will take changes in acquisition policy and law, training across government acquisition and functional communities, and leadership committed to championing new approaches to old, recurring problems. Progress in this direction is being made (e.g., 18F, Digital Services, the TechFar, Modular Contracting and a slowly growing willingness to acquire Agile services under Time & Materials (T&M) and Level of Effort (LOE) contracts).

Government Clients as POs and PMs

Appointing contractors to act as SAFe POs and PMs is an indicator that the SAFe implementation/adoption is subverted (an anti-pattern). It is a throwback to the lack of oversight and control experienced under earlier programs run by LSIs. To ensure that real value is created for every taxpayer dollar spent on federal systems development, the Federal Government must be in the driver’s seat. SAFe enables the creation of an ecosystem where technical and management expertise from the private sector works to fulfill requirements continuously defined and prioritized by government stakeholders. This “just in time” approach to scope and requirements definition is the key to achieving agility, lowering costs, and achieving faster, more frequent releases, while improving quality.

At Large Scale, DevOps is not Enough

For the large-scale, software-intensive, systems-of-systems integration projects typically funded by the Federal Government, DevOps alone is not enough. The Scaled Agile Framework (SAFe) fully incorporates DevOps while providing a more holistic approach to organizing multiple Agile teams into programs that live the DevOps culture.

I am astonished at the speed with which DevOps has gained traction across private industry and government organizations. This is especially surprising to me with respect to the Federal Government, which is often a (very) late adopter of new IT technologies and methodologies. Although, to my knowledge, no definitive studies exist that confirm this, DevOps seems to be gaining acceptance and adoption within the Federal Government at a faster rate than Agile did during the last 16 years.

The general consensus of when DevOps started is 2008, the year Andrew Shafer and Patrick Debois met over a talk on “Agile Infrastructure.” I worked my first Agile project as a developer that year. The Agile Manifesto was published seven years prior and adoption of Agile software development within Federal software development programs was in its infancy. Within private industry, Agile development was not as prominent as it is today, but it was further ahead in acceptance and practice and the momentum was definitely building. Many “Gray Beards” across the Federal Government and contracting industry were convinced that Agile was a fad and that it could not work within a Federal acquisitions environment (Many still believe this today!). As depicted in the chart below (source: “Agile by the Numbers,” Deloitte Insights, 5/5/2017), in 2008, roughly 17% of all major Federal IT projects were described as Agile or iterative. By 2017, that percentage rose to approximately 80%: a 370% increase.

Since the Federal Government does not yet collect information on the DevOps implementations it sponsors, I am making an anecdotal comparison. I started hearing government contractors pitching DevOps to Federal agencies in 2014. As of 2017, most, if not every RFI or RFP for Agile software development services I encounter include requirements for DevOps services. It took Agile roughly 16 years to become a standard practice for Federal software development efforts. Ten years after it was first proposed, a large number of Federal agencies see DevOps as desirable and necessary and are willing to spend significant sums of money to implement it.

I suppose this relatively swift embrace of DevOps by the Federal Government should not surprise me. DevOps is an easy sell because of three main factors. First, Agile laid the groundwork for acceptance of DevOps. The second factor is a realization across the Federal Government that iterative and incremental software development approaches are superior to traditional monolithic (“Big Bang”) development and delivery. Third, the shift to cloud architectures enables and encourages DevOps implementations. Agile and cloud adoption serve as catalysts for DevOps adoption. In addition, DevOps has a compelling value proposition: Increased opportunity for efficiencies, faster and more frequent deployments, and tighter collaboration and alignment between developers, operations, and organizational sponsors.

While DevOps is, indeed, a game changer, for the large-scale, software-intensive, systems-of-systems integration projects typically funded by the Federal Government, it is not enough. Despite DevOps’ core emphasis on developing a collaborative culture between development, operations, and across the organization, there is little guidance on how to do that as part of a DevOps approach. The Scaled Agile Framework (SAFe) fully incorporates DevOps while providing a more holistic approach to organizing multiple Agile teams into programs that live the DevOps culture.

SAFe Approach to DevOps

The Continuous Delivery Pipeline is SAFe’s DevOps approach. This approach tightly integrates with SAFe’s concept of the Agile Release Train (ART). The ART is a cross-functional team/organization of 50 to 125 people that delivers technical capabilities (i.e., software, hardware, firmware, systems). The ART is a “virtual” or matrix organization in the sense that it brings together people who, under traditional software program management practices, are typically siloed apart from each other: Engineers, testers, security, enterprise architects, configuration managers, release managers, program managers, product managers, operations, maintenance, business stakeholders, etc.

The ART aligns very well with the DevOps concept of collaboration. However, collaboration in an ART goes well beyond the work performed by developers and operations. It includes representation from all stakeholders who affect or who are affected by the ART. SAFe also aligns program and portfolio-level concerns with the work performed by Agile teams. DevOps emphasizes the need for collaboration at all levels, but there is next to no guidance on how to foster it.

An ART fulfills the IT capabilities needs of a value stream. A value stream is the chain of processes, systems, and people that that delivers business value (goods or services) to a customer. As an example, consider the steps required for a delivery company to deliver a package. Since value streams operate for as long as customers procure value from them, value streams, and the systems that support them, are long-lived. Just like the flow of value, the flow of IT capabilities is continuous: unencumbered by wasteful handoffs, reviews, queues, delays, and project starts and stops. For a given value stream, developers and operations personnel apply DevOps concepts and practices as part of an ART. This emphasis on eliminating waste and promoting flow aligns with DevOps philosophy and practice.

With respect to SAFe’s Continuous Delivery Pipeline, the ART performs three interlocking and reinforcing activities: Continuous Exploration (CE), Continuous Integration (CI), and Continuous Deployment (CD). The goal of CE is to develop a program-level vision, roadmap, and backlog that the ART fulfills by delivering features. As the name suggests, CI encompasses the activities that develop, integrate, and test prioritized features resulting from CE activities. The CD process deploys validated features into the production environment where they are further tested and prepared for release. SAFe’s CI and CD concepts closely align with the same-named concepts in DevOps. Automation is a critical enabler of CI and CD activities in both DevOps and SAFe. However, SAFe goes further by enabling the creation of portfolio and program-level ecosystems that align with DevOps.

Bottom line: Like Agile, attempts to solve problems and improve performance by applying new technology and practices, while ignoring the cultural and organizational prerequisites that support those approaches, has colored many DevOps implementations. Also like Agile, DevOps is not solely or even primarily about tools, technologies, and technical practices. It is a mindset. Rather than merely exhorting practitioners to change their organizational culture to one that is more collaborative and integrated, SAFe provides a framework that enables the creation of such a culture. SAFe enables and enforces a balance between DevOps technical practices and the cultural and organizational behaviors and structures that enable its success.

Five Things You Must Understand about Agile Team Velocity

Agile team velocity is an often misunderstood concept that is prone to misapplication. This article sets the record straight.

As the Federal Government mandates the use of Agile software development practices in an ever-growing number of software development programs, the proper use of team velocity is becoming a significant point of confusion.  A misconception that seems to be gaining traction is the idea that the government can “buy” Agile software development services à la carte based on team velocities.  This contracting model would involve vendors offering prepackaged teams rated at some velocity.  Based on that velocity, the government could then pay for a certain amount of that team’s time to complete predefined tasks.  Agile teams would, in essence, be treated like plug-and-play units of Agile development capability.  This mentality is evidence of a fundamental misunderstanding of Agile, of what Agile teams require to be highly effective, and of the concept of Agile velocity itself.

What is Agile Team Velocity?

Agile team velocity is a heuristic measure of an Agile-Scrum development team’s throughput (not productivity).  It is the rate at which a team delivers fully completed, tested, and accepted user stories from the product backlog during one iteration/sprint.  Rather than simply adding up the number of completed stories at the end of a sprint, story points previously assigned to those user stories are added together.  At the start of every sprint, the team works with the product owner to clarify, decompose, and assign story points to user stories during sprint planning. Story points are heuristically-determined, numerical proxies assigned to user stories that represent the combination of perceived or estimated user story complexity, risk, uncertainty, and the level of effort required to complete the work.  This combination is often referred to as the “bigness” or “size” of a user story.  More complex user stories typically require more time and effort to complete thus, earning more story points than less complex stories.    Story points are the basis of a team’s velocity and velocity enables team estimates of the amount of capability (value) they can deliver, at a sustainable pace, within a sprint.

Σ Story Points of Completed User Stories During a Sprint = Velocity

Five Concepts that must be Understood about Agile Team Velocity:

1.  Story points are derived through relative sizing/estimation of user stories against each other by the development teams that will implement them.  They are not determined by a chief engineer, systems engineer, or some other senior authority as an estimation exercise prior to the establishment of development teams and the appointment of product owners.

2.  Team velocity is specific to the team.  Velocity cannot be used as a point of comparison between teams.  Team X is not necessary faster or able to achieve a higher throughput than Team Y because they earned a higher number of story points during a sprint.  The story points upon which team velocity is based are specific to the relative sizing performed by the team on their particular backlog of user stories.

3.  A team’s velocity is the end result of many factors, including:
  • Individual team member productivity across the team
  • Team cohesion, collaboration, and effectiveness
  • Team skill set and experience mix
  • Level of programmatic support (i.e., the factors that fall under the purview of program management)
  • Empowerment and level of engagement of product owners
  • The team’s familiarity with the business domain, technology mix, and enterprise architecture
  • The amount of automation leveraged in building, testing, and deploying code
Thus, the make up of the teams and the environment or ecosystem within which they operate determine velocity.  Team velocity is ultimately a proxy for the overall health of the software development program and organization.  When team velocity stagnates or drops, the search for potential causes must be systemic as opposed to a hunt for individual culpability.

4.  It is normal for team velocity to fluctuate.  What should concern us are large fluctuations or downward trends in velocity.  Changes in personnel, technology, organizational structure, etc. will affect development teams in different ways and to varying degrees.

5. Improving team velocity is not the goal!  Improving team capabilities, practices, tools, and programmatic support improves team velocity.  Focusing on improving velocity results in attempts to game the system rather than systemic improvement.

Bottom line:  Agile development teams are composed of people.  People are not machines calibrated to deliver a predetermined rate of output.  Teams need time and stability to gel into highly effective teams.  Helicoptering Agile teams, even experienced ones, in and out of tasks significantly degrades velocity except perhaps for the most mundane and menial of tasks (In which case, why employ Agile at all?).   Decades of reducing people to resources to allocate across Gantt charts has led many to also think of Agile teams as abstract productivity units. Agile requires much more of us.

Article Review: The New New Product Development Game

A look back at a 1986 Harvard Business Review article provides insight into the roots of Agile and its application.

No, there isn’t a typo in the title. 🙂  The title of this blog entry was taken (borrowed?) from a very interesting article published in the Harvard Business Review from January 1986.  Scaled Agile Inc. (the developers of SAFe) references it to make a number of points concerning how to best structure Agile teams.  In particular, Scaled Agile Inc. references the article’s assertion that:

  1.   The “relay race” approach to product development [i.e., Waterfall] conflicts with the need for speed and flexibility.
  2.   That a holistic or “rugby” approach [a la Agile] where, “a team tries to go the distance as a unit, passing the ball back and forth” better serves highly competitive environments.

The things that strike me about this article are:

  1.   It was published in January 1986.  A good 15 years before the Agile Manifesto was published.
  2.   All of the examples are product development efforts outside the software development realm:  Xerox copiers, HP printers, Honda cars, Canon cameras.
  3.   The fact that they likened the process to Rugby (a foreshadowing of Agile-Scrum).
  4.   The focus on self-organizing teams – A major item in the list of Principles of the Agile Manifesto:  “The best architectures, requirements, and designs
    emerge from self-organizing teams.”
  5.   Encouragement of contributor autonomy – A major component of the SAFe principle of unlocking the “intrinsic motivation of knowledge workers.”
  6.   Fostering of “Self-transcendence” – Similar to the concept of Ba: an old Lean term brought to new relevance in Agile-Scrum and SAFe circles.
  7.   Emphasis on “Cross-fertilization,” known in Agile as “Cross Functional Teams.”
  8.   The article’s concept of “Overlapping development phases” roughly translates into the Agile idea that SDLC phases (planning, analysis, design, implementation, and maintenance) should not be separate phases at all but should, instead, be performed concurrently within a specified time-box (iteration/sprint), on a small scope of work, with overall system scope accruing incrementally and iteratively.
  9.   The recognition that for all of this to work, we need management to change away from linear and static product development thinking.
  10.   The recognition that, “A different kind of learning is needed.”  By this they mean engineers need to have both deep skills and broad understanding to be effective in cross-functional teams that execute in less structured environments:  “T-shaped skills.”

I recommend organizational leaders and managers read the article to gain perspective on the thinking that led to Agile, so they may better understand how to adopt it within their respective organizations.  Engineers of all disciplines and current Agilists will also benefit from a better understanding of how real cross-functional, self-organized teams are formed and behave.

Down with the Agilistas!

Are you an “Agilist” or an “Agilista”?

I consider myself an Agilist. I have been working within the Agile software development realm in different capacities/roles since 2008 and I whole-hardheartedly believe in the potential of Agile and Lean to dramatically improve, not just software/systems development, but also organizations that sponsor IT capabilities development. I believe that the success and ongoing effectiveness of any Agile implementation depends on closely hewing to the Agile Manifesto and the 12 Principles of Agile Software.

Now that I have firmly declared my commitment to Agile principles, I differentiate myself from those whom I refer to as “Agilistas.” In my definition of the term, an Agilista is someone who takes a binary, black or white view of what is and is not “Agile.” Believing that deviations from Agile practices they have learned and practiced cannot be Agile, some Agilistas stifle change and innovation in the Agile domain. Others are strict literalists who treat published Agile best practices as instruction manuals that must be followed, to the letter, in every situation, regardless of the realities/constraints imposed on/by projects and organizations. They wish to ignore or change factors external to software/system development concerns to fit their “pure” Agile implementation. Still others confuse attempts to enforce engineering discipline and address complexity in large-scale software/system development efforts with establishing burdensome and unnecessary processes and documentation (i.e., bureaucracy). Many Agilistas decry “Agile frameworks” and “Agile transformations,” feeling that methodologies that go beyond the established canon of team-level Agile practices and techniques are inherently anti-Agile. In their minds, “proper” application of the canon, in the “proper” spirit, with coordination across teams via Scrum-of-Scrums (in the case of Agile-Scrum) is all that is required and desirable. Too often, this desire to keep Agile simple and pure leads to exhortations that others “think” and “be” more Agile, without proposing how to do that in the face of technical, programmatic, compliance, and fiscal realities that constrain and shape all non-trivial system development efforts.

I am sympathetic to the frustration that leads to this brand of reactionary thinking. For a couple of decades, Agilists have had to witness and, sometimes, take part in failed Agile implementations misinformed by Waterfall software/system development prejudices and undermined by an unwillingness on the part of organizational leadership and stakeholders to embrace what I call Agile Zen:

The primary role of management is to empower those who actually deliver value to the customer and to create work environments and customer relationships within which these “value providers” are able to succeed and flourish.

No amount of Agile process mechanics, training, management tools, DevOps tooling, etc. will bring the benefits of Agile to organizations that ignore this maxim. Unfortunately, often it is not just ignored; it is outright rejected by leadership incapable of shifting towards a “servant leadership” orientation. Adoption of an Agile software/system development approach within an enterprise requires alignment between the work performed by Agile software teams and the program and portfolio level organizations that form and sustain them. In other words, Agile development becomes a forcing function for change throughout the enterprise.

This is why I am excited about SAFe. It is a framework for scaling Agile practices to the scale of hundreds of teams while adhering to Agile principles, enforcing engineering rigor through systems thinking, and ensuring long-term continuous improvement and ROI through the application of Lean principles. It includes and fosters approaches to Agile program management and governance based on Agile principles and addresses real-life enterprise/organizational concerns without sacrificing those principles. The future of Agile is in Agile scaling. I hope the Agilistas will embrace it.