Federal “Hybrid Agile” is an ill-conceived and counterproductive approach to Agile rooted in federal acquisitions culture rather than in proven software/systems engineering practices.
A university professor visited a Zen master to learn about the master’s teachings. While the master quietly served tea, the professor lectured him about his studies in Zen philosophy. The master continued pouring tea in the professor’s cup after the tea spilled over the rim. The professor watched the master spill the tea until he could no longer hold back. “It’s full! No more can go in!” the professor blurted out. The master stopped pouring, looked into the professor’s eyes and said, “This is your mind. How can I teach you what I know unless you first empty your cup?”
Many versions of this Zen parable exist. They all boil down to the same moral: People who are unable to set aside their preconceptions and prejudices, who are wrapped up in what they know or think they know, shut themselves off from new knowledge and insights. Often, as we navigate through our lives accumulating knowledge, experiences, and developing points of view about everything, we reach a point where the boundaries of our mental models harden and our thinking becomes similarly sclerotic. In that state, we immediately reject or subconsciously distort anything that challenges our understanding. To truly learn, we must be prepared to understand new concepts on their own terms before interjecting our own perspectives. We must empty the cup so we may taste the new tea.
This parable is not an appeal to suspending critical thinking or disregarding previously acquired education, training, and experience. Indeed, those attributes are critical to the ability to learn new concepts and effectively apply them in the real world. It is a defense of a true open-mindedness that seeks to fully understand unfamiliar ideas before casting judgement on them or attempting to change them to fit preconceived notions or entrenched habits of mind.
“Hybrid Agile”
Most Federal Government Agile system development efforts are really “Scrumfall” implementations, where software development teams perform Agile-Scrum practices/activities within a Waterfall program management structure. This Agile anti-pattern is often colloquially referred to in the Federal Government as “Hybrid Agile.”
Hybrid Agile is not a recognized software/systems development approach. It is an ill-conceived and counterproductive approach to Agile rooted in federal acquisitions culture rather than in proven software/systems engineering practices. It is an attempt to gain the benefits of Agile while avoiding internalizing and applying the “Agile Zen” or principles that make Agile work. Rather than reevaluating existing processes to achieve greater agility, federal project management organizations typically choose to contort Agile processes so they fit within existing management, acquisitions, and contracting culture. This leads to a tailoring of Agile practices that is misinformed by traditional federal management biases. As a result, federal departments and agencies seldom achieve the reputed benefits of Agile software development.
This bias towards Waterfall-based management of software/system development efforts is systemic and rooted in the laws, policies and regulations (LRPs) that govern federal acquisitions and contracting. These precepts will not change overnight. However, waiting until they change is neither desirable nor necessary. Enough flexibility exists in the Federal Acquisition Regulation (FAR) now to reverse this bias and achieve real agility in how software-based capabilities are developed, fielded, and maintained. New acquisition and contracting practices are in place and in use today that align, rather than conflict, with Agile principles and practices. Finally, Agile thinking has evolved beyond its application at the software development team level to scaling it across both the IT and organizational enterprise.
Understanding the difference between product development and system integration projects helps better define the Product Owner role and the Product Management structure necessary for project success.
Agile-Scrum clearly defines the Product Owner (PO) as a key member of a Scrum team. The PO represents the Voice of the Customer. The PO is responsible for fusing the various needs, wants, and perspectives of project stakeholders into one complete, coherent, and cohesive vision. The development team depends on the direction and guidance provided by the PO to understand what business capabilities to develop and how those capabilities must work to provide value to the customer. Thus, PO plays the most important role in the creation of User Stories and their prioritization within the Product Backlog.
However, more often than not, when I ask Agile-Scrum development teams, “Who is your PO?”, they do not know for certain. This is especially true in systems integration projects. Ask different developers within the project who the PO is and you will likely get different answers. Some will say the Enterprise Architect is the PO. Others will name the lead Systems Engineer, or the Chief Engineer, or the Project/Program Manager, or the owner of the company, etc. Often the answer is, “We don’t have one.”
Agile-Scrum was originally conceived within a commercial product development paradigm. The terminology of Agile-Scrum is indicative of this product development bias with terms like Product Ownerand Product Backlog.
In a product development project, the company has significantly more control over the Product Vision and how that vision is actualized than in a systems integration effort. The company decides what capabilities and features to develop, how they will be integrated into the product, when the product will be released, and how much will be spent developing the product. In well-run product companies, all of these decisions are informed by market and user data. However, these decisions are ultimately made by the company.
System integration efforts, by contrast, tend to be customer-driven. Customers decide which capabilities they will pay for, how much they will spend, and when the implementation of those capabilities is due. Thus, system integration efforts are, by nature, service engagements, not product development efforts. Also, software solutions integrated into enterprise environments are typically much more technically and organizationally constrained than than green-field product development efforts.
The control product companies have over their products makes it significantly easier to identify and empower POs than in system integration engagements. Product company POs have fewer stakeholders to answer to and, typically, fewer external dependencies to manage. They also have the advantage of starting with a company-defined Product Vision. In systems integration, Agile-Scrum teams work with a customer PO to define a Product Vision for a solution the PO does not own and that must address many disparate, and often conflicting, needs and wants of internal and external organizational stakeholders.
It quickly becomes clear that the role of PO in a large-scale system integration effort cannot be filled by one person alone. Due to the many stakeholders that need to be included, the Voice of the Customer can easily become a cacophony of voices without some structured way of harmonizing them into a chorus. Adding to this complexity is the fact that large-scale system integration efforts typically involve multiple teams from different companies. Each of these teams requires a PO, assuming they are all executing as Agile-Scrum teams.
Large system integration projects demand a level of planning and coordination that pure “team-level“ Agile cannot handle. I am not proposing engaging in Waterfall-style, Big Planning Upfront. Even in complex system integration engagements, the Product Vision can be defined over time in an iterative approach, as prescribed by Agile.
Large-scale systems integration requires a product management approach that informs and coordinates team-level PO decisions and prioritizations across program and portfolio levels. Each team works with a PO who manages the stakeholder community that depends on the team’s feature set. Each PO also coordinates with other POs through a product management structure. This product management structure is a core feature of the Scaled Agile Framework (SAFe).
Since scope in Agile “floats”, having an empowered and engaged PO is vital to ensuring teams deliver real value within agreed upon budget and time constraints. The definition and realization of the solution’s TO-BE state depends on POs. For Federal software system development projects in particular, the PO role should be played by a government employee who is empowered to make decisions, capable of leading a stakeholder community, and tasked as a full time PO to the teams he/she supports. Anything less introduces risk to Agile projects and breaks the alignment between development teams and the sponsoring organization.
The definition of system features and scope in federal Agile and SAFe development efforts is an inherently governmental responsibility.
Table of Contents
Recently, I came across a job posting for a Scaled Agile Framework (SAFe)product owner (PO) position with a federal government contracting firm. As someone interested in implementing SAFe within the Federal Government, I reviewed it. I wanted to get a sense for whether the job description adhered to the role as described in SAFe and, perhaps, glean the agency’s level of commitment to implementing the framework. Indeed, the duties described were the same as those prescribed by SAFe, however, my initial excitement at seeing yet another example of SAFe adoption within the Federal Government quickly turned into disappointment. The agency was handing over the definition of features and overall work scope to a contractor. This is a fundamental mistake rooted in traditional federal acquisition biases.
The Lead System Integrator Model
The Federal Government tends to outsource the overwhelming majority of its information technology (IT) and digital systems work to contracting firms. This includes the full life-cycle development and operation & maintenance (O&M) of digital systems as well as most of the program management of those activities. The Department of Defense (DoD) often designates contracting firms as Lead Systems Integrators (LSIs) to manage the integration of large-scale, complex, multi-year, digital systems-of-systems (SoS) implementations. Section 805 of the FY2006 National Defense Authorization Act defines two types of LSIs:
LSIs “with system responsibility” – Prime contractors responsible for the technical coordination and managerial execution of a systems development program at the system level. They do not perform technical implementation work at either the system or sub-system levels.
LSIs “without system responsibility” – Contractors who perform acquisition functions closely associated with inherently governmental functions.
Today, many agencies are increasingly assuming the LSI role and instituting modular contracting policies to narrow the scope of large acquisition programs into smaller, more manageable programs and projects that do not necessitate a large LSI role. However, in many parts of the Federal Government, there is still a bias towards limited government involvement in the day-to-day management of contractor activities and capability definition. Typically, this bias exists because of a lack of expertise and/or resources. However, this bias sometimes bleeds into activities that can/are considered inherently governmental. I posit that the product management work performed by SAFe POs and Product Managers (PMs) is inherently governmental.
The Role of Product Owner in Agile
The PO is a member of an Agile software development team. They represent the “face of the customer” who will be using the functionality developed by the team. They bring to the team a deep understanding of customer needs and the business domain/market environment the solution will serve. The PO is primarily responsible for:
Defining solution capabilities and features through the development of Agile epics and user stories and their prioritization within the team’s product and iteration backlogs
Backlogs include technical work that supports the development and implementation of new business/mission capabilities and features. Such technical work mostly revolves around the implementation of physical and software infrastructure (“enablers” in SAFe). While POs are not expected to understand the technical details, they must be able to take into account the scope of these enablers and collaborate with technical contributors in prioritizing the work.
Planning upcoming sprints/iterations
Elaborating/amplifying user stories before an iteration, during iteration planning, and during iterations. They provide additional information necessary to implement stories.
Supporting user story acceptance testing by participating in the development of user story acceptance criteria and formally accepting user stories when they meet the criteria
Participating in team demos and retrospectives
All of these activities are informed by input from project stakeholders, customers, product management, program/project management, and the Agile team itself.
Product Management in SAFe
As a framework founded on Agile principles and Scrum practices, SAFe includes the PO role but goes further by including the concept of a product management team function across every configuration of the Framework: Full, Portfolio, Large Solution, and Essential SAFe. SAFe POs collaborate with their development teams in defining team backlogs and help formulate and articulate a program vision and program roadmap. The program vision and program roadmap inform/guide the development of program features and team user stories. The program vision and program roadmap are themselves shaped by solution and portfolio-level concerns (depending on the SAFe configuration implemented). POs in SAFe are at the most tactical end of a strategic product management regime.
Breaking the Iron Triangle
The most fundamental way Agile and SAFe achieve agility is by breaking the Iron Triangle. Instead of attempting to fix schedule, budget, and scope upfront, we fix schedule (timebox) and budget while letting scope “float.” However, both Agile and SAFe prevent project/work scope from floating too far because POs are empowered to define, prioritize, and approve scope. Moreover, in SAFe, POs and PMs across the SAFe hierarchy enforce scope discipline by:
Communicating with development teams regularly and often
Actively participating as the acceptance authorities of developed functionality vis-à-vis SAFe Agile requirements
Working with teams to plan sprints/iterations, releases, and program increments (PI) – Breaking up scope across timeboxes
Agile and SAFe promote and enforce transparency at all levels. Decisions are made in the open with the input and concurrence of all pertinent stakeholders. In properly run Agile teams, there are no surprises: teams commit to a known set of work; the work is tested as it is developed and integrated and test results are made known; and release candidates are demonstrated to work at the end of each iteration and PI. Issues and impediments that jeopardize delivery are exposed every day through Scrums, automated testing, and frequent integration.
Breaking the Iron Triangle is a big departure from the way federal acquisitions currently works. Federal program management oversight culture and practices must change so that government program managers and project stakeholders can begin to take a much more active role in the definition of functional requirements alongside Agile teams. This will take changes in acquisition policy and law, training across government acquisition and functional communities, and leadership committed to championing new approaches to old, recurring problems. Progress in this direction is being made (e.g., 18F, Digital Services, the TechFar, Modular Contracting and a slowly growing willingness to acquire Agile services under Time & Materials (T&M) and Level of Effort (LOE) contracts).
Conclusion
Appointing contractors to act as SAFe POs and PMs is an indicator that the SAFe implementation/adoption is subverted (an anti-pattern). It is a throwback to the lack of oversight and control experienced under earlier programs run by LSIs. To ensure that real value is created for every taxpayer dollar spent on federal systems development, the Federal Government must be in the driver’s seat. SAFe enables the creation of an ecosystem where technical and management expertise from the private sector works to fulfill requirements continuously defined and prioritized by government stakeholders. This “just in time” approach to scope and requirements definition is the key to achieving agility, lowering costs, and achieving faster, more frequent releases, while improving quality.
For the large-scale, software-intensive, systems-of-systems integration projects typically funded by the Federal Government, DevOps alone is not enough. The Scaled Agile Framework (SAFe) fully incorporates DevOps while providing a more holistic approach to organizing multiple Agile teams into programs that live the DevOps culture.
I am astonished at the speed with which DevOps has gained traction across private industry and government organizations. This is especially surprising to me with respect to the Federal Government, which is often a (very) late adopter of new IT technologies and methodologies. Although, to my knowledge, no definitive studies exist that confirm this, DevOps seems to be gaining acceptance and adoption within the Federal Government at a faster rate than Agile did during the last 16 years.
The general consensus of when DevOps started is 2008, the year Andrew Shafer and Patrick Debois met over a talk on “Agile Infrastructure.” I worked my first Agile project as a developer that year. The Agile Manifesto was published seven years prior and adoption of Agile software development within Federal software development programs was in its infancy. Within private industry, Agile development was not as prominent as it is today, but it was further ahead in acceptance and practice and the momentum was definitely building. Many “Gray Beards” across the Federal Government and contracting industry were convinced that Agile was a fad and that it could not work within a Federal acquisitions environment (Many still believe this today!). As depicted in the chart below (source: “Agile by the Numbers,” Deloitte Insights, 5/5/2017), in 2008, roughly 17% of all major Federal IT projects were described as Agile or iterative. By 2017, that percentage rose to approximately 80%: a 370% increase.
Since the Federal Government does not yet collect information on the DevOps implementations it sponsors, I am making an anecdotal comparison. I started hearing government contractors pitching DevOps to Federal agencies in 2014. As of 2017, most, if not every RFI or RFP for Agile software development services I encounter include requirements for DevOps services. It took Agile roughly 16 years to become a standard practice for Federal software development efforts. Ten years after it was first proposed, a large number of Federal agencies see DevOps as desirable and necessary and are willing to spend significant sums of money to implement it.
I suppose this relatively swift embrace of DevOps by the Federal Government should not surprise me. DevOps is an easy sell because of three main factors. First, Agile laid the groundwork for acceptance of DevOps. The second factor is a realization across the Federal Government that iterative and incremental software development approaches are superior to traditional monolithic (“Big Bang”) development and delivery. Third, the shift to cloud architectures enables and encourages DevOps implementations. Agile and cloud adoption serve as catalysts for DevOps adoption. In addition, DevOps has a compelling value proposition: Increased opportunity for efficiencies, faster and more frequent deployments, and tighter collaboration and alignment between developers, operations, and organizational sponsors.
While DevOps is, indeed, a game changer, for the large-scale, software-intensive, systems-of-systems integration projects typically funded by the Federal Government, it is not enough. Despite DevOps’ core emphasis on developing a collaborative culture between development, operations, and across the organization, there is little guidance on how to do that as part of a DevOps approach. The Scaled Agile Framework (SAFe) fully incorporates DevOps while providing a more holistic approach to organizing multiple Agile teams into programs that live the DevOps culture.
SAFe Approach to DevOps
The Continuous Delivery Pipeline is SAFe’s DevOps approach. This approach tightly integrates with SAFe’s concept of the Agile Release Train (ART). The ART is a cross-functional team/organization of 50 to 125 people that delivers technical capabilities (i.e., software, hardware, firmware, systems). The ART is a “virtual” or matrix organization in the sense that it brings together people who, under traditional software program management practices, are typically siloed apart from each other: Engineers, testers, security, enterprise architects, configuration managers, release managers, program managers, product managers, operations, maintenance, business stakeholders, etc.
The ART aligns very well with the DevOps concept of collaboration. However, collaboration in an ART goes well beyond the work performed by developers and operations. It includes representation from all stakeholders who affect or who are affected by the ART. SAFe also aligns program and portfolio-level concerns with the work performed by Agile teams. DevOps emphasizes the need for collaboration at all levels, but there is next to no guidance on how to foster it.
An ART fulfills the IT capabilities needs of a value stream. A value stream is the chain of processes, systems, and people that that delivers business value (goods or services) to a customer. As an example, consider the steps required for a delivery company to deliver a package. Since value streams operate for as long as customers procure value from them, value streams, and the systems that support them, are long-lived. Just like the flow of value, the flow of IT capabilities is continuous: unencumbered by wasteful handoffs, reviews, queues, delays, and project starts and stops. For a given value stream, developers and operations personnel apply DevOps concepts and practices as part of an ART. This emphasis on eliminating waste and promoting flow aligns with DevOps philosophy and practice.
With respect to SAFe’s Continuous Delivery Pipeline, the ART performs three interlocking and reinforcing activities: Continuous Exploration (CE), Continuous Integration (CI), and Continuous Deployment (CD). The goal of CE is to develop a program-level vision, roadmap, and backlog that the ART fulfills by delivering features. As the name suggests, CI encompasses the activities that develop, integrate, and test prioritized features resulting from CE activities. The CD process deploys validated features into the production environment where they are further tested and prepared for release. SAFe’s CI and CD concepts closely align with the same-named concepts in DevOps. Automation is a critical enabler of CI and CD activities in both DevOps and SAFe. However, SAFe goes further by enabling the creation of portfolio and program-level ecosystems that align with DevOps.
Bottom line: Like Agile, attempts to solve problems and improve performance by applying new technology and practices, while ignoring the cultural and organizational prerequisites that support those approaches, has colored many DevOps implementations. Also like Agile, DevOps is not solely or even primarily about tools, technologies, and technical practices. It is a mindset. Rather than merely exhorting practitioners to change their organizational culture to one that is more collaborative and integrated, SAFe provides a framework that enables the creation of such a culture. SAFe enables and enforces a balance between DevOps technical practices and the cultural and organizational behaviors and structures that enable its success.
