In this video, Frank Velazquez explains how Agile accelerates value delivery.
Many organizational leaders think of Agile as a way to speed up software development, and nothing more. They approach Agile with a traditional project management emphasis on improving efficiency. In doing so, they unwittingly corrupt Agile practices to the point of undermining the very agility they are trying to achieve.
Agile software development teams develop more software functionality faster by focusing on diminishing what Lean terms lead time and cycle time. Agile teams diminish lead time and cycle time by:
Minimizing waste from software development processes
Managing technical scope (increasing flow)
Striving for high quality through technical excellence
Constraints are your friends. Stop treating them like obstacles.
Table of Contents
Developing and fielding software solutions is hard. Turning a set of requirements into a viable and useful solution often feels like embarking on a voyage of discovery. Along the way, we deal with previously known and unknown constraints on what the solution can be, what it can do, and how well it can do it.
Many managers and technical implementers fail to see the difference between constraints and impediments. As a result, we deal with issues we could have planned for ahead of time only after they become impediments to progress. Also, an unwillingness to remain open to alternative ways of fulfilling requirements locks us into “all or nothing” thinking that causes us to fight against constraints instead of leveraging them to better define solutions.
Constraints vs. Impediments
For the purposes of this article, a constraint is anything that restricts the implementation and behavior of a solution. Constraints shape the solution by placing limits or imposing requirements on how we develop, deploy, support, and use the solution. Impediments, on the other hand, are occurrences or circumstances that prevent us from making progress towards developing, deploying, and supporting the solution. We are much more able to plan for constraints than for impediments. Ignoring or not adequately managing constraints leads to impediments, sometimes project wrecking ones.
I categorize constraints as fundamental and organizational. Before discussing how to manage these constraints, let’s understand what they are.
The Fundamental Constraints
The four fundamental constraints on all solutions are solution requirements and the constraints of the Iron Triangle: cost, schedule, and scope.
Solution Requirements
If we do a good job of defining needs, without jumping to specify solutions, we end up with requirements. Requirements are the first constraint on what a solution will offer and how it will perform. Solution capabilities or features are functional requirements. How well the solution performs or behaves is characterized by non-functional requirements (NFRs). Requirements start to narrow the infinite range of approaches we could take towards solving problems or addressing needs.
Figure 1: Small-Scale Manual Catalog Solution
For example, a small town library may be fine using a card catalog to track its holdings. Due to the relatively small amount of content offered and number of library patrons, manual tracking via cards and lists works both from a functional perspective (tracking books/media) and from the perspectives of cost and simplicity (NFRs). The same solution would be woefully inadequate at a large university or the Library of Congress. For large libraries, solutions must be highly automated and digitized to be viable and cost effective. The NFRs associated with managing the holdings of a large library filter out the option of using manually-intensive solutions.
Cost, Schedule, and Scope
The narrowing of solution approaches continues as we account for the constraints of the Iron Triangle. Requirements inform analysis of project scope. The more requirements we address, the larger the solution becomes (greater scope). The number of requirements, and their relative complexity, drive schedule and cost. Smaller budgets and/or tighter schedules typically drive down the amount of scope development teams can deliver. Not aligning scope to budget and schedule constraints risks degrading quality.
NFRs drive the level of quality necessary for the solution to be viable and effective. The Iron Triangle constrains quality indirectly. We can’t buy quality as a system feature delivered by a certain time. Instead, we build quality into the solution by ensuring we build the right things in the right ways.
Organizational Constraints on Solution Development
Organizations that develop and/or sponsor development, fielding, and support of software solutions impose constraints on those solutions. These organizational constraints tend to reinforce the fundamental constraints discussed in the previous section. I categorize organizational constraints as Enterprise Governance, Programmatic, and Implementation constraints. These categories align with organizational structures typically found in large organizations. In large organizations, most software systems development supports systems integration as opposed to product development.
Figure 2: Organizational Constraint Levels
Enterprise Governance Constraints
MITRE defines enterprise governance as:
The structure and relationships that control, direct, or regulate the performance of an enterprise [and its] projects, portfolios, infrastructure, and processes
Constraints at the enterprise level inform constraints at the program and project levels of the organization. Table 1 below lists the most common examples:
Table 1: Common Enterprise Governance Constraints
Programmatic Constraints
Along with ensuring implementation of enterprise governance constraints, program management offices (PMOs) also impose their own constraints on the development, fielding, and support of software solutions. PMOs focus primarily on the day-to-day management of Iron Triangle constraints across development projects and sustainment efforts as well as the associated contractual mechanisms that enforce those constraints.
Implementation Constraints
Individual projects add requirements, design specifications, and technical implementation details to the full weight of constraints flowing down from enterprise and program organizational levels. Considerations directly tied to development of solution capabilities further constrain implementation choices.
This applies to both what development teams develop and how. From a technical implementation perspective, the following are examples of choices often constrained by policies and procedures from higher organizational levels:
Forms of application or system architecture (e.g., service-oriented, monolithic, client-server)
Which technologies development teams employ
How development teams implement business / mission rules
The same applies for how development teams work. For example:
Contractual requirements may specify that teams must work within an Agile framework such as Scrum or SAFe
Developers may be required to use enterprise CM, Agile, and DevOps tooling and deployment environments provided and maintained by the organization
Managing Software Solution Constraints
People often see constraints as inherently bad. They see them as frustrating inconveniences or problems that hamper solutions development. Instead, we should see constraints as criteria that help narrow design and implementation choices to those most likely to foster solution success. Delivering on requirements while aligning solutions to enterprise, programmatic, and implementation constraints achieves fit. By fit I mean that a solution performs its work, at expected levels of performance and quality, in harmony with organizational, contractual, technological, and procedural realities.
Figure 3 and Table 2 below outline a process for managing software solution constraints.
Figure 3: Managing Constraints on Software Systems DevelopmentTable 2: Process for Managing Constraints on Software Solutions
Implementing Workarounds
A workaround is a way of accomplishing the same work or achieving the same results when a preferred approach would violate constraints. Workarounds allow us to implement solutions that would otherwise not align with constraints. There are two ways of working around constraints that can’t be waived or changed:
Technological Workarounds: Implement new or use existing technologies, services, systems, applications, etc. to address constraints. Chosen approaches become part of the solution’s implementation and architecture.
Procedural Workarounds: Implement auxiliary processes or procedures. For example, manual processes may fill requirements gaps not addressed by a solution.
Managing Project Scope
An agile way of managing constraints is through managing project scope. For example, a constraint keeps us from implementing a feature in its entirety. The constraint can’t be waived or changed away. We could break up the feature to only implement functionality that does not break the constraint while we look for alternatives. We can choose to deal with the constraint at a later time while working within the constraint. A limited implementation of the feature may turn out to be good enough.
Conclusion
Traditional project management has conditioned us to think in terms of “getting it all done.” In other words, implementing solutions according to predictive plans and designs that can’t be changed. This is unrealistic because, try as we might, we will not identify all of the constraints we will encounter before we start.
We have to do our best in identifying constraints as soon as possible so we can address them before they become impediments. At the same time, we need the agility to deal with unanticipated constraints. We achieve this balance by not treating constraints as obstacles but as influences that help shape what solutions will become.
Organizations rooted in traditional project management often fail to reap the benefits of Agile. This is primarily because they try to implement Agile approaches in ways that align with traditional project management biases. One of the most common examples of this corruption of Agile practices is developing project schedules that attempt to fix the constraints of the Iron Triangle: Schedule, Scope, and Budget / Resources.
In this article, I explain why project roadmaps are a much better fit for Agile projects than traditional project schedules.
Traditional Project Schedules
Obviously, all projects have a beginning and an end. How much time transpires in between depends on how long it takes to accomplish a certain amount of work based on some amount of resources (e.g., equipment, people). The total cost of the project depends on the per hour cost of all those resources and the number of hours they contribute to the project.
Traditional projects attempt to fix or set the constraints of the Iron Triangle from the start. This entails identifying all of the work that needs to be done, estimating how long each item of work will take, and identifying who will do the work and at what rate. Planners determine the project’s budget from these data. To keep schedule, cost, and scope as close to the original plan as possible, project management works mightily to minimize changes to planned project scope (i.e., minimize “scope creep”). The traditional project management artifact that ties all three constraints together is the project schedule, often referred to in larger traditional projects as the Integrated Master Schedule (IMS).
Thus, in traditional software projects, planners develop schedules not just to set expectations of when teams will deliver and release software functionality, but also to track:
Work task progress
Dependencies between tasks
Estimated schedule impacts of early / late task starts and completions
Costs incurred as well as estimated future costs
The project schedule is the main instrument of traditional project management control. Project managers depend on it to manage tradeoffs between Iron Triangle constraints as the project progresses.
Project Critical Path
As I discussed in the previous section, traditional project managers use project schedules to track dependencies between work tasks. This allows them to track how the actual start and completion dates of any given work task will impact subsequent tasks. It is important to stress, however, that these schedule impacts are based on estimates that may (likely) be inaccurate.
The longest sequence or path of consecutive schedule activities is known as the project’s critical path. Usually, the critical path is the path that determines duration of the project. Tracking a project’s critical path is important in traditional project management because task delays along the critical path impact project schedule more than those not on the critical path.
Figure 1: Critical Path Example (Original from onepager.com)
Agile Projects Break the Iron Triangle
Unlike traditional projects, we manage Agile projects primarily by managing scope. Agile projects hold budget and schedule milestones fixed while managing scope to fit available time and resources throughout the project. This approach to planning aligns with Agile’s principle of welcoming “changing requirements, even late in development.” Rather than defining the total scope of the project upfront, we define it as we go. To pull this off without going over budget or schedule, Agile teams employ timeboxing, sizing, ordering, and frequent delivery of business / mission capabilities.
Timeboxing
Agile planning breaks up the project planning horizon into short time periods or “timeboxes” throughout the project timeline. Instead of creating a plan and schedule for the entire duration of a project, teams break up planning across each and every timebox on the project timeline. Agilists usually call these timeboxes “sprints” or “iterations.”
Figure 2: Waterfall vs. Agile Project Planning Horizons
Sizing
Agile teams plan enough work to fill the timeboxes. They don’t change timebox duration to fit the work. They do so by estimating the “size” of the business or mission features / functionality they plan to build. The size of a given feature (or user story) represents the combination of perceived or estimated:
Feature complexity, risk, and uncertainty
Level of effort required to complete the feature
Teams estimate features by “sizing” them relative to each other. More complex features typically require more time and effort to complete, thus making them relatively “bigger” than less complex features.
Based on these “sizes”, and the team’s record of finishing different sized features during previous timeboxes (known in Scrum as “velocity”), Agile teams determine which features to break down into smaller ones and which will fit into the next timebox or two.
Ordering
Rather than following a detailed and static plan of component deliveries for later integration, Agile teams plan their work based on adaptive planning. Ordering the development, delivery, and release of features / deliverables takes into account both what is most important or valuable from a business / mission perspective and the technical dependencies between the items. I cover ordering, and how it differs from prioritization, here.
Frequent Deliveries
Agile teams deliver small increments of valuable business or mission functionality. Each increment builds on previous ones. The system grows and evolves an increment at a time. Every delivery contains working functionality that is potentially releasable. This means that the functionality is ready for deployment and release when authorized.
Figure 3: Waterfall, Agilefall, and Agile Release Frequencies
Agile teams avoid delivering individual components. They strive to deliver fully functional features. Doing so allows project sponsors, stakeholders, customers, and end users to validate that teams delivered the right things.
The faster and more often teams deliver, the more opportunities exist to validate the solution as it grows and evolves. This Agile “empiricism” provides teams the information they need to plan, not just the next timebox or two, but the direction of the solution / product and its development.
At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
https://agilemanifesto.org/principles.html
No Critical Paths in Agile Projects
Since teams define and refine the project plan continually throughout the project, there is no predefined set of work tasks to form a critical path. Also, in the case of Scrum, for example, teams organize work around implementation of user stories, not accomplishment of developer tasks. Rather than focusing on tracking task completion (as traditional project management does), Agile teams track delivered features.
Working software is the primary measure of progress.
https://agilemanifesto.org/principles.html
So, if traditional schedules don’t align to Agile projects, what should we use instead?
The Project Roadmap
Project roadmaps afford teams and project stakeholders the flexibility to shape the evolution of solutions throughout development. Instead of following and tracking a detailed predictive plan via a project schedule, Agile projects can leverage project roadmaps to chart the course of development. The course charted is informed by ongoing verification and validation of potentially releasable and released functionality throughout development.
Figure 4: Charting a course to a solution while navigating changing requirements, impediments, and risks
A project roadmap should depict the delivery of business or mission capabilities at a coarse-grained level. In the case of Scrum, for example, I recommend keeping the project roadmap at the epic level. The idea here is not to track every feature / user story and all of their associated tasks. Instead, the project roadmap should present the major slices of system functionality that deliver complete business or mission utility. However, some development team collaboration tools offer features that make drilling down below the epic level on the roadmap easy.
As depicted in Figure 5, we arrange the epics along a timeline that depicts their estimated start and end dates. Teams arrange the epics in the order they expect to work them, taking into account business / mission priorities and technical dependencies between them.
Figure 5: Agile Project Roadmap Example
Finally, we include major milestones and deadlines that teams must meet as part of project activities. Since Agile teams separate deployments to production from releases, and every timebox offers an opportunity to release, there is no need to clutter up the roadmap with lower-level details.
Conclusion
Project roadmaps are a better way of tracking planned completion and release of business / mission capabilities by Agile teams than traditional project schedules. They align with Agile scope management principles and provide the planning flexibility necessary for Agile projects. They are also much more lightweight and easier to maintain than complex project schedules. Most important, leveraging project roadmaps helps us avoid the tendency to over-plan and over-analyze that often turns Agile projects into traditional project management efforts.
Increase development and organizational agility, and better serve stakeholders, by integrating Agile, DevOps, and microservices approaches.
Table of Contents
Well-run organizations invest in information systems to achieve specific and measurable goals and objectives. They determine the return on information technology (IT) investments by how much they lower cost, increase productivity, lower risk, and increase quality. This expectation of business or mission value achieved for money spent carries over to how organizations conceive, develop, field, and maintain information systems. Valuable information systems allow users to leverage technology in ways that improve how they work to accomplish organizational goals and objectives.
Improving this alignment between business or mission needs and information systems development has been the holy grail of analysts, engineers, and managers since the dawn of the Information Age. Unfortunately, past engineering and management approaches actually widened the communication gap between business / mission stakeholders and software developers. This state of affairs made information systems:
More expensive to develop and operate;
Take longer to develop than expected;
Less likely to generate expected value after fielding; and
More likely to be cancelled before initial fielding.
Over the last 20 years, innovations in how we develop and field software systems, of all kinds, are breaking down barriers to collaboration between business / mission stakeholders and development teams. Arguably, the three most important innovations are Agile software development, DevOps, and Microservices Architecture.
Here, I discuss how the three approaches can work together to address the aforementioned shortfalls. We aim to foster tighter collaboration between business / mission stakeholders and development teams as we improve development and organizational agility.
The Agility Triad
Today, mature software development organizations can maximize development and organizational agility by integrating Agile software development, DevOps and microservice architectural practices. Agile Cheetah defines agility as:
Agility = Speed + Control + Flexibility
We call this definition “The Agility Triad.” The Agility Triad’s elements apply to organizational, process, and technological agility. The three elements are mutually dependent and reinforcing constraints defined as follows:
Figure 1: Agility Triad Elements Definitions
DevOps and Microservices Contribute to Agility
DevOps and microservice architectures provide the means of implementing the Agility Triad’s elements. DevOps serves as the Agile implementation of the Software Development Lifecycle (SDLC). Microservices help align software deliverables directly to the business / mission domain and requirements. They also help structure software development teams, their work, and what they deploy to directly address business or mission needs.
DevOps as the Agile SDLC
Agile projects perform the same SDLC activities as Waterfall projects. The difference is that software teams execute the Agile SDLC as a continuous flow of activity rather than as a linear, one-time progression through time-bound phases.
Throughout development, Agile teams define, develop, test, deploy, and release features. Rather than waiting until the end of the project for a finished system, project stakeholders, customers, and end users gain access to released features throughout the project. These features, individually and collectively, are valuable business / mission capabilities ready for use upon release.
Every feature goes through its own mini–Agile SDLC iteration. Mature Agile teams leverage automation to continually test and integrate features into complete increments of releasable functionality. Newly deployed and released increments work with all previous increments. The system grows and evolves iteratively and incrementally.
At the project level, the Agile SDLC is essentially a “swirl” or “flow” of activity occurring concurrently across multiple features and increments. The most mature Agile teams release so quickly and often that their releases approach a continuous flow.
DevOps is the implementation of this flow-based approach to software development. DevOps is not an Agile concept, nor did it start with Agile. However, it aligns perfectly with Agile because it reinterprets the SDLC in ways that support Agile’s iterative and incremental value delivery and team-based culture. It also aligns with Lean’s emphasis on achieving flow and promoting continuous improvement.
Ways DevOps Enables Agility
I now explain how DevOps enables the three elements of the Agility Triad.
Figure 3: The Agility Triad
DevOps Contribution to Project Speed
DevOps accelerates software delivery by leveraging modern software development techniques, tools, and technologies and by eliminating wasteful handoffs between development teams and operations staff. This culture of automation and continuous improvement facilitates faster iterations through the SDLC for every feature developed.
DevOps automation goes beyond automating the testing, packaging, and deployment of source code. Automated building and configuration of host environments is also a critical facet of DevOps. This capability is termed, “Infrastructure as Code (IaC).”
IaC allows for the repeatable provisioning and configuration of host environments via execution of code. Rather than incurring the time and cost of manually setting up environments, infrastructure automation platforms provision and configure virtual environments by executing scripts and implementing configuration files.
Project management can make decisions about the state of the project based, not on estimated levels of completion but, on information gained through actual deployment, release, and use of functionality. This arms managers with the information they need to decide whether to continue the current course, update project plans, or terminate the project. All along, teams deliver complete increments of functionality capable of providing value immediately.
DevOps Contribution to Project Flexibility
DevOps contributions to software project flexibility include:
Faster and more frequent iterations through the SDLC that allow experimentation with different approaches and quickly pivoting away from failing or undesirable approaches.
The ability to separate software deployments in production from releases, which enables:
Testing in Production – Teams can deploy and test new software features in production before making them available to users. This allows teams to experiment with different versions of the same functionality and see which performs best.
Releasing on Demand – Rather than deploying and releasing on a fixed schedule, teams can deploy functionality at any time and make it available to users when they are ready to use it.
Monolithic vs. Microservices Architectures
Traditional software projects delivered solutions all at once. In other words, software teams deployed and released all application components as one unit. Changes to an application often required complete redeployment. In microservices parlance, we refer to applications architected this way as monoliths.[1]
While architects and developers typically split monoliths into packages, modules, or components, interdependencies between those components make it difficult to develop, test, and deliver them separately. In systems engineering terms, these system components are “tightly coupled.”
A microservices architecture, on the other hand, builds applications as sets of independently deployable services that represent key entities of a particular business or mission domain. For example, an inventory system may include services that handle transactions associated with customers, invoices, warehouses, payment, etc. All of the information / data associated with each of these entities belongs to its respective service. A service needing information from outside the confines of its own entity (e.g., the warehouse service needs a customer’s address) must get it from the corresponding service.
Three-Tiered (Web) Applications
Since the 1990’s, the predominant architectural model for monolith applications has been the three-tiered (web) application. This distributed architecture grouped together components based on where they fit within each of the following tiers:
Figure 4: Three-Tiered (Web) App Monolith
Front End: Implements the application’s user interface(s) or externally-facing interface(s)/API(s)
Middle Tier: Performs the bulk of the application’s business logic
Datastore: Manages application data and state
Microservices Cut Across Three-Tiered Architectures
In contrast, since microservices are end-to-end slices of business or mission functionality, each microservice includes just enough of the three tiers to perform its job.
Figure 5: Each Microservice Includes Components from Each Tier
Individual microservices are not coupled to the internal workings of other services. Thus, microservices are independent or “loosely coupled.” As long as the interfaces they expose do not change, internal changes to individual microservices will not ripple across the application or system. Teams can deploy and release microservices independently, allowing applications to grow and evolve iteratively and incrementally.
Figure 6: Microservice Composition
Features Map to Microservices
As stated earlier, Agile value delivery is feature-based. This differs from the component-based delivery of functionality performed in traditional software projects.
In traditional software projects, management organizes teams to develop system components which they later integrate into complete solutions. Until that integration happens, nothing of business or mission value exists. Complete and cohesive functionality / features only exist after system integration.
System integration occurs infrequently in traditional software projects. This increases project risk. Validation of an application’s actual business / mission value typically occurs towards the end of the project, when budget and schedule are scarce.
As end-to-end slices of business or mission functionality, microservices help development teams develop, test, and deliver applications, iteratively and incrementally, as features, from the start. A feature may be satisfied by one microservice or through orchestration of multiple microservices.
Ways Microservices Enable Agility
Agile approaches, such as Scrum, help teams organize and manage their work as increments of functionality (features / user stories), developed within short timeframes (timeboxes), as part of frequent releases. DevOps provides the automation and coordination required to continually and sustainably deliver features agilely. Microservices are as close as we have come to directly implementing, deploying, and releasing requested features as features.
Microservices Contribution to Project Speed
Microservices accelerate software applications development and fielding by:
Breaking down application scope into smaller and more manageable units of work – A concept of Lean is that smaller batches of work increase flow. Microservices make it easier for development teams to manage their work through Agile and DevOps practices.
Allowing customers and end users access to valuable features earlier – Rather than waiting until full system deployment to validate and use a solution, customers and end users gain access to useful functionality throughout development. Lessons learned from released functionality help inform ongoing development.
Allowing development teams to work in parallel – Development teams can work on different microservices in parallel with little worry about inter-service dependencies beyond the interfaces the services expose.
Facilitating mixing and matching of technologies and technical approaches – Microservice independence also applies to the technology stacks and platforms used to implement them. Teams can develop microservices in whatever technologies they are most comfortable with and still achieve service integration and interoperability. Since microservices communicate via networks, they do not need to run on the same platform or under the same operating system. This technology independence lowers the amount of learning and coordination required across development teams.
Lowering the risk associated with fielding new functionality – Adding or replacing individual microservices is much easier and less risky than redeploying monolith applications. This flexibility lowers outage times.
Microservices Contribution to Controlling Projects
Microservices help teams and project management control projects by:
Facilitating Development Around Features – Agile teams control projects by managing scope. Breaking up system scope into features helps teams manage their work. Microservices support the iterative and incremental development and release of features. This strategy affords teams multiple opportunities throughout development to evaluate progress and update project plans based on actual use.
Lowering Inter-Team Dependencies – Agile software development teams are cross-functional. Members possess the skills necessary to develop complete, coherent, and deployable features across short timeboxes. This organization around features, rather than components, aligns with the tenet of building microservices that include all of the components necessary to perform as independently deployable services. Organizing teams along the lines of the microservices they develop lowers dependencies across teams and associated integration risks.
Figure 7: Tier-Based / Component Teams vs. Microservices / Feature Teams
Microservices Contribution to Project Flexibility
Spreading application scope across individual microservices simplifies changes and promotes experimentation. Teams can change their piece of the application without fear of affecting the rest. Thus, there is less resistance to trying out different approaches and settling on the most optimal. For example, we can deploy different versions of the same microservice with different user interfaces in production, selectively release them to users, and evaluate them. This technique is known as A/B testing.
Summary of DevOps and Microservices Contributions to Agility
I summarize the contributions DevOps and Microservice Architecture can make to agility below:
Figure 8: DevOps and Microservices Contributions to Agility
Conclusion
Proper integration of Agile, DevOps, and microservice architectural approaches can narrow the gulf between business or mission stakeholders and those who deliver software solutions. We do this by improving development and organizational agility. Under this construct:
Agile planning and execution are supported by DevOps processes
DevOps processes accelerate development, testing, packaging, and deployment of microservice-based applications
Resulting applications are better aligned with Agile delivery and business / mission needs
The decision to adopt any of the three approaches is not be taken lightly. There is no one-size-fits-all approach to making them work and few organizations make all three work well. It may also be the case that any of three approaches are not right for your organization or specific project.
However, these approaches are firmly in the canon of modern software development practice. It is worthwhile investigating how your organization can improve how it practices these approaches or begin adopting them.
[1] Sam Newman provides a comprehensive explanation of the ideas covered in this section in his book: Monolith to Microservices, 2020, O’Reilly, Chp 1
