Improve Product Backlog Ordering and Prioritization
In this video, Frank Velazquez explains what a User Story Map is and how it supports the ordering and refinement of product backlogs.
User Story Maps can help your Scrum Team order and refine its product backlog, plan releases, and communicate status and plans to sponsors and stakeholders. They help ensure teams take into account technical dependencies between product backlog items when ordering and refining product backlogs. They also help ensure that all planned and implemented product backlog items add up to a business or mission-valuable solution.
Organizations rooted in traditional project management often fail to reap the benefits of Agile. This is primarily because they try to implement Agile approaches in ways that align with traditional project management biases. One of the most common examples of this corruption of Agile practices is developing project schedules that attempt to fix the constraints of the Iron Triangle: Schedule, Scope, and Budget / Resources.
In this article, I explain why project roadmaps are a much better fit for Agile projects than traditional project schedules.
Traditional Project Schedules
Obviously, all projects have a beginning and an end. How much time transpires in between depends on how long it takes to accomplish a certain amount of work based on some amount of resources (e.g., equipment, people). The total cost of the project depends on the per hour cost of all those resources and the number of hours they contribute to the project.
Traditional projects attempt to fix or set the constraints of the Iron Triangle from the start. This entails identifying all of the work that needs to be done, estimating how long each item of work will take, and identifying who will do the work and at what rate. Planners determine the project’s budget from these data. To keep schedule, cost, and scope as close to the original plan as possible, project management works mightily to minimize changes to planned project scope (i.e., minimize “scope creep”). The traditional project management artifact that ties all three constraints together is the project schedule, often referred to in larger traditional projects as the Integrated Master Schedule (IMS).
Thus, in traditional software projects, planners develop schedules not just to set expectations of when teams will deliver and release software functionality, but also to track:
Work task progress
Dependencies between tasks
Estimated schedule impacts of early / late task starts and completions
Costs incurred as well as estimated future costs
The project schedule is the main instrument of traditional project management control. Project managers depend on it to manage tradeoffs between Iron Triangle constraints as the project progresses.
Project Critical Path
As I discussed in the previous section, traditional project managers use project schedules to track dependencies between work tasks. This allows them to track how the actual start and completion dates of any given work task will impact subsequent tasks. It is important to stress, however, that these schedule impacts are based on estimates that may (likely) be inaccurate.
The longest sequence or path of consecutive schedule activities is known as the project’s critical path. Usually, the critical path is the path that determines duration of the project. Tracking a project’s critical path is important in traditional project management because task delays along the critical path impact project schedule more than those not on the critical path.
Figure 1: Critical Path Example (Original from onepager.com)
Agile Projects Break the Iron Triangle
Unlike traditional projects, we manage Agile projects primarily by managing scope. Agile projects hold budget and schedule milestones fixed while managing scope to fit available time and resources throughout the project. This approach to planning aligns with Agile’s principle of welcoming “changing requirements, even late in development.” Rather than defining the total scope of the project upfront, we define it as we go. To pull this off without going over budget or schedule, Agile teams employ timeboxing, sizing, ordering, and frequent delivery of business / mission capabilities.
Timeboxing
Agile planning breaks up the project planning horizon into short time periods or “timeboxes” throughout the project timeline. Instead of creating a plan and schedule for the entire duration of a project, teams break up planning across each and every timebox on the project timeline. Agilists usually call these timeboxes “sprints” or “iterations.”
Figure 2: Waterfall vs. Agile Project Planning Horizons
Sizing
Agile teams plan enough work to fill the timeboxes. They don’t change timebox duration to fit the work. They do so by estimating the “size” of the business or mission features / functionality they plan to build. The size of a given feature (or user story) represents the combination of perceived or estimated:
Feature complexity, risk, and uncertainty
Level of effort required to complete the feature
Teams estimate features by “sizing” them relative to each other. More complex features typically require more time and effort to complete, thus making them relatively “bigger” than less complex features.
Based on these “sizes”, and the team’s record of finishing different sized features during previous timeboxes (known in Scrum as “velocity”), Agile teams determine which features to break down into smaller ones and which will fit into the next timebox or two.
Ordering
Rather than following a detailed and static plan of component deliveries for later integration, Agile teams plan their work based on adaptive planning. Ordering the development, delivery, and release of features / deliverables takes into account both what is most important or valuable from a business / mission perspective and the technical dependencies between the items. I cover ordering, and how it differs from prioritization, here.
Frequent Deliveries
Agile teams deliver small increments of valuable business or mission functionality. Each increment builds on previous ones. The system grows and evolves an increment at a time. Every delivery contains working functionality that is potentially releasable. This means that the functionality is ready for deployment and release when authorized.
Figure 3: Waterfall, Agilefall, and Agile Release Frequencies
Agile teams avoid delivering individual components. They strive to deliver fully functional features. Doing so allows project sponsors, stakeholders, customers, and end users to validate that teams delivered the right things.
The faster and more often teams deliver, the more opportunities exist to validate the solution as it grows and evolves. This Agile “empiricism” provides teams the information they need to plan, not just the next timebox or two, but the direction of the solution / product and its development.
At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
https://agilemanifesto.org/principles.html
No Critical Paths in Agile Projects
Since teams define and refine the project plan continually throughout the project, there is no predefined set of work tasks to form a critical path. Also, in the case of Scrum, for example, teams organize work around implementation of user stories, not accomplishment of developer tasks. Rather than focusing on tracking task completion (as traditional project management does), Agile teams track delivered features.
Working software is the primary measure of progress.
https://agilemanifesto.org/principles.html
So, if traditional schedules don’t align to Agile projects, what should we use instead?
The Project Roadmap
Project roadmaps afford teams and project stakeholders the flexibility to shape the evolution of solutions throughout development. Instead of following and tracking a detailed predictive plan via a project schedule, Agile projects can leverage project roadmaps to chart the course of development. The course charted is informed by ongoing verification and validation of potentially releasable and released functionality throughout development.
Figure 4: Charting a course to a solution while navigating changing requirements, impediments, and risks
A project roadmap should depict the delivery of business or mission capabilities at a coarse-grained level. In the case of Scrum, for example, I recommend keeping the project roadmap at the epic level. The idea here is not to track every feature / user story and all of their associated tasks. Instead, the project roadmap should present the major slices of system functionality that deliver complete business or mission utility. However, some development team collaboration tools offer features that make drilling down below the epic level on the roadmap easy.
As depicted in Figure 5, we arrange the epics along a timeline that depicts their estimated start and end dates. Teams arrange the epics in the order they expect to work them, taking into account business / mission priorities and technical dependencies between them.
Figure 5: Agile Project Roadmap Example
Finally, we include major milestones and deadlines that teams must meet as part of project activities. Since Agile teams separate deployments to production from releases, and every timebox offers an opportunity to release, there is no need to clutter up the roadmap with lower-level details.
Conclusion
Project roadmaps are a better way of tracking planned completion and release of business / mission capabilities by Agile teams than traditional project schedules. They align with Agile scope management principles and provide the planning flexibility necessary for Agile projects. They are also much more lightweight and easier to maintain than complex project schedules. Most important, leveraging project roadmaps helps us avoid the tendency to over-plan and over-analyze that often turns Agile projects into traditional project management efforts.
How you use “Agile” support tools could sabotage your agility
Table of Contents
How an organization uses “Agile” requirements management and team collaboration tools like JIRA is an indicator of how well that organization aligns its management of software systems development with Agile software development. The least “Agile-mature” organizations attempt to manage Agile projects with traditional project management biases and expectations. Such organizations often use Agile requirements management and collaboration tools to track and report:
Traditional project management metrics
“To-do” lists of unprioritized and unsized items written like traditional requirements or developer work tasks
All the while, organizational leaders and management assume they are “Agile” because they are using “Agile” tools.
The Place of Tools in Agile
Before expounding on these two general misuses of Agile requirements management and collaboration tools, let’s go over what the Agile Manifesto for Software Development says about tools. Agile values, “Individuals and interactions over processes and tools.” While Agile teams institute processes and use tools to manage their work, they value direct interaction between individuals more. Under this construct, tools are enablers that facilitate communication between team members and project stakeholders. They are not substitutes for direct communication.
Agile prefers face-to-face communication over emails, phone calls, project tools, formal boards, documentation, etc. This way of communicating increases agility by avoiding the use of intermediary channels, documentation, and reporting structures that often slow and distort communication. It also prevents formalizing decisions too early, which unnecessarily narrows options and commits development teams to approaches that may prove wrong after having spent significant time, money, and resources implementing them.
However, not every interaction in and among Agile teams can be face-to-face. Agile development teams with members who work remotely, and enterprises staffed entirely with remote teams, are common. While Agile best practice is to collocate teams and team members, it is not always possible. For example, the project team may be too large to depend solely on direct communication between members.
Agile requirements artifacts are purposely lightweight. They work as reminders for future conversations that will flesh out details. However, even lightweight requirements become difficult to manage as they grow in number. Requirements management and team collaboration tools help document and track requirements, manage work, and report progress. They also help bridge distances and time zones between collaborators.
Project Metrics Tracking
The types of metrics relevant to traditional vs. Agile projects are different. Traditional project management is concerned with managing the constraints of the Iron Triangle: Cost / Budget, Schedule, Scope, and Quality.
Figure 1: The Iron Triangle
Fixing the former three constraints leaves quality as the only variable, opening the door to sacrificing quality to adhere to the other constraints. We can sum up the Iron Triangle with the adage, “I can get it to you cheap, fast, or good. Pick any two.” On the other hand, we manage Agile projects primarily by managing scope. We fix schedule and budget while treating project scope as variable and not sacrificing quality.
Traditional Project Metrics
Keeping the first three constraints of the Iron Triangle fixed throughout a project requires adherence to detailed project plans. Changes to those plans are highly frowned upon since even small changes can ripple across project plans in unexpected ways. Thus, traditional project managers focus on managing people and tasks to keep projects on track with plans. This leads to collecting metrics such as the ones shown in Table 1 below:
Table 1: Common Traditional Project Management Metrics
Agile Project Metrics
Agile projects hold schedule and budget fixed while treating scope as a variable. In essence, we have a span of time, a bucket of money, and the flexibility to build just enough of what is needed to satisfy customers. When schedule or budget run out, we can stop development or pony up more money to build more. We also have the flexibility to stop the project at any time should we decide that we delivered enough functionality or if the project is no longer viable. Cancelling a failing or unnecessary project after spending $10 million is much better than doing so after spending $100 million.
Since Agile teams are self-organizing, they manage their own work. They do not need management to allocate tasks, lead estimation activities, or coordinate between teams. Also, in Agile, “Working software is the primary measure of progress.” Therefore, metrics like percentage complete are unnecessary (and nonsensical) since we regularly evaluate complete and releasable business or mission functionality. In Agile, a feature is either done or it isn’t and teams do not take credit for work on incomplete features. What “complete” means to project teams and stakeholders makes up the “Definition of Done”.
“Agile metrics” help teams:
Manage current work
Plan future work
Communicate project status across teams and project stakeholders
Many Agile approaches / frameworks exist. In the Table 2 below, I provide Scrum metrics and reports as examples. I avoid discussing the details of tracking, reporting, and using these metrics and reports in this article. Instead, I present these examples to drive home the point that what we measure, track, and report in Agile projects is different from that of traditional project management because the two approaches are fundamentally different.
Table 2: Common Scrum Metrics and Reports
Quality Metrics
Measuring the quality of deliverables and acting on findings is crucial to all software development projects, Agile or not. There aren’t Agile-specific measures of quality. However, like traditional requirements, well-formed Agile requirements specify the outcomes and quality characteristics associated functionality must meet. In Scrum, for example, quality criteria inform user story acceptance criteria.
Tracking Traditional Project Management Metrics via “Agile” Tools
Many organizations configure and use tools like JIRA to collect and report traditional project management metrics. This practice demonstrates a fundamental misunderstanding of Agile and how it differs from traditional project management. It is also indicative of organizations that adopt the terminology and mechanics of an Agile approach or framework without embracing its principles. While tools can enable Agile practices, agility is not the result of using tools.
Using Product Backlogs as “To-Do” Lists
Tools like JIRA are primarily designed to support Scrum teams. As such, they organize work through product backlogs. A product backlog is a prioritized list of estimated or “sized” descriptions of business or mission capabilities needed by users or roles (human or automated) to accomplish specific outcomes. Scrum teams typically write these requirements as user stories:
Unfortunately, it is quite common to see user stories written as system components or deliverables or as developer tasks. Many teams also neglect to size and prioritize product backlog items until sprint planning (and sometimes not at all). Rather than managing a product backlog, such teams work from what is ostensibly a to-do list. Doing so takes away the ability of Scrum teams to manage current work, plan future work, and clearly communicate project status.
Conclusion
Requirements management and team collaboration tools like JIRA are useful for documenting and tracking requirements, managing work, and reporting progress. However, using such tools is not the same as being agile, especially when used to support traditional project management processes and reporting. Understanding the differences between Agile approaches and traditional project management is key to ensuring we avoid contorting Agile to support Waterfall processes and expectations.
Increase development and organizational agility, and better serve stakeholders, by integrating Agile, DevOps, and microservices approaches.
Table of Contents
Well-run organizations invest in information systems to achieve specific and measurable goals and objectives. They determine the return on information technology (IT) investments by how much they lower cost, increase productivity, lower risk, and increase quality. This expectation of business or mission value achieved for money spent carries over to how organizations conceive, develop, field, and maintain information systems. Valuable information systems allow users to leverage technology in ways that improve how they work to accomplish organizational goals and objectives.
Improving this alignment between business or mission needs and information systems development has been the holy grail of analysts, engineers, and managers since the dawn of the Information Age. Unfortunately, past engineering and management approaches actually widened the communication gap between business / mission stakeholders and software developers. This state of affairs made information systems:
More expensive to develop and operate;
Take longer to develop than expected;
Less likely to generate expected value after fielding; and
More likely to be cancelled before initial fielding.
Over the last 20 years, innovations in how we develop and field software systems, of all kinds, are breaking down barriers to collaboration between business / mission stakeholders and development teams. Arguably, the three most important innovations are Agile software development, DevOps, and Microservices Architecture.
Here, I discuss how the three approaches can work together to address the aforementioned shortfalls. We aim to foster tighter collaboration between business / mission stakeholders and development teams as we improve development and organizational agility.
The Agility Triad
Today, mature software development organizations can maximize development and organizational agility by integrating Agile software development, DevOps and microservice architectural practices. Agile Cheetah defines agility as:
Agility = Speed + Control + Flexibility
We call this definition “The Agility Triad.” The Agility Triad’s elements apply to organizational, process, and technological agility. The three elements are mutually dependent and reinforcing constraints defined as follows:
Figure 1: Agility Triad Elements Definitions
DevOps and Microservices Contribute to Agility
DevOps and microservice architectures provide the means of implementing the Agility Triad’s elements. DevOps serves as the Agile implementation of the Software Development Lifecycle (SDLC). Microservices help align software deliverables directly to the business / mission domain and requirements. They also help structure software development teams, their work, and what they deploy to directly address business or mission needs.
DevOps as the Agile SDLC
Agile projects perform the same SDLC activities as Waterfall projects. The difference is that software teams execute the Agile SDLC as a continuous flow of activity rather than as a linear, one-time progression through time-bound phases.
Throughout development, Agile teams define, develop, test, deploy, and release features. Rather than waiting until the end of the project for a finished system, project stakeholders, customers, and end users gain access to released features throughout the project. These features, individually and collectively, are valuable business / mission capabilities ready for use upon release.
Every feature goes through its own mini–Agile SDLC iteration. Mature Agile teams leverage automation to continually test and integrate features into complete increments of releasable functionality. Newly deployed and released increments work with all previous increments. The system grows and evolves iteratively and incrementally.
At the project level, the Agile SDLC is essentially a “swirl” or “flow” of activity occurring concurrently across multiple features and increments. The most mature Agile teams release so quickly and often that their releases approach a continuous flow.
DevOps is the implementation of this flow-based approach to software development. DevOps is not an Agile concept, nor did it start with Agile. However, it aligns perfectly with Agile because it reinterprets the SDLC in ways that support Agile’s iterative and incremental value delivery and team-based culture. It also aligns with Lean’s emphasis on achieving flow and promoting continuous improvement.
Ways DevOps Enables Agility
I now explain how DevOps enables the three elements of the Agility Triad.
Figure 3: The Agility Triad
DevOps Contribution to Project Speed
DevOps accelerates software delivery by leveraging modern software development techniques, tools, and technologies and by eliminating wasteful handoffs between development teams and operations staff. This culture of automation and continuous improvement facilitates faster iterations through the SDLC for every feature developed.
DevOps automation goes beyond automating the testing, packaging, and deployment of source code. Automated building and configuration of host environments is also a critical facet of DevOps. This capability is termed, “Infrastructure as Code (IaC).”
IaC allows for the repeatable provisioning and configuration of host environments via execution of code. Rather than incurring the time and cost of manually setting up environments, infrastructure automation platforms provision and configure virtual environments by executing scripts and implementing configuration files.
Project management can make decisions about the state of the project based, not on estimated levels of completion but, on information gained through actual deployment, release, and use of functionality. This arms managers with the information they need to decide whether to continue the current course, update project plans, or terminate the project. All along, teams deliver complete increments of functionality capable of providing value immediately.
DevOps Contribution to Project Flexibility
DevOps contributions to software project flexibility include:
Faster and more frequent iterations through the SDLC that allow experimentation with different approaches and quickly pivoting away from failing or undesirable approaches.
The ability to separate software deployments in production from releases, which enables:
Testing in Production – Teams can deploy and test new software features in production before making them available to users. This allows teams to experiment with different versions of the same functionality and see which performs best.
Releasing on Demand – Rather than deploying and releasing on a fixed schedule, teams can deploy functionality at any time and make it available to users when they are ready to use it.
Monolithic vs. Microservices Architectures
Traditional software projects delivered solutions all at once. In other words, software teams deployed and released all application components as one unit. Changes to an application often required complete redeployment. In microservices parlance, we refer to applications architected this way as monoliths.[1]
While architects and developers typically split monoliths into packages, modules, or components, interdependencies between those components make it difficult to develop, test, and deliver them separately. In systems engineering terms, these system components are “tightly coupled.”
A microservices architecture, on the other hand, builds applications as sets of independently deployable services that represent key entities of a particular business or mission domain. For example, an inventory system may include services that handle transactions associated with customers, invoices, warehouses, payment, etc. All of the information / data associated with each of these entities belongs to its respective service. A service needing information from outside the confines of its own entity (e.g., the warehouse service needs a customer’s address) must get it from the corresponding service.
Three-Tiered (Web) Applications
Since the 1990’s, the predominant architectural model for monolith applications has been the three-tiered (web) application. This distributed architecture grouped together components based on where they fit within each of the following tiers:
Figure 4: Three-Tiered (Web) App Monolith
Front End: Implements the application’s user interface(s) or externally-facing interface(s)/API(s)
Middle Tier: Performs the bulk of the application’s business logic
Datastore: Manages application data and state
Microservices Cut Across Three-Tiered Architectures
In contrast, since microservices are end-to-end slices of business or mission functionality, each microservice includes just enough of the three tiers to perform its job.
Figure 5: Each Microservice Includes Components from Each Tier
Individual microservices are not coupled to the internal workings of other services. Thus, microservices are independent or “loosely coupled.” As long as the interfaces they expose do not change, internal changes to individual microservices will not ripple across the application or system. Teams can deploy and release microservices independently, allowing applications to grow and evolve iteratively and incrementally.
Figure 6: Microservice Composition
Features Map to Microservices
As stated earlier, Agile value delivery is feature-based. This differs from the component-based delivery of functionality performed in traditional software projects.
In traditional software projects, management organizes teams to develop system components which they later integrate into complete solutions. Until that integration happens, nothing of business or mission value exists. Complete and cohesive functionality / features only exist after system integration.
System integration occurs infrequently in traditional software projects. This increases project risk. Validation of an application’s actual business / mission value typically occurs towards the end of the project, when budget and schedule are scarce.
As end-to-end slices of business or mission functionality, microservices help development teams develop, test, and deliver applications, iteratively and incrementally, as features, from the start. A feature may be satisfied by one microservice or through orchestration of multiple microservices.
Ways Microservices Enable Agility
Agile approaches, such as Scrum, help teams organize and manage their work as increments of functionality (features / user stories), developed within short timeframes (timeboxes), as part of frequent releases. DevOps provides the automation and coordination required to continually and sustainably deliver features agilely. Microservices are as close as we have come to directly implementing, deploying, and releasing requested features as features.
Microservices Contribution to Project Speed
Microservices accelerate software applications development and fielding by:
Breaking down application scope into smaller and more manageable units of work – A concept of Lean is that smaller batches of work increase flow. Microservices make it easier for development teams to manage their work through Agile and DevOps practices.
Allowing customers and end users access to valuable features earlier – Rather than waiting until full system deployment to validate and use a solution, customers and end users gain access to useful functionality throughout development. Lessons learned from released functionality help inform ongoing development.
Allowing development teams to work in parallel – Development teams can work on different microservices in parallel with little worry about inter-service dependencies beyond the interfaces the services expose.
Facilitating mixing and matching of technologies and technical approaches – Microservice independence also applies to the technology stacks and platforms used to implement them. Teams can develop microservices in whatever technologies they are most comfortable with and still achieve service integration and interoperability. Since microservices communicate via networks, they do not need to run on the same platform or under the same operating system. This technology independence lowers the amount of learning and coordination required across development teams.
Lowering the risk associated with fielding new functionality – Adding or replacing individual microservices is much easier and less risky than redeploying monolith applications. This flexibility lowers outage times.
Microservices Contribution to Controlling Projects
Microservices help teams and project management control projects by:
Facilitating Development Around Features – Agile teams control projects by managing scope. Breaking up system scope into features helps teams manage their work. Microservices support the iterative and incremental development and release of features. This strategy affords teams multiple opportunities throughout development to evaluate progress and update project plans based on actual use.
Lowering Inter-Team Dependencies – Agile software development teams are cross-functional. Members possess the skills necessary to develop complete, coherent, and deployable features across short timeboxes. This organization around features, rather than components, aligns with the tenet of building microservices that include all of the components necessary to perform as independently deployable services. Organizing teams along the lines of the microservices they develop lowers dependencies across teams and associated integration risks.
Figure 7: Tier-Based / Component Teams vs. Microservices / Feature Teams
Microservices Contribution to Project Flexibility
Spreading application scope across individual microservices simplifies changes and promotes experimentation. Teams can change their piece of the application without fear of affecting the rest. Thus, there is less resistance to trying out different approaches and settling on the most optimal. For example, we can deploy different versions of the same microservice with different user interfaces in production, selectively release them to users, and evaluate them. This technique is known as A/B testing.
Summary of DevOps and Microservices Contributions to Agility
I summarize the contributions DevOps and Microservice Architecture can make to agility below:
Figure 8: DevOps and Microservices Contributions to Agility
Conclusion
Proper integration of Agile, DevOps, and microservice architectural approaches can narrow the gulf between business or mission stakeholders and those who deliver software solutions. We do this by improving development and organizational agility. Under this construct:
Agile planning and execution are supported by DevOps processes
DevOps processes accelerate development, testing, packaging, and deployment of microservice-based applications
Resulting applications are better aligned with Agile delivery and business / mission needs
The decision to adopt any of the three approaches is not be taken lightly. There is no one-size-fits-all approach to making them work and few organizations make all three work well. It may also be the case that any of three approaches are not right for your organization or specific project.
However, these approaches are firmly in the canon of modern software development practice. It is worthwhile investigating how your organization can improve how it practices these approaches or begin adopting them.
[1] Sam Newman provides a comprehensive explanation of the ideas covered in this section in his book: Monolith to Microservices, 2020, O’Reilly, Chp 1
