Why Integrate Agile, DevOps, & Microservices?

Increase development and organizational agility, and better serve stakeholders, by integrating Agile, DevOps, and microservices approaches.

Table of Contents

Well-run organizations invest in information systems to achieve specific and measurable goals and objectives. They determine the return on information technology (IT) investments by how much they lower cost, increase productivity, lower risk, and increase quality. This expectation of business or mission value achieved for money spent carries over to how organizations conceive, develop, field, and maintain information systems. Valuable information systems allow users to leverage technology in ways that improve how they work to accomplish organizational goals and objectives.

Improving this alignment between business or mission needs and information systems development has been the holy grail of analysts, engineers, and managers since the dawn of the Information Age. Unfortunately, past engineering and management approaches actually widened the communication gap between business / mission stakeholders and software developers. This state of affairs made information systems:

More expensive to develop and operate;
Take longer to develop than expected;
Less likely to generate expected value after fielding; and
More likely to be cancelled before initial fielding.

Over the last 20 years, innovations in how we develop and field software systems, of all kinds, are breaking down barriers to collaboration between business / mission stakeholders and development teams. Arguably, the three most important innovations are Agile software development, DevOps, and Microservices Architecture.

Here, I discuss how the three approaches can work together to address the aforementioned shortfalls. We aim to foster tighter collaboration between business / mission stakeholders and development teams as we improve development and organizational agility.

The Agility Triad

Today, mature software development organizations can maximize development and organizational agility by integrating Agile software development, DevOps and microservice architectural practices. Agile Cheetah defines agility as:

Agility = Speed + Control + Flexibility

We call this definition “The Agility Triad.” The Agility Triad’s elements apply to organizational, process, and technological agility. The three elements are mutually dependent and reinforcing constraints defined as follows:

DevOps and Microservices Contribute to Agility

DevOps and microservice architectures provide the means of implementing the Agility Triad’s elements. DevOps serves as the Agile implementation of the Software Development Lifecycle (SDLC). Microservices help align software deliverables directly to the business / mission domain and requirements. They also help structure software development teams, their work, and what they deploy to directly address business or mission needs.

DevOps as the Agile SDLC

Agile projects perform the same SDLC activities as Waterfall projects. The difference is that software teams execute the Agile SDLC as a continuous flow of activity rather than as a linear, one-time progression through time-bound phases.

Agile projects are feature-based. In the context of Agile, features are cohesive and complete slices of system functionality capable of performing useful work from the perspective of customers and end users (human or automated).

Throughout development, Agile teams define, develop, test, deploy, and release features. Rather than waiting until the end of the project for a finished system, project stakeholders, customers, and end users gain access to released features throughout the project. These features, individually and collectively, are valuable business / mission capabilities ready for use upon release.

Every feature goes through its own mini–Agile SDLC iteration. Mature Agile teams leverage automation to continually test and integrate features into complete increments of releasable functionality. Newly deployed and released increments work with all previous increments. The system grows and evolves iteratively and incrementally.

At the project level, the Agile SDLC is essentially a “swirl” or “flow” of activity occurring concurrently across multiple features and increments. The most mature Agile teams release so quickly and often that their releases approach a continuous flow.

DevOps is the implementation of this flow-based approach to software development. DevOps is not an Agile concept, nor did it start with Agile. However, it aligns perfectly with Agile because it reinterprets the SDLC in ways that support Agile’s iterative and incremental value delivery and team-based culture. It also aligns with Lean’s emphasis on achieving flow and promoting continuous improvement.

Ways DevOps Enables Agility

I now explain how DevOps enables the three elements of the Agility Triad.

DevOps Contribution to Project Speed

DevOps accelerates software delivery by leveraging modern software development techniques, tools, and technologies and by eliminating wasteful handoffs between development teams and operations staff. This culture of automation and continuous improvement facilitates faster iterations through the SDLC for every feature developed.

DevOps automation goes beyond automating the testing, packaging, and deployment of source code. Automated building and configuration of host environments is also a critical facet of DevOps. This capability is termed, “Infrastructure as Code (IaC).”

IaC allows for the repeatable provisioning and configuration of host environments via execution of code. Rather than incurring the time and cost of manually setting up environments, infrastructure automation platforms provision and configure virtual environments by executing scripts and implementing configuration files.

DevOps Contribution to Controlling Projects

Faster and more frequent iterations through the SDLC support greater control of project scope, budget, and schedule through increased empiricism. In other words, Agile teams and project management monitor progress, make decisions, and act based on lessons learned from previous releases. The faster and more frequently teams iterate through the SDLC, the faster and more frequently they receive feedback that informs upcoming work plans (scope).

Project management can make decisions about the state of the project based, not on estimated levels of completion but, on information gained through actual deployment, release, and use of functionality. This arms managers with the information they need to decide whether to continue the current course, update project plans, or terminate the project. All along, teams deliver complete increments of functionality capable of providing value immediately.

DevOps Contribution to Project Flexibility

DevOps contributions to software project flexibility include:

Faster and more frequent iterations through the SDLC that allow experimentation with different approaches and quickly pivoting away from failing or undesirable approaches.
The ability to separate software deployments in production from releases, which enables:
- Testing in Production – Teams can deploy and test new software features in production before making them available to users. This allows teams to experiment with different versions of the same functionality and see which performs best.
- Releasing on Demand – Rather than deploying and releasing on a fixed schedule, teams can deploy functionality at any time and make it available to users when they are ready to use it.

Monolithic vs. Microservices Architectures

Traditional software projects delivered solutions all at once. In other words, software teams deployed and released all application components as one unit. Changes to an application often required complete redeployment. In microservices parlance, we refer to applications architected this way as monoliths.[1]

While architects and developers typically split monoliths into packages, modules, or components, interdependencies between those components make it difficult to develop, test, and deliver them separately. In systems engineering terms, these system components are “tightly coupled.”

A microservices architecture, on the other hand, builds applications as sets of independently deployable services that represent key entities of a particular business or mission domain. For example, an inventory system may include services that handle transactions associated with customers, invoices, warehouses, payment, etc. All of the information / data associated with each of these entities belongs to its respective service. A service needing information from outside the confines of its own entity (e.g., the warehouse service needs a customer’s address) must get it from the corresponding service.

Three-Tiered (Web) Applications

Since the 1990’s, the predominant architectural model for monolith applications has been the three-tiered (web) application. This distributed architecture grouped together components based on where they fit within each of the following tiers:

Front End: Implements the application’s user interface(s) or externally-facing interface(s)/API(s)
Middle Tier: Performs the bulk of the application’s business logic
Datastore: Manages application data and state

Microservices Cut Across Three-Tiered Architectures

In contrast, since microservices are end-to-end slices of business or mission functionality, each microservice includes just enough of the three tiers to perform its job.

Individual microservices are not coupled to the internal workings of other services. Thus, microservices are independent or “loosely coupled.” As long as the interfaces they expose do not change, internal changes to individual microservices will not ripple across the application or system. Teams can deploy and release microservices independently, allowing applications to grow and evolve iteratively and incrementally.

Features Map to Microservices

As stated earlier, Agile value delivery is feature-based. This differs from the component-based delivery of functionality performed in traditional software projects.

In traditional software projects, management organizes teams to develop system components which they later integrate into complete solutions. Until that integration happens, nothing of business or mission value exists. Complete and cohesive functionality / features only exist after system integration.

System integration occurs infrequently in traditional software projects. This increases project risk. Validation of an application’s actual business / mission value typically occurs towards the end of the project, when budget and schedule are scarce.

As end-to-end slices of business or mission functionality, microservices help development teams develop, test, and deliver applications, iteratively and incrementally, as features, from the start. A feature may be satisfied by one microservice or through orchestration of multiple microservices.

Ways Microservices Enable Agility

Agile approaches, such as Scrum, help teams organize and manage their work as increments of functionality (features / user stories), developed within short timeframes (timeboxes), as part of frequent releases. DevOps provides the automation and coordination required to continually and sustainably deliver features agilely. Microservices are as close as we have come to directly implementing, deploying, and releasing requested features as features.

Microservices Contribution to Project Speed

Microservices accelerate software applications development and fielding by:

Breaking down application scope into smaller and more manageable units of work – A concept of Lean is that smaller batches of work increase flow. Microservices make it easier for development teams to manage their work through Agile and DevOps practices.
Allowing customers and end users access to valuable features earlier – Rather than waiting until full system deployment to validate and use a solution, customers and end users gain access to useful functionality throughout development. Lessons learned from released functionality help inform ongoing development.
Allowing development teams to work in parallel – Development teams can work on different microservices in parallel with little worry about inter-service dependencies beyond the interfaces the services expose.
Facilitating mixing and matching of technologies and technical approaches – Microservice independence also applies to the technology stacks and platforms used to implement them. Teams can develop microservices in whatever technologies they are most comfortable with and still achieve service integration and interoperability. Since microservices communicate via networks, they do not need to run on the same platform or under the same operating system. This technology independence lowers the amount of learning and coordination required across development teams.
Lowering the risk associated with fielding new functionality – Adding or replacing individual microservices is much easier and less risky than redeploying monolith applications. This flexibility lowers outage times.

Microservices Contribution to Controlling Projects

Microservices help teams and project management control projects by:

Facilitating Development Around Features – Agile teams control projects by managing scope. Breaking up system scope into features helps teams manage their work. Microservices support the iterative and incremental development and release of features. This strategy affords teams multiple opportunities throughout development to evaluate progress and update project plans based on actual use.
Lowering Inter-Team Dependencies – Agile software development teams are cross-functional. Members possess the skills necessary to develop complete, coherent, and deployable features across short timeboxes. This organization around features, rather than components, aligns with the tenet of building microservices that include all of the components necessary to perform as independently deployable services. Organizing teams along the lines of the microservices they develop lowers dependencies across teams and associated integration risks.

Microservices Contribution to Project Flexibility

Spreading application scope across individual microservices simplifies changes and promotes experimentation. Teams can change their piece of the application without fear of affecting the rest. Thus, there is less resistance to trying out different approaches and settling on the most optimal. For example, we can deploy different versions of the same microservice with different user interfaces in production, selectively release them to users, and evaluate them. This technique is known as A/B testing.

Summary of DevOps and Microservices Contributions to Agility

I summarize the contributions DevOps and Microservice Architecture can make to agility below:

Conclusion

Proper integration of Agile, DevOps, and microservice architectural approaches can narrow the gulf between business or mission stakeholders and those who deliver software solutions. We do this by improving development and organizational agility. Under this construct:

Agile planning and execution are supported by DevOps processes
DevOps processes accelerate development, testing, packaging, and deployment of microservice-based applications
Resulting applications are better aligned with Agile delivery and business / mission needs

The decision to adopt any of the three approaches is not be taken lightly. There is no one-size-fits-all approach to making them work and few organizations make all three work well. It may also be the case that any of three approaches are not right for your organization or specific project.

However, these approaches are firmly in the canon of modern software development practice. It is worthwhile investigating how your organization can improve how it practices these approaches or begin adopting them.

[1] Sam Newman provides a comprehensive explanation of the ideas covered in this section in his book: Monolith to Microservices, 2020, O’Reilly, Chp 1

Let us know what you think!!!

Agile Teams Release Frequently

Frequent releases help buy down project risk while making valuable software capabilities available sooner.

Table of Contents

Agile is the default software development approach worldwide. The foundation upon which Agile approaches (e.g., Scrum, Kanban) rest consists of the Agile Manifesto for Software Development and the 12 Principles of Agile Software. We consider practices that align with the Manifesto’s values and underlying principles to be Agile. Unfortunately, too many organizations do not achieve the benefits of agility because they engage in the mechanics of an Agile approach without understanding and implementing its foundational values and principles.

One of the most common examples of this disconnect between Agile software development practice, values, and principles is organizing work into iterations or sprints while releasing infrequently, often only towards the end of a project. By release, I mean making new software functionality available to customers and end users. This practice runs counter to the third principle of Agile software development:

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

The 12 Principles of Agile Software Development: https://agilemanifesto.org/principles.html

Agile Empiricism

Agile software development teams do not immediately release everything they develop. Sometimes they must wait on external technical dependencies. Other times, customers and end users are not ready to accept new functionality. Examples include:

Users may need training on new functionality
Business and mission processes and workflows may need to change before release
Project sponsors and stakeholders may decide that a particular release candidate is not what is needed or desired

However, product management/owners, sponsors, and stakeholders should strive to release valuable business or mission capabilities frequently, cost effectively, and in time to be valuable. You see, it is not possible to determine the actual business or mission value of a software feature or capability until actual customers and end users use it to achieve desired outcomes. Until then, software features are hypotheses of what product teams believe will best address customer and end user needs.

Allowing too much time to pass before exposing an evolving solution to target customers and end users risks developing unwanted and/or under-performing features and capabilities. Since Agile projects are controlled primarily by managing scope, losing touch with customer and end user needs translates into:

Wasted effort
Higher development, testing, and operational costs
Longer development cycles
Less feedback to inform product management, project management, and Software Development Lifecycle (SDLC) activities

Agile development is empirical. In other words, Agile teams monitor progress, make decisions, and act based on lessons learned from previous releases. In Agile efforts, customer and end-user input and feedback inform planning, implementation, testing, and release of software features and capabilities. Thus, Agile teams deliver and release relatively small increments of functionality within short time frames (weeks). They continually gauge the level of business or mission value software release increments provide and how well those increments perform relative to technical specifications.

Agilefall

The less frequently software development teams release, the less “Agile” a software project becomes. The project increasingly resembles a traditional project management / Waterfall project.

This conflating of Agile and Waterfall is often referred to as Agilefall. Agilefall is not a formal SDLC approach. Thus, there is no set of best practices to guide its implementation. In fact, Agilefall is an Agile anti-pattern.

The typical Agilefall scenario involves Scrum Teams developing and testing system components for eventual integration, deployment, and release to production. Software teams develop and deploy components or portions of components every iteration. They may do some degree of integration during development but deliver and release little to nothing of business or mission value to customers and end users until near the end of the project. Customer and end user input and feedback do not inform iteration planning.

Click to Expand — Figure 1: Waterfall, Agilefall, and Agile Release Frequencies

Agilefall typically leads to a divergence between what the solution offers and customer and end user needs. It also leads to costly problems during integration and testing since development teams have fewer opportunities to integrate what they develop and to test those integrations. Agilefall is, in essence, a Waterfall project with software teams performing Scrum “mechanics”.

Agile Delivery

Agile software releases are feature based, not component-based. Instead of delivering system components, Agile software teams deliver features. Features are cohesive and complete slices of system functionality capable of performing useful work from the perspective of customers and end users. This approach to value delivery ensures that:

Customers and end users gain access to valuable functionality early and often
Lessons learned from previous releases inform future releases

The more frequently software teams deliver and release, the more opportunities exist to course correct planning and execution. Requirements are often fuzzy until members of the stakeholder community can see and experience functionality for themselves. Problems are easier and cheaper to fix when caught early in development.

DevOps

DevOps is an Agile implementation of the SDLC. It facilitates Agile’s iterative and incremental delivery model. DevOps leverages automation and a culture of close collaboration between development and operations teams to:

Promote faster and more frequent deployment of useful capabilities through delivery of small increments of business or mission value
Apply lessons learned and insights gained from customer and end-user interaction with released software
Avoid time-wasting hand-offs between phases and teams
Accelerate development, testing, deployment, and release of software features as well as their integration into complete and cohesive systems or solutions

Conclusion

Frequent releases help buy down risk across the entire SDLC while making valuable business or mission software capabilities available sooner. DevOps practices enable this value delivery approach. The best Agile software teams release so frequently, their releases approach a continuous flow.

The promise of Agile is to deliver software capabilities faster and more cost effectively, while improving quality and alignment to customer and end user needs. Keeping this promise requires adherence to fundamental Agile values and principles, regardless of the chosen Agile approach.

Let us know what you think!!!

Traditional vs. Agile Risk Management

System integration software projects require formal risk management. Here’s how to do it agilely.

Table of Contents

If you work in system integration projects, you have likely wondered how to manage project risks within your organization’s chosen Agile approach. This is a topic seldom discussed in certification courses for Agile-based approaches such as Scrum. The two major reasons for this are:

Most of these courses focus exclusively on team-level Agile. In other words, how does a small team of individual contributors work together under the Agile approach? Individual small teams tend to work on smaller efforts that require much less coordination. In most of these situations, formalized risk management is overkill.
Managing risk is baked into Agile because Agile is an inherently empirical way of working. Since Agile development is iterative and incremental, teams narrow the scope of what they build to fit within short time frames. This enables them to, quickly and frequently, build, test, fix, deploy, evaluate and change what they build, and how. Every iteration of value delivery brings multiple opportunities to address risk “in real time.” In other words, Agile spreads risk management across the entire development effort.

However, system integration projects are typically large and complex, with multiple teams working across organizations while managing multiple dependencies. The larger the project, the more risk project sponsors and stakeholders assume. A formalized approach to risk management becomes a priority.

So, what does a formalized Agile risk management approach look like? It looks a lot like a traditional risk management approach with three main differences:

Agile risk management is more adaptive than predictive
Agile projects perform risk management activities more frequently and faster than traditional projects
Risk management activities in Agile projects are much more coupled to project execution than in traditional projects

Before describing a formalized Agile risk management approach, let’s level set understanding about project risk management. Along with the short background provided here, I highly recommend reading this article from pmi.org about risk analysis and management.

Risks, Opportunities, and Issues

The third edition of the Project Management Body of Knowledge (PMBOK Guide) defines risk as:

An uncertain event or condition that, if it occurs, has a positive or negative effect on a project’s objectives.

Project Management Body of Knowledge (PMBOK Guide), Third Edition

By that definition, risks are not just harbingers of negative outcomes. They may also be opportunities to affect positive outcomes. Once a negative risk becomes reality, it becomes an issue.

Risk Categories

Categorizing project risks is helpful. Doing so facilitates risk tracking across the entire project scope. It also communicates that risk management is everyone’s job, not just that of project management. Table 1 lists a typical set of project risk categories and their associated extended categories[1]:

Risks belonging to the external, organizational, and project management categories are typically:

Closely aligned to high-level project concerns outside the control of development teams
Less dynamic and fluid than technical risks
Almost exclusively in the realm of project and program management and/or product management

Much of Agile literature and training narrowly focus on technical risks. This aligns with their focus on Agile execution by small teams. However, external, organizational, and project management risks often impact projects far more.

The Risk Management Cycle

Both traditional and Agile projects execute the activities of the Risk Management Cycle depicted in Figure 1. Table 2 lists and describes each activity of the cycle[2]:

Predictive vs. Adaptive Planning

Risk management is an important part of project planning. Traditional project management engages in predictive planning, while Agile performs adaptive planning. The chosen planning approach affects how projects manage risk.

Traditional project management tries to account for all project tasks and milestones upfront. Agile planning breaks up the planning horizon into increments. These increments facilitate changing plans when circumstances change. To be effective, a project’s risk management approach must align to the chosen project management approach.

While all projects, whether they adopt a predictive or adaptive planning approach, perform every activity of the Risk Management Cycle, they emphasize different activities.

A predictive risk management approach emphasizes risk forecasting (prediction) and analysis. Risk analysis entails determining the probability of occurrence and expected project impact of individual risks. The rationale is that the earlier project managers identify and prioritize project risks, the sooner they can address them before they become issues.

An adaptive risk management approach emphasizes keeping risks small by defining, developing, and delivering system functionality iteratively and incrementally. This allows teams to handle risks while their potential for project impact is small and to continually monitor risks as solutions evolve and grow.

Traditional vs. Agile Risk Management Characteristics

I’ll now discuss the key differences between how traditional and Agile projects manage risk.

Traditional Risk Identification and Evaluation

Traditional projects develop and execute detailed plans that span the entire project timeline. Project management often breaks up larger projects into planning packages as part of “rolling wave planning”, but each of those planning packages are themselves detailed plans. Project plans are based on the resources needed to complete the entire scope of deliverables at a certain cost (The Iron Triangle).

Project plans start with requirements definition. The entire set of approved requirements defines project scope. Changes to requirements mean changes to project scope, which translate into schedule and cost changes.

Similarly, traditional risk management tries to identify and evaluate project risks and opportunities prior to or at the start of projects. While risk management is supposed to happen throughout a project’s timeline, traditional projects tend to perform the bulk of Risk Identification and Evaluation upfront.

The problem with this arrangement is that people cannot predict the future. The larger and more complex the project, the more likely project requirements and risks will change.

Traditional project management develops and executes plans assuming requirements, and the design approaches that implement them, are inherently correct and will not change. This opens projects to the greatest risk of all: building the wrong thing.

Risk Identification and Evaluation activities in traditional projects often cannot keep up with the accelerated pace of modern software development. In many organizations, risk management is relegated to weekly or monthly risk management boards. By the time a board adjudicates a risk it may already be an issue.

Agile Risk Identification and Evaluation

While some degree of upfront Risk identification and Evaluation happens in Agile projects, the planning horizon is typically far shorter than that of traditional projects. Also, most Risk Identification and Evaluation of technical risks happen as part of development, rather than resulting from a separate risk management process.

Traditional Risk Handling and Controlling

Traditional (Waterfall) projects typically get one pass through the Software Development Lifecycle (SCLC). That pass is strictly constrained by schedule, resources, and cost. As a result, there is often little room to add risk management activities to the schedule.

Project managers and developers are incentivized to deliver according to plan. This puts pressure on them to ignore risks until they become issues. They also forgo opportunities in favor of “flying the plan.”

Agile Risk Handling and Controlling

The Agile SDLC is fast moving and iterative. Every feature software development teams deliver goes through its own “mini-SDLC”. Agile planning coupled with DevOps automation support this:

Agile provides frequent opportunities to evaluate what development teams build and how
Adaptive planning allows sponsors, stakeholders, and development teams to change requirements, plans, and approaches
Teams add risks to their work backlogs and prioritize them, thus integrating Risk Handling actions into team workflows
Shorter development timeframes with frequent deliveries / releases provide: 1) More frequent and earlier warning of emerging risks; 2) More feedback from which to gauge the effectiveness of risk management efforts

Recommended Approach

The Agile SDLC improves management of technical risks. However, as stated earlier, external, organizational, and project management risks often impact projects far more than technical risks. Agile teams have the flexibility to manage technical scope. They have far less control over funding and major project milestones.

External, organizational, and project management risks influence the overall business or mission context within which teams develop solutions. Project leadership must actively manage these risks before they impact the work of development teams.

Agile projects should leverage traditional risk management tools and techniques to manage more strategic, slower-moving risks. Project leaders would then work with development teams to translate such risks into actionable work items and prioritize them. All project members should have access to risk management reporting and the ability to submit risk candidates as part of their work.

Conclusion

Too often, risk management gets short shrift, especially in Agile projects. Risk management is an integral part of project planning, regardless of the software development approach employed. Combining Agile requirements definition and planning, with DevOps automation, and a robust risk management process helps teams deliver valuable software capabilities, cost-effectively, and in time to be valuable.

[1,2] Lavanya, N., Malarvizhi, T., “Risk Analysis and Management,” Project Management Institute, 3 March 2008, https://www.pmi.org/learning/library/risk-analysis-project-management-7070

Let us know what you think!!!

Why So Many Developers are Fed Up with Agile (Pt. 4)

Changing Expectations for Developers

Table of Contents

Developers are moving away from functioning as “order takers”, focused almost exclusively on coding, to collaborating with solution sponsors, stakeholders, customers, and end users (i.e., Stakeholder Community) to define, verify, and validate what they develop. Many technical contributors find this level of direct collaboration difficult to manage alongside their growing list of technical responsibilities.

This article is the last of a four-part series about why so many developers are fed up with Agile and why so many organizations fail to achieve Agile’s much-touted benefits. The common thread across the series is that what often passes for Agile is anything but Agile.

The other topics covered in this series are:

The Never-Ending Agile Grind – Developers often feel overwhelmed by the constant pressure to produce under tight deadlines.
Greater Visibility & Oversight with Less Control – The “radical transparency” of Agile approaches, such as Scrum and Kanban, is often exploited by management as an opportunity to micromanage development teams.
Top-Down Driven Agile – Technical contributors often have little say in how their organizations or sponsors implement Agile, leading to the imposition of unnecessary, ineffective, and, often, counterproductive practices on development teams.

Difficulties Collaborating with Stakeholders

One of the defining characteristics of Agile is the fostering of close and ongoing collaboration between software development teams and their Stakeholder Communities. As the fourth Principle of Agile Software Development states:

Business people and developers must work together daily throughout the project

http://agilemanifesto.org/principles.html

Needs, expectations, and feedback from the Stakeholder Community guide what Agile teams build. Delivering features and capabilities iteratively and incrementally affords more opportunities to inspect and adapt deliverables and team processes accordingly.

The ultimate validation of a solution’s value is use by real customers and end users. Prior to release, Agile teams test and validate software increments in collaboration with their respective Stakeholder Communities. After release, development and/or operations teams monitor and evaluate how evolving solutions are used and perform. Lessons learned inform decisions about upcoming development.

Many Agile projects fail because this vital feedback loop breaks down or is never established. Achieving effective collaboration between groups of people who hold different ideas about what is needed or important is difficult. The trust and respect upon which collaboration depends take time and effort to build and are easily undermined. Avoiding or mitigating misunderstandings, disagreements, and conflicts starts with people understanding each other’s perspectives.

The Developers’ Perspective

Below are some of the most common issues developers encounter when dealing with Stakeholder Community members.

There is also the issue of whether development teams have access to the right stakeholders (or any stakeholders at all). Too often, those who provide requirements do not have intimate understanding of relevant business or mission processes or of customer and end-user needs. This is especially true in organizations with traditional project management cultures where project or program managers, architects, and senior engineers are expected to “know” customer needs yet rarely interact with them.

The Stakeholders’ Perspective

Collaboration is a two-way street. Agilists approach requirements from a customer-centric perspective. Unfortunately, too many developers think about solutions from the perspective of what they want to build. They define requirements as development tasks to accomplish or as system functionality to develop. In the eyes of the Stakeholder Community, these attitudes come across in the following ways:

Expanding Technical Responsibilities

Along with increased expectations for Agile development teams to interact with stakeholders, the scope of work done by software developers continues to grow.

Back in the 80s and 90s, a software developer was considered proficient if he or she specialized in one operating system and one programming language. In the 90s and the aughts, many, if not most, developers specialized as front-end or back-end developers. There were much fewer tools to master and most development was custom rather than based on third-party components and frameworks. Also, the work associated with the full lifecycle management of applications was highly specialized and assigned to different people.

The advent of the Cloud and DevOps changed all of that. Now developers are expected to be “full-stack developers” capable of developing both the front and backend components of complete releasable features. Virtual computing and the Cloud broke down operating system-based development siloes. Open-source development wildly increased the number of tools and frameworks available. Great efficiencies brought about by DevOps came at the price of developers learning DevOps tools, developing automation scripts, and managing DevOps pipelines.

A proficient developer today is a master of multiple programing languages, highly conversant in more than one operating system, comfortable using a plethora of tools and frameworks, and regularly contributes to testing and DevOps automation. Software developers now perform a lot of work that used to be set aside for specialists such as system administrators, configuration managers, build engineers, testers, etc. While a lot of that work is now automated or abstracted away from developers, the scope of what is considered development work is much larger than before.

Many software developers feel overwhelmed by expectations to keep up with the ever-increasing pace of technological change, consistently deliver value, and collaborate with stakeholders. Most developers entered the field for the opportunity to develop software for a living and concentrate on becoming better technologists. They become highly proficient at building software but lack the business or mission insight necessary to better align solutions with business or mission needs.

Proposed Solutions

Development teams do not need to choose among better collaboration with stakeholders, maintaining technical excellence, and delivering value cost effectively and in time to be valuable. To achieve that balance, we recommend the following actions.

Promote Strong Product Ownership

While not all Agile teams are Scrum teams, Scrum’s concept of Product Ownership is useful for managing Stakeholder Communities and can be tailored to work with non-Scrum Agile teams. In Scrum, the “product” is synonymous with whatever technical capability a Scrum Team develops, be it a system, solution, application, etc.

A Scrum Product Owner (PO) is responsible for fusing the various needs, wants, and perspectives of product and project stakeholders into one complete, coherent, and cohesive vision. The Scrum Development Team looks to the PO as the primary (but not the only) source of business or mission knowledge and understanding. The PO enlists the help of the development team and the Stakeholder Community in defining the product/solution but is primarily responsible for outcomes.

An empowered PO is able to do most of the heavy lifting associated with managing a solution’s Stakeholder Community. While software developers are free to work individually with any stakeholder, the PO ultimately decides what features make it into the product and the nature of those features. This arrangement provides developers a dedicated Scrum Team liaison to the Stakeholder Community who is well-informed and respected by the community.

Mentor Developers in Customer Engagement Skills

Customer engagement and requirements elicitation are learned skills. While some people have more of a knack for them, everyone can benefit from training and mentoring in those skills. Such training would include topics such as:

Organizing and Leading Meetings
Presentation Skills
Active Listening
Holding User Experience (UX) Workshops

Investment in these “soft skills” pays off in better communications between development teams and stakeholders, better requirements definition, less rework, and greater customer and user acceptance of delivered solutions.

Align Work to Team Capacity

Development teams have more time to interact with stakeholders if the amount of work they commit to is in line with their capacity to complete it. As explained in the first article of this series, software development teams that do not have control over the amount of work they commit to are not Agile teams.

Invest in Technical Training

Another way to open up time for developers to collaborate with stakeholders is by providing access to technical training and setting aside time to take advantage of it “on the clock”. Done well, this investment can bring about better prepared and more knowledgeable technical contributors, better solution implementations, cost savings due to decreased rework, and a boost in employee morale and goodwill towards employers.

Pass Along Knowledge through Technical Consultants

Often, development teams do not possess specific hard-to-find skillsets or may not have the bandwidth to learn them. One way to meet short-term development needs while passing along knowledge is by partnering developers with technical consultants. This learning by doing model accelerates learning, thus providing development teams more time to interact with stakeholders.

Conclusion

Developers and stakeholders benefit greatly from closer collaboration in defining solutions. The keys to improving that collaboration are:

Developing well-managed Stakeholder Communities through a Product Ownership or Management model
Aligning the amount of work done by development teams to their capacity
Leveraging training and mentorship to open up more time for collaboration with stakeholders

This series covered a lot of ground concerning the frustrations with Agile software development shared by many developers. I too have participated in “Agile” implementations that failed to be Agile. What those failed implementations had in common was an inability or unwillingness to let go of traditional project management biases rooted in Taylorism. The challenge is not in training people on the mechanics of individual Agile approaches. It is in chipping away the Industrial Age organizational mentality that stifles progress in today’s cyber-driven world.

The Agile Manifesto is a clarion call to improve how we work by acknowledging the skills, knowledge, talents, and professionalism of today’s knowledge workers. Agile is an ever-evolving domain filled with practitioners dedicated to improving it. It is that ethos that keeps me optimistic about leveraging Agile to improve not just software development, but world of work in general.

Let us know what you think!!!