Managing Software Project Scope: WBS vs. Product Backlog

Both decompose project scope in very different ways.

Many organizations steeped in traditional project management culture struggle to adopt Agile’s product management approach. One of the biggest cultural stumbling blocks is the difference between how traditional project management and Agile treat project scope. This is best exemplified by the differences between the Work Breakdown Structure (WBS) and Scrum’s concept of the product backlog. This article describes both tools, compares and contrasts them, and makes the case for not conflating these two methods of accounting for project scope.

What is a WBS?

The PMBOK® Guide—Third Edition defines a WBS as:

“A deliverable-oriented hierarchical decomposition of the work to be executed by the project team to accomplish the project objectives and create the required deliverables. It organizes and defines the total scope of the project. Each descending level represents an increasingly detailed definition of the project work. The WBS is decomposed into work packages. The deliverable orientation of the hierarchy includes both internal and external deliverables.”
PMBOK® Guide—Third Edition

A WBS is a planning tool that helps identify, document, and present the total scope of a project. Scope is one of the three project management constraints colloquially known as “The Iron Triangle”.

The Iron Triangle represents how scope, schedule, and level of effort/resources act as mutually-dependent project constraints. Traditional project management attempts to predict most, if not the entire, project scope prior to project execution. Later, planners estimate the level of effort required to complete planned tasks and deliverables and build a project schedule based on those estimates. The resulting three values feed calculation of project cost.

WBS Structure

A WBS is most often depicted as a hierarchical tree. Every node in the tree is called an element.

Depending on the type of WBS hierarchy created, project scope may be organized in terms of:

Project/program activities necessary to deliver deliverables (Verb-Oriented)
Product components that make up a deliverable (Noun-Oriented)
Time-bound phases of the project plan
A mix of the previous three types

The root of the tree (WBS level 1) represents the entire scope of the project or product deliverable. From there, the main branches/elements of the tree (WBS Level 2) are structured in accordance with one of the four types of WBS hierarchy types listed above.

WBS Decomposition

The tree expands as higher-level elements are decomposed into smaller constituent elements or components. Each element represents 100% of planned work and deliverables for that element. Child elements collectively represent 100% of the scope represented by their parent. Decomposing across elements just enough to facilitate project scope definition, without over-specifying the project or solution, while accounting for the entirety of project scope, is an art honed by experience.

Mapping to schedule and Resources

At the lowest level of each branch of the WBS hierarchy are work packages. Work packages map project scope (deliverables and activities) to project schedule. A work package includes schedule activities and milestones required to complete the associated deliverable or work element/component. In turn, control accounts map work packages to project budget (cost), facilitate the roll up of activities and milestones into an integrated schedule, and allow planners to assign specific resources (e.g., people) to specific elements of the WBS hierarchy.

Another artifact that ties the scope represented by the WBS to project schedule and resources is the WBS dictionary. The PMBOK® Guide—Third Edition defines a WBS dictionary as:

A document that describes each component in the WBS. For each WBS component, the WBS dictionary includes a brief description of the scope or statement of work, defined deliverables, a list of associated activities, and a list of milestones. Other information may include responsible organization, start and end dates, resources required, an estimate of cost, charge number, contract information, quality requirements, and technical references to facilitate performance of the work.
PMBOK® Guide—Third Edition

The WBS dictionary for figure 3 above may look something like this:

Types of WBS Hierarchies

Let’s delve a little deeper into the different types of WBS hierarchies.

Project Scope WBS

A project scope WBS decomposes the project work or activities necessary to produce deliverables or products. The first word of each WBS element name is a verb, such as, design, develop, test, etc. In figure 4 above, the project is organized by Software Development Lifecycle (SDLC) phases (Define, Design, Develop, and Integrate & Test) and includes the project management work necessary to orchestrate the work.

Product Scope WBS

A product scope WBS defines project work in terms of the components (physical or functional) that make up a deliverable or product. In this case, the first word of each WBS element name is a noun, such as, Module A, Subsystem B, Aircraft Engine, Fuselage, etc. Since each element represents a part of the overall product, this WBS type is sometimes called a “Product Breakdown Structure (PBS)”.

Time-Phased WBS

Large-scale projects can take years to complete. In such cases, it is reasonable and advisable to break up project scope and schedule into discrete phases. A “time-phased” WBS breaks up the project into major phases instead of activities. Rather than planning the entire project upfront, planners follow a “rolling wave” planning approach in which they plan the upcoming phase in detail while postponing detailed planning for later phases. Rolling wave planning affords the opportunity to improve current plans based on lessons learned in previous phases. It also provides a measure of flexibility to deal with changing project realities.

Hybrid WBS

Elements from the three other WBS types can be combined to create a “hybrid” WBS. Defense materiel project planners frequently use this type of WBS. Such projects tend to be long-lived, so time-phased branches may be included. Product-based branches scope complex products/deliverables. For activities that map to specific activity phases (e.g., SDLC phases), project scope elements are appropriate. A hybrid WBS may include project or program activities outside the scope of creating physical or functional product components or managing their creation. Examples of such activities include training, operations & maintenance, and facilities management.

WBS Type Comparison Chart

I summarize the WBS hierarchy types in the table below:

Agile Value Delivery

Before discussing product backlogs, I need to explain how Agile software teams deliver useful capabilities.

Agile software teams deliver iteratively and incrementally. They release valuable increments of software functionality over the entire development effort. These increments are “production ready” or “potentially releasable” software capabilities that customers and end users can start using right away. While not everything released by Agile teams must be user facing, they avoid releasing individual system components or portions of components that do not deliver real business or mission value on their own.

Solutions built by Agile teams grow and evolve over time. Each and every increment delivered is designed to work with all other increments, past and future.

Thus, Agile releases are primarily feature-based, not component-based. Rather than delivering system components, Agile software teams strive to deliver features. Features are cohesive and complete slices of system functionality capable of performing useful work from the perspective of customers and end users.

What is a Product Backlog?

A product backlog is an ordered queue of deliverables. The term product backlog is most associated with the Agile framework called Scrum. According to the Scrum Guide, the product backlog:

…is an emergent, ordered list of what is needed to improve the product. It is the single source of work undertaken by the Scrum Team.
https://www.scrumguides.org/scrum-guide.html

In this definition, the “product” is whatever the Scrum Team develops and delivers (application, system, solution, documentation, reports, etc.). A product backlog is used to track product deliverables.

As depicted in figure 9 below, Scrum teams take into account multiple considerations when ordering product backlog items.

Learn more about ordering product backlogs.

Product backlog items represent features. Scrum teams decompose bigger features into smaller ones to better manage their development.

The highest-ordered items are most ready for implementation. The Scrum Team refined them well enough to implement them. Lower-ordered items are increasingly less ready for implementation the further down the product backlog they rank.

Comparing & Contrasting WBS vs. Product Backlog

Having level set what a WBS and product backlog are, I now discuss the similarities and differences between them.

Similarities

Of the four WBS types discussed in this article, the PBS is most like a product backlog. Both are product scope-focused and decompose larger deliverables into smaller ones.

Neither a WBS or product backlog depict dependencies between elements or items. The main artifact used to show dependencies between work activities and deliverable milestones in traditional project management is the project schedule. User story maps help organize Scrum product backlog items (epics and user stories) based on workflow and interface dependencies.

Differences

The main difference between WBS hierarchies and product backlogs is that the WBS is a tool for linking together project scope with the other two Iron Triangle constraints: schedule and level of effort/resources. A product backlog is strictly focused on tracking and ordering product scope.

Modern Agile productivity tools like JIRA allow development teams to document the work they perform, how long it takes them to do it, etc. However, that information is collected after developers pull the highest priority items from the product backlog and begin work. The product backlog is neither a repository for nor a source of work status information.

Another difference is that tasks and activities count as part of the scope represented by a WBS (e.g., project management tasks, meetings, reporting). While Agile software teams also deal with management and project overhead, those activities are not included in the product backlog. From an Agile perspective, those activities are not part of the product. They are necessary, yet ancillary activities to the work of developing, testing, and releasing software.

Finally, product-oriented WBS elements represent components while product backlog items represent features.

Disadvantages of Decomposing Product Scope into Components

Decomposing product scope into components has three major disadvantages:

Postponing integration and testing
Difficulty measuring progress
Formation of component team silos

Postponing Integration and Testing

In traditional software projects, there is often nothing valuable to test and evaluate, from a business or mission perspective, until most or all components are integrated into a cohesive solution. In traditional software projects, system integration is done infrequently, often occurring only once towards the end of development. Integrating components towards the end of a project often affords insufficient time to address problems.

Difficulty Measuring Progress

Traditional project management focuses on tracking tasks and activities and their level of completion (e.g., percentage complete). The rationale being that if everyone executes all necessary tasks and activities, on time, and in the right order, the project will proceed as planned. Therefore, project management’s job is to identify everything that needs to be done before the start of the project and enforce adherence to the plan.

The problem with applying this approach to software projects is that non-trivial software systems are significantly more complex than physical systems of comparable cost. This is because software systems can be developed and modified in infinite ways. As software systems evolve to serve growing numbers of requirements from growing user bases, their complexity increases, sometimes at a seemingly exponential rate. No amount of predictive planning can overcome this natural slide towards entropy.

There is also the issue of quantifying how much actual progress has been made. What is the difference between a component being 75% complete vs. 95% complete? If we have to wait until it is 100% complete and integrated into the larger solution before we can test and validate it, there is none.

Agile planning is adaptive, with short project planning horizons. Software development teams achieve fast and frequent releases by keeping releases small. Planning and releasing this way allows for more opportunities to test and validate deliverables and to fold lessons learned into ongoing planning. Best of all, the measure of project progress is delivery of valuable capability to customers and end users rather than arbitrary estimates of completion rates.

Formation of Component Team Silos

Each WBS element maps to a set or resources (one or more teams) that do the work. Establishing development teams for specific solution components often stifles cross-team collaboration and creates productivity bottlenecks.

For example, project management forms a dedicated database team to deliver a system database identified as a WBS component. The result is that the team’s allocated hours and skillset are segregated around delivering a single component. Often, the inefficiencies of managing dependencies between component teams outweigh the efficiencies gained from component team specialization.

Forming teams around system features is a staple of Agile software development. While component teams are sometimes necessary in Agile software development efforts, the default bias is towards forming feature teams. Under feature teams, system components form and evolve incrementally and iteratively, one feature at a time. Value delivery no longer depends on delivering the entire system. Instead, value delivery occurs one release at a time throughout development.

Agile Breaks the Iron Triangle

To achieve project agility, Agile breaks the Iron Triangle by making project scope variable while fixing resources and schedule. This involves shifting our thinking about managing projects:

The most optimal solutions reside in the nexus of value, cost-effectiveness, and timeliness:

Agile projects control scope and ensure teams satisfy requirements through ongoing planning and collaboration between development teams, project sponsors, stakeholders, customers, and end users. Sponsors control budgets and approve project roadmaps; stakeholders, customers, and end users work with development teams to define solutions; and development teams plan and build functionality within those constraints. The advantages of controlling projects primarily through managing project scope are:

Not sacrificing quality in favor of adhering to all three constraints of the Iron Triangle.
No budget or schedule surprises (since budgets and schedules are essentially fixed)
Basing project decisions on the actual business or mission value delivered and team performance rather than alignment to detailed (and possibly obsolete) plans

Conclusion

At the heart of the differences between traditional project management and Agile is the fundamental difference between project management and product management. Project management focuses on managing tasks and activities to ensure they complete on schedule and on budget. Product management focuses primarily on managing the product.

Many project and program managers are unaware of this difference. Implementation of Agile practices corrupted by traditional project management biases often leads to failed projects. Attempting to structure product backlogs like WBS hierarchies is one manifestation of this misalignment between traditional project management and Agile mindsets.

Let us know what you think!!!

Handling Non-Functional Requirements in Agile Projects

Yes. You have to address non-functional requirements in Agile software projects. Here’s how.

Table of Contents

In any software project important enough for sponsors to spend significant amounts of money on, the debate over how to handle non-functional requirements (NFRs) sooner or later comes to a head. Many people who sponsor, manage, and execute development of software systems simply do not understand NFRs. This results in increased risks to software projects, Agile or not. This article explains NFRs in the context of requirements management and how to address them in Agile projects.

Requirements Management

Requirements management is the process of documenting, analyzing, tracking, prioritizing, and controlling changes to system requirements. Development teams accomplish this work in collaboration with project sponsors, stakeholders, customers and end users (i.e., the Stakeholder Community). Since software systems change and evolve throughout their useful life, requirements management is a continuous activity, not just something that occurs during development projects.

What is a Requirement?

A requirement is a singular, documented, physical or functional need that a solution (functional or procedural) aims to satisfy. Requirements specify the things a solution must do and how well it must do them. The former are functional requirements. The latter are non-functional requirements.

NFRs Explained

Functional requirements are intuitive for most people. People feel comfortable voicing what they need a solution to do for them. The tricky part is characterizing how well a solution must do those things. This points to how the solution performs or behaves while performing useful work for customers and end users.

For example, a bottle capping machine at a soda plant designed to cap ten bottles per minute would be woefully inadequate if the plant produces 50,000 bottles of soda per day. Assuming the plant operates 24 hours per day, the machine would need to cap bottles at a rate of at least 35 bottles per minute just to keep up. That rate may still be too slow in relation to other plant operations and distribution schedules. No matter how well that capping machine caps bottles (functional requirement) it must do so at an adequate throughput rate (number of bottles per minute).

A software solution example might entail the number of concurrent users an application must support. Perhaps the application experiences large swings in the number of users depending on circumstances. For example, a retail sales website will likely see surges in users around Black Friday. No matter how well that application handles normal traffic, it must be designed to handle such surges seamlessly.

This is when we get into discussions about how a solution behaves. In software engineering, we often characterize solution behavior in terms of “-ilities”. I list examples in the table below:

Click to Expand — Table 1: Non-Functional Requirement “-ilities”

These “-ilities” apply to the entire system or solution, not just to individual system features or components. However, each and every part of the system contributes to, or takes away from, its overall behavior. One underperforming component could degrade the behavior of an entire system. Thus, it is crucial to take NFRs into consideration when designing and implementing system features and components. The more frequently development teams integrate them into the larger system and put them through integration and regression testing, the better.

NFRs constrain functional requirements. In other words, the capabilities designed and developed to address functional requirements must also support or contribute to meeting NFRs at the system level.

Common sources of NFRs include official standards; laws, regulations, and policies (LRPs); and technical specifications (e.g., operating systems, required middleware, hardware specifications).

Handling NFRs Agilely

There is no fundamental difference between how Agile and traditional software development projects handle NFRs. In either context, we handle NFRs by:

Writing well-written NFRs
Ensuring functional requirements address relevant performance/behavioral attributes.

Well-Written NFRs

A well-written NFR describes a system-level behavior or performance expectation both qualitatively and quantitively. There is no universally accepted standard way to write NFRs. However, the following guidelines should help:

Start with the outcome in mind by providing context about the requirement.

For example, why is scalability a desirable goal for our system? The answer is tied to the business or mission processes the system supports. What does scalability look like from a business or mission perspective?

Quantitatively, we must define what the right degree of scalability is. For example:
- How many concurrent users per day, hours, minute, second?
- What data volumes can we expect?
- How much fluctuation in users and data volumes can we expect? When? How often?

The more specific we are about our needs, the better we’ll define functional requirements that realize NFRs. Focus on identifying needs rather than on defining solution approaches or designs.

Addressing NFR Performance & Behavioral Attributes

Functional requirements realize NFRs. Agile functional requirements adhere to the INVEST criteria:

Agile functional requirements are customer and end-user centric. They are written from the perspective of what customers and end users need rather than from the point of view of what the system will provide.

To make this discussion more concrete, I will use the User Story requirements format. A User Story includes:

A description of what the user needs to accomplish an outcome
Acceptance criteria that explain the specific behavior and performance characteristics the resulting functionality must exhibit to comply with requirements

User Story Description

The de facto standard format for User Stories is the following:

As <a user, role, or automated client> I want <some capability> so that <desired outcome>

This simple and concise statement tells us:

Which specific user, role, or automated client needs the capability or feature: User Stories serve as reminders for future conversations. They are not meant to document requirements in detail. User Stories tell us the types of users we need to talk to or learn more about.
The business or mission capability the user needs: User Stories are written from the perspective of the business or mission capability a user needs to accomplish a desired outcome. The resulting software functionality will enable the accomplishment of that business or mission outcome.
The desired business or mission outcome the capability or feature will enable: User Stories answer what tangible business or mission outcome will be achieved as a result of using the implemented feature. The outcome is often the execution of a business or mission process or task.

User Story Acceptance Criteria

The second part of a well-written User Story are acceptance criteria. This is where the NFR attributes defined earlier constrain the design and implementation of the resulting functionality. Acceptance criteria are the results or outcomes the functionality must achieve when executed. We demonstrate adherence to acceptance criteria through testing. So, in essence, acceptance criteria are test criteria for the User Story.

Well-written acceptance criteria are testable. That means that the result is not left to interpretation. The feature either delivered expected results or did not. The behavior and performance the feature demonstrates should contribute to (or not degrade) the overall system-level behavior and performance documented by NFRs.

Here is an example:

NFR: The system shall implement authentication and authorization functionality as per corporate security policy XYZ.

Functional Requirement:

As a system administrator (SA), I need to restrict user access to files so that authenticated users only have access to files for which they have access permissions.

Acceptance Criteria:

The SA can grant specific authorized users access to specific libraries or documents.
The SA can revoke access to specific libraries or documents from specific authorized users.
All file access permissions for specific users are automatically revoked if the SA revokes their authorization to access the system.

Conclusion

It is quite common for people engaged in sponsoring, managing, and developing software systems to misunderstand NFRs and struggle in applying them to Agile requirements management. User Stories make mapping of system-level NFRs to functional requirements easier. Acceptance criteria link functional requirements to tests that demonstrate satisfaction of NFRs. Coupled with frequent integrations and regression testing, this approach to NFRs ensures the delivery of expected system behavior to customers and end users.

Let us know what you think!!!

Why So Many Developers are Fed Up with Agile (Pt. 4)

Changing Expectations for Developers

Table of Contents

Developers are moving away from functioning as “order takers”, focused almost exclusively on coding, to collaborating with solution sponsors, stakeholders, customers, and end users (i.e., Stakeholder Community) to define, verify, and validate what they develop. Many technical contributors find this level of direct collaboration difficult to manage alongside their growing list of technical responsibilities.

This article is the last of a four-part series about why so many developers are fed up with Agile and why so many organizations fail to achieve Agile’s much-touted benefits. The common thread across the series is that what often passes for Agile is anything but Agile.

The other topics covered in this series are:

The Never-Ending Agile Grind – Developers often feel overwhelmed by the constant pressure to produce under tight deadlines.
Greater Visibility & Oversight with Less Control – The “radical transparency” of Agile approaches, such as Scrum and Kanban, is often exploited by management as an opportunity to micromanage development teams.
Top-Down Driven Agile – Technical contributors often have little say in how their organizations or sponsors implement Agile, leading to the imposition of unnecessary, ineffective, and, often, counterproductive practices on development teams.

Difficulties Collaborating with Stakeholders

One of the defining characteristics of Agile is the fostering of close and ongoing collaboration between software development teams and their Stakeholder Communities. As the fourth Principle of Agile Software Development states:

Business people and developers must work together daily throughout the project

http://agilemanifesto.org/principles.html

Needs, expectations, and feedback from the Stakeholder Community guide what Agile teams build. Delivering features and capabilities iteratively and incrementally affords more opportunities to inspect and adapt deliverables and team processes accordingly.

The ultimate validation of a solution’s value is use by real customers and end users. Prior to release, Agile teams test and validate software increments in collaboration with their respective Stakeholder Communities. After release, development and/or operations teams monitor and evaluate how evolving solutions are used and perform. Lessons learned inform decisions about upcoming development.

Many Agile projects fail because this vital feedback loop breaks down or is never established. Achieving effective collaboration between groups of people who hold different ideas about what is needed or important is difficult. The trust and respect upon which collaboration depends take time and effort to build and are easily undermined. Avoiding or mitigating misunderstandings, disagreements, and conflicts starts with people understanding each other’s perspectives.

The Developers’ Perspective

Below are some of the most common issues developers encounter when dealing with Stakeholder Community members.

There is also the issue of whether development teams have access to the right stakeholders (or any stakeholders at all). Too often, those who provide requirements do not have intimate understanding of relevant business or mission processes or of customer and end-user needs. This is especially true in organizations with traditional project management cultures where project or program managers, architects, and senior engineers are expected to “know” customer needs yet rarely interact with them.

The Stakeholders’ Perspective

Collaboration is a two-way street. Agilists approach requirements from a customer-centric perspective. Unfortunately, too many developers think about solutions from the perspective of what they want to build. They define requirements as development tasks to accomplish or as system functionality to develop. In the eyes of the Stakeholder Community, these attitudes come across in the following ways:

Expanding Technical Responsibilities

Along with increased expectations for Agile development teams to interact with stakeholders, the scope of work done by software developers continues to grow.

Back in the 80s and 90s, a software developer was considered proficient if he or she specialized in one operating system and one programming language. In the 90s and the aughts, many, if not most, developers specialized as front-end or back-end developers. There were much fewer tools to master and most development was custom rather than based on third-party components and frameworks. Also, the work associated with the full lifecycle management of applications was highly specialized and assigned to different people.

The advent of the Cloud and DevOps changed all of that. Now developers are expected to be “full-stack developers” capable of developing both the front and backend components of complete releasable features. Virtual computing and the Cloud broke down operating system-based development siloes. Open-source development wildly increased the number of tools and frameworks available. Great efficiencies brought about by DevOps came at the price of developers learning DevOps tools, developing automation scripts, and managing DevOps pipelines.

A proficient developer today is a master of multiple programing languages, highly conversant in more than one operating system, comfortable using a plethora of tools and frameworks, and regularly contributes to testing and DevOps automation. Software developers now perform a lot of work that used to be set aside for specialists such as system administrators, configuration managers, build engineers, testers, etc. While a lot of that work is now automated or abstracted away from developers, the scope of what is considered development work is much larger than before.

Many software developers feel overwhelmed by expectations to keep up with the ever-increasing pace of technological change, consistently deliver value, and collaborate with stakeholders. Most developers entered the field for the opportunity to develop software for a living and concentrate on becoming better technologists. They become highly proficient at building software but lack the business or mission insight necessary to better align solutions with business or mission needs.

Proposed Solutions

Development teams do not need to choose among better collaboration with stakeholders, maintaining technical excellence, and delivering value cost effectively and in time to be valuable. To achieve that balance, we recommend the following actions.

Promote Strong Product Ownership

While not all Agile teams are Scrum teams, Scrum’s concept of Product Ownership is useful for managing Stakeholder Communities and can be tailored to work with non-Scrum Agile teams. In Scrum, the “product” is synonymous with whatever technical capability a Scrum Team develops, be it a system, solution, application, etc.

A Scrum Product Owner (PO) is responsible for fusing the various needs, wants, and perspectives of product and project stakeholders into one complete, coherent, and cohesive vision. The Scrum Development Team looks to the PO as the primary (but not the only) source of business or mission knowledge and understanding. The PO enlists the help of the development team and the Stakeholder Community in defining the product/solution but is primarily responsible for outcomes.

An empowered PO is able to do most of the heavy lifting associated with managing a solution’s Stakeholder Community. While software developers are free to work individually with any stakeholder, the PO ultimately decides what features make it into the product and the nature of those features. This arrangement provides developers a dedicated Scrum Team liaison to the Stakeholder Community who is well-informed and respected by the community.

Mentor Developers in Customer Engagement Skills

Customer engagement and requirements elicitation are learned skills. While some people have more of a knack for them, everyone can benefit from training and mentoring in those skills. Such training would include topics such as:

Organizing and Leading Meetings
Presentation Skills
Active Listening
Holding User Experience (UX) Workshops

Investment in these “soft skills” pays off in better communications between development teams and stakeholders, better requirements definition, less rework, and greater customer and user acceptance of delivered solutions.

Align Work to Team Capacity

Development teams have more time to interact with stakeholders if the amount of work they commit to is in line with their capacity to complete it. As explained in the first article of this series, software development teams that do not have control over the amount of work they commit to are not Agile teams.

Invest in Technical Training

Another way to open up time for developers to collaborate with stakeholders is by providing access to technical training and setting aside time to take advantage of it “on the clock”. Done well, this investment can bring about better prepared and more knowledgeable technical contributors, better solution implementations, cost savings due to decreased rework, and a boost in employee morale and goodwill towards employers.

Pass Along Knowledge through Technical Consultants

Often, development teams do not possess specific hard-to-find skillsets or may not have the bandwidth to learn them. One way to meet short-term development needs while passing along knowledge is by partnering developers with technical consultants. This learning by doing model accelerates learning, thus providing development teams more time to interact with stakeholders.

Conclusion

Developers and stakeholders benefit greatly from closer collaboration in defining solutions. The keys to improving that collaboration are:

Developing well-managed Stakeholder Communities through a Product Ownership or Management model
Aligning the amount of work done by development teams to their capacity
Leveraging training and mentorship to open up more time for collaboration with stakeholders

This series covered a lot of ground concerning the frustrations with Agile software development shared by many developers. I too have participated in “Agile” implementations that failed to be Agile. What those failed implementations had in common was an inability or unwillingness to let go of traditional project management biases rooted in Taylorism. The challenge is not in training people on the mechanics of individual Agile approaches. It is in chipping away the Industrial Age organizational mentality that stifles progress in today’s cyber-driven world.

The Agile Manifesto is a clarion call to improve how we work by acknowledging the skills, knowledge, talents, and professionalism of today’s knowledge workers. Agile is an ever-evolving domain filled with practitioners dedicated to improving it. It is that ethos that keeps me optimistic about leveraging Agile to improve not just software development, but world of work in general.

Let us know what you think!!!

Why So Many Developers are Fed Up with Agile (Pt. 3)

Top-Down Driven Agile

Table of Contents

No one likes being told how to do their job, especially by people less knowledgeable about the work than themselves. One of the defining characteristics of knowledge workers is that they know more about the work they do than their supervisors.

Knowledge workers amass a wealth of education, training, and experience in their specialties. Their success depends on how well they apply those intellectual resources to their work. Knowledge workers do not merely produce tangible or quantifiable products and services. They help employers or clients achieve business or mission outcomes. Thus, they usually enjoy much greater latitude in determining how they work than workers in traditional service or manufacturing sectors. Software development teams are made up of knowledge workers.

Unfortunately, traditional project management culture is still highly influenced by Taylorist notions regarding work organization and worker management. Too many organizations reflexively assume that management should determine how employees work. This predisposition runs directly counter to the Agile principle of:

Build[ing] projects around motivated individuals, [giving] them the environment and support they need, and trust[ing] them to get the job done.

http://agilemanifesto.org/principles.html

Organizations possessing traditional project management cultures often adopt Agile in ways that largely ignore the perspectives and input of those most affected by the change: software development teams.

This article is the third of a four-part series about why so many developers are fed up with Agile and why so many organizations fail to achieve Agile’s much-touted benefits. The common thread across the series is that what often passes for Agile is anything but Agile.

The other topics covered in the series are:

The Never-Ending Agile Grind – Developers often feel overwhelmed by the constant pressure to produce under tight deadlines.
Greater Visibility & Oversight with Less Control – The “radical transparency” of Agile approaches, such as Scrum and Kanban, is often exploited by management as an opportunity to micromanage development teams.
Changing Expectations for Developers – Developers are moving away from functioning as “order takers”, focused almost exclusively on coding, to collaborating with product owners/managers, sponsors and stakeholders, and customers and end users to define, verify, and validate what they develop. Many technical contributors find this level of direct collaboration difficult to manage alongside their growing list of technical responsibilities.

A Reaction Against Traditional Project Management

Agile software development started as a reaction against the decades-old, heavyweight, project management approach to software development commonly referred to as Waterfall. It began as a collection of “lightweight” software development approaches implemented by a few like-minded, software industry thought leaders.

Realizing their approaches shared a common philosophy, they gathered at a ski resort in Utah in February 2001. There they drafted “The Agile Manifesto for Software Development”. Adoption of the Manifesto and its supporting 12 Principles spread quickly across the software industry. That formulation of what constitutes software development agility helped popularize Agile and shape the ongoing evolution and refinement of Agile approaches.

Agile was a bottom-up movement started by software developers. As stated in the first sentence of the Agile Manifesto:

We are uncovering better ways of developing software by doing it and helping others do it.

http://agilemanifesto.org/

Viewed through the lens of management practice at the time, what these early Agilists advocated was disruptive; almost subversive. They carved out space within which they could work without the distractions or frustrations of managerial micromanagement and organizational politics. They broke down organizational silos and included customers and end users in defining, verifying, and validating software solutions. While Agile helped shift software and IT management culture towards becoming more cooperative and participatory, progress has been spotty.

Few organizations fundamentally changed their traditional management cultures to align with Agile. While many digital native companies established during the first two decades of the 21^st century internalized Agile and Lean principles from the start, many (if not most) organizations did not. Traditional project management biases still color how most organizations view Agile. They perceive Agile as a new way of developing software rather than a new way of delivering value through software. For those organizations, Agile is something developers do while the rest of the organization functions as before.

This fundamental mismatch between Agile’s bottom-up culture and top-down traditional project management culture is the primary reason so many Agile implementations go wrong. It is also why those with the most at stake in adopting Agile often have the least say in its implementation.

Aligning Organizations to Agile Delivery

Many organizational leaders are unaware of the need to align their organizations with Agile’s iterative and incremental mode of value delivery. In Agile, value is whatever solution sponsors, stakeholders, customers, and end users (i.e., the Stakeholder Community) care about or makes them happy. This definition equates value with real business or mission value rather than the cost incurred in developing functionality. Agile teams help organizations achieve predefined and measurable outcomes by delivering valuable software capabilities.

Agile teams determine what to build through negotiation and collaboration. The Stakeholder Community is actively engaged with development teams in defining requirements and verifying and validating solution increments throughout development. This way of working affords plenty of opportunities to inspect and adapt emerging solutions as they grow and evolve. Frequent releases provide the most important validation of all: customer and end user acceptance of solutions and achievement of desired outcomes. What teams learn along the way informs future development.

Organizations with strong traditional project management cultures are not organized to participate in this form of value delivery and empiricism. They are organized to outsource the definition and development of software solutions to development teams after a period of requirements definition. They then wait until the solution is finished to validate its value.

Outsourcing Agile Adoption

When organizational leadership sees Agile as a process for developers to follow, instead of a culture change for the entire organization, they are more likely to “outsource” Agile adoption to Agile trainers, coaches, and consultants.

Many organizations spend large sums training developers on the mechanics of an Agile approach but fail to internalize the values and principles behind the approach. The result are development teams that perform the practices of the chosen Agile approach without understanding how those practices foster agility. Meanwhile, management continues to operate under Waterfall assumptions and expectations. Organizations end up with crops of newly Agile-certified employees but no plans to apply what they learned towards achieving greater agility.

Many developers feel they do not get enough value from Agile trainers and coaches. Some trainers and coaches lack enough understanding of the day-to-day work of software developers or the organizational environment to be effective. Others are Ill-equipped to help organizations move beyond executing team-level Agile practices and establish Agile-based organizational cultures.

Another manifestation of organizational leadership outsourcing Agile adoption is hiring consultants to develop “Agile strategies.” Rather than leveraging consultants to help them develop Agile strategy, organizational leadership pays consultants to develop their Agile strategy with little organizational input or understanding of the organizational culture. When this happens, the resulting strategies tend to be long on theory and short on actionable objectives for the organization to pursue.

“Hybrid Agile” (Scrumfall, Agilefall)

Many organizations struggling to transform from traditional project management to Agile decide to go for a “Hybrid Agile” approach. “Hybrid Agile” is not a recognized Agile software development approach. It is an attempt to gain the benefits of Agile while avoiding internalizing and applying Agile values and principles. Rather than reevaluating existing processes to achieve greater agility, these organizations contort Agile approaches to fit within existing organizational culture. This leads to a tailoring of Agile misinformed by traditional project management biases.

Development teams working in “Hybrid Agile” projects experience the worst of both Waterfall and Agile. Teams are expected to:

“Plan” their work while being denied control over the amount of work they commit to
“Self-organize” while having no say over team structure, tools, and processes
Provide “estimates” that are treated as deadlines
Document what they work in “Agile” productivity tools to keep management informed
Develop what they are told to according to an “approved” design while expected to accommodate customer and end-user change requests

Proposed Solution

Developing valuable software solutions and products takes more than writing code. Regardless of the software development approach followed, all projects are accountable to their target Stakeholder Communities and must provide business or mission value, cost effectively, and in time to be valuable. Management and development teams have to support each other to be successful.

Bottom-up Agile adoption typically fails to foster organizational agility beyond development teams. Top-down Agile adoption tends to lack in-depth understanding of the realities of software development and buy-in from development teams. A “meet-in-the-middle” approach accounts for the strengths and weaknesses of the other two approaches. The meet-in-the-middle approach entails:

Prepare Everyone to Work Agilely

While Agile is inherently bottom-up, organizational transformations required to achieve agility are not. Organizational leadership and management must play an active role in leading the transformation. However, to do so in ways that do not crush agility, they must adopt a “Servant Leadership” model. This starts with a recognition that the primary role of management is to empower those who deliver value to the customer and to create work environments and customer relationships in which value providers can succeed and flourish.

A big piece of leading the change involves training, coaching, and mentoring employees, at all levels and roles, to understand Agile values and principles and how to apply them to their work. This may require different types of training/mentoring or individual study for different roles throughout the organization. For example, coaching and mentoring for organizational leadership and management could be geared towards:

Understanding the differences between traditional project management and Agile
Learning and internalizing the tenets of Agile Servant Leadership
Aligning organizations to Agile value delivery in ways that align with Agile values and principles

I stress the need for coaching and mentoring. Setting people loose after a two-day Agile certification class and telling them to “be Agile” guarantees failure. Most people outside of software development have not worked in Agile environments. Thus, they do not understand how an organization rooted in Agile values and principles would impact how they work. Organizational agility is not about everyone organizing themselves as a Scrum team. Like any new skill, Agile thinking and practice must be honed to gain the most benefit.

Hire Knowledgeable and Experienced Trainers & Coaches

This point is obvious but warrants explanation: hire knowledgeable and experienced trainers and coaches. They need experience in more than just teaching classes and holding Agile certifications. The more experience trainers and coaches possess in developing software solutions or leading their development, the better.

I temper this with the caveat that the best trainers and coaches possess subject matter expertise and instruction, facilitation, and communication skills specific to their role. The most technical people are often not the best fit for teaching and coaching Agile. What are needed are generalists who possess a mix of theoretical knowledge, relevant industry or mission experience, exposure to the work of software development, and strong interpersonal and communication skills.

Hire, Educate, Train, & Mentor Developers for Leadership Roles

Employers in technical fields tend to select out “soft skills” while overemphasizing technical skills. They do this at their peril. Agile depends on the development of self-organizing “high-performance” teams.

Agile teams develop leaders. Hiring well-rounded people, who work well in teams, and developing their technical and leadership skills ensures development teams have the skills they need to self-organize and to build productive relationships with their Stakeholder Communities.

Conclusion

Reconciling Agile’s inherent bottom-up perspective to organizational change with the need for transformative leadership from above is difficult. There are no one-size-fits-all approaches for doing so. Successful implementation of any Agile approach requires deep understanding of Agile, the chosen Agile approach, the organizational environment, the Stakeholder Community, and the day-to-day realities of development teams. It requires discernment and a willingness to make tough decisions.

Software developers are on the front lines of Agile transformations because they create solutions that enable business or mission success. They have a huge stake in the direction Agile adoptions take because they are directly affected by the outcome. Thus, they need and deserve a significant voice in their implementation.

In the next and final article of this series, I’ll discuss difficulties software developers face managing Agile and DevOps-driven changes to their role.

Let us know what you think!!!