My Journey to Unravel the Agile Mindset
Why Agile is more than a set of methodologies—it's a mindset crucial for navigating the complexities of software development. How embracing change and valuing people leads to true agility.
"The tactical result of an engagement forms the base for new strategic decisions because victory or defeat in a battle changes the situation to such a degree that no human acumen is able to see beyond the first battle. In this sense one should understand Napoleon's saying: "I have never had a plan of operations."
Therefore no plan of operations extends with any certainty beyond the first contact with the main hostile force."
- Field Marshal Helmuth von Moltke the Elder
I started professional software development sometime around 2013. It was at a time when Agile methodologies like Scrum were already quite established in teams and organizations. It seemed to me that it had morphed into a default choice in software development. Working like that made sense to me. Instead of doing exhaustive planning up front and shipping software after long development and testing phases, we went for shorter feedback cycles that enabled us to incorporate customer feedback earlier. Being new to software development, I thought this was a good thing (spoiler alert: today, I still think it is).
Nevertheless, my understanding was too superficial to assume that Agile methodologies are a good idea simply based on the fact they were the default choice for most teams I worked in. I wanted to understand the mindset and reasoning behind this way of working. I wanted to dig deeper.
Software Development and Complexity
Before we discuss Agile as a mindset and its methodologies, let's take a step back by looking at a conceptual model from complexity theory: The Cynefin Framework categorizes problems into five buckets: simple, complicated, complex, chaotic, and disorder. Its purpose as a sense-making framework is to contextualize problems and choose appropriate solution strategies.
The first section contains clear problems. In this area, solutions to problems are predictable and repeatable. Thus, we have existing best practices (or checklists) to solve them. Complicated problems are a little more difficult. They require expert knowledge to analyze, but we can follow a clear plan to arrive at a solution.
There are some problems that we can grasp only in hindsight. Here, we face complex problems. Outcomes are unpredictable. We must experiment to find out what works. There are also chaotic problems where we cannot gather enough understanding to conduct experiments in the first place. The space of confusion sits in the middle. That means our problem should reside in one of the other spaces, but determining which one presents challenges, so we rely on familiar solution patterns that may not be the best choice.
Software development lives in the space of complex problems. Why is that? First, we do not know beforehand how our users will use the system's functionality and in what combination they will do so. Second, we do not know what the best approach to solving the problem is in the first place. Also, we work in a fast-changing software development ecosystem. New frameworks and dependency versions appear daily. All the reasons above make it very hard to predict what will happen after we have started developing software systems.
Martin Fowler argues that software development is primarily a design activity, a creative process that is hard to plan. Unlike civil engineering's plan-driven approaches, more predictable "construction work" is a small part of software development.
Classical approaches in our discipline heavily relied on up-front plans and up-front design. That neglected the complex nature we are dealing with. It treats software development as a complicated problem without considering the unpredictability of user needs, the context in which the system operates, and the evolution of the software development ecosystem.
Agile Is About Responding to Change
Responding to change is a central pillar. In the face of uncertainty, it makes sense to work in small increments and decrease the amount of up-front planning since the plan would need to change anyway. A better approach is to conduct experiments, fail as fast as possible, and learn from the assumptions we have made. A consequence of this line of thought is that we want to deliver our system early and regularly to get feedback as fast as possible.
But what changes do we need to respond to in the first place? First, requirements change. They change because it is much easier to argue about a system, its features, and the usage workflow when customers can try and experience the system in the first place. Even when our users would perfectly know about the features and how they will use them, we would still need to factor in changing requirements. The reason for that is the evolving world around us. Our company might decide to enter a new market... or new competitors might be emerging. Changes in the business world are highly unpredictable, as every stock trader would tell you.
Second, we have to deal with an evolving software development ecosystem. Operating systems, container orchestrators, cloud environments, frameworks, and libraries are only examples of tools we use in our day-to-day development. All of them evolve independently and sometimes in unexpected directions. Some of them have security issues that need to be fixed urgently. On top of that, new tools emerge, often unpredictably. Could you have predicted the impact of Docker before it was released and widely adopted?
Agile Is About People
As we have discussed above, software development is creative work. As with other creative work, individuals are not replaceable and make unique contributions to the software product. Each individual has a background, experiences, strengths, and weaknesses that can uniquely and positively impact the team and its work results. Therefore, we should treat people as first-class citizens in the software development process instead of handling them like replaceable gears in a factory.
Agile Is a Mindset
After the end of World War II, inhabitants of the Melanesian islands in the South Pacific practiced what anthropologists later coined "Cargo Cult." During the war, Allied forces dropped packaged equipment and food items onto military bases stationed on the islands. This profoundly affected the islanders since the military shared some of the goods with them, significantly increasing their standard of living. After the war, when military bases were removed from the islands, inhabitants were short of supply and the abundance they brought.
The (scientifically disputed) theory is that inhabitants of the island believed technologies (such as airplanes, headsets, and landing towers) were worshipping relics to spiritual entities who dropped equipment and food packages onto the island. What happened was that the islanders began to build their airplanes, headsets, and landing towers out of straw and practiced military formations with wooden weapons as a spiritual act. The spiritual worship did not result in packages being dropped from the air.
The message is that implementing Agile methodologies without adopting Agile principles and mindsets is problematic. Using processes like Scrum or XP only works as far as you adapt the holding beliefs as well. Otherwise, you will land on an island full of buzzwords without embracing Agile as what it truly is: a mindset.
Out of that mindset follow the practices: we write tests and embrace continuous delivery because it makes it easier for us to deliver working software faster and more often. We hold daily stand-ups to determine if team members can help each other clear away impediments because we believe Agile is about people first. We conduct experiments to see what works and what doesn't work. And we want to release software more often because it helps us collaborate with our customers in the first place.
Agile Is No Means to an End
One of the fundamental problems with Agile lies in its adoption. Superficial adoption of the methodologies and thinking it might be a silver bullet is a significant obstacle. We need a deep understanding of the mindset behind the methods—organization-wide, not only technical but also from leadership positions and beyond.
What is essential for Agile teams is to have a safe space to try things out, potentially fail, and have room to learn and adapt. Leadership and management must embrace that. It is fundamental not to see those methodologies as a control mechanism but as a mindset that helps to deliver features faster, embrace people as creatives and make room for experimentation and growth.
Further Reading and References
In “The New Methodology”, Martin Fowler wrote about his thoughts about the Agile movement in 2005.
The "Manifesto for Agile Software Development" states the most central principles of Agile software development.
In ”The Cynefin Framework”, its creator David Snowden describes elements and applications.