23-Mar-2018. Just an FYI that I’m currently re-drafting this article with feedback from a couple of colleagues; you can expect some significant changes at some point – all of which should make the article clearer.
I’d like to share with you an epiphany. I run the risk of it being like a riotously funny party joke, retold later to much less effect: a case of ‘I guess you had to be there’, but that’s seldom stopped me before.
This epiphany is essentially a new way of viewing several things:
- Waterfall vs agile, and agile vs Agile (big A, little A).
- Why people often don’t ‘get’ software development; and probably why your developers and project managers have a fractious relationship, especially whenever estimation and deadlines are hot topics.
In short, the epiphany is based in philosophy. First the recognition that waterfall is inherently rational, whereas agile is inherently empirical. You then realise that as theories within philosophy they both have roots back in antiquity (Plato and so forth) and have essentially been elaborated and compared for significantly longer than waterfall and agile have been around. Therefore, we can contemplate agile and waterfall from a purely philosophical viewpoint (without any pretense of ‘brand’ or additional baggage), and we can also draw on significant body of previous philosophical work.
I also suggest to you that there exists a rationalist-empiricist continuum on which each of us are placed – some people are naturally inclined towards one or the other.
Let’s look at this further and see what we can through this lens.
Short Philosophical Primer
Rationalism and Empiricism are both theories within Epistemology – the branch of philosophy concerned with the theory of knowledge:
- Rationalism is the view that “regards reason as the chief source and test of knowledge”, or in other words, where “the criterion of the truth is not sensory but intellectual and deductive”.
- Empiricism is a theory that states that knowledge comes only or primarily from sensory experience.
Philosophical literature frequently discusses one in contrast to the other. One often also sees references to things being ‘a priori’ or ‘a posteriori’. This isn’t something you need to remember, but it is interesting to note the concept behind these two terms as it’s relevant to our subject and subsequent contemplation:
- A Priori is “what comes before”, as opposed to
- a Posteriori which is “what comes after”.
The question (of whether or not something comes before or after) is significant enough that a priori and a posteriori appear in ‘Elements’ – Euclid’s 300 BCE work which served as the model for precise thinking throughout 15th-18th century Europe.
For those of you less well disposed to latin, you might also come across the terms deductive and inductive:
- A deductive argument is where the conclusion follows from the premise, i.e. via reason. I think, therefore I am.
- In contrast, an inductive argument is based on experience/observation; in such cases the argument may be strongly supported by the conclusion but not necessitated by it.
It’s also interesting to note that empiricism is a key part of the scientific method.
So, key takeaway: Rationalism and Empiricism have been around and thought about for ages, and there’s lots we can draw from that.
Is Waterfall Really Rational and Agile Empirical?
If you already ‘get it’, then you can probably skip this section.
Waterfall, as it is practiced in an Information Technology related project delivery context, is a sequential approach. Typically it consists of phases such as analyse, design, build, test, and so on. Each phase informs the next: analysis informs us what the problem is, based on this we design a solution, based on this design we implement the solution, and so on. Only at the end, when the solution is complete, do we see it working and finally see how well it addresses the original problem.
In other words, the basis for the solutions design and implementation is rational thought – what we can deduce. It’s essentially theoretical.
The planning of waterfall projects is also based on rationalism; at the outset, estimations are made as to how long each phase will take, how much it will cost, who needs to be involved. All of this happens in advance, and is frequently updated. Should reality hit the project, the team is forced to ‘re-baseline’ – delivery schedules, resource estimates and costs are all calculated, based mainly on a rational thought process.
Agile, again as practiced in ‘IT’, is based on an empirical approach where work is done and the results evaluated before proceeding. At the beginning of an agile project effort is spent working out an initial prioritised list of problems to be solved, a small chunk of this is then worked on by the team, the results are evaluated and corrections made – corrections in terms of the base assumptions and the work/output itself. In this way the project proceeds, repeating a consistent cycle of delivery and inspection.
The planning of agile projects is also based on empiricism; teams typically provide relative estimates based on the size of previous work that they have already done, and in this way the estimates are evidence based.
Agile Explained
As I wrote previously, Marty Cagan has a nice simple way of asking people if they are agile:
- How soon can you test?
- Does shipping out a release mean you’re finished?
At face-value this appears fairly straight-forward, but I think we can go further:
- Are you driven by what you have planned or what you can prove through experience?
If you primarily “know” a thing because you have deduced it then you’re not being empirical, and therefore you’re not agile. It’s that simple.
Agile: A vs a; Big A vs Little A
It’s true that in the context of contemporary software development the concept of Agile (as in the big A, as in the Agile Manifesto) is bundled up with additional ideas: individuals over interactions, collaboration, and so on; and whilst these additions clearly add immense value and have made agile more accessible for many, they don’t alter what’s arguably the single most important foundational idea behind agile – empiricism.
Take the agile manifesto and its guiding principles, for example:
- Individuals and interactions over processes and tools.
- Working software over comprehensive documentation.
- Customer collaboration over contract negotiation.
- Business people and developers must work
together daily throughout the project. - Build projects around motivated individuals.
Give them the environment and support they need,
and trust them to get the job done.
Many of these ideas (such as those listed above) are not mutually exclusive with waterfall; any group of people using waterfall can still choose to collaborate, be motivated and trust each other. It’s true (in my experience at least) that we don’t often associate the above with waterfall projects – but that’s inductive, just because it’s often the case does not mean it is bound to always be the case.
Sure, waterfall is based on rationalisation, which is often accompanied by process, requiring tools – but I’d argue that often the use of those processes and tools is often driven by other factors: social norms, people hiding behind process, and so on; it is still possible to put ‘individuals and interactions over processes and tools’ and follow a rational process.
There will no doubt be many for whom these additions are in fact indivisible parts of Agile, and I think that view is fine. My point is that one of the ways to better understand agile/Agile is to mentally remove all of these additions and conceptualise agile/Agile (and waterfall/agile) from a philosophical perspective based on Empiricism and Rationalism.
We can also apply this lens to a more recent phenomenon, the mainstream adoption of Agile as a product to make money off. Don’t confuse agile (and the key part: learning from experience and evidence) with Agile (i.e. specific recipes, brands and elaborate constructs).
State of Mind
Thinking about social norms and human behavior brings us to my second key point: the existence of a rationalist-empiricist continuum, and the premise that everyone is somewhere on this continuum – usually closer to one end than the other.
In my view, many software developers that I have worked with are empiricists – in that they have a greater affinity to an empirical way of thinking rather than a rational one. Conversely, many project managers I have worked with are rationalists – they have a greater affinity to an rational way of thinking.
Are these philosophies, these states of mind, mutually exclusive? I’d say not. For a start this article is itself based on ideas which have emerged through a combination of both. Furthermore, I can think of many occasions when I have personally used each.
Both are clearly useful tools, and like all tools they are most useful when the correct tool is matched to the problem being faced (not the other way around). The problem that rationalists are frequently confronted with in a software project is that the nature of software development is inherently empirical. Or to be a little more precise, the nature of software development when applied to solving a particular real-world problem, is usually inherently empirical.
I say ‘usually’ because I cannot say for certain that all such problems are categorically empirical. If we stretch software out to encompass IT, and therefore hardware, rationalism becomes more relevant.
We must acknowledge that for a lot of people, especially the senior executive who’s paying the bill, the basic question of “when will it be ready and how much will it cost” is not unreasonable. The challenge is how to best answer it – in this regard both Empiricism and Rationalism have their unique challenges.
Peruse Muse Infuse 