Arcturus Mentoring

Abstract:  Under-estimation in technical projects is chronic and widespread.  It arises from bad beliefs and behaviours.  The consequences of under-estimation harm both projects and people.  It also causes under-delivery of promised project features.  It is wiser to choose to properly estimate and fund complex enterprise and development projects.

Yesterday, I had lunch with a dear friend and colleague, for the first time in a decade, who I have known for close to fifty years.  We’ve been good friends, man and boy, but have lived in different countries for several decades.  At one time, I worked for him, in the IT department of a tertiary educational establishment.  Throughout most of our working lives, though, we’ve experienced the world of enterprise Information Technology (IT) and of software development from the perspective of different countries and continents.  What astonished us both was the commonality of experience, in our working lives.  There are global factors that appear to arise, no matter where you practice engineering or technical management.  Perhaps they derive from human nature.

What emerged, early in our catch-up discussion, was that we had both frequently worked on projects whose estimates had been approximately one third of that which, as time would tell, were actually required.  Time and again, we both encountered budgets and timescales which should have been tripled.  With this underestimation problem being so commonplace and seemingly ubiquitous, we began to ponder on why it might be that estimates were so often risibly low and what the knock-on consequences of that perpetual error were.  It was an interesting and far-ranging discussion, drawing on our decades of experience in the industry.

The problem with ludicrously low under-estimates is that the technical and people problems you must solve, during any technical project, must be solved no matter what, or else the project delivers nothing.  There is no way to estimate and budget these problems away, by simply declaring them to be non-existent, on paper.  They happen anyway.  The remedial works you have to do mount up and press on the team and project stakeholders alike.

The issues accumulate, assert themselves and demand answers, which cost time and money.  If there is no time and money available for these unexpected, but wholly predictable and easy to anticipate issues, then other sacrifices are inevitably made.  Usually, the sacrifices involve demanding donated personal time, from those working on the project, or else the deliverables are decimated, resulting in a less than satisfactory outcome for the project’s sponsors and end users.  The opportunity costs, associated with addressing unbudgeted additional tasks, are particularly high.  While you are spending the project budget on fire-fighting or rescue missions, you aren’t using that funding to create needed functionality.

When things go wrong, during a technical project, you have to fix them.  If you don’t fix them, nothing works.  If you do fix them, you need to spend time and money that hasn’t been allocated or budgeted for.  The robbery required to fund the remedies usually takes place at the expense of the people working on the project, or on the delivered functionality.  In the case of a minimum viable solution, taking any functionality away from the deliverables runs the very real risk of causing the solution to be non-viable.

It has to be said: the main cause of under-delivery is under-budgeting, most of the time.  Peter is robbed to pay Paul and then people wonder why Peter is unhappy.

So, why do professional, educated, experienced people perpetually make these hideously silly under-estimates and why are they accepted as immutable gospel?

We both agreed that there is an over-zealous willingness to make “acceptable” promises to project stakeholders and sponsors, which is to blame for chronic under-estimation and hence under-investment in technical solutions.  At the first sign of disapproval, the estimates are reduced.  This is understandable in a hierarchical power-relationship, where the person with the more powerful position (usually the one with the money) can hurt the one making the estimates, if their pronouncements are met with disbelief; feigned or born of ignorance.  The terrible fact is that project sponsors can and do hurt people that steadfastly defend realistic project estimates.  They always want them to be lower.  They are prepared to dispense harm until the estimate they want is the estimate they get.

It is only once that these estimates sound musical to the ears of the sponsors that they become frozen in stone.  The other terrible fact is that, even if the estimator agrees to a musical-sounding, acceptable, low estimate, the practical impossibility of delivering to that estimate will also cause the estimator to be harmed by the project sponsor.  From the point of view of the estimator, you are going to be hurt, whether you low-ball the estimate and fail to deliver on it, or else stick to the realistic estimate, which your experience tells you is the true figure, but displease the sponsor with the discordant sonic nature of your projected numbers.  The duress is merely delayed, in the case of offering a derisory under-estimate.

Project sponsors, for their part, are usually mired in debt and cash flow precarity.  Their imperatives to call for low estimates are born of fear and desperation and a naive belief that software comes from thin air.  They wear their lack of insight into the technical development process as a badge of pride, citing their specialism in money matters as a justification.  It’s a form of wilful ignorance and arrogance and leads to very bad decisions, when it comes to funding technical development.  Their every decision, related to sponsoring technical development projects, is, by necessity, based on hearsay, gut feeling and guesswork.

Younger, more thrusting career ladder climbers see advantage in estimate disputes.  If they are trying to score points and win the approval of superiors, in order to get ahead, they will make extravagant promises of being able to do something for nothing.  If they have credibility, the sponsor will gravitate toward the nirvana-like lies, rather than be supportive of more rigorous, fact-based analyses.

What the young go-getter is rarely aware of are the consequences of their seemingly attractive guarantees.  They’re heedless of the terrible downstream ramifications, which in reality nobody wants.  They are also oblivious to what their promises will require them to do to innocent other people, many of whom are the very creators of the technological solution that everybody depends upon, in order to make good on their impossible pledges.  They’ll end up being hated by everyone and hating themselves.  The psychological damage related to this turn of events cannot be underestimated.

Of course, at the point of the inevitable, predictable outcomes rearing their grotesque heads, not a single person that placed pressure on the estimators to deliver a low estimate will be willing to take responsibility for them.  That particular badge of dishonour is usually placed squarely on the technical team and their commitment and competence called into question.

With this turn of events being so common, it is little wonder that my friend and I have not advised our respective teenage children to pursue careers in technology.  We wouldn’t want them to experience the trauma, as interesting and fulfilling as creating things from scratch can be.

Agreeing to under-estimates sends out a broadcast message, throughout the organisation, that technical work is not valuable or to be valued highly.  It casts the aspersion that the technical developers are lazy, good-for-nothings who perpetually sandbag and pretend the work is much more complex and difficult that it really is.  Of course, in reality, the reverse is true.  Technical developers are dedicated and diligent, working far harder than most of their non-technical colleagues, to deliver work that is far more complex and difficult than even they themselves will publicly admit to their colleagues.

One of the drearily predictable and inevitable consequences of under-estimates is that if you are replacing a number of custom-built, bespoke systems with an enterprise-wide, common platform, individual groups, who once had their perfect system, which did everything they wanted to do, the way they wanted to do it, will lose sometimes vital functionality, as it is sacrificed to meet the needs of the dwindling project budget.  The reason they cannot keep their old, bespoke systems is that the organisation cannot afford the maintenance costs on all of these peculiar, vertical solutions.  Only an enterprise-wide, horizontal suite of software makes economic sense.

However, because of under-estimation, they lose utility, when things they once had fail to materialise in the new system rolled out.  The vociferous backlash, coming from system users who are no longer able to do their jobs in quite the same way as they used to be able to, cannot be underestimated, either.  It can bring a change management programme to its knees.

The fundamental problem is that estimated development budgets seldom seem to include the research or maintenance and solution evolution costs.

Firstly, there is scant acknowledgement and recognition of the need to find out what the development team doesn’t know and to learn it, before the work can be accomplished successfully.  There seems to be a preference to pretend that everyone on the team knows everything they need to know already, at the inception of the project.  This leads to construction of the software with significant and telling gaps in the developers’ knowledge.  Obviously, that can lead to downstream issues.

A reasonably complex enterprise system, once installed and running, is rarely static.  It must be maintained, so that it continues to serve users.  Capacity must be planned, upgrades and patches must be applied at intervals and data must be scrupulously backed up, in the event of disaster recovery being necessary.  The system generally evolves, as the organisation is supports also changes.  Both maintenance and evolution have costs and corners cut, during the initial development of the system, can dramatically increase these costs, which are seldom accurately budgeted for, anyway.  System evolution and extension costs are normally regarded as operating costs, but they are really ongoing development costs.

Technical project funders imagine they can get away with cutting the funding for IT and development projects with few practical effects.  Without being mindful of the fact, often, their mental model is that the software they need will somehow appear, due to diligent programmers working harder, or donating their personal time to the cause, or else that their enterprise system will be miraculously conjured out of the ether, by some form of technical magic which they don’t fully understand.  Such a belief is, of course, utterly nonsensical.  Making anything takes time and effort.  If that effort and time is not paid for, it either won’t happen at all, it will be donated grudgingly and sparingly or else they are expecting people to do real work and create tangible value, for free.

Project stakeholders, who ask developers, already under extreme budgetary pressure, if they can add additional functionality, without additional funding and time, using the incantation that begins with the magic words; “Can you just…?” are particularly frustrating and odious distractions that raise the hackles of any development team.  If anybody asked them to do more work, spending more hours doing it, without any additional pay, funding or deadline extension, they’d be furious, yet they blithely, breezily and happily ask for additional features in the software project, as if they come from a bottomless jar of features, for nothing, in no time.

Once the project funding dwindles and the development team begins to be unfairly seen as obstructive, difficult and incompetent, their work performance is scrutinised minutely.  Their every movement, decision and choice is second-guessed, restricted and commented upon.  These judgements are made, often multiple times, by other people in the organisation, who neither understand what the developers do, nor could do it.  Micromanagement by non-technical managers is thought to be the answer to wringing more software out of less money; the unjustified implication being that the technical team is catastrophically and dysfunctionally disorganised.

Ignorance of how software must be created, for it to have lasting value and low cost of ownership, makes for a very poor project sponsor.  It leads to needless, arbitrary, artificial haste, which is self-defeating anyway.  Pared-to-the-bone staffing levels mean that quality is invariably the first casualty.  Corners cut in testing and quality assurance, the first candidates for cuts, come back to haunt system users in the form of egregious software defects that never seem to get resolved.

This kind of poor corporate behaviours sets up the developers to be permanently on the back foot, fire fighting and under constant, extreme pressure.  Sheer corporate denial of the true costs of development projects leads to corner cutting, short changing the user, making the developers feel bad and to lose pride in their work and in their achievements to date.  It is demoralising and soul-destroying, resulting in a lot of jaded, cynical, burnt out people, who have little residual loyalty to the organisation that has reduced them to this state.  You can’t help getting the sense that people’s parents did not raise them for a working life under these conditions of prolonged psychological stress and social engineering.  It’s the stuff of nightmares.  The emotional and psychological scars are long-lasting.

At the nominal end of the woefully under-funded project (for these projects have a habit of running indefinitely, as the mistakes made have to be patched up somehow, usually by a skeleton maintenance staff), reputations are trashed, pay stagnates, appreciation is never forthcoming, career trajectories are damaged and derailed, stress and depression have been induced, health is damaged and all in the name of delivering something that couldn’t have been built for the time and money available in the first place.

The way that IT and R&D projects are chronically under-funded, poorly run and mismanaged is no way to create anything of aesthetic, technical, business or artistic merit.  It’s a very poor method and model for obtaining utility and growing the value of an organisation.  There’s nothing remotely nurturing about it.

It does make you wonder why anyone with programming or technical talent would want to create anything, under these needlessly Draconian conditions, which arise solely from a fictitious belief that complex, valuable software comes very cheaply.