Why software tools fail and what's needed to succeed

posted by cjh, 13 January 2009

Some of my perceptions of the social dynamic of the use of software tools.

Designers of software are motivated mainly by kudos - if their success might be put down to a tool, they aren’t as motivated to use it. The thing that motivates them is to produce the nice software (that’s already in their head) in the shortest possible time. They like tools when they lessen the work without reducing the quality. Tool output is often seen as a compromise and suboptimal, so there really has to be a big time saving to impress these people. Existing ORM and CASE design tools often haven’t produced the right artifacts to shorten schedules - they’ve been targeted at doing things better and getting them right more often. The developer’s hubris doesn’t allow them to see this as an advantage, since they think if left alone, they could produce perfect software without such tools, and that’s what they want to be admired for.

People who want to be “top dog” in a development team, wielding control in excess of their work output, like to use tools - because their knowledge of the tool gives them a special place, they pull the strings. But such people produce so little of the final artifacts of a project they often contribute little to the success of projects anyhow. Instead they create turmoil by insisting that things be done their way, and redone even if a solution is already working.

These last two paragraphs explain why the CASE tool movement of the 1980’s failed so miserably, not because the tools didn’t work.

Organisations will not invest in software development using tools from companies that may fail, or where the tool is seen as risky or dead-end. A technology has to be established in order to provide an escape route… but these days, a viable escape route is if the tool is open source. Nothing is invested to get started, and no vendor can take you down with them.

The IT function as a whole, is often viewed by the business as having far too much control. In part this is a natural reality, as the business can’t move forward without the IT changes, and they don’t have enough understanding of the challenges of succeeding in software development to trust IT. But on the other part, IT uses its power to gain some control over the business direction, sometimes with legitimacy but not always. So the IT function is further distrusted. In addition, IT often fails to deliver adequate functionality in a timely way, and so are seen as less competent than other areas of the business.

On the other side, the business isn’t often much good at writing specifications. The language used is too vague, and doesn’t reflect an understanding of how the IT systems will support business changes; because the business is concerned with what and why - as it should be. So they employ business analysts, who are meant to bridge the gap, but often fall too much on one side or the other. When IT try to explain that a feature cannot be implemented, or is incompletely specified, they have great trouble explaining why there’s a problem. In part, that’s because they think in terms of how, since that’s the natural tendency of the engineering mind. The failure of the business to understand the problem is seen as legitimizing the degree of control that IT asserts.

All this is down to communication and language. A semantic modelling language must make it easier for the business and IT to work together, not as opponents, engaging in paper warfare, but really collaborating. The best way to do that is to create a single language that both groups can read and write, that can provide both with what they need - precision and consistency for the IT folk, and verifiability against the business rules and process for the business folk.

In the process, the language can also be used to generate the artifacts that both groups need (schemas, code, documentation of business rules) - but it’s main attraction to the business is the way it changes the communication process. The generated artifacts are about reducing the project schedules while ensuring continuous compliance with the specification - but they must be of high quality, and preferably, the generators must be tweakable (open source).

That’s the language I hope CQL will become.