Why are enterprise software products so badly designed?

The question was asked, about a poorly-designed enterprise database, surprisingly enough by an old hand who’s managed major development programs in the software industry:

Why was this system designed to be like this?

If it’s anything like the enterprise databases I’ve encountered, it was never designed at all. It just grew by people throwing one thing after another at it, without anyone ever trying to understand or optimise the big picture. This is exacerbated by success; as soon as a system or especially a product becomes a success, it grows large, which means no-one is willing to understand it all just to add one new feature. In addition, it accrues an entire ecosystem of extensions, add-ons and other kinds of dependencies. No-one can ever know what all the schema dependencies are so nothing can ever be changed – the only path is to add new things. Any mistakes made in the early data modelling are magnified a thousand times as new additions are made that work around the inability to change the basic structures. The result is best described by the acronym BBOM – Big Ball Of Mud.

If, as often happens now-a-days, the original model is a semi-automatic or simplistic mapping of an object-oriented schema, the problems are much worse, even if that O-O schema was carefully designed and really rather good. In fact, perhaps especially if it was rather good, since that reflects the owner’s belief in the supremacy of the O-O model, and hang the mere storage concerns. Chuck that object model over the wall and let Hibernate deal with it.

Adding to this woe is the fact that enterprise markets enshrine mediocrity. The product that just installs and works produces no on-going consulting revenue (from junior “consultants” being charged out at five or ten times their market value as employees). My point? It’s the same consulting companies who get to write the purchasing recommendations, since no CIO will write a seven- or eight-figure cheque without first getting a consultant’s recommendation to protect their career. As a result, enterprises buy the worst product that can, with lots of help and constant ongoing coddling, be coerced into doing most of the job. So the markets are dominated by products like SDM, SAP, and a hundred others equally as horrendous. The purchasing process is simply not rational and informed, as my correspondent seems to wish it was.

How do I know this stuff? Because I’ve spent three decades as an architect of three major software products that each mostly maintained their architectural integrity (thanks to my work) over a decade or more, dozens or hundreds of releases, millions of lines of code, many sub-products and spin-offs, being deployed in mission-critical roles on millions of computers. They ran or are running banks, telcos, stock exchanges, aircraft manufacturers, armies, national postal systems… and yet, because the market consistently preferred far inferior products, the products I designed, though successful by many measures, never became major cash cows and dominated to the point of excluding others from the industry. Can you tell I’m proud of my work? Yet I never made anything more than salary from all my hard work and the risks I (and my co-founders) took.

I’m not bitter about that… just realistic… I wouldn’t have it another way. In fact, I’m still somewhat hopeful that software purchasing will grow up and start saying no to this rubbish, but I doubt it’ll happen in my lifetime. It took four to six hundred years for accounting and banking to grow to the professional calibre we see today, and there’s still such skulduggery there. Until software engineers and product managers start showing the markets that a new standard of behaviour is possible, it will not become accepted and expected.

And that is why I believe that fact-based modelling is the key to the software industry entering its adulthood. The problem of complexification can only be solved by learning how to never take a major modelling mis-step, and by implementing software so as to minimise the exposure of modelling mistakes – so they can be fixed without rewriting half the world. This requires reducing dependencies on schemas to the minimum required to support each transaction’s semantics; which means no visibility of wide tables or objects, just the individual meanings encoded in those tables.

Configuring a modern Linksys ADSL2 router

It took me a number of hours to get a basic ADSL2+ configuration working with a new Linksys X3000, which is a higher-end domestic ADSL2+ router with Wireless, Ethernet (of course) and USB storage. This blog says why, and what the solution was. I hope it helps someone.

But before I get on to the particular challenge of the X3000, I had to get the X3000 to connect to ADSL. This was also a challenge, even after I contacted TPG to find out what DSL mode to use and which of several versions of my usernames and passwords to enter.

Ok, so once the ADSL model connects, the key is this. Until the wireless security and password has been configured, almost no network traffic will be routed between the ADSL and internal networks (wireless, ethernet). This is true even if you aren’t using wireless – which is pretty likely, since the X3000 comes with wireless configuration disabled.

Before the wireless is secured, the X3000 diverts all outbound Internet requests to a web page with a bold warning. In order that the request can be made however, it does allow DNS requests to get through. All other traffic is blocked; so if you configure ADSL and Ethernet then test using the “ping” command, you get your IP addresses but no actual ping responses.

But why didn’t I see the security diversion and the big warning page? Here’s where Chrome came in.

By default, Chrome uses a web service instead of DNS, to help with auto-complete. The trouble with this is that it prevents access to the Internet through the modem in its security-diversion mode. The DNS-replacement web service is blocked, so your browser never finds out where to send requests. Note that this affects you even though you’re still using Ethernet and haven’t even looked at the wireless yet. So “ping” doesn’t work, and neither does web browsing. DNS works however, which causes some hair loss, because you know that most things are working ok. You never see the security diversion page because there’s never a request to divert.

So then you get the IP address for google.com, or whatever website you usually use to test connectivity, and enter that into the Chrome address bar. This should answer you immediately, no?

Weeelll, no. Chrome bites again.

Chrome also has a default mode to foil some phishing attacks. At least I think it was the anti-phishing measures. Either way, Chrome doesn’t want to render pages that have numeric IP addresses until after it has looked up the IP address in its list of known-bad sites, OR roughly 30 seconds has expired. So from the first time you tried to visit the new internal IP address you set for your local network (assuming you changed it from 192.168.1.* (as I did really early on), it takes 30 seconds at least to fetch every configuration URL from the router. And guess what? The security redirect page has a couple of dozen items in it (images etc), so yuo basically never get to see the whole thing.

You can disable both these settings in the Chrome advanced preferences. Or you can switch to Firefox for configuration, which is what I did. Suddenly the router configuration pages started rendering quickly and completely, and when I got to the security diversion page, I was able to figure out why I’d been having troubles all along.

Anyhow, there it is, the results of a number of hours of my life wasted because some helpful software folk thought they’d make my life safer. And I suppose in a way it did; I might have been outside in the sunshine instead, and got hit by a bus.

Are we willing to cross the crevasse?

Dan North’s post on software craftsmanship, and Martin Fowler’s response to it, are right on the money regarding what’s required for programming to become a proper profession.

Ultimately, developers are mostly rather good at putting software together, even those who don’t use the most fashionable methods. Good software can be made using many different methods, not just the ones espoused by the current golden children of the software development fashion cliques, whether it be extreme, lean, agile, or whatever.

Where we continue to fail is in not even attempting to build the bridge that Dan and Martin talked about in The Yawning Crevasse of Doom. We mutter and sneer about how business folk don’t know what they want, and how they’re incapable of applying rational thought to that problem. They feed us endless waffle, and our job is to make it all make sense, to figure out what they need and to meet those needs.

I call BULLDUST! The business knows what it needs, mostly, or they wouldn’t open the dialog. The problem is with developers. We don’t want the business to read, understand or speak our languages, because that makes our speciality less special. Face it, we’ve spent all this time learning out how to describe the structure and function of a computer program in a formal way, and we don’t want someone else telling us why it doesn’t meet their needs. Especially when they don’t even seem to know what they need. It’s bad enough that we often build software that doesn’t meet the business needs, but we certainly don’t want to be told why.

The level of dialog engendered by these attitudes devolves to the business throwing their waffle words at us, and us throwing software back. Agile methods tell us how we can do that really quickly, so that a single interchange in this dialog might take as little as just one week.

ONE WEEK.

And all because all we can speak is software, and all we can hear is waffle. Let me tell you something – the business actually does know what it wants. All we have to do is help them know how to say it. A way to capture the meanings that are in their heads.

If we can find a way to talk the same language, we’ll have dialog interchanges that take only seconds or minutes. Interchanges that don’t require building anything. Interchanges that carry meaning, in each direction, until we can agree on the nature of the things we’re talking about. Only then should we apply all those craftsmanship techniques to build the right thing in the right way.

That’s the kind of dialog a semantic modeling language can facilitate. You might like my one or you might not, but to advance, we must start trying to build such things. I have to say however, the dismissive responses I see from software craftsmen doesn’t improve my confidence that we have the will to fix this problem. Yes, I know that a semantic modeling language doesn’t do anything you can’t already do… except communicate back to your business users before you build the software. I can only fall back on the incredible enthusiasm I get from business folk and regular computer users when I show them what I’m attempting. They understand the nature of the problem that developers are so unable or unwilling to see.