Some of my perceptions of the social dynamic of the use of software tools.

Designers of software are motivated mainly by kudos - if their success might be put down to a tool, they aren’t as motivated to use it. The thing that motivates them is to produce the nice software (that’s already in their head) in the shortest possible time. They like tools when they lessen the work without reducing the quality. Tool output is often seen as a compromise and suboptimal, so there really has to be a big time saving to impress these people. Existing ORM and CASE design tools often haven’t produced the right artifacts to shorten schedules - they’ve been targeted at doing things better and getting them right more often. The developer’s hubris doesn’t allow them to see this as an advantage, since they think if left alone, they could produce perfect software without such tools, and that’s what they want to be admired for.

People who want to be “top dog” in a development team, wielding control in excess of their work output, like to use tools - because their knowledge of the tool gives them a special place, they pull the strings. But such people produce so little of the final artifacts of a project they often contribute little to the success of projects anyhow. Instead they create turmoil by insisting that things be done their way, and redone even if a solution is already working.

These last two paragraphs explain why the CASE tool movement of the 1980’s failed so miserably, not because the tools didn’t work.

Organisations will not invest in software development using tools from companies that may fail, or where the tool is seen as risky or dead-end. A technology has to be established in order to provide an escape route… but these days, a viable escape route is if the tool is open source. Nothing is invested to get started, and no vendor can take you down with them.

The IT function as a whole, is often viewed by the business as having far too much control. In part this is a natural reality, as the business can’t move forward without the IT changes, and they don’t have enough understanding of the challenges of succeeding in software development to trust IT. But on the other part, IT uses its power to gain some control over the business direction, sometimes with legitimacy but not always. So the IT function is further distrusted. In addition, IT often fails to deliver adequate functionality in a timely way, and so are seen as less competent than other areas of the business.

On the other side, the business isn’t often much good at writing specifications. The language used is too vague, and doesn’t reflect an understanding of how the IT systems will support business changes; because the business is concerned with what and why - as it should be. So they employ business analysts, who are meant to bridge the gap, but often fall too much on one side or the other. When IT try to explain that a feature cannot be implemented, or is incompletely specified, they have great trouble explaining why there’s a problem. In part, that’s because they think in terms of how, since that’s the natural tendency of the engineering mind. The failure of the business to understand the problem is seen as legitimizing the degree of control that IT asserts.

All this is down to communication and language. A semantic modelling language must make it easier for the business and IT to work together, not as opponents, engaging in paper warfare, but really collaborating. The best way to do that is to create a single language that both groups can read and write, that can provide both with what they need - precision and consistency for the IT folk, and verifiability against the business rules and process for the business folk.

In the process, the language can also be used to generate the artifacts that both groups need (schemas, code, documentation of business rules) - but it’s main attraction to the business is the way it changes the communication process. The generated artifacts are about reducing the project schedules while ensuring continuous compliance with the specification - but they must be of high quality, and preferably, the generators must be tweakable (open source).

That’s the language I hope CQL will become.

Things don’t have attributes, they have relationships to other things.

Programmers get taught to sort things into objects and their attributes, but that isn’t always helpful. We tend to treat anything we can write down as a value (like a name, a number, or a date) as an attribute of something. But sometimes, a value identifies a thing, and that thing might have other attributes. So the distinction breaks down, and we have to rearrange. In a database, that can mean a lot of extra work.

In a semantic model things don’t have attributes, they have relationships. A relationship might be to a value, and perhaps a given thing may allow only one value in that relationship. That makes it seem like an attribute, but we need to keep those ideas separate.

Is your birth date an attribute? No, it’s just a date to which you have a special relationship - the birth date relationship. Other people have other relationships to that date, and so do other things. Somebody registered their car that same day. The same date plays roles in other relationships, and those roles might even carry meaning in relation to your birth date.

Because you have only one birthdate, it seems obvious to store it as an attribute. That means it’s yours, and not intrinsically related to other things. But what if you made the wrong decision about which concept the birthdate belongs to?

Consider your birth place. Places have their own identity, just as dates do. You only have one birth place, so that could be an attribute too. But there were other people present at your birth… your mother for instance. Your doctor, and nurses. If these things matter in a database you’re designing, you might need to model the birth event. Birth place and birth date are now seen to be attributes of your birth event, not attributes of you at all. The other people involved in your birth also have a relationship to that birth event.

Consider your given name… or is that names? Do the separate names matter, or just the string of names joined up with spaces separating them? That depends on what purpose you have in using the names. You might want to be able to quickly find all the people who have “John” as a middle name - then it might make sense to store the names separately, not as an attribute of the person object.

The same reasoning follows for every kind of attribute. If you start out by thinking about objects (entities) and their attributes, you make assumptions about the way your data will be used.

Instead, start by thinking about how entities are related to values (and to other entities), and make sure you have that clear in your mind. Describe each relationship, using expressions such as “Person was born in Birth” and “Birth is of Person”, “Birth occured on birth-Date”, “Birth occured at birth-Place”, “Person (as Mother) gave Birth”. A hyphen after each adjective will help keep the idea of Place and Date separate from birth date and birth place, while making it clear that birth date is a special role of a Date.

Continue this process of semantic modeling until you have described most of the entities and values that matter, and the relationships (fact types) that join them up.

As you go, you can also record the cardinality of each relationship: “Birth was at exactly one birth- Place”. The “exactly one” isn’t part of the relationship, it’s just a constraint over it. It limits the cardinality of the relationship. Other constraint expressions you might use are “at least one”, or “at most one”. Don’t forget that when you say “exactly one”, or “at least one”, you will need to always know the answer. If you ever need to store information about a Birth but you might not know the birth date, say “at most one”.

When you’re done, or nearly done, you’ll know whether you need a separate Birth table in addition to the Person table, or a separate GivenNames table. Of all the things that you’ve decided matter to you, if Birth is only relevant to one of them (the Person) then since there’s only one Birth per person, then you may be able to absorb the birth date and place as columns of the Person table. It’s a little more complicated than that, but not much.

This is one reason why semantic modeling works better than traditional ER or UML modeling. It’s possible to make a complete model before making decisions about which things are attributes and which aren’t. It still won’t be your final model, but you’ve postponed some bad decisions so you can make good ones instead.

Because of these rules about when you can absorb things, even small changes or additions to your semantic model will cause changes in which tables you need. The shape of your relational database will always be more prone to change than the original semantic structure. For example, a new requirement might be to link up the Birth to the hospital records management system, to be used in paying the medical staff who assisted. Suddenly the Birth details don’t look very Personal any more - even though they haven’t changed!

But because your semantic model hasn’t changed much, you should be able to get away with making only small changes in your code - assuming you followed through properly! As long as the details of the changed tables are hidden under a semantic layer. But that’s a problem for another time…

In the meantime, restrain yourself making from early assignments concerning attributes, and you’ll find you discover new meanings in the information that embodies the rules of your business.

Last night I presented Nathan Sobo’s excellent Treetop packrat parser generator to the Melbourne Ruby community. Some of the material was directly adapted from Nathan’s presentation (thanks Nathan!), but I took a different approach and show my own examples.

I’ve been using Treetop to construct a parser for CQL, a very interesting new development based on restricted natural language. CQL is for data definition and query, and rolls together my natural language approach with concepts from ORM, SBVR, Prolog, and the Web Ontology Language OWL. But I don’t present that language here. Patience doodz, the time for unveiling will come!

If you’re interested in Treetop, you can download the 23MB MP3 audio file and the PDF of my presentation slides. The example code and driver program are also available.

I talked about how we build systems by composing them from elements, not by decomposing monolithic “problem statements”. The elements always depict either states, or transitions between states. These two angles are the information perspective and the process perspective. Process steps always transit between legal states, so the set of legal states must be defined first. It’s not that information is pre-eminent, but it does tend to lead rather than lag the rest of the design. So here I’m focusing on the information part of our design.

Now, we need to build up an overall aggregate picture of what things our system can describe, and what it needs to know about those things. The picture is made of many small elements, and many small constraints on the ways they interact. The sum of these small things forms our conceptual model - they reflect the way we and our clients think about their problem, not its solution. To store them however, we need to group them together for efficient management. That’s what we’re doing when we’re building a database design - writing down all the things that the system needs to know, in a way that will be efficient to manage. There are two goals here: manage all the elements and their interactions without losing track of any, and produce an aggregate structure that is efficient. These goals work against one another.

When we’re done, if we’ve done a good job, we have a normalised database design. “Normalised” basically just means that it provides only one way to represent any of the elementary facts, so that you can’t have two versions that disagree. But there’s another property of normalised data that causes problems: any one “thing” will only have one record, and all facts about that thing for which there is only one value at a given time are stored in that record. This aggregation is a fine principle for creating efficient physical storage structures, but the aggregation leaks into our code.

When we query the data using SQL, there’s one way of accessing a fact that has only one value for each thing - the column - and there’s another completely different and somewhat difficult way of accessing facts that have more than one value for each thing - the join. SQL forces the direct use of the physical database model, while at the same time hiding the true domain model which is present in the elementary form. This prevents the domain expert from properly engaging in verifying the model and ensures communication problems because of the translation and interpretation required.

Being bound to the physical model also tragically limits the agility (evolution) of applications because the physical model is always more unstable than the conceptual model. The mere number of values (or other things) related to a thing in a given fact relationship should be a minor detail, yet it completely controls the physical model. When a requirement changes in tiny ways, we can sometimes end up needing to do a major restructure of the database, potentially across many tables.

Imagine you have a table of users, and one column is the “given names” column. Your client now needs to store information about the reason your parents gave you each name… and all of a sudden you need to move the contents of that column out into a new “given names” table. Every query that fetched a user will probably have also fetched their given names, and so now needs to be rewritten. All we did was add a new fact to an otherwise complete model - why does all our code need to be checked and maybe rewritten? Ok, perhaps “given names” is an uncommon example, but this sort of thing occurs so often in relational databases that for more than twenty years, it has its own name: attribute migration. It’s an example of just one way the addition of a small item to our elementary model causes big impacts on out aggregate design.

So while relational databases are one of the preeminent achievements of computer science, they must move beyond requiring direct dependence on the physical storage structures. SQL is the problem here, because of the gross difference between accessing a column (single value) and another table via a join.

Replacing SQL by a language that has this property of uniformity of reference must be the top priority if the industry is to more forward in solving this critical problem. There is a way out of Vietnam… but only after we replace SQL. Tune in next time for a first peek at the language that can do this, the Constellation Query Language.

Object/Relational mapping has been called the Vietnam of Computer Science, meaning, I think, that it’s become an intractable problem that we never needed to get into in the first place. Actually, it was unavoidable, but there’s a way of hiding the problem, which is the subject of this series of articles.

The core of software design is expression; how do we express what we want a system to do, to be, and to achieve? It’s hard for software folk to think clearly about this. We’re conditioned by having problems handed out on a sheet of paper during our training. We’re taught to break them down, decompose them, by various methods. We worry and argue about the right way to go about decomposing problems.

In reality, we never receive problems fully-formed like this, and so we seldom have to decompose them. Instead, our clients witter on about how this should have one of those, and how a thing is on this list unless that condition holds… and we have to compose a system out of these fragmentary utterances. Composition and aggregation, not decomposition, is our main activity. In the process, we try to distil and create conceptual purity from the original communications.

In choosing how to aggregate things, we take various approaches. Object-Oriented practitioners group things mostly by shared behaviour. Database people struggle to avoid duplication while clustering things to maximise disk throughput and transactional reliability. In both cases, the attempt to maintain purity is moderated by the need to work within the bounds of physical computer hardware - main memory on the one hand, disk drives on the other. One is volatile, the other persistent. These two place very different constraints on the shape of an optimum solution. Both are based in the real world, so the problem is to some extent unavoidable.

It gets worse though… neither solution is very close to the original problem statement, which shuts out the domain expert. We actually have not a two-way problem, but a three-way one, played by three roles:

1. The Business Analyst or domain expert
2. The Software Designer/architect
3. The Data Managers

In general, none of these wants quite the same things or talks the same language as the others, and none really accepts the other’s view on things. Depending on who you ask, they’ll always point to another group as being the origin of the communication problem. So we have a stand-off, and rocks get thrown in all directions. This is the most pernicious and costly communication problem in the software industry.

To get out of Vietnam, we have to create a language in which all three groups can be equally fluent, and which gives each group what they need. We need a language of facts which is at once formal and accessible, and which can be automatically and efficiently mapped to objects and to normalised database designs. It must reflect natural verbalisations, yet have an unambiguous meaning. It’s not UML or Barker ER notation. Think it sounds too hard? Come back to read the next instalment.