Software Engineering as Engineering

Software engineering is engineering by definition. But is this vision still precise and useful enough? [triggered by ICSE'13]

Engineering Viewpoint

Software engineering was originally defined by analogy with older engineering fields, like mechanical engineering, chemical engineering, electrical engineering, and so on. Software engineering combines the following factors essential for an engineering discipline:

• application of scientific principles [grounded in science]
• constructing complex technological systems [object of construction]
• used by the society [practical purposefulness]
• under the economic constraints [time and budget restrictions]

(Check out a report about the seminal NATO Conference on Software Engineering.)

The engineering view on software production is very mainstream among software engineers and researchers. The absolute majority believes that the engineering path is the way to go (Denning and Riehle treat it as a natural assumption), and the only question remains: how to make software development a true engineering discipline. Incremental improvements in methods (e.g., predictive models), processes (e.g., metrics), and product qualities (e.g., fault tolerance) are expected to drive software engineering down that path.

At ICSE'13 I've learned that not everyone shares the same view. But before I explain an alternative, let's look at the software metaphors dominant today.

Software Metaphors

Software, the central object of consideration for software engineering, is quite a complex phenomenon to comprehend. To deal with this complexity, humans compare it to perhaps not less complex yet more familiar objects, of which I know two: buildings and bird wings. 

(I only consider metaphors for software as a whole. You can read about finer-grained metaphors in programming here.)

Building metaphor

A building is, I would say, the main analogy to a software system. Much vocabulary of the software field emerged from it: from software architecture to software construction. Much has been written about where the building metaphor is very accurate and where it breaks down. I'll just give it a brief description.

Just like buildings, successful software starts with a concept: what does it do and how do we want others to see it?

By Brian Adams.

Once you understand the concept well enough, you lay out big parts to get a vague understanding of how the object (building or system) is structured.

From Maiden Rock Creamery.

Then you make detailed blueprints. These enable almost limitless opportunities for analysis and synthesis, both for building and software.

From the Writer's Sherpa.

From Stanford.

At last, you actually do the construction. This stage is very different for buildings, which require every single detail of the design be brought to life, than for software, for which final implementation reminds filling the empty holes.

From P.A. Lawson Ltd.

Next comes exploitation and maintenance. The inherent malleability of software allows much broader changes, and many have fallen victims to this seemingly benign opportunity.

This one is mine. I was looking for an excuse to sneak GHC in.

(Don't let this waterfall-ish description fool you: there's a difference between the conceptual stages and the concrete steps with their durations and ordering.)


Despite all the inaccuracies of the building metaphor, it is very useful for understanding software development. I'd say that the inaccuracies are at least as insightful as the similarities between software and buildings.

Wing metaphor

The wing metaphor is less known. It comes from DSA -- a widely recognized book on software architecture. This metaphor isn't very useful to think about stages of development; instead, it illustrates how humans perceive complex objects.

The main insight here is that a complex system can be studied from different perspectives. A perspective is determined by the concerns of interest, goals, level of abstraction, context of the system, the observer, and many other factors. Choosing a particular perspective helps deal with the otherwise overwhelming complexity.

For example, a wing can be seen as a composition of its constituents -- like a software application may be a composition of its Java classes (or threads, or functions, depending on the perspective and the way it's built).

From the Design in Nature.

Each feather can be looked at individually, and it has its own parts (cf. composition and abstraction).

From Fernbank.

On the other hand, one can focus on how a system/wing is integrated into a larger system or environment (a bird).

From AWAKE.

An entirely different viewpoint leads to studying a wing in motion. Here you pay attention to changes over time as opposed to the structure at a fixed point of time. This relates to the gap between static and dynamic software analysis.

From twootz.

The dynamic viewpoint encompasses a wide variety of concrete contexts: not only the whole wing can be observed, but its parts can be "tested" independently. For example, you can take a single feather and drop it to the ground. From what you see you could infer something about how feathers behave in a wing (i.e., transfer conclusions from one context to another).

From Sciencephotography.

As I said, the wing metaphor is more analytic than process-oriented. It offers more insights about analysis than about synthesis. 

Other metaphors

Sometimes one can hear about concrete biological analogies for software, like genetic algorithms, neural networks, and adaptive behavior (see Marinescu and Boloni). One may talk about systems growing organically, that is incrementally and without major re-designs. I have not seen, however, an overarching elegant metaphor like this.

Beyond Engineering

Now that we've talked about the classic engineering metaphors for software, I can explain the weak sides of the engineering paradigm.

One of the basic premises of engineering is studying an object that is easily conceptually and operationally separable from the outside world (environment, context, ...). A car designer can think of and define the car (function and structure) mostly independently of many contexts where it might be used. A processor design can be tested in a lab because the important variability in the context (temperature, pressure, acceleration) can be simulated easily and repeatedly. An engineer has a great deal of experimental control over the artifact under analysis.

For software engineering the separation between software and its environment goes back to Herb Simon's Sciences of the Artificial: he talks about how the artificial (software) is introduced into the world, how it imitates something about nature, but is designed by humans, and how to think about the interface between the artificial and the world. Jackson evolves the topic in The World and the Machine, still replying on the separability of the two and not addressing such concerns as, for example, mutual recursive influence between the machine (= the artificial) and the world (= the environment). For what it's worth, Simon and Jackson support the classic engineering mindset.

Software, however, is often more context-dependent than products of other engineering fields. There's a fundamental difference between, "this car is good if it either drives fast, looks cool, or uses little gas", and "this software system is good if it reasonably satisfies the goals of whoever happens to use it in the future." It seems that the flexibility of software makes us bend software around the constraints of our life, not vice versa (as it happened with cars, for instance).

A change in the game comes from people like Linda Northrop from the SEI, who pointed this out in her keynote at ICSE'13 that the engineering vision might be too reductionist for success with software. With a broad advent of cyber-physical systems (think home automation and driverless vehicles) and ultra-large scale systems (think unified air traffic control for the whole planet), software cannot be easily separated from the context, nor can it be experimented with easily. I can personally add the systems with rich supply chains to the list of examples: is the supply chain part of the world or part of the machine?

Linda sees scale and the uncertainty arising from it as the driving force for extending the engineering vision. She offered a striking metaphor: instead of thinking about designing a building, think about how a city evolves. Nobody explicitly designs a city up-front, trying to elicit the requirements. Nobody tries to do short iterations of city design to deal with uncertainty. These options would be totally helpless, given how many parties and factors are involved. Instead, they try to constrain and direct the growth through policies and creating the right incentives, accepting their incomplete knowledge about the border between the city and the outside world (and even the difference between them).

NYC. From Tripadvisor.

So, in the situations when software is (unpredictably) ingrained into the (complex) environment, the classic engineering mindset may fail, and we might be better off thinking about software in the ecosystem way: cities not buildings, flocks not wings. The definiting characteristics are:

• Lack of control over the whole system, existence of multiple independent parties.
• Difficulty of experiments, non-reproducible setups. 
• Difficulty of predicting how a system would change the world and therefore would need to be changed to satisfy the new world.

PhD students heard about free food. From the Telegraph.

A skeptic may argue that the issues of scale and uncertainty can be solved by merely more careful requirements engineering. In this case, however, the problems of requirements drift and "dirty" recursive dependencies remain the second nature of software.

So what are the implications for today? I honestly don't know. The answer may vary between the extremes of "it's all a bunch of buzzwords, keep the old ways" to "drop the old ways completely." I would certainly suggest paying more attention to this topic and employing humanities and social sciences to help shed some light on these questions. I'm pretty sure that they cannot be answered with only engineering or natural sciences.

---

If you have an opinion on this or related issues, I'd love to hear it.

---

Update April 2015: an exciting discussion with software practitioners moderated by Mary Shaw (who has worked on the issue of SE as engineering for a while).