5 Readings From A Hidden History Of Software Engineering
(Image: Johann Georg Meyer “Das Lesende Mädchen”, 1871)
Introduction
There is a hidden history to software engineering that some have come to know, but of which most are sadly unaware. Instead, as happens so often in history, a vein of easily graspable but unworkable techniques and ideas has come to dominate the industry. These techniques are bundled with their own superficially plausible justifications and are parroted, easily enough, by amateurs to ensnare other amateurs. And so, we have a generation of software engineers trained by amateurs whose working techniques fall apart in the presence of even modest complexity. Yet, instead of revisiting the techniques themselves, recourse is had to critical theory, chaos theory, and ultimately the ship grounds itself upon a nihilistic skepticism – “if these universally proselytized techniques do not work then no technique at all must work!”
And yet, while these amateur laborers were busy twisting themselves and others into a Gordian Knot, another, quieter, more subterranean tradition had already found the means by which to cut it. Below you will find 5 readings from within this tradition, none of which will have appeared on any best of list, and yet any one of which is worth a whole stack of best-ofs. Each contains contains numerous as yet un-mined veins and places for future extension.
I first stumbled upon this hidden tradition in despair, after having tried the popular techniques, to the letter, and found them wholly inadequate to the demands of professional work – late projects, poor quality, an inability to plan and estimate, all were problems that continually plagued me under their tutelage. Finally, I reached a lowest point, and a for a time I gave myself over whole-heartedly to nihilistic skepticism, to the despair of ever gaining even a modicum of control or professional ability. After several months I could bear this no more, and I resolved to either find others who had seen the need for this work or to forge ahead alone and do it myself.
Luckily, it wasn’t long before I saw that many had indeed seen this need but that their work was not widely read. Several had even labored for decades to produce whole systems of techniques. Slowly, I learned these disciplines and began to apply them to my professional work. Almost immediately, I met with success from their application. Eventually, I could plan projects, estimate work accurately, and reliably deliver quality. Now, after several years of their application, I am confident that they are worth sharing with you, dear reader.
And so, without further ado, I present to you five readings that will change your software engineering for the better.
1. PSP: A Self Improvement Process for Software Engineers
This first book is the best introduction to the hidden tradition. Its author, Watts Humphrey, was the former Vice President of IBM during its heyday where he supervised over 4,000 software professionals across 15 laboratories in 7 countries. It was at IBM that he saw first-hand how talented professionals following disciplined processes could produce extremely complex software at high levels of quality. It was also at IBM where he would witness the damage that can be inflicted when disciplined processes aren’t in place.
Watts left IBM in the early 1980s and in 1986 joined the recently established Software Engineering Institute (SEI) at Carnegie Mellon University. His goal was to explore more deeply the idea of disciplined working processes for managing complex software projects.
During his tenure at the SEI he created the influential Capability Maturity Model (CMM) which would later evolve into the Capability Maturity Model Integrations (CMMI), elaborated most clearly in his book “Managing the Software Process
The PSP, detailed in the book above, is often called a “one-man CMMI level 5” process. It is an extremely disciplined approach to developing software. Learning this process will give you the ability to plan, track, and measure your work as well as the data you need to continually improve. For many software engineers this will also be their first serious exposure to measuring, planning, data tracking, and design. For this reason, the experience of learning the PSP is revelatory in many different ways.
Published shortly before his death in 2010, this book contains the distillation of several decades of theory and practice by one of the most dedicated minds in software engineering. It is a complete package, and will reveal to those disciplined enough to work through its 14 chapters and 8 accompanying exercises, an alien but extremely effective way to develop software. One that produces high-quality work, and one that scales.
2. Cleanroom Software Engineering
Having stepped through the looking glass many methods that would initially have fallen completely below the line of attention begin to rise to the fore. Cleanroom software engineering is one such method. It has been used successfully most notably by NASA to develop space shuttle software, but also found extensive use in embedded systems and telecommunications.
Originally developed by Harlan Mills and his colleagues at IBM, cleanroom engineering flips traditional software engineering on its head. Its main idea is that defect prevention – stopping bugs from getting inserted in the first place – is more cost effective than defect removal – detecting and removing bugs after you’ve already inserted them. Since most modern methods focus almost exclusively on post hoc defect removal through extensive testing, this can come as quite a surprise. By far the most radical consequence of this change in emphasis is that no unit-testing is typically recommended at all on Cleanroom projects, and was found to be largely unnecessary in practice.
The cleanroom process focuses on the idea of “maintaining intellectual control”. This means that designs start high-level and are turned into software in a top-down fashion. Each refinement of the design is linked to the level above it, allowing strict traceability from requirements through to implementation in specific lines of code. This also means forgotten requirements can be easily found in review and that the final conversion into code is usually “obvious” given the preceding refinements. This “box model” is a simple and powerful technique for completely specifying systems.
Another great idea introduced by this approach is that of “intended functions”. The basic idea is that each line of code should be accompanied by a comment that explains, in a rigorous notation, exactly what that line of code is intended to do. This makes code review much easier, as lines of code are simply checked against their intentions. This also allows formal proofs to be easily applied to programs in order to show that they generate their intended outputs.
Alien, contrary, and packed with powerful ideas that can be easily separated from the overall method. The Cleanroom approach provides a wealth of tools to be incorporated into your personal software process.
3. Documenting Software Architectures
If disciplined engineering methods teach you anything, it’s to take up-front design seriously. However, most (probably all) software engineers do not learn how to do professional-level design work in school. Therefore, two big questions immediately confront us: (1) What should we be designing? (2) How should we convey our designs?
This ambitious and highly valuable book attempts to tackle the second of those questions. Its introductory companion, “Software Architecture in Practice“, answers the first first – namely, you should be designing an architecture that meets the quality attribute requirements of your project, and document any trade-offs between quality attribute requirements for future reference. For those unfamiliar, the software architecture is basically all the interesting stuff you might need to communicate to both technical and non-technical folks about how your system works and where it fits together with other systems.
The book provides thorough coverage of the 3 types of views you’ll likely be using to capture your architecture: module views, component-and-connector views, and allocation views. It then shows you how to combine these views in order to create a coherent document that completely covers the architecture of the system you intend to build. I found myself having to take frequent breaks while reading the book to actually go and practice what it was teaching me. Often, it had shown me a simple way to document some functionality that I’d been unable to document before.
Once you’ve started taking disciplined engineering seriously, you’ll find that your practice is still rudimentary. This book will give you the toolset to move your design work from amateur to professional and level-up your personal process.
4. The Rational Unified Process Made Easy
One common trouble area when learning disciplined practices is how to think about scaling them down and up. In addition, working in any organization quickly reveals that a personal process must be part of a larger life-cycle that includes the initial vision, requirements, deployment, and post-release maintenance.
Kroll and Kruchten fill exactly this gap in their wonderful introduction to their own disciplined process – The Rational Unified Process (RUP). The life-cycle of a RUP project is divided into four phases: Inception, Elaboration, Construction, and Transition. Each phase is concluded with a milestone that defines what “done” for that phase looks like. When the milestone is hit, a project is ready to either transition to the next phase or repeat another iteration of the current phase.
What is especially wonderful about this book, however, has little to do with the elaboration of the RUP. Instead, it has to do with the project examples provided. It begins with Project Deimos, an example of applying the disciplined RUP life-cycle to a team of one project. This example shows what disciplined and professional individual working practices look like. Following this solo demonstration, the book then scales up the RUP life-cycle for three other imaginary projects: Project Ganymede – a small green-field project, Project Mars – a large green-field project, and Project Jupiter – the second generation of an existing large project. Each of these demonstrates how the RUP practices can be scaled and adapted to different projects one might encounter in industry.
The example based approach taken in this book combined with its organization-scale view of software process is the perfect antidote to the inward-looking nature of most personal process books. It helps to locate personal processes inside of the organization scale process that support them, and it also demonstrates what disciplined organization-scale processes look like. For these reasons, it deserves a place among the rest of the hidden history.
5. Object-Oriented Software Engineering
Finally, we come to this book, the larval stage of practices that would eventually become use-cases, RUP, UML, etc. The scope of this book is staggering. It covers industrial-scale software engineering, use-case driven development, object-oriented design, and component based engineering. Heck, there’s even a chapter on embedded systems. The overall goal is to teach you how to write software with an eye towards starting your own industrial revolution. You do this by creating a set of organization-wide reusable components as you develop your systems. These components eventually become your competitive advantage, allowing you to outmaneuver competitors in producing novel assemblages of existing functionality.
It’s clear that these ideas have either failed completely in practice or that most organizations have never really tried to use them. Either way, they are ambitious and worth considering. The rise of frameworks lends some credence to its underlying philosophy, and I think more organizations need to be thinking about how to design and architect systems like this. Its not hard to imagine these ideas, when combined with high-quality personal practices, could catalyze a new generation of very high quality frameworks and component systems with which to begin tackling the challenges of the 21st century.
This book is included because of its vision, which is still worth taking seriously almost 30 years later. Perhaps the best vision of the overall plan is presented in Robert Martin’s “Architecture: The Lost Years Talk“.
I hope you’ve enjoyed this view into the hidden history of software engineering, and hope these books come to benefit you as much as they have me.
