Simple HPC Wins – Usually

N.B. This essay also appeared on the Communications of the ACM web blog.

In the 19^th century, writing about his work on mechanical calculating devices, Charles Babbage noted, "The most constant difficulty in contriving the engine has arisen from the desire to reduce the time in which the calculations were executed to the shortest which is possible." Roughly a century later, Daniel Slotnick wrote retrospectively about the ILLIAC IV parallel computing design, "By sacrificing a factor of roughly three in circuit speed, it's possible that we could have built a more reliable multi-quadrant system in less time, for no more money, and with a comparable overall performance."

Babbage's design challenged the machining and manufacturing capabilities of his day, though recently, others were able to build a functioning system using parts fabricated to tolerances achievable with 19^th century processes. Similarly, Slotnick's design challenged electronics and early semiconductor fabrication and assembly. Today, of course, parallel computing designs embodying tens of thousands of processors are now commonplace, leveraging inexpensive, commodity hardware.

Technology Lessons

There is a lesson here, which systems designers repeatedly ignore at their peril. Simple designs usually triumph, and artful exploitation of mainstream technologies usually bests radical change. Or, as Damon Runyon once archly observed, "The race may not always be to the swift, nor the victory to the strong, but that's how you bet."

All of which is to say that incrementalism wins repeatedly, right up to the point when a dislocating phase transition occurs. There are, of course, many paths to failure. One can be too early or too late. Or to put it another way, you want to be the first person to design a successful, transistorized computer system, not the last person to design vacuum tube computer. The same is true of design approaches such as pipelining, out of order issue and completion, superscalar dispatch, cache design, system software and programming tools.

Innovator's Dilemma

Any designer's challenge is to pick the right technologies at the right time, recognizing when inflection points – maturing, disruptive technologies – are near. This is the essence of Clayton Christensen's well-documented innovator's dilemma.

The shift from largely proprietary high-performance computing (HPC) designs to predominantly commodity clusters a decade ago was but one of the most recent transitions. Arguably, we are near another disruptive technology point. The embedded hardware ecosystem offers one intriguing new performance-power-price point, particularly as we consider trans-petascale and exascale designs that are energy constrained. The experiences of cloud providers in building massive scale infrastructures for data analytics and on-demand computing are another possibility.

As I frequently told my graduate students at Illinois, the great thing about parallel computing is the question never changes – "How can I increase performance?" – but the answers do. Babbage would have understood.

Comments

[Gretchen Huizinga]

I liken your take on the innovator's dilemma to a sort of business version of a spiritual phase transition. I paraphrase Matthew 7:13-14:

"Enter through the narrow gate. For wide is the gate and broad is the road that leads to product obsolescence, and many enter through it. But small is the gate and narrow the road that leads to market dominance, and only a few find it."

Now if we could just identify exactly where that particular narrow gate is...