Reed's Ruminations: A Blog by Dan Reed

Analog Computing: Time for a Comeback?

N.B. I also write for the Communications of the ACM (CACM). The following essay recently appeared on the CACM blog.

In the early dates of the automobile, there was a lively competition among disparate technologies for hegemony as the motive power source. Steam engines were common, given their history in manufacturing and locomotives, and electric vehicles trundled through the streets of many cities. The supremacy of the internal combustion engine as the de facto power source was by no means an early certainty. Yet it triumphed due to a combination of range, reliability, cost and safety, relegating other technologies to historical curiosities.

Thus, it is ironic that we are now assiduously re-exploring several of these same alternative power sources to reduce carbon emissions and dependence on dwindling global petroleum reserves. Today's hybrid and electric vehicles embody 21^st century versions of some very old ideas.

There are certain parallels to the phylogenic recapitulation of the automobile now occurring in computing. Perhaps it is time to revisit some old ideas.

Analog History

Use of the word "computer" conjures certain images and brings certain assumptions. One of them, so deeply ingrained that we rarely question it, is that computing is digital and electronic. Yet there was a time not so long ago when those adjectives were neither readily assumed nor implied when discussing computing, just as the internal combustion engine was not de rigueur in automobile design.

The alternative to digital computing – analog computing – has a long and illustrious history. Its antecedents lie in every mechanical device built to solve some problem in a repeatable way, from the sundial to the astrolabe. Without doubt, analog computing found its apotheosis in the slide rule, which dominated science and engineering calculations for multiple centuries, coexisting and thriving alongside the late comer, digital computing.

The attraction of analog computing has always been its ability to accommodate uncertainty and continuity. As Cantor showed, the real numbers are non-countably infinite, and their discretization in a floating-point representation is fraught with difficulty. Because of this, the IEEE floating-point standard is a delicate and ingenious balance between range and precision.

All experimental measurements have uncertainty, and quantifying that uncertainty and its propagation in digital computing models is part of the rich history of numerical analysis. Forward error propagation models, condition numbers, and stiffness are all attributes of this uncertainty and continuity.

Hybrid Futures

I raise the issue of analog computing because we face some deep and substantive challenges in wringing more performance from sequential execution and the von Neumann architecture model of digital computing. Multicore architectures, limits on chip power, near threshold voltage computation, functional heterogeneity and the rise of dark silicon are forcing us to confront fundamental design questions. Might analog computing and sub-threshold computing bring some new design flexibility and optimization opportunities?

We face an equally daunting set of challenges in scientific and technical computing at very large scale. For exascale computing, reliability, resilience, numerical stability and confidence can be problematic when input uncertainties can propagate, and single and multiple bit upsets can disturb numerical representations. How can we best assess the stability and error ranges on exascale computations? Could analog computing play a role?

Please note that I am not advocating a return to slide rules or pneumatic computing systems. Rather, I am suggesting that we step back and remember that the evolution of technologies brings new opportunities to revisit old assumptions. Hybrid computing may be one possible way to address the challenges we face on the intersecting frontiers of device physics, computer architecture and software.

A brave new world is aborning. Might there be a hybrid computer in your hybrid vehicle?

Surrounded By Opportunities

Perhaps the closest I have approached the feeling of intellectual omnipotence (and it has not been often) was when I was a graduate student, teaching undergraduate FORTRAN programming for science and engineering students. Those were the days of punched cards, batch processing and line printer output. It was a time when dinosaurs roamed the earth, my slide rule was still a fond memory, and we were computing on massive machines with less performance than my smartphone has today. (Ignore that part about the dinosaurs, as it was a wee bit earlier.)

During my office hours, students would troop through for help with their programming assignments, bringing program printouts. These printouts began with a "day file" showing the timeline of compilation and execution steps (JCL steps on IBM S/360), followed by the program and then any output. After having seen a few students, you realize that elementary programming errors have just a few equivalence classes, often identifiable from the day file alone. I took great pride in listening to the student description of the problem, predicting the root cause and then immediately locating it on the printout. The student was inevitably amazed. Yes, it was a cheap parlor trick, but as a graduate student, you take gratification where you can find it.

All of this reminds me that sequential programming remains difficult and parallel programming even more difficult, despite all our efforts to tame the unruly beast with syntactic sugar, specification and scripting languages, libraries and tools. I am no expert in parallel programming language design, nor am I a cognitive psychologist, but like many of my readers, I have pulled a few all-nighters writing code. Perhaps that at least qualifies me to have an opinion.

In a world of parallel programming, multicore architectures, dark silicon, and rising software complexity, we face some difficult choices. It's time to learn from the past.

Fading Abstractions

We have been fortunate to date. Although sequential programming at scale is intellectually challenging, we have been riding the wave of ever-increasing processor performance, made possible by semiconductor advances. In turn, these were based on an abstraction – the Mead-Conway design rules –that largely separated device physics from microprocessor architecture. Finally, the von Neumann abstraction has also served us well, allowing decades of software development atop an easily understood model of sequential instruction execution.

Our run of good luck is over. Put another way, we are surrounded by opportunities! Parallelism is now the norm for almost all software developers, not just those of us in high-performance computing. More perniciously, heterogeneous parallelism is here, with disparate cores, accelerators and functional units. To manage dark silicon and exploit continuing transistor density increases, this trend will only accelerate, with a dizzying array of heterogeneous chips and devices challenging code portability and performance tuning. Finally and most worrisome, our device physics abstractions are breaking down, with device physics affecting architecture. We face real silicon reliability, process variation and energy challenges, and near threshold voltage computations emerging.

Holistic Collaboration

The abstractions of device physics, processor architecture and programming models naturally led to a corresponding intellectual specialization. Consequently, those working in materials science and device physics have little in common with those designing software tools. I believe that must change if we are to address our challenges.

We need new, integrated design methodologies and rich collaborations that consider end-to-end design and capability. In many ways, this is a return to the past, necessitated by the desire for continuing increases in performance, mediated by the desire for reliability, scalability, programmability and economic feasibility. It will not be easy, but it is a way forward.

Why We Compute

N.B. I also write for the Communications of the ACM (CACM). The following essay recently appeared on the CACM blog.

Why do we, as researchers and practitioners, have this deep and abiding love of computing? Why do we compute? Superficially, the question seems as innocuous as asking why the sky is blue or the grass is green. However, like both of those childhood questions, the simplicity belies the subtlety beneath. Just ask someone about Raleigh scattering or the quantum efficiency of photosynthesis if you doubt that simple questions can unearth complexity.

At its most basic, computing is simply automated symbol manipulation. Indeed, the abstract Turing machine does nothing more than manipulates symbols on a strip of tape using a table of rules. More deceptively, the rules seem simpler than some board games. Though vacuously true, the description misses the point that symbol manipulation under those rules captures what we now call the Church-Turing thesis.

However, as deep and as beautiful as the notion of computability really is, I doubt that is the only reason most of us are so endlessly fascinated by this malleable thing we call computing. Rather, I suspect it is a deeper, more primal yearning, one that underlies all of science and engineering and that unites us in a common cause. It is the insatiable desire to know and understand.

Lessons from Astronomy

When I recently I stood atop Mauna Kea, looking at the array of telescopes perched there, I was again struck by our innate curiosity. Operated by a diverse array of international partnerships and built on Mauna Kea at great expense, they are there because we care about some fundamental questions. What is the evolutionary history and future of the universe? What are dark matter and dark energy? Why is there anything at all?

Answers to these questions are not likely to address our current economic woes, improve health care or address our environmental challenges. We care about the answers, nevertheless.

As I pondered the gloaming sky, my thoughts turned to Edwin Hubble, who first showed that some of those faint smudges in the sky were "island universes" – galaxies like our own. The universe was a far bigger place than we had heretofore imagined. As Hubble observed about this age-old quest to understand:

From our home on the Earth, we look out into the distances and strive to imagine the sort of world into which we are born. Today we have reached far out into Space. Our immediate neighborhood we know rather intimately. But with increasing distance our knowledge fades, and fades rapidly, until at the last dim horizon we search among ghostly errors of observations for landmarks that are scarcely more substantial. The search will continue. The urge is older than history. It is not satisfied and it will not be suppressed.

Hubble's comment was about the observational difficulties of distance estimation and the challenges associated with identifying standard candles. However, it could just as easily have been a meditation on computing, for we are driven by our own insatiable desires for better algorithms, more flexible and reliable software, new sensors and data analytics tools, and by ever larger and more faster computers.

Computing the Future

Why do we compute? I suspect it is for at least two, related reasons, neither relating to publication counts, tenure, wealth or fame. The first is the ability to give life to the algorithmic instantiation of an idea, to see it dance and move across our displays and devices. We have all felt the exhilaration when the idea takes shape in code, then begins to execute, sometimes surprising us in its unexpected behavior and complexity. Computing's analog of deus ex machina brings psychic satisfaction.

The second reason is that computing is an intellectual amplifier, extending our nominal reach and abilities. I discussed the power of computing to enable and enhance exploration in another CACM blog. (See Intellectual Amplification via Computing.) It is why those of us in computational science continually seek better algorithms and faster computer systems. From terascale to petascale and the global race to exascale, it is a quest for greater fidelity, higher resolution and finer time scales. The same deep yearning drives astronomers to seek higher resolution detectors and larger telescope apertures. We are all chasing searching the ghostly signals for landmarks.

It is our ability to apply our ideas and their embodiment in code to a dizzying array of problems, from the prosaic to the profound, that attracts and compels us. It is why we compute.

Hubble was right. We compute because we want to know and understand. The urge is deep and unsatisfied. It cannot be denied.

Nothing Lasts Forever

Those of us in computing and especially those of us in high-performance computing (HPC) have been received Pavlovian conditioning to believe that computers will always be faster if we just wait eighteen to twenty-four months. (Excuse me while I wipe the saliva off my chin, triggered by thinking about this.) It seems this trend has ever been so and ipso facto, so shall it ever be.

Unfortunately, no exponential continues forever, at least outside the mathematics textbooks. All technology-based advances are ultimately limited by something, usually some physical or economic limit. Innovation then shifts to other metrics, against which advances can be measured and valued. Computing is no exception.

Leavin' On A Jet Plane

In the heady days of aviation, when world speed records were set every year, it must have seemed that the sky was literally the limit on achievable airspeeds. And it was, for supersonic flight proved impractical given the societal response to sonic booms, the fuel inefficiency of high-speed transports and the limiting characteristics of materials at high temperatures. The SR-71 may well have been the apogee of hypersonic flight, Aurora rumors not withstanding.

Once those practical limits on airspeed were reached, aerospace innovation did not stop, it simply changed direction. The Boeing 747 and Airbus A380 fly at subsonic speeds, just as the Boeing 707 and 727, but they carry far more passengers and have longer range. Similarly, improvements in jet engines dramatically increased fuel efficiency and reduced maintainence costs. In turn, glass cockpits and electronics improved safety.

Perhaps most of all, changing expectations and airline deregulation democratized air travel, making it economically accessible to almost all, though with all the ambience one associates with riding a prison bus on a hot summer day. (Parenthetically, there are days I regret the loss of amenities as board yet another airplane. I still remember hot meals on flights, and I mourn the loss of legroom.)

The Need for Speed

There is no reason to believe that computing is exempt from the cruel realities of bounded exponentials. The difficulties of continued Dennard scaling and the rise of dark silicon are both challenging our ability to deliver higher performance single and multicore chips. (See Battling Evil: Dark Silicon) Instead, there is an increasing emphasis on function-specific accelerators and devices, hardware-software co-design and lower power designs for embedded use.

In the HPC world, the performance of the systems on the Top500 list has been rising faster than single processor performance, in large measure due to increasing system sizes. This is not, in the long term, a sustainable trend. Similarly, design studies for exascale systems suggest the need for dramatic improvements in energy efficiency (operations/joule), system reliability and programmability.

These may – or may not – be achievable at economically viable price points. The risk of brute force solutions and "research by acquisition" is that we build a Concorde SST, an economic dead end. Instead, we need to conduct the basic research and explore the new ideas in high-performance computing that might solve some of our fundamental problems. After all, studies of hypersonic scramjets continue.

Lest I be called a naysayer or predictor of doom, let me note that there are plenty of other opportunities for innovation in computing in general and in high-performance computing in particular. In addition to pursuing scramjet designs, we need the HPC equivalent of the Boeing 737 – an everyday workhorse that is reliable, efficient, easy to operate and ubiquitous. We must address the infamous excluded middle, making HPC accessible for everyday use by businesses, researchers and governments. There's plenty of room for innovation in algorithms, software tools, programming models, resource management, and energy efficient design.

Remember, no exponential is forever.

Battling Evil: Dark Silicon

As a boy, I remember reading all the superhero comic books, from the standard Superman and Batman to the Green Lantern, Thor, The Flash and Spider-Man. Some superhero stories and backstories were better than others, but all were escapism from small town life. For me, Thor was always a favorite, though I am not quite sure why. Maybe I just liked the head gear! The summer "action movie" season is upon us, and several of these superheros will grace the screen in new Hollywood fantasies, battling the forces of evil.

All of this made me think about the resident evil now facing us in computing – the rise of dark silicon. The very phrase sounds ominious and it is, for I believe it will profoundly reshape how we think about computing in the next decade. But before we talk about the grand denouement where our Justice League of silicon process experts, computer architects and software gurus do battle with dark silicon, perhaps we should begin with a bit of backstory.

Moore's Law and Silicon Scaling

We are all familiar with Moore's Law, namely that the number of transistors in a given area doubles roughly every two years. It is worth noting that Gordon Moore never really claimed this was a law, merely that it was a statistical truth based on observational data and technology trends. It is also interesting how a rather informal 1965 Electronics paper, entitled "Cramming More Components onto Integrated Circuits," had such impact. Over the past thirty years, the "law" has held true due to the hard work and creativity of a great many people, as well as many billions of dollars of investment in process technology and silicon foundries.

For a detailed understanding of why reducing transistor feature sizes really has been possible, I highly recommend reading the original, seminal paper by Robert Dennard, Fritz Gaensslen, Hwa-Nien Yu, V. Leo Rideout, Ernest Bassous and Andre LeBlanc, entitled "Design of ion-implanted MOSFET's with Very Small Physical Dimensions." It appeared in the October 1974 issue of the IEEE Journal of Solid-State Circuits, and it defined a scaling process that has driven the semiconductor industry ever since.

Simply put, Dennard scaling says that reducing the length, width and gate oxide thickness of transistor features by a constant k results in transistors that are k² times smaller, k times faster, and dissipate k² less power. This was Moore's Law made manifest and has let us move from feature sizes measured in micrometers to nanometers. Since the elucidation of Dennard scaling, all has been well in Metropolis, at least until very recently.

Power and Performance: Evil Lurks

As transistor sizes continued to shrink and clock rates continued to increase, we have seen the rise of high-k dielectrics and silicon on insulator (SOI) to address static current leakage. In that spirit, one of my colleagues recently remarked that transistors are now less like solid doors that keep out the cold (current flow) unless open and more like screen doors that always have some air flow (current leakage) even when closed.

Transistor density, clock rates and chip sizes have also made heat dissipation an ever bigger problem. While at Intel, Pat Gelsinger once remarked that unless things changed, chip temperatures would approach that of the sun's surface, an obvious absurdity only possible in the comic books. All of this has led to multicore designs, bounds on clock rates for most microprocessors and decreases in operating voltages.

We have also seen new semiconductor processes and transistor design options to reduce leakage current. (Intel's recent FinFET announcement is one such example.) Although I expect to see transistor feature sizes continue to shrink, with sub-11 nanometer feature sizes achievable, it will not be easy, nor will it be cheap. Lest there be any doubt, the cheap part really matters, for it is the economics that will determine if scaling continues.

These semiconductor challenges have been mirrored by an increasingly empty bag of general purpose architectural tricks. Almost all of the techniques once found in supercomputer processor designs – superpipelining, scoreboarding, and vectorization – are now in mainstream microprocessors. It is not obvious that we can extract more parallelism while maintaining the illusion of sequential execution. In fact, the evidence is substantial that we cannot.

Battling Evil: Dark Silicon

That's the backstory. Now let's turn to the true evil lurking among us, dark silicon. If one considers process scaling trends and limits on heat dissipation, the conclusion is rather obvious. We soon will have (and in many cases already do have) chips with more transistors than can be concurrently activated. The practical implication is that most of the chip silicon will be "dark" (unpowered) most of the time.

If it's dark most of the time, is it of any value? Yes, but only if we embrace our fears. This is the true end of the hegemony of von Neumann computing and the birth of a new era, one based on chips with a diverse set of specialized functional units, accelerators and tailored architectures that each target specific tasks. The mobile world has already learned this lesson, for functional specialization increases energy efficiency (more operations/joule) and allows designers to mix and match the set of functional units to target domains. Arguably, heterogeneous multicore is one, very tentative step in this direction for client and server computing.

This transition to diverse functional units, accelerators and tailored architectures will also challenge many of our assumptions about programming models and system software, as well as the economics of chip design. In this new world, hardware/software co-design becomes de rigeur, with devices and software systems deeply interdependent.

It is a different ecosystem, with different cultures and different processes. It means far less general-purpose performance increases, much more diversity in the hardware, elevation of multivariate optimization – power, performance, reliability – in programming models, new system software resource management challenges and a plethora of new applications.

More generally, with lots of silicon real estate (dark silicon), chip designers and architects have the space to place functional blocks that are used only part of the time. However, at some point (and we are arguably close to that point now), we will have deployed those functional units that cover the common tasks and adding more will have possibly deleterious effects due to larger chip size. That potentially drives the economics to smaller chip sizes, which, given foundry capacity, leads to lots of small chips. That is the collision of the big chip server world with the small chip mobile and Internet of Things world, a battle already raging.

To address these challenges and battle dark silicon, we need new ideas in computer architecture, system software, programming models and end-to-end user experiences. It's an epic struggle for the future of computing. Giants are battling, and not all will survive, nor will the survivors emerge unscathed. Heroes will also emerge, sometimes from unexpected places.

I think I left my hammer, Mjölnir, around here somewhere. Do you feel your spider sense tingling?