Reed's Ruminations: A Blog by Dan Reed

Raspberry Pi? Why Yes, I’ll Have Pi Cluster

The Raspberry Pi, a credit card sized, $35 computer, is a runaway success, exceeding even the wildest expectations of anyone in the hobbyist or educational communities. The latest version contains a 900 MHz quadcore ARM processor, 1 GB of DRAM, multiple USB ports, a 10/100 Ethernet port, an HDMI interface and 40 GPIO pins (for external device interfaces). Permanent storage is via a Micro SD card slot.

Assuming one has either an HDMI-capable digital television or desktop computer display, the only additional requirements are a USB keyboard/mouse and a Micro SD card of at least 8 GB capacity. Chances are most of us have all of these; if not, another $30 suffices for all but the display. Thus configured, the Raspberry Pi can run several variants of Linux and, as well as the forthcoming embedded version of Windows10.

All of this takes me back to a time in the early 1980s, when I walked to graduate school every day (in the snow, uphill both ways), Pac-Man was a state of the art video game, and the Rubik's Cube was bringing recreational group theory to the masses (though they didn't know it). A DEC VAX 11/780 with 2-8 MB of DRAM running the Berkeley Software Distribution (BSD) of UNIX was the de facto standard research environment for academic computer science.

At a time when batch computing and timeshared university mainframes defined academic research, these BSD VAX systems, along with the National Science Foundation (NSF) Coordinated Experimental Research (CER) program and the CSNET academic network, were transformative. It is no exaggeration to say that modern, experimental computer science can be traced directly to this environment. All of which is why Bill Joy once called the 1 MIPS of the VAX 11/780 the "electron volt of computing."

The Raspberry Pi is a more capable system in almost every way than that departmental VAX – faster, with a larger memory and bigger secondary storage system – as well having a richer base of open source software and access to the global Internet. It is a very tangible and inexpensive manifestation of Moore's Law. However, to quote from the movie Airplane!, that's not important right now. (I am serious. And don't call me Shirley.)

What is important is the transformative effect of the Raspberry Pi and related, inexpensive computing devices on both education and innovation. (The Kickstarter-funded Adapteva Parallela multicore processor is another.) Raspberry Pi's now serve as robot controllers, research sensors, web servers, and software development platforms; they are the darling of hobbyists and the maker movement.

However, the creation nearest and dearest to my heart has been the widespread development of Raspberry Pi Beowulf clusters for parallel computing education. These systems use inexpensive Ethernet switches to create systems similar to the Caltech Cosmic Cube, a seminal design for message-based parallel computing created by Chuck Seitz and Geoffrey Fox in the early 1980s.

Although not fast by today's parallel computing standards, these Raspberry Pi clusters allow students to construct complete parallel systems, ranging from small, development clusters (4-8 nodes) to larger (40-120+ nodes) research systems. More importantly, they expose all aspects of hardware and software configuration, bringing experiential parallel computing to a broad new class of students. (The construction recipe for ORNL's Tiny Titan is one of many such examples.) Even more importantly, the clusters let students run real-world parallel applications.

I would have dearly loved to have one of these clusters for parallel computing research back in the early 1980s. At that time, Richard Fujimoto and I were writing Ph.D. theses at UC-Berkeley and Purdue, respectively, about various aspects of message passing machines. We then collaborated on a book about what we then called multicomputer networks. We have come a long, long way since then.

Yes, I will have some Raspberry Pi. In fact, I'll have a cluster.

Exascale Software: Just a Few Orders of Magnitude

N.B. I also write for the Communications of the ACM (CACM). The following essay recently appeared on the CACM blog.

Petascale high-performance computing (HPC) is here, with multiple machines achieving more than ten petaflops on the Linpack (HPL) Top500 benchmark. These achievements have not been without teething problems, as the scale and complexity of the systems have made debugging, acceptance testing and application scaling ever more challenging. Nevertheless, these systems are now operational, and they are being used for scientific research and national security applications.

In many ways, it amazing that a decade ago we celebrated crossing the terascale threshold. When I christened the NSF Distributed Terascale Facility (DTF) as the TeraGrid in 2002, Ian Foster asked me if we should worry about embedding a performance level in a facility name. I responded that I was confident the capabilities and the name would evolve, and they have.

Planning is now underway for exascale computing, though both the depth of the technical challenges and the straitened economics of research funding have slowed progress. For detailed background on the technical challenges, I heartily recommend reading the 2008-2009 DARPA exascale hardware and software studies, chaired by Peter Kogge and Vivek Sarkar, respectively. Although some of the details have changed in the interim, the key findings are still relevant. (In full disclosure, I was one of several co-authors of the exascale software study.)

Among a plethora of design challenges highlighted by these two reports, three are especially relevant when considered with respect to petascale systems:

• Substantially reduced memory per floating point operation (i.e., reduced memory per processor core due to energy constraints)
• Dramatically higher energy efficiency per floating point operation with minimal data movement, given the high time and energy cost of off-chip data accesses.
• Frequent component failures, given the sheer number of chips required to reach the exascale performance target

Just a Few Orders of Magnitude

All currently envisioned exascale systems would require parallelism at unprecedented scale, and barring new, energy efficient memory technologies; they would be memory starved relative to current systems, even under a 20 MW system design point; and multilevel fault tolerance would be required to achieve acceptable systemic mean time to failure (MTBF). Extraordinary parallelism, unprecedented data locality and adaptive resilience: these are daunting architecture, system software and application challenges for exascale computing.

If we have learned anything in sixty years of software and hardware development, it is that orders of magnitude matter, whether in latencies and access times, bandwidths and capacities, software scale and complexity, or level of parallelism. From file system metadata bottlenecks when opening thousands of files to application performance losses from operating system jitter due to daemon activity, every order of magnitude brings new challenges. Only the naïve or inexperienced believe one can scale any computer system design by factors of ten without exposing unexpected issues.

Knowns and Unknowns

What can we expect at exascale? As always, there are the known knowns, the known unknowns and the unknown unknowns, to use a Rumsfeld phrase. The knowns, of both kinds, include the ever-present issues of scale and locality. Will variants of current scheduling and resource management techniques be effective and usable by application developers? Will the complexity of multilevel memory management, heterogeneous multicore and dark silicon shrink the cadre of ultra-high performance application developers even further, perhaps below a technically and politically viable threshold?

The unknowns are more deep and subtle. How can energy optimization be elevated to parity with performance optimization, both statically and dynamically. The dynamic aspect is crucial, as the hysteresis of thermal dissipation in dark silicon affects chip lifetimes. Equally importantly, how can energy usage be related to code in ways that highlight optimization choices? This is the energy analog of performance measurement and guidance for optimized code, where measurements must be related to the original code in ways that are meaningful and amenable to change.

Finally, there are important open questions about the future of operating system structures themselves. The fundamental lesson of cloud computing – the nearest equivalent in scale – is the importance of weak consistency and loose coordination. Given projected exascale communication costs (energy and time), and frequent component failures, might federated rather than synchronized operation be preferable? Is it time to revisit some of our most cherished HPC assumptions and imagine operating system structures and programming models not based on Linux variants, MPI and OpenMP?

Evolution or Revolution

The exascale hardware and software challenges are real. Do we pursue incremental extensions of current practices or step back and explore more radical and fundamental options? Each has different advantages and disadvantages, which suggests we should probably pursue both, recognizing the costs. To be sustainable, an exascale research and development program must lead to cost effective and usable systems that are an integral part of the mainstream of semiconductor and software industries.

Related articles

Bill Gropp: We Need to Stop Talking Just About Exascale

Data Movement At Exascale

2012 Cray, Kennedy and Fernbach Awards

This year, I again have the honor and privilege to chair the selection committee for the Seymour Cray and Sidney Fernbach awards. I will also be hosting the presentation ceremony at SC12 in Salt Lake City for these two awards, along with the Ken Kennedy award. (My former colleague at the University of Illinois, Professor David Padua, chaired the Kennedy award selection committee.)

These eponymously named awards recognize truly outstanding contributions to high-performance computing, in honor of three leaders of our field. The Seymour Cray award recognizes individuals whose innovative contributions to high performance computing systems best exemplify the creative spirit demonstrated by Seymour Cray. In turn, the Sidney Fernbach award is given to for outstanding contribution in the application of high-performance computers using innovative approaches. Finally, the Ken Kennedy award recognizes outstanding contributions to programmability or productivity in high-performance computing, together with significant community service or mentoring contributions.

I am delighted that the 2012 Seymour Cray Award will be presented to Peter Kogge, who is currently a professor at the University of Notre Dame. Early in his career, Peter developed parallel algorithms for recurrence solutions and parallel prefix operations, resulting in what is now known as the Kogge-Stone adder, the gold standard for fast addition. While at IBM, he also developed a series of HPC systems for military and space applications, including some that were key elements of the U.S. space shuttle. For many people, that would define a career. However, Peter has also been a pioneer in multicore and multithreaded processor design; he was also one of the creators of what we now call "Processor in Memory" (PIM) computing. Today, he is a leader in exascale initiatives. 

I am equally pleased that Professors Laxmikant "Sanjay" Kale and Klaus Schulten, developers of the NAMD biomolecular simulation software, are joint recipients of the 2012 Sidney Fernbach Award. Their collaboration combines the best of computer science and computational science. Utilizing Kale's insights in fine grained parallelism and load balancing, embodied in the CHARM++ library, and Schulten's insights into biophysics, NAMD transformed high-speed molecular dynamics simulations, allowing the motions of large biological systems to be simulated for longer periods of biological time than were previously feasible.

The third of these awards, the Ken Kennedy award, recognizes Professor Mary Lou Soffa for "contributions to compiler technology and software engineering, exemplary service to the profession, and lifelong dedication to mentoring and improving diversity in computing." As the award citation notes, her work spans optimizing compilers and software engineering. Among her many contributions are key insights into quantifying the value of code optimizations within a coherent framework and on register optimization. In addition to seminal contributions to compiler technology, Professor Soffa has also been instrumental in mentoring women in computing and raising awareness of the need for our culture to be more inclusive and supportive.

My thanks to Satoshi Matsuoka for the photographs of the plenary program award presentations.

The IT Ecosystem and Global Competitiveness

I am a member of the U.S. National Academies Board on Global Science and Technology (BGST). As the name suggests, the role of BGST is to examine the shifting nature of the global science and technology enterprise and its implications. These include the global flow of intellectual talent and capital, the interplay of government policies, research and development priorities, innovation and technology transfer, and global competitiveness and security. This is a wide-ranging mandate, which is both exciting and challenging.

As a member of BGST, I recently chaired a study, commissioned by the U.S. Department of Defense, on the issues surrounding the end of Dennard scaling and its implications for U.S. industry, defense capabilities and national security. The report, The New Global Ecosystem in Advanced Computing: Implications for U.S. Competitiveness and National Security, was just released. It is a cautionary tale about the Gordian knot of intellectual and economic interdependencies, of epochal ecosystem shifts, and the global consequences of both.

Lest this seem mere academic punditry, remember that 30-40 percent of U.S. net economic growth in recent decades has been due to advances in information technology. This is not just a tale of Silicon Valley, but also one for the entire country's economy.

Beyond Dennard Scaling

Moore's Law, the notion that the number of transistors in a given silicon area doubles roughly every two years, is not a law or even a theorem. Rather, it was an empirical observation, originally made by Gordon Moore in 1965. For over forty years, it has continued to hold true by virtue of enormous intellectual effort, ongoing architectural and software innovation, and billions of dollars of investments in process technology and silicon foundries. In turn, consumers, businesses and governments have been the happy beneficiaries of faster microprocessors, more powerful, inexpensive and ubiquitous computing devices and a rich and varied suite of software applications.

However, there is bad news. The continued and seemingly miraculous improvement in general processor performance is now over, a consequence of physical limits on transistor shrinkage that was itself a happy consequent of Dennard scaling. To be clear, Moore's law continues, with the number of transistors on a chip continuing to double, but the transistors no longer become more energy efficient as they shrink. The result has been bounds on microprocessor clock rates due to energy dissipation constraints and the concomitant rise of multicore chips and function-specific accelerators such as GPUs. (See Battling Evil: Dark Silicon and Nothing Is Forever for a few reflections and details.)

This radical shift breaks a virtuous cycle of mutually reinforcing benefits, one where software developers created feature-filled applications and stimulated demand for faster general-purpose processors, which then drove the creation of even more advanced applications. Simply put, we are the reluctant, wide-eyed denizens of a brave new world, one where the cherished and popular expectation that applications would run faster without change each time a new backward-compatible processor appeared.

There is a technical way forward, but it means embracing application parallelism and retargeting software to each new generation of non-compatible, heterogeneous multicore processors. As over fifty years of research in parallel computing has shown, this is a path fraught with pain and difficulty. In turn, this has profound implications for the future of the silicon ecosystem and the nature and locus of continued innovation. It is the trillion-dollar inflection point, pivoting on chip performance, business models and global markets.

Ecosystem Implications    

The end of Dennard scaling and the emergence of heterogeneous multicore processors has coincided with another shift, the transition from an era dominated by PCs to one defined by smartphones and tablets. For much of the world, the smartphone is now the primary computing system, and in developing economies, the aspirational feature phone is the only computing device. More to the point, the majority of PC and smartphone users are not in the U.S., nor will they ever be again. Not surprisingly, these two phenomena, the end of Dennard scaling and the rise of smartphones and tablets are deeply interrelated.

The PC ecosystem has long been driven by the phenomenal success of the x86 microprocessor family and successive generations of processors from Intel and AMD, both U.S. companies. Conversely, the smartphone and tablet ecosystem is largely based on the ARM microprocessor family and low power systems-on-a-chip (SoCs) developed by ARM licensees around the world. Beyond the ongoing competition between PC and smartphone vendors, this is a battle of business models, pitting a closed x86 ecosystem with captive silicon foundries against fabless semiconductor design firms that mix and match function-specific accelerators with ARM cores and use Taiwan Semiconductor Manufacturing Company (TSMC) as a foundry.

Enormous resources are being invested in both silicon ecosystems, with x86 designers seeking to "grow down" by reducing power and integrating functionality on chip to compete in the smartphone and tablet market. Conversely, ARM designers are seeking to "grow up" by increasing performance and adding features to compete with x86 designs in the server market. This battle royal is not winner take all. Rather, it is a competition to define the global nexus of innovation, with profound implications for global IT dominance in the next decade.

Global Competitiveness and BGST

It is with this backdrop that the BGST committee examined the technical consequences from the end of Dennard scaling, the cultural and economic challenges of parallelism, the possible shifts of capital and talent, and national and regional investments in IT research. The report's broad conclusions include the following: (a) the U.S. still leads in basic IT research, but the global gap continues to shrink, (b) IT investment strategies and challenges differ markedly across countries and regions, (c) single chip performance is unlikely to continue as the predominate focus of innovation, (d) there are serious risks that growing international markets will diminish U.S. influence and (e) U.S. national security and defense readiness depends on continued rapid uptake and deployment of advanced IT.

I encourage you to download and read the complete BGST report for additional details and insights.

In IT and innovation circles, Gordon Moore is also famous for another dictum, "only the paranoid survive." What he really said is more nuanced and relevant to the global innovation competition, "Success breeds complacency. Complacency breeds failure. Only the paranoid survive." It is worth pondering as one considers the interplay of science and technology, economics, government policy, and business models.

Acknowledgments

I would be remiss if I did not express my sincere thanks to the members of the BGST report committee: Cong Gao (Nottingham), Tai Ming Cheung (UCSD), John Crawford (Intel), Dieter Ernst (East-West Center), Mark Hill (Wisconsin), Steve Keckler (NVidia), David Liddle (U.S. Venture Partners), and Kathryn McKinley (Microsoft). There were thoughtful, dedicated and indefatigable. Finally, all of the committee members are deeply indebted to Bill Berry, Ethan Chiang and Lynette Millett from the National Academies.

Analog Computing: Time for a Comeback?

N.B. I also write for the Communications of the ACM (CACM). The following essay recently appeared on the CACM blog.

In the early dates of the automobile, there was a lively competition among disparate technologies for hegemony as the motive power source. Steam engines were common, given their history in manufacturing and locomotives, and electric vehicles trundled through the streets of many cities. The supremacy of the internal combustion engine as the de facto power source was by no means an early certainty. Yet it triumphed due to a combination of range, reliability, cost and safety, relegating other technologies to historical curiosities.

Thus, it is ironic that we are now assiduously re-exploring several of these same alternative power sources to reduce carbon emissions and dependence on dwindling global petroleum reserves. Today's hybrid and electric vehicles embody 21^st century versions of some very old ideas.

There are certain parallels to the phylogenic recapitulation of the automobile now occurring in computing. Perhaps it is time to revisit some old ideas.

Analog History

Use of the word "computer" conjures certain images and brings certain assumptions. One of them, so deeply ingrained that we rarely question it, is that computing is digital and electronic. Yet there was a time not so long ago when those adjectives were neither readily assumed nor implied when discussing computing, just as the internal combustion engine was not de rigueur in automobile design.

The alternative to digital computing – analog computing – has a long and illustrious history. Its antecedents lie in every mechanical device built to solve some problem in a repeatable way, from the sundial to the astrolabe. Without doubt, analog computing found its apotheosis in the slide rule, which dominated science and engineering calculations for multiple centuries, coexisting and thriving alongside the late comer, digital computing.

The attraction of analog computing has always been its ability to accommodate uncertainty and continuity. As Cantor showed, the real numbers are non-countably infinite, and their discretization in a floating-point representation is fraught with difficulty. Because of this, the IEEE floating-point standard is a delicate and ingenious balance between range and precision.

All experimental measurements have uncertainty, and quantifying that uncertainty and its propagation in digital computing models is part of the rich history of numerical analysis. Forward error propagation models, condition numbers, and stiffness are all attributes of this uncertainty and continuity.

Hybrid Futures

I raise the issue of analog computing because we face some deep and substantive challenges in wringing more performance from sequential execution and the von Neumann architecture model of digital computing. Multicore architectures, limits on chip power, near threshold voltage computation, functional heterogeneity and the rise of dark silicon are forcing us to confront fundamental design questions. Might analog computing and sub-threshold computing bring some new design flexibility and optimization opportunities?

We face an equally daunting set of challenges in scientific and technical computing at very large scale. For exascale computing, reliability, resilience, numerical stability and confidence can be problematic when input uncertainties can propagate, and single and multiple bit upsets can disturb numerical representations. How can we best assess the stability and error ranges on exascale computations? Could analog computing play a role?

Please note that I am not advocating a return to slide rules or pneumatic computing systems. Rather, I am suggesting that we step back and remember that the evolution of technologies brings new opportunities to revisit old assumptions. Hybrid computing may be one possible way to address the challenges we face on the intersecting frontiers of device physics, computer architecture and software.

A brave new world is aborning. Might there be a hybrid computer in your hybrid vehicle?

Surrounded By Opportunities

Perhaps the closest I have approached the feeling of intellectual omnipotence (and it has not been often) was when I was a graduate student, teaching undergraduate FORTRAN programming for science and engineering students. Those were the days of punched cards, batch processing and line printer output. It was a time when dinosaurs roamed the earth, my slide rule was still a fond memory, and we were computing on massive machines with less performance than my smartphone has today. (Ignore that part about the dinosaurs, as it was a wee bit earlier.)

During my office hours, students would troop through for help with their programming assignments, bringing program printouts. These printouts began with a "day file" showing the timeline of compilation and execution steps (JCL steps on IBM S/360), followed by the program and then any output. After having seen a few students, you realize that elementary programming errors have just a few equivalence classes, often identifiable from the day file alone. I took great pride in listening to the student description of the problem, predicting the root cause and then immediately locating it on the printout. The student was inevitably amazed. Yes, it was a cheap parlor trick, but as a graduate student, you take gratification where you can find it.

All of this reminds me that sequential programming remains difficult and parallel programming even more difficult, despite all our efforts to tame the unruly beast with syntactic sugar, specification and scripting languages, libraries and tools. I am no expert in parallel programming language design, nor am I a cognitive psychologist, but like many of my readers, I have pulled a few all-nighters writing code. Perhaps that at least qualifies me to have an opinion.

In a world of parallel programming, multicore architectures, dark silicon, and rising software complexity, we face some difficult choices. It's time to learn from the past.

Fading Abstractions

We have been fortunate to date. Although sequential programming at scale is intellectually challenging, we have been riding the wave of ever-increasing processor performance, made possible by semiconductor advances. In turn, these were based on an abstraction – the Mead-Conway design rules –that largely separated device physics from microprocessor architecture. Finally, the von Neumann abstraction has also served us well, allowing decades of software development atop an easily understood model of sequential instruction execution.

Our run of good luck is over. Put another way, we are surrounded by opportunities! Parallelism is now the norm for almost all software developers, not just those of us in high-performance computing. More perniciously, heterogeneous parallelism is here, with disparate cores, accelerators and functional units. To manage dark silicon and exploit continuing transistor density increases, this trend will only accelerate, with a dizzying array of heterogeneous chips and devices challenging code portability and performance tuning. Finally and most worrisome, our device physics abstractions are breaking down, with device physics affecting architecture. We face real silicon reliability, process variation and energy challenges, and near threshold voltage computations emerging.

Holistic Collaboration

The abstractions of device physics, processor architecture and programming models naturally led to a corresponding intellectual specialization. Consequently, those working in materials science and device physics have little in common with those designing software tools. I believe that must change if we are to address our challenges.

We need new, integrated design methodologies and rich collaborations that consider end-to-end design and capability. In many ways, this is a return to the past, necessitated by the desire for continuing increases in performance, mediated by the desire for reliability, scalability, programmability and economic feasibility. It will not be easy, but it is a way forward.

Nothing Lasts Forever

Those of us in computing and especially those of us in high-performance computing (HPC) have been received Pavlovian conditioning to believe that computers will always be faster if we just wait eighteen to twenty-four months. (Excuse me while I wipe the saliva off my chin, triggered by thinking about this.) It seems this trend has ever been so and ipso facto, so shall it ever be.

Unfortunately, no exponential continues forever, at least outside the mathematics textbooks. All technology-based advances are ultimately limited by something, usually some physical or economic limit. Innovation then shifts to other metrics, against which advances can be measured and valued. Computing is no exception.

Leavin' On A Jet Plane

In the heady days of aviation, when world speed records were set every year, it must have seemed that the sky was literally the limit on achievable airspeeds. And it was, for supersonic flight proved impractical given the societal response to sonic booms, the fuel inefficiency of high-speed transports and the limiting characteristics of materials at high temperatures. The SR-71 may well have been the apogee of hypersonic flight, Aurora rumors not withstanding.

Once those practical limits on airspeed were reached, aerospace innovation did not stop, it simply changed direction. The Boeing 747 and Airbus A380 fly at subsonic speeds, just as the Boeing 707 and 727, but they carry far more passengers and have longer range. Similarly, improvements in jet engines dramatically increased fuel efficiency and reduced maintainence costs. In turn, glass cockpits and electronics improved safety.

Perhaps most of all, changing expectations and airline deregulation democratized air travel, making it economically accessible to almost all, though with all the ambience one associates with riding a prison bus on a hot summer day. (Parenthetically, there are days I regret the loss of amenities as board yet another airplane. I still remember hot meals on flights, and I mourn the loss of legroom.)

The Need for Speed

There is no reason to believe that computing is exempt from the cruel realities of bounded exponentials. The difficulties of continued Dennard scaling and the rise of dark silicon are both challenging our ability to deliver higher performance single and multicore chips. (See Battling Evil: Dark Silicon) Instead, there is an increasing emphasis on function-specific accelerators and devices, hardware-software co-design and lower power designs for embedded use.

In the HPC world, the performance of the systems on the Top500 list has been rising faster than single processor performance, in large measure due to increasing system sizes. This is not, in the long term, a sustainable trend. Similarly, design studies for exascale systems suggest the need for dramatic improvements in energy efficiency (operations/joule), system reliability and programmability.

These may – or may not – be achievable at economically viable price points. The risk of brute force solutions and "research by acquisition" is that we build a Concorde SST, an economic dead end. Instead, we need to conduct the basic research and explore the new ideas in high-performance computing that might solve some of our fundamental problems. After all, studies of hypersonic scramjets continue.

Lest I be called a naysayer or predictor of doom, let me note that there are plenty of other opportunities for innovation in computing in general and in high-performance computing in particular. In addition to pursuing scramjet designs, we need the HPC equivalent of the Boeing 737 – an everyday workhorse that is reliable, efficient, easy to operate and ubiquitous. We must address the infamous excluded middle, making HPC accessible for everyday use by businesses, researchers and governments. There's plenty of room for innovation in algorithms, software tools, programming models, resource management, and energy efficient design.

Remember, no exponential is forever.

Battling Evil: Dark Silicon

As a boy, I remember reading all the superhero comic books, from the standard Superman and Batman to the Green Lantern, Thor, The Flash and Spider-Man. Some superhero stories and backstories were better than others, but all were escapism from small town life. For me, Thor was always a favorite, though I am not quite sure why. Maybe I just liked the head gear! The summer "action movie" season is upon us, and several of these superheros will grace the screen in new Hollywood fantasies, battling the forces of evil.

All of this made me think about the resident evil now facing us in computing – the rise of dark silicon. The very phrase sounds ominious and it is, for I believe it will profoundly reshape how we think about computing in the next decade. But before we talk about the grand denouement where our Justice League of silicon process experts, computer architects and software gurus do battle with dark silicon, perhaps we should begin with a bit of backstory.

Moore's Law and Silicon Scaling

We are all familiar with Moore's Law, namely that the number of transistors in a given area doubles roughly every two years. It is worth noting that Gordon Moore never really claimed this was a law, merely that it was a statistical truth based on observational data and technology trends. It is also interesting how a rather informal 1965 Electronics paper, entitled "Cramming More Components onto Integrated Circuits," had such impact. Over the past thirty years, the "law" has held true due to the hard work and creativity of a great many people, as well as many billions of dollars of investment in process technology and silicon foundries.

For a detailed understanding of why reducing transistor feature sizes really has been possible, I highly recommend reading the original, seminal paper by Robert Dennard, Fritz Gaensslen, Hwa-Nien Yu, V. Leo Rideout, Ernest Bassous and Andre LeBlanc, entitled "Design of ion-implanted MOSFET's with Very Small Physical Dimensions." It appeared in the October 1974 issue of the IEEE Journal of Solid-State Circuits, and it defined a scaling process that has driven the semiconductor industry ever since.

Simply put, Dennard scaling says that reducing the length, width and gate oxide thickness of transistor features by a constant k results in transistors that are k² times smaller, k times faster, and dissipate k² less power. This was Moore's Law made manifest and has let us move from feature sizes measured in micrometers to nanometers. Since the elucidation of Dennard scaling, all has been well in Metropolis, at least until very recently.

Power and Performance: Evil Lurks

As transistor sizes continued to shrink and clock rates continued to increase, we have seen the rise of high-k dielectrics and silicon on insulator (SOI) to address static current leakage. In that spirit, one of my colleagues recently remarked that transistors are now less like solid doors that keep out the cold (current flow) unless open and more like screen doors that always have some air flow (current leakage) even when closed.

Transistor density, clock rates and chip sizes have also made heat dissipation an ever bigger problem. While at Intel, Pat Gelsinger once remarked that unless things changed, chip temperatures would approach that of the sun's surface, an obvious absurdity only possible in the comic books. All of this has led to multicore designs, bounds on clock rates for most microprocessors and decreases in operating voltages.

We have also seen new semiconductor processes and transistor design options to reduce leakage current. (Intel's recent FinFET announcement is one such example.) Although I expect to see transistor feature sizes continue to shrink, with sub-11 nanometer feature sizes achievable, it will not be easy, nor will it be cheap. Lest there be any doubt, the cheap part really matters, for it is the economics that will determine if scaling continues.

These semiconductor challenges have been mirrored by an increasingly empty bag of general purpose architectural tricks. Almost all of the techniques once found in supercomputer processor designs – superpipelining, scoreboarding, and vectorization – are now in mainstream microprocessors. It is not obvious that we can extract more parallelism while maintaining the illusion of sequential execution. In fact, the evidence is substantial that we cannot.

Battling Evil: Dark Silicon

That's the backstory. Now let's turn to the true evil lurking among us, dark silicon. If one considers process scaling trends and limits on heat dissipation, the conclusion is rather obvious. We soon will have (and in many cases already do have) chips with more transistors than can be concurrently activated. The practical implication is that most of the chip silicon will be "dark" (unpowered) most of the time.

If it's dark most of the time, is it of any value? Yes, but only if we embrace our fears. This is the true end of the hegemony of von Neumann computing and the birth of a new era, one based on chips with a diverse set of specialized functional units, accelerators and tailored architectures that each target specific tasks. The mobile world has already learned this lesson, for functional specialization increases energy efficiency (more operations/joule) and allows designers to mix and match the set of functional units to target domains. Arguably, heterogeneous multicore is one, very tentative step in this direction for client and server computing.

This transition to diverse functional units, accelerators and tailored architectures will also challenge many of our assumptions about programming models and system software, as well as the economics of chip design. In this new world, hardware/software co-design becomes de rigeur, with devices and software systems deeply interdependent.

It is a different ecosystem, with different cultures and different processes. It means far less general-purpose performance increases, much more diversity in the hardware, elevation of multivariate optimization – power, performance, reliability – in programming models, new system software resource management challenges and a plethora of new applications.

More generally, with lots of silicon real estate (dark silicon), chip designers and architects have the space to place functional blocks that are used only part of the time. However, at some point (and we are arguably close to that point now), we will have deployed those functional units that cover the common tasks and adding more will have possibly deleterious effects due to larger chip size. That potentially drives the economics to smaller chip sizes, which, given foundry capacity, leads to lots of small chips. That is the collision of the big chip server world with the small chip mobile and Internet of Things world, a battle already raging.

To address these challenges and battle dark silicon, we need new ideas in computer architecture, system software, programming models and end-to-end user experiences. It's an epic struggle for the future of computing. Giants are battling, and not all will survive, nor will the survivors emerge unscathed. Heroes will also emerge, sometimes from unexpected places.

I think I left my hammer, Mjölnir, around here somewhere. Do you feel your spider sense tingling?

SC09 Reflections

SC09 (aka the "Supercomputing Conference") has ended, and my tired feet and aching back are now recovering, albeit slowly. I am confident that tryptophan-induced sleep from massive turkey ingestion, as part of the annual U.S. Thanksgiving ritual, will abet my recovery. But I digress …

The SC09 conference set an attendance record this year – roughly 10,000 attendees at the combined conference and tradeshow – despite the economic malaise of the technology industry and the global economy. One suspects the strong resilience of the conference may be due in part to substantial government investments in very high-performance computing (HPC). The recent announcement that a Japanese government panel recommended reducing investments in ultrascale computing may or may not be a harbinger of things to come.

The Need for Speed …

As always, a new Top500 list was revealed, and exascale computing was (literally) a hot topic, as multiple groups discussed the energy and reliability challenges inherent in building exascale systems. Jack Dongarra and collaborators summarized the results of the International Exascale Software Project (IESP), including a putative roadmap produced following the most recent workshop. A group of national laboratories and universities also announced the Hybrid Multicore Consortium (HMC), whose goal is to "address the migration of existing applications to accelerator-based systems and thereby maximize the investments in these systems."

More generally, GPUs for general purpose scientific computing were a major topic, both on the show floor, where NVIDIA's booth had standing room only presentations and in workshops and paper presentations. The real challenge of exploiting heterogeneous manycore systems lies not in achieving high performance – we know how to do that – but in creating sustainable software infrastructures and tools that embody good software engineering and maintainability principles. The hardware space is in rapid flux, as are the programming models and tools, and applications live far longer than hardware platforms.

I would be remiss if I did not point out that simplifying infrastructure and ensuring resilience and reliability at scale are some of the potential benefits of cloud computing. Clouds were also a big topic of discussion at the conference, with two cloud tutorials attracting large audiences. My colleagues and research collaborators, Rich Wolski (UCSB/Eucalyptus), Dennis Gannon (Microsoft Research) and Roger Barga (Microsoft Research) presented one of these, comparing and contrasting Amazon AWS, Eucalyptus, Windows Azure and Google App Engine as examples of infrastructure as a service (IaaS), Platform as a Service (PaaS) and Applications as a Service (AaaS).

… While Being Visual, Healthy and Green

Dona Crawford (LLNL), Wilf Pinfold (Intel and SC09 conference chair) and I recruited the panel of plenary and keynote speakers: Justin Rattner (Intel CTO), Lee Hood (Institute for Systems Biology) and former U.S. Vice President Al Gore, who gave the conference keynote. (Dona deserves full credit for landing al Gore, as she worked tirelessly to negotiate terms, processes and topics.)

Justin showed an early version of Intel's Larrabee graphics chip, itself another form of GPU, albeit with multiple cores executing the x86 instruction set. Lee highlighted the challenges and opportunities in personalized medicine, based on blood-based diagnostics. Finally, Al Gore's conference keynote wove comments about high-performance computing throughout his climate change talk. In small group conversation, I was also impressed by the former Vice President's perceptive questions about computational modeling, complexity theory and HPC applications.

Recognitions

As Cray/Fernbach awards chair, I was delighted to participate in the presentation of the Seymour Cray award to my friend Kenichi Miura and the Sidney Fernbach award to Roberto Car and Michele Parrinello. All three have made seminal contributions to high-performance computing. Ken talked at length about his experiences as a graduate student in the ILLIAC IV project at Illinois and how it shaped his research, reminding me of the enduring legacy of visionary projects. (See Luck is a Fickle Friend) In addition, I was part of the committee that selected my friend and long-term collaborator, Fran Berman, to receive the first Ken Kennedy award. All of this reminded me how interconnected the world of high-performance computing really is.

That's the news from Lake Wobegon, er SC09. It's time for some turkey.

Language Shapes Behavior: Our Poor Cousin, Data

The Sapir-Whorf Hypothesis holds that language and culture shape behavior and thought. The idea has a long, rich and sometimes controversial history, though recent cognitive linguistic research, together with brain imaging, has lent it new credence. Lately, I have been reflecting on the nature of linguistic relativity in scientific research, particularly for HPC and cloud computing. In particular, how has our focus on floating point operations per second (FLOPS) shaped discourse, science politics and outcomes?

Look, It's Terascale!

If I were to remark, "That is a terascale system," you would likely interpret my comment as an assessment of the system's computing performance, more specifically its floating point capabilities. For the cognoscenti, the sentence likely engenders visions of multicore systems with out of order instruction issue and deep pipelining, small clusters with fast interconnects and (perhaps) some accelerators such as GPUs. You might even reflect on the historical evolution of high-performance computing, when a terascale system was ranked first on the Top500 list about a decade ago.

Given my remark, you would be unlikely to eye your desktop computer with its terascale (terabyte) commodity disk drive. Yet, based on its storage capacity, that desktop is just as surely a terascale platform as is the compute cluster. Ah, you say, I have mixed capacity (bytes) with capability (operations). Terabyte per second storage systems are rare, you say. Absolutely true, yet as this bit of third rate linguistic legerdemain illustrates, language and, perhaps more importantly, connotation really does trump denotation in our discourse.

Whether Petascale and Exascale?

Today, the world's fastest computing platforms have broken the one petaflop mark on the high-performance Linpack (HPL) benchmark, and planning groups across the United States, Europe and Asia are debating political and technical approaches for the construction of exascale computing systems. I cannot help but wonder, though, why we in technical computing talk so little about petascale and exascale data platforms? Why is there no international initiative to build a low latency, high bandwidth (many terabytes/second to even a petabyte/second) data analysis engine with multiple exabytes of storage capacity?

I suspect the reason lies in our background and linguistic heritage. Most of us in high-performance computing came from mathematical backgrounds, where success was defined by a proof and an equation and their embodiment in a computational model, rather than by insight gleaned from large-scale data analysis. I believe it is time we found a lingua franca that bridges FLOPS and bytes and reclaim the full connotation of exascale.

I Want My Exabytes

It is worth remembering that the explosive growth of observational data is itself largely a product of inexpensive CMOS sensors, based on the same semiconductor technologies that begot microprocessors. The examples of high resolution instruments are legion, from the Large Hadron Collider (LHC) and its international hierarchy of data archives to the proposed Square Kilometer Array (SKA) radio telescope, which may produce as much as an exabyte of data every few days. This data tsunami is not limited to the physical sciences; the biological and social sciences are being inundated as well.

One of the major lessons from web search and cloud data centers is the power of truly massive scale, near real-time data analysis. When anyone with a cheap cell phone and a web browser can extract data and insights from a non-trivial fraction of the human knowledge base, behavior and culture are transformed. I would like to believe that we can bring the same data-driven analytics to scientific research as are routinely exploited to find a good lasagna recipe. We need a balanced exascale initiative, in every sense of the word, lest we be a technical confirmation of the Sapir-Whorf hypothesis.