Reed's Ruminations: A Blog by Dan Reed

Paucity to Plethora: Jevon’s Paradox

N.B. This essay, in the same spirit as my previous post, appears in the Communications of the ACM blog.

Those of us of a certain age (i.e., once able to use a slide rule) remember when the university computer (note the singular) was a scientific and engineering shrine, protected by computer operators and secure doors. We acolytes extended offerings of FORTRAN, ALGOL or COBOL via punched card decks, hoping for the blessings that accrued from a syntactically correct program that compiled and executed correctly.

The commonality across all our experiences was the need to husband computer time and plan job submissions carefully, particularly when one's job might wait in the queue for six to ten hours before entering execution. I distinctly remember spending many evenings laboriously examining my latest printout, identifying each syntax error and tracing the program flow to identify as many logic errors as possible before returning to the keypunch room to create a new punched card deck.

Because computing time was scarce and expensive, we devoted considerable human effort to manual debugging and optimization. (The subject of manual memory overlays before virtual memory shall remain for another day.) Today, of course, my wristwatch contains roughly as much computing power as that vintage university mainframe, and we routinely devote inexpensive computing time to minimize human labor. Or do we?

Yes, we routinely use WIMP interfaces for human-computer interaction, cellular telephony is ubiquitous and embedded computers enhance everyday objects – from microwave ovens to thermostats and running shoes. However, I suspect much of computing is still socially conditioned by its roots in computational paucity to recognize fully the true opportunity afforded by computational plethora.

Many of us are still wed to a stimulus-response model of computing, where humans provide the stimulus and computers respond in preprogrammed ways. For example, traditional web search (traditional indeed – how quickly the new becomes commonplace) requires typed or spoken search terms to initiate a search. In a world of plethora, computing could glean work, personal, and even emotional context, anticipating information queries and computing on behalf rather than in response. My computer could truly become my assistant.

In economics, the Jevon's paradox posits that a technological increase in the efficiency with which a resource can be used stimulates greater consumption of the resource. So it is with computing. I believe we are just at the cusp the social change made possible by our technological shift from computational paucity to computational plethora.

Computing Invisibility

N.B. This essay will appear in the March 2010 issue of the CRA newsletter.

Computer science, computer engineering, information technology, informatics, computing and a host of other terms: we have used them all to denote this wonderfully fascinating and diverse thing we do. We have debated connotation and denotation as we seek a clear and compelling exegesis of our field. In so doing, I suspect we have occasionally lost sight of one key aspect, namely the importance of invisibility. What follows is a serious but whimsical look at invisibility's power.

Technological Innovation

If you have read any comic books or watched movies based on them, you know that each superhero has a backstory, a set of superpowers and a stylized routine. As our technologies have evolved, so too have expectations for our fantasy superheroes and their behaviors. After all, technology evolution affects your favorite superhero just as it does you, albeit with slight twists.

In a world of observation satellites, global positioning systems and MEMS-based sensors, it is extraordinarily difficult to maintain a secret hideout. Speech recognition and computer vision systems, along with deep data mining and machine learning, now compromise secret identities. One cannot even step into a nearby telephone booth for a superhero costume change.

Not only did the now ubiquitous cellular telephone render the pay telephone obsolete, it changed everyone's expectations for mobile accessibility. As an early adopter, I once drew a surprised and envious airport crowd when I opened my cellphone to rebook a flight after a cancellation. In years past, all of us would have sought a customer service agent or a payphone to call the airline. Today, we expect wireless communication access almost anywhere the world. Such is the nature of exponential technological change and its cultural impact.

Invisible Technologies

In any enumeration of fantasy superpowers, flight, invulnerability and superstrength rank high, along with invisibility. Like superheroes, successful technologies also become invisible. As technologies mature, market penetration rises, cultural expectations shift and consumer knowledge of the underlying theory and practice generally decline.

No longer need one understand Maxwell's equations, the propagation properties of the Heaviside layer or superheterodyne receiver design to listen to the radio. Nor does one need a deep knowledge of thermodynamics, catalysis theory or electronic fuel injection to drive a car. (A wag might suggest that such a requirement would reduce traffic accidents by severely limiting the number of licensed drivers. A technology realist would look forward to the commercial advent of autonomous navigation systems.)

Today, most computer users know nothing of the halting problem, superscalar pipeline design or object oriented programming. This is success, for it allows all but the technology experts to focus on the raison d'être for computing or any other technology – enabling and expanding the human experience, often in new and unexpected ways.

In that spirit, I humbly suggest that the word computing is often an intellectual blinder that limits our imagination of the possible. Ask anyone how many computers they own, and the instinctive answer is probably a small, positive integer. In truth, most of us own hundreds, if not thousands of invisible computers. They are simply embedded in everyday devices – electronic thermostats, home appliances, portable music players, hearing aids and pacemakers, and even running shoes. You probably drive a mobile computing platform to work each day, replete with computer-controlled engine and antilock brakes, climate controls and communication system. Invisible computing makes these devices more flexible and adaptive.

Empowering Experiences

Although our worldwide deployment of computing remains sadly and unequally distributed, we have crossed a critical capacity threshold from paucity to plethora. Such a plethora of inexpensive and powerful computing devices, high-capacity storage systems and ubiquitous wired and wireless communication brings new opportunities. Fully capitalizing on those opportunities likely requires a shift in psychological mindset, particularly for those of us who entered the field when cycles, bytes and bits were to be managed frugally.

It is time to free our imaginations and embrace computing invisibility as our technological superpower. We can be truly profligate and use invisible computing to enrich the human experience and empower creativity in new and novel ways, exploring the rich interplay of technological possibilities with social expectations and desires. Concomitantly, as computing becomes the critical infrastructure of the 21^st century, we must also be aware of both the risks and the rewards. As one superhero famously remarked, "With great power comes great responsibility."

Innovation Via Client Plus Cloud: Microsoft-NSF Partnership

Update

Here is a bit more background and news coverage:

• New York Times article
• NSF-Microsoft press conference
• Inside HPC article
• Microsoft's own PR article

Today, February 4, Microsoft and the U.S. National Science Foundation (NSF) announced a collaborative project where Microsoft will offer individual researchers and research groups (selected through NSF's merit review process) free access to advanced client-plus-cloud computing. Our focus is on empowering researchers via intuitive and familiar client tools whose capabilities extend seamlessly in power and scope via the cloud.

I am very excited about this, as it is the fruit of nearly two years of planning and collaboration across Microsoft product and research teams, as well as many discussions with researchers, university leaders and government agencies. As part of this project, a technical computing engagement team, led by Dennis Gannon and Roger Barga, will work directly with NSF-funded researchers to port, extend and enhance client tools for data analysis and modeling. We also appreciate the support of the Microsoft Dreamspark, Technical Computing, Windows Azure, Azure Dallas, Public Sector, education and evangelism (DPE) teams, among others, to build and deliver this capability.

By extending the capabilities of powerful, easy-to-use PC applications via Microsoft cloud services, our objective is to broaden researcher capabilities, foster collaborative research communities, and accelerate scientific discovery by shifting the focus from infrastructure to empowerment. This is a potentially profound shift in how we innovate in the 21^st century.

21^st Century Innovation

The brief history of computing is replete with social and technological inflection points, when a set of quantitative and qualitative technology changes led to new computing modalities. I believe we are now at such an inflection point in computing-mediated discovery and innovation, enabled by four social and technical trends:

• Massive, highly efficient cloud infrastructures, driven by search and social networking demands
• Explosive data growth, enabled by inexpensive sensors and high-capacity storage
• Research at the interstices of multiple disciplines, conducted by distributed, virtual teams
• Powerful, popular and easy-to-use client tools that facilitate data analysis and modeling

The first two of these are well documented, and I have written about them before. (See Beyond the Azure Blue and Language Shapes Behavior: Our Poor Cousin Data.) The late Jim Gray also lectured and wrote perspicuously about data-intensive scientific discovery, which he called The Fourth Paradigm. As a logical complement to theory, experiment and computation, the fourth paradigm is based on extracting insight from the prodigious amounts of social, business and scientific data now stored in facilities whose scale now dwarfs all previous computing capabilities.

Climate change and its environmental, economic and social implications; genetics, proteomics, lifestyle, environment, health care and personalized medicine; economics, global trade, social dynamics and security – these are all complex, multidisciplinary problems whose exploration and understanding depend critically on expertise from diverse disciplines with differing cultures and reward metrics.

As our research problems rise in complexity, the enabling tools must rise commensurately in power while retaining simplicity. I have seen far too many multidisciplinary projects founder on the rocks of infrastructure optimization and complexity, when they should have focused on simplicity, familiarity and ease of use.

As Fred Brooks once remarked, "We must build tools so powerful that full professors want to use them, and so simple that they can." Simplicity really, really matters. It is for this reason that Excel spreadsheets, high-level scripting languages and domain-specific client toolkits are now the lingua franca of multidisciplinary innovation, the harbingers of invisibility.

Invisible Simplicity

Any successful technology ultimately becomes invisible, enabling and empowering while user knowledge of the technology's underlying theory and operation declines. This is success, for it allows all but the technology experts to focus on the raison d'être for computing or any other technology: enabling and expanding the human experience. By that metric, technical computing and its subset – high-performance computing— have experienced a long adolescence.

Sadly, our technical computing experiences have been dominated and shaped by a focus on technology and infrastructure, rather than empowerment and simplicity. We talk routinely of data and software repositories, toolkits and packages; of cyberinfrastructure and technology roadmaps. In our technological fascination, it is all too easy to lose sight of the true objective. Infrastructure exists to enable. If it is excessively complex, cumbersome or difficult to use, its value is limited. The mathematician Richard Hamming's admonition remains apt: "The purpose of computing is insight, not numbers."

Empowering the Majority

To address 21^st century challenges, we must democratize access to data and computational models, recognizing that the computing cognoscenti are, by definition, the minority of those who can and should benefit from computing-mediated innovation and discovery. Instead, we must enfranchise the majority, those who do not and will not use the low-level technologies – clusters, networks, file systems and databases – but wish to ask and answer complex questions. Remember, most researchers do not write low-level code; nor should they need to.

I believe we must focus on human productivity, not cyberinfrastructure, and leverage popular and intuitive client tools that hide infrastructural idiosyncrasies. This means deploying tools like Excel that can manipulate data of arbitrary size, reaching into the cloud to access petabytes of data and executing massive computations for epidemiological analysis as easily as one might balance a checkbook. It means coupling multiple, domain-specific toolkits via a script of a dozen lines, and launching a parametric microclimate study as easily as one searches the web. Perhaps more importantly, it means rethinking public and private sector partnerships for innovation, identifying and leveraging core competencies.

It all comes back to simplicity and invisibility. Technical computing can and should be an invisible intellectual amplifier, as easy to use as any other successful consumer technology. Now is the time; and Microsoft is committed to making it a reality. We look forward to working with the community.

Serial Dismay, Parallel Excitement

Multicolored lights are one of the iconic and ubiquitous manifestations of the end of year holidays. They also conjure festive memories of my childhood, albeit with one major exception. That rather less than pleasant memory involved much consternation, veiled (and not so veiled) mutterings and ritual questioning of all things derived from Maxwell and Edison.

I speak, without hesitation or ambiguity, regarding the curse (sometimes literal) of serially wired holiday lights. My father and mother kept a healthy supply of spare bulbs for laborious and sequential replacement and testing whenever a single bulb failed, and a strand of thirty bulbs went dark. I still remember my excitement and delight when we first purchased strands with parallel circuit wiring. What a difference it made, when a bulb failure dimmed only one light, rather than an entire strand. As a child, I quickly gained a visceral appreciation for Kirchhoff's laws, long before I ever knew the name Gustav Kirchhoff.

Systemic Resilience

If there is an Aesop-like moral in this tale from my childhood, it relates to systemic design for resilience rather than component resilience alone. Parallel circuit resilience trumps serial circuit resilience, and the extra cost is repaid in greater systemic reliability. Alas, I fear we have not learned this lesson in parallel computer system design and parallel programming models and applications.

The standard domain decomposition models commonly used to solve the partial differential equations that underlie so much of computational science and that are embodied in parallel MPI implementations of the solvers, implicitly presume that all of the parallel processes (tasks) and the supporting hardware operate without error or failure. Periodic checkpointing is our primary concession to failure and the vagaries of job scheduling.

This is analogous to trusting that the series circuit holiday lights will not fail until we turn them off each evening. It can be effective for small numbers of lights (processes) but increasingly problematic for larger systems. All this suggests we need to rethink our models of hardware and software resilience, and embrace redundancy as the necessary cost of systemic reliability, particularly in the trans-petascale and exascale regimes. It may also mean that we can never use all of the hardware for non-redundant computation. After all, would one rather fail quickly at exascale rates or compute reliably at sustained trans-petascale rates?

Changing the Game

Cloud services now operate on the largest computing systems we have ever built on this planet, with service reliability expectations far higher than what we demand from scientific applications. Thus, I also believe there are lessons from cloud computing that are potentially applicable to computational science applications. Arguably the most important is exploring how to exploit eventual consistency rather than sequential consistency.

To understand this potential shift in perspective, I heartily recommend Werner Vogels' analysis of the power of eventual consistency for large-scale web services at Amazon. Eric Brewer's thoughts on the CAP theorem, drawn from his Inktomi experiences, have also shaped theoretical and empirical assessments of large-scale system reliability. For those not familiar with the CAP theorem, it postulates that one can choose any two of Consistency, Availability or Partition tolerance. More generally, it offers a framework to reason about conflicting objectives.

Lest everyone shudder in numerical and scientific horror at this prospect, remember that parallel, numerical applications are themselves samples drawn from the space of "possible answers." They are not "the answer," and they may not even be an unbiased sample. (See On Getting the "Right" Answer.) Are these weak consistency approaches transferrable to scientific applications? I do not know. I am only sure that we need to find out – quickly. If successful, this would be a profound change, with deep implications for computational science algorithms, software and application reliability.

Two, Five, Ten …

It's not a Fibonacci series and it's not a prime number series. It might be the second three denominations of U.S paper currency, but it's not. (Yes, Virginia, there is a U.S. $2 bill.) Instead, it's a measure of the potential impact and uptake (or lack thereof) of new technologies. To see why, let's move from the abstract to the concrete, considering when new technologies gain credence.

Closing the Deal

Suppose you are presented with a new hardware or software model, one that requires migrating your extant code base to exploit the new model. The migration effort will be substantial, but you are promised a multiplicative increase in one or more metrics – performance, reliability, scalability, maintainability, or usability. What multiplicative factor will provide the benefit to justify the cost?

Will a factor of ten? Yes, history has shown repeatedly that an order of magnitude change in critical metrics offers competitive advantages that offset costs.

Will a factor of five? Perhaps it will, but there are no guarantees.

Will a factor of two? Certainly not, for the costs are real and the benefits are modest. Incremental advances of existing technology are likely to yield comparable benefits and often more quickly. Those who risk change for small factors are often left behind.

Remember the Cray-1

We tend to forget the real reason the Cray-1 was so successful. It was not the innovative vector architecture and memory system, nor was it the plethora of great software. There was so little software on the Cray-1 that Los Alamos launched a project to build an operating system, DEIMOS, for it. Rather, it was that the Cray-1 was by far the fastest scalar processor in the world when it was introduced. Existing scalar code ran much faster than on earlier machines, even without vectorization, creating a powerful impetus to move to the new machine and vectorize the code incrementally.

It Will Scale Linearly

Years ago, I was sitting in the back of a conference room with John Hennessy (now the President of Stanford University), listening to a young researcher talk about their latest parallel software technology. I must confess that John and I weren't paying much attention, until the researcher said something that immediately changed our intellectual focus. In passing, the researcher said, "There are no reasons this technique shouldn't scale linearly with system size." John and I immediately looked at one another, incredulous at what we had just heard. I no longer remember which of us took exception to the comment, but we did.

Rarely do computing system architectures scale by orders of magnitude – in either direction – without exposing either performance bottlenecks or non-scalable design assumptions. Things that consume negligible time at one scale suddenly dominate performance at another, or software structures become unwieldy.

Lessons Learned

Despite our love of self-similarity, orders of magnitude really matter. It's worth remembering this as we contemplate future exascale computing system designs and digest the recent DARPA exascale software study. Lessons from the petascale era may or may not translate directly to exascale designs. Two, five ten – Caveat emptor!