Reed's Ruminations: A Blog by Dan Reed

Nothing Left But the Smile …

Long ago, we in HPC recognized the inevitability of the Attack of the Killer Micros, and with few exceptions, all of today's HPC systems are based on some variant of commodity microprocessors and their commodity cousins, GPUs. A similar revolution swept the secondary storage market. After all, the I in RAID standards for inexpensive, a synonym for commodity.

Like the Cheshire Cat in Alice's Adventures In Wonderland, it seems increasingly clear that in the high-performance, low latency interconnection network space, we will be left with nothing but the smile. Simply put, the last remaining non-commodity component of HPC clusters – the high-performance interconnect – is in grave danger, due to the global economic downturn and the price-performance pressures of commodity Ethernet.

Remember Coaxial Cables and BNC?

The old geezers among us, including yours truly, remember when Ethernet meant coaxial cables and CSMA/CD at 10 megabits/second. (Here's a shout out to my colleague Chuck Thacker, one of the co-inventors of Ethernet.) Since then, we have seen many generations of advances in transport fabrics, switches and routers and speeds.

One gigabit Ethernet is now a commodity item, contained on almost all PC motherboards, 10 gigabit Ethernet is the high bandwidth standard, and 40 gigabit and 100 gigabit Ethernet standards are under development by the IEEE. Infiniband is one of the last major competitors to 10 gigabit Ethernet, and even the Infiniband vendors are increasingly offering Ethernet compatibility (i.e., so-called converged fabrics).

Cloud Data Centers and HPC

I invite you to ponder your response to the following question. What is the difference between a megascale data center and a petascale computing system? Increasingly, the answer is "not much, but a few elements of the software stack."

Of course, this need not be the answer, but it likely will be unless we change our research and development strategies and also our procurement models. Ironically, megascale data centers could also benefit from lower latency, higher bandwidth communication fabrics with scalable bisection bandwidth. It is time we bring these two together, for the HPC community could leverage the economics of megascale data centers and their needs.

NITRD Reauthorization: Enabling the Future

As a computing researcher, as chair of the Computing Research Association (CRA), and as a former member of the President's IT Advisory Committee and the President's Council of Advisors on Science and Technology (PCAST), I have spoken and written repeatedly about the state of computing research in the United States, the importance of long-term, strategic investment and the critical need for strategic, interagency planning.

Today, the U.S. House of Representatives passed H.R. 2020, the Networking and Information Technology Research and Development Act of 2009, which embodies many of those recommendations. As press release from the House Committee on Science and Technology notes, this reauthorization of the Networking and Information Technology R&D (NITRD) program

… strengthens interagency planning, coordination, and prioritization for NITRD by requiring the development and periodic update of a strategic plan informed by both industry and academia. This plan is meant to create a vision for networking and information technology R&D across the federal government, and provide specific metrics for measuring progress toward that vision.

The Road to the Present

As chair of the Computing Research Association (CRA), I was pleased to endorse H.R. 2020. Simply put, H.R. 2020 is the culmination of many years of background work, reports, discussions and Congressional hearings by diverse groups.

In July 2008, i testified before Rep. Gordon and the House Science and Technology Committee ("NITRD: Come, Let Us Reason Together"), summarizing the 2007 recommendations of the President's Council of Advisors on Science and Technology (PCAST) report, Leadership Under Challenge: Information Technology R&D in a Competitive World, whose production I had the privilege to co-chair. The report included the following recommendations ("PCAST, NITRD and the Future"), emphasizing the contributions of information technology to our continued prosperity and well being, something especially timely given current circumstances:

• Address the demand for skilled IT professionals by revamping curricula, increasing fellowships, and simplifying visa processes.
• Emphasize larger-scale, longer-term, multidisciplinary IT R&D and innovative, higher-risk research
• Give priority to R&D in IT systems connected with the physical world, software, digital data, and networking
• Develop and implement strategic and technical plans for the NITRD program

Thanks to the hard work of many people, all of these were addressed in the NITRD reauthorization bill. Specifically, the reauthorization includes creation of a five year strategic plan, to be updated every three years and assessed by an independent committee whose co-chairs are members of PCAST. The reauthorization also emphasizes the importance of long term, multidisciplinary research and identifies cyberphysical systems as a critical element of the research agenda.

These are sufficiently noteworthy that I feel compelled to quote from H.R. 2020, regarding the strategic plan:

(A) foster the transfer of research and development results into new technologies and applications for the benefit of society, including through cooperation and collaborations with networking and information technology research, development, and technology transition initiatives supported by the States;

(B) encourage and support mechanisms for interdisciplinary research and development in high-performance computing, including through collaborations across agencies, across Program Component Areas, with industry, with Federal laboratories (as defined in section 4 of the Stevenson-Wydler Technology Innovation Act of 1980 (15 U.S.C. 3703)), and with international organizations;

(C) address long-term challenges of national importance for which solutions require large-scale, long-term, interdisciplinary research and development;

(D) place emphasis on innovative and high-risk projects having the potential for substantial societal returns on the research investment; and

(E) strengthen all levels of networking and information technology education and training programs to ensure an adequate, well-trained workforce.

Yes, high-performance computing is identified explicitly, as a collaborative activity across government, industry and academia and with international partners.

Cyberphysical Systems

As some of you may recall, cyberphysical systems (i.e., computing systems that interact with the physical world) emerged as the top research priority from the PCAST assessment of NITRD needs. Today, our critical national and international infrastructure (financial systems, telecommunications, transportation, and utility grid), national security, and our personal lives (communications, biomedical devices, household appliances, automobiles and entertainment systems) are all computer enhanced and mediated.

Computing is an inseparable part of our culture and our prosperity, and ensuring the reliable, correct and secure operation of this cyberphysical infrastructure is central to our future. Hence, I am especially delighted that the reauthorization calls for a joint university/industry task force to develop a research and development agenda for cyberphysical systems, together with defining roles and responsibilities, suggesting funding mechanisms and discussing intellectual property (IP) mechanisms. I am especially pleased that IP mechanisms was identified explicitly, as I believe we need to rethink how our public-private sector partnerships are best organized for mutual benefit.

The Road Ahead

The work by the House Science and Technology Committee and the passage of H.R. 2020 by the full House paves the way for the future. We have defined a scaffold for the future. Now we must erect the enabling infrastructure for a knowledge-centric society. I am confident the new incarnation of PCAST, which includes my Microsoft colleague, Craig Mundie, will continue to watch the progress of the NITRD program as our ever-changing field helps shape our future.

A Few Thoughts on Programming

I just posted an essay on the BLOG@CACM site regarding the divergence of programming models for network (distributed) services and for parallel computing.  The details are here.

In the essay, MPI is a placeholder for all of the message passing libraries we have developed over the past thirty years.

On Getting The “Right” Answer

Update:  Jim Pool reminds me of the following book, which touches on this issue.

The parallel application contains millions of lines of code, combining multiple models of physical, engineering, biological, social and/or economic processes, operating over temporal and spatial scales that span ten orders of magnitude. It was written by tens or even hundreds of graduate students, post-doctoral associates, software developers and yes, even a few professors, over a decade. It involves numerical libraries and functions from diverse research groups and companies, and a single execution requires thousands of hours on tens of thousands of processor cores. In short, it's a typical example of an extreme scale high-performance computing code.

And The Answer Is …

What is the probability that any execution of this code produces the "right" answer? Does a "right" answer even exist? If so, how might we know? Or, is this a nonsensical question?

These are not simply Gedanken experiments in software engineering or hardware reliability, nor are they just epistemological questions about the philosophy of knowledge. Rather, they are very practical and real questions about the nature of extreme scale computational science. They are the essence of verification and validation (V&V) processes, and we should be much more rigorous and systematic about their application.

The central lesson of software engineering is that regardless of the rigor of design processes, testing methodologies and boundary condition specifications, large applications, regardless of type, will contain multiple errors. We have all seen and experienced examples, from the blue screen of death on personal computers to the loss of the Mars Climate Orbiter due to mixed unit calculations. Computational science applications have no special dispensation to escape this destiny.

Second, because the raison d'être for multidisciplinary applications is enabling researchers to gain insight into complex and often poorly understood phenomena, testing them can be problematic, as the answers are often known only for simplified, model problems or boundary conditions. What constitutes a rigorous test suite when little or no experimental data is available for independent comparison?

Third, there's the small matter of numerical stability. The IEEE floating point standard balances range (exponent) and precision (mantissa) in a fixed number of bits, and necessarily approximates many real numbers. After large numbers of floating point operations, even a stable algorithm and a well-conditioned problem will contain some degree of error. Rarely do we bound the possible error via interval analysis.

Finally, there's the small matter of hardware errors, an increasingly common phenomenon for large memories constructed from high-capacity DRAMs and for microprocessor datapaths. We have been taught that error correcting codes correct DRAM bit errors. In truth, the standard SECDED code only corrects single bit errors and detects double bit errors. Burst DRAM errors (triple bit or more) are not corrected and are increasingly common. Some iterative algorithms can recover from bit errors, converging to the "correct" answer; others cannot.

Computation As Experiment

Are you afraid? We all should be. It is time to embrace the scientific process for computational science. We must view the execution of a large, multidisciplinary code as what it is – an experiment, with all the possible error sources attendant with any physical experiment. This includes repeating the experiment (computation) to determine confidence intervals on the answer, conducting perturbation studies to determine the sensitivity of the answer to environmental (hardware and software) conditions, identifying sources of experimental bias and defining the experiment rigorously for independent verification.

In the days of analog scientific computing via slide rule, we all understood that the computation was an approximation. It's time to relearn that lesson.