Reed's Ruminations: A Blog by Dan Reed

American Competitiveness: IT and HPC Futures – Follow the Money and the Talent

N.B.  A more formal version of these arguments is posted on arXiv (https://arxiv.org/abs/2203.02544), co-authored with Dennis Gannon and Jack Dongarra.

This is a long post, containing perspectives that may be controversial to some.  That's okay; it is why we have lively intellectual debates.

Charles Dickens’ A Tale of Two Cities opens with this line:

IT was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of light, it was the season of darkness, it was the spring of hope, it was the winter of despair.

It’s an apt metaphor for the state of information technology and the state of U.S. competitiveness in computing and science, technology, engineering, and mathematics (STEM).  In some ways, the hegemony of U.S. IT has never been greater, with Apple and the cloud companies (Google, Microsoft, Amazon, and Facebook) dominating the Nasdaq with trillion dollar (and more) market capitalizations and reshaping the very definitions of computing with global services, deep learning, and advanced AI.

Conversely, the semiconductor shortage has highlighted the economic and national security risks from over-dependence on off-shore fabrication, and the relative influence of traditional computing companies on the IT ecosystem has waned. Meanwhile, the future of U.S. leadership in bleeding edge scientific computing (aka supercomputing) is at a critical crossroads.

 Finally, while the global competition for STEM talent in general and IT talent in particular has never been more intense, the U.S. is substantially underperforming relative to other countries in STEM education.  Let’s take a look at how all of this has happened.  Remember, follow the money and the talent.

Remembering the Past

Let’s start by playing a parlor game. Name the first computer company that comes to mind.  If you are an old timer, you might be inclined to mention IBM, Amdahl, and the BUNCH (Burroughs, UNIVAC, NCR, CDC, and Honeywell).  Although IBM once bestrode the computing world like a colossus, today, even its relevance has faded. Indeed, few now remember that IBM was once locked in a bitter struggle with the U.S. Department of Justice for monopolistic practices under anti-trust laws.  As for the BUNCH, they are either gone from the computer business or niche players at best. Later, others – Compaq, Digital Equipment Corporation (DEC), and SGI – came and went.

As an aside, I learned to program on an IBM System 360/50, wrote lots of code on a DEC PDP-11/45 for my first paid software development job, ported code from a GE 600 timesharing system, and used a CDC 6500 and 6600 during graduate school. I programmed in FORTRAN, COBOL, PASCAL, LISP, and PI/I (in addition to many more recent languages), and I’ve toggled absolute binary into the front panel. I’m also old enough to remember when one could do Mead and Conway VLSI design with colored pencils.  I also used a slide rule, while walking barefoot in the snow to school, uphill (both ways), but I digress. (See You Might Be a Computing Old Timer If …)

There’s a sad, parallel tale (pun intended) in the land of high-performance computing, where the cemetery is filled with departed denizens, some large, some small, but all now permanent members of the Dead Computer Society. Whether by misjudging markets or technologies, each in in its own way proved the truth of the old adage that the best way to make a small fortune in supercomputing is by starting with a large fortune, then shipping one or two generations of product. Sic transit gloria mundi SGI, Cray, and a host of others. (See HPC: Making a Small Fortune.) 

More to the point, the list of high-end supercomputing players keeps getting shorter.  Cray is gone, now part of HPE; and both IBM and Intel are struggling to right themselves, each for different reasons.  In the United States, this leaves HPE as the lone high-end HPC integrator in the United States.  Time will tell if Nvidia, with its recent acquisitions, steps into this space, though its proposed acquisition of ARM seems unlikely.

Meanwhile, the information technology ecosystem and the locus of innovation and global leadership have shifted dramatically. There are deep and important implications for the U.S.

Contemplating the Present

If you are Gen Z or Gen Alpha, the world begins and ends with Apple, Google, and Samsung smartphones and the cloud services behind them.  All of which brings us to the FAANG companies (Meta née Facebook, Amazon, Apple, Netflix, and Alphabet née Google) plus Microsoft and their Asian BAT (Baidu, Alibaba, and Tencent) counterparts.  These companies are economically exothermic in ways perhaps unprecedented since the days of the Standard Oil monopoly.  Doubt me?  Look at their market capitalizations  and free cash flow relative to traditional computer companies. (As I write this, Microsoft just made a $69B all-cash offer for Activision and reported over $50B in quarterly revenue.)

I first watched this power dynamic begin to shift while at Microsoft.  For years, the Wintel duopoly locked Microsoft and Intel in a collaborative/competitive partnership as rough peers.  I sat in more than one “frank and spirited” exchange between leaders of the two companies, where each pushed the other to accommodate the other’s market needs. As server chips, clouds, and infrastructure grew in importance relative to desktops and laptops, the power dynamic began to shift.  Today, Intel is the far weaker player, in many cases forced to respond to cloud vendor demands.  In hindsight, it is all obvious.

Technological and Economic Shifts

Today, Apple and the other cloud service companies dominate the computing hardware and software ecosystem, both in scale and in technical approaches. Initially, they purchased standard hardware for deployment in traditional colocation centers (colos). Then they began designing purpose-built data centers, optimized for power usage effectiveness (PUE), deployed at sites selected via multifactor optimization – inexpensive energy availability, tax breaks and political subsidies, political and geological stability, network access, and customer demand. 

As scale, complexity, and operational experience grew, new optimization and leverage opportunities emerged, including software defined networking, protocol offloads, and custom network architectures (greatly reducing dependence on traditional network hardware vendors), quantitative analysis of processor, memory, and disk failure modes, with consequent redesign for reliability and lower cost (dictating specifications to vendors via consortia like Open Compute), custom processor SKUs, custom accelerators (FPGAs and ASICs), and finally, complete processor design (e.g., Apple silicon, Google TPUs and AWS Gravitons).  In between, they deployed their own global fiber networks.

This virtuous cycle of insatiable consumer demand for rich services, business outsourcing to the cloud, expanding data center capacity, and infrastructure cost optimization has had several effects.  First, it has dramatically lessened – and in many cases totally eliminated -- FAANG/Microsoft/BAT dependence on traditional computing vendors and ameliorated the risks of wholesale transfer pricing. Put another way, Amazon AWS, when necessary, can now negotiate pricing terms with Intel and AMD in the same way Walmart does with Procter and Gamble (i.e., from a position of strength as a dominant customer).

Second, the absolute scale of infrastructure investment, denominated in billions of dollars per year for each cloud service vendor, means these companies are shaping computing technology research and development priorities in ways traditional computing vendors rarely can, driving design and fabrication priorities in every element of the ecosystem.

Bigger is not simply bigger; bigger is different, fueling investment in new technologies at a phenomenal rate. (See The Zeros Matter: Bigger Is Different.) Relatedly, I cannot overemphasize the massive scale of these cloud data center deployments, dwarfing anything seen in academia or national laboratories, with the already large gap widening each year.  The CAGR is simply astounding.

Third, custom infrastructure is about more than economic leverage; it’s about market differentiation and competitive advantage.  Anything not differentiating is standardized and commoditized, driving unnecessary cost out of the system, and concomitantly reducing profits for commodity purveyors.  Conversely, anything differentiating is the focus of intense internal optimization and aggressive deployment.

Equally importantly, client plus cloud services vendors predominately make their money from the services offered atop their infrastructure, much less from the infrastructure itself. Unlike Intel, AMD, and Nvidia, which sell silicon, Apple is not selling A15 SoCs; it is selling iPhones, just as Amazon and Google are selling services atop TPUs and Gravitons.

Fourth, all of this is occurring against a backdrop of big data and machine learning. Large-scale AI itself is fueling a computing revolution, with an insatiable hardware demand to train ever larger machine learning models.  As an example, modern speech recognition models now have hundreds of billions of parameters, and multiple groups have announced models with over one trillion parameters. Similarly, work on generative adversarial networks (GANs) has led to impressive advances in game playing and other tasks. Nor is deep learning limited to “AI workloads;” there are impressive results on solving differential equations as well.  (See Dennis Gannon’s excellent summary of Karniadakis’ work here.) Similarly, look at developments in generative adversarial networks (GANs),

Finally, all of these trends are convolved with the end of Dennard scaling, dark silicon, functional specialization, chiplets, semiconductor fabrication challenges, rising fabrication facility costs, and an uncertain political future for Taiwan and TSMC.  Moreover, the actual cost of transistors (dollars/transistor) is increasing as we move closer to one nanometer scaling.  In fact, the transistor cost minimum occurred at roughly the time (~28 nanometers) we switched from planar CMOS to FinFETs. It will likely only get worse as gate-all-around FETs predominate, given increased fabrication complexity.

In the midst of all this, Intel stumbled, losing its lead in process technology just as EUV finally and belatedly became mature, and TSMC staked a lead. Time will tell if Pat Gelsinger can turn things around.

The Future

Okay, let’s put all the technology and economics puzzle pieces together:

• Advanced computing requires non-recurring engineering (NRE) investment to develop new technologies and systems.
• The FAANG/Microsoft/BAT companies are cash rich (i.e., exothermic), and they are designing, building, and deploying their own hardware and software infrastructure at unprecedented scale.
• The traditional computing vendors are now small – in some cases, tiny – economic players in the computing ecosystem, and many are dependent on federal investment (i.e., endothermic) for NRE to advance the bleeding edge.
• AI is fueling a revolution in how we think about problems and their numerical solution.
• Dennard scaling is over and performance advances increasingly depend on functional specialization via custom ASICs and chiplet-integrated packages.
• Transistor costs are likely to continue to increase as we push to smaller feature sizes.
• Nimble hardware startups are exploring new ideas, driven by the AI frenzy.
• Talent is following the money and the opportunities, which are increasingly in industry.

With this backdrop, what is the future? I believe some of it is relatively obvious and secure, given the current dominance of U.S. cloud service providers. However, broader U.S. leadership in semiconductors, information technology, and bleeding edge HPC is less certain. I believe any viable solutions will require rethinking our approaches and assumptions, including how, where, and when we spend money and what will both grow and retain talent.  Let’s start with the broad issues around the U.S. information technology ecosystem.

Information Technology Futures

The big changes outlined above make it obvious that a computing revolution is underway, one that is reshaping global leadership in critical areas.  All revolutions create opportunity, but only for those prepared to accept the risks and act accordingly. In Far Side cartoon parlance, we must be tall enough to attack the city.

First, we need to be honest about the social, political, economic, and national security risks of losing U.S. leadership and control of the silicon fabrication ecosystem.  Fabless semiconductor firms are important, but we also need onshore, state of the art fabrication facilities, and we need to invest and act accordingly. It is why the U.S. CHIPS Act and its successors are so important, and why the recent announcements by Intel, TSMC, and GlobalFoundries that they are building new chip fabrication faculties in the U.S. (each for different reasons), is so important.

Second, the seismic business and technology shifts have also led to equally dramatic talent shifts. People gravitate to opportunities, and that giant sucking sound is people leaving academia, national laboratories, and traditional computing companies to pursue those opportunities.  Seymour Cray once said, “I wanted to make a contribution,” when asked why he had used so many high risk technologies in the Cray-1. The desire to make a difference is no less powerful today.

The academic brain drain among artificial intelligence researchers is well documented.  After all, in the commercial world one can develop and test ideas at a scale simply not possible in academia, using truly big data and uniquely scaled hardware and software infrastructure.  The same is true for chip designers and system software developers.

Making matters worse, the U.S. talent pipeline is uncertain.  As the recent 2022 Science and Engineering Indicators shows, the U.S. is losing ground compared to the rest of the world, trailing China in both patents and publications, as summarized in this Science story about the recent Science and Engineering Indicators report. 

Our K-12 STEM education performance lags most of Asia and Europe – as shown in the chart to the right, we rank between Hungary and Turkey in mathematics performance by 15-year-olds.  We also have deep and systemic inequalities in STEM education, and we have sadly created an environment that is less attractive to international graduate students and scholars than it once was.  The latter is especially worrisome given that over 60 percent of computer science and mathematics Ph.Ds. working in the United States were born overseas.

Practically, this means we need to recognize the need for determined actions:

• Raise the levels of federal investment in ongoing information technology R&D. This means billions, not millions of dollars.
• Invest in the talent pipelines needed to make academic and laboratory R&D attractive.
• Partner deeply with the U.S. private sector in new and creative ways.

Now, what about advanced scientific computing futures?

Leading Edge HPC Futures

It now seems self-evident that supercomputing, at least at the highest levels, is endothermic, requiring regular infusions of non-revenue capital to fund the NRE costs to develop and deploy new technologies and successive generations of integrated systems. In turn, that capital can come from either other, more profitable divisions of the business or from external sources (e.g., government investment). 

While at Microsoft, I once pitched a technical computing idea to CEO Steve Ballmer.  He listened patiently to my spiel, then simply said, “I’m not interested.”  Being nothing if not persistent, I asked why.  To which he replied, “It’s not that I think you are wrong.  I actually think you are right – we could drive a few hundred million dollars in revenue, but the opportunity cost is too high, and the return on investment (ROI) is too low.  Come back when you have an idea that starts with B (billions), not M (millions).”

I was chastened, but it was a valuable lesson that I fear too many in high-end scientific and engineering computing have not learned.  A business leader must always look at the opportunity cost (time and cost of money) for any NRE investments.  The core business question is always how to make the most money with the money you have, absent some other marketing or cultural reason to spend money on loss leaders, bragging rights, or political positioning.   The key phrase here is “the most money;” simply being profitable is not enough. 

Therein lies the historical “small fortune” problem for HPC.  The NRE costs for bleeding edge supercomputing are now large relative to the revenues and market capitalization of those entities we call “computer companies,” and they are increasingly out of reach for U.S. government agencies, at least under current funding envelopes. The days when a few million dollars could buy a Cray-1/X-MP/Y-MP/2 or a commodity cluster and land in the top ten of the Top500 list are long gone. Today’s game requires hundreds of millions of dollars to deploy a machine at the high end and at least similar, if not larger investments in NRE.

What does this brave new world mean for those of us in the big iron, bleeding edge HPC crowd?  First, as I mentioned over a decade ago, we are really the fast iron crowd, as our biggest systems are now dwarfed by the overall scale of commercial cloud infrastructure deployments. Second, we may still be invited to dinner, but economically, we are increasingly relegated to eating SpaghettiOs and drinking Kool-Aid at the kids table, while the adults are dining on lobster and drinking limited release, estate-bottled wine. 

Put another way, the golden rule applies – buying a $500M supercomputer every five years is not much financial leverage against a multibillion dollar spend each year. Equally importantly, most of the traditional supercomputing vendors are sitting at the kids table with us.  In turn, federal investment (e.g., DOE’s exascale PathForward program) is quite small compared to the scale of commercial cloud investments and their leverage with the same vendors.

First, it means acknowledging that we in bleeding edge HPC have limited leverage; we cannot influence vendors in the same ways we did in the past. The delays and cost overruns for Argonne National Laboratory’s Aurora system and the (resolving) deployment pains for Oak Ridge National Laboratory’s Frontier system speak to these challenges. Meanwhile, China has already deployed two exascale systems, both built using Chinese components.

We also need to recognize that bleeding edge HPC is about more than solving time-dependent partial differential equations on complex meshes.  They will always matter, but other areas of advanced computing are also of high and growing importance.  After all, the Science 2021 Breakthrough of the Year was for AI-enabled protein structure prediction, with transformative implications for biology and biomedicine.  This is a cultural shift that will require us to lessen our hold on some cherished beliefs and acknowledge the world has changed.

Second, it’s time we got serious – really serious – about end-to-end co-design, from device physics to applications, following the lead of the cloud ecosystem operators, and where possible and appropriate, partnering with them. This means more than encouraging tweaks of existing products or product plans.  Rather, it means looking holistically at the problem space, then envisioning, designing, testing, and fabricating appropriate solutions. (See Advanced Computing: Integrative Thinking for the Future.)  As a contact sport, it means substantially expanded investment in basic research and development, something that unfortunately suffered as the U.S. Exascale Computing Project (ECP) diverted limited funds to timeline-driven deployments

Third, we will need to embrace the chiplet ecosystem and work collaboratively with the commercial players to establish interoperability standards (e.g., the Open Domain-Specific Architecture (OSDA) project within the Open Compute Project). Chiplets are more than a way to mix and match capabilities from multiple sources, they are an economic and engineering reaction to the interplay of chip defect rates, the cadence of feature size reductions, and semiconductor manufacturing costs (CAPEX and OPEX). Concomitantly, we must both work with the new chiplet design and integration companies while also designing chiplets, both driven by the specific needs of quantified scientific workloads.

There ARE interesting things happening in the hardware space, but many of them are in the startup world.  One need look no further than companies like Graphcore, Groq, and Cerebras for inspiration, and at the ferment in edge devices and AI. I will not name other companies, as several are still in stealth mode. Part of this requires some flexibility and imagination.  (Full disclosure, I started NCSA on the GPU accelerator push by building a cluster of Sony Playstations and formed the first partnership with NSF on cloud computing.)

Fourth, we will need to build real hardware and software prototypes at scale, not just incremental ones, but ones that truly test new ideas using custom silicon and associated software. In turn, that means accepting the risk of failure at substantial scale (e.g., that $50M prototype didn’t work, but we learned some important things, and we are going to build another one).  (See Luck Is a Fickle Friend.)

To do these things, we will need to invest in recruiting and sustaining research teams – chip designers, system software developers, and packaging engineers – in an integrated end-to-end way AND create opportunities than make it intellectually attractive to work on science and engineering problems. Simply put, we will need to create integrated teams and build integrated prototypes at scale, both at national laboratories and in academia.

Finally, and I realize this may be heretical to some, we need to ask when and where we would be best served by embracing commercial cloud services for our HPC needs.  The performance gaps between cloud services and HPC have lessened substantially over the past decade.  See, for example, a recent comparative analysis by the UC-Berkeley group. HPC as a service is now both real and effective.

I realize some in academia and national laboratory community will immediately say, “But, we can do it cheaper, and our systems are bigger!”  Perhaps., though proving such claims means being dispassionately honest about technological innovation, NRE costs, opportunity costs, staff career goals, and the true total cost of ownership (TCO). In turn, this requires a mix of economic and cultural realism in making build versus use decisions and asking what cloud service vendors would be willing to do for academic and government consortia at large scale (e.g., a billion dollar annual partnership). 

In summary, it is increasingly unlikely that future high-end HPC systems will be procured and assembled solely by commercial integrators from commodity components.  Rather, they are likely to be (a) collaborative partnerships with cloud providers, (b) increasingly bespoke, designed and built collaboratively to support key scientific and engineering workload needs, or (c) a combination of these two.

Put another way, bleeding edge, exascale class computing systems are no different than the Vera Rubin Observatory, the LIGO gravity wave detector, or the Advanced Photon Source.  Each contains commercially designed and constructed technology, but they also contain large numbers of custom elements for which there is no sustainable business model.  Instead, we build these instruments because we want them to explore open scientific questions, and we recognize that their design and construction requires government investment.

Coda

As our U.S. political divisions illustrate, investing in the future will not be easy, but it is critical if the U.S. is to remain globally competitive. Let’s be clear.  The price of innovation keeps rising, the talent is following the money, and many of the traditional players – companies and countries – are struggling to keep up.  Andy Grove was only partially right.  It is true that only the paranoid survive, but they’d best have deep pockets and enticing intellectual opportunities as well.  That needs to be us.

N.B. My thanks to Jack Dongarra, Doug Burger, Dennis Gannon, Tren Griffin, Tony Hey, and Jim Larus for thoughtful feedback on drafts of this post.  Any errors of fact and all opinions are my own. In full disclosure, I am a former Microsoft corporate vice president and a current stockholder.

Technical Potpourri

Edge Computing

SC17, the annual gathering of the supercomputing community, was held in mid-November, the week before U.S. Thanksgiving. As I mentioned in a previous post (Come to the Supercomputing Big Data Edge), we presented the results of the Big Data and Extreme-scale Computing (BDEC) workshops on the technical and social convergence of big data, machine learning, high-performance computing, and computing hardware and software. You can find the report here. At SC17, we had a lively discussion about edge computing, global competitiveness, and the nature of scientific revolutions.

HPC Data Center Energy Efficiency

In addition to the BDEC discussion, I also chaired an SC17 panel on the energy efficiency of HPC data centers. The panel was organized with the help of the Energy Efficient High Performance Computing Working Group (EEHPCWG). The panel participants were Sadaf Alam (CSCS/ETH), Bill Gropp (NCSA/Illinois), Satoshi Matsuoka (GSIC/Tokyo Tech), and John Shalf (NERSC/LBL). You can find the slides and a video of the panel here, courtesy of InsideHPC.

Autonomous Vehicles

Finally, I was recently a guest on Iowa Public Radio's lunchtime conversation, River to River. We had a lively and thoughtful conversation about the potential social consequences of autonomous vehicles. You can listen to that audio here.

Related articles

Exascale Computing and Big Data: Time to Reunite

Come to the Supercomputing Big Data Edge

SC17, the annual gathering of the supercomputing community is nigh. As always, there will be talk of next-generation technology – hardware, software, algorithms – and new applications. All of this will take place against the backdrop of the race to exascale computing – bigger and bigger, faster and faster. We will all geek out, and we will love it.

While we discuss operations/joule, quantum and neuromorphic computing, and new and exotic memory technologies to enable ever faster systems, there is another computing revolution underway, at the other extreme. As Richard Feynman famously said about nanoscale devices, "There's plenty of room at the bottom." Folks, I'm talking edge devices, the world of streaming data sensors that is changing the nature of computing and increasingly, of science as well. This is the subject of the forthcoming Big Data and Extreme-scale Computing (BDEC) report.

Computing History 101

The history of computing is one of punctuated equilibrium, with each pendulum swing, from centralized to distributed, bringing new and unexpected changes. It has also been a story of disruption from below, as successive generations brought order of magnitude shifts in price-performance ratios and sales volumes. Mainframes beget minicomputers, which begat workstations, which begat personal computers, which begat smartphones. The same has been true in scientific computing. Monolithic supercomputers (think the original Cray series) gave way to symmetric multiprocessors (SMPs) and distributed memory supercomputers. These gave way to clusters, now augmented with accelerators. Mind you, I still miss the lights on the Thinking Machines CM-5.

In both technical and "mainstream" computing, our largest systems are of unprecedented scale, whether globe spanning networks of massive cloud data centers or petascale and soon-to-be exascale HPC systems. As the Internet of Things sweeps across the computing landscape like a tsunami, one should ask about its analog in technical computing – the burgeoning networks of sensors that now capture science, engineering, environment, health and safety, and social data. (See Mobility, Transduction, and Garages)

The Sensor Edge

Today's smartphone-cloud ecosystem places most of the intelligence and data in the cloud. Indeed, all of the intelligent assistants – Siri, Cortina, and others – run in the cloud, dependent on precious wireless network bandwidth and relying on both behavioral history and deep learning algorithms. Limited by smartphone processing speeds, memory capacity, power budgets, and battery lifetime, and human expectations and physiology, there are good economic and technical reasons for this design.

There are also disadvantages, particularly in science. First, transmitting raw, time sensitive data to central sites places major demands on national and international networks. When the Large Hadron Collider (LHC) was under construction, this bandwidth and data processing demand was a major concern, triggering an international collaboration to build a distributed data storage hierarchy; those concerns remain as luminosity upgrades to the HLC are being planned.

The planned data pipeline for the Large Synoptic Survey Telescope (LSST) will send the raw data from the telescope in Chile to a dedicated supercomputer at NCSA, with a sixty second (maximum) delay before global alerts of transient phenomena must be distributed. Although this centralized approach can and will work when the daily data volume is measured in terabytes, the data deluge of the Square Kilometer Array (SKA) will require in situ reduction.

My friends in Chicago's Array of Things project grappled with another aspect of this local/remote tradeoff when designing urban sensors to capture and analyze human interactions. Communities were happy to support social dynamics analysis, but they did not want Big Brother video to leave the sensors. The privacy and social constraint requires low power, high-performance machine learning solutions on the sensors themselves.

Scale Challenges

Centralization also brings operational and security risks. My former Microsoft colleague, James Hamilton (now at Amazon), when asked why cloud data centers are not even bigger, said, "… the cost savings from scaling a single facility without bound are logarithmic, whereas the negative impact of blast radius is linear." We have all experienced the blast radius effect, at least metaphorically; when a cloud data center goes down, it affects both social networks and business, and it is international news. There is, perhaps, a salutatory lesson here for the ultimate, maximum size of supercomputers, but that is a discussion for another day.

In addition, the stateless design of the venerable TCP/IP stack includes no notion of data geo-location or intermediate data processing. Everyone who has debugged "last mile" network performance problems knows this all too well. To remediate this, research groups and companies have designed a variety of domain-specific network overlays, data caches, publisher-subscriber systems, and content delivery networks (CDNs).

Dramatic declines in computing costs mean it is now possible to deploy truly large numbers of inexpensive sensors. How dramatic? I have posed the following question on my campus. What would you do with 10,000, disposable ~$10 sensors? I was met with surprise at the scale of my question. For many of us, even those who came of age in the smartphone era, this scaling is a difficult question to answer because it defies our worldview of the possible. It is not a science fiction fantasy; it is a present reality.

A traditional hub and spoke model of data storage and sensor data analysis is increasingly neither scalable nor reliable. Amazon's Greengrass architecture is an early example of a generic, distributed approach, while NVidia's Metropolis system targets video in smart cities. There is an even more general model, where generic services support distributed data caching and processing throughout the network. Micah Beck (Tennessee) has made a thoughtful case for the design of such an approach, which he calls an Exposed Buffer Processing Architecture. Similarly, the Array of Things leaders have argued that edge processing is essential when

• The data volume overwhelms the network capability
• Privacy and security encourage short-lived data
• Resilient infrastructure requires local actuation/response
• Latency constraints demand the response times of dedicated resources

Simply put, these requirements imply a much more complex, distributed data storage and processing architecture for data-driven scientific computing than we have developed to date.

Supercomputing Edge Manifesto

This brave new world of streaming data has profound implications for the future of supercomputing. Here are just a few:

• Streaming data is continuous and requires a reconceptualization of workflow and scheduling systems. Periodic, batch processing is necessary but not sufficient.
• Sensor scale and complexity span many orders of magnitude, from inexpensive, disposable devices to nation-scale instruments. New orders of magnitude always expose new bottlenecks and necessitate architectural change to identify and decompose domain-specific and domain-independent elements.
• Systemic resilience takes on new urgency, extending beyond central HPC to include networks and distributed devices.
• Machine learning, modeling, and data assimilation apply concurrently at many sites and multiple levels, subject to energy and packaging constraints at the edge.
• We in HPC are not the only players in this space. Much as we learned and adapted much from the open source Linux community, we must learn from and embrace distributed data analytics from the sensor community.

As the name implies, the Big Data and Extreme-scale Computing (BDEC) project has been examining this nexus and the changing nature of scientific discovery enabled by the fusion of big data – from sensors large and small – machine learning, and computational modeling. It is a rapidly shifting environment, with new hardware and software ecosystems evolving rapidly.

Much of it remains largely unknown to the traditional HPC community, just as software containerization and machine learning were just a few years ago. That will change very soon. If you are attending SC17, come to the Birds of a Feather (BOF) discussion, late Wednesday, November 15.

A Literary Coda

In his book, Hell's Angels, the inimitable gonzo journalist Hunter S. Thompson once wrote about living on the edge – riding flat out – in the dark – on a motorcycle:

The Edge…There is no honest way to explain it because the only people who really know where it is are ones who have gone over. The others–the living–are those who pushed their control as far as they felt they could handle it, and then pulled back, or slowed down, or did whatever they had to when it came time to choose between Now and Later.

In supercomputing, we have always felt the need for speed, exploiting every new technology in the endless search for higher performance and greater capacity. It's time for the supercomputing community to come to the edge, where sensors and streaming data meet HPC, computational models, and machine learning.

There's lots of room at the edge for innovation and discovery. Let's not crash in the dark.

Related articles

Exascale Computing and Big Data: Time to Reunite

Effervescent Clouds: Nimble Computing Infrastructure

N.B. A version of this essay appeared as a guest blog post for the EU Cloud Lightning project

What's a cloud? The word means many things to many people. In true Airplane! movie parody style, one is tempted to say, "It's a visible mass of condensed water vapor floating in the atmosphere. But, that's not important right now."

Some might say that our cloud confusion is indicative of a fog, a low hanging cloud. Indeed, in public perception, the cloud is a vaguely understood and amorphous capability that lurks behind our smartphones, enabling electronic communications, web searches, social networks, e-commerce, and 24x7 news and streaming media. It is a mysterious "something, somewhere" that is only noticed when it is inaccessible.

For those of us in computing, the cloud is more than a buzzword or meme, it is a set of technologies that are transforming the provisioning and delivery of computing cycles, storage and services, with profound implications for government services, research discovery and innovation, and businesses large and small. Given the public confusion and sometimes overhyped marketing, it seems worthwhile to dispel the fog and illuminate a few compelling features of clouds.

In a very real sense, public clouds – those hosted by cloud providers for on-demand access – have much in common with the timesharing systems of the 1960s and 1970s, albeit at unprecedented scale. They shift the capital costs of provisioning, along with the expertise required for operation, from users to providers, allowing users to focus on their core expertise (e.g., research, business, government) and pay for only what computing they need, when they need it.

In turn, cloud elasticity allows servers to be provisioned and de-provisioned dynamically in response to changing interest and demand. One of the few things worse than having your brilliant idea ignored is having it fail publicly from too much attention. Many an organization has first been thrilled to see their product or service be reported in the press and go viral, only to be horrified when on-premise servers collapse under the load of success; this is known as the slashdot effect, named after one of the popular "news for nerds" web sites. Without cloud elasticity, the global phenomenon of Pokémon Go would not be possible.

Although not required, many clouds also virtualize the underlying computing, allowing multiple services and software stacks to be co-resident on the same physical hardware, further reducing costs. With Infrastructure as a Service (IaaS), Platform as a Service (PaaS), Software as a Service (SaaS) and containerization, developers can choose the level of abstraction, from low level to high level, needed to support existing and new applications.

By allowing innovators to focus on their ideas rather than computing infrastructure, the pay-as-you-go cloud model has made it easier, faster and cheaper to bring ideas to life, unleashing a second dot.com boom of cloud-based services, unprecedented citizen access to government data, and new approaches to scientific discovery via big data.

The latter has sometimes been called the fourth paradigm, where new sensors, data analytics and deep learning have shifted the research approach from "what experiment should I conduct" to "what insights can I extract from available data." It has also stimulated interest in convergence architectures for high-performance computing (HPC) and data analytics that combine the best elements of cloud and HPC capabilities. (See my essay, Exascale Computing and Big Data: Time to Reunite.)

Implicit in this client-cloud model is ubiquitous, reliable and inexpensive wired and wireless broadband services. Without broadband access, clouds are simply isolated sets of computers and data, unconnected to the world of mobile devices and the nascent Internet of Things. To further ensure reliability, most public cloud providers geo-distribute their pools of servers. Even private clouds (i.e., those operated by governments and companies for their own use) often geo-distribute their servers at multiple data centers. Should disaster strike at one site, services can be shifted automatically to another site, perhaps even in another country or continent.

Finally, the unprecedented scale of cloud deployments, with commercial data centers each costing over $1B, has stimulated new approaches to energy efficiency and cooling, component failures and reliability, customized and open hardware, software defined networks (SDNs) and software defined storage (SDS), and system provisioning and management. In short, we are in a time of radical change in the computing ecosystem, all against a backdrop of Moore's law limitations.

What's a cloud? It's an elastic, on-demand, often geo-distributed, frequently virtualized, flexible hardware, software and services infrastructure that powers the 21^st century knowledge economy. And that is really important right now.

Related articles

Exascale Computing and Big Data: Time to Reunite

Exascale Computing and Big Data: Time to Reunite

N.B. I also write for the Communications of the ACM (CACM). The following essay recently appeared on the CACM blog. In addition, you can find the video of an interview and the full text of the CACM article, co-authored with Jack Dongarra.

In other contexts, I have written at length about the cultural and technical divergence of the data analytics (aka machine learning and big data) and high-performance computing (aka big iron) communities. I have euphemistically called them "twins separated at both." (See HPC, Big Data and the Peloponnesian War and Scientific Clouds: Blowin' in the Wind.) Like all twins, they share technical DNA and innate behaviors, despite their superficial differences. After all, in a time long, long ago, they were once united by their use of BSD UNIX and SUN workstations for software development.

Since then, both communities have successfully built scalable infrastructures using high-performance, low cost x86 hardware and a rich suite of (mostly) open source software tools. Both have addressed ecosystem deficiencies by developing special-purpose software libraries and tools (e.g., SLURM and Zookeeper for resource management and MPI and Hadoop for parallelism), and both have optimized hardware for their problem domains (e.g., Open Compute hardware building block standardization, FPGAs for search and machine learning, and GPU accelerators for computational science).

Like many of you, I have seen this evolution firsthand, as a card-carrying geek in both the HPC and cloud computing worlds. One of the reasons I went to Microsoft was to bring HPC ideas and applications to the nascent world of cloud computing. While at Microsoft, I led a research team to explore energy-efficient cloud hardware designs and new programming models, and I launched a public-private partnership between Microsoft and the National Science Foundation on cloud applications. Now that I am back in academia, I am seeking to bring cloud computing ideas back to HPC.

In that spirit, Jack Dongarra and I recently co-authored an article for the Communications of the ACM on the twin ecosystems of HPC and big data and the challenges facing both. Entitled, Exascale Computing and Big Data, the article examines the commonalities and differences, and discusses many of the unresolved issues associated with resilience, programmability, scalability, and post-Dennard hardware futures. Most importantly, the article makes an impassioned plea for hardware and software integration and cultural convergence.

The possibilities for this convergence are legion. The algorithms underlying deep machine learning would benefit from the parallelization and data movement minimization techniques commonly used in HPC applications and libraries. Similarly, the approaches to failure tolerance and systemic resilience common in cloud software have broad applicability to high-performance computing. Both domains face growing energy constraints on the maximum size of feasible systems, necessitating shared focus on domain-specific architectural optimizations that maximize operations per joule.

Perhaps most important of all, there is increasing overlap of application domains. New generations of scientific instruments and sensors are producing unprecedented volumes of observational data, and intelligent, in situ algorithms are increasingly required to reduce raw data and identify important phenomena in real time. To see this, one need look no further than applications of machine learning to astronomy, which now include automated object identification. Conversely, client plus cloud services are increasingly model-based, with rich physics, imaging processing and context that depend on parallel algorithms to meet real-time needs; augmented reality applications are one such exemplar.

The explosive growth of Docker and containerized software management speaks to the need for lightweight, flexible software configuration management for increasingly complex and rich software environments. My hope is that we can develop a unified hardware/software ecosystem that leverages the technical and social strengths of each community. Each would benefit from the experiences and insights of the other. It is past time for the twins to have a family reunion.

A Sticky WCIT Indeed

Warning: This is a long post, reflecting the complexity and nuance of Internet governance.

Just before the 2012 holidays, the World Conference on International Telecommunications (WCIT), pronounced as “wicket” by the cognoscenti, concluded in Dubai. In the buildup to the WCIT, there was much handwaving and fearmongering, political position jockeying, and backroom negotiations by multinational companies, governments, non-governmental organizations (NGOs), technology policy wonks and political pundits. There were frequent stories in the trade and policy rags and the mainstream press, including the New York Times. How, you might ask, could such a seemingly obscure conference engender such an international frenzy?

In the spirit of full disclosure, I should reveal that I spent a good portion of the last three years working on this issue, while heading Microsoft's Technology Policy Group. I traveled the world, meeting telecommunications policy leaders, trade associations, companies and senior representatives of international governments, including ones from the United States Departments of State, Commerce and Federal Communications Commission (FCC). With that disclosure, as well as noting that what follows are my personal opinions, a bit of background and history is in order.

Telecommunications History

The International Telecommunications Union (ITU), which is a United Nations organization, organizes the WCIT. The ITU began as the International Telegraph Union, which suggests some its origins and history. Originally responsible for coordinating global use of the radio spectrum and, more recently, assisting in international assignment of satellite orbits, the ITU now operates under three units, the ITU-R (radiocommunications), ITU-T (standardization) and ITU-D (development).

As a UN organization, the ITU's members are countries, rather than individuals, companies or NGOs. Despite the one country, one vote governance model, there is multistakeholder participation in some informal aspects of the ITU. I was a frequent visitor to the ITU, and I participated in ITU retreats and CTO roundtables. I know the ITU leadership well.

Because major ITU policies and approaches, including tariffs, are codified as international treaties – the International Telecommunication Regulations (ITRs) – policy changes are fraught with all the complexities that accompanies any international treaty. Indeed, before the 2012 WCIT, the ITRs were last updated in 1988. The languid pace of treaty change is but one of many challenges posed by the ITRs. It has also brought into question the relevance of the ITU and its policies and approaches. The country voting model and UN culture do not match the freewheeling Internet world.

Internet Time

Looking back twenty four years to the previous WCIT takes one almost to prehistory in Internet time. The TCP/IP protocols were in place, but international networking was largely a research or private network curiosity and dialup modems defined the consumer network experience. The World Wide Web was still a gleam in the eye of Tim Berners-Lee; the Mosaic web browser had not burst on the scene; the dot.com frenzy was yet to come; mobile phones were rare, expensive and bulky; and plummeting international telephony charges due to IP-based audio and video calling were still over the horizon. In short, the telecommunications world of 2012-2013 is radically different from the staid landscape of 1988.

The technological and economic changes wrought by the Internet in those fourteen years are equaled by the social and cultural changes. Geographically and politically separated regions are now digitally interconnected in ways unthinkable just a few years ago. The global flow of information has challenged governments, companies, NGOs and individuals to adapt legal frameworks, regulations, technical operations, social expectations and national security operations. (See Globally Connected, Globally Worried and Information Privacy: Changing Norms and Expectations for a bit of perspective.)

Multifaceted Internet Governance

One can parse Internet governance challenges in several ways: technical, legal and economic, social and ethical and national security. Let's begin with the technical issues, which are perhaps the simplest. Operating the global Internet effectively requires de facto adherence to an evolving set of technical standards. The Internet Corporation for Assigned Names and Numbers (ICANN) manages the domain name system (DNS) and IP address blocks on behalf of the international community.

Although not without controversy, ICANN and its sister organization, the Internet Society's Internet Engineering Task Force (IETF), have generally been effective in creating a multistakeholder governance model for Internet operations. However, the tizzy over generic top-level domain names (gTLDs) and intellectual property protection has heightened government desires to control ICANN.

Shifting to economics, companies, countries, NGOs and privacy advocates all worry about the transnational flow of data, differing laws and intellectual property protections, jurisdictional constraints and safe harbors. If you are a Kenyan national working for a German company with operations in Brazil, and you travel to India, just whose laws govern you and your company? The answer is murky, but generally all of the above, plus some others.

Nevertheless, Internet service operators must respond to civil and criminal legal requests for data every day. For those subject to multiple legal jurisdictions, it is sometimes impossible to satisfy the laws of countries involved. For those interested and suffering from insomnia, I highly recommend reading the Bank of Nova Scotia case (United States v. Bank of Nova Scotia, 740 F.2d 817 (11th Circuit, 1984).

Mixed with all of these legal and economic issues are elements of international trade, protectionism and economic development. The rise of cloud computing, whose economics have driven massive scale and data consolidation, has exacerbated these concerns about extraterritorial jurisdiction and control worldwide, particularly when U.S. companies dominate cloud computing and many fear the U.S. PATRIOT Act.

Then there are the crucial issues of human rights and free speech. How does one reconcile global communication, which brings differing international norms and expectations for freedom of speech into direct and day-to-day conflict? It is more than an abstract question for millions of Internet users, and one with no simple answers.

One can debate the ethics, human rights records and legitimacy of certain governments, but the sovereignty of nations and their right to establish laws within their territory is an unquestioned aspect of international law. I have seen senior representatives of governments with widely divergent views on freedom of expression all agree that their governments have a critical role to play in Internet governance. Those same representatives differed markedly in their delineation of appropriate governmental roles, their definitions of free speech and its appropriate limitations.

Finally, there are issues of national security, information security and cyberwarfare that are beyond the scope of this essay. In a knowledge economy, information is advantage, and economic or technical disruption via the Internet itself can be a form of low-grade warfare. Likewise, with military capabilities themselves increasingly dependent on smart, network-connected weapons, they are themselves objects of defense and attack.

Back to the WCIT

This interplay of technology, international law, economics and trade, social norms and human rights, and national security is a witch's brew of complexity, with diverse stakeholders and perspectives. Many of them encamped in Dubai at last December's WCIT.

Thus, it is not surprising there was acrimony and controversy, with claims and counterclaims; one could hardly expect otherwise. Central to these debates were concerns, some say unfounded, though others disagree vehemently, that the U.N., via the ITU, might assume greater control of Internet governance, shifting from the multistakeholder model toward greater centralization and government control.Some of this was also tied up in the ITU’s own search for new relevance.

What emerged from two weeks of painful negotiations was what can best be described as an uneasy peace. After much debate, 89 of 152 countries signed an amended version of the ITRs. The U.S., Japan, Canada, Germany, India, and the U.K. were not among the signatories, objecting to attempts to curtail multistakeholder governance. In short, it was an ugly mess.

Futures

What's next? The global community is strongly divided along ideological lines. Thus, we are likely to see even greater dichotomy in government intervention, more uncertainty in international law, and limits on global information flow. Despite this, I believe the Internet will continue to grow and evolve organically. Too many stakeholders want that to happen, and their voices must be heard.

Thus, I am a strong proponent of the multistakeholder model. I do recognize, though, that governments have an important role to play, just as they do in other domains. This is a messy process, and it will undoubtedly continue that way. Such is the nature of debate.

Related articles

WCIT collapses: US, UK, allies refuse to sign treaty after Africa wins floor vote

WCIT and Internet Governance: Harmless Resolution or Trojan Horse?

Little’s Law in the Exascale Era

N.B. I also write for the Communications of the ACM (CACM). The following essay recently appeared on the CACM blog.

A substantial fraction of my professional life has been devoted to analyzing and optimizing the dynamic behavior of some of the world's largest high-performance computing (HPC) systems. During the past thirty years, spanning my time at Illinois, North Carolina, Microsoft and now Iowa, my students, post-doctoral associates, research staff and I have used and developed many performance, energy consumption, and reliability instrumentation and measurement tools and techniques, embodied them in scalable data capture and reduction software, and created correlation tools that connect measurements to potential remediation opportunities. Multiple instantiations of these tools have been released as part of the Pablo performance analysis toolkit and Autopilot toolkit for adaptive performance control.

Just a Couple of Weeks

Most of this work originated in a challenge from my Ph.D. thesis supervisor, Herb Schwetman, issued while we were both at Purdue. After listening to me expound excitedly about the virtues of yet another interconnection network for parallel systems, he challenged me to prove that the object of my excitement would deliver higher performance than other alternatives. I dutifully promised him that I would spend a few weeks on the performance analysis problem, then return to the core parallel systems design question.

What ensued changed my thesis direction dramatically, as the two-week project took three years to complete. Over thirty years later, I find myself still working to answer generalizations my advisor's basic question: "How fast is system A compared to system B and what are the bottlenecks in each?" These are beguilingly simply questions, but they cut to the heart of system design and optimization. They are also especially pertinent as we consider the design of putative exascale systems with billion-way parallelism, and millions of components that must operate reliability, all while consuming tens of megawatts of power.

It Seems Simple, But It Is Not

Beneath all this complexity is a simple, yet subtle law that that relates performance (throughput), delay (response time) and number of interacting units (customers). It has been my friend and nemesis throughout my research career.

Generally known as Little's Law, it relates throughput (X), response time (R) and customers (n) in a stable system in a linear way: n=XR. This general relationship holds for computers, airline reservation systems, store checkout lines, or any other system where entities (customers or computer jobs) compete for service from a resource-limited system.

Little's Law also says there is no free lunch. Optimizing for high system utilization leads to large wait times for customers, as anyone who has waited on a busy telephone reservation line knows all too well. Conversely, ensuring short waiting lines requires system utilization to be low. Simply put, one must choose between high system efficiency and short customer wait times. One cannot (in general) have both.

As simple as this may seem, in reality, the best optimization strategy depends on many other constraints and the broader environment. For many systems, the relative costs of the components providing and receiving service determine the appropriate optimization axis. When humans are involved, their psychology and expectations must be managed appropriately. That is why many telephone response lines provide waiting time estimates and periodic feedback during the wait. Those of us who have managed batch-scheduled HPC systems also know that users and system operators constantly balance system utilization against the product of queue wait time and parallelism.

Coming Full Circle

Exascale computing brings many of these issues into focus in new ways. When a multiple megawatt energy spike is imposed on the electrical grid, utility operators notice. Thus, exascale system energy requirements may constrain when parallel job execute, as jobs with high floating-point computation intensity draw more energy those dominated by pointer chasing. Likewise, component reliability may limit maximum parallelism within exascale jobs; the mean-time-to-failure (MTBF) of processors, memory and storage systems, and interconnect elements really matters. Although more job parallelism normally reduces execution time, component reliability bounds effective parallelism and maximum computation time without either resilience or recovery mechanisms.

Performance, delay and parallelism at large scale: Little's Law speaks to all of these issues and more. When performance optimization, reliability requirements, and energy management are convolved with component costs, device physics, system software services and application characteristics, the constraint-based optimization problems become dauntingly complex. Their solution is a bit more than a two-week project, especially at exascale.

Related articles

HPC, Big Data and the Peloponnesian War

John Barr on the European Strategy for HPC and Exascale

China moves to beat U.S. in exascale computing

Exascale unlikely before 2020 due to budget woes

Research {preposition} Infrastructure

N.B. I also write for the Communications of the ACM (CACM). The following essay recently appeared on the CACM blog.

Computing research and advanced computing infrastructure – each is dependent on the other in a myriad of subtle and complex ways, yet each is profoundly different in culture, process, skills and metrics. As a computer scientist who has often conducted research on, toward and with large-scale infrastructure, both for computational science and for big data analytics in clouds, and as someone who has overseen large-scale, production infrastructure, both as director of a national facility and as a university CIO, I have seen this creative tension and intellectual dichotomy firsthand.

Convolving the two inappropriately can be frustrating at best and disastrous at worst. Conversely, when each of the constituent communities acts appropriately, there are enormous benefits to both. As we think about next-generation capabilities and the balance of investment in basic research and deployed infrastructure for data analytics and computational science, it is salutatory to consider some insights from our experiences, reflected, sometimes unthinkingly, in the language we use. The prepositions, such simple words, connecting the two, make all the difference. About, on, before, beyond, for, toward, with: they are all different, and we confuse them at our peril.

Metrics and Culture {before and beyond}

First and perhaps most obviously, the metrics of success in research and infrastructure differ markedly, though both operate in a highly competitive world. The research culture rewards novelty and new ideas, intellectual risk taking, rapid prototyping and experimentation, and early publication. At its best, research pursues basic questions and long-term outcomes. Students and post-doctoral associates learn as they are doing, and the resulting prototypes are often buggy and usable only by the researchers themselves. Often, though not always, the experimental scale is small, limited by budgets and staff, with localized consequences for failure and little expectation that others will depend on the outcomes.

Conversely, the production infrastructure culture, whether for big data analytics or computational science, rewards operational stability; quality of service, both technical and human; and long-term compatibility, continuity and interoperability. It also prizes consultation with others, operational experience, planning, process, testing and documentation. Operationally, the production culture is inherently conservative, for failure at scale can affect thousands to millions of users. Thus, change for change's sake is rare. Rather, adoption and deployment are driven by shifting costs and technological capabilities, user expectations and needs and by competitor behavior. These competitors can be in the business world or in the world of international scientific instrumentation and facilities.

Collaborations {on and with}

Computing researchers rarely build and successfully operate production infrastructure; they lack the skills and production mindset. Likewise, infrastructure operators rarely ask the fundamental questions; they rightly focus on operational priorities and their demanding user community. Yet each can benefit from the other. Lest these characterizations seem unnecessarily pedantic or abstract, consider again the prepositions and their implications.

Conducting research on or with production infrastructure requires collaboration and deep partnership with the infrastructure operators. Moreover, under no circumstances can it compromise infrastructure security, reliability or quality of the service. Thus, there must be a clear value proposition for the infrastructure operator and its user community, other than just the goodwill of the computing research community. The payoff might be optimized production infrastructure based on experimental insights, data to shape next-generation vendor capabilities, or better algorithms for data analysis, but it must be weighed against the risks.

As a personal example, my research group spent years exploring performance measurement and analysis techniques for large-scale HPC systems, using systems at the NSF and DOE supercomputing centers for testing and measurement. Because these centers supported a national base of computational scientists, it was critical that my team's prototype tools not adversely affect either the HPC system or its users. Because these tools typically captured data at many levels, from hardware, system software, runtime libraries and applications, this was no small task. It required careful planning, testing and scheduling. The rewards, however, were substantial – data and insights at a scale simply unattainable in the laboratory and influence on next generation capabilities.

When I became director of the National Center for Supercomputing Applications (NCSA), one of those NSF-funded national facilities, I found myself on the other side of the equation, responsible for deploying and operating a portion of the national research infrastructure. This sharpened my awareness of fiduciary responsibility and the risk asymmetry. Make no mistake; in research collaborations, the infrastructure operator bears the majority of the risk. These span a wide range, but include compromised functionality from faulty experiments and security or privacy breaches due to data analysis.

There is a reason firms with web search logs and other social network data rarely release even sanitized and anonymized data sets for academic research. Even inside a company, production infrastructure and data are accessible to only a few.

Ideas and Implementation {for and toward)

What are the best ways to transfer innovative ideas and form mutually beneficial partnerships? It is neither easy nor simple. (See Technology Transfer: A Contact Sport for a few thoughts on this topic.) Crossing the "valley of death" from idea to realization requires understanding the entire set of optimization constraints and considerable patience. Indeed, repeated U.S. National Academies studies have shown that the time lag to translate basic research ideas in computing into a billion dollar industry is over a decade. (See the famous "tire tracks" diagram.) From supply chains and availability through competitive pressures and time to market to ecosystem economics and labor force, these constraints may seem prosaic or mundane to the research community, but they are very real.

As a researcher, this often leads to a difficult lesson, that of idea valuation. Simply put, an intellectually interesting idea is not necessarily a helpful idea or even an economically valuable one. This is an often-painful realization. I have seen many cases where a new and improved algorithm or hardware design was rejected because it was not backward compatible with existing infrastructure; it solved a problem but not the most pressing one; it was too costly to implement; it would have required retraining and a cultural shift; or it was simply the right idea at the wrong time.

As the scale, scope and influence of computing grow ever larger, the stakes are high, for both computing researchers and those who build and deploy large-scale computing infrastructure. Each needs to understand the constraints and needs of the other to find common ground.

Lessons for HPC and Big Data

What does all of this mean for today's world of HPC and big data? First, we must continue to support the basic research needed to advance the state of the art. There are deep and unresolved research questions regarding resilience and reliability, energy efficiency and performance, programmability and expressivity, privacy and security, among many others, which span algorithms, software, hardware and architecture.

Second, advanced HPC and big data infrastructure enable discovery across all scientific and engineering domains. There are important problems in biology and medicine, physics and materials sciences, weather prediction and climate modeling, and business and consumer services that depend on continued advances in our large-scale infrastructure. It is crucial that we continue to fund and deploy such capabilities at the largest scales possible.

Third and critically, we should not confuse the two. The two communities, computing research and research infrastructure for science and engineering, can and should learn from the other, but they are different, with divergent constituencies and cultures; disjoint timescales and needs; and complementary outcomes and rewards. One cannot be sacrificed for the other.

Remember the prepositions. They really matter.

It Is Fall: There Must Be Clouds

Today, I testified to the Subcommittee on Technology and Innovation of the U.S. House of Representatives Committee on Space, Science and Technology. The topic of the hearing was "The Next IT Revolution?: Cloud Computing Opportunities and Challenges." At the hearing, I was joined by Mike Capellas (CEO, VCE), Nick Combs (Federal CTO, EMC), and David McClure (Associate Administrator, GSA).

This hearing drew on some of the content from the TechAmerica Commission on the Leadership Opportunity in U.S. Deployment of the Cloud (CLOUD²), on which I had the privilege to serve as the commercial vice-chair. In addition, my own remarks were augmented by Microsoft's cloud engagement experiences with the U.S. National Science Foundation and other research funding agencies. (See Innovation Via Client Plus Cloud: Microsoft-NSF Partnership for background.)

My complete written testimony will ultimately appear on the Subcommittee web site here, and the hearing charter, the subcommittee press release, and the hearing video capture most of context. However, there are a few points from my testimony that seem appropriate to repeat here. These concern research empowerment via the cloud, support for basic research and education and broadband expansion.

Accelerating Scientific Discovery for Research via the Cloud

Throughout the history of science, data has been scarce and precious. Indeed, the modern scientific method is defined by a careful cycle of hypothesis and experiment, which gathers experimental data to test the hypothesis. In a few short years, scientists and engineers have gone from scarcity to an incredible richness, necessitating a significant change in how they manage and extract insight from all this data. In a parallel shift, many of our scientific, engineering and societal questions increasingly lie at the intersections of traditional disciplines.

Increasing data volumes and the complexity of collaboration on interdisciplinary problems are challenging our historical approaches to discovery and innovation via computing. Most researchers and research institutions are ill-prepared for the large-scale computing infrastructure management challenges posed by large data sets and complex models. The cloud and associated applications and tools offer a possible solution to this challenge by letting scientists be scientists.

The U.S. government can accelerate this transition by encouraging the purchase of cloud services, in addition to the acquisition of local IT infrastructure, and by supporting new tools that facilitate distributed collaboration and simplify access to multidisciplinary scientific data. As I have noted before, Microsoft is acting on this belief, working in partnership with the National Science Foundation.

Fostering Continued Support for Computing Research and Education

Today's cloud technology is derived from basic computing research conducted over the past four decades. To ensure that the U.S. continues to remain at the forefront of cloud technology, continued investment in basic research is critical. There are deep and open questions in areas as diverse as the future of silicon scaling and system-on-a-chip design, energy-efficient systems, primary and secondary storage, data mining and analytics, wired and wireless networks, system resilience and reliability, privacy and security, and user interfaces and accessibility, to name just a few. Insights and innovations from this research will spawn new companies, create jobs and reshape our future.

In addition to continued research investment, it is critical to support the pipeline that produces researchers, and others who will able to invent new uses of the cloud and information technology. The U.S. Bureau of Labor Statistics estimates that the computing sector will have 1.5 million job openings over the next 10 years, yet the number of graduates receiving Bachelors, Masters or Ph.D. computer science degrees is far short of that. In addition, we must strengthen the quality of and access to computing education at all levels. Consistent with these concerns about the IT workforce and computing education, Microsoft is a founding member of the Computing in the Core coalition, which supports computer science education, particularly at the K-12 level.

Broadband: The Need for Speed

The web and cloud services depend on broadband communications. Without them, service and information sharing are impossible. Concomitantly, ensuring reliable wired and wireless connectivity, with adequate bandwidth and latency, is critical to ensuring successful adoption of the cloud and realization of its benefits. The phenomenal growth of digital data, the rise of streaming media services, and the explosive growth of Internet-connected devices are all straining our nation's broadband infrastructure.

It is critical that we continue to design and deploy new backbone networks that support higher data rates, develop and deploy new protocols and infrastructure for the next generation of wireless networks and define the global standards that will shape the future of the globe-encircling cloud. We must also remember that digital access to information and services is increasingly the enabler of economic competitiveness, of lifelong education in a rapidly changing world, and of government efficiency and service delivery.

21^st Century Innovation

This is an exciting time, when the future becomes the present. Computing can be a great equalizer, providing access to the world's knowledge base to individuals, anywhere, anytime; empowering entrepreneurs and companies large and small to sell their products and ideas globally; and enabling scientists and engineers to discover and innovate in ways that will define the future.

Battling Evil: Dark Silicon

As a boy, I remember reading all the superhero comic books, from the standard Superman and Batman to the Green Lantern, Thor, The Flash and Spider-Man. Some superhero stories and backstories were better than others, but all were escapism from small town life. For me, Thor was always a favorite, though I am not quite sure why. Maybe I just liked the head gear! The summer "action movie" season is upon us, and several of these superheros will grace the screen in new Hollywood fantasies, battling the forces of evil.

All of this made me think about the resident evil now facing us in computing – the rise of dark silicon. The very phrase sounds ominious and it is, for I believe it will profoundly reshape how we think about computing in the next decade. But before we talk about the grand denouement where our Justice League of silicon process experts, computer architects and software gurus do battle with dark silicon, perhaps we should begin with a bit of backstory.

Moore's Law and Silicon Scaling

We are all familiar with Moore's Law, namely that the number of transistors in a given area doubles roughly every two years. It is worth noting that Gordon Moore never really claimed this was a law, merely that it was a statistical truth based on observational data and technology trends. It is also interesting how a rather informal 1965 Electronics paper, entitled "Cramming More Components onto Integrated Circuits," had such impact. Over the past thirty years, the "law" has held true due to the hard work and creativity of a great many people, as well as many billions of dollars of investment in process technology and silicon foundries.

For a detailed understanding of why reducing transistor feature sizes really has been possible, I highly recommend reading the original, seminal paper by Robert Dennard, Fritz Gaensslen, Hwa-Nien Yu, V. Leo Rideout, Ernest Bassous and Andre LeBlanc, entitled "Design of ion-implanted MOSFET's with Very Small Physical Dimensions." It appeared in the October 1974 issue of the IEEE Journal of Solid-State Circuits, and it defined a scaling process that has driven the semiconductor industry ever since.

Simply put, Dennard scaling says that reducing the length, width and gate oxide thickness of transistor features by a constant k results in transistors that are k² times smaller, k times faster, and dissipate k² less power. This was Moore's Law made manifest and has let us move from feature sizes measured in micrometers to nanometers. Since the elucidation of Dennard scaling, all has been well in Metropolis, at least until very recently.

Power and Performance: Evil Lurks

As transistor sizes continued to shrink and clock rates continued to increase, we have seen the rise of high-k dielectrics and silicon on insulator (SOI) to address static current leakage. In that spirit, one of my colleagues recently remarked that transistors are now less like solid doors that keep out the cold (current flow) unless open and more like screen doors that always have some air flow (current leakage) even when closed.

Transistor density, clock rates and chip sizes have also made heat dissipation an ever bigger problem. While at Intel, Pat Gelsinger once remarked that unless things changed, chip temperatures would approach that of the sun's surface, an obvious absurdity only possible in the comic books. All of this has led to multicore designs, bounds on clock rates for most microprocessors and decreases in operating voltages.

We have also seen new semiconductor processes and transistor design options to reduce leakage current. (Intel's recent FinFET announcement is one such example.) Although I expect to see transistor feature sizes continue to shrink, with sub-11 nanometer feature sizes achievable, it will not be easy, nor will it be cheap. Lest there be any doubt, the cheap part really matters, for it is the economics that will determine if scaling continues.

These semiconductor challenges have been mirrored by an increasingly empty bag of general purpose architectural tricks. Almost all of the techniques once found in supercomputer processor designs – superpipelining, scoreboarding, and vectorization – are now in mainstream microprocessors. It is not obvious that we can extract more parallelism while maintaining the illusion of sequential execution. In fact, the evidence is substantial that we cannot.

Battling Evil: Dark Silicon

That's the backstory. Now let's turn to the true evil lurking among us, dark silicon. If one considers process scaling trends and limits on heat dissipation, the conclusion is rather obvious. We soon will have (and in many cases already do have) chips with more transistors than can be concurrently activated. The practical implication is that most of the chip silicon will be "dark" (unpowered) most of the time.

If it's dark most of the time, is it of any value? Yes, but only if we embrace our fears. This is the true end of the hegemony of von Neumann computing and the birth of a new era, one based on chips with a diverse set of specialized functional units, accelerators and tailored architectures that each target specific tasks. The mobile world has already learned this lesson, for functional specialization increases energy efficiency (more operations/joule) and allows designers to mix and match the set of functional units to target domains. Arguably, heterogeneous multicore is one, very tentative step in this direction for client and server computing.

This transition to diverse functional units, accelerators and tailored architectures will also challenge many of our assumptions about programming models and system software, as well as the economics of chip design. In this new world, hardware/software co-design becomes de rigeur, with devices and software systems deeply interdependent.

It is a different ecosystem, with different cultures and different processes. It means far less general-purpose performance increases, much more diversity in the hardware, elevation of multivariate optimization – power, performance, reliability – in programming models, new system software resource management challenges and a plethora of new applications.

More generally, with lots of silicon real estate (dark silicon), chip designers and architects have the space to place functional blocks that are used only part of the time. However, at some point (and we are arguably close to that point now), we will have deployed those functional units that cover the common tasks and adding more will have possibly deleterious effects due to larger chip size. That potentially drives the economics to smaller chip sizes, which, given foundry capacity, leads to lots of small chips. That is the collision of the big chip server world with the small chip mobile and Internet of Things world, a battle already raging.

To address these challenges and battle dark silicon, we need new ideas in computer architecture, system software, programming models and end-to-end user experiences. It's an epic struggle for the future of computing. Giants are battling, and not all will survive, nor will the survivors emerge unscathed. Heroes will also emerge, sometimes from unexpected places.

I think I left my hammer, Mjölnir, around here somewhere. Do you feel your spider sense tingling?