|DOWNLOAD THE REPORT|
Carolyn Fu is a PhD Candidate in the Economic Sociology program at MIT’s Sloan School of Management. She researches innovation strategy, exploring how firms can best leverage their ecosystems to discover new sources of value. She previously worked in defense research in Singapore, and has a Bachelor’s in Mechanical Engineering from the University of Pennsylvania and a Master’s in Mechanical Engineering from Stanford University.
Lars Frølund is the Research Director of MIT Innovation Initiative and a Visiting Fellow at the MIT Sloan School of Management. His research focuses on the success factors for university-industry partnerships, innovation ecosystems, and mission-driven research and innovation agencies such as DARPA. He is on the board of the Danish Innovation Fund and was a Fulbright Scholar at MIT in 2016/17.
Fiona Murray is the Associate Dean of Innovation at the Massachusetts Institute of Technology (MIT) Sloan School of Management, William Porter (1967) Professor of Entrepreneurship, and an associate of the National Bureau of Economic Research. She is also the co-director of MIT’s Innovation Initiative. She serves on the British Prime Minister’s Council on Science and Technology and was made Commander of the British Empire (CBE) for her services to innovation and entrepreneurship in the UK. She is an international expert on the transformation of investments in scientific and technical innovation into innovation-based entrepreneurship that drives jobs, wealth creation, and regional prosperity.
Governments and foundations around the world are urgently seeking strategies to optimize their investments across a range of distinctive missions targeted towards societal challenges. How should such investments be made, from early R&D spending to later-stage acceleration, to most effectively fuel the full lifecycle of innovation from ideas to impact? We answer this fundamental and urgent question by analyzing the boundary conditions of the most influential model of mission-driven innovation thus far, the much lauded Defense Advanced Research Project Agency (DARPA). We argue that the highly successful DARPA model is best suited to pursuing high-risk, high-reward research investment when applied to a targeted problem space that can be matched to a dense innovation ecosystem of potential solution providers. Such an environment allows for the identification and selection of the most promising avenues of research, from which extraordinary breakthroughs can emerge to address critical mission requirements. DARPA’s success therefore relies not only on the amount of funding available or the application of the agency’s specific system of complementary management practices, as has been the focus of prior analyses, but also on the nature of the innovation ecosystem to which it is being applied. Based on these insights, we develop the notion of four different Mission Arenas defined along two dimensions: the scope of the problem space and the density of potential solution providers within it, as a framework to understand when the DARPA model can be most usefully applied. We describe how some of these Mission Arenas can be actively managed towards a form more amenable for the DARPA model. We then focus on the Mission Arena that represents a nascent innovation ecosystem (e.g. an emerging research area where the scope of the problem is broad and solution providers are few) and argue for an entirely different approach to high-risk, high-reward research investment altogether. We introduce the Ecosystem Growth Model, which transforms the DARPA model from one focused on strategic selection to one focused on strategic growth instead, by emphasizing program iteration, solution provider incentivization, portfolio integration, and organizational bandwidth.
As nations prepare to mount a robust response to the complex challenges ahead of us, ranging from climate change to Covid-19 preparedness, it is timely to understand the most effective way to organize mission-driven innovation. Governments around the world (both national and supra-national) and foundations are urgently seeking strategies to optimize their investments across a range of distinctive missions and mission time horizons. How should investments be made, from early R&D spending to later-stage acceleration, to most effectively fuel the full lifecycle of innovation from ideas to impact? We must learn from history to determine the boundary conditions that have helped to shape the success and failure of such efforts in the past, in order to understand how we should guide our efforts going forward.
In this paper we focus on the much lauded Defense Advanced Research Project Agency (DARPA) to understand its organizational characteristics, and more importantly, assess the boundary conditions that characterize its much heralded mission-driven approach. Specifically, we examine how nascent innovation ecosystems can be nudged towards a form more amenable for the DARPA model.
DARPA’s model has long been considered to be the gold standard of mission-driven innovation management. Founded more than 60 years ago, it remains a touchstone of the conversation for thought leaders from Dominic Cummings to Peter Thiel, who uphold its famed high-risk, high-reward approach as the panacea for mission-driven R&D investment today. Looking at DARPA’s track record, it is easy to see why it has been so widely lauded. The military research agency has been credited for the creation of numerous transformative technologies, including the internet, GPS, Siri, the personal computer, lasers, and autonomous navigation, to name just a few. By creating a portfolio of audacious bets on promising yet unconventional early research ideas, DARPA – an organization designed to fulfill difficult, mission-critical objectives – made all of these breakthroughs possible.
Inspired by DARPA’s success, many organizations have sought to recreate its model (or parts of it); the Wellcome Trust recruited former DARPA Director Regina Dugan to spearhead its “LEAP Fund” to accelerate global health innovation. In Europe, a “civilian DARPA” is being implemented via the European Innovation Council and, more recently, Ursula von der Leyen, President of the European Commission, announced the ambition to create a European BARDA (Biomedical Advanced Research and Development Authority). In the United Kingdom the role and likely effectiveness of a proposed UK “ARPA” is the source of considerable debate.
Despite such desires to apply this acclaimed mission-driven model around the world, one is hard-pressed to find evidence of successful imitation thus far. The EU’s $1.3 billion USD investment in the “Human Brain Project”, which sought to “accelerate brain research, brain medicine and brain-inspired technology”, has led critics to conclude that that it “did not succeed”, and to wonder if this failure was due to “poor management,” or if “something [is] fundamentally wrong with Big Science”. Closer to home, DARPA itself has floundered in recreating its own success across other parts of the U.S. government. For example, ARPA-E, which applies the DARPA model to tackle high-risk, high-reward projects in the energy sector, acknowledged its challenges in transitioning projects to commercialization. On the other hand, with a mission in national security that is distinctive from defense, I-ARPA (the ARPA for Intelligence) has, on occasion, been considered to be one of the government’s most creative agencies.
What might explain the fact that these organizations have yet to live up to the standard established by its DARPA predecessor? Have we perhaps misunderstood the factors that actually enabled DARPA to be successful? Or was the success of the DARPA model merely a function of time and space, driven by the idiosyncratic and un-replicable practices of a small number of people? As important, what role did the wider defence ecosystem – comprising solution providers (in the jargon of DARPA they are called Performers) ranging from large corporations, to start-ups, to risk capital to universities – play in facilitating DARPA’s outcomes? To pose the question more academically, what are the boundary conditions and underlying factors that have allowed for DARPA’s success?
With these questions in mind, we take a closer look at some of DARPA’s famed successes, and contextualize them within the organizational system in which they were delivered, and as importantly, within the innovation ecosystem(s) in which they were able to flourish. Through this, we will begin to understand the ways in which today’s efforts to replicate DARPA have been confounded by the complexity and scale of the innovation ecosystems within which mission innovation is now taking place. DARPA’s success relies not only on the amount of funding available or the application of the agency’s specific system of complementary management practices as has been the assumption thus far, but on the nature of the innovation ecosystem to which it is being applied. The success of DARPA’s model of high-risk, high-reward research investments rests on the ability to select the most valuable research avenues from a rich ecosystem of options. Thus, a dense innovation ecosystem of potential solution providers is in fact DARPA’s boundary condition – its condition sine qua non.
From these insights, we will then learn how to adapt the design of innovation funding approaches for contexts that do not meet this boundary condition, and see how nascent innovation ecosystems can be nudged towards a form more amenable to DARPA’s selection model. Hence this paper lays the critical foundations of revised strategies to most effectively ensure that spending on high-risk, high-reward research supports progress on today’s mission critical challenges, regardless of the nature of the innovation ecosystem in which they are being solved.
2. UNDERSTANDING DARPA AND ITS CONTEXT
DARPA’s successful outputs have been widely documented, and yet the specific features of the agency’s organizational system that have enabled them remain less well understood. Attention is usually drawn to its sizable $3 billion annual budget for research funding, leading aspiring imitators to assume that it is simply a matter of allocating sufficient funds – “throwing money at the problem.” Alternatively, others have focused on the role played by its highly autonomous program managers, who independently select research areas to develop and projects to fund. In the subsequent sections, we will gain a more holistic insight into DARPA’s organizational system that has enabled its success, and critically, the wider environment in which this has been accomplished.
2.1. DARPA’s Organizational System
Azoulay et al. (2019) provide a comprehensive breakdown of the key management capabilities that constitute what has come to be termed the “ARPA model” of innovation. They identify the following four complementary features of the organizational system:
a. Bottom-up program design
Program Managers (PMs) are the backbone of the DARPA program. The Agency encompasses about one hundred PMs (roughly $30M per year per PM), who each run a roughly 3-year program with a research agenda of their own design. PMs design their programs by identifying a need or challenge within the overall defence mission, then defining a “technological white-space” – an area in which little research is currently being done, but if filled, could enable significant progress in meeting the mission-critical need. PMs identify such areas by interacting closely with military users (i.e. problem owners) who clarify mission requirements, as well as with academics who are familiar with the state of the art who can clarify research-led opportunities in the potential solution space. This often takes place through meetings, community engagement and curated conversations across stakeholders. As they are typically hired from leading academic institutions or from successful military careers, PMs often leverage their own background in connecting with relevant solution providers.
b. Discretion in project selection
Having identified a useful technological white space to which they can contribute, PMs then undertake a selection process to pick an initial set of projects which, if developed further, could help to fill this important gap. Using their own expertise to guide selection, they have discretion to allocate funds to academic researchers who propose promising solutions to military problems, as well as to contractors (in start-ups as well as larger corporations) who can prototype potential solutions. Through this process, PMs serve to integrate ideas, build teams and leverage resources and solution providers across an ecosystem. Their ability to not only select but also co-create and curate projects is essential as they build a portfolio that can meet critical mission needs.
c. Active portfolio management
As the projects progress, PMs are able to redistribute funding across the portfolio. This capability – often referred to as active portfolio management – is generally achieved using a “real options” approach, where PMs continuously reallocate funding so as to maintain a balanced portfolio of risk and reward across projects. PMs and recipients establish specific milestones against which to assess a project’s progress, based on the mission need established at the program’s outset, and use these metrics at regular junctures to decide if a project’s risk is still worth its potential reward. Projects are thus continued with more (or less) funding, stopped, or redirected. Constant readjustment ensures that system resources are always being allocated as strategically as possible.
d. Organizational flexibility
PMs are given flexibility to manage programs and achieve their objectives. As a whole, DARPA reports directly to the Secretary of Defense, and is thus given a significant latitude in setting its own targets, developing programs and methods for achieving them. In short, DARPA has a high degree of ‘freedom to operate’ within the wider innovation system of the DoD. Internally, DARPA maintains a flat organization, with the PMs only two steps removed in management from the DARPA director. PMs are hired outside of regular civil service requirements, as are the contractors/researchers whom they work with on research and prototype development. This ensures that the DARPA system can attract the necessary talent to design and run their research programs, and PMs can easily acquire the resources they require. In addition, PMs are hired with appointments of four to five years which ensures turnover and fresh thinking, and a certain amont of time pressure to achieve the objectives of a program.
What makes these four features of the DARPA organizational system particularly effective is the fact that together, they “comprise a bundle of complementary practices” (Azoulay et al., 2019). For example, one might note that without sufficient organizational flexibility, PMs would be hard-pressed to start and stop projects at will; or that without the ability to change project funding allocation over time, PMs would find it far more onerous to select suitable projects at the outset and likely build a more low risk (low reward) portfolio. In particular, without the specification of a clear mission objective during program design, subsequent program management becomes much more difficult – making it harder to identify where the “technological white space” lies, what resources might help to fill it, or the extent to which ongoing projects are making progress towards it.
Such complementary, mutually reinforcing activities, that build toward a successful mission objective, have been described as DARPA’s “right-to-left” model. PMs identify a mission objective on the right, which then guides all of their key activities on the left – namely, selecting projects based on the extent to which they are likely to satisfy the objective on the right, and actively manage projects along the way (e.g. stopping or further funding them) depending on their progress toward this objective. The better specified the right-side objective that is established during the process of bottom-up design, the easier it becomes for the PM to later assess the progress of different research projects relative to that objective.
2.2. The DARPA system in practice – GPS
DARPA’s development of the miniaturized GPS provides an example of this right-to-left model in action – specifically, how a clear objective was established, and how projects were subsequently assessed against it. DARPA’s initial interactions with troops on the ground indicated a clear need for a hand-held version of the early satellite-based navigation technology that had thus far proved too heavy to bring on missions. Program Manager Dr. Sherman Karp thus established the goal of the program as “realizing a battery-operated, hand held receiver with military P-code capability”.
With miniaturization as a key objective, the program was able to select research projects that focused on digitizing the GPS signal, enabling it to be fabricated on miniaturized semiconductor chips to replace multiple analogue hardware components. Importantly, against a specific weight limit of 10lbs, DARPA was able to find five different defense contractors who could be actively managed based on their ability to design prototypes to meet this objective. Two – Magavox Research Labs and Rockwell Collins – were ultimately chosen for full scale development of the device.
In summary, DARPA’s handheld GPS program was able to establish a set of very clear goals, through their direct interaction with users on the ground. These goals were by no means straightforward – significant technological risks were taken to eventually achieve a handheld GPS that remains very similar to the version we carry in our own phones today. Yet, with a clear right-side objective, multiple contractors and researchers could be easily engaged, the potential returns to taking this risk could be systematically re-assessed along the way, and stages of projects (i.e. basic research, prototyping, full scale development) could build upon one another towards achieving the ultimate objective. The figure below illustrates this right-to-left model for the case of DARPA’s GPS program.
Figure 1. Different project investments by DARPA’s GPS program built toward the mission objective of a handheld 10lbs receiver on the right.
On the left hand side the large dot represents the program design and size of the budget for the portfolio that will become the solution space. On the right hand side we have the problem space i.e. mission specification, as defined by the PM. The small dots between represent the specific projects that are funded, closed, evolved, and actively managed.
2.3. GPS versus Graphene
Contrast this example with that of the EU’s Graphene Flagship program. Established in 2013, its goal was to “advance graphene commercialization and take graphene and related materials from academic laboratories to society within 10 years, while revolutionizing entire industries and creating economic growth and new jobs in Europe”. This overall goal was divided into 11 smaller “Work Packages”, each one akin to DARPA’s various programs, which guided the selection of potential projects to invest. One such “Work Package” was optoelectronics, a field of research which explores the use of light in electronics. It aimed to “establish a new field of graphene photonics and electronics”, to develop “innovative technological applications in long-haul optical communications, inter- and intra-chip optical interconnects, wireless communications, security and surveillance applications, environmental monitoring, and energy harvesting”.
Several research projects were funded under this mandate, which made valuable scientific strides – such as the experimental demonstration of a graphene-insulated silicon capacitor to drastically improve speed and accuracy in optical data transmission, and the development of a flexible graphene film with quantum dots to optically monitor heart rate and respiratory rate.. Yet these advancements are still far from the Flagship’s overall goal to “advance… commercialization” and take graphene “from academic laboratories to society”. Compared to DARPA’s handheld GPS program, the Flagship’s various Work Packages have made significantly less progress in bringing a product to users on the ground – leading Terrance Barkan, director of the Graphene Council, to state that “for the money applied and for all the resources rallied, the Graphene Flagship is underperforming from a commercial development perspective”.
Two key differences are immediately apparent in the examples of GPS versus Graphene. Firstly, the early graphene research that formed the basis of the European efforts (i.e. the left hand side) was significantly further removed from any clear objective of full scale development and problem-solving on the right hand side, as compared to the GPS case. Moreover, where GPS technology had already been in use for some time before the commencement of DARPA’s miniaturization program, research in graphene was still largely lab-based at the advent of the Graphene Flagship. Consequently, the time taken to progress from left to right would naturally be greater.
Secondly, a significant challenge also arose from the breadth of the objective which the Graphene Flagship set out to achieve. As depicted in the diagram above, the right-side objective to “establish a new field of graphene photonics and electronics” with “innovative technological applications” in a variety of fields was significantly broader than the precisely defined objective of a 10lb handheld GPS receiver. The result of such a broad objective was that any progress made towards it was necessarily diffuse, so as to yield research advances in fields ranging from telecommunications to healthcare. While advances were individually significant, their diversity limited the extent to which they could build on one another in order to reach the objective on the right, as was possible for GPS. The figure below illustrates the case of the EU’s Graphene Flagship program using a similar representational approach as Figure 1, but illustrating the broad scope of the “problem domain” for graphene and the breadth of the potential portfolio of solutions.
Figure 2. Range of projects funded by the Graphene Flagship, which were far removed from the objective to “advance graphene commercialization”, and which led in multiple different directions.
A cursory analysis of these differences may conclude that one simply need apply a better funded, longer term version of the DARPA model to the challenge of graphene commercialization. However, we will see in the following section that the GPS and Graphene cases represent fundamentally different types of problem spaces, which have significant consequences for the effectiveness of the key tenets of the DARPA model of strategic investment. We will see that the nature of the problem space being funded serves as an important boundary condition at the ecosystem level for the applicability of the DARPA model.
3. DARPA’S BOUNDARY CONDITIONS – THE ECOSYSTEM
3.1. Assessing the Problem Space
Understanding details of both the GPS and Graphene cases help demonstrate that the Graphene Flagship’s Optoelectronics program covered a much larger problem space than that of DARPA’s GPS program, owing to the broader and more distant right-side objective that it sought to achieve. Consequently, investments made toward achieving GPS’ right-side objective yielded solutions that were more densely concentrated than those yielded from the Graphene Flagship’s investments.
This is illustrated in figure below: On the left hand side we illustrate a narrower mission specification against which a similarly sized solution portfolio is built (in financial terms) but as illustrated, with a clustered set of projects within the portfolio. On the right hand side we observe a broader and more distant right-side objective, with the same budget, expands the space of possible solutions that are viable in addressing the problem but at the same time the lower density of the portfolio limits opportunities for learning, exchange, project interaction etc.
Figure 3. Spread of possible solutions as a function of the size of a problem space.
Unfortunately, this greater variation among possible solutions makes it markedly more challenging to assess and shape their progress towards the program’s right-side objective. As detailed above, a key feature of the DARPA model is the need to rebalance a portfolio of investments, to ensure that projects maintain an acceptable risk to reward ratio. This requires that PMs be able to assess the progress of a given project against the desired program outcome, and compare this progress against that of other projects. The subsequent paragraphs illustrate how the clarity of the right-side objective and the size of the problem space begin to impose significant challenges on the ease and effectiveness of this key portfolio aspect of the DARPA system.
- Challenging project evaluation: DARPA’s GPS program laid out the specific desired outcome of a 10lb miniaturized receiver. Against a focused objective and the clear outcome metric it afforded, it was possible for the PM to assess whether research projects or contractors were making headway. It was clear, for example, that miniaturization required digitizing the GPS signal; thus research projects that could not achieve this were abandoned. Solution providers could likewise be curated according to the weight of their prototypes. Ultimately, the project was being developed for soldiers on the ground, thus prototypes could be tested for usability and customer satisfaction along the way.Conversely, against the broad objective of “commercializing graphene”, the extent to which projects are making progress is less clear. With the starting point so far removed from the objective, it was difficult to determine what technologies would eventually turn into something commercially viable – for example, whether it will prove effective to manufacture silicon-insulated graphene, or graphene quantum dots. In addition, the significant variation amongst projects made it challenging to compare their progress– how might one compare advances in the speed of data transmitted to the accuracy of heartbeat recorded? Choosing one project to stop over another becomes untenable, since all can be said to be making progress towards the broad, general objective.
- Reduced Program Manager Capacity: DARPA selected its PMs based on their deep technical or operational expertise. Their scientific or military experience would enable them to more accurately identify promising research, and assess their progress along the way. Unfortunately, as has already been established above, the further and broader the objective on the right, the wider the variety of possible solutions that can achieve it. PMs in a larger solution space need to be familiar with a far wider range of research, in order to adequately assess the progress of the varied possible solutions developed. Furthermore, with the problem space expanded to include commercialization as opposed to end use in a specific operational setting, PMs seeking to fill their chosen “technological white space” would require not just technical expertise but expertise in the many possible routes to commercialization as well. Consequently, a larger problem space implies that any single PM is much less likely to have all the necessary expertise to assess and curate the increasingly heterogeneous and sparse solutions that fall within their program’s problem space.
3.2. Assessing Solution-Provider Availability
Another critical, and orthogonal, consideration regarding the size of the problem space is the availability of solution-providers within the ecosystem. We find that this is an important boundary condition for DARPA’s success. Specifically – it seems that a given problem space must have a sufficient concentration of potential solution providers (e.g. researchers or companies) that a PM can leverage to develop potential solutions to fill the problem space. We refer to the availability of solution providers as how dense or sparse the ecosystem across problem space is. This distinction is critical as a sparse concentration of solution providers in the ecosystem implies that there are fewer possible investments that a program can make in attempting to solve the problems in such a space.
Figure 4. A problem space of the same size can be more densely or sparsely populated by potential solution providers
A problem space with a sparse availability of solutions providers in the ecosystem is generally characteristic of nascent ecosystems – newly developing industries or fields such as graphene, where there are relatively few researchers or entrepreneurs who might be tackling problems in this space.
Conversely, DARPA’s GPS program was set against the backdrop of a dense, mature defence ecosystem – which we now understand as an important boundary condition. As is explained below, this enabled PMs to easily identify potential solution providers conducting research or manufacturing in areas sufficiently proximate to the program’s objectives. In addition, this density of solution providers allowed for the successes produced by one set of solution providers to be built upon by future investments in other solution providers. In contrast, a PM making investments in a problem space with a sparse availability of solution providers would encounter the following challenges:
- Limited solution provider availability: In the GPS scenario, Program Manager Dr. Sherman Karp was able to reach out to the director of his own division, Dr. Anthony Tether, to explore the possible digitization of the GPS signal. Tether was already conducting research in a proximate area, and was therefore well positioned to push the program closer to its goal. In developing the prototype, five defence contractors were approached, each with the relevant manufacturing capability. The defence ecosystem had already been developing GPS for the past 30 years, thus many of the necessary solution providers for the miniaturization program were likely already present.Conversely, with the more expansive problem space in Graphene, and far fewer existing solution providers to be leveraged, the portfolio was more limited. In addition those providers that may exist are spread more widely throughout the ecosystem. Hence even after promising lab-based projects are funded, potential manufacturers who can develop prototypes were few and far between. Identifying one capable of producing a working integration of silicon and graphene, or of graphene and quantum dots, would involve a far wider search process than simply reaching out to a set of pre-existing contractors. Furthermore, should a prototype be developed, fewer opportunities would exist to test it with end users.
- Reduced solution provider complementarity: Even if relevant solution providers can be successfully identified, their likely ‘distance’ to one another shapes the extent to which their efforts can be effectively combined. In the GPS case, existing solution providers likely worked with one another before, as well as within the same overall defence ecosystem, allowing them to better coordinate their activities with one another. Thus, when the defence contractor Rockwell Collins needed to reduce the cost of its $5,000 prototype, it was able to leverage complementary developments in the MIMIC (millimetre wave monolithic integrated circuits) Chip, the work of yet another DARPA program, which offered a cheaper alternative to the gallium arsenide chip.Conversely, with solution providers sparsely distributed in a wider problem space, the solutions they explore can be very different, and have less opportunities to contribute to one another’s progress. Where one research project may pattern graphene onto silicon via plasma etching, another may pattern quantum dots onto graphene using lithography, making it difficult for their understanding of manufacturing methods to build upon one another, so as to get closer to the right-side objective. As such, a program’s multiple investments are curtailed because of their inability to aggregate towards something greater than the sum of its parts.
In summary, a problem space with a dense population of solution providers ensures that at every stage of a program’s progress, there is a ready supply of individuals or institutions who are conducting proximate R&D, and who can be leveraged to undertake projects that move the program closer to its right-side objective.
The concepts of the size of the problem space and density of the solution providers are considered independently, as a given size of the problem space can either be densely or sparsely populated with potential solution providers. For example, the biopharmaceutical and renewable energy industries can be thought of as having similarly sized problem spaces – with one seeking to cure all manner of ills, and the other to develop novel sources of energy. However, the biopharmaceutical industry is far more densely populated with potential solution providers than is the renewable energy industry.
Furthermore, the size of the problem space versus density of solutions providers each carry independent implications for project management. For example, one might infer from the prior discussion on problem space that the Graphene Flagship should simply have focused on a much narrower problem space. However, narrowing the problem space can also narrow the number of potential solution providers to draw potential solutions from, making it no easier for a program to select and manage projects.
Finally, it is critical to note that the notion of leveraging a dense problem space does not imply that a DARPA-like program can only operate in an ecosystem in which solutions essentially already exist, i.e. that the boundary condition would be an innovation ecosystem with providers of already existing solutions. A distinction must be made between the availability of solutions, versus potential solution providers. The fundamental value of the DARPA model is that it invests in high-risk, high-reward research which would not otherwise take place, and thus brings about novel, revolutionary solutions. However, it does require as a boundary condition at the ecosystem level that solution or proto-solution providers exist in the first place, with necessary capabilities in the field, who are willing and able to take that risk.
3.3. The DARPA Approach to Graphene
The necessity of strategically selecting the problem space and understanding the shape and density of the ecosystem is exemplified by DARPA’s own manner of graphene investment. Given the promise of this exciting new material, DARPA made several investments in graphene related research. But rather than operating within this nascent ecosystem wholesale, they instead applied their model toward a dense mature ecosystem that only tangentially intersected with the nascent developments in graphene.
Specifically, recognizing graphene’s value for optoelectronics, DARPA’s PM focused their activities on two existing programs. First, under DARPA’s Wafer Scale Infrared Detectors (WIRED) program, which sought to “[develop] a high-performance, low-cost detector technology using wafer-scale fabrication techniques” they invested in projects such as a graphene-enhanced infrared detector with greater light absorption and tunability, to which it provided $1.3M USD of funding. In other words, DARPA applied its model within the denser ecosystem of wafer-scale technology, investing in graphene as only one of the many projects in the program’s portfolio of possible solutions. Hence DARPA operated within the boundary conditions where its research investment model would be most successful. In parallel, other graphene investments undertaken by DARPA were focused narrowly on the development of an implantable graphene-based electrode capable of measuring both optical and electrical brain signals. This was funded under DARPA’s Reliable Neural-Interface Technology (RE-NET) program, which sought to develop “high-performance neural interfaces to control the… advanced prosthetic limbs” created in complementary DARPA programs. Here, graphene was again one of several alternatives in an overall portfolio of possible solutions, all of which existed within a targeted, dense problem space defined by the right side objective of a neural interface.
3.4. DARPA as a “Selection” vs “Seeding” Model
A better understanding of the DARPA model, and the problem spaces to which it has been applied, helps to illuminate that the DARPA model manages high-risk, high-reward research investments within its organizational system using selection practices (Goldstein & Kearney, 2020). Its success relies on the ability to prioritize possible research avenues across an ecosystem, select the most valuable ones to pursue further, and down select others to be stopped.
The initial phase of Project Selection, for example, assumes that there are multiple projects to select from, in order to initialize a portfolio of research investments and indeed the PM curates such a portfolio. This assumption is further entrenched in DARPA’s use of “real options” in its active program management. Possible program investments are viewed as constituting a balanced portfolio, where there is a distribution of resources that range from low to high risk. The task of the PM is then to ensure that this risk remains balanced across the portfolio over time. Such an approach again assumes, however, that there is a distribution of resources to select from at all.
These assumptions were valid for dense, mature ecosystems with a range of potential solution providers (e.g. researchers or corporations participating in relevant areas). DARPA was able to achieve ground-breaking, unprecedented advances by strategically selecting a set of high risk projects to invest in and develop further.
In contrast, in the sparse solution space of a nascent ecosystem, the already challenging task of strategic selection becomes an almost impossible one, when there are in fact few or no potential options to select from. PMs in nascent ecosystems would be hard-pressed to identify relevant research projects or possible solution providers, let alone a whole portfolio of them. In this case, a program seeking to make high-risk, high-reward investments must first seed the ecosystem with potential solution providers before having any to select from and invest in. This requires a significantly different approach to research investment than has been prescribed thus far by the DARPA model – specifically, one that focuses on seeding the ecosystem with potential solution providers, establishing them with the necessary capabilities and resources, and integrating their activities with one another.
By understanding the exact challenges that research investment can face in a nascent ecosystem, we can develop a version of the DARPA model that successfully overcomes them. In Section 4, we identify how some ecosystems can be nudged towards a form more amenable for the DARPA model. In Section 5, we lay the foundations of a revised strategy to enable high-risk, high-reward research in nascent ecosystems today.
4. FOUR MISSION ARENAS
We have highlighted that the DARPA model for mission-driven innovation is best suited to pursuing high-risk, high-reward research investment when applied to a narrow problem space that can be matched to a dense ecosystem of potential solution providers. Such an environment allows for DARPA’s strengths as a selection system to identify the most promising avenues of research, and encourage them towards an extraordinary breakthrough that addresses a critical mission requirement on the right. The phenomenal success of DARPA in the late 20th century can be attributed as much to its revolutionary model of research investment, as to the dense, mature ecosystem for which it was optimally designed. Unfortunately, as the DARPA model has been adopted across increasingly varied settings, the attendant impact of this boundary condition has not been well recognized. Consequently, investors seeking to copy the DARPA model have, time and again, fallen short of DARPA’s fabled success. Armed with our understanding of how the problem space and the potential solution providers affects the outcomes of the DARPA model, we can now identify the adaptations necessary to facilitate its successful application.
4.1. Assessing the Suitability of the DARPA Model
The key dimensions established above were the scope of the problem space and the density of potential solution providers within it. These two attributes are represented in the diagram below, revealing what we will conceptualize as four general types of ‘Mission Arenas’ in which one might seek to make high-risk, high-reward research investments.
Figure 5. Four types of Mission Arenas which can be invested in.
Mission Arena 1 represents a densely populated, narrowly targeted problem space, best exemplified by the defence ecosystem surrounding the miniaturized GPS program. Here, there is a precisely specified right-side objective, and a density and diversity of potential solution providers (e.g. GPS researchers, chip manufacturers) leveraged to develop potential projects bringing the program from left to right.
Mission Arena 2 represents a broad problem space, but with many solution providers, such as the biopharmaceutical industry. The industry is guided by the diffuse and distant right-side goal of drug development, but many solution providers populate the space, who seek to achieve this for many different disease types, through many different methodologies.
Mission Arena 3 represents a problem space with clear specified objectives, but where there are very few solution providers working towards this objective. Such spaces can exist where there is a clear unmet consumer demand, but a large upfront investment is required by potential solution providers. One example is the commercial space industry, with well-defined objectives such as low orbit space tourism, satellite deployment or space logistics, but where only a very limited number of solution providers can be found, such as Tesla or Blue Origin.
Mission Arena 4 represents the broad problem space with a sparse solution ecosystem – for example, the graphene industry. The potentially revolutionary applications of the lightweight, strong and highly conductive material fuel a broad right-side objective of diverse commercial applications. However, this nascent ecosystem is still sparsely populated, with potential applications and manufacturing methods being explored primarily at the lab stage, and very few commercial graphene companies in existence.
The diagram reminds us that the further removed a solution space is from the dense, targeted ecosystem on the bottom left, the less applicable the DARPA model becomes as a tool for research investment. In an overly broad problem space, the more diffuse the objectives with which to direct and evaluate efforts, the less valuable a model of optimal selection becomes. Thus, in order to apply the DARPA model in Mission Arenas 2-4, these Arenas must first be transformed – either by i) more narrowly targeting an appropriate problem space, or ii) incentivizing entry into solution space. We will see that there are two immediate, short-term solutions to transforming a Mission Arena 2 to 1, and from 3 to 1; but 4 to 1 requires a longer-term and more intentional strategy.
4.2. Managing Mission Arenas
Narrowing the Problem Space (Misison Arena 2 → 1)
One straightforward solution is that a broad problem space, such as that in Mission Arena 2, should be narrowed before a DARPA model can be suitably applied. As alluded to before, if the objective remains broad, it becomes difficult to select relevant projects into the program, or compare the progress of their diverse approaches to determine which should be funded further. A narrowed objective hence enables the PM’s duties in project selection and active portfolio management become tractable. At the same time, the availability of solution providers is sufficiently dense, such that this narrowing does not come at the cost of the number of potential solutions that can be drawn upon.
A timely example of such a strategy in action is the USA’s current accelerated search for a COVID-19 vaccine, “Operation Warp Speed”. The biopharmaceutical industry covers a large problem space – seeking to develop drugs for all manner of disease, ranging from Alzheimer’s to HIV. This broad problem space is also dense, with many solution providers pursuing a variety of strategies to reach the same right-side objective of drug development. Furthermore, solution providers have come to be organized into a hierarchy of institutions, whose activities are coordinated within the overall structure of clinical trials. Thus, not only is there a density of solution providers, but they are also well poised to build off one another’s research through their interconnectivity.
Consequently, the specific articulation of the COVID-19 vaccine as a right-side objective serves to define a narrow problem space from a broader one, creating one that resembles the defence ecosystem supporting DARPA’s miniaturized GPS program. Thus Operation Warp Speed can, and is, being managed very similarly to a DARPA program, where different potential vaccine projects have been selected for initial funding, with investments (e.g. funding to researchers, or investments in large scale production) being readjusted depending on projects’ success at different milestones.
It should be noted that vaccine development is not within the typical purview of biopharmaceutical firms. As mentioned above, the requirement of a problem space with a dense availability of solution providers does not imply that the DARPA model can only be applied in cases where solutions already exist, which would render the notion of high-risk, high-reward research moot. Drug and vaccine development each involve unique manufacturing challenges and economic incentives, which haven traditionally driven commercial firms to under invest in vaccines. Operation Warp Speed’s clarification of and investment in this targeted, narrow right-side objective prompts them to explore solutions in a focused area they would have otherwise found too risky, despite its potential rewards.
Incentivizing Stakeholder Entry (Mission Arena 3 → 1)
The challenge with Misison Arena 3 is that despite its well defined objectives, there is insufficient interest from solution providers to enter this space, typically due to the large upfront investments required. Entry can thus be incentivized through a ‘competition’ model of investment, where monetary rewards are offered for working towards a right-side objective that might not otherwise be attractive. An example of such a competition model in action is the XPrize, a nonprofit organization which sponsors prizes on the order of tens of millions to solution providers who are able to achieve a specified “mission”. These missions are very narrowly and precisely defined, such as the requirement to “rapidly train 500 individuals in 60 days or less at no entry cost” and “ensure job retention of at least 90 days” (among others) in order to win the “Work Reimagined” prize.
The goal of the competition model is not necessarily to achieve this objective per se; rather, it is to populate the mission innovation space with relevant solution providers instead. As stated by the XPrize, despite their clear articulation of a desirable future, their goal is to “focus the resources, talent and technology required to enable those breakthroughs and accelerate that future” – in other words, to increase the availability of potential solutions and solution providers within a specific problem space. XPrize projects are given dedicated mentorship over the course of the competition and go through many rounds of guidance and solution prototyping with judges who are familiar with the field. Such mentorship ensures that new solution providers not only enter the problem space, but are retained as well. Eventually, these efforts can transform a Mission Arena 3 into a Mission Arena 1, to which a DARPA model can then be suitably applied.
Developing a Nascent Ecosystem (Mission Arena 4)
Mission Arena 4 is less straightforward than those proposed for Mission Arena 2 and Misison Arena 3. While Mission Arena 2 and 3 could be nudged towards a state amenable to the implementation of the DARPA model, a nascent ecosystem like that of Mission Arena 4 is sufficiently far removed as to require the specification of a new investment strategy altogether.
As noted, Mission Arena 4 is characteristic of a nascent ecosystem – a region or industry in which there is some burgeoning research, and very little commercial activity taking place. In a nascent ecosystem, there is no targeted objective which can be used to incentivize stakeholder entry; nor has the problem space been sufficiently explored to inform what a feasible right-side objective might be. Consequently, as demonstrated by the Graphene Flagship case, investments made in such an area likely result in a dispersion of effort. Even though the availability of funding can incentivize solution providers to enter the problem space (as with the competition model), what progress they make is likely to be diffuse, with many different solutions being pursued in many different directions. This makes it difficult for different projects to learn from one another, given their varied methods and applications. Based on these disparate efforts, PMs will also find it difficult to surmise what a feasible “technological white space” should be. Hence progress towards the right, and the articulation of what that right should be, is far slower than what it would be in Mission Arena 1.
The strategy for Mission Arena 4 therefore lies in concentrating the efforts of its disparate solution providers, enabling them to learn from one another and jointly determine a cohesive objective on the right – something that the DARPA model is hardly designed for. Hence an entirely new approach is required to enable high-risk, high-reward research in the context of nascent ecosystems. In the next section, we move from DARPA’s model of strategic selection, towards one of strategic growth.
5. ACCELERATING NASCENT ECOSYSTEMS
The ultimate objective of any research investment model is to accelerate the achievement of a high-risk, high-reward right-side objective directly related to achieving the mission. However, different strategies must be applied depending on the state of the ecosystem in which this is being attempted. The DARPA model is based on principles of selection, where funds are optimally allocated towards the most promising projects among multiple contenders. In a scenario in which there are limited projects to select from, a model based on principles of growth is required instead.
5.1. Ecosystem Solution providers
Before diving into the specifics of the model, we require a better understanding of what the growth of solution providers entails. Thus far, we have spoken in abstract terms about the different solution providers responsible for undertaking the projects that incrementally progress an ecosystem towards the right-side objective. However, understanding their specific nature and requirements can help us to understand how best to enable their integration in a nascent ecosystem. The following are the types of solution providers that play an essential role in nascent ecosystems as based on Budden & Murray (2019):
- Universities: These essential solution providers contribute much of the early research into a nascent ecosystem, such as explorations of the properties of different types of graphene based materials. However, universities may lack sufficient expertise to develop more talent (e.g. a dearth of materials science faculty with graphene expertise), or lack transfer practices that enable inventions to leave the lab. These drawbacks can limit universities’ abilities to pursue relevant research that can push the ecosystem further to the right.
- Entrepreneurs: These solution providers play an important role in enabling innovations to leave the lab, which again serve to develop solutions that bring the ecosystem further to the right. However, an ecosystem may be set in a country where there is a culture against risk taking, or where it is difficult to incorporate a company, thereby reducing the availability of entrepreneurs willing to explore potential solutions further.
- Corporates: Similar to entrepreneurs, existing large corporations can help to support the development of novel projects in the ecosystem. They may also have large scale lab or manufacturing facilities that aid in the production of potential prototypes. However, if they are unwilling to invest in innovative projects, or are unaware of relevant research, this can stymie their potential contributions.
- Risk Capital: Within the dense defence ecosystem to which the DARPA model was applied, it was not necessary to consider the role of risk capital. This is because the Department of Defence itself provided all of the necessary monetary investment into potential projects. However, in a much larger ecosystem, far more complementary investments should be undertaken by other parties. Risk capital providers can thus be incentivized to invest in an ecosystem as well, provided that there are sufficient ventures to invest in.
- Government: Similarly, the government was not taken into consideration in a defense ecosystem, given that the problem space was primarily helmed by the Department of Defence, and many other solution providers involved such as end users and defence contracts fall within their purview. However, in a larger, nascent ecosystem, the government’s role in enabling the activities of and interactions between solution providers becomes significant – such as the extent to which non-compete laws and enforced can affect the ability of talent to flow to promising projects, or the stringency of IP laws can affect researchers’ willingness to embark on projects which they may not be able to commercialize.
This elaboration of solution providers demonstrates not only their individual importance in a nascent ecosystem, but the importance of their integration with one another. For example, corporate and entrepreneurial product development depends on universities’ research; the rate of business formation by entrepreneurs depends on government policies for incorporation; the presence of risk capital is heavily dependent on the availability of entrepreneurs to invest in, and the incidence of entrepreneurship is likewise dependent on the availability of risk capital.
Thus, the effective integration of solution providers’ activities in a nascent ecosystem is essential, as they can enable positive feedback loops that accelerate progress towards the right-side objective. This was no different in the application of the DARPA model to GPS, where different solution providers built off one another’s efforts. The key difference is that where the DARPA model was able to rely on a pre-existing density of and interconnectivity between solution providers, this must be actively encouraged in a nascent ecosystem. The revised model below will outline new guidelines for a PM’s role in achieving this – specifically their role in creating opportunities for limited solution providers to learn from one another, and work together to refine a right-side objective.
5.2. Ecosystem Growth Model
We lay out a revised model for research investment in a nascent ecosystem, focusing primarily on the role of the PM within it. The key features of this Ecosystem Growth Model are mapped to those of the DARPA model identified above.
Figure 6. Adaptation of DARPA model into an Ecosystem Growth model
1. Program Iteration
The initial role of the PM in the DARPA model is to work with solution providers to identify a feasible right-side objective that orients all of the program’s activities to achieve the mission – such as the identification of a clear end user demand like a miniaturized GPS. Unfortunately, PMs in a nascent ecosystem will be hard-pressed to work with users to discover a similarly clear objective, given that in nascent ecosystems, end users may not even exist, for a market that has yet to be created. The limited number of solution providers in the ecosystem, as well as their limited experience with relevant technology, means that any right-side objective that can be articulated at this stage may prove itself to ultimately be unachievable or unprofitable. Fixating on a single narrow objective to invest in from the get go is thus woefully premature.
Consequently, the right-side objective for a nascent ecosystem must be prototyped in conjunction with the solution providers that populate it. One example of this in action is XPrize’s “Impact Roadmaps”, where a very broad goal, such as securing the future of food or the future of housing, is analyzed by a panel of existing solution providers, and subdivided into potential sub-goals. These sub-goals then serve as the temporary right-side objective to guide their competitions which, as introduced before, serve to attract solution providers. As more solution providers enter the space, and gain more experience with the challenges and opportunities of a given problem space, the current roadmap is then reassessed to define a more accurate right-side objective over time.
In prototyping the right-side objective, it is also important to ensure that different types of solution providers are adequately represented in its design. The Graphene Flagship originally began under the guidance of a consortium comprising primarily (80%) academic institutions – a likely driver of the large proportion of early stage research which was funded as the Flagship commenced. However, the representation of solution providers has changed over time. As stated by Jari Kinaret, director of the Graphene Flagship, “[t]he consortium was originally made up of mostly academic groups, whereas today about 40 percent of its members are companies.” This has enabled the Flagship to now focus on more targeted investment directions, through the identification of “Spearhead Projects” – “initiatives with well-defined, application-oriented objectives that are motivated by market opportunities”.
In the DARPA model, PMs could rely largely on their own personal prior experience or personal connections to craft a suitable right-side objective to define the program going forward. However, PMs in a nascent ecosystem must create strategies to constantly tap into the fast changing knowledge of a fast changing pool of solution providers, so as to enable constant iteration on what the right-side objective should be.
2. Stakeholder Incentivization
PMs in a nascent ecosystem can no longer rely on being able to select a pool of potential projects to invest in, as they would have in the original DARPA model. Rather, as elaborated upon in the suggestions for Mission Arena 3, PMs must incentivize the entry of solution providers, before they can even have projects to select from.
In addition to the competitions introduced in the prior section, hackathons offer a valuable means by which potential solution providers can be introduced to a nascent ecosystem, with a very low level of commitment. These hackathons are run for several days, and bring together individuals with a variety of backgrounds, who may be interested in solving problems in a given ecosystem. The barrier is low as no relevant expertise is assumed, nor is it necessary to have a potential solution going in. One such example is MIT’s Hacking Medicine annual “Grand Hack”, which gathers individuals with backgrounds ranging from medical professionals to product designers to engineers, to solve healthcare challenges. Participants pitch novel innovations, and team up to form a basic prototype or business plan is created over several days. Each team is assigned a mentor who is familiar with healthcare innovation, such as an existing entrepreneur or investor, to guide the team’s progress. Large healthcare corporations typically sponsor the event, and also serve on the hackathon’s judging panel.
Not only do these events incentivize stakeholder entry, they also enable key solution providers in a nascent ecosystem to form relationships with one another (such as entrepreneurs with risk capital, or university research with corporate funding), and allow the continuous exchange of what valuable, sparse knowledge exists in an ecosystem. Furthermore, hackathons can serve as an avenue to quickly understand the state of solution providers in an ecosystem, and identify several innovation challenges that may be worth pursuing. As such, they are also a useful mechanism for the process of Program Iteration explored above.
3. Portfolio Integration
In the DARPA model, as projects progress, the PM’s role is to redistribute funds between them toward the most promising projects, and culling those projects which are unsuccessful. However, as has been emphasized in this article, PMs may not have a distribution of projects to select from, and culling projects may dwindle the already limited stakeholder participation in the ecosystem.
Rather than pitting projects against one another, PMs in a nascent ecosystem should enhance their ability to learn from one another instead, so as to accelerate their development of relevant knowledge. One example of this in action is the clean energy incubator Greentown Labs, in Cambridge, Massachusetts. It was founded on the realization that different clean energy startups were finding it difficult to prototype the specialized components they required, as there was a dearth of manufacturing firms with the necessary expertise. Greentown Labs thus serves as a common venue where they can share learnings about manufacturing techniques, and invest in novel manufacturing methods and technology together.
In a similar fashion, the Graphene Flagship has added an additional “Industrialization” Work Package in recent years, one objective of which is to “[develop] consensus-based and accepted international standards for properties and characterisation of graphene”. Their establishment of the “Graphene Flagship Standardisation Committee” helps to integrate the multifarious efforts of different projects, by facilitating the use of common measurement, manufacturing or regulation methods. This eases the transfer of knowledge across projects, and enables them to build upon one another.
4. Organizational Bandwidth
For the DARPA model, the key organizational principle that underlay all of the PM’s activities was organizational flexibility. Given their need to be discerning in project selection, and their role in actively redistributing funds, it was necessary that PMs had the flexibility to invest in projects as they saw fit, without the typical constraints of defence contracting. Flexibility is also just as important in a nascent ecosystem, especially when there is limited choice amongst projects that can be invested in. However, an organizational property which becomes much more essential in this large, sparse problem space is the need to have a greater bandwidth, in order to remain cognizant of the varied activities of solution providers who are distributed widely across the ecosystem.
In DARPA, PMs were chosen based on their individual specialized technical or military knowledge, which allowed them to use their personal understanding and networks to identify promising areas of investment. However, with potential solutions spread over a far larger space in a nascent ecosystem, PMs can no longer rely on their individual knowledge. Furthermore, the factors affecting projects’ progress are now much more varied than in a dense defense ecosystem – encompassing not just the basic scientific viability of the underlying technology, but market demand, government policies, and even the culture of a society. A single PM is no longer likely to possess all the technical and commercialization expertise necessary to guide a project all the way from left to right.
Consequently, the program must now be structured to enable it to tap into a wide array of solution providers. As mentioned above, hackathons are one easy way to do this ad hoc. However, on an ongoing basis, it can be useful to recruit a diverse team of ecosystem solution providers instead – namely, representatives from entrepreneurs, risk capital, universities, large corporations, and government. All of these solution providers have a purview over different aspects of the resources necessary in an ecosystem, and can work together to better identify gaps that can prevent projects from going from left to right.
One example of this in use is in fact DARPA’s more recent development of Siri. We focused heavily on the very specific DARPA case study of GPS, to demonstrate the dense ecosystem in which it was developed, and thus how its success was enabled. However, DARPA’s development of Siri provides a valuable example of a case in which DARPA ventured into a nascent ecosystem to make an impact.
Figure 7. DARPA leveraged SRI’s ecosystem familiarity to bridge efforts toward a narrowed objective.
Siri’s success was enabled specifically by leveraging a team of solution providers beyond the defence ecosystem to manage the program. The Personalized Assistant that Learns (PAL) program was initiated with the objective of reducing soldiers’ cognitive loads. Despite its similarity to GPS as an operationally-grounded objective, this right-side was less precisely defined, and required resources that lay outside the defence ecosystem. The non-profit research institute SRI International was therefore hired under the program, in order to aid in tapping into the necessary ecosystem resources which DARPA itself had less familiarity with. SRI International effectively assumed the role of the PM, engaging solution providers in the Palo Alto AI ecosystem – such as Stanford researchers across different AI departments, early entrepreneurs who tapped into the emerging technology, and venture capital firms who were willing to invest. It was through this pivotal involvement of SRI International, and its integration of ecosystem resources, that DARPA’s initial $150 million investment ultimately resulted in Siri 7 years later.
The DARPA model for mission-driven innovation is best suited to pursuing high-risk, high-reward research investment when applied to a targeted problem space that can be matched to a dense innovation ecosystem of potential solution providers. Such an environment allows for the identification and selection of the most promising avenues of research, from which extraordinary breakthroughs can emerge to address critical mission requirements. The successful pursuit of mission-driven research therefore relies not only on the amount of funding available or the application of specific management practices, but also on the nature of the innovation ecosystem to which it is being applied. Based on this insight into the boundary conditions of the lauded DARPA model, it is possible to describe four different Mission Arenas defined along the two dimensions of the scope of the problem space and the density of potential solution providers within it, with varying degrees of applicability of the DARPA model. We introduce strategies for how two of these Mission Arenas can be actively managed towards a form more amenable to the use of the DARPA model for high-risk, high-reward research investment. In the specific case of a Mission Arena with a nascent innovation ecosystem, the selection principle behind the DARPA model must be changed towards one of strategic growth. We thus introduce the Ecosystem Growth Model, with a focus on program iteration, solution provider incentivization, portfolio integration, and organizational bandwidth. For governments and foundations around the world engaged in a range of missions these insights give an answer to the urgent question of how investments should be made, from early R&D spending to later-stage acceleration, to most effectively fuel the full lifecycle of innovation from ideas to impact.
 See Frolund et al :https://ec.europa.eu/info/news/support-breakthrough-innovations-european-innovation-council-eic-should-take-hands-challenge-driven-approach-according-international-expert-group-2020-nov-11_en
 National Academies of Sciences, Engineering, and Medicine 2017. An Assessment of ARPA-E. Washington, DC: The National Academies Press. https://doi.org/10.17226/24778.
 https://www.iarpa.gov/index.php/newsroom/iarpa-in-the-news/2016/833-success-and-failure-not-all-iarpa-programs-transition-to-the-field https://medium.com/@ODNIgov/iarpa-embraces-intelligence-communitys-toughest-challenges-560a38d77336 but see https://www.nytimes.com/2013/03/22/opinion/brooks-forecasting-fox.html
 See e.g. Fuchs, E., ‘Rethinking the role of the state in technology development: DARPA and the case for embedded network governance’, Research Policy, Vol. 39, 2010, pp. 1133–1147. On a similar note, but more focussed on the decentralized role of Programme Managers in relation to management in the public sector see Piore, M. (2009) “Sociology, Street-Level Bureaucracy, and the Management of the Public Sector”. Available here: https://economics.mit.edu/files/4288.
 Azoulay, P., Fuchs, E., Goldstein, A. P., & Kearney, M. (2019). Funding breakthrough research: promises and challenges of the “ARPA Model”. Innovation policy and the economy, 19(1), 69-96.
 Bonvillian, W. B. (2009). “The Connected Science Model for Innovation—The DARPA Model”, in 21st Century Innovation Systems for the U.S. and Japan, ed. S. Nagaoka, M. Kondo, K. Flamm, and C. Wessner. Washington, DC: National Academies Press.
 Alexandrow, Catherine (2015). The Story of GPS. 50 Years of Bridging the Gap, DARPA p 54-55.
 Graphene Flagship. Graphene Flaship in Work Packages, October 10th 2013.
 Sorianello, V., Midrio, M., Contestabile, G. et al. Graphene–silicon phase modulators with gigahertz bandwidth. Nature Photon 12, 40–44 (2018). https://doi.org/10.1038/s41566-017-0071-6
 Polat, E. O., Mercier, G., Nikitskiy, I., Puma, E., Galan, T., Gupta, S., … & Konstantatos, G. (2019). Flexible graphene photodetectors for wearable fitness monitoring. Science advances, 5(9), eaaw7846.
 Johnson, D. (2019). “Europe Has Invested €1 Billion Into Graphene—But For What?” IEEE Spectrum, July 2019.
 Safaei, A., Chandra, S., Vázquez-Guardado, A., Calderon, J., Franklin, D., Tetard, L., … & Chanda, D. (2017). Dynamically tunable extraordinary light absorption in monolayer graphene. Physical Review B, 96(16), 165431.
 Park, D., Schendel, A., Mikael, S. et al. Graphene-based carbon-layered electrode array technology for neural imaging and optogenetic applications. Nat Commun 5, 5258 (2014). https://doi.org/10.1038/ncomms6258
 Goldstein, Anna P., and Michael Kearney. “Know when to fold ‘em: An empirical description of risk management in public research funding.” Research Policy 49.1 (2020): 103873.
 Institute of Medicine (US) Committee on the Evaluation of Vaccine Purchase Financing in the United States. Financing Vaccines in the 21st Century: Assuring Access and Availability. Washington (DC): National Academies Press (US); 2003. 5, Vaccine Supply.
 Kremer, M., & Snyder, C. M. (2003). Why are drugs more profitable than vaccines? (No. w9833). National Bureau of Economic Research.
 Budden and Murray: “MIT’s Stakeholder Framework for Building & Accelerating Innovation Ecosystems. Available at: https://innovation.mit.edu/assets/MIT-Stakeholder-Framework_Innovation-Ecosystems.pdf.
 D. Johnson, “Where does graphene go from here? Experts weigh in on whether the EU’s €1 billion Graphene Flagship can get the “wonder material” past the Valley of death – [News],” in IEEE Spectrum, vol. 56, no. 7, pp. 10-11, July 2019, doi: 10.1109/MSPEC.2019.8747299.