Ideaflow: The Only Business Metric That Matters
Jeremy Utley & Perry Klebahn
304 pages, Portfolio, 2022
We all want great ideas, but few understand how great ideas are born. Innovation is not an event, a workshop, a sprint, or a hackathon. It’s a result of mastering ideaflow, a practice that elevates everything else you do. As Marc Randolph put it in his fantastic memoir, That Will Never Work, the origin story of Netflix, “It was not about having good ideas. It was about building this system and this process and this culture for testing lots of bad ideas.”
In Ideaflow: The Only Business Metric That Matters, we (Stanford’s Jeremy Utley and Perry Klebahn, codirectors of the renowned Hasso Plattner Institute of Design, aka the “d.school”) offer a proven strategy for routinely generating and commercializing breakthrough ideas.
“Is this idea any good?” We get this question hundreds of times a year from students at Stanford. In what has become something of a pilgrimage at the university, aspiring entrepreneurs make their way to LaunchPad Office Hours to see if they have what it takes to build a new company, wondering whether their idea is good enough. But it’s not just start-up founders who wonder about the merits of their ideas. It’s a question that plagues individual contributors, managers, and executives in commercial settings, too.
Think about it: How many times have you wondered whether a particular new idea was worthy of resources? Forget the financial investment a new direction might involve, most of us struggle to know whether a new idea deserves even our attention, or our time, or our social capital. We are all resource-constrained, and any new idea can threaten to be a distraction at the very least.
But how else can we make progress, without doing new things? And so the question continues to linger, “Is this idea any good? Is this the one to which I ought to give extra effort?”
Lucky for them, and for all of us, that’s not the question that matters. What distinguishes world-class entrepreneurs and business leaders from the rest is not having better ideas, but having better methods for evaluating ideas. A better question is, “How do I learn whether any of my ideas is worth incremental effort?”
In the following excerpt, we explain why having a solid method for evaluating ideas can help to build a better innovation pipeline, as well as knowing whether an idea is worth testing to leverage the full value of your organization’s ideas (while minimizing the uncertainty and risk involved in making them real).—Jeremy Utley and Perry Klebahn
* * *
Silicon Valley Bank is a large commercial bank headquartered in Santa Clara, California. Since its founding in 1983, SVB has specialize in a local commodity: high-tech start-ups. Today, SVB is one of the largest banks in the country, with operations worldwide. For all its scope, however, technology and venture capital—innovation—still play a vital role in its success. Eager to drive growth, CEO Greg Becker brought us into work with high-potential leaders at the bank in 2016. Becker assembled nine teams from around the organization and gave each one a strategic area of opportunity to explore.
Working with a new group, we’ll usually introduce a hypothetical project for teaching purposes. Since we already had Becker’s nine strategic areas, we decided to start with one of those. Debt financing for start-ups struck us as an ideal demo, one with plenty of relevance to SVB’s core business. That team would be looking at how founders borrow money to launch new businesses. What kinds of terms do these customers expect? What problems do they commonly face? How might SVB make its debt financing offerings more compelling? The answers might have major ramifications.
After explaining the idea-generation process described in the previous chapter, we divided the group into nine teams and set them to work. For three days, each team generated ideas, prototyped concepts, and gathered user feedback. At the end of the third day, the entire group gathered to share results. The team tasked with eventually tackling this problem would act as a jury, selecting the most promising suggestions for testing and validation.
Though the portfolio review got off to a good start, it became increasingly clear that the jury was prioritizing risk avoidance over upside. At the end of the presentation, they went to the front of the room and announced their choice. As we’d feared, it was the one we’d seen as the least risky, interesting, and promising option. Realizing that the audience wasn’t reacting as expected, the jury faltered. We asked for a show of hands. “Outside of the jury,” we asked, “who would have voted for this particular idea?” After a pause, two hands out of forty hands went up.
“Are you serious?” one of the jury members exclaimed.
“Are you?” someone called from the back.
Everyone on the jury thought they’d picked the obvious winner. Everyone else thought otherwise. What happened?
Why Picking Is Difficult
We all know someone who shouts the “right” plays at the screen during the Superbowl. You find armchair quarterbacks in every arena. Some of us love second-guessing a decision ... when we face none of the consequences of making the wrong one.
This isn’t necessarily a bad thing. Knowing that you’ll be the one to face the consequences of a decision can’t help but shape your thinking about it. With skin in the game, your options look different. If you’ll have to follow through, you will instinctively narrow the scope and shorten the horizon. It may be obvious from the outside that your friend needs to quit that toxic job driving them to an early grave. When it’s your own lousy job, however, the picture isn’t as clear. The effort and risk involved in leaving a job or even changing careers are far more intimidating. Maybe your boss isn’t so crazy after all. It’s tough to think big when you’re the one carrying the load.
The showdown at Silicon Valley Bank illustrates this tension. If you don’t have to worry about logistics, bandwidth, or risk, why think small? There were more than a handful of genuinely novel ideas put forward that piqued everyone’s curiosity. Validating any idea requires real-world experimentation, but these ideas brimmed with potential. Even if they weren’t viable, exploring them would have led in interesting directions.
The jury group at SVB steered to the lowest- potential idea because it was the most feasible one on the list. We are hard-wired to evade sabertooth tigers, not maximize the growth potential of debt-financing programs at Bay Area banks. This is the cognitive bias known as loss aversion showing up once again. When our necks are on the line, risks outweigh rewards. Under pressure, the mind relies more heavily on instinct. Then it justifies these intuitive decisions retroactively, adding a gloss of logic and reason to choices that were driven by cognitive biases.
The safest, least interesting idea genuinely seemed like the right choice to the people on the jury. That’s why they were surprised by the audience’s reaction. On an emotional level, they had latched onto a sure thing: a manageable amount of effort required with clear line of sight to the goal and alignment with the status quo. Only then had their rational minds entered the picture to make the business case. As our good friend at Stanford Professor Baba Shiv says, “The rational part of the brain is excellent at rationalizing decisions made elsewhere.”
Since perceived effort and risk stymie the capacity to think big, it helps to lower the pressure. We do this by establishing a testing pipeline for ideas. A validation process gives ideas an outlet, someplace to go other than two buckets labeled Yes and No. When you hear testing, forget the expensive and bureaucratic corporate “pilot program.” Instead, think rapid, scrappy tests straight out of high school science class. Hypothesis to results in an hour and then off to lunch.
When you select ideas for testing, as opposed to full-on implementation, all you’re committing to is a quick, scrappy test. This mindset frees you to evaluate your ideas on their merits alone. Creating a pipeline to validate your ideas is crucial to maintaining ideaflow. When an expensive and scary green light is the only option, most ideas seem too risky and resource-intensive to even consider. That’s why your most ambitious ideas tend to stagnate as you keep reaching for ones that are easier and less risky to implement. Once your creative mind recognizes this logjam, it usually stops generating more big ideas.
Where Israel’s Dead Sea is famously salty, the freshwater Sea of Galilee, ninety miles to the north, supports a diverse ecosystem. Though both bodies of water are fed by the Jordan River, the Dead Sea has no outlet while the Sea of Galilee supplies ten percent of Israel’s water needs. Flow is essential to life and vitality. A low-stakes outlet for ideas restores the flow of creativity. If we get stuck in the mindset that we must generate a ton of ideas at the start and then pick the “right” one, the pressure of perfection leads to sterile and safe choices. With the binary approach, we don’t try things out, let alone use what we’ve learned to adapt our thinking.
Reality is an excellent source of creative input. Through experimentation, your ideas will benefit from the lessons of costs, clients, and customers. This is why you don’t generate a mountain of ideas in a vacuum, decide which one wins, and go make it happen. From now on, you will test ideas in the real world, using the data you gather to refine the ones you have and spark better ones along the way. This is how we move, step by step, down the path from inspiration to conviction.
Never Stop Testing
Even if you’re an acknowledged expert in your field, you simply aren’t qualified to decide which ideas to pursue absence of real-world data. Nobody is! There are too many unknowns. Without real-world testing, you’re leaving the success of your pursuit to luck to one degree or another. Innovation without validation the equivalent of pointing your car in the direction of home, closing your eyes, and hitting the gas pedal. You might make it back to the house, but chances are you’ll end up in a ditch. It makes no sense, yet companies routinely drive home blindfolded, throwing barrels of money and time into engineering solutions before even validating desire. When it turns out that nobody wants whatever the product or service is, blame falls on the sales department, or on changing market conditions. Never on the faulty innovation process where it belongs. Thus, the cycle begins again.
General Motors saw early traction when it deployed its car-sharing service, Maven, in 2016. Thousands of people signed up for the pilot effort in New York City to rent GM cars for hourly or daily use. The right next step would have been to expand Maven to the surrounding area or, even better, try a different locale altogether. A second pilot program in Phoenix, Arizona, would have tested GM’s assumptions in a very different way. Instead, eager to stake a claim in the space, GM burned through millions in a misguided effort to expand Maven to more than a dozen cities at once. What GM’s leaders discovered too late was that there were critical gaps in the concept that hadn’t been exposed by its Big Apple test run. Those flaws only became obvious in other markets with different conditions on the ground. Unfortunately, scaling up meant addressing all these problems simultaneously to keep the concept afloat. As it turned out, there was no time to fix everything before the initiative hit the end of its runway. GM shuttered Maven only four years after its promising launch.
A similar thing happened when a promising concept was proposed at Keller Williams, according to our friend John Keller, the company’s head of transformation. As Keller explained to us, local insights are currency in real estate. Anyone can put up a shingle and list some properties. Coming to know a place well takes time and effort. It goes far beyond doing a Yelp search for good coffee shops. The best real estate agents amass encyclopedic knowledge about their areas, and this hard-earned expertise becomes a competitive advantage. People learn to trust in an agent’s knowledge of school districts, noise pollution, residential streets that get used as shortcuts by commuters, and other relevant factors. Over time, this expertise gets rewarded with loyalty and referrals. When a real estate agent saves you from buying the wrong house for a reason you would never have spotted on your own, you remember.
Collectively, Keller Williams agents possessed an enormous reservoir of local knowledge. However, they had no way of sharing this knowledge amongst themselves. To better leverage this valuable resource, the company wanted to create an internal database. Keller Williams agents would share their knowledge. In return, they could draw on that database whenever they needed answers. For example, they would be able to get up to speed much more quickly when moving to a new area themselves. Onboarding would also be easier with all that collective knowledge in one place. You wouldn’t have to keep explaining the same local quirks to each new agent. The database idea was promising, but it raised questions. What kinds of information would qualify as local insights: Dining options? Good pediatricians? Reliable contractors? In terms of implementation, would off-the-shelf software work, or would the company need to invest in a pricey custom solution? When it came to using the database, how easy would it be to contribute and find information, particularly on a mobile phone? Good real estate agents aren’t known for sitting still.
Every idea comes with a set of questions attached. The only way to proceed is to make assumptions about what the answers might be. You won’t know whether you’ve guessed right, however, until you’ve tested your assumptions in the real world. Leaving that testing process to the public launch of a completed product or service is a critical error made by far too many organizations. Keller Williams was savvy enough to start with a pilot version of the database, opening it up to agents in a single area. Within a short time frame, thousands of insights were entered into the new repository. The participants seemed happy with it. What had threatened to be a hassle was actually easy to use and fairly valuable in practice. The rate of adoption exceeded the company’s expectations.
As with Maven at GM, the right next step would have been to expand the offering to a very different area in order to test the same assumptions from another angle. And again, that isn’t what happened. If the institution doesn’t have a culture of testing and an established innovation pipeline, it’s just too tempting to run with promising ideas rather than “waste time with endless tests.”
As Keller ruefully admitted to us years after the fact, Keller Williams rolled the database out nationally after that one successful test. But leaders hadn’t grasped that scale magnifies complexity. Making an idea even a little bigger can make it a lot more complicated. During the pilot, real estate agents could easily police the database, educating each other about best practices and clearing out any low-quality contributions cluttering up search results. On a national scale, the database was suddenly flooded with contributions at a pace that far exceeded the users’ capacity for self-moderation. Once the database had ballooned to half a million entries, sifting valuable insights from the accumulated dross became impossible. The best contributors grew tired of watching their carefully written insights get diluted by a sea of one-sentence remarks. The software hadn’t yet been optimized for sorting vast amounts of data either. Finding anything at all became difficult as the overloaded database grew buggy and slow. The database seemed to run into a brick wall as tens of thousands of users almost simultaneously stopped using the product. Attempts to fix the system failed. When all the frustration threatened to become a distraction from the work of selling properties, leaders canceled the project. In its eagerness to reap the benefits of a promising idea, Keller Williams had killed the golden goose.
John Keller sees this as the biggest innovation flop in the company’s history because the core concept had held such potential. If the company had taken the time to validate its assumptions through multiple stages of iteration and testing, it might have homed in on a workable approach, avoiding what Keller called “a hectic scramble that came from not having a plan in place.” Once an idea goes south at scale, however, there is rarely any institutional willingness to wind it back down and start refining it from an earlier stage of development. By the time Keller Williams had pulled the plug on its insights database, users had lost interest in investing time in this voluntary, unpaid activity. This project’s failure illustrates the danger of going all- in on an idea, even one with a successful test behind it.
The right validation process is cyclical. You don’t just generate a bunch of ideas, test one, and then scale it up like crazy if it works. Instead, you go through stages: test, analyze results, refine, test again. Our observation, confirmed many times over, is that organizations are so eager to scale promising ideas that they compulsively skip this effort. In their hurry to score a win, they unwittingly undermine their efforts. This is doubly true of companies that are perennially starved for innovation. Be particularly cautious of the urge to run with promising ideas when ideaflow is just ramping up in an organization. There are always problems that must be resolved at each stage of growth before you can move forward. Pace yourself. People blame poor execution when a project goes off-course, but even great drivers can’t steer blind. The sooner you let go of the delusion that you can “eyeball it” or “do it on the fly,” the more consistent your success will be. Test before you invest, not once but at every stage. Testing is forecasting. It’s how you see your success before you achieve it.
There are several reasons for the institutional reluctance to test, false incentives chief among them. If you don’t know any better, testing seems like lots of work for little reward. The innovation process generally begins with a mandate from a leader to make or fix something. No one rises in the ranks of an organization by telling leaders something won’t work, just as scientists don’t win Nobel Prizes for publishing a lack of positive results. If testing is seen as simply ruling something out, a binary act ending in either victory or defeat, the stakes are too high. If you’re going to risk defeat, you might as well aim for the moon. That’s why most people are reluctant to look at any viable-sounding idea too closely before implementing it. Meanwhile, leaders tend to interpret caution and curiosity at the developmental stage as skepticism and procrastination. No one wants to be seen as a momentum-killer.
This resistance fades once everyone understands what testing actually entails. A rapid, scrappy test should take hours, tops. Not weeks, and certainly not months. As we saw with Thomas Edison’s efforts to develop a long-lasting light bulb, experiments aren’t for killing ideas. They’re for filtering the best from the rest. He succeeded so frequently by cramming as many scrappy tests as possible into every twenty-four-hour period. When you boost ideaflow, test-based filtration becomes a necessity. There are too many ideas to consider and, as we’ve seen, our biases tend to steer us from the winners even if it were possible to identify them without real-world data. A good test rules out lots of options that won’t work while homing in on the ones that might, vastly reducing the risk of failure. Reframing testing as a scrappy process of learning, refinement, and validation is the key to circumventing the reluctance to experiment.
As for the effort required, running a test can and should be quick and easy relative to implementing the idea in its final form. You’re going to be testing a lot, so you’re always looking for the biggest bang for your experimental buck. When we design tests, whether for bootstrapped start-ups or multinational corporations with massive R&D budgets, we always optimize for experimental efficiency. The best experiments return lots of actionable data in exchange for a minor investment of time and energy. Why invest months and millions in a new product when a few days and a few hundred dollars might reveal that nobody wants to buy it as currently envisioned? In fact, why pursue any new idea seriously if you don’t have credible evidence of desire?
Test, refine, and test again until you’ve zeroed in on a solution that works. With the right validation process, you’ll know whether an idea has wings long before you hit the end of the runway. In the case of a product, you’ll even know things like how much to charge and how much inventory to keep on hand before you go to market. In this way, you can leverage the full value of your ideas while minimizing the uncertainty and risk involved in making them real.