Innovating systematically in complex conditions through guided trial and error
by Marvin Cheung, Head of Research and Strategy
Trial and error in complex conditions
Architecture professors Horst Rittel and Melvin Webber from UC Berkeley observed some of the properties of complexity in 1973 and called the type of challenges that involve complexity as “wicked problems”. At a basic level, it is when a challenge has factors that are deeply interwoven and cannot be understood in isolation. There is also no way to fully account for the solution’s knock-on effects until it is executed.
Common innovation challenges, such as finding product-market-fit or scaling, can be considered a wicked problem. Even with the most sophisticated technologies, there is no way to know for certain whether a new product will succeed without testing it. Its success will depend on many factors ranging from the quality of the product or its marketing to its final design.
We can use variations of a puzzle as a metaphor for wicked problems and its inverse, “tame” problems. A tame problem is like a thousand-piece puzzle from a box: you can follow the picture, start with the edge pieces, and work towards the middle; it is not easy, but there is a definitive goal and the relationship between the parts is clear.
We can model this through a tree diagram. There are one thousand pieces originally. You can start with any piece. Each time you find a match, the number of possible pieces decreases by one. Like any other games, there are established best practices that can facilitate the process. For example, you can start with the corner pieces and work your way into the center.
A wicked problem is like a hundred ten-piece puzzles mixed into the same box. Some of the ten pieces will produce a more desirable picture than the others, but you only have time to put one or two complete puzzles together. Immediately, we begin to see some characteristics of real world challenges. First, there is very little indicator of what a successful end result looks like. Second, there are resource constraints.
By definition, we know that some form of trial and error is required. There are, however, different kinds of trial and error, as outlined in Donald T. Piele and Larry E. Wood’s essay “Thinking Strategies with the Computer” as part of the anthology The Best of Creative Computing Volume 3 published in 1980 by Creative Computing Press. The three, with varying degrees of effectiveness, are random, systematic, and guided trial and error.
The most basic strategy is random trial and error. This would be equivalent to moving the algebraic symbols around randomly when you are stuck on a math question. You pick up a random piece, build the first puzzle, then pick up another random piece and build the second puzzle. At the end of this process, you have two complete puzzles. The chances of you liking both puzzles is the same as the chances of you hating both of them.
The better strategy is systematic trial and error. Instead of picking pieces at random, we set a parameter of trying to build a puzzle with at least one blue piece, and proceed to list the rest of the colours in order of preference in case there are no blue pieces. In this scenario, we decrease the likelihood of getting a puzzle you really hate, but we still have very little control over the outcomes.
The most effective strategy is guided trial and error. Say we start with trying to build a blue race car puzzle but fail to find any blue pieces. You then decide that orange is your next favourite colour and consider building a monotone puzzle. You discover fifteen orange pieces - either enough for one orange monotone ten-piece puzzle, fifteen puzzles with one orange piece each, or anything in between. Unfortunately, none of the fifteen orange pieces work together. You see the opportunity to build a puzzle with an orange motorcycle and complete it. In this scenario, though the top choice was not available, you manage to find a close second.
The idea of guided trial and error is further observed by mathematician Keith Devlin in 2006 from Stanford’s Center for the Study of Language and Information, with reference to how gamers are reshaping the business world, and by the Headwaters Science Institute on the process of scientific research.
While guided trial and error may seem like an intuitive strategy, we are only just scratching the surface of the challenge. The bigger question remains: How do we apply a guided trial and error process to innovation, and even more importantly, is there a repeatable process we can use to help us innovate systematically? The answer, as it turns out, is not quite so simple.
Existing innovation best practices
We will start with common best practices and build up to our new recommendations. For example, in the guided trial and error scenario, we have already introduced a popular idea in lean startup: iterations. Rapid trial and error will increase the chances of discovering a desirable outcome. We have also introduced the idea of Minimum Viable Product: build only as much as you need to get a sense of the prototype’s desirability.
To introduce the next best practice, we need to develop the metaphor further to better reflect the challenges of innovating in the real world: imagine trying to tackle the hundred ten-piece puzzle challenge in a team of five, where not everyone is allowed to see the puzzle pieces. Your colleagues, investors, advisors, and customers all have different motivations and viewpoints, yet the ultimate success of the project requires some alignment of your stakeholders.
As a response to the complexity - intertwined stakeholder needs, Human-Centered Design (HCD), commonly associated with IDEO and the Stanford Design School, gained recognition. One of the hallmarks of HCD is the use of post-it notes and mind maps to facilitate communications and help secure stakeholder buy-in.
If we return to the original guided trial and error scenario, we would still see two missing pieces in our current understanding of innovation: (1) how we choose our parameters, and (2) how we evaluate solutions within the parameters. We will address (2) in the next section of this Coursebook.
To understand the knowledge gap, we need to recognize the limitations of the hundred ten-piece puzzles metaphor in representing innovation challenges. For one, constraints are rarely as convenient as “puzzle pieces with only orange (monotone)”, even “puzzle pieces with some orange” would have increased the level of difficulty significantly. We have also taken perfect eyesight and perfect information for granted here. In real life, we have to work with imperfect instruments, as well as incomplete, inaccurate, and even incorrect information.
At its core, (1) is the more abstract part of how we think about problems: how should we guide the thinking process in a way that enables innovation? From a practical point of view, how do we connect the many frameworks available and choose the right parameter at the right time?
We found the answer in the field of design. Design is of interest to the field of innovation, because designers produce creative outcomes in every project. We can observe the properties of wicked problems within a project: there are clients and stakeholders with different needs, and at each point there is an infinite number of possibilities. An architecture can take many forms, and an empty canvas can carry whatever image you put on it. This is elaborated on in Richard Buchanan’s essay from 1992, “Wicked Problems in Design Thinking”, published by MIT Press.
Design thinking in this context is not the same as HCD. It comes as a surprise to many people including designers, that a lot of literature on design thinking predates IDEO. In fact, while the design thinking literature was being canonized in 1991 at the first symposium on Research in Design Thinking held at TU Delft, nowhere does David Kelley, co-founder of IDEO, mention the phrase “design thinking'' at his Ted Talk, titled “Human-centered Design” in 2002. Stanford’s D. School, co-founded by David Kelley as well, only began teaching ‘design thinking’ in 2005, according to Design Thinking: Understand - Improve - Apply, published by Springer in 2011. As a further demonstration of the difference between Pre-HCD Design Thinking and the current understanding of Design Thinking, none of the literature quoted below is included in IDEO’s article on the history and evolution of design thinking.
It seems unavoidable that we discuss, however briefly, the term “design thinking”: to think like a designer. As you may have noticed, design thinking has not been included in the titles of this essay. This is not only because of the semantic shift from the original meaning to HCD, but also because the term design thinking itself misses the point. All of the previous attempts to “scientise” design, as observed by Nigel Cross in his essay “Design Discipline versus Design Science” from 2001 published by MIT Press, have inevitably failed. Attempts to codify design ignores a fundamental truth of the discipline: art and design challenges the boundaries of the norms. The ways of thinking are not static. Here, we are not so much interested in the subject of design itself or how we can define a way of thinking exclusive to designers, but the subject of innovation and what we can learn from pre-HCD design thinking literature.
We can begin by reformulating (1) in the language of design. “In design, ‘the solution’ does not arise directly from ‘the problem’; the designer’s attention oscillates, or commutes, between the two, and an understanding of both gradually develops, as Archer (1979) has suggested [...] Designers use alternative solution conjectures as a means of developing their understanding of the problem,” notes Nigel Cross in the proceedings of Research in Design Thinking published in 1992.
While it may seem at first glance that the creative process is like a pendulum swinging between two distinct ends with no apparent start or end, we know this to be untrue. As Cross observes, “a design solution is not an arbitrary construct - it usually bears some relationship to the problem, as given, which is, after all, the starting condition for considering solution possibilities.”
We can cut through the pendulum swing and frame the creative process instead as a starting condition, followed by a series of problem-solution pairs. We have seen this at work in the guided trial and error scenario when we started with the parameter “blue racecar”, moved to the solution “no blue pieces”, and then to the second parameter “orange monotone puzzle” with the second solution “fifteen orange pieces”. We can also see how “blue racecar”, for example, serves as a parameter, a solution conjecture, and a hypothesis of a possible and highly desirable outcome.
We can map our starting condition as well as problem-solution pairs in a tree diagram too. At the top layer, we have one starting condition, then we branch off to our second layer with the first problem-solution pair “blue racecar” and “no blue pieces”. Also on the second layer is the problem-solution pair “orange monotone puzzle” and “fifteen orange pieces”, which branches off into the third layer with the problem-solution pair “none of the fifteen orange pieces work together” and “build the puzzle with an orange motorcycle”.
Much of what we described above coincides with existing thinking in business. DMAIC in lean six sigma advocates for a continuous effort to Define, Measure, Analyze, Improve, and Control. The McKinsey Mind, published by McGraw-HIll Education in 2001, discusses in detail the importance of a hypothesis-led investigation, so we do not have to “boil the ocean” looking for solutions, and the use of logic trees with issues and sub-issues. Where we see a clear point of divergence is the MECE principle (Mutually Exclusive, Collectively Exhaustive) in the making of the logic tree. By definition, complexity involves non-mutually exclusive parts that are closely interconnected.
Innovating systematically in complex conditions
This is when we need to leave the metaphor behind and instead consider the real world example described in Richard Buchanan’s essay from 1992: “Managers of a large retail chain were puzzled that customers had difficulty navigating through their stores to find merchandise. Traditional graphic design yielded larger signs but no apparent improvement in navigation - the larger the sign, the more likely people were to ignore it. Finally a design consultant suggested that the problem should be studied from the perspective of the flow of customer experience. After a period of observing shoppers walk through stores, the consultant concluded that people often navigate among different sections of a store by looking for the most familiar and representative examples of a particular type of product. This led to a change in display strategy, placing those products that people are most likely to identify in prominent positions.”
We can see many of the prior principles apply here:
they have iterated and created small tests
there is a starting condition: customers have difficulty navigating the client’s stores, and two problem-solution pairs
they have taken a hypothesis-led approach
The larger question still remains: how did they move from signage and graphic design, to product placement and customer experience? Is that just a coincidence? Pre-HCD Design thinking literature says otherwise. Specifically, we observed three types of flexibility that enable innovation. First, flexibility with the starting condition. Second, flexibility with exploring different layers of abstraction. Thirdly, flexibility with the frameworks used. We also found that designers add in information and perspective to manage the immense flexibility of the process.
The first strategy we found was having flexibility with the starting condition. Quoted from Cross’ 1992 essay, “Thomas and Carroll (1979) concluded that ‘Design is a type of problem solving in which the problem solver views the problem or acts as though there is some ill-definedness in the goals, initial conditions or allowable transformations”. The literature makes explicit that there is not just flexibility in how we ideate hypotheses, but that we should be ready to accept that the original starting condition is renegotiable. A starting condition is not written in stone and we can pivot.
The second strategy we found was allowing flexibility with exploring the different layers of abstraction. Because the parts and the wholes are interconnected in a complex challenge and the strict boundaries between the parts and the whole is unclear, we can and should be able to move between different levels of abstraction. Elaborated upon in the Doctrine of Placements by Buchanan in his 1992 essay, Buchanan observes how many designers work across the four broad areas he identified, including symbolic and visual communications, material objects, activities and organized services, as well as complex systems or environments to deliver creative outcomes.
Specifically, Buchanan notices the way the designers treat the areas as interconnected “with no priority given to any single one”. While the “sequence of signs, things, actions, and thought could be regarded as an ascent from confusing parts to orderly wholes”, “there is no reason to believe that parts and wholes must be treated in ascending rather than descending order.”
One of the implications of this, is the idea that when we structure our thinking, we can at any point, move up and across branches in the tree. Unlike when we are executing a project where going down a certain path requires us to commit significant amounts of time and resources, creating a unidirectional navigation pattern down a decision tree, you can change paths in the thinking process with far fewer commitments, creating a multi-directional navigation pattern in a tree.
The third strategy, flexibility with frameworks used, was also identified in Buchanan’s 1992 publication: “Although the [retail chain navigation challenge] is a minor example, it does illustrate a double repositioning of the design problem [...] There are so many examples of conceptual repositioning in design that it is surprising that no one has recognized the systematic pattern of invention that lies behind design thinking in the twentieth century [...] Understanding the difference between a category and a placement is essential if design thinking is to be regarded as more than a series of creative accidents. Categories have fixed meanings that are accepted within the framework of a theory or a philosophy, and serve as the basis for analyzing what already exists. Placements have boundaries to shape and constraint meaning, but are not rigidly fixed and determinate.”
Creatively switching between disciplines, frameworks, and layers of abstraction all help illuminate the relationship between the parts and the wholes and enable us to deliver innovation in complex situations. This applies to both the problem formulation process and the solution generation process. Emerging literature in Cross Domain Deterrence reveals the efficiency of leveraging capabilities in one domain to compensate and strengthen the capabilities of another. Quite simply, formulating solution combinations across departments gives us more flexibility, options, and control.
Having too many options can feel overwhelming. Indeed, this seems to add to the inherent ambiguity of the innovation process. Pre-HCD Design Thinking literature has in fact observed ways in which designers manage the immense flexibility. Cross in the 1992 publication notes, “In early observational studies of urban designers and planners, Levin (1965) realized that they ‘added information’ to the problem as given, simply in order to make a resolution of the problem possible [...] Darke (1979) from her interviews with successful architects [...] also concluded that the architects had all found, generated or imposed particular strong constraints, or a narrow set of objectives, upon the problem, in order to help generate the early solution concept.” In innovation, these can include your values: what you will and will not do, it can be a vision: how your venture will align with a well researched projection of the future, or it can be anything that you discover using different frameworks.