Note: If you browse this site it will use temporary 'session cookies' to assist navigation and keep an anonymous record of page visits. More info…

Design as a Verb

2

Design as a verb is the process of going from initial ideas to a fully realized product. It is sometimes referred to as design and development, but is sometimes abbreviated with just the term design. The term ‘educational design’ is typically concerned with design as a noun (e.g., Davis & Krajcik, 2005)—that is with the thing being produced, be it a unit of study, an assessment, a whole curriculum, a professional development workshop, an educational building, or some combination of these products into a larger system. In attending to the products themselves, much attention is given to relatively general design elements that should be contained in a good design (e.g., a good design should help teachers recognize and build upon conceptual diversity in student thinking). The discussion of good design elements sometimes focuses on particular subsystems of the larger product (e.g., assessment, lecturing within a curriculum), but also on principles and values that cut across subsystems (e.g., scaffolding as a principle and critical thinking as a value).

However, just as the word ‘design’ has two main meanings (as a noun and as a verb), so too does ‘educational design’. To further unpack the verb meaning, consider the verb definitions of ‘design’ from Marriam-Webster ("Design," 2008): “1. To create, fashion, execute, or construct according to plan. 2. To conceive and plan out in the mind. 3. To indicate with a distinctive mark, sign, or name. 4. To make a drawing, pattern, or sketch of.” The first meaning highlights the planful nature of designing—there is system to the madness. The second meaning highlights the important mental activities of the designer(s). The third meaning is archaic and perhaps less relevant; however, design is very ego-involving and the designer often tries to bring a distinctive mark to the project. The fourth meaning highlights that the process of design is often focused on creation/manipulation of prototypes of the final product rather than directly on the final product itself.

None of these meanings of design as a verb reveal much about the subcomponents of design as an activity: Does a design process have meaningful steps along the way, and is there a sensible organization to these steps? It is likely that design is not further unpacked in the definition for two reasons. First, people generally do not have good direct access to or awareness of mental processes that they follow even when there is strong regularity and organization to their behavior (Anderson, Lebiere, & Lovett, 1998). Second, it may be very difficult to find components that are true of all designing, across the range of things that can be designed and the range of quality of designers.

However, despite our inability to fully and directly reflect on design processes, and despite the likely variety of design processes that occur, I would like to submit that there is great value in thinking more rigorously about design processes. In all areas of human activity, processes that people follow have a great influence on both the quality of products that are produced and the speed with which the products are produced (Ericsson & Charness, 1994). Further, there are very large individual differences (Hayes, 1985; Simonton, 1997): some individuals regularly produce high quality products quickly and others take a long time to produce very mediocre products at best. Because these individual differences repeat across instances, there must be something reliably different about the processes being followed (rather than just stochastic strokes of good or bad luck). If we are to generally improve the quality and time-to-completion of designs, we must find out what better designers are doing and transfer those practices to weaker designers.

Why Educational Design Should Consider The Analogy To Engineering Design

3

How do educational designers develop their expertise? The lack of scholarship on the process of educational design suggests that it is primarily a craft-based kind of expertise. That is, it is likely that educational designers develop their process expertise through trial-and-error and through apprenticing with other educational designers. Disciplinary education (e.g., education in mathematics, physics, composition, etc.) pays little attention to educational issues and no attention to the design of educational objects. Some books on the educational design process exist and are popular (e.g., Wiggins & McTighe, 2005), but the research base underlying the content of those books is not clear. Schools of education rarely offer courses on educational design, and when they do, the content is craft-based knowledge and sometimes the focus is on design as a noun—what makes for a good or bad product rather than a process that leads to good products.

Craft-based performance can be very good. Or said another way, some people can become amazing performers through apprenticeship or craft-based education. However, it is not a very efficient process of training—the lessons learned by one artisan travel very slowly to other artisans within the same organization and perhaps not at all to artisans in other organizations. As an extreme example, chicken sexing used to be treated in an apprenticeship fashion and required many years to develop the ability to reliably determine visually the sex of a chick. However, through explicit instruction regarding the process used by experts, that same high level of skill can be obtained in less than a day of instruction and practice (Biederman & Shiffrar, 1987).

In contrast to the lack of scholarship and formal education on educational design processes, there is considerable scholarship and formal education on engineering design processes: there are undergraduate and graduate courses on engineering design, there are textbooks that support these courses (e.g., popular offerings include The Mechanical Design Process (Ullman, 2003), Product Design (Otto & Wood, 2001), Product Design and Development (Ulrich & Eppinger, 2008), and the Design of Things to Come (Vogel, Cagan, & Boatwright, 2005)), and there are journals in which scholarship on design processes is published (e.g., Journal of Engineering Design, Journal of Mechanical Design, Design Studies, and Journal of Research in Engineering Design). Thus, engineering design is a rich model to examine for improving scholarship and practice in educational design.

Clearly engineering products and educational products have differences and thus the processes of engineering design and educational design are also likely to have differences. However, there are also many similarities between engineering products and educational products, and thus the differences in effective design processes may not be so large. For example, in both settings the products are usually complex systems (with multiple, complex interacting subsystems) being designed for competing constraints (i.e., cannot all be perfectly met in one design) to be used by a range of users (whose requirements are not well understood or entirely fixed in advance), and whose design requires bringing together expertise from a variety of disciplines. Further, engineering covers a very wide ranged of designed objects—products (e.g., iPods), processes (e.g., a more efficient way of manufacturing a given chemical or scheduling flights within a network of cities), complex systems (e.g., an office building or a space shuttle), or services (e.g., an offshore call center). Design processes of relevance to that range of designed objects will likely transfer to at least a range of educational design.

This paper describes some major lessons learned in the scholarship on engineering design processes and explores the mapping of these lessons to education. I have contributed to that scholarship on engineering design and have led the design of educational products in science education, writing education, and math education. The mapping exercise in this paper is heavily influenced by those experiences, and in-depth linked examples from these experiences are provided to help make more concrete the way in which these engineering processes might be realized in educational design.

Why Design Process Has a Large Influence on Design Outcomes

4

I begin with an abstract analysis of why the design process can have such a large influence on the success of design outcomes. There are two related factors to consider. The first factor is an analysis of costs and flexibility of design change. Early on in the design process, there is great flexibility in the possible designs that can be pursued—there is relatively little design time or monetary cost in changing design concepts early on. However, as time goes on, as particular design concepts are developed in more detail and with increasing amounts of lab testing, field testing, marketing, and creation of production facilities, the cost of changing the fundamental design concepts becomes overwhelming (see Figure 1). Thus, it is extremely important to generate and select a good design concept relatively early on in the process—committing to weak choices through an ineffective design process can have disastrous consequences.

At the same time, the early design phase is fraught with uncertainty. The marketplace may change by the time the product is released and the underlying technology may need to be developed and thus is not yet fully understood. Even worse, the range of possible design concepts is impossibly large for the mind to fully consider—it is very tempting to re-employ an existing design or pursue the first seemingly workable design that comes to mind.

In sum, it is very important to do a good job in the early phases of design and it is very hard to do so. Scholarship on formal engineering design processes has paid considerable attention to improving conceptual design processes, facilitating more reliable decision making early in the process.

Processes That Have Been Shown To Be Important In Engineering Design

5

Mehalik and Schunn (2006) reviewed the empirical literature on engineering design to determine which design processes were most reliably associated with expert designers / higher quality designs. Note the plural on processes. Just as science can be best characterized as a set of processes that come together in different ways in different situations (Chinn & Malhotra, 2002), design can also be best characterized as a set of processes that are used in varying combinations depending upon the situation. In fact, one of the hallmarks of good engineering design is using an iterative/interactive design process, rather than a simple linear process.

Although the design process should be iterative/interactive, there is a rough ordering of steps. I discuss the other important processes in the rough order they would typically occur, including some mapping to the educational design context. Two of them (Develop a subsystem decomposition, and Develop analytic models), however, are complex big ideas that will be discussed in major sections of their own.

Create Requirements & Metrics

One big idea in educational design is to be clear about the learning goals in advance: what do you want students to know and be able to do (Wiggins & McTighe, 2005)? From the engineering design literature, we know that thinking about requirements like this is a good early step. However, thinking in terms of “requirements” more generally reveals that learning goals are just one element of the design goals that should be clearly specified at the beginning. Formally specifying all the requirements brings attention to the full range of dimensions that must influence the choice of designs, rather than allowing the designers to forget important factors (e.g., time to market or material costs or training costs). The specific requirements are developed by interviewing customers, analyzing the market place, and through expert input. That is, they are chosen carefully to reflect what dimensions are necessary for success, and thus can place important constraints on proposed designs. Possible additional factors that might relevant include: Is it important for design to be effective with teachers with relatively little content knowledge? Is the product meant to influence student identity (e.g., desire to become a scientist or engineer)?

Also note that general requirements specified in vague terms are not helpful because different specifications can require substantially different designs. For example, is it important to do well on a particular kind of test, such as a state test that is essay-based or multiple-choice-based? Specifying the requirements in terms of specific values on particular metrics (e.g., a certain effect size of learning on a particular test) allows the design team to find possible solutions that are satisfycing (i.e., good enough on all dimensions). (See Appendix A)

There are several common mistakes in putting together requirements for a design task. The first common mistake is giving a solution instead of a requirement. The requirements are about the ends that must be met (e.g., students must learn X) and the constraints on resources (e.g., material cost, development time, or amount of teacher professional development that can be assumed). Sometimes, however, the designer may mistakenly list a particular way of achieving those ends as part of the requirements. For example, a science unit must include a hands-on activity, or a math unit must include a particular kind of diagram. These are solutions. Although they may be good solutions to the actual design problem at hand, listing them as requirements prevents the designer from considering alternatives (including minor variations) that might prove to be more effective overall.

The second common mistake is getting confused between absolutely necessary requirements (called ‘must-have’ requirements) and useful but not actually necessary requirements (called ‘nice-to-have’ requirements). Many widely used curricula become difficult for teachers to learn how to use because so many bonus features get added (e.g., each lesson can be enacted in very different ways, each requiring different instructions). One can think of this problem as having added too many nice-to-have requirements while significantly weakening some must-have requirements (e.g., the requirement that most teachers are supported in focusing on big ideas, or the requirement that most teachers understand how to enact the curriculum with rigor). As a variation of this issue, requirements are specified with both minimal values and ideal values. In thinking about tradeoffs among alternative solutions, it is important to not give up on a minimal value on one dimension in order to obtain an ideal value on another dimension. For example, it would be a bad idea to change the duration of a curricular unit to a length of time that is simply not supportable by typical school districts even if it would improve learning to some more ideal level.

Explore Alternatives

Across a wide variety of design problems, one of the best predictors of success in design is the number of different designs that were considered, although considering too many possibilities without carefully exploring any of them is bad, too. Alternatives can be considered virtually/mentally or they can be explored empirically in prototype form. Exactly which form is best depends upon the accuracy of evaluations on the virtual/mental form and the time cost of the empirical tests — in most areas of educational design, our theories and past experiences are not highly predictive, and thus some form of empirical testing is likely to be important. But regardless of format, multiple possibilities should be explored in the early phases of design before a main choice is selected. Further, these alternatives must be explored far enough that they can be evaluated in some way with respect to the key requirements, even if only through expert judgment of how well they are likely to meet the key requirements (See Appendix B).

A controversial issue related to exploring multiple alternatives early in design is group brainstorming. The typical form of brainstorming involves generation in a group while judgment is suspended. The research literature has repeatedly found that this kind of brainstorming is very ineffective, most likely because the group tends to converge on shared ideas that are not necessarily good ideas. More effective is to allow some critical evaluation or, even better, to have individuals brainstorm alone, and then have the group critically evaluate the longer listed of collectively generated ideas (Paulus & Yang, 2000).

Explore Problem Representation

Developing a solution that is noticeably better than past solutions is a kind of insight problem. A common feature of insight problems is that there is a large space of alternatives and no clear feedback from virtual or empirical tests of each considered option on whether one is getting closer to or further from finding a significantly better solution (Perkins, 1994). Imagine digging for gold in a random spot, finding nothing, and then having no idea about which direction to search for a better digging location. In other words, one could search a long time and still not find something better. Rather than relying on luck in finding the needle-in-the-haystack through blind search, a much more effective method is to develop a more helpful representation of the problem that highlights where solutions are likely (and unlikely) to be (Kaplan & Simon, 1990; Lovett & Schunn, 1999). To develop this better representation, some heuristics are useful: 1) what do failures (or successes) tend to have in common?, and 2) can I discard some features that tend not to matter? Good representations help predict success. For example, in the design work in our own lab, we find it useful to separate designs into the more abstract feature of curricula that act to separate the teacher from the students (bad) vs. curricula that act to bring the teacher in close regular contact with the students (good) rather than in terms of more superficial features like many different variations of video-based demonstrations (good and bad) or paper-based instructions (good and bad).

Explore End-User Perspective

Usually designers are not designing something for themselves or often not even for people who are like them. In educational design, we usually design materials for people much younger than us, often for people from different demographics from us, and certainly for people who know less that we do about the topic at hand. Further, design teams are often multidisciplinary, and thus each member of the design team is removed from the end-user in a different way. Under these circumstances, it is rather easy to design a solution that is good for the designers as users but not for the actual users. For example, several members of most educational design teams have been teachers at one point, and likely teachers with strong content knowledge and strong pedagogy skills. These expert-teacher designers are often prone to design products that would work with them as teachers, but they are not very representative of teachers in general. Thus, it is regularly useful for the designers to explore the perspective of the end-users during the design process, to re-ground themselves in the needs and resources of the end-user. Some visits to physical sites early in the process is a useful experience because it enables the designer to more easily mentally simulate their design in the target environment—mental simulations are a very common way in which designers resolve uncertainties in their design process (Christensen & Schunn, 2008). In educational design, the early physical visits will often involve observing classrooms to understand the physical layout constraints, common teacher/student interactions, student background knowledge, and/or student goals related to the topic at hand. Just involving people who represent the classroom voice on the design team is often not good enough—it has been amazing to me how little classroom observation was done by various school district personnel, who in theory represent the classroom perspective on design teams.

Important Design Process Concepts

6

In addition to thinking about processes of design, there are also important design concepts that are intimately related to the design process. There is not sufficient space here to go into great detail on all of the relevant concepts, but there are two big concepts that organize many of the detailed concepts: 1) Systems design and 2) Optimizing over constraints (Silk & Schunn, 2008).

Central Engineering Design Concept 1: Systems Design

Engineers conceive of all designed objects (watches, cars, planes, buildings) as systems, essentially a black box with inputs and outputs, as well as subsystems that help achieve the overall system functions. This kind of “systems thinking” focuses on the functional nature of the products, which is critical to improving functionality.

Design of individual subsystems. With respect to the design process, one very important aspect of systems thinking is the decomposition of the whole system into functional subsystems. For example, any alarm system breaks down into detector, indicator, power, and control subsystems. The advantage of thinking in terms of functional subsystems (rather than in terms of the whole system or in terms of parts) is that the designer can make more rapid progress overall by working separately for significant periods of time on each subsystem, even though some attention to integration will also be required. For example, one reason that the Wright brothers made rapid progress on motorized flight when many other designers made no progress at all is because they alone worked separately on lift, propulsion, and control subsystems (Bradshaw, in press). Early on, they focused on the lift subsystem, testing many different wing shapes in a wind tunnel to see which shapes gave the most lift. Then they turned to different mechanisms for controlling the plane. Finally, they explored different propulsion mechanisms. By contrast, their competitors always tested whole airplanes. It took much longer to construct and test an entire airplane, and failures were very hard to diagnose in the larger system (e.g., was the wing shape wrong or was the control mechanism off?).

To provide an application of this incremental focus on subsystems in educational design, one might think about separately working on how new ideas/skills are introduced from how those ideas/skills are practiced to fluency if one thought of those as different subsystems (i.e., a knowledge introduction subsystem and a practice subsystem). A design team might first focus on the challenge of introducing the ideas. What prior knowledge do the students bring to the table? What is conceptually challenging about the underlying ideas? How can initial understanding best be measured? What level of initial understanding is reasonable to require of the designed subsystem? The design team could empirically test various methods for introducing the ideas relatively rapidly when it is decoupled from the practice subsystem. Later, with a solid method for introducing the ideas in hand, the design team could turn to development and testing of the practice subsystem. It would be hard to efficiently evaluate the practice subsystem without a functioning method for introducing the concepts. Similarly, there is no point to evaluating methods for training teachers without having core elements of student materials in place.

What are all the pieces of the whole? A second important aspect of systems thinking in design is that it brings attention to all the critical subsystems and the ways in which they depend upon one another. Consider Edison’s invention of the light bulb (Hughes, 1987). In fact, Edison had to invent a whole system rather than just the light bulb: a way of generating electricity, a way of getting electricity from the power plant to the consumer, a way of distributing electricity in the house, and a way of metering electricity consumption so that cost could be fairly distributed among consumers. Of course, in many cases, the engineering is designing one piece (e.g., a more energy efficient light bulb) that fits into an existing system rather than designing the whole system. However, the engineer needs to understand the overall system, which pieces can be used, and which pieces must be redesigned (e.g., a halogen light bulb require different kinds of fixtures).

In educational design, there are many ways of breaking down the overall system into functional subsystems because there are many theoretical models of teaching and learning. In general, a good subsystem decomposition is one that supports separate design activities on each component: 1) clear functional goals of each subsystem, 2) clear metrics for each subsystem, and 3) opportunities for separate testing of each subsystem. Figure 2 presents one subsystem decomposition that I have used in the design of middle and high school science curricular units (see Appendix C). Note that each subsystem is organized more by functional roles than by physical identity. For example, it includes a teacher support system rather than specific teacher professional development, and teachers themselves are actually part of two different subsystems.

Central Engineering Design Concept 2: Optimizing over Constraints

A second design concept is the general idea of optimizing over constraints. In all real design tasks, there will be competing constraints such that designs that produce the best outcome on one constraint will not be the designs that produce the best outcome on another constraint. For example, the cheapest solution is never the highest functioning solution. In general, there will be many tradeoffs between constraints. Engineers find it very useful to think in terms of tradeoffs between these constraints because it enables decision-making rather than whining about missing features or suboptimal results on a particular dimension: if you want to improve results on one dimension, where else are you likely to have to make concessions? For example, if you want a car that goes faster, are you willing to pay more money? If you want to use materials that demand higher levels of student thinking, are you able to spend more time on teacher professional development? If one focuses entirely on user requests for features, it is very tempting to try to include everything. In educational settings, curriculum adoption is often driven by the presence of many relatively superficial features (e.g., coverage of my favorite contents, inclusion of my favorite teaching methods, or support for my favorite special needs populations). But including all these extra components comes at significant costs: the quality of every component must go down or the cost (in development time and monetary cost) must go up significantly.

To facilitate the tradeoff decisions, explicit requirements (and approximate weightings of each requirement) become very important in selecting a relatively optimal solution to the competing constraints. For example, zero development cost would be best, but perhaps development costs of $100,000 are acceptable and there is no need to worry about reducing value on other dimensions in order to go below that development cost.

Another important element of optimizing designs is specifying a particular user niche (or at least range of target niches). One cannot build a car that is optimized for a family of 8 and optimized for a young, unmarried individual. Similarly, it is impossible to have a curriculum that is optimized for very different learning environments (e.g., consider the needs of a physics classroom in which every student is very fluent with calculus versus a physics classroom in which most students are struggling with elementary algebra). One can choose to optimize over several different niches, but one should do that explicitly recognizing the extent to which that will reduce the overall value to each niche.

The concept of optimizing sounds very mathematical. In engineering, it often is very mathematical and is part of what is called engineering analysis—some engineers argue that this mathematical analysis is the heart of true engineering (versus what they pejoratively call tinkering). When working on some subsystem of an overall design, engineers will carefully read up on the relevant bits of physics, chemistry, biology, etc. (or conduct new fundamental science themselves when it does not already exist for the particular situation at hand) to understand precisely how to predict the outcomes of different design decisions on various dimensions. With a mathematical function in hand, the designer (usually via analysis software) can then determine exactly which choices produce the optimal results. As one impressive example, the Boeing 777, with its millions of parts, was designed entirely through virtual design (i.e., no physical prototypes had to be developed to debug design options) . This kind of mathematical analysis and optimization is much more efficient than an empirical test-and-evaluate blind search. Further, the mathematical analysis can reveal the unfortunate situation of having no possible solutions within the space of considered options, which would then trigger the development of whole new classes of solutions. (Boeing actually became nervous during the virtual design process and ordered a mockup of the nose section for tests. However, the tests were so successful that all other mockup tests were cancelled).

One might argue that education is not so amenable to mathematical optimization because we are very far from having precise mathematical functions that accurately predict outcomes of different educational design choices. However, even when exact outcomes cannot be predicted, often one can predict the approximate outcome relative to specific requirements on each dimension (e.g., above or below the threshold). In this way, the designer can optimize a simple function that computes the number of dimensions over which the minimal or ideal requirement values were met (e.g., does this design choice give me a product that is inexpensive enough to develop, fast enough to develop, and provide learning outcomes that meet the given goal for the selected target population?). Further, there will be some areas of educational design where some mathematical functions are known, at least within a particular niche. Development cost functions are often known by educational design houses. Learning functions are becoming known for some settings. For example, overall we are coming to learn that teaching quality (and student learning) do not change as a linear function of teacher professional development hours; rather it appears to be an S-curve in which hours below some threshold are all equally useless, and then diminishing returns are obtained above some value. The exact crossover points are coming to emerge for particular situations (e.g., fewer than 20 hours of professional development appears to be useless for secondary math and science teachers in the US (Garet, Porter, Desimone, Birman, & Yoon, 2001; Porter, Garet, Desimone, & Birman, 2003). See Appendix D.

Cognitive Bottlenecks in Design: Design Fixation

7

Another way to think about design processes is to consider the designers themselves and how (and why) the processes are necessary from a cognitive psychology perspective. Although there are many jokes about how engineers are very different people from everyone else, in all likelihood the same cognitive limitations that are true for engineers in their design work will likely be problematic for educational designers in their design work. One particularly salient cognitive challenge in design is design fixation. That is, designers often get stuck on their current idea for the design and have a very difficult time thinking of other possibilities, even when it is clear that current idea has very large problems. Cognitively, this design fixation has been connected to a very general memory problem (Smith, Ward, & Schumacher, 1993) in which one idea in working memory limits access to other ideas in long-term memory (e.g., when trying to retrieve the name of a person, being given one incorrect name can block the retrieval of the correct name).

One classic study of design fixation highlights how powerful the effect can be (Jansson & Smith, 1991). In this study, advanced engineering undergraduates and practicing mechanical engineers were to design a new product (e.g., a spill-proof coffee mug). Some designers where shown an initial example which included some undesirable features (e.g., a straw)—participants were told what the undesirable features where and why they were undesirable (e.g., a straw will always leak). Nonetheless, designers shown the initial example produced worse solutions, frequently having many aspects from the provided example in their final solutions, including the specific aspects that were described as undesirable (i.e., a straw).

In educational design settings, there are usually plenty of prior examples that one could examine (e.g., other textbooks on the same topic). On the one hand, examining these prior examples can produce fixation in the designers. Even elements of the prior design that were thought to be poor design choices might leak into the newly designed curriculum. On the other hand, prior designs have found ways of solving many complex problems; it is foolish to start over from scratch in every educational design project.

The solution, it turns out, lies in how and when one thinks about prior examples. There is a famous Russian design methodology called TRIZ (which roughly means “The theory of inventor's problem solving”) that embodies a way around the dilemma (Altshuller, 1973). Among the various elements of TRIZ, one element involves going from a specific design problem to an abstract description of the problem. Then design principles can be applied to the abstract problem to retrieve general solutions that can be adapted to the specific design problem (See Appendix E).

As a variation of this idea, designers can consider several instances of the particular problem at hand, and then try to infer that abstract class to which the instances belong. Often the deep structure of a problem is hidden among many superficial features; comparison of cases is known to be a very effective way of highlighting the deep structure, even for novices (Holyoak & Koh, 1987; Loewenstein, Thompson, & Gentner, 2003). The more abstract, functional representation is particularly useful for helping the designer think of other prior examples from different domains (i.e., analogies) that might suggest new solutions for the current design task. In general, analogies are thought to be very useful for innovative design (Christensen & Schunn, 2007; Dahl & Moreau, 2002) and overcoming fixation (Christensen & Schunn, 2005). Abstract representations (e.g., that use functional labels) appear to help the designers generate useful analogies (Linsey, Laux, Clauss, Wood, & Markman, 2007; Tseng, Moss, Cagan, & Kotovsky, 2008).

Is Educational Design Really Like Engineering Design? Consideration Of Possible Differences

8

Ability to Do Design Analysis and Optimization

I have already touched upon one huge difference between engineering design and educational design, namely the extent to which precise quantitative analysis can be done to reliably guide design decisions and optimization. In education, much research uses qualitative dependent measures (e.g., interviews) rather than quantitative dependent measures (e.g., test scores), and the research that does use quantitative outcome measures generally tries to establish the absence or presence of causal effects of other variables rather than trying to specify the magnitude of the effects, never mind the shape of the output function across many different particular choices (e.g., the exact learning gains as a function of students in a team). Further, many, many educational designs are poorly evaluated or completely unevaluated (at least in publicly available format), and thus there is very little that can be said a priori with great confidence. Thus, it will be a long, long time before we can sensibly design an educational system entirely virtually. However, as I noted earlier, there still can be some quantitative thinking in educational design to help organize thinking about design tradeoffs. Finally, there are many processes within engineering design that are not mathematical in nature (e.g., design work by subsystem, considering multiple options, developing more functionally relevant representations of the design problem), and the applicability of those processes to educational design will not depend upon this issue of mathematical analysis.

Ability to Do Rapid Prototyping

Although considerable design work can be done virtually in engineering, a lot of testing is also done on physical prototypes. Modern engineering labs have very sophisticated equipment for building prototypes. For example, highly accurate plastic prototypes can be constructed in just a few minutes from a 3D drawing. This ability to rapidly prototype physical objects is a nice compliment to the virtual design; in either case, testing of many designs can be done quickly and cost effectively. Perhaps educational design is less amenable to rapid testing because the testing requires access to students (or other relevant users) for testing; rapid testing is key for rapid prototyping to be of any use. Students tend not to be hanging out in a lab available for random testing, or even worse, might be at the relevant moment of learning only once a year. That being said, some testing of prototypes is possible in educational design. Major curricula take many years to fully develop, and variations of prototypes can be tested in parallel with different student groups. The key here is that empirical testing of simplified pieces can provide very useful feedback, and that remains true of educational design (Nieveen, 1999).

The Physicality of the Designed Objects

Engineering design is often focused on physical objects that depend heavily on physical laws / physical interactions with users. By contrast, educational design is physically often relatively simple and the complexity lies in symbols whose physical manifestation is almost irrelevant. One might argue that symbols are much more contextually fluid in how they function and thus educational design is inherently more contextually fluid (i.e., hard to predict a priori across contexts of implementation). However, the fluidity of the learning challenges across contexts is likely exaggerated. Physics misconceptions are remarkably robust across ages, cultures, and instruction. Good instructional methods appear to be useful in many different contexts (consider for example the significant interest in the US in educational methods developed in Japan). Furthermore, much of the research into design processes involved the study of software engineering practices (Mehalik & Schunn, 2006), which is certainly more symbolic than physical in nature.

Complexity of the Designed System

A simple toilet is complex enough to be functionally opaque to most adults (Keil, 2003). But the complexity of even an airplane or an office building seems small relative to the complexity of the mind of just one child—reverse engineering the brain is one of the grand engineering challenges declared by the US National Academy of Engineering (http://www.engineeringchallenges.org/). By extension, the complexity of many classrooms of teachers and learners seems overwhelming complex at times.

It is hard to know how to objectively compare apples (bridges) and oranges (classrooms) in terms of complexity. But this argument seems to be a grass-is-greener argument—the challenges faced by engineers likely seems more complex to them than the challenge of educational design. Airplanes have millions of parts that must function together in very precise ways with disastrous consequences to design errors. Industrial engineers use well-structured design methods to engineer hospital systems and factories, and hospital systems and factories also have many humans as a core part of the engineered object. Thus, relative complexity of educational settings is not likely to be a key difference that undermines the applicability of engineering design processes.

The Road Ahead

9

I have examined possible mappings of engineering design processes into educational design processes. My educational design group has found this analogy to be very productive. The examples described in this article are only a few cases of this kind of engineering thinking applied to educational design. Across examples, the elements that we have used most regularly in our successes include: Being explicit about requirements and using those requirements to drive design decisions, a subsystems decomposition that paid close attention to the curriculum and professional development, and designing many feedback opportunities into the testing process.

In this paper, I have just scratched the surface of the scholarship on engineering design. In addition to considerably more research on design processes, there are other aspects of engineering design that might be fruitfully brought over to educational design (e.g., methods specific to complex systems design, methods for dealing with teamwork and interdisciplinary collaboration, methods for designing for sustainability, and entrepreneurship to name just a few). Additional work should examine the mapping of those aspects of engineering design into educational design.

Finally, the ultimate test of the ideas in this paper will come from empirical examinations of educational design. It could be that good educational designers already do many of the things that good engineering designers do. But for us to find out, we will need to study educational designers. That literature simply does not currently exist. I hope that some of the readers of this paper will take up the gauntlet and do that empirical work. I also hope that educational designers will agree to participate in studies of their practice because their participation in the empirical efforts is critical building this critically necessary scholarship.

Acknowledgements

10

Work on this paper was supported by grant SBE-0738071 from the US National Science Foundation. I would like to acknowledge helpful comments by Hugh Burkhardt and Alan Schoenfeld on earlier drafts of the manuscript.

Appendices

11

Appendix A

In 2002–2005, I was part of a large project designed to improve K-12 science performance in large urban districts in the US. We had developed the general idea of immersion units during a grant proposal as part of the school reform work. Immersion units were thought to be project-like experiences for students in which they become immersed in the work of scientists or engineers. But that definition was rather vague and needed further specification to produce a successful outcome; we knew that a number of existing educational products existed that met this general definition but could be generally considered failures to produce widescale improvements in urban settings, either because they only worked in very rarified settings or because they became changed into ineffective forms as they went to scale. Rather than dive straight into design, we spent approximately 8 months refining the immersion unit concept to a set of much more specific requirements along a very wide range of dimensions. To develop the requirements, we worked with a wide range of developers, researchers, and users. The developers and researchers represented experienced individuals with very different views of what immersion units might look like, including individuals who thought the entire concept was unlikely to work. The users represented multiple layers in the educational system, from individuals in the school district central office (e.g., science supervisor) to science teachers. Some of the requirements we developed included:

• Must cover core, difficult concepts in the curriculum, in order to justify the time required. Operationally this involved improving student performance on high stakes, mostly multiple-choice state tests.
• Must increase student interest in later science courses and science/engineering careers, especially in traditionally unrepresented students.
• Must include at least one full science cycle (from hypothesis to experiment conclusions) or engineer cycle (from ideas to prototype), and students must legitimately engage with each step in that cycle.
• Must cost less than $10/student in materials
• Must work with teachers with little experience with inquiry or inquiry-based teaching
• Must work with teachers with weak science content knowledge
• Must be scalable to as many as thousands of teachers in a few years in a given large school district (e.g., 2000 4th grade teachers in Los Angeles Unified School District).

Note several features of this requirements list. First, it conceives of the immersion unit as part of the core curriculum, rather than bonus content material. Existing immersion units tended to cover advanced ‘bonus’ content rather than core content. Second, it conceives of immersion units as being for everyone (all teachers, all students), rather than the best teachers and “honors” students, which was the standard market for immersion-like units. Third, it specifies a range of activities with which students must engage, ruling out excessively cookbook approaches that are common in large-scale science materials. Fourth, it includes both cognitive and affective outcomes, and both in relatively specific terms (change in test scores, growth in interest in further coursework and later careers). Finally, it provides a very concrete cost target, which derived from an analysis of the costs of existing competitors and actual school district budgets for materials. With these requirements in hand, we then were able to develop units relatively quickly that were almost immediately successful and widely implemented.

Appendix B

In educational design (as in all design), it is tempting to go with the hammer in hand rather than develop a new product from scratch. At the same time, it can also be tempting to develop something from scratch purely for the novelty of it—designers are often very creative people with high need for novelty, and sometimes it is easier to sell something that has the appeal of superficial novelty. Building from experience and exploring new methods can both be legitimate design strategies, but it is better to generate some alternatives before focusing on or ruling out prior work.

At the beginning of the design process of the overall educational product, at the beginning of the design of a subsystem, or when we get stuck on a dysfunctional design element, a core design process that we engage with in my educational design teams is of explicitly developing two to four design alternative ideas before selecting a likely best choice to develop in detail. For example, in developing an immersion unit for high school physics classrooms focused on teaching Newton’s laws of motion, we spent a full quarter of our design time exploring substantially different possible foci of the unit:

1. A launcher-system unit we had developed as a larger group the year before that used PVC pipes and springs
2. An existing lift-systems unit we had developed several years before using Lego materials
3. A robotic-arm unit one of the teachers on the design team had built previously using PVC pipes and springs
4. A trebuchet unit (based on a mediaeval siege engine) which another teacher on the design team had built previously
5. A reverse egg-drop unit we brainstormed in one meeting.

Some of the ideas were problematic in terms of safety concerns. Some of the ideas were strong in controlling which science needed to be learned but were very uninteresting to students as a design task given the cost constraints. Some of the ideas resulted in the wrong science concepts being foregrounded — each engineering design task might involve some connection to many different science ideas, but often a smaller focal set of science concepts did the most work for the engineering design task. In the end, we selected the robotic arm unit because it met the overall requirements most effectively. The evaluation was not based on new, large scale evaluations of prototypes of each unit, but rather based on the judgment of the design team members, which were based on prior experiences with the different units, early on-paper writeups of the unit details, and some physical trial of units with other design team members. In other words, there was a rational basis to the selection process, but it was not so expensive/complicated as to make it undoable.

By exploring all of the ideas through systematic evaluation against our requirements, we were able to build consensus on a strong approach. We were also able to borrow some strong ideas from one unit into another. For example, there were some material choices and measurement techniques borrowed from the other units to develop a much improved robotic-arm unit.

Appendix C

The following table presents the subsystems in our Alarm Unit for teaching electronics concepts to 8th graders (Mehalik et al., 2008). We had to build an entire system that included the hands-on and paper materials, the activities and their sequence, the professional development, and a system for sustainability of materials and training. Rather than thinking about isolated parts, we thought about the functioning system and what solutions were required to be part of the functioning system. Then classroom observation and analysis of classroom video were used to examine whether each component was being successfully implemented for its specific intended function. For example, we saw in early implementations that team presentations to the whole class were not functioning properly as assessment and feedback, which was the intended function of team presentations. This problem was then addressed in later teacher workshops to change the way in which team presentations were facilitated and evaluated. This kind of focused debugging happened on every single component of each subsystem.

Subsystem	Component	Examples from Alarm Unit
Student	Knowledge introduction	Students generate hypotheses for observed phenomena in groups and then share these ideas in full-class discussion; teachers add/elicit missing key ideas if necessary.
	Assessment/feedback	Students get primary feedback through reflective testing of ideas in embedded science and design activities in the unit. Students also get feedback from other student teams and the teacher via regular presentations to the whole class of team progress.
	Practice	Warm-up exercises at the beginning of class refresh ideas from prior classes. Students apply ideas learned through systematic testing to improve their initial design ideas.
Teacher in the class	Student monitoring	Teachers must walk from team to team during their hands on work to monitor student progress. Teachers must elicit ideas from multiple groups in whole class discussions.
	Student guidance	Teachers should provide small suggestions or hints during teamwork activities, but not directly provide the answer. Via whole class discussion, teachers must make sure that target understandings of each curricular unit emerge as the final word at the end of that curricular unit.
Teacher out of the class	Teacher learning	4-hour teacher workshops every two weeks were used to provide just-in-time exposure to unit materials and pedagogical strategies. Teachers were encouraged to share adaptations and reflections on the successes/failures of these adaptations in each workshop.
	Lesson planning	Teacher materials included focal goals of each lesson along with space to be filled-out by the teacher prior to the lesson with detailed plans for the lesson.
Teacher support	Materials refurbishment	Detailed plans were provided to the school district central office regarding what materials were likely to require annual refurbishment along with cost-effective supplier information. In the Alarm unit, the primary refurbishment concern (other than new printed teacher and team worksheets) was new 1.5v batteries.
	Teacher learning	Teachers who emerged as strong implementers and good communicators with other teachers at the teacher workshop were identified to the school district. These teachers were then provided with professional development workshop materials (e.g., powerpoint slides) such that they could guide the professional development workshops (with central office support) with later cohorts of teachers.

Appendix D

One case that has benefited greatly from explicit tradeoff calculations was a project that involved the creation of online training materials for green/sustainable behaviors in a newly built green hospital building. The hospital system had developed a very long list of features and behaviors that they wanted to train a broad range of hospital staff and patients on to maximize the effectiveness of their sustainability initiative as well as generate positive press for the hospital corporation. Providing training on the whole long list was deemed a poor overall solution: 1) it was very unlikely that any individual would sit through training on that long a list or be able to remember that long a list of actions given how much training time was likely to be allocated; 2) some of the actions were questionable as to whether they actually helped the environment; and 3) some of the actions did not involve most people who would receive the training. However, there was a complex set of constraints on why items were on the list (e.g., high visibility corporate-level initiatives, consistency with prior training initiatives, common expectations of staff members, etc.). So we constructed decision matrices to winnow down the long list to a more manageable and important list. The particular dimensions and the rating system we used are given below:

Level of impact	Corporate priority	0=low or none 1=medium 2=high
	Percent of operating budget	1=0% - 30% 2=31% - 60% 3=61% - 100%
	Potential for cost savings	0=neutral 1=long payback 2=short payback
	Users decision-making control	0=no choice 1=choices affect building performance
Users	Does this action affect clinical staff?	0=no 1=yes
Transferability	Is this action / behavior easily transferred outside of building?	0=no 1=yes
Education	Information availability	0=nothing to teach 1=general "green" concepts 2=existing instruction on corporate policies and procedures
	Training already provided?	0=yes 1=no

We then collected information on each of the major initiatives on each dimension. To calculate a total score for each initiative, we considered 5 different weighting schemes, representing the relative importance of different stakeholders: equal weighting, double-weighting of training provided, reversing the relative value of short and long payback, and removing transferability. We then selected out the items that were consistently in the top three highest total scores across each weighting scheme; there was pretty high consistency across weighting schemes, but explicitly computing the different weighting scheme assured each stakeholder that their perspective was valued.

Appendix E

Consider the example of developing interim assessments in our Alarm Systems unit. A typical feature of science curriculum units is regular end-of-unit paper assessments, usually with multiple choice and short answer items. Rather than simply copy this practice into our unit design, we stepped back and asked the follow sequence of questions: 1) what function(s) do end-of-unit assessments serve? 2) What other possible solutions exist in other instructional materials that serve these same functions? 3) What other possible solutions could we imagine to meet these same functions? The answers to these three questions that we developed were: 1) guidance to teacher on later instruction and self-regulation information for students; 2) portfolio assessment and peer assessment; 3) classroom presentations. In the end, we decided that classroom presentations served as a more immediate and actionable guide to teachers and students than paper, portfolio, or peer assessments (see Figure E1 for an example of student presentation content). We also were able to use this functional role framework in our professional development workshops to convince teachers that it was not necessary to have traditional paper assessments as part of this unit.

References

12

© ISDDE 2008 - all rights reserved