Design experiments in educational research

ements of a learning ecology typically include the tasks or problems that stu- dents are asked to solve, the kinds of discourse that are encour- aged, the norms of participation that are established, the tools and related material means provided, and the practical means by which classroom teachers can orchestrate relations among these elements. p. 2

. Elements p. 2

Beyond just creating designs that are effective and that can sometimes be affected by “tinkering to perfection,” a design theory explains why designs work and sug- gests how they may be adapted to new circumstances. Therefore, like other methodologies, design experiments p. 2

ign experiments are crucibles for the generation and testing of theory. p. 2

are p. 2

Although the term design exper- iment is most closely associated with Brown (1992) and Collins (1992), pedagogical design has informed the development of theories of instruction for well over a century. p. 2

One-on-one (teacher-experimenter and student) design ex- periments in which a research team conducts a series of teaching sessions with a small number of students. The aim is to create a small-scale version of a learning ecology so that it can be studied in depth and detail (Cobb & Steffe, 1983; Steffe & Thompson, 2000).

• Classroom experiments in which a research team collaborates with a teacher (who might be a research team member) to as- sume responsibility for instruction (Cobb, 2000; Confrey & Lachance, 2000; Gravemeijer, 1994).
• Preservice teacher development experiments in which a re- search team helps organize and study the education of pro- spective teachers (Simon, 2000).
• In-service teacher development studies in which researchers collaborate with teachers to support the development of a professional community (Lehrer & Schauble, 2000; Stein, Silver, & Smith, 1998).
• School and school district restructuring experiments in which a research team collaborates with teachers, school ad- ministrators, and other stakeholders to support organiza- tional change (Confrey, Bell, & Carrejo, 2001). p. 2

Design experiments are conducted to develop theories, not merely to empirically tune “what works.” p. 2

” These theories are rel- atively humble in that they target domain-specific learning processes. For example, a number of research groups working in a domain such as geometry or statistics might collectively de- velop a design theory that is concerned with the students’ learn- ing of key disciplinary ideas in that domain. A theory of this type would specify successive patterns in students’ reasoning together with the substantiated means by which the emergence of those successive patterns can be supported. p. 2

We interpret processes of learning broadly to encompass what is typically thought of as knowledge, but also the evolution of learning-relevant social prac- tices and even constructs such as identity and interest. p. 3

The use of prior research to both specify a design and justify the differentiation of central and ancillary conditions is central to the methodology. p. 3

design experiments always have two faces: prospective and reflective. These two faces are familiar to all empirical scientists, but the forms they take in design experiments are somewhat specialized. On the prospec- tive side, designs are implemented with a hypothesized learning process and the means of supporting it in mind in order to ex- pose the details of that process to scrutiny. An equally important objective is to foster the emergence of other potential pathways for learning and development by capitalizing on contingencies that arise as the design unfolds. p. 3

design experiments are conjecture-driven tests, often at several levels of analysis. The initial design is a con- jecture about the means of supporting a particular form of learn- ing that is to be tested. During the conduct of the design study, however, more specialized conjectures are typically framed and tested. For example, during a classroom design experiment, an initial conjecture about a prospective interaction between charac- teristics of tasks as they are realized in the classroom and student responses may be tested. If this conjecture is refuted, alternative conjectures can be generated and tested. p. 3

iterative design. As conjectures are generated and perhaps refuted, new conjectures are developed and subjected to test. The result is an iterative de- sign process featuring cycles of invention and revision. p. 3

The intended outcome is an explanatory framework that specifies expectations that become the focus of investigation during the next cycle of inquiry. p. 3

Theories developed during the process of ex- periment are humble not merely in the sense that they are con- cerned with domain-specific learning processes, but also because they are accountable to the activity of design. The theory must do real work. p. 3

Rather than grand theories of learning that may be difficult to project into particular circumstances, design ex- periments tend to emphasize an intermediate theoretical scope (diSessa, 1991) that is located between a narrow account of a spe- cific system (e.g., a particular school district, a particular classroom) and a broad account that does not orient design to particular con- tingencies. F p. 4

To achieve the instructional agenda, the team identifies current student capabilities, current practices, and other resources on which it might be able to build. p. 4

How- ever, in less researched areas, the team typically needs to conduct pilot work to document these understandings and, thus, the con- sequences of students’ prior instructional histories. p. 4

In the course of this pilot work, the team might also develop new methods for assessing aspects of student reasoning that need to be documented, given the purposes of the experiment. p. 4

en the purposes of the experiment. When the conjectured starting points, elements of a trajectory, and prospective endpoints have been specified, the challenge is to formulate a design that embodies testable conjectures about both significant shifts in student reasoning and the specific means of supporting those shifts. p. 4

Most classroom design experiments are conceptualized as cases of the process of supporting groups of students’ learning in a par- ticular content domain. The theoretical intent, therefore, is to identify and account for successive patterns in student thinking by relating these patterns to the means by which their develop- ment was supported and organized. p. 4

In addition to clarifying the theoretical intent of the experi- ment, the research team must also specify the significant discipli- nary ideas and forms of reasoning that constitute the prospective goals or endpoints for student learning. p. 4

generation of data on both learning and the means by which that learning was generated and supported. p. 5

There are at least four important functions that require ongo- ing direct engagement in the research setting and the associated planning and interpretive activities. p. 5

First, a clear view of the anticipated learning pathways and the potential means of support must be maintained and communicated within the research team, even while responding to contingency. p. 5

Much of the cleverness of excellent design experiments resides in how the team handles is- sues of measurement. p. 5

It is therefore important that the team generates a comprehensive record of the ongoing design process. It is stan- dard procedure in most engineering disciplines to keep records to support the retrospective analysis of the experiment (Edelson, 2002). Accordingly, the research team may employ audio records of meetings and logs to document the evolving conjectures, to- gether with the observations that are viewed as either supporting or questioning a conjecture. p. 5

In addition to self-consciously building and documenting the design and its rationale, the team members, like all researchers, have a responsibility for communicating what they learn in ways that are open to public scrutiny. This implies a commitment to generate data that support the systematic analysis of the phe- nomenon under investigation. At a minimum, this entails th p. 5

In summary, design experiments are extended (iterative), in- terventionist (innovative and design-based), and theory-oriented enterprises whose “theories” do real work in practical educational contexts. p. 6

As part of this process, it is important to be explicit about the criteria and types of evidence used when making particular types of inferences so that other researchers can understand, monitor, and critique the analysis. p. 6