Expert-Novice Studies

PROTOTYPICAL STUDIES

Chase and Simon (1973) recruited three participants of varying chess skill for their study (one master chess player, one Class A player, and one beginner). Each participant completed two different tasks involving side-by-side chessboards with a divider between them. The left board presented a different configuration each trial, and the right board was used by the participants to make responses. For the perception task, the participants were asked to recreate 14 game configurations, completing each as quickly and accurately as possible. For the memory task, the participants viewed an additional 14 game configurations for only five seconds and needed to recreate each configuration from memory. Each configuration was repeated until the participant correctly recreated the configuration. The game configurations varied by whether they were selected from the middle of a game or the end of the game and whether they were actual game configurations or random organizations of the pieces (i.e., unlikely to occur in a game).

Chase and Simon found that recall ability increased with chess skill ability. This relationship only occurred in recreations of actual configurations, not with random configurations. Chase and Simon hypothesized that the expert memory advantage was specific to actual configurations because experts saw chessboards in terms of familiar chunks, and random configurations did not have familiar chunks. Chase and Simon empirically estimated chunk size by using the time intervals between the placements of each chess piece as participants were recreating the configurations. Using this information to parse the chunks for each participant, they found that master chess players indeed had more chunks and larger chunks than the less skilled players. Even though the chunk size was larger, the chunks still fit within the accepted memory span–that is to say, chess masters did not have better memory ability, just bigger chunks.

Another classic expert-novice study involved the study of representations rather than chunks. Chi, Feltovich, and Glaser (1981) recruited 16 participants of varying physics knowledge (eight experts included advanced PhD students from the physics department and eight novices included undergraduates who had just completed a semester of mechanics). Each participant sorted 24 physics problems selected from eight chapters of a physics textbook that were copied onto note cards. They were instructed to make their categorizations based on the similarities of the solutions. After the sorting process, they explained why they chose to group certain problems.

While there were no differences in the number of categories the experts and novices used in the sorting task, there was very little overlap in the category labels used by the experts and novices. From both the categorizations and the explanations, it was clear that novices grouped by what the problems looked like (e.g., incline planes or pulley problems) and experts grouped by the major physics principle necessary to solve each problem (e.g., conservation of energy problems or Newton's second law).

ROLE OF EXPERT-NOVICE STUDIES IN EDUCATIONAL RESEARCH

Expert-novice studies have many important roles in educational research. First, they define the educational end point. The core of educational research is about defining how to structure teaching and learning rather than in defining what should be taught or learned, at least with respect to the end point of learning. From somewhere else, there must be input on the appropriate targets of learning. Expert-novice studies are a very important method of specifying those targets, and they are a method that has systematic advantages over simply asking disciplinary experts to list the target knowledge. The reason is that one hallmark of expertise is that much of the knowledge is tacit and situationally evoked (Patel, Arocha, & Kaufman, 1999)—experts don't know all the things they know and do, and it is hard to systematically extract the things they do know.

Second, expert-novice studies define the educational start point. Young children are remarkably adept at some things (e.g., extracting patterns from statistics in the environmental input) and young adults are remarkably weak at other things (e.g., reasoning about confounded variables in very simple situations). Research on where novices sit with respect to experts establishes empirically where the biggest gains need to be made by the learner.

This start-to-endpoint view of expert-novice studies connects with a knowledge decomposition view of curriculum design: what are all the individual knowledge and skill components that make up the goals of educational interventions (Gagne, 1962)? Here the assumption is that expertise is made up of many independent skills and knowledge, and instruction must cover each of these skills and knowledge in some logical order.

A variation of this decomposition perspective, derived from expertise research that emphasized the con-textualized nature of knowledge and skills, is that experts not only know facts and have separable skills but are also better able to selectively and appropriately apply facts and skills in particular contexts (Lemaire & Siegler, 1995). For example, the early expertise studies on chess suggested that experts had tens of thousands of chess chunks, but the research also showed that experts had associated moves with these chunks that allowed them to quickly “see” which moves were worth considering. The educational implication is that students must learn these connections with the correct contexts (and perhaps break connections with the wrong contexts).

A third important use of expert-novice studies for educational research is to provide models of some form of education that was clearly successful. Expertise research has generally ruled out the genetic or talent perspective on expertise, which makes especially salient the question of what environmental factors were in place to develop that expertise (Ericsson, Krampe, & Tesch-Roemer, 1993). That is, can we develop better models for instruction on the basis of these success stories? For example, some researchers have considered the ways in which experts and novices interact in many settings that lead to high levels of expertise (e.g., in a professional or sport setting), namely apprenticeship. The observation is that novices are asked to participate in authentic disciplinary activities from the beginning (e.g., playing in full basketball games), and move gradually from peripheral, supporting roles to central, independent roles (Collins, Brown, & Newman, 1989). Another important educational implication from this angle on expertise research is the central importance of focused practice: expertise is developed through thousands of hours of focused, regular practice, and curricula that devote much less time to an important topic are unlikely to be successful (Ericsson et al., 1993).

A fourth focus involves the importance of some expertise for learning in addition to performance. For example, research on learning by analogy has found that more expert students are better able to find relevant features in examples and learn by analogy to these examples than more novice students, who tend to encode problems in more superficial ways (Novick, 1988). This kind of research suggests which knowledge and skills should be placed earlier in the curriculum because they enable or accelerate later learning.

EXPERT-EXPERT EXTENSIONS OF EXPERT-NOVICE RESEARCH

Expertise is often treated as a single dimension, such that there is only one way of being more expert in a given domain. However, expertise is actually more nuanced than that single dimension view, and individuals can be more or less expert on different dimensions. For example, a person might be an expert on facts of a domain, but not expert in the skills of a domain, such as in the case of a U.S. Civil War buff with no training in historical thinking. Another person might have considerable training in the skills relevant to a particular domain but relatively little knowledge of the facts relevant to that domain, such as in the case of a physics researcher working on biology research. Studies that compare different kinds of experts allow for some teasing apart of the importance of different kinds of knowledge or skill on performance in different kinds of circumstances. For example, Voss, Tyler, and Yengo (1983) compared political scientists with expertise in the Soviet Union against political scientists with expertise on other regions, as well as comparing them to chemistry faculty and undergraduates. They found that some aspects of reasoning performance (on a problem relating to solving a political science problem involving the Soviet Union) depended upon training in political science in general whereas other aspects of reasoning performance depended specifically on being an expert on that exact region.

Most importantly for educational research, expertexpert studies can address the central educational question of generality and transfer of knowledge. A tension in curriculum design is the extent to which reasoning skills can be taught outside of a context or in some arbitrary context, or whether reasoning skills must be taught specifically in the context in which they will be later used. An example would be statistics or research methods. These are often stand-alone courses, with the idea that students will later be able to apply the skills learned in those courses to whatever later contexts they need to apply them in, even if those contexts involve reasoning about very different content. The alternative approach is to teach only more specialized reasoning courses that are specific to subgroups of students' goal reasoning domain, or maybe have them take many such courses if they are unsure about their target domain. The general reasoning course approach is more efficient, but depends upon there being successful transfer. Expert-expert studies help address this question methodologically: if an expert in one domain can successfully solve problems in another domain, then the general training approach is a viable one. In fact, some recent studies have found generality to many aspects of training (e.g., Schunn & Anderson, 1999).

STATISTICAL ANALYTICAL TECHNIQUES USED IN EXPERT-NOVICE RESEARCH

Although expertise is really a continuum rather than true categorical stages as the term “expert-novice” implies, expert-novice studies tend to focus on extreme group comparisons. Thus, the independent variables tend to be categorical. Dependent variables used in expert-novice research can be quite diverse, ranging from simple quantitative performance measures (such as accuracy or solution time) to more qualitative measures such as protocol analysis. In the case of simple performance metrics, basic univariate statistics such as ANOVA and t-tests are commonly used. In the case of protocol analysis, statistical techniques relevant to analysis of frequencies are used (e.g., Chi-Squared or other non-parametric tests). Because experts are often hard to find in large quantities, the minimum N assumptions of many statistical tests are not met, and low N variations of those tests are required (e.g., Fisher-exact instead of Chi-Squared).

STRENGTHS AND WEAKNESSES OF EXPERT-NOVICE RESEARCH

In abstract terms, expert-novice research generally involves high external validity at the cost of low internal validity. To be more specific, expert-novice contrasts tend to involve participants similar to ultimate situations of interest working on tasks of ultimate interest, in contrast to using participants of convenience working on highly simplified tasks with questionable levels of motivation to perform the given experimental tasks. However, the downside is that the statistical power of the studies is often low because experts are hard to find and the statistical method is by necessity the lower powered, between-subjects method. There are also many confounded variables bundled with expertise because one cannot randomly assign participants to expertise. Thus, expert-novice research is an excellent methodological tool that should work in conjunction with other techniques that have better statistical power and methodological control but perhaps lower external validity.

Another important methodological feature of expert-novice research involves confirmatory versus exploratory research. Because expertise is a natural variable rather than an experimentally controlled one, expertise research tends to be more exploratory. Exploratory research has the advantage of being able to provide many unexpected findings, but it often involves looking at many possible dependent variables to find which of the many complex ways in which expertise could express itself is relevant to the given domain/task being studied. This “fishing expedition” nature of data analysis in expert-novice studies is why qualitative data is so often collected—it allows the researcher to explore many different dimensions of behavior after the data is collected. It also explains why expert-novice studies are slow to be analyzed and published.