Peering into large lectures: examining peer and expert mark agreement using peerScholar, an online peer assessment tool

As class sizes increase, methods of assessments shift from costly traditional approaches (e.g. expert-graded writing assignments) to more economic and logistically feasible methods (e.g. multiple-choice testing, computer-automated scoring, or peer assessment). While each method of assessment has its merits, it is peer assessment in particular, especially when made available online through a Web-based interface (e.g. our peerScholar system), that has the potential to allow a reintegration of open-ended writing assignments in any size class – and in a manner that is pedagogically superior to traditional approaches. Many benefits are associated with peer assessment, but it was the concerns that prompted two experimental studies (n = 120 in each) using peerScholar to examine mark agreement between and within groups of expert (graduate teaching assistants) and peer (undergraduate students) markers. Overall, using peerScholar accomplished the goal of returning writing into a large class, while producing grades similar in level and rank order as those provided by expert graders, especially when a grade accountability feature was used. p. 1

The best way to ask students to think critically and to communicate clearly is via open-ended writing assignments (Ennis 1993; White 1993), p. 2

That traditional implementation is what we will term the ‘expert marker’ approach, and it involves students first composing a written assignment, and then submitting it to an expert for grading and feedback. p. 2

problem both cost and time between handing-in and receiving feedback, which makes it less meaningful (Heywood 1989) p. 2

However, thinking and communication skills, like virtually every other cognitive ability, develop with repeated practice (Schneider & Shiffrin 1977; Shiffrin & Schneider 1977) and develop best with distributed rather than massed practice (Melton 1970; Glenberg 1979; see Seabrook et al. 2005 for a demonstration in an educational context). p. 2

there is another less obvious pedagogical shortcoming of the expert marker approach. Namely, students do not see any assignments other than their own and therefore do not experience both poorly and well-written work, preventing an occasion to understand what makes a composition relatively strong (or weak), thereby diminishing an opportunity for students to improve the quality of their work (Sadler 1989). p. 2

Automated marking systems can produce grades that are similar to human markers (Williamson et al. 1999; Landauer et al. 2003), can be used economically in any class size, and can provide prompt numeric feedback (see Miller 2003 for an overview). p. 2

Peer assessment, sometimes called peer evaluation or peer review, is a process wherein peers evaluate each other’s work, usually along with, or in place of, an expert marker (e.g. Topping 1998; Woolhouse 1999; Rada & Hu 2002). p. 2

it is the pedagogical benefits inherent in having students grade the work of their peers that are especially exciting (Sims 1989; Liu et al. 2002). Specifically, seeing and critiquing peers’ work is said to encourage deeper analyses of the student’s own work (Sims 1989), a process that is required for the improvement of work quality (Sadler 1989). The use of peer assessment has also been shown to be a useful tool in exposing students to the real world of scientific discourse (Towns et al. 2001; Venables & Summit 2003), a world where peer assessment is a normal and accepted process. Peer assessment can also be used to teach students how to provide both quantitative and qualitative feedback properly (Bloxham & West 2004), p. 3

The peerScholar system was originally developed to address the need for writing and critical thinking assessments in a specific class setting; the Introductory Psychology course at the University of Toronto Scarborough that has enrolments of over 2000 students per year. p. 3

Successful attempts have been made with students marking each other’s work using paper and pencil methods (Sims 1989; Liu et al. 2002) and via electronic systems such as online forums (Towns et al. 2001; Trautmann et al. 2003) in smallto medium-sized class settings, but these approaches become problematic as class size increases. p. 3

The marking phase (see Fig 2) is where peer assessment occurs. Once this phase opened, peerScholar randomly assigned five abstracts and five critical thinking pieces to each student who participated in the first phase. p. 4

As each student logged on to peerScholar, they were presented with their assigned 10 anonymous peer-written answers and were required to rate each of them on a scale from 1 to 10. A general rubric was provided for students explaining what the grading criteria were. A number of other references were available from the first phase to serve as a reminder on what was to be included in each written piece. These included the original reading material, the student’s own answer, and the abstract and critical thinking guidelines. Students were also asked to provide feedback in the form of a positive and a constructive comment to support the numeric grade given to their peers’ assignments. p. 4

Specifically then, we required our students to read two peerreviewed articles based on opposing views of the same issue, for example, two articles discussing opposite sides of the topic ‘animal use in research’ p. 4

After reading the articles, students proceeded to write two essay-type pieces based on those readings through the peerScholar interface. As a way for students to show their understanding of the readings through summary and to become acquainted with scientific writing practices, they were asked to write an American Psychological Association (APA) formatted abstract for one of the articles. In the second writing piece, students were encouraged to think about the issue presented in the assigned articles and decide where their own opinions lie. They were then asked to write a critical thinking piece that supported their perspective on the issue based on arguments from each article, personal experience, and information acquired from external sources. p. 4

The results and feedback phase (see Fig 3) is where students accessed the final grade and comments they received on their assignment. Final grades were calculated automatically by peerScholar and were based on averaging the middle three values of the five peerassigned marks for each written piece, and then averaging the two written piece marks. p. 4

A total of 1143 students from the University of Toronto Scarborough completed the writing assignment as part of their required course work in the Introductory Psychology class (PSYA01). This assignment was worth 5% of participants’ final grade in the course. p. 8

Experiment 1 directly assessed the agreement between peer-given marks obtained within peerScholar and those provided by experts. p. 8

One aspect of this work involved an assessment of the agreement level of these expert markers with each other. p. 8

Of the expected 2286 written pieces submitted in the reading and writing phase, 120 were randomly selected to be graded by both expert markers and undergraduate peers. p. 8

Four psychology graduate students from the University of Toronto participated in the experiment as expert markers p. 8

students were grading on a scale ranging from 1 to 10 and were asked to try and maintain an average mark of p. 8

approximately 7 across each set of written pieces they were grading. However, students were told that if presented with 5 poorly written pieces, they would be justified in an average mark below 7 and similarly if they were presented with 5 above average pieces they should have an average above 7. These instructions were included as an attempt to keep peer-given grades in line with the class average without forcing students to grade assignments on the extreme ends of the quality scale with an undeserved mark to meet the average. Second, students were told to take the marking phase seriously given the success of the system, and their own grade, relied on the students’ support. To encourage proper grading, students were told that their marks would automatically be monitored by the system for inconsistencies such as patterns (e.g. giving out all 7s) and deviations (e.g. giving a 3 when all other markers gave 9s). p. 9

With an index of mark agreement levels within groups of expert markers, mark agreement within the three sets of peer markers should also be examined. Here, an intraclass correlation coefficient (ICC) was calculated because each assignment was graded by a set of three random peer markers. The ICC for the averaged marks from the sets of three peer markers was 0.41, P < 0.001. p. 10

An independent-samples t-test was conducted using the expert and peer mark means and a significant difference was found, t(226) = 2.00, P = 0.047, with the peers’ mean mark being slightly higher than that of the experts’ mean mark. p. 10

Given we now have a standard agreement level to contrast, we can establish mark agreement between expert markers and peer markers to determine if the two agreement levels are comparable. An analysis was done using Pearson’s correlation coefficient. A significant positive correlation was found between averaged expert marks and averaged peer marks, r(113) = 0.27, P < 0.003. It should be pointed out that although this correlation is not high, the fact that it is statistically reliable implies that expert markers and peer markers have a tendency to agree on the quality of written pieces being marked. p. 10

our next step was to compute the significance of the difference between the two correlations to determine if the agreement levels are significantly different from each other. To do this, we used Fisher’s z-score transformation of Pearson’s r for both pairs of expert markers’ correlations and the expert/peer correlations and found that the z value of the difference was 1.20 and 1.35, respectively, both smaller than 1.96 and thus not significant at the 0.05 level. p. 10

mark agreement within both pairs of expert markers was inspected using Pearson’s correlation coefficient. The first pair of experts’ marks was found to have a significant positive correlation, r(59) = 0.46, P < 0.001. Similarly, the second pair of experts’ marks was found to have a significant positive correlation, r(59) = 0.44, P < 0.001. These results indicate that there is significant agreement between expert markers on marks given to each writing assignment. Although these correlations are only moderate, they are typical of agreement levels when grading open-ended written assignments (Blok 1985; Miller 1996). p. 10

a replication of the first experiment was done with an additional accountability feature. This feature was labelled ‘Mark the Marker’, and allowed an outlet for students to show praise or concern for a mark they were given by an anonymous marker. It was also expected to add an element of perceived accountability by peers for the marks they gave. p. 11

Averaged expert marks and averaged peer marks were analysed and found to have a significant positive correlation, r(115) = 0.45, P < 0.001. p. 12

results from the questionnaires found that students strongly recognized the need for writing assessments in the course. It was also found that students generally liked the concept of peerScholar, they were pleased with the look and feel of the system, and they would like to use it in other university courses. However, there were concerns with peer grading and its fairness. It was partially because of these concerns that the described research was born. Having the ability to tell students the mark they received through peerScholar is comparable to the one that they would have been given by an expert marker lends immense credibility to the process, while simultaneously easing student concerns. p. 13

All instructions and information related to the peerScholar system – including emails, videos, and website material – has been digitally archived at the Web address: http://www.peerScholar.com/articles/grading p. 14

References p. 14