{"coupon":"","discount":"","doc_name":"Psychology manuscript on peer ratings - disucssion","doc_id":68588,"editorId":"454","email":"violette_c@yahoo.com","message":"<h1><strong>Introduction<\/strong><\/h1>\r\n<h2><strong>The Present Study<\/strong><\/h2>\r\n<p>In this paper, we describe a modified peer rating system and test the system in five small Advanced Research Methods (ARM) classes. Similar to the existing Peer Rating System, students completed reading assignments, wrote reflections online, engaged in class discussions and rated their peers&rsquo; reflections. Similar to Zhang et al. (2008), the group&rsquo;s overall reliability was analyzed to gauge the degree to which peer ratings varied from group mean. Unique to our modified peer rating system, the data were also analyzed to offer tailored feedback to each rater regarding the degree to which their own judgment departed from group consensus. The purpose of providing individualized feedback was to evaluate and improve rating skills. We hypothesized that performance feedback to student raters mid-semester would induce students to become better raters in the second half of the semester.<\/p>\r\n<p>The first study included the instructor as a rater, and thus addressed both validity (the agreement between instructor and student raters) and reliability (the agreement among student raters). In light of the large body of research on the validity of peer rating systems (see Falchikov &amp; Goldfinch, 2000), the rest of our studies focused on reliability. In the method and results sections, we provide step-by-step instructions for how to implement the system in a class as well as how to statistically conduct the relevant tests. We also provide tables for how to communicate to a class about the data obtained so that students can interpret the validity and reliability indicators.<\/p>\r\n<p>&nbsp;<\/p>\r\n<h1><strong>Discussion<\/strong><\/h1>\r\n<p>The peer rating system used here advances the quantitative literacy goals outlined in social sciences, business and related disciplines. From this perspective, peer rating exercises are educational pursuits that are inherently meaningful in their own right. The benefit of peer rating skills is not confined to meeting a particular learning outcome in a particular discipline but has far reaching consequences beyond graduation (Boud &amp; Falchikov, 2006). For students in occupation oriented majors, they can appreciate the importance of acquiring coding skills at school before they can handle certain data types on a job (U.S. Department of Education, 2012). The case for career preparation is harder to make for academic oriented majors who have limited job prospects in their disciplines. However, a general trend is promising in the era of big Data. A large quantity of data is collected in all areas and these data will have to be processed in order to inform treatment or service choices. An emphasis on curriculum or vocation may be a topic of debate, but the difference is small in the present context. Both herald a paradigm shift in how peer rating is perceive in classrooms; they embody a student centric approach (that champions learning), instead of the instructor centric approach (that champions the time-saving feature).<\/p>\r\n<p>With a learning goal in place, we used one of the most challenging data types in our classes. Open-ended comments and reflections may be intimidating to judge at first, but all five classes competently evaluated their peers&rsquo; work at the end of the semester even though their learning curves were somewhat different. To document the mastery of coding skills, we first analyzed each class separately (to allow for shared characteristics on a class level, such as the instructor or the students&rsquo; understanding of the grading rubric) and then conducted an omnibus test across the five classes. Our data, aggregated this way, showed that students reached a greater consensus in judging their peers&rsquo; work in the second half of the semester than in the first half.<\/p>\r\n<p>The overall improvement in rating skills in our study is in concert with Zhu (1995) who found significantly higher quantity and quality of comments from trained peers than those from untrained peers. The intervention in Zhu&rsquo;s (1995) experimental research design was to offer teacher-student conferences to selected course sections whereas the intervention in our longitudinal study was a designated class period mid-semester. In our study, we also instituted the individual accountability feature in order to further incentivize skill acquisition not only during the designated class period but also through office hours and other informal channels. Given that learning skills are nascent on the learning agenda, future studies are needed to uncover the details of how students learn and what they learn. Students may be gravitated towards different teaching modalities, from teacher-student conference to experience sharing. Students may gain different component skills, from a better grasp of the rating rubric or a sensitivity to the cues in the data (Cathey, 2007).<\/p>\r\n<h2><strong>Individual Feedback and Assessment<\/strong><\/h2>\r\n<p>Individual accountability is essential if rating skills were to be regarded as a learning outcome. Accountability is the most effective if its derivation and meaning are straightforward to everyone in the system. We chose Cronbach&rsquo;s alpha as our main index because it is an assessable analytical method in psychology. The method of &ldquo;Cronbach's alpha if item deleted&rdquo; is particularly familiar to (aspiring) psychologists who use measurement scales. Cronbach&rsquo;s alpha was not only indicative of individual accountability but also a marker for group responsibility. The well accepted .70 rule of thumb is a convenient cutoff to judge class success. One of our future ambitions is to involve students in the actual numerical analysis so they can practice their statistical training in class. Other disciplines and course formats may require a different reliability index but the same principle applies. For example, Intraclass correlation coefficient (ICC) may be considered for large online classes where a flexible match between raters and ratees is desired (Luo, Robinson, &amp; Park, 2014).<\/p>\r\n<p>Cronbach&rsquo;s alpha also has strategic advantages for the present study which involves multiple small classes. Our small class sizes (designed to enroll 10-18 students) were most at risk for falling short of the .70 cutoff. However, with the exception of Time 1 of Study 1, the rest of our studies reached a satisfactory level of inter-rater reliability. Having succeeded in small samples, our modified system ought to be applicable to large samples. Rather than large class size, the modified peer rating system should be tested with other class characteristics, such as non-traditional students or online courses, will be more informative test cases in the future.<\/p>\r\n<h2><strong>Capstone as an Ideal Setting<\/strong><\/h2>\r\n<p>Capstones and advanced research classes may be particularly suitable platforms to turn peer rating system into educational opportunities. Our own journey started with a pedagogical need in this type of classes where so many aspects of the course rely on students keeping up with reading assignments. The negative repercussions stemming from students&rsquo; failure to read before class are well documented (Burchfield &amp; Sappington, 2000; Lineweaver, 2010). Time spent reviewing what should have been read in class is a wasted opportunity for instructors to promote deeper engagement with the course materials. Some of the past solutions proposed to enforce reading assignments are effective but have unintended side effects. Threats of pop-quizzes are often perceived as punitive by the students (Combs, 1976; Crouse, Deffenbacher, &amp; Frost, 1985; Thorne, 2000). Reading and grading weekly assigned writing assignments (Angelo, 1995; Connor-Greene, 2005; Dunn &amp; Smith, 2008; McDermott, 2010; Wade, 1995) solely for the purpose of having students keep up with readings, seems like an untenable solution especially for instructors who do not have graduate teaching assistants as a teaching support. Online learning management systems provide a variety of platforms for students to share their comments (Lineweaver, 2010) without offering any quality assurance features.<\/p>\r\n<p>A recurring question to an instructor is whether the accountability for reading course material can be enforced by peers. This pedagogical question was at the forefront when we looked to the existing peer rating system for an answer. We initially planned to simply present past evidence on the validity of peer rating systems and ask students to blindly trust their peer&rsquo;s judgments in our class, but given the reality of the various efforts students actually invest in rating exercises, we reconsidered this approach. We decided to analyze both Cronbach&rsquo;s alpha on a group level and &ldquo;Cronbach's alpha if item deleted&rdquo; for individual students to incentivize their engagement. However, once student engagement was brought to par, student actual rating skills fell short of our expectations given their advanced education level. Our focus then shifted from the product (of fair grades) to the process (of rating skills). We believe that both are accomplished in the end although future research is needed to show the interconnection between the ability to render ratings and the confidence in receiving ratings.<\/p>\r\n<p>&nbsp;<\/p>","price_select":"2","temp_doc_name":"upload_5792b4ae49c705.29461362.docx","word_count":"1373","subtotal":27.46,"total_amt":27.46,"firstName":"Violet","lastName":"Cheung","lastfour":"7158","exp":"0418","address":"5220 Anza Street","city":"San Francisco","zip":"94121","state":"CA","country":"United States","transactionReference":"20068362787"}