The reliability of peer assessment in a final-year information systems course
Abstract
The perceived lack of reliability is one aspect of peer assessment that makes educators reluctant to use it in their teaching. This study was done among third‒year information systems students who had to assess two assignments of their peers by using rubrics. The rubrics differed in the sense that the one was more detailed requiring less subjective judgement while the other one was less detailed and required more subjective judgement from assessors. To determine the reliability of the peer assessment process, all the assignments of the participants in the study were re‒assessed by the lecturer and the marks compared. While the reliability of the peer assessment of both projects was acceptable, the assessment of the assignment requiring less judgement was more reliable compared to the assignment requiring more subjective judgement. A survey done among the participants revealed that they were happy with their marks and with the effort put in by their peers in assessing them. A significant number of students believed their peers awarded them higher marks for both assignments; however, this was only found to be the case for the assignment requiring more subjective judgment. It was also found that while the ratings of one peer rater resulted in reliable marks when the detailed rubric was used, it was not the case for the rubric requiring more subjective judgement. This study provides motivation for educators to use peer assessment as fears about the reliability thereof is unwarranted provided that properly designed rubrics are used, and students are adequately guided through the process.
Downloads
References
Andrade, H.G. 2005. Teaching with rubrics. College Teaching 53(1):27–30.
Ballantine, J.A. and P. M Larres. 2007. “Final year accounting undergraduates’ attitudes to group assessment and the role of learning logs.” Accounting Education 16(2):163–183.
Barac, K., M. Kirstein and R. Kunz. 2021. “Using peer review to develop professional competencies: an Ubuntu perspective.” Accounting Education 30(6):551–577.
Bennett, C. 2016. “Assessment rubrics: thinking inside the boxes.” Learning and Teaching 9(1):50–72.
Berry, F.C., W. Huang and M. Exter. 2023. “Improving Accuracy of Self‒and‒Peer Assessment in Engineering Technology Capstone.” IEEE Transactions on Education 66(2):174–187.
Bloxham, S., B. den‒Outer, J. Hudson and M. Price. 2016. “Let’s stop the pretence of consistent marking: exploring the multiple limitations of assessment criteria.” Assessment and Evaluation in Higher Education 41(3):466–481.
Boud, D., R. Cohen and J. Sampson. 1999. “Peer Learning and Assessment.” Assessment & Evaluation in Higher Education 24(4):413–426.
Charamba, E. and N. Dlamini‒Nxumalo. 2022. “Same yardstick, different results: efficacy of rubrics in science education assessment.” EUREKA: Social and Humanities (4):82–90.
Cho, K., C. D. Schunn and Wilson, R.W. 2006. “Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives.” Journal of Educational Psychology 98(4):891–901.
Cockett, A. and C. Jackson. 2018. “The use of assessment rubrics to enhance feedback in higher education: An integrative literature review.” Nurse Education Today 69:8–13.
Dale‒Jones, G., P. Hancock and K. Willey. 2013. “Accounting Students in an Australian University Improve their Writing: But How Did it Happen?” Accounting Education 22(6):544–562.
Darvishi, A., H. Khosravi, S. Sadiq and Gašević, D. 2022. “Incorporating AI and learning analytics to build trustworthy peer assessment systems.” British Journal of Educational Technology (December 2021):844–875.
Fagot, R.F. 1991. “Reliability of Ratings for Multiple Judges: Intraclass Correlation and Metric Scales.” Applied Psychological Measurement 15(1):1–11.
Falchikov, N. and J. Goldfinch. 2000. “Student Peer Assessment in Higher Education: A Meta‒Analysis Comparing Peer and Teacher Marks.” Review of Educational Research 70(3):287–322.
Gielen, S., F. Dochy, P. Onghena, K. Struyven, and S. Smeets. 2011. “Goals of peer assessment and their associated quality concepts.” Studies in Higher Education 36(6):719–735.
Hassan, O., A. Fox and G. Hannah. 2014. “Self‒ and Peer‒Assessment: Evidence from the Accounting and Finance Discipline.” Accounting Education 23(3): 225–243.
Jones, I. and L. Alcock. 2014. “Peer assessment without assessment criteria.” Studies in Higher Education 39(10):1774–1787.
Karaman, P. 2024. “Effects of using rubrics in self‒assessment with instructor feedback on pre‒service teachers’ academic performance, self‒regulated learning and perceptions of self‒assessment.” European Journal of Psychology of Education (June, 29).
Kilgour, P., M Northcote, A. Williams, and A. Kilgour. 2020. “A plan for the co‒construction and collaborative use of rubrics for student learning.” Assessment and Evaluation in Higher Education 45(1):140–153.
Kollar, I. and F. Fischer. 2010. “Peer assessment as collaborative learning: A cognitive perspective.” Learning and Instruction 20(4):344–348.
Koo, T.K. and M.Y. Li. 2016. “A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research.” Journal of Chiropractic Medicine 15(2):155–163.
Langan, A.M., C.P. Wheater, E.M. Shaw, B.J. Haines, W.R. Cullen, J.C. Boyle, D. Penney, J.A. Oldekop et al. 2005. “Peer assessment of oral presentations: Effects of student gender, university affiliation and participation in the development of assessment criteria.” Assessment and Evaluation in Higher Education 30(1):21–34.
Li, H., Y. Xiong, X. Zang, M.L. Kornhaber, Y. Lyu, K.S. Chung and H.K. Suen. 2016. “Peer assessment in the digital age: a meta‒analysis comparing peer and teacher ratings.” Assessment and Evaluation in Higher Education 41(2):245–264.
Lin, G.Y. 2018. “Anonymous versus identified peer assessment via a Facebook‒based learning application: Effects on quality of peer feedback, perceived learning, perceived fairness, and attitude toward the system.” Computers and Education 116:81–92.
Liu, N.F. and D. Carless. 2006. “Peer feedback: The learning element of peer assessment.” Teaching in Higher Education 11(3):279–290.
Luaces, O., J. Díez and A. Bahamonde. 2018. “A peer assessment method to provide feedback, consistent grading and reduce students’ burden in massive teaching settings.” Computers and Education 126(July):283–295.
Lundstrom, K. and W. Baker. 2009. “To give is better than to receive: The benefits of peer review to the reviewer’s own writing.” Journal of Second Language Writing 18(1):30–43.
Matherly, M. and L. Burney. 2009. “Using peer‒reviewed writing in the accounting curriculum: A teaching note.” Issues in Accounting Education 24(3):393–413.
McDonald, B. and D. Boud. 2003. “The impact of self‒assessment on achievement: The effects of self‒assessment training on performance in external examinations.” Assessment in Education: Principles, Policy and Practice 10(2):209–220.
McGarr, O. and A.M. Clifford. 2013. ““Just enough to make you take it seriously”: Exploring students’ attitudes towards peer assessment.” Higher Education 65(6):677–693.
McIsaac, C.M.P. and J.F. Sepe. 1996. “Improving the writing of accounting students: A cooperative venture.” Journal of Accounting Education 14(4):515–533.
Nawas, A. 2020. “Grading anxiety with self and peer‒assessment: A mixed‒method study in an Indonesian EFL context.” Issues in Educational Research 30(1):224–244.
Nicol, D. and D. MacFarlane‒Dick. 2006. “Formative assessment and selfregulated learning: A model and seven principles of good feedback practice.” Studies in Higher Education 31(2):199–218.
Nicol, D., Thomson, A. and C. Breslin. 2014. “Rethinking feedback practices in higher education: a peer review perspective.” Assessment and Evaluation in Higher Education 39(1):102–122.
O’Donovan, B., M. Price and C. Rust. 2004. “Know what I mean? Enhancing student understanding of assessment standards and criteria.” Teaching in Higher Education 9(3):325–335.
Orsmond, P., S. Merry and K. Reiling. 1996. “The Importance of Marking Criteria in the Use of Peer Assessment.” Assessment and Evaluation in Higher Education 21(3):239–250.
Orsmond, P., S. Merry and K. Reiling 2000. “The use of student derived marking criteria in peer and self‒assessment.” Assessment and Evaluation in Higher Education 25(1):23–38.
Ortega‒Ruipérez, B. and J.M. Correa‒Gorospe. 2024. “Peer assessment to promote self‒regulated learning with technology in higher education: systematic review for improving course design.” Frontiers in Education 9(1):1‒11
Panadero, E., A. Jonsson., L. Pinedo and B. Fernández‒Castilla. 2023. “Effects of Rubrics on Academic Performance, Self‒Regulated Learning, and self‒Efficacy: a Meta‒analytic Review.” Educational Psychology Review 35(4).
Petkov, D., O. Petkova, M. D’Onofrio and A.T. Jarmoszko. 2008. “Using Projects Scoring Rubrics to Assess Student Learning in an Information Systems Program.” Journal of Information Systems Education 19(2):241–251.
Phillips, F. 2016. “The power of giving feedback: Outcomes from implementing an online peer assessment system.” Issues in Accounting Education 31(1):1–15.
Ramon‒Casas, M., N. Nuño, F. Pons and T. Cunillera, T. 2019. “The different impact of a structured peer‒assessment task in relation to university undergraduates’ initial writing skills.” Assessment and Evaluation in Higher Education 44(5):653–663.
Riley, T. J., and K.A. Simons. 2013. “Writing in the Accounting Curriculum: A Review of the Literature with Conclusions for Implementation and Future Research.” Issues in Accounting Education 28(4):823–871.
Saito, H. and T. Fujita. 2004. “Characteristics and user acceptance of peer rating in EFL writing classrooms.” Language Teaching Research 8(1):31–54.
Shrout, P.E. and J.L. Fleiss. 1979. “Intraclass Correlations: Uses in Assessing Rater Reliability.” Psychological Bulletin 86(2):420–428.
Son, M. and M. Ha. 2024. “Effects of Descriptive Peer Assessment on the Learning of Middle and High School Students in South Korea.” SAGE Open 14(2).
Sridharan, B., M.B. Muttakin and Mihret, D.G. 2018. “Students’ perceptions of peer assessment effectiveness: an explorative study.” Accounting Education 27(3):259–285.
Sridharan, B., J. Tai, and D. Boud. 2019. “Does the use of summative peer assessment in collaborative group work inhibit good judgement?” Higher education 77:853–870.
Sudhakar, A., J. Tyler and J. Wakefield. 2016. “Enhancing student experience and performance throuh peer‒assisted learning.” Issues in Accounting Education 31(3):321–336.
Sullivan, G.M. 2011. “A Primer on the Validity of Assessment Instruments.” Journal of Graduate Medical Education 3(2):119–120.
Sung, Y.T., K.E. Chang, T.H. Chang, and W.C. Yu. 2010. “How many heads are better than one? The reliability and validity of teenagers’ self‒ and peer assessments.” Journal of Adolescence 33(1):135–145.
Taylor, B., F. Kisby, and A. Reedy. 2023. “Rubrics in higher education: an exploration of undergraduate students’ understanding and perspectives.” Assessment and Evaluation in Higher Education 48(4)
To, J., E. Panadero and D. Carless. 2022. “A systematic review of the educational uses and effects of exemplars.” Assessment and Evaluation in Higher Education 47(8):1167–1182.
Topping, K. 1998. “Peer assessment between students in colleges and universities.” Review of Educational Research 68(3):249–276.
Van Zundert, M., D. Sluijsmans and J. Van Merriënboer. 2010. “Effective peer assessment processes: Research findings and future directions.” Learning and Instruction 20(4):270–279.
Westermann, K.D., J.C. Bedard and C.E. Earley. 2015. “Learning the “Craft” of Auditing: A Dynamic View of Auditors’ On‒the‒Job Learning.” Contemporary Accounting Research 32(3):864–896.
Willey, K. and A. Gardner. 2010. “Investigating the capacity of self and peer assessment activities to engage students and promote learning.” European Journal of Engineering Education 35(4):429–443.
Yan, D. 2024. “Rubric co‒creation to promote quality, interactivity and uptake of peer feedback.” Assessment and Evaluation in Higher Education 49(4)
Zhang, W., Y. Li and W. Zhang. 2024. “More pain, more gain? Extricating the effect of student involvement in rubric co‒construction.” Assessment and Evaluation in Higher Education 49(4)
Copyright (c) 2025 Stephanus Jacobus Kruger
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
This journal is an open access journal, and the authors and journal should be properly acknowledged, when works are cited.
Authors, copyright holders, may use the publishers version for teaching purposes, in books, theses, dissertations, conferences and conference papers.
A copy of the authors' publishers version may also be hosted on the following websites:
- Non-commercial personal homepage or blog.
- Institutional webpage.
- Authors Institutional Repository.
The following notice should accompany such a posting on the website: This is an electronic version of an article published in SAJHE, Volume XXX, number XXX, pages XXX “XXX", DOI. Authors should also supply a hyperlink to the original paper or indicate where the original paper (http://www.journals.ac.za/index.php/SAJHE) may be found.
Authors publishers version, affiliated with the Stellenbosch University will be automatically deposited in the University Institutional Repository SUNScholar.
Articles as a whole, may not be re-published with another journal.
The following license applies:
Attribution CC BY-NC-ND 4.0
