Students' evaluation of the quality of teaching using generalisability theory: A case of a selected university in Ghana
Abstract
Students’ evaluation of lecturers’ quality of teaching has been a common practice in universities in Ghana and beyond. Data gathered from students are used to make vital decisions about lecturers such as promotion, training and development needs, among others. In recent times, the accuracy of students’ ratings of teaching quality has been questioned by stakeholders due to several reasons. Previous studies have attempted to investigate this issue using classical test theory (CTT) which comes with its own flaws. Little attention has been paid to the applicability of Generalisability theory (GT) to students’ evaluation of teaching in Africa. This study aims to assess the reliability of students’ rating of teaching through the lens of GT. A three-facet partially nested random balanced design [(r x i x o): l] was adopted for this study. Student (rater), item, and occasion served as the facets and lecturer served as the object of measurement. Both G (generalisability) and D (decision) studies were conducted. The institution’s evaluation questionnaire was adapted, validated and used for data collection. The sources of measurement error were accounted for by raters and lecturer-by-occasion interaction. Generally, the dependability index for the students’ evaluation of teaching quality was low, signalling little trust for such data. It was recommended that a minimum of 25 students should be permitted to rate lecturers for each class using at least 20 evaluation items. Suggestions for further studies were made based on the findings.
References
Benton, S. L. and K. R. Ryalls. 2016. Challenging misconceptions about student ratings of instruction. IDEA Paper 58(1): 1‒22.
Brennan, R. L. 2001. Generalizability theory. New York, NY: Springer-Verlag.
Briggs, D. C. and M. Wilson. 2007. Generalizability in item response modelling. Journal of Educational Measurement 44(2): 131–155.
Casabianca, J. M., J. R. Lockwood and D. F. McCaffrey. 2015. Trends in classroom observation scores. Educational and Psychological Measurement 75(2): 311–337.
Feistauer, D. and T. Richter. 2016. How reliable are students’ evaluations of teaching quality? A variance components approach. Assessment & Evaluation in Higher Education 42(8): 1263‒1279.
Geiser, C. 2012. Data analysis with Mplus. New York, NY: Guilford Press.
Ginns, P., M. Prosser and S. Barrie. 2007. Students’ perceptions of teaching quality in higher education: The perspective of currently enrolled students. Studies in Higher Education 32(5): 603–615.
Goos, M. and A. Salomons. 2016. Measuring teaching quality in higher education: Assessing selection bias in course evaluations. Research in Higher Education 58(4): 341‒364.
Gravestock, P. and E. Gregor-Greenleaf. 2008. Student course evaluations: Research, models, and trends. Toronto, Canada: Higher Education Quality Council of Ontario.
Hativa, N. 2013. Student ratings of instruction: A practical approach to designing, operating, and reporting. Tel Aviv: Oron Publications.
Hill, H. C., C. Y. Charalambous and M. A. Kraft. 2012. When rater reliability is not enough: Teaching observation systems and a case for the generalizability study. Educational Researcher 41(2): 56–64.
Iyamu, E. O. S. and S. E. Aduwa-Oglebaen. 2005. Lecturers’ perception of student evaluation in Nigerian Universities. International Education Journal 6(5): 619‒625.
Li, G., G. Hou, X. Wang, D. Yang, H. Jian and W. Wang. 2018. A multivariate generalizability theory approach to college students’ evaluation of teaching. Frontiers in Psychology 9(1065): 1‒11.
Lord, F. M. and M. R. Novick. 1968. Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
Machingambi, S. and N. Wadesango. 2011. University lecturers’ perceptions of students’ evaluation of their instructional practices. Anthropologist 13(3): 167‒174
Manary, M. P., W. Boulding, R. Staelin and S. W. Glickman. 2013. The patient experience and health outcomes. The New England Journal of Medicine 368(3): 201‒203.
Marsh, H. W., B. Muthén, T. Asparouhov, O. Lüdtke, A. Robitzsch, Alexandre J. S. Morin and U. Trautwein. 2009. Exploratory structural equation modelling, integrating CFA and EFA: Application to students’ evaluations of university teaching. Structural Equation Modelling: A Multidisciplinary Journal 16(3): 439–476.
Oermann, M. H., J. L. Conklin, S. Rushton and M. A. Bush. 2018. Student evaluations of teaching (set): Guidelines for their use. Nursing Forum 53(3): 280‒285.
Ogbonnaya, U. I. 2019. The reliability of students’ evaluation of teaching at secondary school level. Problems of Education in the 21st Century 77(1): 97‒109.
Quansah, F., V. R. Ankoma-Sey and D. Asamoah. 2019. The gap between the academia and industry: Perspectives of university graduates in Ghana. International Journal of Education and Research 7(3): 63‒72.
Quansah, F., E. Appiah and V. R. Ankoma-Sey. 2019. University students’ preparation towards building knowledge economy in Africa: A case of universities in Ghana. International Journal of Social Sciences & Educational Studies 6(1): 38‒48.
Rantanen, P. 2013. The number of feedbacks needed for reliable evaluation: A multilevel analysis of the reliability, stability and generalisability of students’ evaluation of teaching. Assessment & Evaluation in Higher Education 38(2): 224–239.
Salem, M. I. 2014. The role of universities in building a knowledge-based economy in Saudi Arabia. International Business & Economics Research Journal 13(5): 1047‒1056.
Samian, Y. and N. M. Noor. 2012. Students’ perception on good lecturer based on lecturer performance assessment. Procedia-Social and Behavioural Sciences 56(1): 783‒790.
Shavelson, R. J. and N. M. Webb. 1991. Generalizability theory: A primer. Thousand Oaks, CA: Sage Publications.
Shin, Y. and S. W. Raudenbush. 2012. Confidence bounds and power for the reliability of observational measures on the quality of a social setting. Psychometrika 77(3): 543–560.
Spooren, P. 2010. On the credibility of the judge: A cross-classified multilevel analysis on students’ evaluation of teaching. Studies in Educational Evaluation 36(4): 121–131.
Spooren, P., B. Brockx and D. Mortelmans, 2013. On the validity of student evaluation of teaching: The state of the art. Review of Educational Research 83(4): 598–642.
Spooren, P., D. Mortelmans and W. Christiaens. 2014. Assessing the validity and reliability of a quick scan for student’s evaluation of teaching. Results from confirmatory factor analysis and G Theory. Studies in Educational Evaluation 43(1): 88–94.
Staufenbiel, T. 2000. Students course assessment questionnaire for evaluation of university courses. Diagnostica 46(4): 169–181.
Staufenbiel, T., T. Seppelfricke and J. Rickers. 2016. Predictors of student evaluations of teaching. Diagnostica 62(1): 44–59.
Wolfer T. A. and M. M. Johnson. 2003. Re-evaluating student evaluation of teaching. Journal of Social Work Education 39(1): 111–121.
VanLeeuwen, D. M., T. J. Dormody and B. S. Seevers. 1999. Assessing the reliability of student evaluations of teaching (SETS) with generalizability theory. Journal of Agricultural Education 40(4): 1‒9.
This journal is an open access journal, and the authors and journal should be properly acknowledged, when works are cited.
Authors, copyright holders, may use the publishers version for teaching purposes, in books, theses, dissertations, conferences and conference papers.
A copy of the authors' publishers version may also be hosted on the following websites:
- Non-commercial personal homepage or blog.
- Institutional webpage.
- Authors Institutional Repository.
The following notice should accompany such a posting on the website: This is an electronic version of an article published in SAJHE, Volume XXX, number XXX, pages XXX “XXX", DOI. Authors should also supply a hyperlink to the original paper or indicate where the original paper (http://www.journals.ac.za/index.php/SAJHE) may be found.
Authors publishers version, affiliated with the Stellenbosch University will be automatically deposited in the University Institutional Repository SUNScholar.
Articles as a whole, may not be re-published with another journal.
The following license applies:
Attribution CC BY-NC-ND 4.0
