Improving competency-based assessment in WIL using programmatic assessment

Case Studies

Institution:

Edith Cowan University

Discipline:

Health

Model/s of WIL activity:

Industry/community based placement

Level activity is delivered:

Individual unit/course of study

Improving competency-based assessment in WIL using programmatic assessment

Engaging WIL stakeholders to improve assessment practices.

Using participatory action research, we engaged stakeholders to redesign the competency-based assessment (CBA) for the WIL component of the Master of Nutrition and Dietetics. Stakeholders and academics collaborated in six workshops that evaluated current practices and redesigned WIL assessment in accordance with the principles of programmatic assessment (Jamieson et al. 2017).

In the new assessment, students collect evidence of authentic learning tasks (self-reflection, patient care, reports). With university guidance, assessments are aggregated into an e-portfolio (PebblePad©). A panel of university staff are responsible for evaluating the portfolio against National Competency Standards and use consensus building to determine WIL outcome i.e., pass or fail.

Length of time the WIL activity has been/was in operation

Since 2016.

How does the WIL activity demonstrate good practice and/or innovation?

Programmatic assessment is the prevailing approach to CBA in medical education (Heeneman et al. 2021). Programmatic assessment recognises the inherent limitations of a single tool when applied to the complex phenomenon of competence in the WIL setting. Programmatic assessment uses a suite of fit-for-purpose instruments to longitudinally collect data on student performance. Rather than the traditional formative-summative dichotomy, assessment is conceptualised on a spectrum of increasing stakes, proportional to student outcomes (low to high-stakes). Low-stakes assessments occur regularly and are optimised for meaningful student feedback. Low-stakes data are purposefully combined to provide an information-rich picture that informs high-stakes decisions (course progression or graduation).

Who benefits from the WIL activity and how?

Our research (Jamieson et al. 2021), and that of others, demonstrates all stakeholders benefit from programmatic assessment. Academics are provided student performance data to enable credible and defensible (high-stakes) assessment decisions. WIL supervisors are reoriented to a collaborative teaching role which focuses attention on educating students and feedback dialogues, leveraging their skillsets. This removes the conflict of interest between being a teacher and assessor, reducing their burden and stress. Students are empowered to lead their assessment which drives learning and prepares them for the workforce. Student ownership provides academics and WIL supervisors with valuable insight into student comprehension that can assist remediation, where required.

How does the activity embed successful evaluation processes?

We ran a focus group with the WIL stakeholders to evaluate the assessment redesign process and found that it enabled a shared vision and that stakeholders were empowered to implement, and had ownership over, the new assessment. After implementation, we undertook a mixed method evaluation to understand the impact upon WIL supervisors. This revealed that programmatic assessment improved the student-supervisor relationship leading to a productive learning environment. In accordance with programmatic assessment principles, we collect annual student and WIL supervisor feedback and integrate this into the system for future improvements in a continuous quality improvement process.

What are the broader/longer term impacts for stakeholders? 

    We found that WIL stakeholders who participated in the redesign of the assessment experienced a philosophical shift in their views towards assessment, which better aligned with best practice. Programmatic assessment helped WIL supervisors become supportive student mentors who were relaxed without the burden of high-stakes assessments. These changes cultivated a lower-stress productive learning environment in which students thrived. Students were better prepared for the workforce as they gained control of their own learning, developing their initiative and sense of responsibility. In the long-term, these factors will work towards improving WIL attitudes and culture, and the profession more broadly.

    How is the WIL activity integrated into curricula? Include how it incorporates the preparation, implementation, and reflection/debriefing phases of WIL.

    Programmatic assessment necessitates considered alignment between teaching and assessment. Assessment tools have been mapped to the National Competency Standards and the relevant university Learning Outcomes. During WIL, students, supervisors and academics participate in regular student-led ‘Reflective Progress Meetings’ where student performance and learning goals are collaboratively discussed. Students and supervisors are equal stakeholders that are (i) trained by academic staff prior to the WIL, (ii) provided exemplars and instructions, (iii) supported throughout, and (iv) included in evaluation. Following the WIL activity, students participate in a survey and focus group where their honest opinions and reflections are welcomed by academic staff.

    Describe how the case study is informed by relevant theoretical or empirical literature, research and/or scholarship. Include relevant references.

    The WIL assessment has been designed in accordance with the published principles of programmatic assessment (recently updated by Heeneman et al. 2021) and implementation is aligned with the constructivist-interpretivist worldview of the programmatic assessment movement (Pearce & Tavares, 2021). The focus is on assessment for learning, rather than assessment of learning. Assessments are maximised for student feedback and to provide the university with trustworthy and robust evidence of student performance upon which to make high-stakes decisions (van der Vleuten et al. 2012).

    What are the plans for the WIL activity in the future?

    Programmatic assessment is firmly established as the model for WIL assessment in the Master of Nutrition and Dietetics at Edith Cowan University. Given the recent emergence of programmatic assessment, particularly in allied health, we will continue to research and share our own experiences to expand collective understanding. Human judgement is an inescapable component of competency-based assessment, and we will continue working with our WIL supervisors to embrace this subjectivity and develop a shared understanding for the profession.

    References

    Heeneman, S., de Jong, L.H., Dawson, L. J., Wilkinson, T. J., Ryan, A., Tait, G. R., Rice, N., Torre, D., Freeman A., & van der Vleuten, C. P. M (2021). Ottawa 2020 consensus statement for programmatic assessment – 1. Agreement on the principles, Medical Teacher, DOI: 10.1080/0142159X.2021.1957088.

    Jamieson, J., Hay, M., Gibson, S., & Palermo, C. (2021). Implementing programmatic assessment transforms supervisor attitudes: an explanatory sequential mixed methods study. Medical Teacher, DOI. 10.1080/0142159X.2021.1893678

    Jamieson, J., Jenkins, G., Beatty, S., & Palermo, C. (2017). Designing programmes of assessment: a participatory approach. Medical Teacher. DOI: 10.1080/0142159X.2017.1355447

    Pearce, J & Tavares, W (2021). A philosophical history of programmatic assessment: tracing shifting configurations. Advances in Health Science Education, DOI: 10.1007/s10459-021-10050-1.

    van der Vleuten, C. P. M, Schuwirth, L. W. T., Driessen, E. E., Dijkstra, J., Tigelaar, D., Baartman, L. K. J., & Tartwijk, J. V. (2012). A model for programmatic assessment fit for purpose, vol. 34, pg. 205-214.

    Case Study Leader

    Associate Professor Denise Jackson (ECU)

    Janica Jamieson