The four partner TEOs were engaged to undertake pilot projects that aimed to:

• determine the suitability and utility of the Framework across different sectors; and
• provide recommendations and insights into ways of improving and implementing the Framework.

Impacts on learner outcomes and equity will only be measurable once the processes and changes required become more mature and entrenched within the partner TEOs.

Waikato Institute of Technology (Wintec)

Project commenced 30 June 2019 and finished in December 2019.

Total funding - $430,000 plus an additional $100,000 to work with the US organisation Achieving the Dream (ATD) to support TEO coaching activities.

Wintec was the first and longest-running of the partner projects. They tested the first four phases of the Framework – from diagnostic to strategy development. By using human-centred design methodologies, Wintec developed a deep and comprehensive student journey mapping process.

This approach has given Wintec key insights into their learners’ needs and formed the basis of a new operating model for the organisation. Wintec is in the process of developing culturally situated momentum strategies that leverage data, leadership, people and partnerships to drive continuous improvement.

Based on their learner success-focused approach, preliminary feedback from Wintec shows improvements in retention and course completions for Māori and Pacific learners during the COVID-19 level four lockdown in 2020. The gains have halted and reversed historical declines for these measures.

• Watch Warwick Pitts, Wintec Executive Director, Products and Planning, talk about their learner success project

Te Wānanga o Raukawa (TWoR)

Project commenced 26 July 2019 and finished in July 2020.

Total funding - $375,000

Changing learner demographics and demand necessitated TWoR to undertake a radical shift in the way they design and deliver their education offerings, by moving most of their content into a blended delivery model. We partnered with TWoR to support them in undertaking this business transformation project, which entails them using a TWoR-derived change management model which closely aligns with the Framework. This provided us with an unprecedented opportunity to test the framework in a mātauranga Māori context.

The high alignment between their model and the Framework meant that TWoR had already made considerable progress towards developing a holistic understanding of their learners and a learner-centred operating model. The Framework has assisted in aligning the learner success practices with their own guiding kaupapa. TWoR is driving organisational and learner change by leveraging technology and continuous reflection and improvement strategies.

TWoR is now partnering with local and global IT industry leaders to look at further opportunities to refine their own operating model and also deliver IT programmes of learning.

University of Waikato (UoW)

Project commenced 30 June 2019 and finished in August 2020.

Total funding - $297,500

UoW’s project focus was to create a robust system of learner insights by building their data analytical capability. This approach enabled them to identify teaching and learning gaps, trends, and patterns to build responsive information technology systems that allow pro-active approaches to strengthen student success. While working through the Framework stages such as undertaking a learner journey mapping project, it was realised that a broader whole of organisation change was required.

This meant the approach evolving from “just an IT project” to more of a holistic change project. While the deep qualitative evidence required for a comprehensive learner needs analysis to determine profile characteristics was lacking, extensive work had been done to develop an artificial intelligence-driven data and insights platform to support future student-centred business planning.

Manukau Institute of Technology (MIT)

Project commenced 31 January 2020 and finished in May 2021.

Total funding - $331,200

We partnered with MIT to test the first four phases of the Framework, from diagnostic to strategy development. The project builds on MIT’s successful Hōkūle’a Project, which increased the retention rates of Māori and Pacific learners in levels 1-3 provision, through an early assessment of needs and provision of holistic support (academic, pastoral and cultural). MIT researched specific approaches and interventions used at Georgia State University, such as applying the process of ‘meta-majors’ to reduce confusion in learner subject choice, and developing technology options to support early alerts.

Note: Meta-majors group individual majors under a larger academic umbrella. These programmes provide learners with a clear pathway and help them make connections between their studies and different career tracks. Learners also have set schedules depending on their meta-major. A learner interested in health care, for example, may enrol in a meta-major that includes prescribed health-related courses in science, communications and statistics, opening up career opportunities ranging from nursing to pharmacy.

The intermittent Auckland-region lockdowns resulting from COVID-19, as well as internal changes, have caused significant disruptions and delays in MIT’s ability to effectively progress their learner success initiative in 2020. Despite these factors, they have made progress in setting up quantitative and qualitative data collection processes to inform the learner journey mapping and learner needs analyses.

During the COVID-19 level four lockdown period, MIT developed an online version of their learner success approach. The recent transferring of the learner success project to the Deputy Chief Executive Academic to drive the learner success approach is a further strong indication of organisational commitment.

From September 2019 to early 2021, a developmental evaluation was undertaken (in two phases) alongside the four Learner Success Framework pilots in different contexts. This evaluation was led by the TEC strategy and evaluation team, with support from the Ōritetanga team and TEO partner organisations.

The aim of this evaluation was to provide the TEC with:

• an understanding of how the Framework and tools could be implemented, based on its unique iterative development and use,
• the key findings of the piloting of the Framework in different educational contexts; and
• the key learnings at TEO and system level to consider in scaling Framework approaches.

The current pilot projects were aimed at determining the suitability/utility of the adapted framework to our local context. Impacts on learner outcomes and equity will only be fully measurable once the learner success processes and changes required become more mature and entrenched.

Overarching findings and themes

The suitability and utility of the Framework was endorsed

The preliminary overall view based on the evaluative findings is that the Framework has been successfully tested and implemented within the tertiary education context in Aotearoa New Zealand. This was clearly shown in the enthusiasm with which TEOs engaged with the process and worked through the stages (in whichever permutation they used). The engagements with TEOs were seen as beneficial and useful. These provided a structure for the TEOs to use and adapt to embed new processes to become more learner-focused. These concrete steps along the learner success pathway clearly showcase the perceived utility and applicability of the Framework in Aotearoa New Zealand.

TEOs took ownership of learner success approaches

The open and collaborative approach used throughout allowed all four pilot TEOs to implement the Framework in their own idiosyncratic ways. This allowed TEOs to:

• adapt the Framework to their own unique contexts; and
• internalise the approach within which they situate future work on understanding learner success and improving learner outcomes.

TEOs shifted their mindsets towards different ways of doing business

A significant development already ocurring was the initiation of organisation-wide cultural shifts to become more learner-centric and tailor organisational processes around learner needs, not business processes. Most TEOs required a mind shift to a more learner-centric way of operating, on top of developing data systems to help them holistically understand learners and needs. Data-driven insights cannot be used effectively if operational contexts do not allow them to inform strategies, policies or processes.

While the full effects of the initiatives will take some time to be seen, the pilot TEOs had already achieved meaningful progress. This includes learner journey mapping and creation of learner data systems and insights systems, as well as business cases for reviewing and changing organisational processes.

Large-scale work required for large-scale change

A common issue encountered was an underestimation of the scale of work required. The learner success approach is a complex and lengthy process to work through. All pilot TEOs underestimated the intensity of the process, the resources required and the time needed.

This was mostly due to underestimating the holistic whole-of-organisation nature of the process. The scope of the organisational change required is often not clear at the outset of the project and only becomes clear once a TEO has completed the diagnostic and problem definition stages of the Framework. Extra guidance or upfront information on the resourcing requirements for effectively engaging in the learner success process would be beneficial.

Flexibility is required or needs to be better emphasised in a future iteration of the Framework

One pilot TEO identified more dimensions than the four used in the diagnostic tool of the Framework. While the extra dimensions did not conflict with those in the Framework, this may reflect differing emphases or organisational processes unique to the TEO. This underscores the need to emphasise flexibility in the Framework when applied to different TEOs or contexts.

Learner success approaches made meaningful progress in challenging times

The extraordinary events of 2020 caused significant disruptions and obstacles to the pilot projects. Regardless of these challenges, all TEOs engaged with and continued building their learner success initiatives. Meaningful progress is already evident in creating and implementing the cultural and organisational shifts required for effective systems that will help identify, engage and support not only high-needs learners, but all learners.