Objectives
Our evaluation aims to understand the success of the ExCHANGE Collaboration in achieving the overarching project aims detailed above. We will adopt a theory-driven realist approach using qualitative methods [21] to examine in what circumstances and how the model of collaboration works, and for whom. The realist approach enables ongoing adjusting of the underlying theories and their associated mechanisms.
The evaluation will aim to address the following research questions:
-
a.
How does the ExCHANGE Collaboration develop capacity, under which circumstances, and for which care home and research staff?
-
b.
How does the ExCHANGE Collaboration mobilise knowledge, under which circumstances, and for which care home providers?
-
c.
How does the ExCHANGE Collaboration identify research ideas and implementation problems for future investigation, under which circumstances, and for which care home providers?
-
d.
What are the implications for theorising how academic-practice collaborations work, under which circumstances, and for whom?
Our evaluation of the study does not seek to be summative and so will not focus on assessing if it ‘works’ as a collaboration programme; rather we seek to generate insights and learning that can be used locally and more widely to others attempting similar collaborative research initiatives.
Theoretical framework
A realist evaluation approach assumes that programs are “theories incarnate”: that is, whenever a program is implemented, it is testing a theory about what ‘might cause change’ [21]. This evaluation will test and refine an initial programme theory of how the ExCHANGE Collaboration ‘might cause change’. The research team will bring existing theories from the literature (Closer Collaboration, Co-production, Knowledge Brokering) to provide an initial idea (programme theory) about how the collaboration may achieve its aims. This will then be shared and discussed in a team meeting (which will include all key stakeholders) to capitalise on researcher, residents’ family members, and care home staff knowledge and expertise to adapt these theories to this specific ExCHANGE Collaboration. The resulting ‘initial programme theory’ will guide data collection and analysis to understand how contexts may influence which mechanisms work, or not, to produce outcomes.
Study setting
The evaluation will be carried out over the course of the 24-month project by the research team delivering the wider ExCHANGE Collaboration project. The project will be co-ordinated within the University of Exeter College of Medicine and Health, which supports applied health research and implementation projects. Data collection will take place remotely, via phone or video call, or in person at the University of Exeter or one of the participating care homes in Devon.
Patient & Public Involvement
Care home residents’ family members and residents themselves (where appropriate) will be supported to engage and contribute to the wider ExCHANGE Collaboration in addition to the evaluation. Two representatives will be invited to join the ExCHANGE Collaboration Management and Delivery Groups and be involved in the decisions that affect the design and delivery of the collaborative model, and the subsequent analysis and dissemination of its findings; through these meetings, the evaluation of the overall project will also be managed. Evaluation tools will be shared with the Delivery Group prior to use and reviewed to ensure they are appropriate for the audience.
The project team will follow the UK National Standards for Public Involvement (NIHR, 2018) [19], including payment and reimbursement of expenses for patient advisors. In line with these, information and support will be provided to the family member(s)/residents(s) to enable their participation, such as holding pre-meetings before Delivery Group meetings to go over the agenda and explain any technical aspects which will aid involvement. Standard two of the document, about ‘working together’, underpins our principle of valuing different kinds of knowledge that people bring to the ExCHANGE Collaboration meetings and workshops. All communication will be in Plain English, and PPI representatives will both be encouraged to contact the researchers directly if anything is unclear, and approached by the project researcher to ensure they are fully able to participate in meetings and workshops. PPI representatives are involved in the governance of the project and its evaluation, by sitting on the project delivery group.
Data collection
The evaluation will employ mixed qualitative methods, including observations of events/activities, interviews and analysis of documents such as feedback forms, a Knowledge Broker reflective journal/diary kept by the Researcher (a semi-structured document for Research field notes made following activities, also including reflections on how activities were received etc.), event attendance records, risks and issues logs and other relevant paperwork gathered as part of project delivery. Care home providers, staff, residents, family members and University researchers who are involved in the project will be invited to take part in a short structured interview – this includes those involved in project design and delivery as well as those attending events or taking part in other project activities (see Fig. 1). All data collection tools will be designed by the Knowledge Broker and reviewed by the project Delivery group.
Documents generated
A range of documents useful for the evaluation will be gathered and stored electronically (see Fig. 1). Project management documents will be created and updated at regular intervals by the study team. Documents seeking anonymous feedback about how well a project activity/event was received by participants will be developed in consultation with the Delivery Group and handed out after each event. Records of project ideas generated during the project and connections made will also be recorded and regularly updated by the project team. Further, the Knowledge Broker will keep a journal for the course of the project, noting reflections as well as personal experience and learning regarding the role of the knowledge broker and the barriers and facilitators of working in this way.
Observations
Observations will be concerned with activities such as learning events and Project Generation Meetings. In addition to researcher observations, project PPI representatives will also be invited to take notes to add to the evaluation data collected and to support the interpretation of events. We have not pre-specified events to be observed because we want to ensure representation of those delivered across the course of the ExCHANGE Collaboration and to allow for the evolving formative model. We will use field notes to record (on paper) the roles of those who were in attendance, the purpose of the activity/event, the focus of discussions, any feedback received verbally or in writing from those involved, and the reflections of the observer on the success of the activity in achieving its goal(s). Prompts focused on refining the programme theory and mechanisms will also be added as the evaluation progresses.
We will seek consent using an opt-in approach whereby attendees will be provided with a consent form at the start outlining what participation involves. For anyone who does not opt in, we will refrain from gathering any information from them or about their participation in the event/activity being observed. Attendees will be able to approach the researcher at any time up to the end of the activity to opt in or out. The researcher will wear a badge so that they can be identified.
Interviews
Individual, semi-structured telephone interviews (or face-to face where possible and ethical) lasting up to one hour will be conducted by the project Researcher, with:
-
Between 5 and 10 care home staff who have been involved in one or more activities/events held as part of the ExCHANGE Collaboration.
-
Between 2 and 5 care home residents or family members who have been involved in one or more activities/events held as part of the ExCHANGE Collaboration or who have been involved in the design and/or delivery of the project (e.g. through involvement with the Delivery Group).
-
Between 3 and 5 staff involved in the design and delivery of the project.
-
Between 2 and 5 research staff who have been involved in one or more of the project’s activities/events as recipients.
Participants will be identified following their involvement in project activities. Individuals will be invited to take part directly by the Researcher via their preferred communication method (as outlined in their registration for the event/activity they took part in). Where individuals are unwilling or unable to take part, alternatives will be approached until the target number has been achieved. Residents and family members will be recruited independently. We will ensure that the study materials and the interview questions are accessible as recommended by the Enabling Research in Care Homes (ENRICH) network [6]. Where it is not clear that the resident would be able to provide consent, they will be excluded from the research.
Potential participants will be given an information sheet about the evaluation and what the interview will involve, and will be given at least 24 h to decide if they would like to take part. If they agree, informed consent will be obtained. This will make clear that taking part is voluntary and they can choose not to take part without any disadvantage or giving a reason. Where residents or other individuals may require assistance to participate, this will be managed on a case-by-case basis, but participation will be supported wherever possible. If individuals appear distressed during the interview, they will be asked if they wish to stop or take a break. The researcher will let the relevant care home staff know if a resident was tired or distressed.
Interview schedules will be informed by the developing programme theory to learn more about mechanisms, contexts and outcomes of particular interest, and to explore gaps in the emerging theory where more information is needed. This will include: individuals’ experience of the project activities; views on the value of the activity, how well it achieves its aims, how well it was organised and delivered, and any improvements that can be made; views about the ExCHANGE Collaboration model itself – how well it was designed, managed, adapted and delivered over the course of the project – as well as any learning identified, including reflections on the role of the Knowledge Broker and applicability of other programme theories; and the overall barriers and facilitators noted to deliver the model. Prompts or questions focused on refining the programme theory and mechanisms will be added as the evaluation progresses.
For the interviews, each participant will be assigned a participant identification number for all files and transcripts. This number will be recorded on the consent form which will be the only point that links to the participant’s personal details. The interviews will be digitally recorded using an encrypted recording device or by handwritten notes and as soon as possible after uploaded or scanned, following which the recording/notes will be destroyed. Transcripts will be done by a professional transcription service for which a Non-disclosure Confidentiality Agreement and formal contract will be drawn up by the University of Exeter Legal Services Department to ensure that appropriate standards for data handling and transfer are met. Transcripts will be reviewed against the recording for accuracy.
Data anonymization, storage and analysis
Data will be gathered, recorded, coded and analysed by the study researcher (KW), with support from the project manager (JD) and other members of the Delivery Group, where necessary. All notes and collected data will be pseudonymised and later anonymised, and stored electronically on a University computer that is encrypted and password-protected. Paper notes will be destroyed once typed up.
In analysing the data we will use Framework Analysis, a matrix-based method for ordering and synthesising data [25] that facilitates rigorous and transparent data management and analysis. We will develop a thematic coding framework informed by the initial programme theory and use this to classify and organise the data. Our analysis will be strategic and focus on areas that seem important in relation to the developing theory of how the ExCHANGE Collaboration might achieve change. To increase rigour, coding and interpretations will be discussed between the researcher, project manager and chief investigator (IL), as well as key stakeholders involved in the project. Findings will also be sense-checked with the family members who are part of the project Delivery group, and where possible, we will seek feedback from residents via these family members, or via staff working in the care homes owned by our provider partners. The final step in our analysis will be to consider what our findings from this evaluation add to the existing theories of Closer Collaboration, Co-production and Knowledge Brokering.
We will anonymise quotes or data used in dissemination outputs (i.e. reports, journal publications, presentations, oral feedback/presentations and lay summaries) to ensure that participants and participating organisations cannot be identified, and we will ask respondents before using any quotes from them in these outputs.