A steering committee that included a patient partner, an education specialist, an evaluation expert, and 2 scientists met quarterly to oversee the direction and operationalization of this study.
Designing the PiR course
The course design was informed by various learning theories and best practices in education including transformative and social learning theories [11,12,13], critical pedagogy and adult learning theory [14, 15], as well as online education models such as the Community of Inquiry Model and the Teaching Excellence Competencies Model [16,17,18,19,20]. The Canadian Institutes of Health Research SPOR Patient Engagement Framework was also used to guide the course design [5]. This framework outlines core principles of POR such as inclusiveness, support, mutual respect, and co-building. These guiding principles were used to meaningfully engage patient and research partners in the co-design, delivery and evaluation of the PiR course.
We created a working group composed of two patient partners, two research partners, an adult education specialist and a research assistant. Patient partners who sat on the steering committee or the working group were recruited from within our existing network of patient partners, established via previous research activities led by our research team.
Over a four-month period, the working group convened to operationalize the course design, delivery, and evaluation. The working group met monthly using an online meeting platform (Cisco Webex) and communicated routinely via email and telephone. Patient and researcher partners provided direction on the optimal course content, structure and needed learning supports. The course content focused on defining meaningful POR and distinguishing it from tokenistic engagement; common barriers and facilitators to POR, and the skills and attitudes that support POR in practice. The group iteratively revised each course offering based on course participant feedback. An additional file of the course syllabus outlines a complete overview of the PiR course components (see Additional file 1).
Delivering the PiR course
PiR was a 2-month, online course that included 4 bi-weekly real-time webinars, self-assessment quizzes, problem-based learning scenario (PBLS) assignments, and online group coaching. Each PiR module topic was co-presented in real-time by a patient and a researcher both of whom had expertise in POR. This effectively provided participants with the opportunity to engage in dialogue with POR experts and fellow learners. Each module also included discussion boards housed on the online learning platform, Canvas, that were co-facilitated by multiple researcher and patient coaches with POR experience. Course presenters (patients and researchers with expertise in POR who delivered real-time online presentations) and coaches (patients and researchers with expertise in POR who facilitated asynchronous online discussions) were recruited through various channels including the KT Program network, patient collaborator networks, and the PiR course participants who had successfully completed the course. Additionally, presenters and coaches of each course iteration tailored the pre-designed webinar slides and group discussion forums to include their own POR lived experiences and insights.
Coaches were provided with a comprehensive handout on coaching objectives and best practices and were connected to a course facilitator who was available to support the development of discussion posts and responses. Presenters were also supported by a course facilitator to prepare and deliver their presentations. A facilitator met with each presenter to discuss the overall topic to be presented, to share pre-developed slides and key talking points, and to encourage presenters to tailor the presentation to highlight their own experiences. Additionally, the course facilitator scheduled and facilitated meetings to connect the researcher and patient co-presenters to foster social connection and meaningful collaboration.
In the 2-week intervals between each webinar, participants were expected to complete small group PBLS assignments and participate in online group discussion boards. The PBLS assignments provided opportunities for small groups of patients and researchers to collectively reflect on common POR challenges and identify potential solutions. Similarly, the online discussion boards provided opportunities for participants to engage in discussions about barriers and facilitators to engaging in POR.
Participant recruitment
PiR was offered to four cohorts of patients, family members, and caregivers (hereafter referred to as patients for brevity) and healthcare researchers between May 2017 and December 2018. We invited a maximum of 40 participants (20 patients, 20 researchers) to participate in each cohort [21]. This number of participants was large enough to allow for a multiplicity of ideas and experiences to contribute to the co-production of knowledge about POR while small enough to facilitate meaningful participant engagement.
Course recruitment materials were disseminated via the KT Program’s, research organizations’ and patient partners’ networks (e.g., the Patient Advisors Network [22]). Participants were eligible to participate if they were either a healthcare researcher or a patient or a patient’s family member/caregiver. Both researcher and patient participants were not required to have POR experience, but were required to submit an application to attend the course that outlined their learning goals and interests in POR. The course was open to participants from across Canada, but recruitment was prioritized in Ontario. PiR course participants were invited to participate in the evaluation study via email invitation during the PiR course onboarding process. Those who agreed to participate in the course evaluation were emailed an online study information sheet and consent to participate.
Study design
We used a longitudinal, uncontrolled before and after study design to conduct a process and outcome evaluation of the PiR course. Patient partners contributed to the evaluation design and were consulted in the preparation of the study protocol. The course evaluation was informed by Kirkpatrick’s four-level training evaluation model [23].
Study outcomes
The primary outcomes of this study were to determine the perceived impact of the PiR course on participants’ knowledge, self-efficacy, intentions to use and use of POR. These outcomes were evaluated using online surveys, which were administered at three time points: baseline (i.e., before the start of the course), immediately post-course, and six months after course completion. Surveys were designed to measure participants’ self-identified knowledge of POR concepts (17 questions), self-efficacy to engage in POR (17 questions), intentions to engage in POR (8 questions) and use of POR (3 questions). Participants rated their level of agreement with each statement on a 7-point likert scale. Open-ended questions were included to gather additional information on participant-level outcomes. The COM-B Behaviour Change Theory [24] and the Theoretical Domains Framework (TDF) [25] were used to inform survey questions related to knowledge and self-efficacy of POR engagement. COM-B is a widely used model of behavior change identifying factors that need to be present for behavior change to occur [24]. The TDF is a comprehensive framework used to identify determinants of behavior [25]. Additionally, Légaré’s 12 item intention assessment scale was used to design survey questions about participants’ intentions to engage in POR [25]. Légaré’s scale is a theory-based instrument that shows adequate validity and reliability [26].
The secondary outcome of implementation quality assessed reach of the PiR course, dose of the PiR course components, and participant responsiveness. These indicators were informed by the Durlak and DuPre framework, which indicates that the quality of implementation affects the intended outcome and therefore relevant indicators should be used to assess intervention implementation quality [27]. Course reach was measured by the number of patients and researchers who participated in each cohort. Participant responsiveness (or level of engagement with the course) was reported descriptively as number of responses to each discussion board post, and participant satisfaction scores. Participants were asked to rate their satisfaction with course format (6 questions), course materials (7 questions), quality of delivery (9 questions), and their level of engagement (i.e., responsiveness, 3 questions) via a 7-point Likert scale in a post-course survey. Open-ended questions were also included to identify opportunities to improve the course and determine how the course compared to other POR training. Finally, via survey, participants were asked to describe barriers and facilitators to engaging in POR. These secondary outcomes align with Kirkpatrick’s training evaluation model [23]. An additional file outlines the survey items in further detail (see Additional file 2).
Data collection
Outcome data were collected via surveys as well as a review of course administrative documents and discussion boards. All surveys were delivered online using Qualtrics software [28]. To optimize survey response rate, the Total Design Method (TDM) was used, whereby reminders were delivered to non-responders at 1-week, 3-weeks and 7-week intervals [29]. Accordingly, participants had a total of 7-weeks to respond to the survey at each timepoint. An additional file further outlines the data sources (see Additional file 3).
Data analysis
Survey data from all 4 cohorts were aggregated and analysis was stratified by participant type (patients vs. researchers). Analysis was conducted using Stata 16 [30]. Descriptive statistics were used to summarize survey scores for participant groups across all 3 survey time points. Given the data were not normally distributed, a non-parametric two-sample Wilcoxon rank-sum test was used to determine whether there was a statistically significant difference in primary outcomes between time points. In addition, a two-sample Wilcoxon rank-sum test was completed to determine if patient and researcher groups significantly differed in primary outcomes scores across time points.
Open-ended survey responses were exported to Microsoft Excel software for qualitative analysis. Survey responses were aggregated across cohorts and stratified by participant type. Each timepoint was analyzed separately and thematic differences between timepoints were recorded. Data were double coded independently by two researchers; coding discrepancies were resolved through consensus meetings. The complete coding framework is presented in an additional file, but in brief, participant responses were coded into the primary outcome constructs of knowledge, self-efficacy, intentions and use of POR and secondary outcome constructs of implementation quality and participant responsiveness (see Additional file 4). Participant-reported barriers and facilitators to POR were analyzed to identify emergent, common themes. Illustrative quotes were abstracted to provide rich descriptions of participant feedback.