Lessons learned in measuring patient engagement in a Canada-wide childhood disability network

Abstract

Background

The CHILD-BRIGHT Network, a pan-Canadian childhood disability research Network, is dedicated to patient-oriented research, where numerous stakeholders, including patient-partners, researchers, and clinicians are involved at different levels. The Network is committed to continuously improving the level of engagement and partnerships’ impact. Measuring patient engagement is therefore important in reflecting on our practices and enhancing our approaches. We aimed to measure patient engagement longitudinally and explore in greater depth the perceived benefits, barriers and facilitators, and overall satisfaction with patient engagement, from the perspectives of the different stakeholders.

Methods

Patient engagement was measured using online surveys. In a longitudinal study design over a 3-years period (2018–2020) the Community-Based Participatory Research (CBPR) questionnaire was used. To enrich our understanding of patient engagement in Year 3, we employed the Public and Patient Engagement Evaluation Tool (PPEET) in a cross-sectional, convergent parallel mixed-method study design. Descriptive statistics and a thematic-based approach were used for data analysis.

Results

The CBPR questionnaire was completed by n = 167 (61.4% response rate), n = 92 (30.2% response rate), and n = 62 (14.2% response rate) Network members in Years 1, 2, and 3, respectively. Ninety-five (n = 95, 21.8% response rate) members completed the PPEET in Year 3. CBPR findings demonstrate a stable and high satisfaction level with patient engagement over time, where 94%, 86%, and 94% of stakeholders indicated that the project is a “true partnership” in Years 1, 2, and 3, respectively. In Years 2 and 3, we noted an improvement in patient-partners’ comfort level in sharing their views and perspectives (92% and 91% vs. 74%). An increase in critical reflective trust (i.e., allowing for discussing and resolving mistakes) from Year 1 to 3 was found, both from the perspectives of patient-partners (51–65%) and researchers (48–75%). Using the PPEET, patient engagement factors (i.e., communications and supports for participation, ability to share views and perspectives) and impact were highly rated by most (80–100%) respondents. PPEET’s qualitative responses revealed several patient engagement advantages (e.g., increased projects’ relevance, enhanced knowledge translation), barriers (e.g., group homogeneity), facilitators (e.g., optimal communication strategies), and solutions to further improve patient engagement (e.g., provide clarity on goals).

Conclusion

Our 3-years patient engagement evaluation journey demonstrated a consistent and high level of satisfaction with patient engagement within the Network and identified advantages, barriers, facilitators, and potential solutions. Improvements were observed in members’ comfort in sharing their views and perspectives, along with an increase in critical reflective trust. These findings underscore the Network's commitment to enhancing patient engagement and provide valuable insights for continued improvement and optimization of collaborative efforts.

Plain English summary

The CHILD-BRIGHT Network, a Canadian childhood disability research Network, is dedicated to patient-oriented research. It engages more than 300 diverse stakeholders, including patient-partners, researchers, and healthcare professionals. We conducted a 3-years study aimed to measure patient engagement over time and delve into the perceived benefits, barriers, and facilitators from the perspectives of the different members. We administered the Community-Based Participatory Research (CBPR) questionnaire in Years 1–3 (completed by 167, 92, and 62 members, respectively) and the Public and Patient Engagement Evaluation Tool (PPEET) in Year 3 (completed by 95 members). Through the CBPR, we identified in which research processes were Network members involved (e.g., defining the research question, results dissemination), appraised the partnership between researchers and other stakeholders such as patient-partners, and determined the type of trust in this partnership. The use of the PPEET allowed us to explore patient engagement impact and what factors facilitate and limit patient engagement (e.g., communication and supports). CBPR results showed a consistently high satisfaction level with patient engagement, with increased comfort among patient-partners in expressing their views over time, showcasing positive collaborative dynamics. Most stakeholders reported a “true partnership” in their engagement, indicating widespread belief in equitable relationships. Additionally, critical reflective trust, allowing for discussing and resolving mistakes in collaborative working activities, increased over the years, with the highest endorsement in Year 3, demonstrating growing trust among stakeholders. The PPEET findings showed positive ratings for communication, support, and impact of patient engagement. Its qualitative responses identified advantages (e.g., increased project relevance), barriers (e.g., lack of diversity in members’ demographic characteristics), facilitators (e.g., effective communication), and suggested improvements (e.g., ensuring goal clarity). In conclusion, our project showed that the partnership between researchers and patient-partners was beneficial, satisfactory and evolved positively over time. The findings are encouraging provided the breadth of the Network, where hundreds of members are primarily connected virtually. We learned that:

• (1) It is possible to measure patient engagement in a large Network, both at one point in time and over time, and multiple tools can be used together to get a better picture.

• (2) Regular evaluations are important to optimize the partnership and its impact.

• (3) The partnership can be improved and strengthened with time through ongoing collaboration, open communication, and a commitment to address the evolving needs and dynamics of all stakeholders involved.

Background

The CHILD-BRIGHT Network (www.child-bright.ca) is a pan-Canadian patient-oriented research Network that works to create brighter futures for children and youth with brain-based developmental disabilities and their families. Created in 2016, the Network is funded by the Canadian Institutes of Health Research (CIHR) under Canada’s Strategy for Patient-Oriented Research (SPOR) and numerous funding partners across the country. It engages over three hundred members, including researchers and trainees, clinicians, decision-makers, industry partners, youth with disabilities, and caregivers of children/youth with disabilities.

The CHILD-BRIGHT Network is dedicated to patient-oriented research. As Canada’s SPOR endorses the active partnership of patients, researchers, health professionals and decision-makers in research to build a sustainable and accessible health care system that optimizes the health of Canadian citizens, related initiatives require authentic involvement of patients and the public in all phases of investigations [1]. The engagement of patient-partners is recommended to improve the relevance and the overall quality of research, through ensuring that the research team targets issues that are meaningful and important to patients [2, 3]. These invaluable team members can be engaged in all phases of the research process, including but not limited to: developing and prioritizing patient-centred research questions and objectives; guiding the study design and procedures; selecting and adjusting outcome measures and intervention materials; developing parts of the intervention methodologies; advising on optimal recruitment strategies; and guiding as well as actively participating in knowledge mobilization activities [4, 5]. In relation to that, CHILD-BRIGHT’s extensive research program was developed based on research priorities identified by youth with disabilities, caregivers of children/youth with disabilities, and other knowledge users such as frontline clinicians and health care administrators. From 2016 to 2022, CHILD-BRIGHT carried out thirteen multi-centre and SPOR-based projects focusing on three childhood disability themes: (1) Early intervention to promote brain and child development; (2) Strategies to support the mental health of children and families; and (3) Service delivery redesign to address gaps in service.

At CHILD-BRIGHT, we recognize that patients and families are experts on their health experiences and needs; they provide unique expertise on their everyday experiences living with a condition and can share their experiences with the health care system. Various stakeholders including patient-partners (youth with disabilities, parents/caregivers of children/youth with disabilities), researchers and clinicians are involved at different levels of the Network and contribute as committee members and/or as research project team members. Committee members are generally engaged over long-term (i.e., at least 1 year in duration) and contribute to important decision-making within the administration of the Network. For instance, this includes the Citizen Engagement Council, where patient-partners (parents/caregivers, family members, and youth-partners) make up the core of this committee. Its purpose is to offer guidance about ways to initiate, improve and sustain meaningful engagement with the Network’s essential partners and constituencies. Research projects members may have variable engagement (i.e., long- and short-term) and collaborate closely with researchers to ensure their voices and experiences are heard and reflected in the research initiatives.

Patient-oriented research and patient engagement measurement are rapidly evolving fields [6]. Previous work on patient engagement in health care research revealed several perceived benefits and certain challenges of these partnerships from various team members (reviewed in [4]). For example, as partners’ perspectives, preferences, and diverse experiences are considered, it can enrich research pertinence and relevance. Moreover, patient-partners’ insights can enhance participant recruitment and retention, increase trust and engagement between the research team and the community, as well as facilitate and optimize knowledge mobilization. On the other hand, challenges related to power dynamics and communication, balancing partners’ perspectives with scientific rigor, as well as training and education can surface and require proactive management [4]. Overall, balancing these factors is essential to ensure the success and impact of patient-oriented research initiatives. In addition, it highlights the need to evaluate these efforts and their impact to ensure that team members are satisfied with the role and that it aligns with their expectations.

The CHILD-BRIGHT Network is committed to continuously enhancing the partnership with patient-partners throughout the entire research project cycle, spanning from protocol development to findings dissemination. Therefore, measuring the Network’s patient engagement and its impact is an essential ongoing activity. Additionally, findings might inform other teams and/or networks as they plan for or navigate their patient engagement journeys. Our objectives were to: (1) Measure patient engagement longitudinally as the research projects evolved through their cycle (from research protocol and intervention co-development to recruitment, data collection, analysis, manuscript preparation and dissemination); and (2) Explore the perceived benefits, barriers and facilitators and overall satisfaction with patient engagement from the perspectives of the different stakeholder groups, including researchers, committee members, patient-partners, trainees, and youth advisors, among others.

Methods

Measuring patient engagement—journey overview

Our patient engagement measurement journey spans over a 3-years period. In 2018 (Year 1), we launched our first evaluation using the Community-Based Participatory Research (CBPR) questionnaire and continued using the CBPR questionnaire in 2019 (Year 2) and 2020’s (Year 3) patient engagement evaluations. In 2020, we aimed to enrich our understanding of patient engagement and proceeded to assess it using the Public and Patient Engagement Evaluation Tool (PPEET). In addition, we conducted individual interviews of researchers and patient-partners [7] and completed a project-specific patient engagement evaluation using the PPEET for one of the CHILD-BRIGHT’s projects [8].

Engagement of patient-partners in the current project

Patient-partners were engaged in various ways in the current project. One co-author of the present manuscript (FG) is a patient-partner. His role included the conceptualization of the project, selecting measurement tools (including the CBPR questionnaire and the PPEET), providing feedback on the methodology and data collection, validating findings (quantitative and qualitative), guiding and providing feedback on the visualization of findings (specifically, how these were displayed as part of published reports, info briefs, and visual summaries [9,10,11,12]), and reviewing and providing valuable feedback on the current manuscript. In addition, in our project-specific patient engagement evaluation [8], patient-partners were involved in modifying the PPEET, and that modified version was used in the current project. More specifically, the following three statements were added by patient-partners in the section of Communication and supports for participation:

• “I am satisfied with the compensation I received for my role on the committee.”

• “The committee is generally prepared for meetings (e.g. agenda/questions for the upcoming meeting and minutes of previous meeting are provided ahead to time to review, timelines are outlined, etc.).”

• “I receive information with enough time to read and respond within the context of my work/life schedule.”.

Study design

For the CBPR questionnaire, we employed a longitudinal survey design with three data collection points in Year 1 (2018), Year 2 (2019), and Year 3 (2020). For the PPEET, we used a cross-sectional, convergent parallel mixed-method study design, including quantitative and qualitative approaches.

Study population

To ensure a comprehensive assessment, partners within all 13 CHILD-BRIGHT projects, including the parent-advisory group (PAG, caregivers of children/youth with disabilities), youth-advisors (youth/young adults with disabilities), researchers/principal investigators, trainees, committee members, clinicians/health professionals and education professionals (e.g., teachers), consultants, policy makers, and industry partners, from across Canada were invited to participate. This inclusive approach allowed for a thorough patient-engagement evaluation of the SPOR-based initiatives within the context of childhood disability themes. All Network members were eligible to participate, irrespective of their engagement duration. Given the Network's establishment in 2016, the initial patient-engagement evaluation in 2018 primarily involved members who were relatively novice.

Study procedures

Informed consent was obtained prior to completing the surveys. Approval was granted by the Ethics Committee of the Research Institute of the McGill University Health Center (study no. 2017-2850) and was performed in line with the principles of the Declaration of Helsinki. Participants’ contact information was not connected to their survey responses and their email or Internet protocol addresses were not recorded by the survey system. The survey system assigned a unique identifier to the survey link sent to each participant and recorded only the assigned identifier with each person’s responses. Identifiers were used to connect data but not to connect data to individuals. For all measures, we used an online survey created in the REDCap platform (https://www.project-redcap.org/). Surveys were active for a period of 6 weeks. Two weeks after the initial mail out, a follow-up email reminder was sent to those who had yet to complete the survey. A second email reminder was sent 4 weeks following the initial mail out. These were sent by the CHILD-BRIGHT administration team.

Respondents were asked to indicate the stakeholder group they identified most with (e.g., researcher/principal investigator, PAG, committee member). For the CBPR questionnaire, principal investigators (PIs)/researchers were directed to the Principal Investigator version of the CBPR questionnaire, and all other respondents were directed to the Stakeholder version. For the PPEET, the participants were directed to one of the four versions of the PPEET, depending on their main stakeholder group (PI, PAG, youth advisor, committee member).

Measurement of outcomes

CBPR

The online survey included the CBPR questionnaire and a section on sociodemographic questions (e.g., education, ethnicity, work status, income). The CBPR is an evaluation tool that examines the degree of stakeholder involvement and partnership in the research process as well as how researchers are engaging stakeholders in their projects [13]. The CBPR questionnaire has been used to assess stakeholder involvement and partnerships across different research areas (e.g., nutrition, substance use, cancer) and with diverse groups of stakeholders (e.g., white, Latino/a, Asian communities) [13]. There is strong factor analytic support for the constructs underlying the CBPR questionnaire [14, 15], and the questionnaire is freely available and used by the research team who developed the tool as well as others conducting CBPR projects.

We used two versions of the CBPR questionnaire: one for PIs (lead researchers involved in CHILD-BRIGHT research projects) and one for stakeholders who interact in different ways with research projects (CHILD-BRIGHT patient-partners or family members, staff or trainees, co-investigators, and committee members). The Stakeholder version of the CBPR asked stakeholders about their engagement experiences to assess partnership development (involvement in each phase of the project), partnership appraisal, and partnership trust. By delving into these aspects of partnership, our study aimed to provide a comprehensive understanding of the intricate dynamics between researchers and patient-partners.

• Partnership development was assessed across seven involvements, each representing a phase of the research process: (1) Defining the problem; (2) Deciding on which issue to research; (3) Developing research questions; (4) Creating research instruments or guidelines; (5) Collecting data; (6) Analyzing and/or interpreting the data; and (7) Disseminating findings. Stakeholders were asked to rate how involved they have been in each step of the research process using a 5-point Likert scale ranging from “Not at all involved” to “Very involved”. Respondents can also select “Does not apply” for any of the items. To facilitate interpretation, responses were recorded as “No involvement” (Not at all involved or Not very involved) or “Involved” (Somewhat involved, Involved, Very involved).

• For partnership appraisal, respondents were asked to rate their agreement across seven items: (1) When the different partners come together, I feel comfortable sharing my opinion; (2) This project is a true partnership; (3) So far, the work of the partnership has been good for the community; (4) So far, the partnership has been good for me personally; (5) I am satisfied with my level of involvement in this project; (6) It is important to me to have my name on presentations and articles; and (7) It is important to have my work acknowledged on presentations and articles. Respondents were asked to rate their level of agreement with each statement using a 5-point Likert scale ranging from “Strongly disagree” to “Strongly agree”. A “Does not apply” response category could be selected for each item. Responses were re-coded to “Disagree” (Strongly disagree, Somewhat disagree) and “Agree” (Somewhat agree, Strongly agree) to create more meaningful groupings for analysis and reporting. Questions about partnership appraisal aim to gauge the perceptions and evaluations of respondents regarding the quality and effectiveness of their collaborations. This information is crucial for identifying strengths and areas for improvement in the collaborative processes.

• To assess trust in the partnership, respondents were asked to rate the type of trust they have in the partnership using the question: “What type of trust do you think you have now in the partnership?”. Respondents could select from the following response categories: critical reflective (i.e., trust that allows for mistakes and where differences can be talked about and resolved), proxy (i.e., partners are trusted because someone who is trusted invited them), functional (i.e. partners are working together for a specific purpose and timeframe, but mistrust may still be present), neutral (i.e., partners are still getting to know each other; there is neither trust nor mistrust), unearned (i.e., trust is based on member’s title or role with limited or no direct interaction), proxy mistrust (i.e., mistrust in representatives involved in the research process), and no trust. We were interested in the percentage of respondents who noted critical reflective trust, where trust is at the place where mistakes and other issues resulting from differences (in culture; power) can be talked about and resolved. Exploring partnership trust is essential as it underlines the importance of a trusting and respectful relationship between researchers and patient-partners. Trust is fundamental for effective collaboration, open communication, and the successful achievement of research goals.

The PI version asked researchers about the extent to which they had engaged stakeholders in their project, partnership development (stakeholder involvement in the research process), stakeholder involvement in the partnership, and perceived partnership trust. Instead of asking PIs to reflect on their own involvement, the CBPR questionnaire for PIs asks them to rate how involved stakeholders have been engaged in each step of the research process using the single item with seven response categories described above. The same question used to assess stakeholder trust in the partnership was also used to assess PIs’ trust in the partnership.

PPEET

Data collection consisted of administering a modified version of a standardized questionnaire, the Public and Patient Engagement Evaluation Tool (PPEET) [16]. This tool has been previously validated, applied in different health care organizations, and more recently refined after a feasibility assessment [17,18,19]. The PPEET is designed to explore existing enablers and barriers related to patient engagement processes, as well as the impact and influences of patient engagement. The ‘Ongoing/Long Term Engagement Initiative’ section of the PPEET was used in the current study. The questionnaire contains 21 statements or questions for PAG (5/21 are open-ended questions, 3/21 were added as per feedback by our parent-advisors who reviewed the questionnaire before deployment as described above) and 18 questions/statements for the research team in four categories: (1) Communication and support for participation; (2) Sharing views and perspectives; (3) Impacts and influence of the engagement initiative; and (4) Final thoughts/satisfaction. Sixteen statements were rated using a 5-point Likert scale, ranging from ‘strongly disagree’ to ‘Strongly agree’ in both versions used. Each category also includes one to two open-ended questions to further explore respondents’ perspectives (e.g., in Communication and support for participation: “What else would you like us to know about how your participation in was supported”).

Data management and analysis

The data was securely stored in REDCap, a password-protected platform, and accessed or reviewed by members of the research team and CHILD-BRIGHT Network staff. To ensure confidentiality, all study participants were anonymized using numerical identifiers, preventing the display or linkage of any identifying information to their profiles, which may contain such details. The CBPR questionnaire’s and PPEET’s quantitative results were explored using univariate descriptive statistics in IBM SPSS Statistics 27. All responses to the open-ended questions of the PPEET were transferred into the NVivo 12 (QRS International, Doncaster, VIC, Australia) software for qualitative analysis. An inductive thematic-based approach [20] was used to analyse the qualitative data as follows: one reviewer (TO) examined all the responses and identified general emerging themes and subthemes. Themes and subthemes were determined based on inductive reasoning, where utterances revealed key descriptors of the phenomenon. The reviewer (TO) then coded each response under the most suitable theme/subtheme (s). Additional subthemes were created or collapsed (in case of overlap in emerging idea) where necessary. The final coding was then verified for accuracy by a second reviewer (MG). Disagreements in categorization of utterances were discussed between the two reviewers; if a decision could not be reached, it was resolved by a third reviewer (AM).

To ensure strong quality in reporting about patient and public involvement, we followed the Guidance for Reporting Involvement of Patient and the Public short form (Additional file 1: S1) [21]. In accordance with these guidelines, the present manuscript provides information on engagement objectives and methods, positive and negative results, impacts and influences of patient engagement, and the team’s critical perspective on the experience.

Results

Table 1 displays the distribution of stakeholder groups per year. The Network included n = 272, 305, and 436 members in Years 1, 2, and 3, respectively, demonstrating consistent Network growth over time. This expansion was primarily driven by a rise in patient-partners, including youth advisors (+ 51 new members between Years 1 and 3), researchers (+ 12), research staff (+ 38), trainees (+ 33), and committee members (+ 10).

Table 1 Stakeholder groups per year

CBPR

Survey respondents

The Total number of respondents who completed the CBPR by stakeholder group per year is outlined in Table 2. Response rates were 61.4%, 30.2%, and 14.2% for Years 1, 2, and 3, respectively. In Year 1 (2018), many respondents still were unsure of their role in the Network, thus selected ‘Not specified’. While we show the number of PIs as a stakeholder group, we report on their results separately from other stakeholders, given their different position within the CHILD-BRIGHT Network and their use of the PI version of the CBPR tool.

Table 2 CBPR survey respondents by stakeholder group and year

Stakeholder involvement and appraisal of the partnership

Table 3 presents the percentage of stakeholders (other than PIs) who rated the CBPR items of interest at each of the three data time points. In Year 1, the most common involvements in research process occurred in areas of collecting data (65%) and creating instruments or guidelines (55%). The highest level of positive appraisal of the partnership was reflected by respondents “[being] satisfied with [the] level of involvement in this project” (83%). In Year 2, the most common involvements in research process were in areas of in collecting data (63%), followed by creating research instruments or guidelines (57%), while the highest level of positive appraisal of the partnership was reflected by respondents “feeling comfortable sharing [their] opinion when the different partners come together” (91%); being “satisfied with [the] level of involvement in this project” (83%); and reporting that “so far, the work of the partnership has been good for the community” (83%). In 2020, the most common involvements in research process were in the areas of creating research instruments or guidelines (66%) and disseminating findings (64%). Similarly, the highest level of positive appraisal of the partnership was reflected by respondents “feeling comfortable sharing [their] opinion when the different partners come together” (92%); being “satisfied with [the] level of involvement in this project” (81%); and reporting that “so far, the work of the partnership has been good for the community” (85%). Over the years, most stakeholders (94, 86, 94% in Year 1, 2, and 3 respectively) reported felt that the project was a “true partnership” (i.e., collaborative, and equitable relationship between stakeholders) while endorsements of critical reflective trust (i.e., trust that allows for mistakes and where differences can be talked about and resolved) was highest in Year 3.

Table 3 Stakeholder involvement and appraisal of the partnership

PIs’ appraisal of stakeholder involvement and partnership trust

Table 4 summarizes how PIs rated the CBPR items of interest per year. Over the 3 years, 92–100% PIs reported that patient-partners’ involvements in research process were in the areas of defining the problem, deciding on which issue to research, and developing research questions. Also, in Year 2 and 3, PIs reported common patient-partner involvements in the areas of collecting data and creating research instruments or guidelines. Appraisals of the partnership were also positive across items (ranging from 70 to 100%), while the endorsement of the highest level of trust was uppermost in Year 3 (75%).

Table 4 Researchers/principal investigator appraisal of stakeholder involvement and partnership trust

PPEET

Survey respondents

Table 5 outlines the PPEET survey respondents’ stakeholder group distribution and their engagement duration. While Network 106 members (n = 31 committee members, n = 9 patient-partners on committees, n = 18 patient-partners on projects, n = 44 researchers, and n = 4 youth advisors) started the survey, the responses of 95 participants (21.8% response rate) were recorded and considered in the analysis. They included researchers (n = 44 across 12 research projects), committee members (n = 31, across 6 committees), patient-partners on projects (n = 18/61, across 8 projects), patient-partners on committees (n = 8/26, across 4 committees), and youth advisors (n = 4). The response rate per stakeholder group was modest at 39.6% for researchers, 29.5% for patient-partners on research projects, 30.7% for patient-partners on committees, 57.4% for committee members, and 40% for youth advisors. We noted that 67.3% of respondents were members of their respective stakeholder group since the beginning of the project, whereas 23.2% were novice members who have joined the teams within the year preceding recruitment. Exceptionally, all youth advisors were novice members. Researchers included, but were not limited to, principal investigators (n = 10, 23.2%), collaborating investigators (n = 11, 25.5) and research assistants (n = 11, 25.5%). Most patient-partners on projects were parents of children/youth with disability (n = 13, 76.4%).

Table 5 PPEET survey respondents by stakeholder group and their project engagement duration

Quantitative outcomes

Figure 1 illustrates the average response frequencies for different group members. We note that most stakeholders (88.5–100% of survey respondents) ‘agreed’ to ‘strongly agreed’ with patient engagement elements assessed through the PPEET, suggesting positive and strong patient engagement processes and impact in the Network. The average ‘agreed’ to ‘strongly agreed’ response rate across all PPEET questions for all respondent groups combined is high at 92.7%.

Fig. 1

Average “Agree”—“Strongly Agree” response frequencies across all PPEET questions

Additional file 1: S2 provides detailed response frequencies to all PPEET questions for the four different stakeholder groups. Patient-partners on committees generally ‘agreed—strongly agreed’ (83.3–100%) with all PPEET items related to communication and supports for participation, except for their “[satisfaction] with the compensation [they] receive for [their] role on the committee”, where 66.7% ‘agreed-strongly agreed’. In sharing views & perspectives, 100% of patient-partners on committees ‘agreed-strong agreed’ that they were “able to express their views freely” and reported feeling that “their views are heard”, while 83.3% ‘agreed—strongly agreed’ that “a wide range of views are shared on the topics discussed during meetings/activities”, and that they have “[brought] forward a broad range of perspectives of the discussion topics”. In impacts & influences of engagement initiative, 80–100% of patient-partners ‘agreed-strongly agreed’ that the “committee has achieved or is on the right path to achieve its objectives”, that they are “confident that [their] input provided as a patient-partners will be used by the committee”, and that “the input provided will make a difference in the work of the committee”. Lastly, in final thoughts,100% of patient-partners on committees reported that they are better informed about SPOR, while 80% are “satisfied with their role” and report that their “role as a patient-partner on the committee is a good use of [their] time”. For patient-partners on projects (n = 18), PPEET results were very similar to those on committees, ranging from 80 to 100% of participants indicating an agreement. However, one patient-partner reported that they ‘strongly disagree’ with being “able to express their views clearly”, and one respondent “disagreed” that “the individuals participating as patient-partners in the project bring forward a board range of perspectives on the discussion topics”. Youth advisors (n = 4, 100%) agreed with all the PPEET items. Similarly, 85–100% of researchers and 83.3–95.8% of committee members ‘agreed—strongly agreed’ with all the PPEET items.

Qualitative outcomes

Tables 6, 7, 8 and 9 summarize the emergent themes/subthemes. Examples of most salient quotes for each theme and subtheme are available in Additional file 1: S3. Four main themes were identified: (1) Advantages of patient engagement (PE), (2) Barriers to PE (3) Facilitators to PE and (4) Strategies to improve PE. Two-hundred-and-forty-eight (n = 248) utterances were analysed, where 44% were related to advantages of PE, 21.8% and 16.1% were barriers and facilitators respectively, and 18.1% were improvement strategies. Amongst the groups, researchers provided the most responses with 57.7% of the utterances, followed by the committee members (32.3%), and patient-partners (10%).

Table 6 PPEET results: Advantages of patient engagement (PE)

Table 7 PPEET results: Barriers to patient engagement (PE)

Table 8 PPEET results: Facilitators to patient engagement (PE)

Table 9 PPEET results: Strategies to improve patient engagement (PE)

Advantages of patient engagement included research project benefits (valuable input making research more relevant and representative of end users’ needs, improvement of recruitment, methodology, relevance of knowledge translation and dissemination, keeping research team on track, and pilot testing and modification to the protocol); learning and having a voice (having a chance to advocate for self and peers, learning about others’ perspectives and about strategies for patient-oriented research, and adapting to changes in interactions related to the pandemic); and connections (collaboration opportunities, close contact with research team, and bridging gaps between people involved). These benefits were most reported by researchers and committee members (92.5% of statements), and less by patient-partners (7.3%). Most patient-partners’ reported benefits were related to increasing the research relevance and representativeness (50% of patient-partners’ utterances), while researchers mainly reported on methodological/recruitment and knowledge mobilization benefits (62.5% of researchers’ utterances).

Barriers to patient engagement were diverse. They were mainly identified by researchers (59.3% of statements) and committee members (29.6%), and less by patient-partners on projects (11.1%). Most reported limitations were group homogeneity, lack of clarity regarding roles, deciding what perspectives from patient-partners are to be incorporated and how, as well as scheduling and organizing meetings that suit everyone’s availabilities.

In terms of facilitators, patient-partners were more outspoken (20.0% of statements) than in other sections of the PPEET (6.7–11.1%). The main facilitator identified by all groups was related to effective communication strategies. More specifically, optimal meeting organization and planning was reported as an important enabler. Furthermore, it emerged that one main facilitator is ensuring that opinions are being welcomed, acknowledged, heard, and respected.

Lastly, nineteen (n = 19) different strategies to improve patient engagement were conveyed by all key stakeholder groups. One commonly reported solution by committee members and researchers was to promote the clarity on goals, outputs, and contributions by being transparent with expectations and timelines (22.2% of utterances). Moreover, recruiting more diverse members to enhance the existing patient-partner group’s heterogeneity was proposed by all groups, including patient-partners (15.6%). Further strengthening communication channels and ensuring appropriate renumeration/other incentives were reported as other possible strategies by researchers and committee members.

Discussion

Our objectives were to measure patient engagement longitudinally as the research projects evolved through their cycle in the CHILD-BRIGHT Network, and to explore the perceived benefits, barriers and facilitators and overall satisfaction with patient engagement from the perspectives of the different stakeholder groups in the Network. Our process began with measuring patient engagement using the CBPR questionnaire, where it was applied yearly over a 3-years period and was suitable to define the research processes and activities of the research project that Network members were involved in, to appraise the perceived level of the partnership, and the type of trust in this partnership. Understanding how partnerships are formed and evolve provided us with insights into the dynamics of collaboration within the Network. Assessing partnership development helped us uncovering potential factors that contribute to successful and sustainable research partnerships. In Year 3, we aimed to deepen our understanding of patient engagement and implemented the PPEET in addition to the CBPR questionnaire.

Overall, as indicated by all the key stakeholder groups, including researchers, committee members, patient-partners, and youth advisors, data suggest that over the 3-years period we have been effectively implementing patient-oriented research strategies. Our findings show that patient-partners were satisfied with their level of engagement in the Network’s research and governance, and the quality of patient engagement was highly rated across the different stakeholder groups. Notably, in Year 1, our findings show a large proportion of stakeholders expressing satisfaction with their level of involvement in the project, which remained robust in subsequent years. Importantly, stakeholders highlighted feeling comfortable sharing their opinions when partners came together, showcasing positive collaborative dynamics. More specifically, in appraising the partnership, we noted that with time, patient-partners gained comfort in sharing their opinions. This was shown by a more prevalent agreement among respondents in Years 2 (91%) and 3 (92%) compared to Year 1 (74%) on the CBPR questionnaire. This encouraging finding is further supported by what we determined using the PPEET, where more than 80% of patient-partners (81.3–100%) reported being able to express their views, felt that they were being heard, that a wide range of topics were discussed in meetings, and that they contributed to a broad range of perspectives in these discussions. To the contrary, previous research in the fields of cancer and low-back pain indicated that patient-partners felt that they were not being listened to [22,23,24] and reported feeling excluded from regular interactions with the research team [22, 25]. We propose that by ensuring regular communications with our patient-partners and researchers on findings from our Network-wide measurement efforts, offering training opportunities and feedback to the teams, and enabling patient-partners to share experiences with our parent peer mentor, the patient engagement was supported and strengthened.

Moreover, throughout the study period, a substantial majority of patient-partners viewed the project as a "true partnership”, suggesting a widespread belief in the collaborative and equitable nature of relationships among stakeholders. Furthermore, endorsement of critical reflective trust, characterized by an environment where mistakes can be discussed and resolved, increased over the years. Our findings indicated that the highest endorsement was observed in Year 3, revealing a growing level of trust among stakeholders. Similarly, our study determined that researchers consistently appraised the partnership positively across various items and the highest level of trust endorsement was most prevalent in Year 3, indicating a strengthening of trust over time. Overall results from the CBPR questionnaire suggest a positive trajectory in stakeholder involvement, partnership dynamics, and trust within the CHILD-BRIGHT Network. The consistent positive appraisals and the increasing endorsement of critical reflective trust underscore the success of the collaborative efforts and the establishment of a true partnership within the Network.

Additionally, our findings highlighted a fundamental facilitator of patient-oriented research, specifically in the realm of communication strategies. This is in line with previous findings from another research network in the field of rheumatologic conditions [26], where communication was identified as the key facilitator. Similarly, within one of the 13 research projects in our Network (BRIGHT Coaching, Theme 3: Service delivery redesign to address gaps in service), researchers and patient-partners recognized facilitating communication as a crucial factor in enabling patient engagement [8]. Specifically, the BRIGHT Coaching group administered the PPEET midway within the randomized trial to members of their research team (researchers and patient-partners). In this “project-specific” patient engagement evaluation, fewer patient-partners agreed with statements related to communication, sharing views and perspectives (on the PPEET) than in the overall Network’s evaluation in 2021 (50–74% vs. 83.3–100% “Strongly agree”). Following the 2019 project-specific evaluation, we have implemented strategies to improve patient engagement, particularly in this period of data collection. These included increasing accountability of how and what patient-partners’ feedback was being incorporated; involving patient-partners more actively in discussing what factors should be considered as important in data analysis and developing plans for knowledge dissemination of results; implementing a shared directory of ongoing and planned tasks involving patient-partners that allows team members to view task assignments and avoid surcharging patient-partners; and offering more time/date options for virtual meetings. We suggest that these measures had a positive impact on patient engagement by fostering a more inclusive and transparent collaboration and addressing communication-related concerns.

In our longitudinal evaluation of patient engagement, we identified that the lack of role clarity was one of the main reported barriers. Another Canadian network in the field of cardiovascular research also found that the lack of role clarity was one of the emerging barriers to patient engagement [27]. In our investigation, the most commonly proposed solution to improve patient engagement was to clearly inform all team members of the goals of their group (e.g., committee, project), their roles, expectations, and timelines. We acknowledge that these factors are not static and can change over time. For instance, in using the CBPR questionnaire, we confirmed that patient-partners’ engagement in the research process evolved with time, where they were mostly involved in processes related to project development, recruitment and data collection in the first 2 years and progressed to mainly disseminating findings in Year 3. Within the CHILD-BRIGHT Network, several projects have now identified an individual or individuals (e.g., patient-partner, research coordinator, and/or researcher) who connects regularly with their patient-partner advisory group to clarify their roles as the research project evolves.

Our longitudinal evaluations have also demonstrated other areas for improvement. Some respondents highlighted the need to enhance the diversity of our patient-partner members. Efforts are underway to recruit more diverse members into the Network to ensure we are getting input from underrepresented groups. One strategy that is being implemented is the creation of the Equity, Diversity, Inclusion, Decolonization & Indigenization (EDI-DI) Program. The EDI-DI program aims to ensure all voices, bodies, and experiences are included in all aspects of CHILD-BRIGHT's work by deploying initiatives (e.g., consultation, training) based on EDI-DI principles and embedding them in the Network’s programs and governance structure. We anticipate that the EDI-DI program can transform the landscape of patient engagement by fostering inclusivity, cultural competence, and equity, enhance the experience of patients involved in the Network and positively contribute to the effectiveness and relevance of patient engagement initiatives. Moreover, future evaluations of patient engagement within this context can benefit from considering the unique impacts and outcomes across diverse communities.

One positive finding from our longitudinal evaluations is that over 80% of patient-partners reported that their group (research project, program committee) was on the right path to achieving its objectives and reported having high confidence that their input was used and made a difference in the work related to their group. In fact, ‘research project-related benefits’ was the most frequently mentioned in the patient engagement benefits sub-theme. Specifically, Network members indicated that patient engagement was important in improving the impacts of knowledge translation and dissemination; as well as in providing valuable input to ensure that the research was more relevant and representative of end-users’ needs. This is consistent with previous research on the benefits of patient engagement [4, 5, 28,29,30] and the CIHR SPOR mandate [1].

A working group of patient-partners, researchers, trainees, and staff oversaw the implementation and analysis of the patient engagement measurement strategies. Results were carefully reviewed, and our Citizen Engagement Council and our National Youth advisory panel developed tipsheets for researchers [9], parent-partners [10] and youth-partners [11] that describe the ‘how to’ for successful patient-oriented research. We also summarized the key findings of this study on a blog highlighting key lessons learned in a visual summary format and full report [12]. As mentioned above, the Network adjusted its efforts in support of patient engagement over time. We also conducted interviews with researchers and patient-partners to further understand barriers and facilitators to patient-oriented research, to inform our engagement strategies moving forward [7].

This study has limitations, particularly with a relatively modest response rate (average across all evaluations: 31.9%), as not all Network members participated in the patient engagement survey(s) at every measurement timepoint. In addition, we have observed a decrease in the response rate on the CBPR (61.4%, 30.2%, 14.2% for Years, 1, 2, and 3), which could be the result of using another tool in Year 3 (PPEET) and the COVID-19 pandemic. While efforts were made to encourage participation, the varying response rates may impact the generalizability of the findings. Moreover, it was not possible for us to conduct a single-subject repeated measures design to confirm changes in individual perspectives related to the patient engagement process, satisfaction with the experience, and its impact. Nevertheless, we determined that on average, there was a high and stable satisfaction level with patient engagement processes. Importantly, our measurements indicated an improvement in communication strategies and patient-partners’ comfort level in sharing their opinions over time.

The most important lessons learned from our patient engagement evaluation journey include: (i) it is feasible to evaluate patient engagement in a large Network both cross-sectionally and over time, (ii) there are several standardized measures available to carry out these evaluations and these could be used jointly to complement findings, and (iii) these periodic assessments are critical in enabling the Network to reflect on the best practices of patient-partnership and ways to optimally support authentic engagement practices [7].

Conclusion

In conducting our evaluation approaches over time, we determined that patient engagement was positively rated. Several areas were highlighted for some improvement, which allowed the Network to develop strategies to support authentic engagement more optimally. These results are encouraging given that we partner with members across a large, nation-wide Network, where connections are primarily virtual, and most members were novices to the patient-oriented research process at the onset of the Network’s activities.