Using a collaborative action-research approach , a national coalition of patient-oriented research leaders, including SPOR SUPPORT Units and Research Networks, patient and community partners, engagement experts, and health system leaders, will collaborate to co-design this evaluation framework. Hence, as much as this project will build on the existing international science of PPE evaluation, the consensus building endeavour will ensure that PPE evaluation is tailored to the national context and will consolidate ongoing collaboration amongst the PPE research community in Canada.
The main research phases are depicted in Fig. 1, including: 1) development of a logic model (via consensus meetings), 2) agreement on a core set of process and outcome indicators (via a Delphi process) and 3) development of implementation guidelines (via co-led expert working groups on implementation and adaptation to seldom-heard populations).
Our evaluation framework development process will be supported by international evidence based on systematic reviews of evaluation frameworks for patient and public engagement in research [9, 11, 12, 15, 18], a systematic review of existing evaluation instruments , evaluation principles for evaluating patient and public engagement in research , and a systematic review of patient and public engagement outcome indicators and evaluation methods . This preparation work will be used to produce background evidence documents for expert working group members, to develop Delphi questionnaires and prepare consensus meeting participants.
With this general understanding of what PPE in research looks like and what the literature broadly considers as successful PPE processes and impacts as a base (e.g. [16, 17]), this project will take the next steps to define what success in PPE looks like for the Canadian research community in particular. Consensus-generating participatory methodologies will be used (Nominal Group Technique and Delphi Method) to consolidate a logic model and build consensus on a core set of PPE evaluation standards. These steps will help us answer the following questions:
What is meaningful PPE in research for the Canadian research community? (Process indicators);
What does successful engagement look like for stakeholders of the SPOR? (Outcome indicators); and
How should we approach the evaluation of PPE engagement in research? (Evaluation criteria)
As Rowe and Frewer  described, defining the term effectiveness/success/meaningfulness is paramount to sound evaluation. Unless there is a clear definition of what it means for a PPE initiative to be effective/successful/meaningful in a given context, it is not possible to develop measures and methods that enable the evaluation of patient and public engagement. However, not universal in nature, the definition of what represents success in a given context needs to be adopted by the key potential users of the evaluation framework. For this reason, our project will work collaboratively with engagement leads, patient partners and researchers from the Canadian SPOR community specifically to define the parameters of these terms.
We recognize that different evaluation perspectives may exist amongst interested parties. Although competing evaluation purposes may coexist, we assume that key partners in patient-oriented research are more likely to measure their engagement activities against a common set of goals if they set those goals together, as a community. Previous studies have documented research partners’ interest in documenting whether their collaborative efforts make a difference in the research process (see ). Our research team will build on the ongoing Canadian collaborations around PPE within SPOR to bring about such consensus and to create a common understanding around what a SPOR Patient Engagement Evaluation Framework should look like.
Virtual consensus meeting
The first consensus-building exercise, employing the nominal group technique, will consist of a structured facilitation strategy for group brainstorming that will allow for a larger conversation to occur between research stakeholders, including patient partners, engagement experts, and researchers. This will fulfill three purposes: 1) building research community mobilization around the construction of the framework; 2) fostering common understanding of PPE evaluation, and 3) generating ideas about core evaluation domains of the logic model (i.e. aligning objectives of PPE in research with evaluation criteria; PPE process dimensions with categories of process indicators; PPE outputs dimensions with categories of outcome indicators). The nominal group technique is a structured variation of small group discussion methods . This structured process aims at preventing discussions from being dominated by a single person or group, by encouraging less vocal group members to participate in the prioritization of solutions and recommendations.
Collaboration with 18 national SPOR Networks, SUPPORT Units and research organizations (from all Canadian provinces and covering research focus in primary care, diabetes, chronic pain, mental health, developmental disability, chronic diseases, kidney diseases) will provide broad representation from the Canadian patient-oriented research community, mobilizing participants with experience on PPE in research and its evaluation. Our consensus panel for the nominal group technique will involve approximately 50 individuals identified through partner organizations (Acknowledgement section). All SPOR funded-entities will be invited to nominate two or three participants – one patient or community partner and one engagement practitioner (eg. research professional, engagement facilitator or researcher with engagement experience), with a focus on identifying participants with engagement expertise and a diversity of perspectives. In addition, the meeting will include other specialists in the field of PPE, including representatives from the Patient Advisors Network, a patient-led community of practice bringing together experienced patient partners in research, and the Canadian Foundation for Healthcare Improvement, a pan-Canadian organisation dedicated to fostering research and healthcare improvement with expertise in patient and public engagement .
This event, that will consist of four virtual meetings held at different dates, was originally intended to be held in person. However, the advent of COVID-19 required us to re-evaluate our original strategy. Within the COVID-19 context, we will hold this consensus process online, in accordance with best practice recommendations for virtual consensus-building (e.g. [24, 30]). The consensus meeting will still include the 8 key phases of a nominal technical group, but it will occur in sequence over 4 virtual meetings, each of which will be carried out with about 12 to 15 participants to facilitate group discussion and sharing. Sub-groups will run in parallel (meetings #1-#2-#3), with summaries from discussions shared among all participants at the plenary discussion phase (meeting #4), as described below.
Meeting 1- preparation phase
A preparation webinar will be offered to participants. This webinar will provide a quick overview of what is currently known about evaluating PPE in research. This will allow participants to have a common knowledge and understanding of the concepts being discussed, and to clarify the project goal and set the stage for discussion rounds. The webinar will address what an evaluation criterion is and what evaluation dimensions are, providing specific examples to the participants. The webinar will be given live with the possibility of a questions and answers period at the end. It will also be recorded in order for participants who won’t be able to attend to watch it asynchronously. Complementary documents will also be available to participants to help them prepare for the following discussions: an outline of project objectives and timeline, a jargon buster, and a synthesis of key articles about PPE evaluation. Ongoing communication with the team is key during the Virtual Consensus Meeting, hence a Slack channel (www.slack.com) will be created and facilitated by a project team member for the duration of Phase 1 to answer questions and orient participants.
Meeting 2- ideas generation phase
Virtual Zoom meetings of 120 min will be conducted with each sub-group to generate ideas on evaluation criteria, process and impact indicators. To identify evaluation criteria, participants will be asked to define principles of PPE in research, to describe their perspectives on what makes PPE in research successful. To identify process dimensions, participants will be asked about standards to determine whether a process of PPE in research is going as planned, and about factors that enable meaningful engagement. To identify impact dimensions, participants will discuss the changes they feel should be brought about when patients are engaged in research and on whom or what these change should be observed. Participants will be asked to identify standards that help determine whether a research project involving patients has achieved the desired outcomes.
Research team members will facilitate these meetings on several dates to accommodate a maximum number of participants. We will create sub-groups composed of equal number of professionals and patient partners. Other options will be made available to participants to ease participation in the discussions. They will have the opportunity, after each meetings, to send their ideas and comments in: 1) writing (format chosen by the participant - email, word, etc.) and 2) audio (with recordings sent to the project team and transcribed by a team member).
Round Robin phase
Verbatim transcripts of participants’ dialogue (recordings of zoom meetings, transcripts and written statements) will be analyzed with the help of Nvivo to identify specific sets of ideas, themes, and remaining questions/needs for clarification. A summary report will be produced by the project team and circulated amongst participants to help start the next facilitated round.
Meeting 3 - clarification phase
At meeting 3, each idea/set of ideas/themes will be discussed one at a time in order to clarify them and/or explain disagreement with emerging positions. These will be 120-min virtual Zoom meetings within sub-groups. Research team members will facilitate the meetings, with several dates proposed to accommodate a maximum number of people. We will create sub-groups composed of equal number of professionals and patient partners.
At the end of the clarification phase, participants will begin to narrow the list of potential ideas. Building on the clarification phase, each member will rank the ideas they consider from most to least important. Zoom provides a survey application built into the plate-form that we will use to ask for a virtual simultaneous vote.
The top ideas from each subgroup will be consolidated by the research team, combining any duplicate ideas, and a master list of ideas will be generated for the plenary discussion and final vote. This list will be circulated amongst participants prior to the last closing meeting.
Meeting 4 - plenary discussion
A 120-min zoom meeting (one date - set in advance to maximize participation) will be held. Briefly, the facilitator will present to participants each idea for each question and their previous ratings in order to examine inconsistent voting patterns and provide an opportunity for a discussion around these outlier ideas. Ideas with strong support will be maintained whereas those with weak support will be eliminated. Participants will be able to use both the microphone and the chat box to participate in the discussion.
A final vote will be called at the end of the meeting on the three sets of ideas (criteria, process dimensions, impact dimensions), asking participants to rank ideas generated by the group.
Meeting 2 and 3 will involve the same sub-groups of participants to increase their sense of belonging and ease discussion. We will try as much as possible to recreate online the same dynamics the face-to-face setting would have provided. For each step of the virtual consensus meetings, a facilitator and note taker from the project team will be facilitating the discussion. The meetings will be recorded to ease note taking and maximize content analysis. Using the notes and transcriptions of the meetings, we will proceed to a thematic content analysis using Nvivo. Analysis will focus on: 1) identifying emerging evaluation criteria, processes and indicators (to generate items for the Delphi priority-setting exercise); 2) participants’ interpretation of the meaning of those criteria and indicators (to inform narrative description of indicators and criteria); and 3) areas of consensus and dissent regarding the relative importance of emergent indicators (to inform our analysis of whose voices were dominant or not in generating potential indicators and criteria).
While the virtual consensus meeting (above) will be used to generate potential evaluation criteria and dimensions, a modified Delphi process will seek to identify those criteria and dimensions that are most important for the community of patient-oriented research experts. The Delphi Technique is a well-known method for structuring (organizing and facilitating) a group communication and consensus process. It is characterized by iterative rounds of questionnaires with systematic feedback between each round. The method allows for large pools of individuals to be involved asynchronously at times and locations convenient to them, while preserving their anonymity. It also allows identification of areas of convergences and dissension among stakeholder groups. For that purpose, two rounds of online questionnaires will seek to determine which standards (process and impact indicators) are most important for the SPOR community. This activity will provide a minimal evaluation structure to be implemented throughout the SPOR environment.
To increase the variety of expertise engaged, the participants invited to take part in the Delphi will cover a broad representation of the Canadian community of patient partners in research and researchers supported/involved within the SPOR environment. We will aim to solicit 200 participants and will be helped by SPOR entities to reach out to their respective members and collaborators to identify potential Delphi participants. SPOR entities will be asked to identify 5 researchers who conduct patient-oriented research in partnership with patients as well as 5 patient partners involved in research. Co-leads of our equity, diversity and inclusion working groups (described below) will also be solicited to identify prospective participants from seldom-heard populations. Purposeful selection of participants will be carried by the research team to ensure that a diversity of perspectives are represented, both from a sociodemographic point of view and in terms of patient experience pathway.
In Round One, we will share our interim report from the consensus building exercise. Panelists will then be asked to prioritize indicators for each dimension, among the list identified during the nominal group technique exercise. Specific prioritization techniques (rating, ranking, choice of scales) will be determined by the research team, based on the number of items emerging from the consensus meetings and pilot testing of the Delphi survey. Participants will be invited to provide optional qualitative justifications for their choices.
In Round Two, participants will receive a summary of Round One results, and will be asked to provide their final, informed priorities, using the same evaluation techniques as used in Round One. These questions are designed to establish the indicators that are the highest priority for the SPOR community. In addition to identifying and ranking indicators, the respondents will also provide input on the chosen wording to identify PPE evaluation standards. This will provide the research team with information to complete common parts of the PPE evaluation framework.
Turnaround time between questionnaires will be 6 weeks. Only two rounds of questionnaires will be sent to participants, as the initial round will already be informed by the results of the consensus meeting. This should increase participation and limit the attrition rate over the process.
We will analyze results for all participants, and by sub-groups (eg. patient partners vs. researchers, sociodemographic characteristics) to identify areas of consensus and dissent among all participants and for distinct groups. These stratified analyses will inform discussions of expert working groups on adaptation of the evaluation framework for specific populations (below).
Expert working-groups on implementation and adaptation
While prior phases of this project will focus on general PPE process and outcomes indicators, this approach does not address the reality that many will face in determining the relative importance to assign to each of these elements in the evaluation of PPE initiatives for specific populations. We know that approaches taken to evaluating PPE differ conceptually and methodologically in response to the different populations engaged as well as the purpose of patient and public engagement in a given context. Abelson and Gauvin  raised these crucial questions: “how evaluation criteria or ‘elements of success’ should be weighted in evaluation, by whom, and whether some criteria are more important than others in terms of their contribution to the evaluation”. (p. 7).
Therefore, in order to answer these questions, six working groups of experts – each co-led by one researcher and one patient/public partner – will work with their respective research and patient/public partner communities in defining recommendations for the implementation and adaptation of the framework. With the results and analysis of the consensus exercises in hand, co-applicants (scientific and patient experts) and collaborators (SPOR Networks and SUPPORT Units) will be involved in designing implementation strategies and outlining fundamental elements of an evaluation design, namely 1) stakeholders’ involvement in the evaluation process, 2) specific evaluation question(s), 3) recommended approaches to evaluation; 4) preferred wording of terms used and formulation of criteria. The six expert working groups will focus on:
General recommendations for implementation of the evaluation framework
Evaluation adaptation with Indigenous communities
Evaluation adaptation with caregivers
Evaluation adaptation with immigrant populations
Evaluation adaptation for patients with physical and/or intellectual challenges/disabilities
Evaluation adaptation for patients with low literacy levels
These six working groups are by no means extensive, other pertinent groups may have included participation from other hardly heard communities as defined by religion, language, gender identity and expression, sexual orientation, age, socio-economic status, and from stigmatized groups such as people using drugs, people living with HIV, or people experiencing homelessness. Our work may thus serve as a template to future teams to adapt the framework to their particular research focus.
The six working groups mobilized within our research will discuss the experience of their own community of patient/public partners in order to identify the contexts and processes that could likely be obstacles or facilitators to contributing engagement practices, and thus outline how to adapt the PPE evaluation framework for these populations. Discussions around the adaptation and implementation of the general evaluation framework for PPE will focus on evaluation indicators and standards that should be emphasized, added or removed, when engaging with specific populations. Those discussions will be informed by population-specific analysis of responses within the Consensus Building Exercise and Delphi process.
Overall, the consensus meetings will have led to the construction of the logic model, the Delphi will have set the SPOR community’s choices for priority process and outcome indicators, and the expert working-groups will provide recommendations on adaptation and implementation of the national framework to specific populations.
Equity, diversity and inclusion
There is strong evidence from the literature that some patient groups face more barriers than others to engage in research [25, 27]. For instance, we know that women are overrepresented in the patient communities and more so in some health research areas than others (e.g. [20, 21]). Conversely, people from ethnic minority groups, living with poverty, or with intellectual disability face specific barriers to engagement that may hamper their meaningful involvement in research .
Our equity, diversity and inclusion strategy has three complementary goals:
That participants in the development of the national evaluation framework include a diversity of perspectives and is inclusive of seldom-heard populations;
That the content of the national evaluation framework addresses evaluation dimensions that are informed by equity, diversity and inclusion considerations;
That implementation of the national evaluation framework is adapted to specific needs and perspectives of seldom-heard populations.
Therefore, from the start of the project, the research team will make sure equity is intertwined in each stage of its design and development. This will involve co-designing each phase of research with patients from seldom heard populations and experts in the field. Our research team actively recruited experts on equity in PPE to foster an inclusive design process. Although inclusion of all perspectives is not feasible, we will pay special attention to 5 seldom-heard groups of particular importance for PPE in research within the Canadian context: Indigenous communities, new immigrants, people with intellectual and physical disability, people with low literacy levels, and caregivers.
At phase 1 – Consensus Meeting – a self-identification questionnaire will help to characterize the sociodemographic profiles of consensus-building participants. At this stage of the process, the focus is on the involvement of participants with expertise and experience in PPE in research, as identified by SPOR research entities. We expect this recruitment strategy will reflect the fact that certain socio-demographic groups are over or under-represented. By highlighting these profiles, it will be easier to take note of potential exclusions and to purposely recruit participants for phases 2 and 3 of the project. We will also have the opportunity to code the transcripts of the meetings in order to analyze whether certain issues of diversity, equity and inclusion emerged, by whom they were raised and whether these elements were reflected in the dimensions and criteria chosen to constitute the logic model.
At phase 2 – Delphi – the research team will recruit purposefully to ensure that a diversity of perspectives are represented, both from a sociodemographic point of view and in terms of patient experience pathway. This includes recruitment of participants from identified seldom heard groups. We will work to adapt the Delphi questionnaire in collaboration with representatives of the five seldom-heard communities to see what kind of adaptations will need to be made to ease participation as much as possible. Among other things, we could consider to roll-out the questionnaire in multiple ways, including by telephone, to help include participants with lower literacy or disabilities. A series of identification questions will help us clarify the profile of Delphi participants in order to better understand the differences and similarities between specific seldom-heard populations preferences and the general group. In addition to allowing the expert working groups to prepare and guide their work, this information will help us to better distinguish between areas of consensus (core evaluation framework) and areas of dissent (informing guidelines for implementation and adaptation).
At phase 3 – Experts-working groups – five of six expert working groups will analyze the simultaneous interactions between different social categories that make up a patient’s social identity, as well as the impact of power imbalances on the involvement of individuals who often carry the burden of illness. Using the list of questions defined by Shimmin et al. , these working groups will discuss the experience of different communities of patient partners in research and answer three questions with the angle of better understanding the specificities of embarking in PPE evaluation with these populations:
Are the evaluation criteria and dimensions representative of their experience? If yes, why? If no, what would be more relevant?
What evaluation criteria and dimensions should be emphasized, added or removed when evaluating PPE in conducting research with your specific group?
Are there evaluation approaches that need to be adopted and protocols implemented in order to adequately embark in the evaluation of PPE?
This project is supported by a research team composed of experts who have been working with these five communities for several years and who have built trusting relationships with participants. The working groups will all require a specific strategy to enhance equity and inclusion. Thus, the first step will be to determine the terms of the collaboration and to define the best ways of working together. We do not want to pre-determine this strategy at the protocol stage. Co-construction is key and will require us to allocate it time and plan this phase of the project accordingly.
Adding to the aforementioned analysis, sex- and gender-based analysis will be led by the research team at the 3 phases to better understand the gendered nature of PPE in Canada and how these factors can affect the process and outcomes of PPE. This health equity exploration will allow for a more inclusive and reflective approach to PPE evaluation and ease the adaptation and implementation of the common framework to specific population needs.
Patient and public engagement
The overarching goal of our engagement strategy within the project is to enhance the relevance, value and usefulness of the proposed evaluation framework to help patients and researchers reflect together on their research partnership. Patient and public partners are involved at every level (strategic, tactic and operational) and phases (research preparation, execution, and knowledge translation) of this research project. Research agenda-setting and framing of the initial project was conducted with the SPOR national patient and public engagement working group, which includes patient and public partners from across the country with extensive experience in supporting engagement in research. Patient partners with extensive experience in research have been engaged since the onset as co-principal investigators (AG) and co-investigators (CC) on the research team and contributed to study design, protocol development and funding application; they will also contribute to ongoing development of study instruments (eg. consensus meeting questions, Delphi survey), recruitment strategies, data interpretation and knowledge translation. Patient and public partners from the 18 national SPOR Networks, SUPPORT Units and research organizations collaborating on this project will be involved in consensus meetings to design the initial evaluation framework template and logic model, alongside 2 national patient leaders from the Patient Advisors Network, an independant community of people who have received health services or cared for those who are, committed to improving healthcare as advisors across Canada. These patient and public collaborators will further be engaged in the recruitment of participants in the Delphi survey, as well as in the dissemination and implementation of the framework within their organizations and communities. Finally, patient and public partners will act as co-leads of the working groups on equity, inclusion and diversity (described above), contributing to its design, structure, recruitment, facilitation and approval of recommandations. Evaluation of our patient and public engagement dynamics (team interaction, support, contribution, feeling valued, convenience, research environment, procedural requirements and benefits) will be conducted at the end of every stage of the project, using the Patient Engagement in Research Scale [16, 17] with additional questions on the experiences of online engagement, in order to support reflexivity and improvement of our own engagement practices within the project.
Research in time of COVID-19
Current health challenges around the world have asked us to reconsider the timeline of this project to allow for the logistical adaptations necessary to comply with public health guidelines while maintaining methodological rigor. At the time of publication of this protocol, the first phase of the project will be well underway and consensus meetings should be completed and the transcripts being analyzed. The transition from a face-to-face to a virtual setting will have lengthened this phase of the project from what was originally planned, delaying the Delphi phase by approximately 3 months. The deployment of Delphi is planned for the spring of 2021 for the first round and the summer of 2021 for the second round. Its preparation (recruitment strategy and logistics, questionnaire development, participant recruitment, etc.) is therefore planned for the winter of 2021. As for the expert working groups, all six of them ask for a specific strategy to strengthen and promote the participation and inclusion of the communities involved. In light of this, this last phase will take place between the months of May and December 2021. This 6-month period should give us enough time to adapt the form and pace of the work to the needs of each of these groups of participants. The complete project was originally planned for a one-year period; the COVID-19 pandemic therefore required us to re-plan for an 18-month period.