Key issues for stakeholder engagement in the development of health and healthcare guidelines
Research Involvement and Engagement volume 9, Article number: 27 (2023)
Established in 2015, the Multi-Stakeholder Engagement (MuSE) Consortium is an international network of over 120 individuals interested in stakeholder engagement in research and guidelines. The MuSE group is developing guidance for stakeholder engagement in the development of health and healthcare guideline development. The development of this guidance has included multiple meetings with stakeholders, including patients, payers/purchasers of health services, peer review editors, policymakers, program managers, providers, principal investigators, product makers, the public, and purchasers of health services and has identified a number of key issues. These include: (1) Definitions, roles, and settings (2) Stakeholder identification and selection (3) Levels of engagement, (4) Evaluation of engagement, (5) Documentation and transparency, and (6) Conflict of interest management. In this paper, we discuss these issues and our plan to develop guidance to facilitate stakeholder engagement in all stages of the development of health and healthcare guideline development.
Plain English summary
A group of international researchers, patient partners, and other stakeholders are working together to create a checklist for when and how to involve stakeholders in health guideline development. Health guidelines include clinical practice guidelines, which your healthcare provider uses to determine treatments for health conditions. While working on this checklist, the team identified key issues to work on, including: (1) Definitions, roles, and settings (2) Stakeholder identification and selection (3) Levels of engagement, (4) Evaluation of engagement, (5) Documentation and transparency, and (6) Conflict of interest management. This paper describes each issue and how the team plans to produce guidance papers to address them.
In recent years, interest in stakeholder engagement in the development of health and healthcare guidelines has increased. This is demonstrated by the increasing number of tools being developed to assist guideline developers with involving certain groups in their guideline processes, particularly patients and members of the public [1, 2]. We define a stakeholder as people and groups who are responsible for or affected by health and healthcare-related decisions . We recognize that this term is problematic for some populations and are actively working with relevant groups to select and suitable alternative term (https://theoche.ca/f/muse-work-and-terminology).
By ‘engagement’, we mean the approach to gather input or contributions from stakeholders resulting in “informed decision-making about the selection, conduct, and use of the research” .
In addition to patients and the public, there are other stakeholder groups who are responsible for or affected by health and healthcare-related decisions, such as payers/purchasers of health services, peer review editors, policymakers, program managers, providers, principal investigators, product makers, and payers of health research. Appropriate participation by each of these groups should be considered through discussions amongst the groups themselves in addition to engagement with the guideline development panel [4,5,6]. This ensures that feedback from each group is shared with the other groups which may help improve the guideline and its relevance.
In 2015, research teams from Canada, the US, and the UK met virtually to discuss mutual interests in improving stakeholder engagement in research and guidelines. The group identified gaps in guidance related to how to engage different groups in a meaningful way across all types of research and guideline development. To address these gaps, we established the Multi-Stakeholder Engagement (MuSE) Consortium which has grown to become an international network of over 120 individuals from 20 countries. All members have an interest and expertise in different aspects relevant to stakeholder engagement in research. The group is governed by a core team who manage the daily tasks of the group as well as stakeholder group co-leads. Members of the consortium are invited by email and newsletters to contribute to projects or tasks if they are interested and can use the network to share related work.
We are currently developing guidance for multi-stakeholder engagement in the development of health and healthcare guidelines. To inform the production of this guidance, we held a number of exploratory discussions with members of the MuSE Consortium and individuals representing our identified stakeholder groups. The aim of this paper is to report and discuss key issues for multi-stakeholder engagement in guideline development.
The stakeholder groups identified within our protocol, based on published work [3, 7, 8], include patients, payers of health research, payers of health services, peer review editors, policymakers, principal investigators and members of the research team, providers, product makers, program managers, and members of the public (See Table 1).
We recruited 2–4 co-leads for each identified stakeholder group. Co-leads were selected for their specific, recognized expertise relevant to their stakeholder group (for example, lived experience or research experience). These individuals were identified through snowballing starting with suggested contacts from our core team and the suggestions of those who were contacted. We aimed to ensure co-leads were balanced between high- and low- and middle-income countries.
The objective of this project is to develop a Stakeholder Engagement Checklist Extension of the GIN-McMaster Guideline Development Checklist , to be used to gain broad feedback from relevant stakeholders. In February 2021, we held a 3-day virtual meeting with all of our stakeholder group co-leads and other members of the MuSE Guidelines project team. Several cross-cutting issues were identified which warrant further consideration as the Stakeholder Engagement Checklist Extension is developed.
Six substantive issues were identified as being important for further exploration.
Definitions, roles and setting
Stakeholder identification and selection
Levels of engagement
Evaluation of engagement
Documentation and transparency
Conflict of interest management
A description of key considerations for stakeholder engagement in health and healthcare guideline develop is provided below.
Definitions, roles and setting
Stakeholders may include not only those directly involved in guideline development but also those involved in preparing research, conducting research, sharing evidence, or implementing that evidence.
Our team has previously reviewed a number of frameworks for categorizing stakeholder groups in research [3, 10]. There are many similarities, and several differences related to preferences for lumping, splitting, refining the groups or tailoring to optimize for the specific context of guidelines. For example, in some research settings patients and the public are combined into one stakeholder group, but for the MuSE guidelines projects our public and patient stakeholders asserted that these two groups had different perspectives and would be interested in engaging in different aspects  of the guideline development process, therefore should be considered separately.
Similarly, our stakeholders agreed that we should separate peer-reviewed journal editors from principal investigators and the research team. Academic journals often have requirements for the methods, rigor, and format of publishable guidelines which will require different input than the research team. The research team may be contributing from the perspective of those who have conducted primary research or as systematic reviewers who have assessed the body of evidence on a topic requiring engagement in different aspects of guideline development.
Where other frameworks have separated payers and purchasers of health services into two distinct groups, we have lumped these together because there are many health care systems in different countries that do not make this distinction and therefore their role for guideline development would be the same.
We suggest that there are 10 stakeholder groups that warrant individual attention for health and healthcare guideline development. The roles that each of these groups play, whether providing feedback or contributing to decision-making, may depend on the guideline and its setting. We recognize that the list provided in Table 1 is not exhaustive and there may be other important stakeholders for guideline development, depending on the context and setting.
Our activities prompted considerable debate on whether the organizations that commission guidelines and the guideline secretariats, which we call Px, are stakeholders that meet our definition of interested people and groups. Although we agree that the Px group are key actors and their interactions with the stakeholder groups are essential, we have decided to keep them separate from our 10 stakeholder groups for two reasons. First, this group likely has a decision-making role throughout all stages of guideline development. Second, the guidance that we intend to develop is aimed at how the Px group should engage with other important stakeholder groups. Therefore, the decisions made by the Px group are guided by other documents, such as organizational handbooks, and our guidance is intended to assist the Px with engaging other groups.
Stakeholder identification and selection
The selection of specific individuals to represent each group requires consideration. To address these challenges around identification, we have developed a list of factors to consider when selecting individuals to engage in health research (see Box 1) .
Clarity is needed when one individual can represent more than one stakeholder group, for example a policymaker who is also a patient, or a principal investigator who is also a provider. A ‘positionality statement’, which allows each member of the guideline panel to declare which stakeholder group(s) they are representing in addition to the groups they could have represented, may be helpful . This could be developed by each individual with guidance from the guideline secretariat to ensure that the roles are filled as needed for the specific guideline.
Levels of engagement
The level of engagement of stakeholders in guideline development can vary. Previous stakeholder engagement work identified 4 levels of engagement adapted from other sources [14,15,16]: (1) Communication: Stakeholders receive information but have no role in contributing, (2) Consultation: Stakeholders provide their views, thoughts, feedback, opinions or experiences but without a commitment from the guideline developers to act on them, (3) Collaboration: Stakeholders are engaged to influence the production of the guideline (e.g. commenting, advising, ranking, voting, prioritizing, and reaching consensus) without direct control over decisions, and (4) Coproduction: Stakeholders are equal members of the guideline development team and have a key role in decision-making in the guideline development process. However, our experience assessing papers reporting on stakeholder engagement in guidelines identified challenges in operationalization of these four levels. Based on the detail provided in the studies we assessed, the distinction between ‘communication’ and ‘consultation’ was unclear. Similarly, the details defining ‘collaboration’ from ‘coproduction’ were missing in published reports. Therefore, we have opted to simplify this by categorizing engagement into two levels. Our first level is ‘advice/feedback’ which includes both the communication and consultation levels. Stakeholder opinions, perspectives, experiences, or values are sought and considered by the guideline development team. Our second level of engagement is ‘decision-making’ which includes collaboration and co-production; stakeholders actively contribute to making decisions at the different stages of the guideline’s development and recommendations.
While there are nuances within each of our two levels of engagement, we have decided that the use of two clearly distinct levels is more feasible for the assessment of stakeholder engagement and also simplifies the guidance we will include in our planned extension of the GIN-McMaster Checklist.
Evaluation of engagement
Engagement can improve the usefulness of guidelines and recommendations and increase uptake. Engagement works best when it is multi-directional , meaning that all stakeholders are engaged with each other as well as the guideline development group. For engagement to be meaningful, there have to be benefits to the guideline being developed and the stakeholders need to feel that they benefited from the development process.
It is important to have a process and tools to evaluate whether the stakeholder engagement goals of guideline developers were met. This requires clarifying the rationale for engaging stakeholders, who was engaged, the engagement activities, as well as how effective the engagement was . It also requires assessing the characteristics of the engagement and whether feedback was accounted for, and whether contributions to decision making were effective. We plan to develop tools to evaluate stakeholder engagement and use them to evaluate our own attempt at stakeholder engagement throughout this project. We will build on existing tools for evaluating stakeholder engagement in research, such as the Patients Active in Research and Dialogues for an Improved Generation of Medicines (PARADIGM) Patient Engagement Monitoring and Evaluation Framework (https://onlinelibrary.wiley.com/doi/10.1111/hex.13191), and the Patient Engagement in Research Scale (PEIRS) (https://onlinelibrary.wiley.com/doi/full/10.1111/hex.13227). We also plan to produce guidance for using these processes and tools to evaluate stakeholder engagement in guideline development.
Documentation and transparency
Including multiple stakeholders in the guideline development process will necessitate an increased attention to transparency around the methods of engagement as well as potential conflicts of interest (see next section). Recommendations for transparently reporting guidelines already exist . The RIGHT (Reporting Items for practice Guidelines in HealThcare) checklist includes 22 items related to reporting of basic information, background, evidence, recommendations, review and quality assurance, funding, and declaration and management of interests. There are two items relevant to reporting of stakeholder engagement; reporting how all contributors were selected and their roles and responsibilities as well as describing how conflicts of interest were evaluated and managed.
We plan to develop an extension of the RIGHT checklist to include additional items related to engaging different stakeholder groups to address this gap and encourage complete and transparent reporting.
Conflict of interest management
While the goal of multi-stakeholder engagement is to bring a multitude of views and perspectives to the table, it can bring additional conflicts of interest. A conflict of interest exists when “a past, current or future interest creates a risk of inappropriately influencing an individual’s judgment, decision, or action when carrying out a specific duty” (Fig. 1). The duty of a stakeholder representative in the context of guideline development is to develop recommendations that guide clinical treatment decisions and safeguard the interests of the groups the recommendations are intended to serve (typically patients). It is important to distinguish between stakeholder representatives’ conflicts of interests and their ‘legitimate interests’. The latter refer to the interests of the stakeholder group that the individual is representing (e.g., ensuring the recommendation reflects the values and preferences of patients). Those interests should not be used to restrict the contribution of the stakeholder representatives to the guideline development process.
A recently published framework categorizes conflicts of interest according to their level (personal or institutional) and their type (financial, intellectual, personal, and cultural). These conflicts might vary across the different stakeholders. For example, principal investigators could be intellectually conflicted if their research is relevant to specific recommendations. Patient advocates might have institutional conflicts of interest emanating from their organizations’ interests . The same applies to providers who are members of professional medical associations .
Once stakeholder representatives disclose their interests, the guideline developing organization may assess whether the risk associated with each disclosed interest qualifies as a conflict (based on the relevance, nature, magnitude, and recency of the interest). The final step is to manage any conflicts of interests. Such management should strike a balance between ensuring representativeness of the different stakeholders while minimizing bias. While guidance for conflict of interest management in guideline development is available [21, 22], guidance that specifically addresses the management of conflict of interest in the context of multi-stakeholder engagement is lacking. We plan to fill this gap by producing guidance around managing stakeholder conflicts of interest in guideline development.
To address all the considerations identified in our discussions with stakeholders, we have planned for a series of papers. We will develop guidance for managing conflicts of interest across multiple stakeholders in guideline development; a reporting guideline to encourage transparent reporting of stakeholder engagement in guidelines; and finally, we will describe the methods we have used throughout this project to engage with our own stakeholders, reporting on the barriers and facilitators as well as lessons learned.
Our overall goal is to develop a GIN-McMaster Guideline Development Checklist Extension for Stakeholder Engagement. Our previous meetings informed the development of an international survey to obtain broad opinions about the engagement of each stakeholder group in the stages of guideline development and we gathered in depth insight into the engagement of each group through interviews. These will help us contextualize the feedback we have received and to finalize the checklist extension. We have already published a list of criteria for selecting individuals to represent identified stakeholder groups .
Our planned checklist extension will allow for variability in views and flexibility for different settings and guideline contexts and resources. The guidance will focus on which stages of guideline development each stakeholder group should be engaged in; whether the engagement should be in a decision-making or feedback role; and the optimal engagement of stakeholder across the different stages of guideline development. Guidance for engaging with stakeholders, especially patients, or providers [26, 27] as well as other stakeholder groups such as product makers exists related to research and medical product development . However, to our knowledge, our project is the first to develop guidance for engaging with multiple stakeholder groups throughout health and healthcare guideline development.
The work of MuSE and the guidance produced may be generalizable or adaptable for stakeholder engagement in other areas, such as health technology assessment (HTA) and systematic reviews.
Availability of data and materials
GIN. GIN Public Toolkit: patient and public involvement in guidelines. Scotland: Guidelines International Network; 2021.
WHO. WHO Handbook for Guideline Development. 2nd Edition. World Health Organization; 2014.
Concannon TW, Meissner P, Grunbaum JA, McElwee N, Guise JM, Santa J, et al. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J Gen Intern Med. 2012;27(8):985–91.
Mitchell G, Hartelius E, McCoy D, KM M. Deliberative stakeholder engagement in person-centered health research. Social Epistemology. 2021.
Slutsky J, Sheridan S, Selby J. Getting engaged. J Gen Intern Med. 2014;29(12):1582–3.
Frank L, Forsythe L, Ellis L, Schrandt S, Sheridan S, Gerson J, et al. Conceptual and practical foundations of patient engagement in research at the patient-centered outcomes research institute. Qual Life Res. 2015;24(5):1033–41.
Petkovic J, Riddle A, Akl EA, Khabsa J, Lytvyn L, Atwere P, et al. Protocol for the development of guidance for stakeholder engagement in health and healthcare guideline development and implementation. Syst Rev. 2020;9(1):21.
Rader T, Pardo Pardo J, Stacey D, Ghogomu E, Maxwell L, Welch V, et al. Update of strategies to translate evidence from cochrane musculoskeletal group systematic reviews for use by various audiences. J Rheumatol. 2014;41(2):206–15.
Schünemann HJ, Wiercioch W, Etxeandia I, Falavigna M, Santesso N, Mustafa R, et al. Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ. 2014;186(3):E123-42.
Concannon TW, Grant S, Welch V, Petkovic J, Selby J, Crowe S, et al. Practical guidance for involving stakeholders in health research. J Gen Intern Med. 2019;34(3):458–63.
Akl EA, Hakoum M, Khamis A, Khabsa J, Vassar M, Guyatt G. A framework is proposed for defining, categorizing, and assessing conflicts of interest in health research. J Clin Epidemiol. 2022;149:236–43.
Parker R, Tomlinson E, Concannon TW, Akl E, Petkovic J, Welch VA, et al. Factors to consider during identification and invitation of individuals in a multi-stakeholder research partnership. J Gen Intern Med. 2022;2022:1–7.
Holmes AGD. Researcher positionality–a consideration of its influence and place in qualitative research–a new researcher guide. Shanlax Int J Educ. 2020;8(4):1–10.
Pollock A, Campbell P, Struthers C, Synnot A, Nunn J, Hill S, et al. Development of the ACTIVE framework to describe stakeholder involvement in systematic reviews. J Health Serv Res Policy. 2019;24(4):245–55.
Oliver SR, Rees RW, Clarke-Jones L, Milne R, Oakley AR, Gabbay J, et al. A multidimensional conceptual framework for analysing public involvement in health services research. Health Expect. 2008;11(1):72–84.
Crowe S. Who inspired my thinking?—Sherry Arnstein. Res All. 2017;1(1):143–6.
Concannon T, Palm M. Monitoring and evaluation of stakeholder engagement in health care research. In: Lerner DP, M, Concannon T, editors. Broadly Engaged Team Science in Clinical and Translational Research: Tufts Medical Center, RAND Corporation; 2022. p. 129–38.
Chen Y, Yang K, Marušic A, Qaseem A, Meerpohl JJ, Flottorp S, et al. A reporting tool for practice guidelines in health care: the RIGHT statement. Ann Intern Med. 2017;166(2):128–32.
Khabsa J, Semaan A, El-Harakeh A, Khamis AM, Obeid S, Noureldine HA, et al. Financial relationships between patient and consumer representatives and the health industry: a systematic review. Health Expect. 2020;23(2):483–95.
Fabbri A, Gregoraci G, Tedesco D, Ferretti F, Gilardi F, Iemmi D, et al. Conflict of interest between professional medical societies and industry: a cross-sectional study of Italian medical societies’ websites. BMJ open. 2016;6(6):e011124.
Schünemann HJ, Al-Ansary LA, Forland F, Kersten S, Komulainen J, Kopp IB, et al. Guidelines international network: principles for disclosure of interests and management of conflicts in guidelines. Ann Intern Med. 2015;163(7):548–53.
Lo B, Field MJ. Conflict of interest in medical research, education, and practice. Washington, USA: National Academies Press. 2009.
Akl EA, Meerpohl JJ, Elliott J, Kahale LA, Schünemann HJ, Network LSR. Living systematic reviews: 4. Living guideline recommendations. J Clin Epidemiol. 2017;91:47–53.
El Mikati IK, Khabsa J, Harb T, Khamis M, Agarwal A, Pardo-Hernandez H, et al. A framework for the development of living practice guidelines in health care. Annals Internal Med. 2022;175(8):1154–60.
Vernooij RW, Sanabria AJ, Solà I, Alonso-Coello P, Martínez GL. Guidance for updating clinical practice guidelines: a systematic review of methodological handbooks. Implement Sci. 2014;9:3.
Benavent PG, Fernandez CM. Methodological Handbooks & Toolkit for Clinical Practice Guidelines and Clinical Decision Support Tools for Rare or Low-Prevalence and Complex Diseases Handbook #4: Methodology for the Development of Clinical Practice Guidelines for Rare or Low-Prevalence and Complex Diseases. Aragon Health Sciences Institute; 2020.
FDA. Patient-focused drug development: Collecting comprehensive and representative input. Guidance for industry, food and drug administration staff and other stakeholders. Silver Springs, Maryland: US Department of Health and Human Services. Food and Drug Administration. Center for Drug Evaluation and Research. Center for Biologics Evaluation and Research; 2018.
Diaz A, Barbareschi G, Bruegger M, Dr Schryver D. Working with Community Advisory Boards: Guidance and tools for patient communities and pharmaceutical companies. PARADIGM Patient Engagement Toolkit. 2020.
The authors are grateful to Pearl Atwere, Tammy Clifford, Christine Laine, Regina Greer-Smith and Maureen Smith for their contributions to the discussions informing this manuscript.
This project is funded by a Canadian Institutes for Health Research Project Grant.
Ethics approval and consent to participate
Consent for publication
PT is funded by the Canada Research Chair Program. VW and PT are co-principal investigators on the CIHR grant which funds this project. TC is the recipient of grants or contracts from: Centers for Disease Control and Prevention/NIOSH; National Institutes of Health/NCATS; Patient-Centered Outcomes Research Institute; National Evaluation System for health Technology; Center for Medicare and Medicaid Innovation and has 100 shares of Moderna, Inc. Other authors have nothing to declare.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Peer Review Editor
Asma Ben Brahem
Payer/Purchaser of health services
Peer Review Editor
Peer Review Editor
Imad Bou Akl
Janet Hatcher Roberts
Jordi Pardo Pardo
Peer Review Editor
Payer/Purchaser of health services
Section and topic
Reported on page No
Section 1: Abstract of paper
Report the aim of the study
Describe the methods used by which patients and the public were involved
Report the impacts and outcomes of PPI in the study
Summarise the main conclusions of the study
Include PPI, “patient and public involvement,” or alternative terms as keywords
Secion 2: Background to paper
Report the definition of PPI used in the study and how it links to comparable studies
2b: Theoretical underpinnings
Report the theoretical rationale and any theoretical influences relating to PPI in the study
2c: Concepts and theory development
Report any conceptual or theoretical models, or influences, used in the study
Section 3: Aims of paper
Report the aim of the study
Section 4: Methods of paper
Provide a clear description of methods by which patients and the public were involved
4b: People involved
Provide a description of patients, carers, and the public involved with the PPI activity in the study
4c: Stages of involvement
Report on how PPI is used at different stages of the study
4d: Level or nature of involvement
Report the level or nature of PPI used at various stages of the study
Section 5: Capture or measurement of PPI impact
5a: Qualitative evidence of impact
If applicable, report the methods used to qualitatively explore the impact of PPI in the study
5b: Quantitative evidence of impact
If applicable, report the methods used to quantitatively measure or assess the impact of PPI
5c: Robustness of measure
If applicable, report the rigour of the method used to capture or measure the impact of PPI
Section 6: Economic assessment
6: Economic assessment
If applicable, report the method used for an economic assessment of PPI
Section 7: Study results
7a: Outcomes of PPI
Report the results of PPI in the study, including both positive and negative outcomes
7b: Impacts of PPI
Report the positive and negative impacts that PPI has had on the research, the individuals involved (including patients and researchers), and wider impacts
7c: Context of PPI
Report the influence of any contextual factors that enabled or hindered the process or impact of PPI
7d: Process of PPI
Report the influence of any process factors, that enabled or hindered the impact of PPI
7ei: Theory development
Report any conceptual or theoretical development in PPI that have emerged
7eii: Theory development
Report evaluation of theoretical models, if any
If applicable, report all aspects of instrument development and testing (eg, validity, reliability, feasibility, acceptability, responsiveness, interpretability, appropriateness, precision)
7 g: Economic assessment
Report any information on the costs or benefit of PPI
Section 8: Discussion and conclusions
Comment on how PPI influenced the study overall. Describe positive and negative effects
Comment on the different impacts of PPI identified in this study and how they contribute to new knowledge
Comment on the definition of PPI used (reported in the Background section) and whether or not you would suggest any changes
8d: Theoretical underpinnings
Comment on any way your study adds to the theoretical development of PPI
Comment on how context factors influenced PPI in the study
Comment on how process factors influenced PPI in the study
8 g: Measurement and capture of PPI impact
If applicable, comment on how well PPI impact was evaluated or measured in the study
8 h: Economic assessment
If applicable, discuss any aspects of the economic cost or benefit of PPI, particularly any suggestions for future economic modelling
8i: Reflections/critical perspective
Comment critically on the study, reflecting on the things that went well and those that did not, so that others can learn from this study
About this article
Cite this article
Petkovic, J., Magwood, O., Lytvyn, L. et al. Key issues for stakeholder engagement in the development of health and healthcare guidelines. Res Involv Engagem 9, 27 (2023). https://doi.org/10.1186/s40900-023-00433-6