A scoping review of methods to measure and evaluate citizen engagement in health research
Research Involvement and Engagement volume 8, Article number: 72 (2022)
Citizen engagement, or partnering with interested members of the public in health research, is becoming more common. While ongoing assessment of citizen engagement practices is considered important to its success, there is little clarity around aspects of citizen engagement that are important to assess (i.e., what to look for) and methods to assess (i.e., how to measure and/ or evaluate) citizen engagement in health research.
In this scoping review, we included peer-reviewed literature that focused primarily on method(s) to measure and/or evaluate citizen engagement in health research. Independently and in duplicate, we completed title and abstract screening and full-text screening and extracted data including document characteristics, citizen engagement definitions and goals, and methods to measure or evaluate citizen engagement (including characteristics of these methods).
Our search yielded 16,762 records of which 33 records (31 peer-reviewed articles, one government report, one conference proceeding) met our inclusion criteria. Studies discussed engaging citizens (i.e., patients [n = 16], members of the public [n = 7], service users/consumers [n = 4], individuals from specific disease groups [n = 3]) in research processes. Reported methods of citizen engagement measurement and evaluation included frameworks, discussion-based methods (i.e., focus groups, interviews), survey-based methods (e.g., audits, questionnaires), and other methods (e.g., observation, prioritization tasks). Methods to measure and evaluate citizen engagement commonly focused on collecting perceptions of citizens and researchers on aspects of citizen engagement including empowerment, impact, respect, support, and value.
Discussion and conclusion
We found that methods to measure and/or evaluate citizen engagement in health research vary widely but share some similarities in aspect of citizen engagement considered important to measure or evaluate. These aspects could be used to devise a more standardized, modifiable, and widely applicable framework for measuring and evaluating citizen engagement in research.
Patient or public contribution
Two citizen team members were involved as equal partners in study design and interpretation of its findings.
Systematic review registration
Open Science Framework (10.17605/OSF.IO/HZCBR).
Plain English Summary
Involving members of the public (citizens) in health research is important. It helps make sure that research focuses on issues that are most important to citizens. It also helps ensure that the research done is respectful of citizen participation and most likely to provide benefit. However, the best way to engage citizens in research is unclear. In this scoping review, we examined existing studies that assessed citizen engagement in health research. We found that citizen engagement was often assessed by asking for feedback from both citizens and researchers. Feedback was collected in person (one on one interviews or group discussions) or in writing (using surveys or audits). Frameworks (organized ways of thinking about an issue) were also sometimes used to measure empowerment, impact, respect, support, and value of engaging citizens. It was clear from the frameworks that there is a need to develop clearer roles for citizens in research. The two citizen members of our research team who helped interpret our study findings felt that a set of guidelines for citizens to help them best participate in health research needs to be developed. We believe these observations could be used to create a more standard method for assessing citizen engagement in research.
Citizen engagement in health research is an increasingly common approach to conducting biomedical, clinical, health system and services, social, cultural, environmental, and population health research with citizens as collaborators rather than subjects [1, 2]. Often referred to using diverse terminology (e.g., community based participatory research, public participation, patient and public involvement), citizen engagement recognizes citizens, defined as interested or affected members of the general public including patients, caregivers, advocates, or representatives of the community as “knowledge users”, or individuals who are affected by the processes and results of health research and can use their lived experience to influence research to be more relevant and useful [3, 4]. Specifically, citizen engagement encompasses meaningful involvement of citizens in various aspects of the health research process such as: membership in advisory groups or steering committees for priority-setting, co-application on funding grants, and research planning, decision-making, conduct, implementation, evaluation, and dissemination [3,4,5].
Engaging citizens in research has the potential to improve the relevance of study findings, minimize waste by facilitating stewardship over resources, create mutual learning and understanding, and build trust in research findings by improving relationships between communities and researchers [6, 7]. Additionally, citizen engagement has shown the ability to provide individuals with opportunities to acquire new skills and knowledge, enjoyment and satisfaction through support and friendship, and financial rewards to compensate their efforts . Due to these documented benefits of citizen engagement, national funding bodies and healthcare organizations worldwide encourage and sometimes mandate citizen engagement in research design and practice [9, 10].
Despite the push towards incorporating citizen engagement in research by funding bodies, citizen engagement is often tokenistic and lacks the clarity and guidance needed to facilitate it’s meaningful use [11,12,13,14,15,16,17,18,19]. Existing guidance on citizen engagement is often provided within a “stakeholder engagement” context which is not specific to citizens and can include health care practitioners, policymakers and industry members and may not directly address the needs of citizens . Finally, literature around citizen engagement tends to focus reiterating benefits, risks, and impact of citizen engagement in health research without detailing specifics on how to appraise citizen engagement in health research [1, 21,22,23,24,25]. As such, there is need for an evidentiary foundation to enable assessment of the degree (i.e., level of engagement in research processes which can vary from participation in research planning committees to recruitment of participants and dissemination of data) and quality (i.e., determining quality of involvement, which may be ascertained collecting citizens’ experiences with the engagement or perceived impact of engagement) of citizen engagement in health research, building upon current guidance provided by national funding agencies and peer-reviewed literature. To develop a high-level understanding of methodology used to appraise citizen engagement in health research and determine aspects of citizen engagement valuable to assess, we conducted a scoping review of literature focused on:
Methods to measure (i.e., determine degree of) citizen engagement in health research; and
Methods to evaluate (i.e., determine quality of) citizen engagement in health research.
We designed and conducted a scoping review to map the existing literature on methods to measure and evaluate citizen engagement in health research according to the Arksey and O’Malley  and Levac  recommendations for scoping reviews. We used the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) to guide the reporting of this scoping review . A detailed description of the proposed methods has previously been published .
Identification of the research question
As degree and quality of engagement are closely related features of citizen engagement, we developed the research question: “what is the state of knowledge on methods to measure and evaluate citizen engagement in health research?” to capture a broad range of potentially relevant literature. Our research question was developed to also shed light on the any relationship between measurement and evaluation of citizen engagement. We ensured that our research question defined the scope of inquiry with respect to population, concept, and outcomes of interest and would direct the subsequent steps .
We defined our target population as “citizens”, or consumers of health services (e.g., patients, families of patients, informal caregivers), advocates and representatives from community organizations, and members of the general public. Our target concept was “engagement”, “involvement”, or “participation” in health research. We used the CIHR definition of health research, which encompasses the biomedical, clinical, health systems and services, social health, cultural health, environmental health, and population health fields . To complement our usage of the CIHR definition of health research with discussion around citizen engagement, we adopted the CIHR definition of citizen engagement or “the meaningful involvement of individual citizens…that is interactive and iterative with an aim to share decision-making power and responsibility for those decisions”. This definition encompassed activities such as priority-setting, planning, acquiring funding, research decision-making, research conduct (e.g., commenting on and developing research materials, interacting with research participants, and/or carrying out research activities), implementation, evaluation, or dissemination to be means of engagement .
Identification of relevant studies
We identified relevant literature on citizen engagement in research using a pre-determined plan for data sources and search strategy, including search terms, languages, and dates of search. As per recommendations, we designed the search strategy to return reasonably relevant results while considering time and personnel workload as limiting factors [26, 27].
Our study team, which included multiple stakeholders and knowledge users including health services researchers (KMF, HTS, JPL), trainees (AS, BKR), patient partners (BGS, SL), a health care professional (HTS), and a health sciences research librarian developed the search strategy. The search strategy (Additional file 1: Item S1) was independently reviewed by a second health sciences research librarian uninvolved with this project using the Peer Review of Electronic Search Strategies (PRESS) checklist .
Using the previously published search strategy , we searched the MEDLINE (Ovid) database from January 1, 2000 to February 1, 2021. We then adapted this strategy to each database to be searched: EMBASE (Elsevier) (January 1, 2000–February 1, 2021), Cochrane Library (Cochrane) (January 1, 2000–February 1, 2021), CINAHL Plus with full-text (Ebscohost) (January 1, 2000–February 1, 2021), APA PsycINFO (ProQuest) (January 1, 2000 – February 1, 2021), Scopus (Elsevier) (January 1, 2000–February 1, 2021), and Web of Science Core Collection (Clarivate) (January 1, 2000–February 1, 2021). We included subject headings, keywords and relevant synonyms related to three concepts:  citizens (e.g., community member, lay person, public, stakeholder), (2) engagement (e.g., collaboration, engagement, participation, involvement), and (3) health research (e.g., biomedical research, clinical research, public health research, environmental health research). We excluded studies published before the year 2000 to capture a modern viewpoint of the ever-evolving practice of citizen engagement in health research. We did not place any exclusion criteria on language. We screened the reference lists of included studies and related systematic reviews to identify additional potentially relevant literature.
We developed inclusion and exclusion criteria a priori through meetings with the study team to refine the study selection process at the beginning, midpoint, and endpoint of the citation screening process in case any unforeseen considerations arose . We screened and selected relevant studies for inclusion in the scoping review independently and in duplicate.
We included articles if they: (1) were primary (e.g., observational or interventional studies) or secondary (e.g., systematic or scoping reviews) research, frameworks, reviews, or reports, (2) primarily focused on citizen engagement in health research (including biomedical, clinical, health systems and services, and social, cultural, environmental and population health, as defined by CIHR) and (3) reported method(s) to measure or evaluate citizen engagement. We did not place any restrictions on language. Non-English language studies were screened using Google Translate . We excluded any literature that discussed methods to measure or evaluate citizen engagement in non-research processes, including health promotion, health education, health system or service delivery and governance (including decision-making), or health program implementation. To gain a focused insight on existing methods determining the degree and quality of citizen engagement in health research, we omitted literature focusing primarily on areas other than measurement and evaluation of citizen engagement in health research.
We imported retrieved articles into Covidence (Veritas Health Innovation, Melbourne, Australia) for title & abstract screening, which was completed independently and in duplicate by two reviewers (AS, BKR, KP, KK, RK, MA, ML, LH, KM). Reviewers conducted a pilot screening of 50 titles and abstracts to ensure consistency in application of inclusion and exclusion criteria. Once a Kappa (inter-rater agreement) of ≥ 0.8 was achieved, reviewers proceeded to screen the remaining articles. If one reviewer indicated an article as potentially relevant at the title & abstract screening phase, the article advanced to full-text review to ensure inclusivity. Following title & abstract screening, we exported a list of included articles into Endnote X9 (Clarivate Analytics, London, United Kingdom). We retrieved full-text versions of included articles using a combination of Endnote X9’s ‘find full text’ feature, Endnote Click online, and the local university online libraries. If a full-text version of an article was not available, a search was conducted of the publishing journal’s website to gain access. Following the search for full-text articles, the Endnote library was re-uploaded to Covidence and full-text screening was completed independently and in duplicate by two reviewers. Reviewers (AS, IL, RK) pilot screened the full text of 20 articles, and screened the remaining articles once a Kappa ≥ 0.8 was achieved. Both reviewers agreed on inclusion status and reason for exclusion at this stage. Any disagreements were resolved by discussion among the reviewers or the involvement of a third reviewer (IL or RK), if required.
Charting the data
We developed an initial data charting form for data elements to be abstracted from the articles, then refined the form through discussion with the study team. The data charting form [Microsoft Excel (version 16.29.1)] was piloted by two reviewers (AS, IL) with ten included studies, and revised as needed. Once the final data charting form was developed, all relevant articles were abstracted independently and in duplicate by two reviewers (AS, IL). Abstracted variables in the initial data charting form included study characteristics, participants, type and goals of citizen engagement, method(s) used to measure citizen engagement, method(s) used to evaluate citizen engagement, features of method(s) used, and observed benefits or risks of the measurement or evaluation method(s) used.
Collating, summarizing, and reporting the results
We collated, summarized, and reported the results by (1) analyzing the data, (2) reporting results, (3) and evaluating their meaning and aimed to contextualize our findings within existing literature and research, practice, and policy [26, 27]. We analyzed qualitative data specifically by having two research team members trained in qualitative methods (AS, IL) inductively code major components of each framework using an analytical approach informed by thematic analysis . First, the researchers independently reviewed each framework to develop a list of relevant terms and associated concepts. Then, the researchers compared lists and discussed discrepancies with a third qualitative expert (JPL). The initial two researchers then deductively analyzed the agreed upon concepts into shared and unique framework features by expanding and collapsing shared meanings through a series of three meetings. Concepts deemed too vague (no label/definition provided) were excluded from analysis.
We present a descriptive summary of characteristics of the included documents, and the characteristics of the intended participants or audiences for these documents alongside a narrative synthesis of abstracted data variables.
Involvement of citizens in literature synthesis is recommended by funding organizations such as the NIHR  and CIHR  and is becoming increasingly commonplace [35, 36]. We involved citizens (BS, NF) in study conception and design including the search strategy. As per recommendations, we involved citizens (BS, SL) in the interpretation and contextualization of the data .
Our search strategy returned 28,353 total results (16,762 results after duplicates were removed). Of these, studies were excluded because they: (1) reported aspects of citizen engagement other than methods of measurement or evaluation (n = 713), (2) reported outcomes or discussion on specific diseases/interventions (i.e., not citizen engagement) (n = 520), (3) had an ineligible study design (i.e., editorials, letters, commentaries) (n = 504), (4) included citizens as health research subjects only (n = 452), (5) engaged citizens, but in non-research (i.e., health service design or delivery) (n = 437), (6) were not health research related (n = 337), (7) focused on non-citizen stakeholders in health research (i.e., clinicians, policy makers) (n = 285), or (8) and/ or had inaccessible full-texts (n = 169). Thirty records met inclusion criteria and were included in our scoping review. Three additional records were found through searching reference lists and included for a total of 33 records. A study flow diagram including reasons for exclusion is shown in Fig. 1.
A majority of the included literature was published within the last 10 years (2011–2021, n = 26, 78.8%). Literature included peer-reviewed journal articles (n = 31, 94.0%), a report by a government organization (n = 1, 3.0%), and a conferencing proceeding (n = 1, 3.0%). Most literature reported work that was conducted in the United Kingdom (n = 16, 48.5%), Canada (n = 6, 18.2%), and other European nations (i.e., Ireland, Germany, Sweden) (n = 6, 18.2%). Most included literature pertained to health services or health research (n = 18, 54.5%), and clinical research (particularly primary care and mental health) (n = 12, 36.4%). Characteristics of the included literature are shown in Table 1.
Citizen engagement: participants, engagement activities and terminology
Included records varied in type of citizens engaged, research engagement activities, and terminology used to describe the engagement process (Table 1). Most studies reported engaging patients or individuals with lived experience of a given illness (n = 16, 48.5%), members of the public (n = 7, 21.2%), service users/consumers (n = 4, 12.1%), or individuals from community groups such as refugees  or children [38, 39] (n = 3, 9.1%). Citizens were engaged in specific research activities such as priority-setting [40,41,42], advisory board/ steering committee membership [38,39,40, 43], grant  and research proposal development [42, 43], study design [42, 45,46,47,48], study materials development [43, 49, 50], study administration and conduct [40, 47, 48, 51], interpretation of findings [46, 50, 52], and dissemination [40, 42, 46, 50,51,52,53]. Other studies described citizen engagement activities more generally as informing citizens of research goals so engagement may occur , collaboration [55, 56], consultation [55,56,57,58,59], co-production [46, 53], co-research , participation , and user control . A word-cloud depicting descriptions of citizen engagement in the included studies is shown in Fig. 2.
Terminology used to describe citizen engagement activities (in order of most common to least common) was: patient and public involvement [37, 39, 47, 49, 53, 58, 59, 61, 62], public involvement [38, 42,43,44, 46, 55], consumer involvement [40, 48, 51, 63], patient and public engagement [64, 65], patient involvement [41, 45], and community-based participatory research [60, 66]. Table 1 shows specific studies, citizen engagement terminology, engagement activities, and types of citizens engaged in further detail.
Methods used to measure and/or evaluate CE
Of the 33 included studies, 20 (61.0%) presented method(s) to evaluate citizen engagement, five (15.2%) presented method(s) to measure citizen engagement, and eight (24.2%) presented method(s) to both measure and evaluate citizen engagement. Methods for the measurement and/or evaluation of citizen engagement included frameworks, discussion-based methods (i.e., focus groups, interviews, workshops), survey-based methods (i.e., audits, questionnaires), and other methods (i.e., indicators, observation, prioritization tasks). Many studies utilized and reported on more than one method to measure and/or evaluate citizen engagement. A summary of these methods is presented in Table 2 and described narratively below.
Citizen engagement strategies: frameworks
Five studies presented frameworks [55, 57, 58, 60, 67] designed to measure and/or evaluate citizen engagement in health research. Frameworks focused on various aspects of citizen engagement including reflection on and impact of citizen engagement activities in research, and recommendations for improvement. The five included frameworks explored measurement and evaluation of citizen engagement through gauging (1) empowerment (i.e., citizens should feel comfortable in voicing their opinions), (2) impact (i.e., research should be positively shaped by citizen engagement), (3) respect (i.e., citizens should feel respected), (4) support (i.e., citizens should have training and supports available), and (5) value (i.e., citizens should feel important to the process). Included frameworks also highlighted the importance of capacity building (i.e., funds, personnel to support engagement in research) , assessing the degree of engagement of researchers and citizens , clarity in roles (i.e., of citizens when engaged) , and involvement of citizens in critical aspects of research (i.e., protocol development, analysis, outputs) . More detail on each framework is provided in Table 2, and similarities and differences between the included frameworks are highlighted in Fig. 3.
Citizen engagement strategies: discussion-based methods
Methods using discussion to measure and/or evaluate citizen engagement utilized focus groups (n = 7) [41, 43, 45, 46, 48, 49, 53], interviews (n = 5) [38, 41, 43, 48, 68], or workshops (n = 1) . Most discussion-based measurement or evaluation of citizen engagement was conducted in a semi-structured manner, with pre-defined topics or question lists [38, 41, 43, 45, 46, 48, 49, 53, 68]. Common focus group discussions included experiences with and perceptions of involvement [46, 48, 53], nature and impact of involvement [45, 46, 53], and recommendations for improving involvement in research [43, 46, 53]. Common topics of discussion in interviews were experiences with involvement [38, 41], perspectives on the project in which involvement occurred , challenges in involvement [38, 41], and opportunities for improvement .
Citizen engagement strategies: survey-based methods
A number of studies used survey-based methods to measure and/or evaluate citizen engagement in health research. These methods included an audit , questionnaires [37, 51, 59, 66, 67, 69], and surveys [48, 50, 51, 59, 63, 65] and varied in number and type of questions asked, content of the questions, and intended recipients of the questions. Open-ended questions [44, 48, 51, 56, 61,62,63, 65, 66, 68, 69] and Likert scale-based statements [37, 48, 49, 65, 66, 68] were commonly used. Many studies used a combination of open-ended questions, closed-ended questions, and/or statements for which a degree of agreement could be declared [48, 50, 51, 59, 63, 65, 66, 68, 69]. Questions were intended for (1) citizens involved in research [43, 44, 48, 50, 52, 61, 63, 65, 67, 69], (2) researchers who involved citizens in their work [44, 51, 52, 56, 59, 62, 63, 68, 69], or less frequently (3) other research or grant administrative personnel [52, 69]. The content of many questions focused on reflections on involvement (i.e., feedback on activity or study involved in) [44, 48,49,50, 52, 59, 61, 65], motivation for involvement in research [43, 48, 56, 61], perceived impact of involvement on the research [44, 49, 51, 61, 69], and recommendations or comments on future involvement in research [43, 63, 68].
Citizen engagement strategies: other methods
A number of studies presented other methods to measure and/or evaluate citizen engagement. These included indicators of user involvement such as documentation of citizen roles in research and availability of training to citizens to facilitate their involvement in research , prioritization tasks focusing on outcomes of the research considered important by participating citizens , and citizen observation of any study steering group meetings and scrutiny of any study documentation . One study used a method to appraise existing frameworks for supporting citizen engagement (the Canadian Centre for Excellence on Partnerships with Patients and Public evaluation tool) , however many of the frameworks discussed intended to support and report rather than measure or evaluate citizen engagement, falling out of the scope of this review.
Our scoping review produced two main findings. First, we found that multiple methods (i.e., audits, focus groups, interviews, frameworks, surveys) have been used, often in combination, to measure and evaluate citizen engagement in health research. These methods collect perceptions of citizens, researchers, and/or research support personnel on many aspects of citizen engagement including reasons, type, and impact of engagement, any challenges encountered in engagement (including project-specific issues), and recommendations for improving future citizen engagement in health research. Secondly, we identified that existing frameworks to measure and evaluate citizen engagement commonly assess perceived empowerment, impact, respect, support, and value. Together, these findings summarize the nature of citizen engagement in health research and itemize citizen engagement aspects that are considered important to assess the degree and quality of citizen engagement in health research.
In addition to our main findings, we identified that the terminology used to define citizen engagement and describe its activities varies widely. Citizen engagement is referred to as patient and public involvement (often in the United Kingdom), patient engagement, public engagement, consumer or service user-involved research, and community based participatory research depending on location and context. Varying terminology may pose a challenge for individual researchers to identify and utilize methods to appraise citizen engagement in research. Standardization of terminology could add to the accessibility and applicability of current and future methods to incorporate and evaluate citizen engagement in health research.
In the process of screening literature for inclusion in this scoping review, we found that much of the current guidance on appraising citizen engagement in research exists in the form of editorials, letters to the editor, commentaries, and perspectives from experienced researchers in the field. We noted that this type of literature does not routinely discuss the merit of discussion-based methods that were used to evaluate citizen engagement in included studies. This could reflect a repeated dismissal of discussion-based qualitative research methods to measure and evaluate citizen engagement in research and warrants further investigation. Despite limited discussion-based methods to appraise citizen engagement, this literature emphasizes the context and process of engagement , clarity, reflexivity, methodological rigour, transparency, pragmatism, and reciprocity as key principles to evaluating citizen engagement in research  and highlights the need for evaluation as an ongoing part of the research process . These elements of citizen engagement complement our main findings and should be taken into consideration when appraising citizen engagement. Our findings also align with previous work emphasizing the importance of evaluating citizen engagement activities as a necessary step in building a strong evidence base for utilizing citizen engagement in health research . Furthermore, previous literature has emphasized a need for standardization in measurement and evaluation of engagement processes as methods to measure or evaluate citizen engagement are seldom utilized beyond the groups that develop them . In light of our findings, we postulate this could occur due to (1) lack of accessibility (i.e., method difficult to find) or (2) lack of perceived applicability/modifiability (i.e., method viewed as unsuitable or too specific to a certain project or type of research and unmodifiable).
As per recommendations by Levac and colleagues, we invited citizen team members (BS, SL) to help interpret the findings of this scoping review and provide insights beyond those in the literature . These citizen team members (BS, SL) remarked that empowerment, impact, respect, support, and value, common to frameworks identified by our study, were important to them in their experiences of participating in research. Additionally, they stated that the ability to openly communicate their concerns about the research project and their involvement has been important to them as members of a research team. Finally, they expressed a desire for an accessible lay resource to help people like them (i.e., citizens) be a meaningful part of research and stated that such a resource would vastly improve their comfort level with participating in research.
Strengths and limitations
This scoping review was designed to form an evidence basis for future work to advance and standardize appraisal of citizen engagement in health research. This study has strengths and limitations to consider. Strengths of our scoping review include: (1) co-development of the study protocol with a multidisciplinary team including researchers, health professionals, and health sciences librarians, and (2) citizen involvement in its design and interpretation. These elements helped to create a comprehensive synthesis and discussion of the existing literature on measuring and evaluating citizen engagement in health research.
Our study also has limitations. A significant number of the methods we summarize in this scoping review are focus groups, interviews, and closed- and open-ended discussions and questions. These methods were often described in the literature with varying levels of detail, presenting difficulty in assessing the rigour of each method. While the level of detail available on included methods is variable, we do not perceive this as a limitation but rather an accurate snapshot of the currently utilized discussion-based methods to appraise citizen engagement in research. Another limitation to this study is possible unintended omittance of relevant literature due to (1) our definition of citizen engagement adapted from the CIHR , which may not align with all citizen engagement activities reported in the international literature, and (2) our approach to including only studies which discussed a method of measuring or evaluating citizen engagement as a major aim of the work. We recognize that as a scoping review designed to provide a high-level mapping of the literature, our search strategy will likely have missed some studies. Thirdly, we only searched and included peer-reviewed literature (i.e., omitted grey literature) around methods to measure and/or evaluate citizen engagement in health research to capture studies with higher methodological quality and minimize surplus complexity in the results. Lastly, like previous reviews of citizen engagement [74,75,76,77], much of the literature we captured reflects United Kingdom-based practices around citizen engagement in health research. This is due to targeted NIHR efforts to set standards for patient and public involvement  (i.e., citizen engagement), making the United Kingdom a leader in participatory health research. While this is a potential weakness to our study, we have clearly stated the geographical location of included studies to highlight any practices distinct to the United Kingdom, in order to avoid misrepresenting worldwide citizen engagement practices.
While there has been an increase in published methods to measure and evaluate citizen engagement over the past decade, there remains a need for standardized guidelines on appraising citizen engagement in research. Extensive variation in terminology used around citizen engagement contributes to a lack of unified principles or criteria that comprise effective citizen engagement and development of a single set of core principles that indicate degree (i.e., measurement) and quality (i.e., evaluation) of citizen engagement is necessary. This set of principles could be impactful if further developed as guidelines to suit specific types of research (e.g., clinical, health services, pre-clinical) and varying audiences (i.e., citizens, patients, researchers, other stakeholders). Commitment to citizen engagement in research by funding bodies, research institutions, and scientific journals could create a shift in research culture promoting use of standardized practices, helping citizen engagement move away from tokenism into an efficient and unified process.
We recommend standardization of terminology (i.e., citizen engagement rather than a multitude of other terms) used to describe participation of lay individuals in health research.
We recommend development of a specific framework for the measurement and evaluation of citizen engagement in health research, built to foster empowerment, impact, respect, support, and value in citizen engagement.
Availability of data and materials
Russell J, Fudge N, Greenhalgh T. The impact of public involvement in health research: what are we measuring? Why are we measuring it? Should we stop measuring it? Res Involv Engagem. 2020;6:63.
Research CIoH. CIHR background: Government of Canada; 2019. https://letstalk-cihr.ca/about.
Canadian Institutes of Health Research. CIHR’s Framework for Citizen Engagement. Ottawa, ON; 2012.
Research CIoH. Health Research in Canada and You Ottawa, Canada2015 [updated 2015-01-12. https://cihr-irsc.gc.ca/e/43753.html.
The Value of Engagement Washington, DC: Patient-Centered Outcomes Research Institute; updated October 30, 2018. https://www.pcori.org/engagement/value-engagement.
Minogue V, Cooke M, Donskoy A-L, Vicary P, Wells B. Patient and public involvement in reducing health and care research waste. Res Involv Engagem. 2018;4(1):5.
Research NIfH. Going the extra mile: Improving the nation’s health and wellbeing through public involvement in research.
INVOLVE. Exploring Impact: Public involvement in NHS, public health and social care research. 2009.
Research NIfH. A brief guide to public involvement in funding applications 2020. 1.0. https://www.nihr.ac.uk/documents/a-brief-guide-to-public-involvement-in-funding-applications/24162.
NIH National Heart L, and Blood Institute. Community engagement builds needed trust in science and scientists: National Institutes for Health; 2020. https://www.nhlbi.nih.gov/news/2020/community-engagement-builds-needed-trust-science-and-scientists.
Rose D, Fleischmann P, Tonkiss F, Campbell P, Wykes T. User and carer involvement in change management in a mental health context: review of the literature. Report to the National Co-ordinating Centre for NHS Service Delivery and Organisation R & D; 2002.
Dyer S. Rationalising public participation in the health service: the case of research ethics committees. Health Place. 2004;10(4):339–48.
Carrick R, Mitchell A, Lloyd K. User involvement in research: power and compromise. J Community Appl Soc Psychol. 2001;11(3):217–25.
Duffett L. Patient engagement: what partnering with patient in research is all about. Thromb Res. 2017;150:113–20.
Hibbard JH, Greene J. What the evidence shows about patient activation: better health outcomes and care experiences; fewer data on costs. Health Aff (Millwood). 2013;32(2):207–14.
Luna Puerta L, Bartlam B, Smith HE. Researchers’ perspectives on public involvement in health research in Singapore: the argument for a community-based approach. Health Expect. 2019;22(4):666–75.
Crockett LK, Shimmin C, Wittmeier KDM, Sibley KM. Engaging patients and the public in Health Research: experiences, perceptions and training needs among Manitoba health researchers. Res Involv Engagem. 2019;5(1):28.
Snape D, Kirkham J, Britten N, Froggatt K, Gradinger F, Lobban F, et al. Exploring perceived barriers, drivers, impacts and the need for evaluation of public involvement in health and social care research: a modified Delphi study. BMJ Open. 2014;4(6): e004943.
Menichetti J, Libreri C, Lozza E, Graffigna G. Giving patients a starring role in their own care: abibliometric analysis of the on-going literature debate. Health Expect. 2014;19:516–26.
Martinez J, Wong C, Piersol CV, Bieber DC, Perry BL, Leland NE. Stakeholder engagement in research: a scoping review of current evaluation methods. J Comp Eff Res. 2019;8(15):1327–41.
Modigh A, Sampaio F, Moberg L, Fredriksson M. The impact of patient and public involvement in health research versus healthcare: a scoping review of reviews. Health Policy. 2021;125(9):1208–21.
Collins M, Long R, Page A, Popay J, Lobban F. Using the public involvement impact assessment framework to assess the impact of public involvement in a mental health research context: a reflective case study. Health Expect. 2018;21(6):950–63.
Crocker JC, Ricci-Cabello I, Parker A, Hirst JA, Chant A, Petit-Zeman S, et al. Impact of patient and public involvement on enrolment and retention in clinical trials: systematic review and meta-analysis. BMJ. 2018;363: k4738.
Mathie EWP, Poland F, McNeilly E, Howe A, Staniszewska S, Cowe M, Munday D, Goodman C. Consumer involvement in health research: a UK scoping and survey. Int J Consum Stud. 2014;38:35–44.
Camden C, Shikako-Thomas K, Nguyen T, Graham E, Thomas A, Sprung J, et al. Engaging stakeholders in rehabilitation research: a scoping review of strategies used in partnerships and evaluation of impacts. Disabil Rehabil. 2015;37(15):1390–400.
Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.
Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5(1):69.
Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.
Shahid A, Rosgen BK, Krewulak KD, Lorenzetti DL, Foster N, Sept BG, et al. Incorporating and evaluating citizen engagement in health research: a scoping review protocol. Syst Rev. 2021;10(1):260.
Research CIoH. Health Research in Canada and You. Citizen engagement. Ottawa, Canada; 2015.
McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS peer review of electronic search strategies: 2015 guideline statement. J Clin Epidemiol. 2016;75:40–6.
Google Translate: Google.com; 2021. https://translate.google.com/.
Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8:45.
Vale C. Public involvement in systematic reviews: supplement to the briefing notes for researchers. National Institute for Health Research: INVOLVE; 2012.
Oliver K, Rees R, Brady LM, Kavanagh J, Oliver S, Thomas J. Broadening public participation in systematic reviews: a case example involving young people in two configurative reviews. Res Synth Methods. 2015;6(2):206–17.
Boote J, Baird W, Sutton A. Public involvement in the systematic review process in health and social care: a narrative review of case examples. Health Policy. 2011;102(2–3):105–16.
Warner G, Baghdasaryan Z, Osman F, Lampa E, Sarkadi A. ’I felt like a human being’—an exploratory, multi-method study of refugee involvement in the development of mental health intervention research. Health Expect. 2021;24(Suppl 1):30–9.
Brady L-M, Preston J. Evaluating the extent and impact of young people’s involvement in National Institute for Health Research (NIHR) studies: an assessment of feasibility. 2017.
Costello W, Dorris E. Laying the groundwork: Building relationships for public and patient involvement in pre-clinical paediatric research. Health Expect. 2020;23(1):96–105.
Lindenmeyer A, Hearnshaw H, Sturt J, Ormerod R, Aitchison G. Assessment of the benefits of user involvement in health research from the Warwick Diabetes Care Research User Group: a qualitative case study. Health Expect. 2007;10(3):268–77.
Abma TA, Pittens CA, Visse M, Elberse JE, Broerse JE. Patient involvement in research programming and implementation: a responsive evaluation of the dialogue model for research agenda setting. Health Expect. 2015;18(6):2449–64.
Greenhalgh T, Hinton L, Finlay T, Macfarlane A, Fahy N, Clyde B, et al. Frameworks for supporting patient and public involvement in research: systematic review and co-design pilot. Health Expect. 2019;22(4):785–801.
Howe A, Delaney S, Romero J, Tinsley A, Vicary P. Public involvement in health research: a case study of one NHS project over 5 years. Primary Health Care Res Dev. 2010;11(1):17–28.
Boote JD, Twiddy M, Baird W, Birks Y, Clarke C, Beever D. Supporting public involvement in research design and grant development: a case study of a public involvement award scheme managed by a National Institute for Health Research (NIHR) Research Design Service (RDS). Health Expect. 2015;18(5):1481–93.
Brutt AL, Meister R, Bernges T, Moritz S, Harter M, Kriston L, et al. Patient involvement in a systematic review: development and pilot evaluation of a patient workshop. Z Evid Fortbild Qual Gesundhwes. 2017;127–128:56–61.
Johnson H, Ogden M, Brighton LJ, Etkind SN, Oluyase AO, Chukwusa E, et al. Patient and public involvement in palliative care research: what works, and why? A qualitative evaluation. Palliat Med. 2021;35(1):151–60.
Stocks SJ, Giles SJ, Cheraghi-Sohi S, Campbell SM. Application of a tool for the evaluation of public and patient involvement in research. BMJ Open. 2015;5(3): e006390.
Wyatt K, Carter M, Mahtani V, Barnard A, Hawton A, Britten N. The impact of consumer involvement in research: an evaluation of consumer involvement in the London Primary Care Studies Programme. Fam Pract. 2008;25(3):154–61.
Seeralan T, Harter M, Koschnitzke C, Scholl M, Kohlmann S, Lehmann M, et al. Patient involvement in developing a patient-targeted feedback intervention after depression screening in primary care within the randomized controlled trial GET.FEEDBACK.GP. Health Expect. 2021;24(Suppl 1):95–112.
Vat LE, Warren M, Goold S, Davidge EB, Porter N, Schuitmaker-Warnaar TJ, et al. Giving patients a voice: a participatory evaluation of patient engagement in Newfoundland and Labrador Health Research. Res Involv Engagem. 2020;6:39.
Hanley B, Truesdale A, King A, Elbourne D, Chalmers I. Involving consumers in designing, conducting, and interpreting randomised controlled trials: questionnaire survey. BMJ. 2001;322(7285):519–23.
Shikako-Thomas K, Gonzales M, Weiss J, Putterman C, Majnemer A, editors. Measuring patient engagement in a national research network. American Academy for Cerebral Palsy and Developmental Medicine; 2020: Developmental Medicine & Child Neurology.
Giebel C, Hassan S, McIntyre JC, Corcoran R, Barr B, Gabbay M, et al. Public involvement in the dissemination of the North West Coast Household Health Survey: experiences and lessons of co-producing research together. Health Expect. 2019;22(4):643–9.
Archana S, Karmacharya BM, Rashmi M, Abhinav V, Meghnath D, Natalia O, et al. Stakeholder engagement in planning the design of a national needs assessment for cardiovascular disease prevention and management in Nepal. Glob Heart. 2019;14(2):181–9.
Oliver SR, Rees RW, Clarke-Jones L, Milne R, Oakley AR, Gabbay J, et al. A multidimensional conceptual framework for analysing public involvement in health services research. Health Expect. 2008;11(1):72–84.
Wright D, Foster C, Amir Z, Elliott J, Wilson R. Critical appraisal guidelines for assessing the quality and impact of user involvement in research. Health Expect. 2010;13(4):359–68.
Payne JM, D’Antoine HA, France KE, McKenzie AE, Henley N, Bartu AE, et al. Collaborating with consumer and community representatives in health and medical research in Australia: results from an evaluation. Health Res Policy Syst. 2011;9:18.
Gibson A, Welsman J, Britten N. Evaluating patient and public involvement in health research: from theoretical model to practical workshop. Health Expect. 2017;20(5):826–35.
Maccarthy J, Guerin S, Wilson AG, Dorris ER. Facilitating public and patient involvement in basic and preclinical health research. PLoS ONE. 2019;14(5): e0216600.
Greer AM, Amlani A, Pauly B, Burmeister C, Buxton JA. Participant, peer and PEEP: considerations and strategies for involving people who have used illicit substances as assistants and advisors in research. BMC Public Health. 2018;18(1):834.
Jewell A, Pritchard M, Barrett K, Green P, Markham S, McKenzie S, et al. The Maudsley Biomedical Research Centre (BRC) data linkage service user and carer advisory group: creating and sustaining a successful patient and public involvement group to guide research in a complex area. Res Involv Engagem. 2019;5:20.
Meyrick J, Gray D. Evidence-based patient/public voice: a patient and public involvement audit in the field of sexual health. BMJ Sex Reprod Health. 2018;44(4):267–71.
Crossing SA, Hunt L, Leigh L. Consumer involvement in research (CIR): does it work? An evaluation of cancer voices’ CIR Program. Milsons Point: Cancer Voices NSW Inc; 2016.
Boivin A, L’Esperance A, Gauvin FP, Dumez V, Macaulay AC, Lehoux P, et al. Patient and public engagement in research and health system decision making: a systematic review of evaluation tools. Health Expect. 2018;21(6):1075–84.
Pelletier C, Pousette A, Fox G, Keahey R, Ward K, Faulkner G, et al. Move the north: evaluation of a regional stakeholder engagement initiative to support the development of a community-partnered physical activity research agenda. Res Involv Engagem. 2019;5:37.
Arora PG, Krumholz LS, Guerra T, Leff SS. Measuring community-based participatory research partnerships: the initial development of an assessment instrument. Prog Community Health Partnersh. 2015;9(4):549–60.
Morrow E, Ross F, Grocott P, Bennet J. A model and measure for quality service user involvement in health research. Int J Consum Stud. 2010;34(5):532–9.
Joosten YA, Israel TL, Head A, Vaughn Y, Villalta Gil V, Mouton C, et al. Enhancing translational researchers’ ability to collaborate with community stakeholders: lessons from the Community Engagement Studio. J Clin Transl Sci. 2018;2(4):201–7.
Abelson J, Li K, Wilson G, Shields K, Schneider C, Boesveld S. Supporting quality public and patient engagement in health system organizations: development and usability testing of the Public and Patient Engagement Evaluation Tool. Health Expect. 2016;19(4):817–27.
Esmail L, Moore E, Rein A. Evaluating patient and stakeholder engagement in research: moving from theory to practice. J Comp Eff Res. 2015;4(2):133–45.
Boivin A, Richards T, Forsythe L, Gregoire A, L’Esperance A, Abelson J, et al. Evaluating patient and public involvement in research. BMJ. 2018;363: k5147.
Edelman N, Barron D. Evaluation of public involvement in research: time for a major re-think? J Health Serv Res Policy. 2016;21(3):209–11.
Manafo E, Petermann L, Vandall-Walker V, Mason-Lai P. Patient and public engagement in priority setting: a systematic rapid review of the literature. PLoS ONE. 2018;13(3): e0193579.
Miah J, Dawes P, Edwards S, Leroi I, Starling B, Parsons S. Patient and public involvement in dementia research in the European Union: a scoping review. BMC Geriatr. 2019;19(1):220.
Modigh A, Sampaio F, Moberg L, Fredriksson M. The impact of patient and public involvement in health research versus healthcare: a scoping review of reviews. Health Policy. 2021.
Flynn R, Walton S, Scott SD. Engaging children and families in pediatric Health Research: a scoping review. Res Involv Engagem. 2019;5:32.
Bird M, Ouellette C, Whitmore C, Li L, Nair K, McGillion MH, et al. Preparing for patient partnership: a scoping review of patient partner engagement and evaluation in research. Health Expect. 2020;23(3):523–39.
National Institute for Health Research CSO, Health and Care Research Wales, HSC Public Health Agency. UK Standards for Public Involvement. In: Dissemination NCfEa, editor. Wales, United Kingdom: UK Public Involvement Standards Development Partnership 2019.
A model framework for consumer and community participation in health and medical research. In: Council NHaMR, editor. Canberra, Australia; 2005. p. 1–59.
We would like to acknowledge Diane Lorenzetti for helping to develop the search strategy utilized in this scoping review. Additionally, we would like to acknowledge team members Kara Plotnikoff, Dr. Karla Krewulak, Laura Hernandez, Kira Makuk, Melanie Anglin, and Madison Leia for their contributions in the title and abstract screening of this scoping review. We would like to acknowledge Regan King for her contribution to the title and abstract screening and full-text screening of this scoping review.
This study is unfunded.
Ethics approval and consent to participate
Consent to publish
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Shahid, A., Lalani, I.N., Rosgen, B.K. et al. A scoping review of methods to measure and evaluate citizen engagement in health research. Res Involv Engagem 8, 72 (2022). https://doi.org/10.1186/s40900-022-00405-2