Skip to main content
Download PDF
Download ePub

A scoping review of methods to measure and evaluate citizen engagement in health research

Research Involvement and Engagement volume 8, Article number: 72 (2022) Cite this article

Abstract

Background

Citizen engagement, or partnering with interested members of the public in health research, is becoming more common. While ongoing assessment of citizen engagement practices is considered important to its success, there is little clarity around aspects of citizen engagement that are important to assess (i.e., what to look for) and methods to assess (i.e., how to measure and/ or evaluate) citizen engagement in health research.

Methods

In this scoping review, we included peer-reviewed literature that focused primarily on method(s) to measure and/or evaluate citizen engagement in health research. Independently and in duplicate, we completed title and abstract screening and full-text screening and extracted data including document characteristics, citizen engagement definitions and goals, and methods to measure or evaluate citizen engagement (including characteristics of these methods).

Results

Our search yielded 16,762 records of which 33 records (31 peer-reviewed articles, one government report, one conference proceeding) met our inclusion criteria. Studies discussed engaging citizens (i.e., patients [n = 16], members of the public [n = 7], service users/consumers [n = 4], individuals from specific disease groups [n = 3]) in research processes. Reported methods of citizen engagement measurement and evaluation included frameworks, discussion-based methods (i.e., focus groups, interviews), survey-based methods (e.g., audits, questionnaires), and other methods (e.g., observation, prioritization tasks). Methods to measure and evaluate citizen engagement commonly focused on collecting perceptions of citizens and researchers on aspects of citizen engagement including empowerment, impact, respect, support, and value.

Discussion and conclusion

We found that methods to measure and/or evaluate citizen engagement in health research vary widely but share some similarities in aspect of citizen engagement considered important to measure or evaluate. These aspects could be used to devise a more standardized, modifiable, and widely applicable framework for measuring and evaluating citizen engagement in research.

Patient or public contribution

Two citizen team members were involved as equal partners in study design and interpretation of its findings.

Systematic review registration

Open Science Framework (10.17605/OSF.IO/HZCBR).

Plain English Summary

Involving members of the public (citizens) in health research is important. It helps make sure that research focuses on issues that are most important to citizens. It also helps ensure that the research done is respectful of citizen participation and most likely to provide benefit. However, the best way to engage citizens in research is unclear. In this scoping review, we examined existing studies that assessed citizen engagement in health research. We found that citizen engagement was often assessed by asking for feedback from both citizens and researchers. Feedback was collected in person (one on one interviews or group discussions) or in writing (using surveys or audits). Frameworks (organized ways of thinking about an issue) were also sometimes used to measure empowerment, impact, respect, support, and value of engaging citizens. It was clear from the frameworks that there is a need to develop clearer roles for citizens in research. The two citizen members of our research team who helped interpret our study findings felt that a set of guidelines for citizens to help them best participate in health research needs to be developed. We believe these observations could be used to create a more standard method for assessing citizen engagement in research.

Peer Review reports

Background

Citizen engagement in health research is an increasingly common approach to conducting biomedical, clinical, health system and services, social, cultural, environmental, and population health research with citizens as collaborators rather than subjects [1, 2]. Often referred to using diverse terminology (e.g., community based participatory research, public participation, patient and public involvement), citizen engagement recognizes citizens, defined as interested or affected members of the general public including patients, caregivers, advocates, or representatives of the community as “knowledge users”, or individuals who are affected by the processes and results of health research and can use their lived experience to influence research to be more relevant and useful [3, 4]. Specifically, citizen engagement encompasses meaningful involvement of citizens in various aspects of the health research process such as: membership in advisory groups or steering committees for priority-setting, co-application on funding grants, and research planning, decision-making, conduct, implementation, evaluation, and dissemination [3,4,5].

Engaging citizens in research has the potential to improve the relevance of study findings, minimize waste by facilitating stewardship over resources, create mutual learning and understanding, and build trust in research findings by improving relationships between communities and researchers [6, 7]. Additionally, citizen engagement has shown the ability to provide individuals with opportunities to acquire new skills and knowledge, enjoyment and satisfaction through support and friendship, and financial rewards to compensate their efforts [8]. Due to these documented benefits of citizen engagement, national funding bodies and healthcare organizations worldwide encourage and sometimes mandate citizen engagement in research design and practice [9, 10].

Despite the push towards incorporating citizen engagement in research by funding bodies, citizen engagement is often tokenistic and lacks the clarity and guidance needed to facilitate it’s meaningful use [11,12,13,14,15,16,17,18,19]. Existing guidance on citizen engagement is often provided within a “stakeholder engagement” context which is not specific to citizens and can include health care practitioners, policymakers and industry members and may not directly address the needs of citizens [20]. Finally, literature around citizen engagement tends to focus reiterating benefits, risks, and impact of citizen engagement in health research without detailing specifics on how to appraise citizen engagement in health research [1, 21,22,23,24,25]. As such, there is need for an evidentiary foundation to enable assessment of the degree (i.e., level of engagement in research processes which can vary from participation in research planning committees to recruitment of participants and dissemination of data) and quality (i.e., determining quality of involvement, which may be ascertained collecting citizens’ experiences with the engagement or perceived impact of engagement) of citizen engagement in health research, building upon current guidance provided by national funding agencies and peer-reviewed literature. To develop a high-level understanding of methodology used to appraise citizen engagement in health research and determine aspects of citizen engagement valuable to assess, we conducted a scoping review of literature focused on:

  1. (1)

    Methods to measure (i.e., determine degree of) citizen engagement in health research; and

  2. (2)

    Methods to evaluate (i.e., determine quality of) citizen engagement in health research.

Methods

We designed and conducted a scoping review to map the existing literature on methods to measure and evaluate citizen engagement in health research according to the Arksey and O’Malley [26] and Levac [27] recommendations for scoping reviews. We used the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) to guide the reporting of this scoping review [28]. A detailed description of the proposed methods has previously been published [29].

Identification of the research question

As degree and quality of engagement are closely related features of citizen engagement, we developed the research question: “what is the state of knowledge on methods to measure and evaluate citizen engagement in health research?” to capture a broad range of potentially relevant literature. Our research question was developed to also shed light on the any relationship between measurement and evaluation of citizen engagement. We ensured that our research question defined the scope of inquiry with respect to population, concept, and outcomes of interest and would direct the subsequent steps [26].

We defined our target population as “citizens”, or consumers of health services (e.g., patients, families of patients, informal caregivers), advocates and representatives from community organizations, and members of the general public. Our target concept was “engagement”, “involvement”, or “participation” in health research. We used the CIHR definition of health research, which encompasses the biomedical, clinical, health systems and services, social health, cultural health, environmental health, and population health fields [30]. To complement our usage of the CIHR definition of health research with discussion around citizen engagement, we adopted the CIHR definition of citizen engagement or “the meaningful involvement of individual citizens…that is interactive and iterative with an aim to share decision-making power and responsibility for those decisions”. This definition encompassed activities such as priority-setting, planning, acquiring funding, research decision-making, research conduct (e.g., commenting on and developing research materials, interacting with research participants, and/or carrying out research activities), implementation, evaluation, or dissemination to be means of engagement [3].

Identification of relevant studies

We identified relevant literature on citizen engagement in research using a pre-determined plan for data sources and search strategy, including search terms, languages, and dates of search. As per recommendations, we designed the search strategy to return reasonably relevant results while considering time and personnel workload as limiting factors [26, 27].

Search strategy

Our study team, which included multiple stakeholders and knowledge users including health services researchers (KMF, HTS, JPL), trainees (AS, BKR), patient partners (BGS, SL), a health care professional (HTS), and a health sciences research librarian developed the search strategy. The search strategy (Additional file 1: Item S1) was independently reviewed by a second health sciences research librarian uninvolved with this project using the Peer Review of Electronic Search Strategies (PRESS) checklist [31].

Using the previously published search strategy [29], we searched the MEDLINE (Ovid) database from January 1, 2000 to February 1, 2021. We then adapted this strategy to each database to be searched: EMBASE (Elsevier) (January 1, 2000–February 1, 2021), Cochrane Library (Cochrane) (January 1, 2000–February 1, 2021), CINAHL Plus with full-text (Ebscohost) (January 1, 2000–February 1, 2021), APA PsycINFO (ProQuest) (January 1, 2000 – February 1, 2021), Scopus (Elsevier) (January 1, 2000–February 1, 2021), and Web of Science Core Collection (Clarivate) (January 1, 2000–February 1, 2021). We included subject headings, keywords and relevant synonyms related to three concepts: [1] citizens (e.g., community member, lay person, public, stakeholder), (2) engagement (e.g., collaboration, engagement, participation, involvement), and (3) health research (e.g., biomedical research, clinical research, public health research, environmental health research). We excluded studies published before the year 2000 to capture a modern viewpoint of the ever-evolving practice of citizen engagement in health research. We did not place any exclusion criteria on language. We screened the reference lists of included studies and related systematic reviews to identify additional potentially relevant literature.

Study selection

We developed inclusion and exclusion criteria a priori through meetings with the study team to refine the study selection process at the beginning, midpoint, and endpoint of the citation screening process in case any unforeseen considerations arose [27]. We screened and selected relevant studies for inclusion in the scoping review independently and in duplicate.

Eligibility criteria

We included articles if they: (1) were primary (e.g., observational or interventional studies) or secondary (e.g., systematic or scoping reviews) research, frameworks, reviews, or reports, (2) primarily focused on citizen engagement in health research (including biomedical, clinical, health systems and services, and social, cultural, environmental and population health, as defined by CIHR) and (3) reported method(s) to measure or evaluate citizen engagement. We did not place any restrictions on language. Non-English language studies were screened using Google Translate [32]. We excluded any literature that discussed methods to measure or evaluate citizen engagement in non-research processes, including health promotion, health education, health system or service delivery and governance (including decision-making), or health program implementation. To gain a focused insight on existing methods determining the degree and quality of citizen engagement in health research, we omitted literature focusing primarily on areas other than measurement and evaluation of citizen engagement in health research.

We imported retrieved articles into Covidence (Veritas Health Innovation, Melbourne, Australia) for title & abstract screening, which was completed independently and in duplicate by two reviewers (AS, BKR, KP, KK, RK, MA, ML, LH, KM). Reviewers conducted a pilot screening of 50 titles and abstracts to ensure consistency in application of inclusion and exclusion criteria. Once a Kappa (inter-rater agreement) of ≥ 0.8 was achieved, reviewers proceeded to screen the remaining articles. If one reviewer indicated an article as potentially relevant at the title & abstract screening phase, the article advanced to full-text review to ensure inclusivity. Following title & abstract screening, we exported a list of included articles into Endnote X9 (Clarivate Analytics, London, United Kingdom). We retrieved full-text versions of included articles using a combination of Endnote X9’s ‘find full text’ feature, Endnote Click online, and the local university online libraries. If a full-text version of an article was not available, a search was conducted of the publishing journal’s website to gain access. Following the search for full-text articles, the Endnote library was re-uploaded to Covidence and full-text screening was completed independently and in duplicate by two reviewers. Reviewers (AS, IL, RK) pilot screened the full text of 20 articles, and screened the remaining articles once a Kappa ≥ 0.8 was achieved. Both reviewers agreed on inclusion status and reason for exclusion at this stage. Any disagreements were resolved by discussion among the reviewers or the involvement of a third reviewer (IL or RK), if required.

Charting the data

We developed an initial data charting form for data elements to be abstracted from the articles, then refined the form through discussion with the study team. The data charting form [Microsoft Excel (version 16.29.1)] was piloted by two reviewers (AS, IL) with ten included studies, and revised as needed. Once the final data charting form was developed, all relevant articles were abstracted independently and in duplicate by two reviewers (AS, IL). Abstracted variables in the initial data charting form included study characteristics, participants, type and goals of citizen engagement, method(s) used to measure citizen engagement, method(s) used to evaluate citizen engagement, features of method(s) used, and observed benefits or risks of the measurement or evaluation method(s) used.

Collating, summarizing, and reporting the results

We collated, summarized, and reported the results by (1) analyzing the data, (2) reporting results, (3) and evaluating their meaning and aimed to contextualize our findings within existing literature and research, practice, and policy [26, 27]. We analyzed qualitative data specifically by having two research team members trained in qualitative methods (AS, IL) inductively code major components of each framework using an analytical approach informed by thematic analysis [33]. First, the researchers independently reviewed each framework to develop a list of relevant terms and associated concepts. Then, the researchers compared lists and discussed discrepancies with a third qualitative expert (JPL). The initial two researchers then deductively analyzed the agreed upon concepts into shared and unique framework features by expanding and collapsing shared meanings through a series of three meetings. Concepts deemed too vague (no label/definition provided) were excluded from analysis.

We present a descriptive summary of characteristics of the included documents, and the characteristics of the intended participants or audiences for these documents alongside a narrative synthesis of abstracted data variables.

Consultation

Involvement of citizens in literature synthesis is recommended by funding organizations such as the NIHR [34] and CIHR [3] and is becoming increasingly commonplace [35, 36]. We involved citizens (BS, NF) in study conception and design including the search strategy. As per recommendations, we involved citizens (BS, SL) in the interpretation and contextualization of the data [27].

Results

Our search strategy returned 28,353 total results (16,762 results after duplicates were removed). Of these, studies were excluded because they: (1) reported aspects of citizen engagement other than methods of measurement or evaluation (n = 713), (2) reported outcomes or discussion on specific diseases/interventions (i.e., not citizen engagement) (n = 520), (3) had an ineligible study design (i.e., editorials, letters, commentaries) (n = 504), (4) included citizens as health research subjects only (n = 452), (5) engaged citizens, but in non-research (i.e., health service design or delivery) (n = 437), (6) were not health research related (n = 337), (7) focused on non-citizen stakeholders in health research (i.e., clinicians, policy makers) (n = 285), or (8) and/ or had inaccessible full-texts (n = 169). Thirty records met inclusion criteria and were included in our scoping review. Three additional records were found through searching reference lists and included for a total of 33 records. A study flow diagram including reasons for exclusion is shown in Fig. 1.

Fig. 1
figure 1

Flow diagram of included studies and reasons for exclusion

Full size image

Study characteristics

A majority of the included literature was published within the last 10 years (2011–2021, n = 26, 78.8%). Literature included peer-reviewed journal articles (n = 31, 94.0%), a report by a government organization (n = 1, 3.0%), and a conferencing proceeding (n = 1, 3.0%). Most literature reported work that was conducted in the United Kingdom (n = 16, 48.5%), Canada (n = 6, 18.2%), and other European nations (i.e., Ireland, Germany, Sweden) (n = 6, 18.2%). Most included literature pertained to health services or health research (n = 18, 54.5%), and clinical research (particularly primary care and mental health) (n = 12, 36.4%). Characteristics of the included literature are shown in Table 1.

Table 1 Characteristics of the included literature (n = 33)
Full size table

Citizen engagement: participants, engagement activities and terminology

Included records varied in type of citizens engaged, research engagement activities, and terminology used to describe the engagement process (Table 1). Most studies reported engaging patients or individuals with lived experience of a given illness (n = 16, 48.5%), members of the public (n = 7, 21.2%), service users/consumers (n = 4, 12.1%), or individuals from community groups such as refugees [37] or children [38, 39] (n = 3, 9.1%). Citizens were engaged in specific research activities such as priority-setting [40,41,42], advisory board/ steering committee membership [38,39,40, 43], grant [44] and research proposal development [42, 43], study design [42, 45,46,47,48], study materials development [43, 49, 50], study administration and conduct [40, 47, 48, 51], interpretation of findings [46, 50, 52], and dissemination [40, 42, 46, 50,51,52,53]. Other studies described citizen engagement activities more generally as informing citizens of research goals so engagement may occur [54], collaboration [55, 56], consultation [55,56,57,58,59], co-production [46, 53], co-research [60], participation [57], and user control [56]. A word-cloud depicting descriptions of citizen engagement in the included studies is shown in Fig. 2.

Fig. 2
figure 2

Word cloud of citizen engagement descriptions, activities, and key terms described in included literature

Full size image

Terminology used to describe citizen engagement activities (in order of most common to least common) was: patient and public involvement [37, 39, 47, 49, 53, 58, 59, 61, 62], public involvement [38, 42,43,44, 46, 55], consumer involvement [40, 48, 51, 63], patient and public engagement [64, 65], patient involvement [41, 45], and community-based participatory research [60, 66]. Table 1 shows specific studies, citizen engagement terminology, engagement activities, and types of citizens engaged in further detail.

Methods used to measure and/or evaluate CE

Of the 33 included studies, 20 (61.0%) presented method(s) to evaluate citizen engagement, five (15.2%) presented method(s) to measure citizen engagement, and eight (24.2%) presented method(s) to both measure and evaluate citizen engagement. Methods for the measurement and/or evaluation of citizen engagement included frameworks, discussion-based methods (i.e., focus groups, interviews, workshops), survey-based methods (i.e., audits, questionnaires), and other methods (i.e., indicators, observation, prioritization tasks). Many studies utilized and reported on more than one method to measure and/or evaluate citizen engagement. A summary of these methods is presented in Table 2 and described narratively below.

Table 2 Methods used by included literature to measure and/ or evaluate citizen engagement
Full size table

Citizen engagement strategies: frameworks

Five studies presented frameworks [55, 57, 58, 60, 67] designed to measure and/or evaluate citizen engagement in health research. Frameworks focused on various aspects of citizen engagement including reflection on and impact of citizen engagement activities in research, and recommendations for improvement. The five included frameworks explored measurement and evaluation of citizen engagement through gauging (1) empowerment (i.e., citizens should feel comfortable in voicing their opinions), (2) impact (i.e., research should be positively shaped by citizen engagement), (3) respect (i.e., citizens should feel respected), (4) support (i.e., citizens should have training and supports available), and (5) value (i.e., citizens should feel important to the process). Included frameworks also highlighted the importance of capacity building (i.e., funds, personnel to support engagement in research) [60], assessing the degree of engagement of researchers and citizens [55], clarity in roles (i.e., of citizens when engaged) [57], and involvement of citizens in critical aspects of research (i.e., protocol development, analysis, outputs) [60]. More detail on each framework is provided in Table 2, and similarities and differences between the included frameworks are highlighted in Fig. 3.

Fig. 3
figure 3

Similar and unique features of the five frameworks for measuring and/or evaluating citizen engagement included in this scoping review

Full size image

Citizen engagement strategies: discussion-based methods

Methods using discussion to measure and/or evaluate citizen engagement utilized focus groups (n = 7) [41, 43, 45, 46, 48, 49, 53], interviews (n = 5) [38, 41, 43, 48, 68], or workshops (n = 1) [58]. Most discussion-based measurement or evaluation of citizen engagement was conducted in a semi-structured manner, with pre-defined topics or question lists [38, 41, 43, 45, 46, 48, 49, 53, 68]. Common focus group discussions included experiences with and perceptions of involvement [46, 48, 53], nature and impact of involvement [45, 46, 53], and recommendations for improving involvement in research [43, 46, 53]. Common topics of discussion in interviews were experiences with involvement [38, 41], perspectives on the project in which involvement occurred [48], challenges in involvement [38, 41], and opportunities for improvement [38].

Citizen engagement strategies: survey-based methods

A number of studies used survey-based methods to measure and/or evaluate citizen engagement in health research. These methods included an audit [62], questionnaires [37, 51, 59, 66, 67, 69], and surveys [48, 50, 51, 59, 63, 65] and varied in number and type of questions asked, content of the questions, and intended recipients of the questions. Open-ended questions [44, 48, 51, 56, 61,62,63, 65, 66, 68, 69] and Likert scale-based statements [37, 48, 49, 65, 66, 68] were commonly used. Many studies used a combination of open-ended questions, closed-ended questions, and/or statements for which a degree of agreement could be declared [48, 50, 51, 59, 63, 65, 66, 68, 69]. Questions were intended for (1) citizens involved in research [43, 44, 48, 50, 52, 61, 63, 65, 67, 69], (2) researchers who involved citizens in their work [44, 51, 52, 56, 59, 62, 63, 68, 69], or less frequently (3) other research or grant administrative personnel [52, 69]. The content of many questions focused on reflections on involvement (i.e., feedback on activity or study involved in) [44, 48,49,50, 52, 59, 61, 65], motivation for involvement in research [43, 48, 56, 61], perceived impact of involvement on the research [44, 49, 51, 61, 69], and recommendations or comments on future involvement in research [43, 63, 68].

Citizen engagement strategies: other methods

A number of studies presented other methods to measure and/or evaluate citizen engagement. These included indicators of user involvement such as documentation of citizen roles in research and availability of training to citizens to facilitate their involvement in research [40], prioritization tasks focusing on outcomes of the research considered important by participating citizens [49], and citizen observation of any study steering group meetings and scrutiny of any study documentation [48]. One study used a method to appraise existing frameworks for supporting citizen engagement (the Canadian Centre for Excellence on Partnerships with Patients and Public evaluation tool) [42], however many of the frameworks discussed intended to support and report rather than measure or evaluate citizen engagement, falling out of the scope of this review.

Discussion

Our scoping review produced two main findings. First, we found that multiple methods (i.e., audits, focus groups, interviews, frameworks, surveys) have been used, often in combination, to measure and evaluate citizen engagement in health research. These methods collect perceptions of citizens, researchers, and/or research support personnel on many aspects of citizen engagement including reasons, type, and impact of engagement, any challenges encountered in engagement (including project-specific issues), and recommendations for improving future citizen engagement in health research. Secondly, we identified that existing frameworks to measure and evaluate citizen engagement commonly assess perceived empowerment, impact, respect, support, and value. Together, these findings summarize the nature of citizen engagement in health research and itemize citizen engagement aspects that are considered important to assess the degree and quality of citizen engagement in health research.

In addition to our main findings, we identified that the terminology used to define citizen engagement and describe its activities varies widely. Citizen engagement is referred to as patient and public involvement (often in the United Kingdom), patient engagement, public engagement, consumer or service user-involved research, and community based participatory research depending on location and context. Varying terminology may pose a challenge for individual researchers to identify and utilize methods to appraise citizen engagement in research. Standardization of terminology could add to the accessibility and applicability of current and future methods to incorporate and evaluate citizen engagement in health research.

In the process of screening literature for inclusion in this scoping review, we found that much of the current guidance on appraising citizen engagement in research exists in the form of editorials, letters to the editor, commentaries, and perspectives from experienced researchers in the field. We noted that this type of literature does not routinely discuss the merit of discussion-based methods that were used to evaluate citizen engagement in included studies. This could reflect a repeated dismissal of discussion-based qualitative research methods to measure and evaluate citizen engagement in research and warrants further investigation. Despite limited discussion-based methods to appraise citizen engagement, this literature emphasizes the context and process of engagement [70], clarity, reflexivity, methodological rigour, transparency, pragmatism, and reciprocity as key principles to evaluating citizen engagement in research [71] and highlights the need for evaluation as an ongoing part of the research process [72]. These elements of citizen engagement complement our main findings and should be taken into consideration when appraising citizen engagement. Our findings also align with previous work emphasizing the importance of evaluating citizen engagement activities as a necessary step in building a strong evidence base for utilizing citizen engagement in health research [73]. Furthermore, previous literature has emphasized a need for standardization in measurement and evaluation of engagement processes as methods to measure or evaluate citizen engagement are seldom utilized beyond the groups that develop them [42]. In light of our findings, we postulate this could occur due to (1) lack of accessibility (i.e., method difficult to find) or (2) lack of perceived applicability/modifiability (i.e., method viewed as unsuitable or too specific to a certain project or type of research and unmodifiable).

As per recommendations by Levac and colleagues, we invited citizen team members (BS, SL) to help interpret the findings of this scoping review and provide insights beyond those in the literature [27]. These citizen team members (BS, SL) remarked that empowerment, impact, respect, support, and value, common to frameworks identified by our study, were important to them in their experiences of participating in research. Additionally, they stated that the ability to openly communicate their concerns about the research project and their involvement has been important to them as members of a research team. Finally, they expressed a desire for an accessible lay resource to help people like them (i.e., citizens) be a meaningful part of research and stated that such a resource would vastly improve their comfort level with participating in research.

Strengths and limitations

This scoping review was designed to form an evidence basis for future work to advance and standardize appraisal of citizen engagement in health research. This study has strengths and limitations to consider. Strengths of our scoping review include: (1) co-development of the study protocol with a multidisciplinary team including researchers, health professionals, and health sciences librarians, and (2) citizen involvement in its design and interpretation. These elements helped to create a comprehensive synthesis and discussion of the existing literature on measuring and evaluating citizen engagement in health research.

Our study also has limitations. A significant number of the methods we summarize in this scoping review are focus groups, interviews, and closed- and open-ended discussions and questions. These methods were often described in the literature with varying levels of detail, presenting difficulty in assessing the rigour of each method. While the level of detail available on included methods is variable, we do not perceive this as a limitation but rather an accurate snapshot of the currently utilized discussion-based methods to appraise citizen engagement in research. Another limitation to this study is possible unintended omittance of relevant literature due to (1) our definition of citizen engagement adapted from the CIHR [3], which may not align with all citizen engagement activities reported in the international literature, and (2) our approach to including only studies which discussed a method of measuring or evaluating citizen engagement as a major aim of the work. We recognize that as a scoping review designed to provide a high-level mapping of the literature, our search strategy will likely have missed some studies. Thirdly, we only searched and included peer-reviewed literature (i.e., omitted grey literature) around methods to measure and/or evaluate citizen engagement in health research to capture studies with higher methodological quality and minimize surplus complexity in the results. Lastly, like previous reviews of citizen engagement [74,75,76,77], much of the literature we captured reflects United Kingdom-based practices around citizen engagement in health research. This is due to targeted NIHR efforts to set standards for patient and public involvement [78] (i.e., citizen engagement), making the United Kingdom a leader in participatory health research. While this is a potential weakness to our study, we have clearly stated the geographical location of included studies to highlight any practices distinct to the United Kingdom, in order to avoid misrepresenting worldwide citizen engagement practices.

Conclusions

While there has been an increase in published methods to measure and evaluate citizen engagement over the past decade, there remains a need for standardized guidelines on appraising citizen engagement in research. Extensive variation in terminology used around citizen engagement contributes to a lack of unified principles or criteria that comprise effective citizen engagement and development of a single set of core principles that indicate degree (i.e., measurement) and quality (i.e., evaluation) of citizen engagement is necessary. This set of principles could be impactful if further developed as guidelines to suit specific types of research (e.g., clinical, health services, pre-clinical) and varying audiences (i.e., citizens, patients, researchers, other stakeholders). Commitment to citizen engagement in research by funding bodies, research institutions, and scientific journals could create a shift in research culture promoting use of standardized practices, helping citizen engagement move away from tokenism into an efficient and unified process.

Recommendations

  • We recommend standardization of terminology (i.e., citizen engagement rather than a multitude of other terms) used to describe participation of lay individuals in health research.

  • We recommend development of a specific framework for the measurement and evaluation of citizen engagement in health research, built to foster empowerment, impact, respect, support, and value in citizen engagement.

Availability of data and materials

Not applicable.

References

  1. Russell J, Fudge N, Greenhalgh T. The impact of public involvement in health research: what are we measuring? Why are we measuring it? Should we stop measuring it? Res Involv Engagem. 2020;6:63.

    Article  Google Scholar 

  2. Research CIoH. CIHR background: Government of Canada; 2019. https://letstalk-cihr.ca/about.

  3. Canadian Institutes of Health Research. CIHR’s Framework for Citizen Engagement. Ottawa, ON; 2012.

  4. Research CIoH. Health Research in Canada and You Ottawa, Canada2015 [updated 2015-01-12. https://cihr-irsc.gc.ca/e/43753.html.

  5. The Value of Engagement Washington, DC: Patient-Centered Outcomes Research Institute; updated October 30, 2018. https://www.pcori.org/engagement/value-engagement.

  6. Minogue V, Cooke M, Donskoy A-L, Vicary P, Wells B. Patient and public involvement in reducing health and care research waste. Res Involv Engagem. 2018;4(1):5.

    Article  Google Scholar 

  7. Research NIfH. Going the extra mile: Improving the nation’s health and wellbeing through public involvement in research.

  8. INVOLVE. Exploring Impact: Public involvement in NHS, public health and social care research. 2009.

  9. Research NIfH. A brief guide to public involvement in funding applications 2020. 1.0. https://www.nihr.ac.uk/documents/a-brief-guide-to-public-involvement-in-funding-applications/24162.

  10. NIH National Heart L, and Blood Institute. Community engagement builds needed trust in science and scientists: National Institutes for Health; 2020. https://www.nhlbi.nih.gov/news/2020/community-engagement-builds-needed-trust-science-and-scientists.

  11. Rose D, Fleischmann P, Tonkiss F, Campbell P, Wykes T. User and carer involvement in change management in a mental health context: review of the literature. Report to the National Co-ordinating Centre for NHS Service Delivery and Organisation R & D; 2002.

  12. Dyer S. Rationalising public participation in the health service: the case of research ethics committees. Health Place. 2004;10(4):339–48.

    Article  Google Scholar 

  13. Carrick R, Mitchell A, Lloyd K. User involvement in research: power and compromise. J Community Appl Soc Psychol. 2001;11(3):217–25.

    Article  Google Scholar 

  14. Duffett L. Patient engagement: what partnering with patient in research is all about. Thromb Res. 2017;150:113–20.

    Article  CAS  Google Scholar 

  15. Hibbard JH, Greene J. What the evidence shows about patient activation: better health outcomes and care experiences; fewer data on costs. Health Aff (Millwood). 2013;32(2):207–14.

    Article  Google Scholar 

  16. Luna Puerta L, Bartlam B, Smith HE. Researchers’ perspectives on public involvement in health research in Singapore: the argument for a community-based approach. Health Expect. 2019;22(4):666–75.

    Article  Google Scholar 

  17. Crockett LK, Shimmin C, Wittmeier KDM, Sibley KM. Engaging patients and the public in Health Research: experiences, perceptions and training needs among Manitoba health researchers. Res Involv Engagem. 2019;5(1):28.

    Article  Google Scholar 

  18. Snape D, Kirkham J, Britten N, Froggatt K, Gradinger F, Lobban F, et al. Exploring perceived barriers, drivers, impacts and the need for evaluation of public involvement in health and social care research: a modified Delphi study. BMJ Open. 2014;4(6): e004943.

    Article  CAS  Google Scholar 

  19. Menichetti J, Libreri C, Lozza E, Graffigna G. Giving patients a starring role in their own care: abibliometric analysis of the on-going literature debate. Health Expect. 2014;19:516–26.

    Article  Google Scholar 

  20. Martinez J, Wong C, Piersol CV, Bieber DC, Perry BL, Leland NE. Stakeholder engagement in research: a scoping review of current evaluation methods. J Comp Eff Res. 2019;8(15):1327–41.

    Article  Google Scholar 

  21. Modigh A, Sampaio F, Moberg L, Fredriksson M. The impact of patient and public involvement in health research versus healthcare: a scoping review of reviews. Health Policy. 2021;125(9):1208–21.

    Article  Google Scholar 

  22. Collins M, Long R, Page A, Popay J, Lobban F. Using the public involvement impact assessment framework to assess the impact of public involvement in a mental health research context: a reflective case study. Health Expect. 2018;21(6):950–63.

    Article  Google Scholar 

  23. Crocker JC, Ricci-Cabello I, Parker A, Hirst JA, Chant A, Petit-Zeman S, et al. Impact of patient and public involvement on enrolment and retention in clinical trials: systematic review and meta-analysis. BMJ. 2018;363: k4738.

    Article  Google Scholar 

  24. Mathie EWP, Poland F, McNeilly E, Howe A, Staniszewska S, Cowe M, Munday D, Goodman C. Consumer involvement in health research: a UK scoping and survey. Int J Consum Stud. 2014;38:35–44.

    Article  Google Scholar 

  25. Camden C, Shikako-Thomas K, Nguyen T, Graham E, Thomas A, Sprung J, et al. Engaging stakeholders in rehabilitation research: a scoping review of strategies used in partnerships and evaluation of impacts. Disabil Rehabil. 2015;37(15):1390–400.

    Article  Google Scholar 

  26. Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.

    Article  Google Scholar 

  27. Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5(1):69.

    Article  Google Scholar 

  28. Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

    Article  Google Scholar 

  29. Shahid A, Rosgen BK, Krewulak KD, Lorenzetti DL, Foster N, Sept BG, et al. Incorporating and evaluating citizen engagement in health research: a scoping review protocol. Syst Rev. 2021;10(1):260.

    Article  Google Scholar 

  30. Research CIoH. Health Research in Canada and You. Citizen engagement. Ottawa, Canada; 2015.

  31. McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS peer review of electronic search strategies: 2015 guideline statement. J Clin Epidemiol. 2016;75:40–6.

    Article  Google Scholar 

  32. Google Translate: Google.com; 2021. https://translate.google.com/.

  33. Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8:45.

    Article  Google Scholar 

  34. Vale C. Public involvement in systematic reviews: supplement to the briefing notes for researchers. National Institute for Health Research: INVOLVE; 2012.

  35. Oliver K, Rees R, Brady LM, Kavanagh J, Oliver S, Thomas J. Broadening public participation in systematic reviews: a case example involving young people in two configurative reviews. Res Synth Methods. 2015;6(2):206–17.

    Article  Google Scholar 

  36. Boote J, Baird W, Sutton A. Public involvement in the systematic review process in health and social care: a narrative review of case examples. Health Policy. 2011;102(2–3):105–16.

    Article  Google Scholar 

  37. Warner G, Baghdasaryan Z, Osman F, Lampa E, Sarkadi A. ’I felt like a human being’—an exploratory, multi-method study of refugee involvement in the development of mental health intervention research. Health Expect. 2021;24(Suppl 1):30–9.

    Article  Google Scholar 

  38. Brady L-M, Preston J. Evaluating the extent and impact of young people’s involvement in National Institute for Health Research (NIHR) studies: an assessment of feasibility. 2017.

  39. Costello W, Dorris E. Laying the groundwork: Building relationships for public and patient involvement in pre-clinical paediatric research. Health Expect. 2020;23(1):96–105.

    Article  Google Scholar 

  40. Lindenmeyer A, Hearnshaw H, Sturt J, Ormerod R, Aitchison G. Assessment of the benefits of user involvement in health research from the Warwick Diabetes Care Research User Group: a qualitative case study. Health Expect. 2007;10(3):268–77.

    Article  Google Scholar 

  41. Abma TA, Pittens CA, Visse M, Elberse JE, Broerse JE. Patient involvement in research programming and implementation: a responsive evaluation of the dialogue model for research agenda setting. Health Expect. 2015;18(6):2449–64.

    Article  Google Scholar 

  42. Greenhalgh T, Hinton L, Finlay T, Macfarlane A, Fahy N, Clyde B, et al. Frameworks for supporting patient and public involvement in research: systematic review and co-design pilot. Health Expect. 2019;22(4):785–801.

    Article  Google Scholar 

  43. Howe A, Delaney S, Romero J, Tinsley A, Vicary P. Public involvement in health research: a case study of one NHS project over 5 years. Primary Health Care Res Dev. 2010;11(1):17–28.

    Article  Google Scholar 

  44. Boote JD, Twiddy M, Baird W, Birks Y, Clarke C, Beever D. Supporting public involvement in research design and grant development: a case study of a public involvement award scheme managed by a National Institute for Health Research (NIHR) Research Design Service (RDS). Health Expect. 2015;18(5):1481–93.

    Article  Google Scholar 

  45. Brutt AL, Meister R, Bernges T, Moritz S, Harter M, Kriston L, et al. Patient involvement in a systematic review: development and pilot evaluation of a patient workshop. Z Evid Fortbild Qual Gesundhwes. 2017;127–128:56–61.

    Article  Google Scholar 

  46. Johnson H, Ogden M, Brighton LJ, Etkind SN, Oluyase AO, Chukwusa E, et al. Patient and public involvement in palliative care research: what works, and why? A qualitative evaluation. Palliat Med. 2021;35(1):151–60.

    Article  Google Scholar 

  47. Stocks SJ, Giles SJ, Cheraghi-Sohi S, Campbell SM. Application of a tool for the evaluation of public and patient involvement in research. BMJ Open. 2015;5(3): e006390.

    Article  Google Scholar 

  48. Wyatt K, Carter M, Mahtani V, Barnard A, Hawton A, Britten N. The impact of consumer involvement in research: an evaluation of consumer involvement in the London Primary Care Studies Programme. Fam Pract. 2008;25(3):154–61.

    Article  Google Scholar 

  49. Seeralan T, Harter M, Koschnitzke C, Scholl M, Kohlmann S, Lehmann M, et al. Patient involvement in developing a patient-targeted feedback intervention after depression screening in primary care within the randomized controlled trial GET.FEEDBACK.GP. Health Expect. 2021;24(Suppl 1):95–112.

    Article  Google Scholar 

  50. Vat LE, Warren M, Goold S, Davidge EB, Porter N, Schuitmaker-Warnaar TJ, et al. Giving patients a voice: a participatory evaluation of patient engagement in Newfoundland and Labrador Health Research. Res Involv Engagem. 2020;6:39.

    Article  Google Scholar 

  51. Hanley B, Truesdale A, King A, Elbourne D, Chalmers I. Involving consumers in designing, conducting, and interpreting randomised controlled trials: questionnaire survey. BMJ. 2001;322(7285):519–23.

    Article  CAS  Google Scholar 

  52. Shikako-Thomas K, Gonzales M, Weiss J, Putterman C, Majnemer A, editors. Measuring patient engagement in a national research network. American Academy for Cerebral Palsy and Developmental Medicine; 2020: Developmental Medicine & Child Neurology.

  53. Giebel C, Hassan S, McIntyre JC, Corcoran R, Barr B, Gabbay M, et al. Public involvement in the dissemination of the North West Coast Household Health Survey: experiences and lessons of co-producing research together. Health Expect. 2019;22(4):643–9.

    Article  Google Scholar 

  54. Archana S, Karmacharya BM, Rashmi M, Abhinav V, Meghnath D, Natalia O, et al. Stakeholder engagement in planning the design of a national needs assessment for cardiovascular disease prevention and management in Nepal. Glob Heart. 2019;14(2):181–9.

    Article  Google Scholar 

  55. Oliver SR, Rees RW, Clarke-Jones L, Milne R, Oakley AR, Gabbay J, et al. A multidimensional conceptual framework for analysing public involvement in health services research. Health Expect. 2008;11(1):72–84.

    Article  Google Scholar 

  56. Wright D, Foster C, Amir Z, Elliott J, Wilson R. Critical appraisal guidelines for assessing the quality and impact of user involvement in research. Health Expect. 2010;13(4):359–68.

    Article  Google Scholar 

  57. Payne JM, D’Antoine HA, France KE, McKenzie AE, Henley N, Bartu AE, et al. Collaborating with consumer and community representatives in health and medical research in Australia: results from an evaluation. Health Res Policy Syst. 2011;9:18.

    Article  Google Scholar 

  58. Gibson A, Welsman J, Britten N. Evaluating patient and public involvement in health research: from theoretical model to practical workshop. Health Expect. 2017;20(5):826–35.

    Article  Google Scholar 

  59. Maccarthy J, Guerin S, Wilson AG, Dorris ER. Facilitating public and patient involvement in basic and preclinical health research. PLoS ONE. 2019;14(5): e0216600.

    Article  CAS  Google Scholar 

  60. Greer AM, Amlani A, Pauly B, Burmeister C, Buxton JA. Participant, peer and PEEP: considerations and strategies for involving people who have used illicit substances as assistants and advisors in research. BMC Public Health. 2018;18(1):834.

    Article  Google Scholar 

  61. Jewell A, Pritchard M, Barrett K, Green P, Markham S, McKenzie S, et al. The Maudsley Biomedical Research Centre (BRC) data linkage service user and carer advisory group: creating and sustaining a successful patient and public involvement group to guide research in a complex area. Res Involv Engagem. 2019;5:20.

    Article  Google Scholar 

  62. Meyrick J, Gray D. Evidence-based patient/public voice: a patient and public involvement audit in the field of sexual health. BMJ Sex Reprod Health. 2018;44(4):267–71.

    Article  Google Scholar 

  63. Crossing SA, Hunt L, Leigh L. Consumer involvement in research (CIR): does it work? An evaluation of cancer voices’ CIR Program. Milsons Point: Cancer Voices NSW Inc; 2016.

    Google Scholar 

  64. Boivin A, L’Esperance A, Gauvin FP, Dumez V, Macaulay AC, Lehoux P, et al. Patient and public engagement in research and health system decision making: a systematic review of evaluation tools. Health Expect. 2018;21(6):1075–84.

    Article  Google Scholar 

  65. Pelletier C, Pousette A, Fox G, Keahey R, Ward K, Faulkner G, et al. Move the north: evaluation of a regional stakeholder engagement initiative to support the development of a community-partnered physical activity research agenda. Res Involv Engagem. 2019;5:37.

    Article  Google Scholar 

  66. Arora PG, Krumholz LS, Guerra T, Leff SS. Measuring community-based participatory research partnerships: the initial development of an assessment instrument. Prog Community Health Partnersh. 2015;9(4):549–60.

    Article  Google Scholar 

  67. Morrow E, Ross F, Grocott P, Bennet J. A model and measure for quality service user involvement in health research. Int J Consum Stud. 2010;34(5):532–9.

    Article  Google Scholar 

  68. Joosten YA, Israel TL, Head A, Vaughn Y, Villalta Gil V, Mouton C, et al. Enhancing translational researchers’ ability to collaborate with community stakeholders: lessons from the Community Engagement Studio. J Clin Transl Sci. 2018;2(4):201–7.

    Article  Google Scholar 

  69. Abelson J, Li K, Wilson G, Shields K, Schneider C, Boesveld S. Supporting quality public and patient engagement in health system organizations: development and usability testing of the Public and Patient Engagement Evaluation Tool. Health Expect. 2016;19(4):817–27.

    Article  Google Scholar 

  70. Esmail L, Moore E, Rein A. Evaluating patient and stakeholder engagement in research: moving from theory to practice. J Comp Eff Res. 2015;4(2):133–45.

    Article  Google Scholar 

  71. Boivin A, Richards T, Forsythe L, Gregoire A, L’Esperance A, Abelson J, et al. Evaluating patient and public involvement in research. BMJ. 2018;363: k5147.

    Article  Google Scholar 

  72. Edelman N, Barron D. Evaluation of public involvement in research: time for a major re-think? J Health Serv Res Policy. 2016;21(3):209–11.

    Article  Google Scholar 

  73. Manafo E, Petermann L, Vandall-Walker V, Mason-Lai P. Patient and public engagement in priority setting: a systematic rapid review of the literature. PLoS ONE. 2018;13(3): e0193579.

    Article  Google Scholar 

  74. Miah J, Dawes P, Edwards S, Leroi I, Starling B, Parsons S. Patient and public involvement in dementia research in the European Union: a scoping review. BMC Geriatr. 2019;19(1):220.

    Article  Google Scholar 

  75. Modigh A, Sampaio F, Moberg L, Fredriksson M. The impact of patient and public involvement in health research versus healthcare: a scoping review of reviews. Health Policy. 2021.

  76. Flynn R, Walton S, Scott SD. Engaging children and families in pediatric Health Research: a scoping review. Res Involv Engagem. 2019;5:32.

    Article  Google Scholar 

  77. Bird M, Ouellette C, Whitmore C, Li L, Nair K, McGillion MH, et al. Preparing for patient partnership: a scoping review of patient partner engagement and evaluation in research. Health Expect. 2020;23(3):523–39.

    Article  Google Scholar 

  78. National Institute for Health Research CSO, Health and Care Research Wales, HSC Public Health Agency. UK Standards for Public Involvement. In: Dissemination NCfEa, editor. Wales, United Kingdom: UK Public Involvement Standards Development Partnership 2019.

  79. A model framework for consumer and community participation in health and medical research. In: Council NHaMR, editor. Canberra, Australia; 2005. p. 1–59.

Download references

Acknowledgements

We would like to acknowledge Diane Lorenzetti for helping to develop the search strategy utilized in this scoping review. Additionally, we would like to acknowledge team members Kara Plotnikoff, Dr. Karla Krewulak, Laura Hernandez, Kira Makuk, Melanie Anglin, and Madison Leia for their contributions in the title and abstract screening of this scoping review. We would like to acknowledge Regan King for her contribution to the title and abstract screening and full-text screening of this scoping review.

Funding

This study is unfunded.

Author information

Authors and Affiliations

  1. Department of Critical Care Medicine, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada

    Anmol Shahid, Inara N. Lalani, Brianna K. Rosgen, Bonnie G. Sept, Shelly Longmore, Jeanna Parsons Leigh, Henry T. Stelfox & Kirsten M. Fiest

  2. Department of Community Health Science and O’Brien Institute for Public Health, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada

    Brianna K. Rosgen, Henry T. Stelfox & Kirsten M. Fiest

  3. Alberta Health Services, Calgary, AB, Canada

    Henry T. Stelfox

  4. Dalhousie University, Halifax, NS, Canada

    Jeanna Parsons Leigh

  5. Department of Psychiatry and Hotchkiss Brain Institute, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada

    Kirsten M. Fiest

Authors
  1. Anmol Shahid
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Inara N. Lalani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Brianna K. Rosgen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bonnie G. Sept
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shelly Longmore
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jeanna Parsons Leigh
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Henry T. Stelfox
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Kirsten M. Fiest
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

AS designed the study and performed the literature search, data analysis, and led the interpretation and dissemination of the data. IL performed the literature search and assisted in data analysis. BS interpreted the data from a patient partner perspective. SL interpreted the data from a patient partner perspective. JPL provided methodological expertise and participated in interpretation of data. HTS provided methodology expertise and led interpretation of data. KMF designed the study and interpreted the data. All authors have contributed to the conception and design of this study. All authors read and approved the final manuscript.

Authors' information

KMF is the Director of Research and Innovation for the Department of Critical Care Medicine and an associate professor in the Departments of Critical Care Medicine, Community Health Sciences, and Psychiatry at the University of Calgary. HTS is Scientific Director of the O’Brien Institute, a member of the Canadian Academy of Health Sciences, and a professor in the Department of Critical Care Medicine at the University of Calgary. JPL is an assistant professor in the Department of Critical Care Medicine at Dalhousie University and an adjunct professor in the Department of Critical Care Medicine at the University of Calgary.

Corresponding author

Correspondence to Henry T. Stelfox.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent to publish

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

Item S1. Medline (Ovid) search strategy.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shahid, A., Lalani, I.N., Rosgen, B.K. et al. A scoping review of methods to measure and evaluate citizen engagement in health research. Res Involv Engagem 8, 72 (2022). https://doi.org/10.1186/s40900-022-00405-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s40900-022-00405-2

Keywords

Research Involvement and Engagement

ISSN: 2056-7529

Contact us