Table 2 Methods used by included literature to measure and/ or evaluate citizen engagement

Framework

–

Four dimensional, theoretical

Weak voice/strong voice; One way to be involved/many ways to be involved

Organization’s concerns/public’s concerns; Organization changes/organization resists change

[58]

Peer Engagement Process Evaluation Framework

Four process goals

Supportive environment, equitable participation, capacity building and empowerment, involvement in protocol developments, analysis, and outputs

[60]

Quality involvement Framework

Six dimensions

Ability to (i.e., access research resources, contribute, express views); Potential to (i.e., take up roles, identify and organize interests); Sense of being (i.e., valued as a partner not controlled, empowered); Research relationships (i.e., expectations); Ways of doing research (i.e., roles available to be taken up), Research structures (i.e., research organizations, ethics, governance)

[67]

Evaluation framework

Based on A Model Framework for Consumer and Community Participation in Health and Medical Research [79]

Impact and value of engagement, compliance with Telford’s principles

[57]

–

Conceptual framework

Degree of researcher versus public engagement

[55]

Method (discussion based)

Focus group

Semi-structured

Experiences, impact, perceived purpose of patient and public involvement, recommendations

[46]

Enjoyment/satisfaction from participation, perceived impact, impact of involvement on study

[45]

Patients' need for information and layout needs

[49]

Experiences, activities, perception of involvement (positive or negative), recommendations for involving others

[53]

Issues of the restricted socio-demographics of the group, the perceived lack of feedback from researchers, and the effectiveness of current and future means to maximize volunteer support

[43]

Perspectives on the project

[48]

Semi-structured; with presentation of results

Reflection on practical guidelines, based on the preliminary results

[41]

Interview

Semi-structured

Views on monitoring and evaluation systems used to record involvement activities, feasibility of systematically collecting/collating data on the nature and impact of young people’s involvement, key opportunities and challenges

[38]

Researchers views on impact of community expert input on their attitudes and practices

[68]

Experiences with initiating and formulating the research agenda, verification of data obtained by study, experiences with programming and implementation, barriers and facilitators for patient involvement in programming and implementation

[41]

Need for public involvement as a means to funding and recruitment, value of efficiency and the speed of turnaround of queries

[43]

Perspectives of the project

[48]

Workshop

Included introduction of framework

Experiences of involvement as members of a public group within their parent organizations; mapping these along framework

[58]

Method (question based) i.e., audit, questionnaire, survey)

CBPR Tool

Information not available

Satisfaction with involvement, trust, comfortability in sharing opinions

[52]

PPI audit tool

12 Questions; open-ended

Identifying PPI work (e.g., what is the PPI for? What is your current activity to involve users?)

Effectiveness of PPI work (e.g., What is the goal of your PPI? Who is involved (how diverse) in PPI work at what) Resourcing PPI work (e.g., Who looks after PPI in your organization? What is their role?)

[62]

–

7 Questions; open-ended†

Motivation for involvement, feedback meeting structure and length, feedback on group, training required, project feedback, impact of involvement, value of involvement, general comments

[61]

–

5–6 Questions; open-ended†

Impact of public involvement, nature of involvement, information level, satisfaction

[44]

Partnership Assessment In community-based Research (PAIR)

31 Close-ended items (Likert scale) and one open ended item

communication, collaboration, partnership values, benefits, and evaluation, with an open-ended final item included to assess respondents’ view of whether they believe completion of the measure will impact how they work with their partner in the future

[66]

Public and Patient Engagement Evaluation Tool (PPEET)

Closed and open-ended questions†

Principles of ‘quality engagement’: (i) integrity of design and process; (ii) influence and impact; (iii) participatory culture; and (iv) collaboration and common purpose. Three unique questionnaires were developed to assess each of these four evaluation domains from the following perspectives: (i) those who participate in PPE activities; (ii) those who plan, execute or sponsor PPE activities within organizations; and (iii) those who provide the leadership and capacity for PPE within their organizations

[69]

Quality involvement questionnaire

4–7 Statements in each part; agreement rated on not at all to high

Multi-part questionnaire on perceived ability, potential, sense of being, research contexts, ways of doing research, research structure

[47, 67]

PPI assessment survey (Open-source evaluation tool)

19 Questions with 1–10 scale an open-ended option; 30 questions (not true at all to moderately true)

Satisfaction on (1) facilities, (2) level of contribution, (3) comfort, (4) understanding, (5) support and personal factors

[59]

–

12 Observable behaviours, paper-based assessment form using Likert scale

Interpersonal relations, nature of advisor contributions, how advisors guided research development

[37]

–

Structured; closed- and open- ended questions

Nature of this involvement in general terms, impact of that involvement, details of specific trials in which consumers had been involved, background of the consumers who had been involved, and the researcher's views about impact of consumer involvement

[51]

–

7 With statements based [1–10 scale], yes/no questions, one open-ended question †

Perceived value of consumer contribution, perceptions of consumer contribution, information about consumer contribution in previous projects (Researcher’s survey)

Perceived value of contribution, perceived respect, empowerment, information, previous experience with contribution (Consumer’s survey)

[63]

–

13–16 Items; Forced or multiple choice (Likert scale), open-ended questions †

What changes they made to their research proposal, research project, or community engagement practices as a result of the input they received from the community experts, community expert experiences, process and outcomes, suggestions to improve the engagement process

[68]

Adapted from Patients as partners in research survey

Yes/no questions, close-ended questions, qualitative items †

Engagement evaluation, views on understanding, sensitivity, feasibility of project, appropriateness of project, satisfaction

[50]

Post-meeting survey

9 Questions (Likert scale); 4 open-ended questions

Evaluation of workshop, satisfaction, level of understanding, empowerment, support provided

[65]

–

Information not provided

Satisfaction, reasons for engagement (i.e., interest in research, desire to improve research to benefit patients, desire to use own experiences in a valuable way, personal learning), nature of involvement (i.e., advice on lay summaries, training activities for volunteers, membership of specific activity/ research group, support received from the staff of the project)

[43]

–

Open-ended questions, attitudinal statements (Likert scale) *

Motivation for involvement in research, role in research, reflections on consumer involvement

[48]

–

8–15 questions (5-point Likert scale) †

Content (format and structure, scope and meaning, characteristics of lecturers)

Communication and supports for participation, sharing views and perspectives, impacts and influence, final thoughts, integrity and design of process, final reflections

[49]

Critical appraisal guidelines

9 Multi-part questions; open-ended

Rationale for involving users, recruitment strategy, training, methodology, value of citizen involvement, evaluation of impact

[56]

Method (other)

Canadian Centre for Excellence on Partnerships with Patients and Public (CEPPP) evaluation tool

Tool to evaluate frameworks

Types of frameworks found (categorized) power‐focused; priority‐setting; study‐focused; report‐focused; and partnership‐focused

[42]

Observation

Observation (attending individual project steering group meetings)

[48]

PPI-ready planning canvas

Barriers, concerns, worries, challenges, value and impact, challenges in the implementation and adoption of PPI into research practice

[39]

Prioritization tasks

Prioritization of needs for feedback; evaluation of drafts

[49]

Telford’s Indicators of user involvement

16 Indicators, 8 principles

Roles of consumers agreed upon by researchers and consumers, budgeting appropriately for cost of consumer involvement, respecting consumers, support and necessary skills, consumers involved in decision making and dissemination, CE is reported on, research findings are made available and easy to understand)

[40]