Skip to main content

The impact of public involvement in health research: what are we measuring? Why are we measuring it? Should we stop measuring it?

Abstract

As public involvement in the design, conduct and dissemination of health research has become an expected norm and firmly enshrined in policy, interest in measuring its impact has also grown. Despite a drive to assess the impact of public involvement, and a growing body of studies attempting to do just this, a number of questions have been largely ignored.

This commentary addresses these omissions: What is the impact of all this focus on measuring impact? How is the language of impact shaping the debate about, and the practice of, public involvement in health research? And how have shifting conceptualisations of public involvement in health research shaped, and been shaped by, the way we think about and measure impact? We argue that the focus on impact risks distorting how public involvement in health research is conceptualised and practised, blinding us to possible negative impacts.

We call for a critical research agenda for public involvement that [a] considers public involvement not as an instrumental intervention but a social practice of dialogue and learning between researchers and the public; [b] explores how power relations play out in the context of public involvement in health research, what empowerment means and whose interests are served by it, and [c] asks questions about possible harms as well as benefits of public involvement, and whether the language of impact is helpful or not.

Peer Review reports

Plain English summary

The involvement of patients and the public in health research is common practice and a key requirement made by research grant funding bodies. As public involvement has grown, so too has a desire to measure and demonstrate its impact – this could be the impact involvement activities have on research processes and outcomes, the impact involvement has on participating members of the public and researchers, or the impact involvement has on addressing power imbalances between the researcher and the researched.

In this commentary, we look at what effect this focus on impact is having on rationales for doing public involvement in the first place and how public involvement is done. Measures of public involvement which are easy to count, such as numbers of people involved, tend to be favoured. However, this tells us little about how patients and the public have changed the course of research studies. Certain kinds of impact, such as how do public involvement activities change power relations and empower the public, are largely not being captured.

We call for a critical research agenda for public involvement – one that acknowledges:

  • different rationales for public involvement, and how the desire to measure effects might be shaping these rationales.

  • that there might be negative impacts of public involvement as these are largely unreported.

  • the impact of power and in particular how power relations play out in research settings.

Background

As public involvement in the design, conduct and dissemination of health research has become an expected norm and firmly enshrined in policy in the UK [1, 2] and internationally [3], interest in measuring its impact has also grown. Advocates of public involvement in health research are keen to see its benefits demonstrated; sceptics anticipate having their doubts confirmed; and others would like to better understand whether and how public involvement impacts on research processes and outcomes, on those involved, and any broader societal effects [4, 5].

The international literature on how to evaluate the impact of public involvement has more than tripled in recent years [6]. The question has been addressed in systematic reviews [7,8,9,10,11,12,13,14], realist evaluations [15,16,17] and studies of stakeholder views [18, 19]. Some studies have developed practical guidance for researchers to incorporate assessment of the impact of public involvement in their research [20], and at least 65 frameworks (summarised in a recent systematic review [21]) have been developed for assessing the nature and impact of public involvement in health research.

Together these studies offer a substantial evidence base on certain aspects of the impact of public involvement in health research. They have identified, for example, how public involvement can increase recruitment to clinical trials [8, 22, 23], make research more relevant and appropriate for users [14, 24], help to formulate research questions and shape and reshape study design [25, 26], and provide insights to inform and develop analysis [27]. However, a significant proportion of the available evidence about impact is considered to be anecdotal and weak [28, 29], and there are frequent and widespread calls for more robust methods and instruments to capture and measure impact [30,31,32]. A recent BMJ editorial on the subject of impact of public involvement in health research concluded: “if we are serious about involvement, we need to be equally serious about evaluation [of impact]” [6].

Questions that have been largely ignored by the existing evidence base, and which provide the starting point for this paper, include the following: What are the consequences of all this focus on measuring impact? How is the language of impact shaping the debate about, and the practice of, public involvement in health research? And how have shifting conceptualisations of public involvement in health research shaped the way we think about and measure impact?

Critical academics, prompted by the inclusion of impact in the Research Excellence Framework (a system for assessing research quality in UK universities), have argued against measuring impact. Formal measurement of impact may distort practice and draws us into the mindset and practices of performativity. As Fielding put it, “it valorises what is short-term, readily visible and easily measurable... it has difficulty comprehending and valuing what is complex and problematic” [33]. Reports tend to emphasise positive impacts and to under-report or neglect what might be construed as negative [34]. Rarely considered – perhaps because they are hard to measure – are long-term but potentially far-reaching influences (positive and negative) on the culture of research itself. Impacts may get overblown in an attempt to secure further funding – a phenomenon referred to as ‘impact sensationalism’ [35]. An insistent focus on measuring impact can mean that the collection of other equally important evidence, such as data on implementation processes, is compromised [36].

In this commentary, we contend that there are comparable arguments to be made in relation to measuring the impact of public involvement in health research. The current emphasis on impact, we believe, risks distorting how public involvement in health research is conceptualised and practised, and blinds us to possible negative impacts of public involvement in health research. In making these arguments it is not our intention to deny the importance of exploring the effect of public involvement in health research, nor to undermine the value of work undertaken by researchers in this field. Rather, we wish to draw attention to the particular ways in which the debate about impact has been framed, and to highlight the associated risks.

The arguments we present have emerged from an historical analysis that two of us (JR and TG) undertook of the intended impact of the National Institute for Health Research (NIHR) public involvement policy, commissioned by NIHR to provide a baseline for further work on assessing impact of public involvement [37]. Our focus is therefore primarily, though not exclusively, oriented to NIHR policies and practices, widely cited as exemplars, both nationally and internationally, of public involvement in health research [3, 10, 16, 38]. We also draw on the second author’s ethnographic work on involving stroke survivors in service development and research [39]. Throughout our commentary we follow NIHR in adopting the term ‘public involvement’ to include patients, service users, survivors, carers and family members [1], while mindful of the distinctions that are drawn between the categories of ‘patient’ and ‘public’ [40]. The broad term of public involvement intersects with, but does not necessarily equate with, approaches such as patient-oriented research, integrated knowledge translation research, lay representation, citizen engagement, co-production and co-design.

How public involvement in health research is conceptualised

The majority of studies of the impact of public involvement in health research have focused on its impact on the research process (for example, on recruitment and retention rates), and on the quality, validity, relevance and utility of research [7, 8]. Public involvement is conceptualised as a means to the end of achieving better research – or, to put it in philosophical terms, public involvement is seen through a ‘consequentialist’, or ‘benefits-based’ lens [41]. In line with the norms of evidence-based medicine, which is seen as the gold standard for measuring effectiveness in health research, public involvement is positioned as an intervention that has a potentially fixed and measurable effect size [28].

This conceptualisation of public involvement has appeal and persuasive power to researchers – it focuses on what is familiar to them in terms of scientific method, and what is likely to matter to them most: the question of whether public involvement leads to better research [8]. However, the risk is that other, equally important, conceptualisations of public involvement are obscured. Mathie et al. argue that: “defining consumer involvement outcomes solely in terms of research quality ignores the rights of those being researched or likely to benefit from the research” [10].

An alternative ‘democratic’ or ‘rights-based’ framing of public involvement draws on philosophies of justice, human rights and empowerment, and sees public involvement not so much as a means to an end (better research) but as an end in itself [42]. From this perspective, public involvement is “primarily concerned with people having more say in agencies, organisations, and institutions which impact upon them and being able to exert more control over their own lives” [43]. This conceptualisation of public involvement necessarily shifts the attention of impact studies from the research endeavour onto patients and the public and the wider community. For example, studies have highlighted how involvement in health research may provide a life focus for patients and the public, can impact positively on self-confidence, and on knowledge about best evidence relating to treatment and care [14, 44, 45]. Furthermore, it can provide support and friendships, and facilitate the learning of new skills, such as communication, presentation, and research skills [18]. Whilst these sort of positive impacts on individual experience may be a first step towards the democratising of research processes, direct forms of participation do not necessarily address the democratic deficit at a collective level, and arguably might perpetuate inequalities if only certain people benefit from involvement. Studies have suggested that organisation-led involvement, practiced within the ‘norms of bureaucracy’, encourages a certain type of patient or citizen to participate, those who feel comfortable with the business meeting or committee format [46]. This means that those who get involved in health research are predominantly white, middle class, retired people, often from a health or research background, constituting an ‘unrepresentative minority’ [47, 48]. Citizens and community members who routinely go unheard – those with extremes of age, complex health/social care needs, limited English speakers, the homeless, those without full citizenship rights, those whose illness makes communication or understanding difficult, or with stigmatising illnesses - are further actively discouraged to participate [46].

Some argue that if involvement is conceptualised as a democratic right, with intrinsic value, and an end in itself, no further evaluation of impact is needed [41]. Others have suggested that it is inappropriate to consider public involvement in health research as an intervention with measurable impacts, questioning “the appropriateness of applying scientific enquiry to a social, collaborative partnership, where mutual learning takes place during personal interactions” [18]. In this vein, Staley and Barron suggest that we should instead conceptualise public involvement as conversations between researchers and the public that support two-way learning. The learning that takes place from such conversations, they suggest, can be captured through in-depth qualitative methods, but is always going to be difficult to quantify and measure. Staley and Barron consider public involvement in health research, and the learning to emerge from it, as evolutionary, unpredictable, subjective and context-specific. They argue that we need to move away from an instrumental conceptualisation of patients and the public as ‘the intervention’, and from the view that experimental methods, and particularly randomised controlled trials, are an appropriate way of assessing impact. They conclude: “trying to standardise involvement processes as ‘methods’ and to objectify the outcomes, may be akin to ‘forcing a square peg into a round hole’. The richness and value of subjective learning needs greater recognition” [28].

Friesen and colleagues make similar criticisms of the effect of the impact agenda on conceptualisations of public involvement [5]. They argue that the dominant focus on measuring the impact of public involvement is harmful because of the way in which it has diverted attention away from ethical reasons for public involvement, towards a narrow focus on epistemic justifications. Epistemic justifications for public involvement, which approximate to the benefits-based conceptualisation of public involvement discussed above, focus predominantly on the impact that involvement has on the research process and outputs, on what Friesen et al. refer to as the three Rs: rigour, relevance and reach. But, they argue:

“Service users have not fought for a voice at the table merely to help improve the research process, but because they have a right to be there. This suggests that what justifies involvement is much larger than that captured by the epistemic focus of those seeking to evaluate the impact of participatory research” [5].

Whilst Friesen and colleagues are talking specifically about participatory research in psychiatry, we suggest that their arguments are applicable to public involvement in health research more generally. As these authors point out, the ethical reasons for involvement are to do with democratic rights and empowerment. Paying serious attention to this conceptualisation of involvement, they argue, means looking beyond questions of impact, to broader questions about imbalances of power.

Changing power relations between researchers and the researched has long been a motivating aim of user social movements, particularly those of people with disabilities and mental health service users [43]. The aim has been for public involvement to progress up Arnstein’s ‘ladder of participation’, a conceptual framework for understanding involvement dating from the 1960s [49]. Low down on the ladder are activities such as informing and consultation, frequently perceived as tokenistic forms of public involvement, moving up to more meaningful and genuine forms of power-sharing such as partnership, delegated power and ultimately citizen control of the research agenda.

While some commentators argue that the past decade has seen real shifts towards the democratisation of health research [4], others are less confident that there has been any fundamental shift in power relations from the scientific community to the public. From her analysis of NIHR activity, Green for example argues that, despite an increasingly progressive rhetoric, as evidenced in the attention given to methods such as co-production and co-design, and despite a body of evidence demonstrating that it is increasingly the norm for the public voice to be incorporated into various stages of the research process, “there has not, however, been a concomitant transformation of the social relations of research, as envisaged by the emancipatory research movement” [42]. Boaz and colleagues’ interview study of researchers involved with NIHR Biomedical Research Centres concluded that despite “changing currents on the surface”, there has remained active resistance to sharing power and control in the process of knowledge generation [50]. Similarly, a recent study of the development of public involvement in a London based mental health biomedical research centre reported that:

“PPI remained localised and under resourced and there was a reluctance to change working practices which resulted in perceptions of tokenism. Service users faced conflicting expectations and were expected to assimilate rather than challenge the organisation’s ‘biomedical agenda’” [51].

Notwithstanding these snippets of evidence, we still know very little about the impact of public involvement on power relations between researchers and the public, because this is rarely the focus of impact research. A reviewer of an earlier draft of this paper commented that whereas researchers and research funders have to a large extent accepted and come to value a certain kind of patient involvement (in the sense of contributing the lived experience of illness to the research agenda of a particular condition [52]), the place of public involvement (in the sense of accountability to critical and questioning voices from wider society) remains awkward.

Indeed, some studies have even suggested signs of movement away from the democratisation of health research. Our historical analysis of NIHR policy suggested that whereas earlier policy documents emphasised a rights-based discourse of involvement (epitomised by the phrase ‘nothing about us without us’), we detected a more recent concern with the duties and responsibilities of users and citizens [37]. For example, the strategic goal of NIHR policy identified in ‘Going the Extra Mile’ is an expectation that by 2025 everyone using health and social care “to choose to contribute to research” through a number of avenues. The public, as users of health services, are to be ‘empowered’ to ‘seize the opportunities’ to engage and become actively involved in research:

"By 2025 we expect all people using health and social care, and increasing numbers of the public, to be aware of and choosing to contribute to research by:

  • Identifying future research priorities and research questions

  • Informing the design and development of innovations

  • Participating in research studies

  • Advocating for the adoption and implementation of research in the NHS" [2].

It seems that public involvement policy here is ‘empowering’ people to become responsible ‘active citizens’ who contribute to the national research endeavour [53]. We can see a subtle shift in the conceptualisation of empowerment, away from any emancipatory understanding of the term, towards a restricted one to do with ‘participating in research studies’ and ‘informing’, activities that are more amenable to measurement. And as a consequence, questions about how public involvement might enable the sharing of power, who power is being shared with, and in what ways, are easily side-stepped [5].

How public involvement in health research is practised

Approaches to measuring impact of public involvement in health research shape not only how we conceptualise public involvement but how it is practiced. The language of impact demands that effects can be formally measured. In other fields the arguments of how a preoccupation with measurement shapes/distorts practice have been well rehearsed – ‘teaching to the test’ in education [54], the ‘gaming’ of quality targets in health care [55], and so on [56,57,58].

In the field of public involvement in health research we see how, for example, in NIHR reporting on public involvement activity, the focus is invariably on quantifying impact – the number of training workshops and conferences held each year, the number of people employed as public involvement leads, facilitators and advisers, and quantifying the work of the NIHR Research Design Service [59,60,61]. What gets recorded is what can be measured (and therefore defined as impact) and the ‘backwash’ effect of this is that what gets done is what can be measured. Wilsdon and colleagues argue that in public policy the focus on “the ‘hardware’ of engagement – the methods, the focus groups, the citizens’ juries that can give the public a voice in science policy and decision-making” has too often been at the expense of “the ‘software’ – the codes, values and norms that govern scientific practice, but which are far harder to access and change” [62]. These “software” elements are also far harder to measure and demonstrate impact, and therefore tend to get neglected in favour of easier aspects of practice to measure and quantify. Chubb and Derrick’s analysis of research impact in higher education suggests that there may also be a gender dimension at play here, given the gendered associations with notions of ‘hard’ and ‘soft’:

“gender may play a role in the prioritisation of ‘hard’ Impacts (and research) that can be counted, in contrast to ‘soft’ Impacts (and research) that are far less quantifiable, reminiscent of deeper entrenched views about the value of different ‘modes’ of research” [63].

A recent development with public involvement in health research in the UK has been the setting of national standards by NIHR [1]. While the authors of these standards explicitly say that “They are not designed as rules, or to provide fixed ideas about public involvement in research” [64], the danger is that they will do exactly that. The advantage of standards is that they offer concrete examples of how public involvement can be undertaken. But their disadvantage is that they can become prescriptive; they define and bound activity in a constraining way, institutionalising the rules of the game [65]. McCoy and colleagues go further in their criticism of the standards, arguing that they “fail to address fundamental questions about when, why and with whom involvement should be undertaken in the first place”. This lack of justificatory context, they argue, feeds the problem of tokenism in public involvement practice – it encourages researchers to undertake involvement activities without necessarily being clear about or reflecting fully on their purpose. The danger is that the standards promote “an unreflective ‘more the merrier’ attitude in relation to involvement” [66].

The impact of public involvement in health research – what is not being captured?

Public involvement is typically presented as being unquestionably a good thing in relation to healthcare in general and health research in particular [67]. Consequently, impact tends to be unproblematically equated with benefit. This can be seen in the NIHR Public Involvement Impact Working Group’s definition of impact of public involvement in NIHR health and social care research: “The changes, benefits and learning gained from the insights and experiences of patients, carers and the public when working in partnership with researchers and others involved in NIHR initiatives” [68]. In this definition there is no mention of negative impacts. Interestingly, in one of the few and now somewhat old reviews of negative impacts (from 1997 to 2009), Staniszewska et al. found that more discussion of negative impact occurred in the earlier papers they reviewed. It is unclear, they suggest “why a reduction has occurred at a time when interest in impact has expanded” [69]. One explanation, we would suggest, is that as public involvement has become more formalised into institutional structures and research practices, it has perhaps become harder to speak critically of it. And while the NIHR national standards for public involvement at least acknowledge that “we can learn from both positive and negative impacts”, it is disappointing that there is no further discussion of possible negative impacts [1].

A few researchers have identified some of the negative impacts of public involvement on those who get involved: feelings of overwork and frustration at the limited opportunities to influence the direction of research; feelings of being marginalised, confusion and conflict due to lack of clarity about the lay role, the burden of responsibility and duty, as well as time and financial burdens [18, 44, 70]. Ashcroft and colleagues suggest that more needs to be done to mitigate the possible negative effects of involvement, by for example ensuring language is clear and not excluding, giving regular feedback, and ensuring people involved have sufficient time to digest information [45]. Concerns have also been expressed by both the public and researchers that public involvement can sometimes appear to amount to little more than a ‘tick box’ exercise, with researchers going through the motions of involvement to satisfy the reporting requirements of funding bodies and ethics committees, rather than meaningful and robust engagement with the processes of involvement [71].

Others have drawn attention to the possible negative impact of involvement on researchers themselves, in terms of the additional resources, time, and skills required, and the stress and tensions that can arise in negotiating the divergent interests involved. Moreover, it is suggested that public involvement challenges the ‘science’ of research and whose knowledge counts; Bekkum and Hilton for example found “numerous and sometimes conflicting concerns about public knowledge deficits and their biases, emotions and personal interests potentially damaging the integrity of science” [19]. Similarly, Oliver et al. suggest that coproduced research can be regarded by the academy as “partisan and biased” and thus of lower quality than “‘real’ or ‘pure’ research” [72].

Perhaps the most important criticism of public involvement in health research is that, rather than being empowering or emancipatory, it runs the danger of having precisely the opposite effect. Fudge’s research of stroke survivors’ involvement in service development and research suggests that the way in which the resources of service users are sought through public involvement mechanisms can ultimately inhibit their capacity for protest, and thus, she argues, has the overall effect of containing and quietening radical social movements [39, 73] Mechanisms such as time-control, allotted slots for patients to speak, and pre-set agendas served to ensure patient discussions were contained and did not go astray. The ritual structure of meetings that public involvement organisers employed (in terms of orientation, time and content), ensured that public involvement was directed towards researchers’ own productive aims, namely generating grant income and research papers [73]. Similarly, Madden et al. assert that current public involvement policy “creates PPI as a form of busywork in which the politics of social movements are entirely displaced by technocratic discourses of managerialism” [30] And for Oliver and colleagues, “far from empowering people politically, participation in research can lead to personal narratives and experiences being dominated by senior interests, leading to lack of motivation to engage again, and the ‘subjugation’ of participants” [72]. These critical analyses of public involvement again bring us back to the question of power and the impact of how power relations play out in research settings. But such questions about impact, whilst urgent and necessary, are in the main missing from the mainstream impact agenda.

Conclusion

This conceptual review supports our call for a more critical research agenda on public involvement in health research. We are encouraged that some groups have begun to move away from an instrumental conceptualisation of public involvement as an ‘intervention’ that has a measurable effect, and are recognising the value of continuous reflection as part of the research process [74]. We align with Staley and Barron [28], who conceptualise public involvement as a social practice of dialogue and learning between researchers and the public: an end in itself, not merely a means to an end (at worst, measured superficially as ‘bums on seats’). Critical public involvement research should explore the complexity and richness of this relationship, using methods that emphasise illumination rather than measurement [75] and asking when, why and with whom the dialogue happens or fails to happen.

We still know very little about whether and how public involvement changes power relations between researchers and the public, because this is rarely the focus of impact research. A systematic review identified some potentially useful power-focused frameworks for supporting and evaluating public involvement in research, including questions such as ‘who gets to define what empowerment is?’ and ‘whose interests are served by so-called empowerment?’ [21, 76]. And we still know very little about the wider influence of involvement on the culture of research itself, a potentially fruitful avenue for critical research.

Our findings suggest that we need more research that acknowledges, investigates and reports on the negative impacts of attempts at public involvement in health research and metrics that measure such involvement. We must ask questions about ways in which public involvement could increase inequalities, distort and suppress rather than amplify particular voices and agendas, and about how focusing on measuring impact could overshadow engagement with ethical reasons for involvement. Finally, we need to question whether the language of measurement and impact is helpful or not, and if not, what alternative language might better serve the goal of improving public involvement in health research. For example, there is a critical distinction to be drawn between ‘measuring the impact of …’ and ‘evaluating’. The former is technocratic, instrumental, largely quantitative, and summative. The latter might be reflexive, imaginative, dialogic and formative.

Much has been achieved in the field of public involvement in recent years. Much critical research remains to be done.

Availability of data and materials

Not applicable.

Abbreviations

NIHR:

National Institute for Health Research

References

  1. National Institute for Health Research. National Standards for public involvement in research. London; 2018. https://www.invo.org.uk/posttypepublication/national-standards-for-public-involvement/ Accessed 12 Mar 2020.

  2. Denegri S. Going the extra mile: improving the nation’s health and wellbeing through public involvement in research. London: INVOLVE; 2015.

    Google Scholar 

  3. Richards T. Patient and public involvement in research goes global; 2017. thebmjopinion. https://blogs.bmj.com/bmj/2017/11/30/tessa-richards-patient-and-public-involvement-in-research-goes-global/ Accessed 20 Sep 2020.

    Google Scholar 

  4. Becker S, Sempik J, Bryman A. Advocates, agnostics and adversaries: researchers’ perceptions of service user involvement in social policy research. Soc Policy Soc. 2010;9(3):355–66.

    Article  Google Scholar 

  5. Friesen P, Lignou S, Sheehan M, Singh I. Measuring the impact of participatory research in psychiatry: how the search for epistemic justifications obscures ethical considerations. Health Expect. 2019;(September):1–8.

  6. Boivin A, Richards T. Forsythe L. Evaluating patient and public involvement in research. Br Med J.Br Med J. 2018. https://doi.org/10.1136/bmj.k5147.

  7. Brett J, Staniszewska S, Mockford C, Seers K, Herron-Marx S, Bayliss H. The PIRICOM study: a systematic review of the conceptualisation, measurement, impact and outcomes of patients and public involvement in health and social care research: University of Warwick; 2010. https://www.ukcrc.org/wp-content/uploads/2014/03/Piricom+Review+Final+2010.pdf Accessed: 15 Jan 2020.

  8. Crocker J, Ricci-Cabello I, Parker A, Hirst J, Chant A, Petit-Zeman S, et al. Assessing the impact of patient and public involvement on recruitment and retention in clinical trials: a systematic review. Br Med J. 2018;363:k4738.

    Article  Google Scholar 

  9. Domecq JP, Prutsky G, Elraiyah T, Wang Z, Nabhan M, Shippee N, et al. Patient engagement in research: a systematic review. BMC Health Serv Res. 2014. https://doi.org/10.1186/1472-6963-14-89.

  10. Mathie E, Wilson P, Poland F, Mcneilly E, Howe A, Staniszewska S, et al. Consumer involvement in health research: a UK scoping and survey. Int J Consum Stud. 2014;38(1):35–44.

    Article  Google Scholar 

  11. Mockford C, Staniszewska S, Griffiths F, Herron-marx S. The impact of patient and public involvement on UK NHS health care: a systematic review. International J Qual Health Care. 2012;24(1):28–38.

    Article  Google Scholar 

  12. Nilsen ES, Myrhaug HT, Johansen M, Oliver S, Oxman AD. Methods of consumer involvement in developing healthcare policy and research, clinical practice guidelines and patient information material. CochraneDatabaseSystRev. 2006;19(3):CD004563.

    Google Scholar 

  13. Shippee ND, Domecq Garces JP, Prutsky Lopez GJ, Wang Z, Elraiyah TA, Nabhan M, et al. Patient and service user engagement in research: a systematic review and synthesized framework. Health Expect. 2015;18(5):1151–66.

    Article  Google Scholar 

  14. Staley K. Exploring impact: public involvement in NHS, public health and social care research. London; 2009. https://www.invo.org.uk/posttypepublication/exploring-impact-public-involvement-in-nhs-public-health-and-social-care-research/ Accessed 23 Aug 2018.

  15. Evans D, Coad J, Cottrell K, Dalrymple J, Davies R, Donald C, et al. Public involvement in research: assessing impact through a realist evaluation. Health Services Delivery Res. 2014;2(36):1–128.

    Article  Google Scholar 

  16. Wilson P, Mathie E, Poland F, Keenan J, Howe A, Munday D, et al. How embedded is public involvement in mainstream health research in England a decade after policy implementation? A realist evaluation. J Health Serv Res Policy. 2018;23(2):98–106.

    Article  Google Scholar 

  17. Forsythe LP, Carman KL, Szydlowski V, Fayish L, Davidson L, Hickam DH, et al. Patient engagement in research: early findings from the Patient-Centered Outcomes Research Institute. Health Affairs (Project Hope). 2019;38(3):359–67.

    Article  Google Scholar 

  18. Barber R, Boote JD, Parry GD, Cooper CL, Yeeles P, Cook S. Can the impact of public involvement on research be evaluated? A mixed methods study. Health Expect. 2012;15(3):229–41.

    Article  Google Scholar 

  19. van Bekkum JE, Hilton S. UK research funding bodies’ views towards public participation in health-related research decisions: an exploratory study. BMC Health Serv Res. 2014;14:318.

    Article  Google Scholar 

  20. Popay J, Collins M. PiiAF the public involvement impact assessment framework guidance; 2014. http://piiaf.org.uk Accessed 2 Dec 2019.

    Google Scholar 

  21. Greenhalgh T, Hinton L, Finlay T, Macfarlane A, Fahy N, Clyde B, et al. Frameworks for supporting patient and public involvement in research: systematic review and co-design pilot. Health Expect. 2019. https://doi.org/10.1111/hex.12888.

  22. Carter P, Beech R, Coxon D, Thomas MJ, Jinks C. Mobilising the experiential knowledge of clinicians, patients and carers for applied health-care research. Contemp Soc Sci. 2013;8(3):307–20.

    Article  Google Scholar 

  23. Ennis L, Wykes T. Impact of patient involvement in mental health research: longitudinal study. Br J Psychiatry. 2013;203(5):381–6.

    Article  Google Scholar 

  24. Staley K, Kabir T, Szmukler G. Service users as collaborators in mental health research: less stick, more carrot. Psychol Med. 2013;43(6):1121–5.

    Article  CAS  Google Scholar 

  25. Wykes TIL. Blue skies in the journal of mental health? Consumers in research. J Ment Health. 2003;12(1):1–6.

    Article  Google Scholar 

  26. Wykes T. Great expectations for participatory research: what have we achieved in the last ten years? World Psychiatry. 2014;13(1):24–7.

    Article  Google Scholar 

  27. Rose D. Telling different stories: user involvement in mental health research. Res Policy Plan. 2004;22(2):23–30.

    Google Scholar 

  28. Staley K, Barron D. Learning as an outcome of involvement in research: what are the implications for practice, reporting and evaluation? Res Involve Engage. 2019;1:1–9.

    Google Scholar 

  29. Staley K. ‘Is it worth doing ? ’ Measuring the impact of patient and public involvement in research. Res Involve Engage. 2015;1(6):1–10.

    Google Scholar 

  30. Madden M, Speed E. Beware zombies and unicorns: toward critical patient and public involvement in Health Research in a neoliberal context. Front Sociol. 2017;2(June):1–6.

    Google Scholar 

  31. Petit-Zeman S, Locock L. Health care: bring on the evidence. Nature. 2013;501(7466):160–1.

    Article  Google Scholar 

  32. Staniszewska S, Adebajo A, Barber R, Beresford P, Brady L-M, Brett J, et al. Developing the evidence base of patient and public involvement in health and social care research: the case for measuring impact. Int J Consum Stud. 2011;35(6):628–32.

    Article  Google Scholar 

  33. Fielding M. The impact of impact. Camb J Educ. 2010;33(2):289–95.

    Article  Google Scholar 

  34. Colley H. What (a) to do about ‘impact’: a Bourdieusian critique. Br Educ Res J. 2014;40(4):660–81.

    Article  Google Scholar 

  35. Chubb J, Watermeyer R. Artifice or integrity in the marketization of research impact ? Investigating the moral economy of (pathways to) impact statements within research funding proposals in the UK and Australia. Stud High Educ. 2017;42(12):2360–72.

    Article  Google Scholar 

  36. Gugerty MK, Karlan D. Ten reasons not to measure impact-and what to do instead. Stanf Soc Innov Rev. 2018;Summer:41–7 www.ssir.org Accessed 20 Jun 2020.

    Google Scholar 

  37. Russell J, Greenhalgh T, Taylor M. Patient and public involvement in NIHR research 2006–2019: policy intentions, progress and themes: Oxford BRC; 2019. https://oxfordbrc.nihr.ac.uk/wp-content/uploads/2019/05/NIHR-and-PPI-report-Feb_2019.pdf Accessed 2 Mar 2019.

  38. Staniszewska S, Denegri S, Matthews R, Minogue V. Reviewing progress in public involvement in NIHR research: developing and implementing a new vision for the future. BMJ Open. 2018. https://doi.org/10.1136/bmjopen-2017-017124.

  39. Fudge N. The participation of stroke survivors in service development and research an ethnographic study. PhD thesis: King’s College London; 2013. https://kclpure.kcl.ac.uk/portal/en/theses/the-participation-of-stroke-survivors-in-service-development-and-research(f2040766-ba80-4860-951b-7363863a957f).html Accessed 23 Jul 2018.

  40. Fredriksson M, Tritter JQ. Disentangling patient and public involvement in healthcare decisions: why the difference matters. Sociol Health Illn. 2017;39(1):95–111.

    Article  Google Scholar 

  41. Edelman N, Barron D. Evaluation of public involvement in research: time for a major re-think? J Health Serv Res Policy. 2016;21(3):209–11.

    Article  Google Scholar 

  42. Green G. Power to the people: to what extent has public involvement in applied health research achieved this? Res Involve Engage. 2016;2(1):28.

    Article  Google Scholar 

  43. Beresford P. User involvement in research and evaluation: liberation or regulation. Soc Policy Soc. 2002;1(2):96–105.

    Article  Google Scholar 

  44. Brett J, Staniszewska S, Mockford C, Herron-Marx S, Hughes J, Tysall C, et al. A systematic review of the impact of patient and public involvement on service users, researchers and communities. The Patient. 2014;7(4):387–95.

    Article  Google Scholar 

  45. Ashcroft J, Wykes T, Taylor J, Crowther A, Szmukler G. Impact on the individual: what do patients and carers gain, lose and expect from being involved in research? J Ment Health. 2016;25(1):28–35.

    Article  Google Scholar 

  46. Cowden S, Singh G. The “user”: friend, foe or fetish? A critical exploration of user involvement in health and social care. Crit Soc Policy. 2007 Feb;27(1):5–23.

    Article  Google Scholar 

  47. Maguire K, Britten N. “How can anybody be representative for those kind of people?” forms of patient representation in health research, and why it is always contestable. Soc Sci Med. 2017;183:62–9.

    Article  Google Scholar 

  48. Healthtalk. Patient and public involvement in research; 2016. http://healthtalk.org Accessed 15 Feb 2018.

    Google Scholar 

  49. Arnstein SR. A ladder of citizen participation. J Am Inst Plann. 1969;35(4):216–24.

    Article  Google Scholar 

  50. Boaz A, Biri D, Mckevitt C. Rethinking the relationship between science and society: has there been a shift in attitudes to patient and public involvement and public engagement in science in the United Kingdom? Health Expect. 2016;19(3):592–601.

    Article  Google Scholar 

  51. Evans J, Papoulias S. Between funder requirements and “jobbing scientists”: the evolution of patient and public involvement in a mental health biomedical research Centre - a qualitative study. Res Involve Engage. 2020;6(1):1–2.

    Article  Google Scholar 

  52. NCRN/NCRI. Impact of patient, Carer and public involvement in cancer research contents. London: Nihr; 2012. http://www.ncri.org.uk/wp-content/uploads/2013/07/2012-NCRI-PPI-report.pdf Accessed 20 Aug 2020.

    Google Scholar 

  53. Martin GP. “Ordinary people only”: knowledge, representativeness, and the publics of public participation in healthcare. Sociol Health Illn. 2008;30(1):35–54.

    Article  Google Scholar 

  54. Vaughan R. UK among world’s worst for “teaching to the test”, research finds. 2015. Times Educational Supplement. https://www.tes.com/news/uk-among-worlds-worst-teaching-test-research-finds Accessed 20 Jun 2020.

  55. Bevan G, Hood C. What’s measured is what matters: targets and gaming in the English public health care system. Public Adm. 2006;84:517–38.

    Article  Google Scholar 

  56. Fukuda-Parr S, Yamin AE. The MDGs, capabilities and human rights: the power of numbers to shape agendas. London: Routledge; 2017.

    Book  Google Scholar 

  57. Porter T. Funny numbers. Culture Unbound: J Curr Cult Res. 2013;4(4):585–98.

    Article  Google Scholar 

  58. Power M. The audit society: rituals of verification. Oxford: OUP; 1997.

    Google Scholar 

  59. National Institute for Health Research. NIHR annual report 2015–16; 2016. https://www.nihr.ac.uk/about-us/documents/CRNperformance reports/2015–16NIHRCRNHighLevelObjectivesAnnualReport_v1.0Public.pdf Accessed 3 Feb 2018.

    Google Scholar 

  60. National Institute for Health Research. NIHR annual report 2013/14; 2014. http://www.nihr.ac.uk/documents/about-NIHR/NIHR-Publications/NIHR-Annual-Reports/NIHR Annual Report 2014-2015.pdf Accessed 3 Feb 2018.

    Google Scholar 

  61. National Institute for Health Research. NIHR report annual 2014/15; 2015. http://www.nihr.ac.uk/documents/about-NIHR/NIHR-Publications/NIHR-Annual-Reports/NIHR Annual Report 2014-2015.pdf Accessed 20 Jan 2020.

    Google Scholar 

  62. Wilsdon J, Wynne B, Stilgoe J. The Public Value of Science: Or how to ensure that science really matters. 2005. Demos. http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:We+need+to+infuse+the+culture+and+practice+of+science+with+a+new+set+of+social+possibilities+really+matters#0 Accessed 19 Jun 2015.

  63. Chubb J, Derrick GE. The impact a-gender: gendered orientations towards research impact and its evaluation. Palgrave Communications. 2020;6(1):1.

    Article  Google Scholar 

  64. NIHR. New national standards launched across the UK to improve public involvement in research; 2018. https://www.nihr.ac.uk/news/new-national-standards-launched-across-the-uk-to-improve-public-involvement-in-research/10885 Accessed 20 Jan 2019.

    Google Scholar 

  65. Mckevitt C. The possibilities and limits of “ co-producing” research. Front Sociol. 2019;4(23):1–5.

    Google Scholar 

  66. McCoy MS, Jongsma KR, Friesen P, Dunn M, Neuhaus CP, Rand L, et al. National Standards for public involvement in research: missing the forest for the trees. J Med Ethics. 2018;44:801–4.

  67. Greenhalgh T, Humphrey C, Woodard F. User involvement in health care. Oxford: Wiley-Blackwell; 2011.

    Google Scholar 

  68. Denegri S. Downloadable definition of the impact of public involvement in research; 2019. https://simondenegri.com/2019/07/03/downloadable-definition-of-the-impact-of-public-involvement-in-health-research-feat-patients-carers-and-the-public/ Accessed 20 Jan 2020.

    Google Scholar 

  69. Staniszewska S, Mockford C, Gibson A, Herron-Marx S, Putz B. Moving forward: understanding the negative experiences and impacts of patient and public involvement in health service planning, development and evaluation. In: Barnes M, Cotterrell P, editors. Critical perspectives on user involvement. Bristol: Policy Press; 2011.

    Google Scholar 

  70. Cotterell P, Harlow G, Morris C, Beresford P, Hanley B, Sargeant A, et al. Service user involvement in cancer care: the impact on service users. Health Expect. 2011;14(2):159–69.

    Article  Google Scholar 

  71. Snape D, Kirkham J, Britten N, Froggatt K, Gradinger F, Lobban F, et al. Exploring perceived barriers, drivers, impacts and the need for evaluation of public involvement in health and social care research: a modified Delphi study. BMJ Open. 2014;4(6):e004943.

    Article  CAS  Google Scholar 

  72. Oliver K, Kothari A, Mays N. The dark side of coproduction: do the costs outweigh the benefits for health research ? Health Res Policy Syst. 2019;3:1–10.

    Google Scholar 

  73. Komporozos-Athanasiou A, Fudge N, Adams M, Mckevitt C. Citizen participation as political ritual: towards a sociological theorizing of “health citizenship”. Sociology. 2018;52(4):744–61.

    Article  Google Scholar 

  74. Hickey G, Brearley S, Coldham T, Denegri S, Green G, Staniszewska S, et al. Guidance on co-producing a research project. Southampton; 2018. https://www.invo.org.uk/wp-content/uploads/2019/04/Copro_Guidance_Feb19.pdf Accessed 20 Sep 2020.

  75. Tsoukas H. Don’t simplify, Complexify: from disjunctive to conjunctive theorizing in organization and management studies. J Manag Stud. 2017;54(2):132–53.

    Article  Google Scholar 

  76. Morrow E, Ross F, Grocott P, Bennett J. A model and measure for quality service user involvement in health research. Int J Consum Stud. 2010;34(5):532–9.

    Article  Google Scholar 

Download references

Acknowledgements

We acknowledge Mark Taylor as a co-author of the report of a historical analysis of the intended impact of National Institute for Health Research (NIHR) public involvement policy, from which the ideas in this paper emerged.

Funding

The ideas in this paper emerged from a historical analysis of the intended impact of National Institute for Health Research (NIHR) public involvement policy funded by NIHR and Oxford Biomedical Research Centre (BRC-1215-20008).

Author information

Authors and Affiliations

Authors

Contributions

This commentary draws on and extends the arguments in a report co-authored by JR, TG, and Mark Taylor. JR drafted the paper with the assistance of NF and TG. The authors approved the final manuscript.

Corresponding author

Correspondence to Jill Russell.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Russell, J., Fudge, N. & Greenhalgh, T. The impact of public involvement in health research: what are we measuring? Why are we measuring it? Should we stop measuring it?. Res Involv Engagem 6, 63 (2020). https://doi.org/10.1186/s40900-020-00239-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s40900-020-00239-w

Keywords