This article has Open Peer Review reports available.
Evaluating public involvement in research design and grant development: Using a qualitative document analysis method to analyse an award scheme for researchers
© Baxter et al. 2016
Received: 4 December 2015
Accepted: 23 March 2016
Published: 1 April 2016
Plain English summary
The National Institute for Health Research (NIHR) Research Design Service (RDS) for Yorkshire and Humber has been running a public involvement funding scheme since 2008. This scheme awards researchers a small amount of money to help them get involvement from patients and/or the public. Involvement activities take place at the time when researchers are planning studies, and when they are completing application forms to request funding for a proposed research project. After the public involvement activities researchers are asked to write a report for the RDS describing what they did with the public involvement funding.
This study analysed those reports using an approach which included members of a public involvement panel in the data analysis process. The aim of the work was to see what the views and experiences of researchers who received funding were, and what might be learned for the future of the scheme. Twenty five reports were analysed. Four main themes were identified, these described: the added value of public involvement; aspects to consider when planning and designing public involvement; different roles of public contributors; and aspects of valuing public member contributions. The group approach to analysis was successful in enabling involvement of a variety of individuals in the process. The findings of the study provide evidence of the value of public involvement during the development of applications for research funding. The results also indicate that researchers recognise the variety in potential roles for the public in research, and acknowledge how involvement adds value to studies.
A regional Research Design Service, funded by the National Institute for Health Research, introduced a small grant in 2008, to support public involvement (often known as patient and public involvement [PPI]) activities during the development of applications for research funding. Successful applicants are requested to submit a report detailing how the grant money was used, including a description of the aims and outcomes of the public involvement activities. The purpose of this study was to analyse the content of these reports. We aimed to find out what researcher views and experiences of public involvement activities were, and what lessons might be learned.
We used an innovative method of data analysis, drawing on group participatory approaches, qualitative content analysis, and Framework Analysis to sort and label the content of the reports. We developed a framework of categories and sub-categories (or themes and sub-themes) from this process.
Twenty five documents were analysed. Four main themes were identified in the data: the added value of public involvement; planning and designing involvement; the role of public members; and valuing public member contributions. Within these themes, sub-themes related to the timing of involvement (prior to the research study/intended during the research study), and also specific benefits of public involvement such as: validating ideas; ensuring appropriate outcomes; ensuring the acceptability of data collection methods/tools and advice regarding research processes. Other sub-themes related to: finding and approaching public members; timing of events; training/support; the format of sessions; setting up public involvement panels: use of public contributors in analysis and interpretation of data; and using public members to assist with dissemination and translation into practice.
The analysis of reports submitted by researchers following involvement events provides evidence of the value of public involvement during the development of applications for research funding, and details a method for involving members of the public in data analysis which could be of value to other researchers The findings of the analysis indicate recognition amongst researchers of the variety in potential roles for public members in research, and also an acknowledgement of how involvement adds value to studies.
Public involvement (often known as patient and public involvement [PPI]) has been increasingly advocated as an essential element of high quality and clinically relevant research . By involvement in this context we are referring to patients, carers and/or members of the public actively contributing to the design, delivery, management and/or dissemination of research. However, there is limited evidence available regarding the processes and impact of public involvement in research . A review in 2009  reported that public involvement may increase study recruitment, improve trial design and outcome measures, and benefit the people involved. A review of published examples of involvement in research design  found that members of the public were reported to contribute to research design by: reviewing patient information sheets/consent procedures; suggesting outcome measures; reviewing data collection procedures; and recommending timing of recruitment and follow up.
Researchers who apply for funding to carry out studies are commonly expected to demonstrate how they have involved lay individuals in preparing their applications, to outline what changes were made to the application as a result of public involvement, and to outline in detail how they will involve members of the public in the design and conduct of the research if funded . The involvement of the public at an early stage in developing proposals can present a challenge to potential applicants as, without the funding being in place, there is no identified resource to draw upon to support involvement activities. Guidelines from the advisory body INVOLVE emphasise that the time, skills and expertise of members of the public should be recognised, and that accurate budgeting for costs is essential . Costs may relate for example to reviewing a research proposal, attending a meeting, preparing for a meeting, or reading relevant documents.
The National Institute for Health Research (NIHR) created 10 regional Research Design Services (RDSs) to provide design and methodological support to health and social care researchers across England, when developing grant applications to the NIHR and other national peer-reviewed funding programmes . The service provides advice and expertise, and builds “bridges and connections across research groups”  to enable high quality research proposals. As part of their role, the RDSs provide advice to researchers on public involvement at all stages of the research study. Many of the RDSs have also developed schemes whereby they offer funding to support service user involvement at the research design stage [8, 9].
The NIHR RDS for Yorkshire and Humber has an established Public Involvement funding scheme to assist researchers to gain involvement with the public during the development of research grant applications. Four calls are held each year, and the amount requested can be up to £500. Following receipt of an award, researchers are asked to write a short report outlining how the payment was used, and the extent and ways in which the public contributed to the development of the application. Researchers are provided with guidelines for the structure of the report (see Appendix). Papers describing schemes in other RDS regions [2, 8] have provided evidence of a link between public involvement and successful funding outcomes.
A previous paper which examined the Yorkshire and Humber scheme  provided details of the usage of the scheme, described the characteristics of applications and outcomes, and outlined three case examples of how the award contributed to successful grant applications. The purpose of the current study was to examine in detail the reports that are completed by researchers following public involvement activities. In this study we aimed to analyse reported views and experiences regarding the processes of public involvement, in order to investigate elements that might be barriers or facilitators to public involvement in research proposals.
We intended that the analysis process should fully include public involvement, and therefore adopted a group participatory approach to analysis of the reports on public involvement events, as outlined below.
Approach to analysis
The approach to analysis drew on techniques of qualitative content analysis and Framework Analysis. These techniques aim to derive meaning from text, by systematic classification and coding, to identify recurring patterns or themes . Content analysis can be used to describe a range of analysis techniques, including a simple counting of sections of text that contains similar words or phrases . Qualitative content analysis aims to go beyond counting words, to examine the language for meaning. In addition to qualitative content analysis, we drew on techniques of Framework Analysis  to ensure that the process of analysis was systematic and comprehensive. In this approach the data are organised into a chart of columns and rows (with each row being an individual document, and each column a theme) to assist with the identification and labelling of text into meaningful chunks.
Method of analysis
A workshop was arranged for the purpose of analysing the reports. Four members of the public with experience of being involved in research, and five local public involvement advisers and facilitators were invited to attend. Prior to the workshop the first author (SB) sourced and read all the reports that were available. As the length of the reports ran to many pages and included information relating to background literature and financial information, an initial extraction was carried out to copy sections of the reports that were relevant to a new document to reduce the burden on public members. There remained a large quantity of text following this process, and in order to share out the workload, six individual documents with different sets of data were created. SB broadly grouped the content of the reports into: input to the research topic; input to the proposed research methods; potential roles of public contributors in the planned study; benefits of public involvement; methods to recruit public contributors; and barriers and facilitators to public involvement. Each report was sent to a pair of workshop participants (one member of the public and one public involvement staff lead) two weeks before the event, with a request to read the text and note down anything that was interesting or important.
Ten participants attended the workshop. The session began with a brief introduction and warm up categorisation activity (on the topic of food and drink) to give participants practice and confidence in grouping written text together to develop themes and sub-themes. Following this, the group divided into the pairs who had received the same text. They discussed the data and wrote down themes that they had identified on sticky notes. Key themes across all the reports were discussed as a whole group and recorded on flip chart sheets. In an interactive process of framework development participants discussed and stuck their notes on the flip chart sheets, with sub-categories/themes developed by grouping sticky notes together.
Two weeks following the workshop the completed framework of themes and sub-themes was circulated via email to attendees asking for feedback on the categories, any themes that were unclear, or did not appear to be representative of the data that they had scrutinised. SB then re-read the data from the reports to populate the framework with text, and this was re-circulated for input from the group to develop the final analysis.
Framework of themes developed from the data
Added value of public involvement
Prior to study commencing
Input in regard to appropriate terminology
Ensuring clarity/user friendliness of written information
Ensuring appropriate outcomes
Acceptability of data collection methods/tools
Alerting to potential ethical/patient safety issues
Advice regarding research processes such as recruitment/drop out/follow up
Ensuring research conducted with sensitivity/empathy
Sharing experiences/emphasising value of individual experiences in study
Planning/designing public involvement
Prior to study commencing
Recruitment to public involvement event
Format of sessions
Recruitment to study advisory groups
Feedback to members on outcome/findings
Role of public members
Prior to study commencing
Feedback on accessibility of language
Intended during study
Project steering group/management panel
Scrutinise conduct of research
As patient interviewers
Involvement in analysis and interpretation
Review of documentation to be used
Assist with ethical approval documentation
Consult regarding any issues regarding recruitment/involvement
Help with any developing or evolving issues
Intended after study
Assist with dissemination
Assisting with writing up findings
Assist with translation into practice
Valuing public member contributions
Recognising expertise of members
Value for public members
Added value of public involvement
“They felt that research attempting to understand what people can do to maintain wellbeing would be very beneficial to patients”.
“Comments were made that it [the research topic] resonated very strongly with their own experiences”.
“It prompted us to think we should focus, in our proposal, only on improving post-diagnostic support, since this was a more discrete issue”.
“Based on these suggestions, I decided the next step will be to modify these relaxation methods” (the content of the relaxation tape provided to participants was changed).
“The intervention would be acceptable to patients”.
As a consequence, we have included this as a secondary outcome in the plan of care that all patients recruited into the study would receive” (an additional wellbeing outcome was included).
“Participants recommended some word changes and alternative phrasing to ensure it is fully understood by patients”.
“Other suggestions have been incorporated into a revised version of the Lay Summary”.
“From talking to attendees I understood how people had very different views on using outcome measures…therefore in the research proposal I have developed flexibility in the process of testing using outcome measures”.
“All patients were willing to undergo the extra tests”.
During these discussions some researchers reported that potential ethical or patient safety issues were raised, such as how abnormal test results should be communicated.
“As a direct result of discussions the focus of recruitment was altered, this had a sizeable impact on the design of the trial and altered the aims and objectives of the project”.
“There was a strong preference for recruitment to be initiated via a face-to-face approach in clinic because this would allow patients to ask questions about the study straight away”.
“All parents agreed that ideally the study follow up period should be at least 12 months because of changes over a 6–12 month period in a child’s life that can influence the sleep pattern”.
“All agreed they would rather complete the questionnaires at home”.
Researchers reported that public involvement had highlighted the need to ensure that studies were conducted with sensitivity and empathy. Involvement activities had also emphasised the importance of capturing varying individual experiences of healthcare in order to inform the study design.
As these reports were written prior to research studies commencing, this explains the lack of data reporting researcher perceptions of added value after a study. However, intended roles for public members within studies were described, and these data are outlined within the theme of public member roles below.
Planning/designing public involvement
“We had originally planned to involve 6–8 patients in this PPI work. Due to recruitment difficulties we only ended involving four patients. In the future, we will allow more time for recruitment and identify patients from more than one surgeon’s clinic”.
“Some families could not take part in our discussion groups as they were unable to make care arrangements for their other children. In addition, we also received late notice cancellations due to illness and other engagements”.
“Arranging the session in the holiday period probably accounted at least partly for the low attendance”.
“Maybe being longer as I felt there was a great deal more we could have discussed”.
“People would have liked to have seen the hand-outs before the meeting”.
“A member of staff or the researcher translated the meeting content for those participants and service users that did not speak English”.
“They were encouraged to contribute to discussion, and during the meeting their views were valued”.
“Two separate PPI meetings were convened to minimise the anxiety and stress”.
The importance of having well-developed facilitation skills was mentioned, in order to ensure that all participants had a chance to contribute, and to create an atmosphere that was enjoyable and interesting.
“A range of training would be needed including interpersonal skills and self-confidence in addition to training on research processes and skills”.
“Payment for time and out of pocket expenses will be provided”.
“Participants expressed suspicion of the idea of payment for participation” (involvement in advisory groups).
Researchers found that email was a popular way of communicating with public contributors, with some also recommending the use of texting to send reminders about meetings.
Role of public members
“I used this feedback to improve the application”.
“The research team are confident that the bid has been improved by the rich data collected during this meeting”.
“A second meeting was held, after a draft grant application and copies of the data collection tools had been circulated, to obtain feedback on specific aspects of the study design and grant application such as the proposed recruitment strategy, the choice of outcomes, and the content of the plain English summary”.
Another potential role for public members prior to a study was as a co-applicant. The possibility of this was mentioned by three researchers, although none of the reports described public members taking on this role.
“We plan for two patients from the PPI group to attend each trial management group meeting”.
“We also explained our plan for patient involvement activities if the project is funded, and invited the participants to form a patient involvement group that would support the research study”.
“To scrutinise the conduct of the research, ensure that resources are used appropriately and to monitor progress”.
“We intend to involve one or two of the participants from this focus group in the design of the trial; especially in the formulation of information sheets and consent forms”.
“Having pregnant women on the panel will enable us to discuss and address issues around recruitment and concordance from their perspective. This will enable us to alter or improve recruitment strategies”.
“The planned events are designed to engage a wide range of stakeholders and will form a key part of our definitive trial proposal development, dissemination, and translation into practice plan”.
“The original PPI group felt it would be acceptable to have a fluid membership of pregnant women in the PPI group”.
“They agreed that having more than one expert patient on the project group meant individuals could ‘share the load’ and liked the potential for ‘peer support’ between expert patients”.
Valuing public member contributions
“Interestingly our fears [regarding the potential for distress to participants] were largely unsupported by the expert patients. Instead the expert patients raised other issues they felt to be of greater importance”.
“I have assured the panel that all comments have been reflected upon and the comments have been included in the submission”.
“A lay summary document was produced which detailed the key messages to be taken from the discussion and how the proposed research will be developed as a result”.
“I felt good in the meeting and it gave me a chance to get to know people with these issues; and it gave me an insight into how other people are and I got to talk openly, as well as listening to other people’s feedback”.
“This is certainly helping me anyway, being part of this…knowing that it’s not just me in this world that it affects”.
“I hope my input was helpful for all the (those) involved”.
This analysis of reports written by researchers following public involvement activities in developing applications for research provides evidence regarding the perceived value of involvement during the research design process. The views and experiences of researchers described in this study provide further data to support the value of involvement of patients and the public at an early stage in developing proposals.
The research findings are also consistent with existing guidance for researchers on public involvement issued by organisations such as INVOLVE , and the Research Design Services . The results of the study provide a useful framework for researchers to consider public involvement at all stages when developing proposals, and enable them to draw on the experiences of other researchers. The group method that we used to carry out the data analysis was a useful approach to achieving involvement of individuals who have no previous experience of qualitative data analysis. The potential for lay individuals to contribute to data analysis is commonly recommended [12–14]. However, we have found few detailed examples of studies reporting methods used to facilitate members of the public contributing to data analysis (a notable exception is described by Gillard et al. 2012 ).
The Framework Method has been suggested as being appropriate for use in research teams where not all members have previous experience of conducting qualitative research , and our experience concurs with this. Our data related to a discrete topic, which may have made the process of analysis more straightforward. Other researchers have cautioned that the Framework Method deals less well with highly heterogeneous data .
A short debrief was held at the end of the analysis workshop, where participants were encouraged to reflect on the participatory process used. All participants valued the warm up exercise where they were asked to categorise different types of food as this demonstrated the categorisation process in an accessible way and helped to build confidence.
Group members reported that working in pairs to look at specific sections of data worked well. Pairs reported that each person brought slightly different insights and perspectives to the data. Breaking the data into manageable sections was also appreciated, as this made both the workshop tasks and the preparatory work more manageable. Having one person who had read all the reports in full provided valuable oversight, and meant that individual sections could be put into context if needed. One potential negative aspect of splitting the text into sections before the workshop, is that the data had already been filtered by a researcher, which could impact on people’s interpretations. Feedback from participants following the workshop was that reading larger sections of data would have been overly onerous, and that different interpretations of data were still possible from reading and discussing each other’s sections.
Everyone present had some previous experience of public involvement. This was positive in that it allowed the group to easily understand the topic and the aims of the project. However, this also created a challenge in that people moved easily into discussing their own experiences, and at times made some assumptions about the activities being described in the reports. Workshop participants had to challenge each other regarding this, and bring discussions back to the data. The group had worked together in the past, and therefore felt able to challenge each other when needed. Group members reported that this would be more difficult with a group who did not have a prior relationship. A learning point here, is that a certain amount of preparation and relationship building is necessary within a workshop where people are coming from different backgrounds, and where challenging each other’s perceptions is central to the process.
We recognise a limitation of the study is that the data came from researchers who had applied for, and achieved funding for public involvement activities. Their views and perceptions may therefore be more positive than researchers who have not recognised the potential benefits, or have not considered public involvement. The researchers were also responding to their funder, and may therefore be more inclined to be positive. The data were also analysed by a group of people who already had public involvement experience, and pre-existing views and experiences may have influenced their interpretation of the data.
It is also important to note that while the reports provide evidence regarding how involvement activities had contributed to development of the research proposal, the potential expected impact on the study itself (if it was successful in receiving funding) was based on researcher intentions rather than actual experience.
This work provides evidence of the value of public involvement in the development of proposals however, the impact during studies themselves requires further investigation. It would be helpful for example to gain a further understanding regarding how PPI elements of proposals have informed funding decisions, and how the intentions for PPI of researchers prior to a study, related to the actual involvement achieved. Research examining the implementation of plans for PPI in clinical trials  reported that while most of the plans for PPI had been implemented, some researchers spoke of using it as a means of “ticking the right boxes” during the application process. This study also found considerable variability regarding the extensiveness of planned PPI activity, and varying clarity with which plans were described. This lack of consistency regarding the quality and implementation of PPI in studies which have been accepted for funding, suggests that further investigation of PPI elements throughout the entire course of a research project would be valuable.
A recently completed study  has provided evidence that PPI during studies can have positive outcomes in the form of changes to study design, and improvements to recruitment and dissemination when enabling contexts are present. As the authors of this study suggest, it would be interesting to track PPI involvement not only through the course of a project, but also from the earliest stage of research proposals to longer term impact on practice.
Following examination of reports the RDS has continued its consideration regarding ongoing delivery of the funding scheme, and its evaluation . There are plans to follow up a sample of the award recipients including members of the public who had been actively involved in the research, to explore what happened later in the process, and whether the planned public involvement happened when studies commenced. We are also considering ways to seek views on the involvement activities from funders.
The findings of our study indicate that, while public involvement is highly valued, there may be a need for greater support regarding carrying out involvement activities, and the best methods for accessing potential lay members. Some researchers described the need for skills in facilitation, as the public involvement process was very different to their routine clinical interactions, for example one researcher noted “I need to improve my facilitation skills to learn how to better manage more dominant members of the group.” The RDS is considering how best to provide additional advice and support in order that researchers have skills to make the most of public involvement opportunities. For example information on how to structure sessions, examples of questions that could be asked, tips on managing group discussions, may assist researchers in preparing for sessions.
The findings of the study provide evidence of the value of public involvement during the development of applications for research funding. The results also indicate recognition amongst researchers of the variety in potential roles for the public in research, and the acknowledgement of how involvement adds value to studies. The use of the participatory approach to analyse the reports that we describe could be of value to others wanting to carry out collaborative data analysis.
Ethical approval was not required as the study analysed anonymous documentary data only.
The Research Design Service Yorkshire and Humber is funded by the National Institute for Health Research. The views and opinions expressed are those of the authors and do not necessarily reflect those of the NIHR, NHS or the Department of Health.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Boote J, Telford R, Cooper C. Consumer involvement in health research: a review and research agenda. Health Policy. 2002;61:213–36.View ArticlePubMedGoogle Scholar
- Carter P, Beech R, Coxon D, Thomas M, Jinks C. Mobilising the experiential knowledge of clinicians, patients and carers for applied health-care research. Contemporary Soc Sci: J Acad Soc Sci, 2013, doi:10.1080/21582041.2013.767468.
- Staley K. Exploring impact: public involvement in NHS, public health and social care research. Eastleight: INVOLVE. 2009.Google Scholar
- Boote J, Baird W, Beecroft C. Public involvement at the design stage of primary health research: a narrative review of case examples. Health Policy. 2010;95:10–23.View ArticlePubMedGoogle Scholar
- Mental Health Research Network/INVOLVE, Budgeting for involvement: Practical advice on budgeting for actively involving the public in research studies. MHRN and INVOLVE: Eastleigh, 2013. http://www.invo.org.uk/posttypepublication/budgeting-for-involvement. Accessed 21/8/15.
- National Institute for Health Research. http://www.rds.nihr.ac.uk. Accessed 21/8/15.
- Green G, Rein M. Building research capital to facilitate research. Health Res Pol Syst, 2013, 11:12, doi:10.1186/1478-4505-11-12.
- Walker D, Pandya-Wood R. Can research development bursaries for patient and public involvement have a positive impact on grant applications? A UK-based small-scale service evaluation. Health Expect, 2013, doi:10.1111/hex.12127.
- Boote JD, Twiddy M, Baird W, Birks Y, Clarke C, and Beever D. Supporting public involvement in research design and grant development: a case study of a public involvement award scheme managed by a National Institute for Health Research (NIHR) Research Design Service (RDS). Health Expect 2013, doi:10.1111/hex.12130.
- Hsiu-Fang H, Shannon SE. Qual Health Res. 2005;15:1277–88.View ArticleGoogle Scholar
- Ritchie J, Lewis J. Qualitative Research Practice: A Guide for Social Science Students and Researchers. London: Sage Publications; 2003.Google Scholar
- Hayes H, Buckland S, Tarpey M. Briefing notes for researchers: public involvement in NHS, public health and social care research, INVOLVE: Eastleigh; 2012.Google Scholar
- Research Design Service Yorkshire and Humber/National Institute for Health Research. Patient and public involvement in health and social care research: A handbook for researchers. https://www.rds-yh.nihr.ac.uk/ppi-handbook-2/. Accessed 24/8/14.
- TwoCan Associates. A review of patient and public involvement at the British Paediatric Surveillance Unit. London: British Paediatric Surveillance Unit, 2011. http://www.twocanassociates.co.uk/perch/resources/files/PPI%20Review%20Final%20Report%20TwoCan%20Report.pdf. Accessed 25/8/15.
- Gillard S, Simons L, Turner K, Lucock M, Edwards C. Patient and public involvement in the coproduction of knowledge: reflection on the analysis of qualitative data in a mental health study. Qual Health Res. 2012;22:1126–37.View ArticlePubMedGoogle Scholar
- Gale NK, Heath G. Cameron E, Rashid S. Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol 2013, 13:117, doi:10.1186/1471-2288-13-117.
- Buck D, Gamble C, Dudley L, Preston J, Hanley B, Williamson PR et al. From plans to actions in patient and public involvement: qualitative study of documented plans and the accounts of researchers and patients sampled from a cohort of clinical trials. BMJ Open 2014;4:12 e006400, doi:10.1136/bmjopen-2014-006400.
- Wilson P, Mathie E, Keenan J, McNeilly E, Goodman C, Howe A et al. ReseArch with Patient and Public invOlvement: a RealisT evaluation - the RAPPORT study. Health Serv Deliv Res. 2015;3:38.Google Scholar