We have categorised our findings in relation to the contextual and process factors that influenced outcomes and impacts. These are discussed in turn.
Contextual factors influencing PSP impact
The JLA PSPs in our sample had been established by different groups for different reasons, and this was an important factor explaining their outputs and impacts. The PSP’s goals were influenced by the mission, capacity, practical constraints and influence of the lead organisation, a charity, patient organisation or academic institution. Some PSPs had set out to raise the profile of an underfunded research area. Others had aimed to broaden the scope or rank competing priorities in an area that was already popular or where priorities were contested. What counted as success therefore depended on the PSP’s starting point.
The wider political context to a PSP also influenced its process and outcomes. For example, in the chronic Lyme Disease PSP (a clinical condition whose origin and treatment is contested), the patient organisation was unable to recruit any clinical researchers for the Steering Group. The PSP Lead from Lyme Disease Action, reported that that some clinical researchers seemed to feel that the PSP “shouldn’t even be happening”. Seven years later, the Lyme Disease PSP priorities have yet to be addressed by any research in the UK, although some interest has been shown by other countries. Simply running a PSP cannot resolve historical or existing tensions in a research community, though it may help to move some debates forward.
Post-PSP processes influencing impact
We have summarised these findings under seven broad themes related to post-PSP activity: planning for the end of the PSP, disseminating the Top 10, persuading funders and researchers to respond to the priorities, using JLA PSP research priorities to influence strategy and funding decisions, translating priority topics into research questions, evaluating the impact of a PSP, and the post-PSP impacts on people and organisations.
Planning for the end of the PSP
Following the recommendations in the current JLA guidebook [5], most PSP Leads in our sample had developed a comprehensive communications plan to raise awareness of their Top 10 research priorities and often the shortlist- considered at their final workshop. However, many PSP Leads questioned whether simply informing others of the outputs was sufficient to have impact. On reflection, they concluded they needed a more strategic approach to their post-PSP activity:
“ … there's no point in identifying your Top 10, if you don’t have a plan of what you're going to do with them afterwards. We underestimated the work that’s required if you want to really make the best use of the results.” - PSP Lead, Charity, Funder
Some reported having overlooked issues that could have been addressed in early planning, such as: considering how to achieve each of the PSP’s strategic goals; agreeing who will ‘own’ the PSP priority list; considering how the PSP will respond to non-research questions (for example, statements about deficiencies in current services); and capitalising on the skills, experience and networks that had been developed through the partnership.
Others described having made such plans. For example, the charity-led Mental Health in Children and Young People PSP (2018) aimed to raise professional and public awareness of mental health issues in young people as an additional outcome. They therefore included an All Party Parliamentary Group event in Parliament in their dissemination plan. The Type 2 Diabetes PSP (2017), led by Diabetes UK, built alliances within their organisation to facilitate dissemination of priorities (via the communications department) and address topics relating to service improvement outside the remit of the PSP (via the quality department). They also engaged external partners at the beginning with a view to developing joint funding calls at the end.
Disseminating the top 10 priority list
Typically, all or most partner organisations involved in a PSP helped with dissemination of the outputs. Large organisations leading PSPs usually had communications teams to provide expert advice and resource for this. Target audiences included patients, carers and the public, research funders, policy makers, health professionals, lay media and researchers. Interviewees felt that it was important to convey details of how Top 10 lists are generated. They considered that few other researchers, journal editors or academic reviewers fully understood or valued the rigour of the process:
“I don’t think they realise that the process has been quite long, very involved, and involved hundreds of people in different ways.” - Researcher
Successful dissemination strategies included: a press release summary of the Top 10 embargoed until launch day; a launch event e.g. at a scientific conference; a plain English summary for patients and carers; an academic journal article; social media activity e.g. releasing one priority per day over 10 days, or a guest blog; hosting a web page; using email lists of PSP survey participants; giving talks to patient, clinical and scientific groups; a funding organisation alerting current and past grant holders; using the channels and networks of partner organisations e.g. their newsletters; and promoting findings by word of mouth.
Many PSPs involved patients, carers and clinicians in their dissemination activity, based on the rationale that this gave the messages greater credibility and authenticity.
Some PSPs in our sample had successfully published their findings in academic journals, hoping to reach researchers in the field. However, some had experience of having papers rejected, because the identified priorities were not considered to be ‘new knowledge’. Some interviewees suspected this was because the JLA approach was no longer seen as novel. In some cases, the PSP team lacked the expertise to write academic articles and navigate the publication process. They saw this as a missed opportunity.
“ … all we could do was link to the article on our website … a clinician, if they see that the link is to a patient website, they’re not going to take it so seriously, but if we had had an [academic] paper, that would have been an anchor, more trustworthy from their point of view.” - PSP Lead, Charity, Non-funder
One interviewee expressed uncertainty about the status of priority setting as a topic for academic publication, especially in relation to the narrow formats deemed appropriate for many clinical journals:
“I don’t think any journal’s going to be interested in our Top 10 … It doesn’t seem scientific, there’s no hypothesis. There isn’t really a research question … it's more social science isn't it?” - PSP Lead, Academic Institution
PSP Leads reported varied experience in achieving national news coverage. Several had been told that their priority list was not “newsworthy”. They felt that science journalists were more interested in reporting scientific ‘breakthroughs’ rather than how the research question came about. The PSPs that had received mainstream media coverage had made links with reporters with personal or family experience of the condition.
Some interviewees concluded that the main audience for Top 10s was researchers and more effort was needed to reach them. While some had produced a lengthy report for their patient/carer community, they were doubtful of its value and concluded that a short report would be sufficient.
“It’s this thing about charities constantly churning out PDF reports and you need to make sure you’re asking yourself ‘What is the point, right? Like who is reading this?’” PSP Lead, Charity, Funder
Persuading others to respond to the top 10
The key stakeholders that PSPs aimed to influence were funders, researchers and donors. In order to get their priorities addressed by external funders, PSP partners first needed to identify whom to influence and then target activities towards these individuals or organisations. However, this sometimes proved challenging – for example, if the funding body did not recognise the JLA process or value its outputs.
Interviewees from external funding bodies emphasised that JLA priority lists were only one of many influences on their portfolio, which typically took account of the organisation’s remit and mission, its accountabilities to a wide range of stakeholders, prevailing scientific and policy priorities, and the opinions of review panels – as well as the amount of funding available: “Sometimes it’s a question of who is the best funder of this research … and often that’s not us.” – Funder.
Involving external funders directly by inviting them to Steering Group meetings or workshops, or involving them indirectly (e.g. by using their premises for meetings) was not found to be sufficient by PSP Leads to influence those funders’ decisions. At best, it helped to start a relationship. Greater value seemed to come from PSP partners working closely with funders in the post-PSP phase – for example to develop themed calls (see below).
PSP Leads emphasised the value of using influencing skills at this stage, which might require involving different people from the team that ran the PSP. Whilst ideally a PSP would include individuals with all the relevant skills, several PSPs in our sample either had not planned for this, or lacked resources. PSP Steering Groups sometimes felt exhausted by the time they “got their PSP over the finishing line” (Advisory Group Member), and because funding had ended, key individuals had little time or energy for follow-up.
Loss of momentum at the post-PSP stage was a particular problem for PSPs which lacked a strong sponsor such as patient organisation or charity. Cellulitis, for example, is a potential complication of numerous other conditions. No organisation existed for cellulitis itself. As the PSP Lead explained, this meant that the Cellulitis PSP, “… didn’t have a critical mass of people willing to keep it up in the air.” This was in contrast to the same interviewee’s experience of leading the Eczema PSP which had the strength of the Eczema Society behind it.
In terms of influencing researchers, our interviews suggested that researchers tended to use a Top 10 priority to strengthen the case for a study they already planned to do. Some said the JLA PSP gave them confidence and motivation to persist with applying for a grant on a prioritised topic. We heard no reports of researchers changing their research plans to pick up a PSP priority, nor of researchers stopping pursuing an area of interest because it was not on a priority list.
"It [the Top 10] was great for including in these sections of the grant and fellowship applications about what is the impact of your research going to be, and what’s the need for this research. - Researcher
Where an academic institution had led a PSP, the partnership was keen to emphasise that that the Top 10 list was not ‘owned’ by them, but meant for all researchers in their field. In one such example, working in partnership with a charity was found to be an effective means of reaching and potentially influencing others. In this case, the charity’s CEO provided the missing skills and links with the wider research community, as well as acting as an ‘honest broker’ from "an organisation that doesn’t have any skin in the game" [PSP Lead, Charity, Funder].
Finally, in relation to donations, some of the patient interviewees felt that a Top 10 could provide an effective focus for fundraising by patients and carers. Some charity-led PSPs reported that the JLA process had indeed helped to convince their donors that specific research projects were worthy of their support:
If you can show with clarity that you’re representing the priorities of your people … that is enormously powerful and it gives you enormous confidence in terms of going out there and speaking about what you do, especially with donors … -PSP Lead, Charity, Funder
Using the priorities to influence funder strategy and decisions
When a PSP was led by an organisation that funds research, there were several ways in which priorities were used to influence its own funding decisions and those of other funders: integrating these into organisational research strategy; directly funding research studies; sharing the list with other related funders and seeking their input to joint programmes of research; putting out themed calls for research proposals; integrating priority topics into open calls for funding applications; or using the priorities as a criterion when judging the importance of research proposals:
“[The JLA PSP has] given us justification to prioritise and rationalise with the research community, because people with [this condition] have told us what they want us to do. What was also important was making sure our Board of Trustees knew about those ten priorities, because now we've got a Board that understands and wants us to report against them.” - PSP Lead, Charity, Funder
Some funders had successfully worked together to develop research calls on shared priorities e.g. Parkinson’s UK joined forces with Marie Curie and other charity PSPs where a question about managing incontinence featured in their Top 10. Several of the PSP Leads interviewed from charities, were also exploring the possibility of their organisation putting out a joint call with the NIHR.
Some Top 10-themed research calls generated a poor response leading to the view that the research community lacked the skills, capacity or willingness to address the topic. Charity funders have taken additional steps to bring researchers together to carry out foundational work to develop new areas of research. For example, the MS Society worked with research groups to develop a plan for a clinical trial that was ultimately funded by a charity in the USA. In another PSP, prevention and minimising risk emerged as a priority topic, but capacity in that area was limited, so the charity organised an international workshop to develop recommendations for follow-up work.
Whilst one or two interviewees felt that all funded research should reflect priority topics, others felt that patients, carers and clinicians involved in the process may lack insight about potentially valuable research avenues: “There will be some basic science questions that were never going to be brought up in a PSP which could be key to scientific and medical progress” – PSP Lead, Academic Institution. None of the charities interviewed for this study had made the decision to only fund JLA PSP priorities. In open funding calls, JLA priority lists were one of several criteria used by reviewers to assess applications.
Some interviewees raised concerns that researchers might use a JLA PSP priority to ‘game the system’. For example, by squeezing their preferred research idea into a broad umbrella priority topic, or claiming that their proposal addresses a JLA PSP without justifying this in depth. Some charity funders assumed that the involvement of patients and carers in the review process provided an adequate check. However, they were uncertain how any grant reviewers assessed the researchers’ claims in practice. Many interviewees also commented that addressing a Top 10 topic did not mean that researchers could then ignore the patient voice in the subsequent research process. On the contrary, as one PSP Lead described, “the JLA is the beginning of an involvement process [with patients]” - PSP Lead, Charity, Funder.
Translating a priority topic into a research project
Translating a Top 10 priority into a research project has three steps: (1) mapping which aspects of the topic remain unanswered; (2) developing a focused research question from the broad priority (where necessary); and (3) designing a project to address the question. Many of the PSPs in this sample had hosted workshops with researchers to carry out this translation work. To our surprise, few had involved patients and carers, though several had involved clinicians. Yet our data suggested that patient and carer involvement at this stage could reduce the risk of priorities being misinterpreted and translated back into conventional clinician and researcher framings.
“Question eight on our [Top 10] list is about how parenting styles affect treatment outcomes for young people with mental health problems. And it seemed when we were talking to funders that they got the wrong end of the stick about this … A lot of the research that is out there is about how parents who have a mental health problem affect their children. That’s not what this question is getting at all … ” PSP Lead, Mental Health in Children and Young People
One PSP Lead described a situation in which a funder had not worked with anyone from the PSP, and had developed a poor-quality call as a result:
“The funder did end up putting out a call on one of the questions, but they didn’t speak to anyone beforehand … and the call was so badly worded that no one could put a research proposal together to answer it. It just went unfunded.” - PSP Lead, Charity, Funder
In another example, the NIHR put out a commissioning call in response to the priority topic ‘the management of continence problems’ from the Childhood Disability PSP. The call asked for a survey of current NHS practice in this area. Parents of disabled children who worked with researchers to respond to the call pointed out that the proposed survey would fail to identify any measures taken by parents and families themselves. They included this omission in their successful application, and two of the parents became co-investigators on the funded study.
Evaluating the difference JLA PSPs are making to research
Interviewees from research charities reported assessing the impact of their PSP by monitoring how many grant applications addressed a priority topic. Some did this by including a tick-box in their application form and suggested that public funders such as NIHR follow suit. Many PSP Leads had kept track of which priorities had been addressed by funded research. For example, the Mild to Moderate Hearing Loss PSP identified nine such projects, including studentships, fellowships, systematic reviews and a feasibility study. However, it is not helpful to judge the impact of JLA PSPs simply in terms of how much funding is allocated to their priorities, since some research topics can be addressed with modestly-funded studies, while others require an expensive clinical trial.
Some PSPs had been able to share details of research progress with their patient, carer and researcher communities. For example, the Tinnitus PSP (2012) produced a research update report for its members 5 years later [17]. Two of the patient interviewees said they found it hard to find the information they would like about progress. It was unclear who had responsibility for keeping such information up to date and whether it should be held by the JLA Secretariat, PSPs, researchers or funders.
Transforming: the impact of PSP involvement on individuals and organisations
The experience of going through the JLA process was reported to have a profound and mostly positive impact on the people and organisations who took part. Patients and carers said they felt they had made a unique and important contribution to their PSPs. They described gains in knowledge and confidence which had affected their own lives and work, and had led to an appetite for subsequent involvement in research and campaigning, although not all had been able to find such opportunities:
“It was the first time that I realised that patients should and could have a say in the whole healthcare ecosystem. They weren’t just victims or the ‘done to’ but that they could positively impact what’s going on.” - Patient participant in a PSP
Clinicians who had been involved in PSPs reported that the experience had changed their clinical practice, making them more aware of patients’ concerns and highlighting aspects of healthcare that need improving. In some cases it enhanced their own visibility, status and credibility amongst peers. They also felt more able and confident to involve patients and carers in their work, and expanded their ‘involvement’ networks:
“The PSP brought me into contact with a patient charity … I heard stories that I would never have heard otherwise, about how doctors relate to patients, and how the words used are important such as giving people fear, hope or being realistic.”
-Clinician lead, Inherited Anaemias PSP
Similar transformational changes were described by organisations that had taken part in a PSP. The experience was reported to have enhanced their status and credibility, for example, one charity described being seen as “the” patient voice for their condition in the UK after their PSP. Others reported a change in their organisations’ cultures and values towards more patient-centred and collaborative ways of working. PSPs often left a legacy of infrastructure, policies and projects to support ongoing patient involvement. Newly built or strengthened networks encouraged partner organisations to continue working together on areas of common interest, and a shift to a more collaborative way of working led to new approaches to working with funded researchers.
Finally some PSPs reported success in transforming national policy. The Tinnitus PSP identified an absence of clinical guidelines for tinnitus in children and drew on the PSP networks to convene an inter-professional group to draft new guidance. The Canadian Dementia PSP directly influenced the Canadian government’s dementia strategy to highlight the issue of stigma and the need for early treatment [18].
