Why were workshops on including patients in COS development needed?
As noted above, we are aware that patients are increasingly being included in ongoing COS development studies particularly as participants [9]. But less is known about the best methods for doing this. We therefore decided to run some workshops with delegates and patients at meetings of the COMET Initiative (see glossary of terms in Table 1). COMET meetings are held annually and attended by international COS developers and other stakeholders to discuss different approaches to COS development and to learn from each other. The purpose of the workshops on including patients was to start to explore what principles, methods and strategies COS developers might want to consider when seeking patient input into the development of a COS. This article describes the discussions that took place during three interactive workshops on ways to include patients in COS development, and to make sure that COS really do reflect the perspectives of patients.
How were the workshops organised and facilitated and who took part?
The three “Involving patients in core outcome set development: identifying the challenges and potential solutions” workshops were organised as part of the COMET Initiative meetings in Manchester (2013), Rome (2014) and Calgary (2015). The workshops were open to all COMET delegates who selected a preferred workshop from a list of available workshops. A total of 115 patients, researchers, clinicians and other stakeholders from 11 different countries registered for the “Involving Patients” workshops. We did not take a register of those who actually attended the workshops, or enquire whether attendees were patients, carers, researchers, clinicians etc., so we cannot give further details.
Before the workshops delegates received a brief description of the workshops emphasising the importance of including patients in COS development. The description also explained that workshops aimed to raise awareness of the challenges in including patients in COS development and to explore potential solutions, rather than ‘to teach’ participants ‘correct’ ways to do this. Each of the three workshops lasted 90 min and began with a 5–10 min presentation to set the scene. Delegates then organised themselves into breakout groups comprising approximately 4–6 individuals to discuss the challenges and solutions. A group of three or four patient facilitators circulated around each of the breakout groups to prompt discussion; each workshop had a different group of patient facilitators. A plenary session lasting approximately 20 min at the end of each workshop enabled the breakout groups to share ideas and experiences. The workshops were designed jointly by Heather Bagley (HB) and Bridget Young (BY), who also facilitated the workshops. The patient facilitators were recruited through various routes, including adverts on the UK National Institute for Health Research People in Research website [16] and through internet searches and personal contacts. All had experience of PPI in research and interests in COS, although very few had direct experience of COS development.
What topics were discussed?
Before the workshops HB and BY sent briefing notes to the patient facilitators to support them in their roles. The notes listed the five main questions (see questions 1–5 below) to structure the breakout discussions, as well as follow up prompts to help facilitators to stimulate discussion. Copies of the COMET plain language summaries on COS and Delphi exercises were also made available to the patient facilitators (see http://www.comet-initiative.org/resources/PlainLanguageSummary). HB and BY met with the patient facilitators before each of the workshops to clarify the aims and discuss how the sessions would be run. The workshops evolved over time. For example, after the first workshop the briefing notes were refined to steer future discussion away from simply revisiting general guidance on patient and public involvement in research (which is increasingly available, for example, from INVOLVE [17] in the UK) and more towards the particular challenges of seeking patient input in COS development. The subsequent workshops also aimed to focus discussion on the challenges in COS studies of both: a) patient participation (i.e. where patients contribute data) and b) patient involvement (i.e. where patients take on research partner roles). At the end of each workshop we collected notes from breakout groups. We used these and some notes that a facilitator had made on each of the plenaries to prepare summaries of the discussions in each workshop. These summaries were checked by HB and BY and have informed the preparation of this article. Some of the patient facilitators also reviewed and commented on drafts of this article and we have listed their names in the acknowledgments section. In the sections that follow we take each of the five main questions in turn and outline the discussions that took place across the three workshops in response to these questions.
Question 1: what principles and practicalities are important to consider in accessing and sampling patients?
Delegates pointed to the need to distinguish between including patients as participants in COS development and including them as research partners in the process of planning, running and disseminating a COS study. Groups discussed how COS developers may find the patient participant role best suited to individuals who can offer a personal experience of the condition at hand, whereas the patient partner role might be more suited to those who, for example, have knowledge and skills relevant to providing a patient perspective on the design and delivery of a COS study. Delegates felt that training and support should be provided for patient partners in COS development so that they are in a position to give meaningful and credible input. However, for some delegates the idea of formally training patient participants in COS development was controversial due to fears that it might influence or undermine the patient voice. It was nevertheless recognised that patient participants would need some form of support, particularly for ‘remote’ methods of COS development, such as online or postal Delphi surveys, that do not involve face-to-face or telephone contact. Potential for bias in COS development from conflicts of interest arising from sources of funding for certain organisations or activities, were discussed. Delegates felt that these could apply equally to patients, researchers, clinicians and other stakeholders, and so should be managed similarly across these groups.
There was considerable uncertainty regarding questions such as “how many patient participants are needed” in COS development or “what proportion of patients relative to other stakeholders should be included”. It is particularly important to gain a diversity of perspectives when patients and other stakeholders participate in COS development. There is currently little guidance on how many stakeholders are needed in total, or on the size and composition of each stakeholder group relative to the others, but there are indications that these factors can influence what outcomes are rated as important [18, 19]. We intend to explore these issues in future workshops. In contrast there was strong agreement on the importance of the principles of inclusivity and fairness for guiding COS development. This was reflected in extensive discussions on how to address the practicalities of making COS development activities accessible to patient participants and research partners (e.g. accommodating commitments to work, school and clinic appointments) and sensitive to the range of needs that particular groups might have arising from their illness, condition or social circumstances (e.g. need for COS development materials to be in different formats). These considerations also extended to the choice of routes for accessing patients as participants. One group suggested that community outreach and the technique of maximum variation sampling (whereby patients are sampled to include those from a wide range of backgrounds and with a wide range of relevant experience) might help in achieving diversity. Others focussed on the trade offs between accessing patient participants via health clinics versus patient organisations. In countries with publically funded healthcare systems like the UK, health clinics tend to be used by most patients and so may help in ensuring diversity. Patient organisations may in theory be open to all, yet in practice these might be accessed by a relatively narrow spectrum of patients. Social media was identified as another option for accessing patients, yet there are indications that social media samples lack diversity [20]. As COS development aims to produce generalizable knowledge, it is usually regarded as a research activity. Therefore, the need to seek ethical approval for all participants, regardless whether they are patients, clinicians or other stakeholders, was also emphasised.
Question 2: what principles and practicalities are important in selecting a method for patient input in COS development?
It was acknowledged that different types of methods such as surveys, interviews, focus groups and consensus meetings may be needed and that COS development would usually comprise several different stages. Delegates advised that it was essential for COS developers to make sure the methods suited the needs and preferences of the particular patient groups whose input was being sought and the sensitivity of the topic. Involving patients from an early stage as partners in COS study development was regarded as crucial in making sure both the method and the mode of patient participation in COS development (e.g. whether face-to-face, online, by phone etc.) were suitable. Some groups favoured methods of participation such as qualitative interviews that allowed two-way conversations over Delphi surveys, particularly at the start and end of the process. Delegates who favoured qualitative methods thought these were good for capturing the complexity of the patient perspective, the language that patients used and understanding why some outcomes mattered more than others. Some delegates also commented that Delphi surveys were intimidating for patient participants. However, others pointed out that Delphi surveys could help to widen patient participation and promote transparency, and by offering anonymity, Delphi surveys helped to minimise the influence of power differentials between different stakeholders. Concerns were voiced about the long number of outcomes included in some Delphi surveys that patient participants would have to go through and score, and it was thought that this could be off putting for many. When working with patient groups such as children, people with learning disabilities, cognitive impairments or brain injuries, delegates advised that care was needed to avoid conflating the perspectives of patient and carer, and warned of how carers can easily ‘drown out’ the perspectives of patients.
Question 3: what are the best ways of explaining the concept of a ‘core outcome set’ to patients and what types of questions can be used to elicit their perspectives?
The extent to which patients need a detailed explanation of COS will depend on the methods used to seek their input and whether they are acting as participants or as research partners. An advantage of qualitative interviews or focus groups for seeking the input of patient participants is that these methods allow patients to ‘set the agenda’ and talk about their experiences as they would in everyday life. This means that patient participants can take part meaningfully in COS development without needing a detailed understanding of what an outcome is, or the reasons why COS are needed [21]. Other types of input to COS development, such as participation in Delphis or consensus meetings and patient partner roles, require patients to engage more directly with concepts such as outcomes and research processes. Delegates felt strongly that patient participation within Delphis and consensus meetings needs to be meaningful and on an equal footing with other stakeholders. It follows that COS developers need to ensure that COS concepts, and methods of COS development such as Delphi studies, are explained in ways that patient participants can understand, yet without unduly influencing their responses.
Delegates advised that patient research partners had an important role in developing clear explanations of COS and associated concepts. They recognised how challenging it could be for COS developers, who were familiar with the concepts, to convey these, particularly when the concepts were far removed from the experience of most patients. In explaining COS to patients, delegates advised developers to focus on the overall goal, that is, how COS would benefit patients. Simple analogies, such as the impossibility of comparing ‘apples and pears’, were thought to be useful in explaining COS, and some delegates felt it might also be helpful for patients, particularly those in partner roles, to learn about systematic reviews and clinical trials to really understand why COS are needed. The importance of being transparent about the goal of identifying only ‘core’ outcomes was emphasised, with delegates commenting on the difficulty that patients and other participating stakeholders might otherwise have in ‘letting go’ of certain outcomes if this was not clear from the outset.
Linked to this, delegates pointed to some fundamental points of tension between the perspectives of patients and those of researchers or clinicians, and the challenges these different perspective posed for COS development. One delegate astutely observed that many patient participants and partners might understandably assume that researchers had ‘worked out what to measure years ago’ and pointed to how patients might be bewildered on first hearing that this is not the case. Similarly, when asked what is most important to them, delegates anticipated that some patient participants might respond by saying “a cure” and that nothing else really mattered to them. Delegates felt that COS developers needed to be able to respond to the other issues and queries that COS development might raise for patient participants and partners. For example, it is natural for patients to want to be cured. Yet, knowing that cure is a very distant prospect for many chronic illness, it is natural for COS developers to want to focus on more achievable outcomes. Developers need to be prepared to acknowledge the importance of cure to patients, while supporting them to consider what other outcomes matter and without inadvertently influencing them.
Methods such as online or postal Delphis were thought to require particularly careful preparation if these were to make sense to patient participants, and delegates again pointed to how patient research partners could help with developing the content and meaningfulness of items and fine-tuning the language used. As we elaborate below, delegates felt that face-to-face methods of consensus development called for first-rate facilitation skills, as well as careful preparation to ensure that patient participants were well supported. Delegates suggested a number of questions to elicit patient perspectives on what outcomes are important. These included: ‘what is the most limiting aspect of your condition’, ‘what would you hope for from a new treatment’, ‘what sort of things would you think about in making a decision about what treatment to take’ or ‘what sorts of changes in your symptoms/condition/life would tell you that a treatment was actually helping you’.
Question 4: How can we maintain the input of patients over time?
While interviews or focus groups often require input from patient participants on only one occasion, other types of input such as patient participation in Delphi surveys and patient partner roles, require input on two or more occasions. So, maintaining the input of patients over time either as participants or partners is vital. Delegates thought that careful planning at an early stage was key here, and reiterated the importance of ensuring that the methods, design and mode of participation were suited to the patient group of interest. Managing expectations from the outset about timescales and keeping patient participants and partners informed of progress were seen as other important ways to guard against drop out or disengagement from the process. Beyond this, delegates thought that attending to common courtesies could go a long way to maintaining the input of patients as both participants and partners. These included building rapport with patients, showing appreciation for their contributions, reimbursing costs and ensuring that feedback was provided and written in plain language. Delegates also advised COS developers to consider what might incentivise patient participants to remain over the course of a project. For some participants this might entail doing the COS quickly and minimising the gap between Delphi rounds. The offer of payments or vouchers were thought to be helpful in maintaining patient input, but delegates also pointed to how building a sense of curiosity and excitement about COS development, as well as a sense of ownership of the process, might also help to maintain interest over time.
Question 5: should patients be brought together with other stakeholders to achieve consensus? if so, how should this be done?
Delegates differed in their views about whether patient participants should be brought together in face-to-face meetings with other participating stakeholders to achieve consensus, or whether it was best for patients to meet separately. Those who favoured a separate approach were concerned about more powerful voices dominating over patients. Those who favoured bringing patient participants together to meet with other stakeholders felt it was important to avoid assuming that patients would necessarily be dominated by other groups and commented that if meetings provided an enabling environment and strong support and facilitation, patients would be capable of putting their perspectives across. Achieving an enabling environment in consensus meetings was thought to depend on good preparation, and crucially, on the skills of the facilitator. Delegates tended to favour the idea of consensus meetings being facilitated by someone who was an expert in facilitation, rather than by an expert in the particular condition or treatment. A few mentioned the possibility of having a patient partner and researcher co-facilitate meetings. Linked to this, delegates raised questions about how divergences between the perspectives of patients and other participating stakeholders would be reconciled in practice. They counselled that all parties might feel uncomfortable when their opinions were challenged. Nevertheless, they were clear that all participating stakeholders needed to be genuinely open to challenge, prepared to explain the reasons behind their opinions and to reconsidering their opinions.
