Skip to main content

Defining and evaluating novel procedures for involving patients in Core Outcome Set research: creating a meaningful long list of candidate outcome domains

Plain English summary

Outcome domains are aspects of a condition that matter to patients and clinicians and can be measured to assess treatment effects. For tinnitus, examples include ‘tinnitus loudness’ and ‘ability to concentrate’. This study focuses on the first stage of agreeing which outcome domains should be measured in all clinical trials of tinnitus. Crucially, it involves identifying outcome domains, prior to a voting process. This article describes how we effectively involved patients in that study design process, and reflects on the impact of their input.

The study first compiled a long list of all possible outcome domains before asking interested parties, including patients, to vote which ones to include. Ensuring patients fully participate in this process holds unique challenges as it can be long, repetitive and its purpose far removed from their needs. These challenges may be addressed by involving patients in designing the research. There is evidence that other research teams are doing this, but its reporting is not detailed enough to guide others. Our paper seeks to address this.

We describe how we involved patients (people living with tinnitus) in creating a long list of outcome domains that we included in our study. We also reflect on the benefits this brought. Two patients partnered with us in designing the survey. We also consulted an independent patient review panel. Involving patients reduced the list of domains included in the survey and made domain names and associated descriptions clearer. Our resulting survey performed well in recruiting and retaining patients as participants.


Background Tinnitus is a complex audiological condition affecting many different domains of everyday life. Clinical trials of tinnitus interventions measure and report those outcome domains inconsistently and this hinders direct comparison between study findings. To address this problem, an ongoing project is developing a Core Outcome Set; an agreed list of outcome domains to be measured and reported in all future trials. Part of this project uses a consensus methodology (‘Delphi’ survey), whereby all relevant stakeholders identify important and critical outcome domains from a long list of candidates. This article addresses a gap in the patient involvement literature by describing and reflecting on our involvement of patients to create a meaningful long list of candidate outcome domains.

Methods Two Public Research Partners with lived experience of tinnitus reviewed an initial list of 124 outcome domains over two face-to-face workshops. With the Study Management Team, they interpreted each candidate outcome domain and generated a plain language description. Following this, the domain names and descriptions underwent an additional lay review by 14 patients and 5 clinical experts, via an online survey platform.

Results Insights gained from the workshops and survey feedback prompted substantial, unforeseen modifications to the long list. These included the reduction of the number of outcome domains (from 124 to 66) via the exclusion of broad concepts and consolidation of equivalent domains or domains outside the scope of the study. Reviewers also applied their lived experience of tinnitus to bring clarity and relevance to domain names and plain language descriptions. Four impacts on the Delphi survey were observed: recruitment exceeded the target by 171%, there were equivalent numbers of patient and professional participants (n = 358 and n = 312, respectively), feedback was mostly positive, and retention was high (87%).

Conclusions Patient involvement was an integral and transformative step of the study design process. Patient involvement was impactful because the online Delphi survey was successful in recruiting and retaining participants, and there were many comments about a positive participatory experience. Seven general methodological features are highlighted which fit with general principles of good patient involvement. These can benefit other Core Outcome Set developers.

Peer Review reports


The core outcome measures in tinnitus (COMIT) initiative

This methodology article is based on experience involving the public in the COMIT’ID study. COMIT’ID stands for ‘Core Outcome Measures in Tinnitus – International Delphi’ and used a modified Delphi survey to seek consensus between different stakeholder groups about what outcome domains are critically important when deciding if an intervention for tinnitus is working. The COMIT (Core Outcome Measures in Tinnitus) initiative is led by members of the European TINnitus NETwork (TINNET) to develop standards for clinical trials in tinnitus [1, 2]. A core outcome set establishes standards for outcome selection and reporting in clinical trials of interventions to improve the wellbeing of people with this lifelong condition. An outcome can be viewed in two parts. First, the outcome domain refers to what aspects of tinnitus matter to patients and clinicians. Outcome domains refer to any aspect of tinnitus that is or can be experienced by a patient, such as ability to concentrate, sense of control, or impact on work. Second, the outcome instrument refers to how that domain is to be measured. Throughout this report, the term “outcome” refers to the general construct which includes both concepts of what and how to measure, while the term “outcome domain” or “domain” is restricted to the concept of what to measure. The list of critically important outcome domains - known as a Core Outcome Set (COS) - forms a minimum set of outcome domains that should be measured and reported in every clinical trial [3, 4]. A minimum set means that researchers must report on the same core outcome domains, whilst remaining free to collect and explore other outcomes too. COMIT’ID aims to develop three separate COSs for tinnitus: one for sound-based intervention outcome domains; one for psychology-based intervention outcome domains; and one for pharmacology-based intervention outcome domains. These COSs will be relevant for adults with chronic subjective tinnitus that should be measured and reported in every clinical trial of these interventions [5]. The long-term goal is to standardise what is measured when sound-based, psychology-based, or pharmacology-based tinnitus interventions are tested, so that data can be compared or combined across studies.

There are different ways that patients can be involved in the process of designing, running and disseminating a COS study. While the term “patient” typically refers to a person who is receiving or registered to receive medical intervention, “patient” is used here to refer to anyone who has lived experience of tinnitus. This article documents a set of methods for involving patients in creating the long list of tinnitus-related outcome domains for round 1 of the Delphi survey. These methods form the first stage of the consensus process to create the three COSs. The article also reflects on the impact of that particular involvement. Patients who took this role are referred to throughout as ‘Public Research Partners’, although others may use terms such as ‘advocates’, ‘representatives’, ‘contributors’, ‘surrogates’ or ‘community stakeholders’ etc. [6, 7].

Why are core outcome sets (COS) needed for tinnitus?

Tinnitus is a subjective condition for which patients experience a diversity of complaints. Hence, there is no straightforward outcome. Some tinnitus outcomes are specific (for example, the perceived pitch of the tinnitus) while others are broad (for example, tinnitus-related quality of life). One of the challenges is that different studies evaluating interventions for tinnitus often measure and report different outcomes [8]. This makes it very difficult to compare results between studies. What is urgently needed are specific discussions around the major therapeutic approaches for tinnitus (namely sound therapies, psychological interventions, and pharmacological therapies) because they do not necessarily target the same tinnitus-related complaints [2, 5]. For example, if all sound-based intervention studies reported the same outcomes, then all the results could be compared and combined. A methodological consensus regarding what aspects of tinnitus should be measured would help to make sense of all the knowledge produced and thus improve the rate of progress in developing interventions [2]. In addition, involving patients in developing these outcome reporting standards would go a long way to ensuring its relevance to end users [2, 5].

Defining candidate outcome domains

Defining a COS involves working with various stakeholders, such as health care users, health care practitioners and commercial representatives, to prioritise large numbers of outcome domains and achieve consensus about the minimum set. Various methods have been used to identify important outcomes when developing a health care COS and there is insufficient evidence to determine which is the most appropriate or efficient [9]. Delphi survey methods are one of the most frequent approaches used [9]. In a conventional Delphi survey, participants nominate outcomes in round 1 to be considered in subsequent rounds, but a modified Delphi survey is becoming a popular way to reduce the burden to participants [9, 10]. In a modified Delphi survey, a comprehensive list – called a ‘long list’ – of all possible outcomes is first identified through a scoping process such as a systematic review of previous clinical trials, and/or patient-centred responses collected through interviews or focus groups. This long list forms the first of several rounds in which participants rate the importance of each and every outcome [9, 11]. As part of round 1, it is generally recommended that participants be given opportunities to suggest new outcomes which they think are missing and submit feedback or personal perspectives about specific outcomes [3, 9, 10]. This enables the team to be confident that they are representing the perspectives of all relevant stakeholder groups and to gain a greater understanding of why a particular outcome might be deemed important.

Procedures for involving patients in COS development

From a series of discussion-based workshops with patients, Young and Bagley [7] reported on how including patients as participants and as Public Research Partners holds unique challenges. Workshop delegates advised that Public Research Partners have an important role in developing clear explanations of COS and associated concepts because these can seem far removed from the experience of patients. Delegates also cautioned that the long list of candidate outcome domains found in some Delphi surveys can be off putting for many patient participants, and so Public Research Partners can again play an important role in highlighting this risk and helping to work out ways to minimise its occurrence. If not adequately addressed at the study design stage, these challenges could have a negative impact on accessibility of the Delphi survey to patients. Indeed, low rates of recruitment have been reported in a previous hearing-related COS [12, 13]. Increasingly, taking a patient-centred approach can improve the relevance and interest to patients. There is recent evidence that investigators are incorporating public involvement in a variety of different stages of COS development. Examples include appointing Public Research Partners to the Project Steering Group [14], reviewing and reducing the long list [14, 15], reviewing and suggesting new outcomes for the long list [16], deciding the outcome name [17], creating a plain language description for each outcome domain to ensure all participants interpret concepts as intended [18, 19], creating age-appropriate outcome domain descriptions with children and parents [20], creating ‘lay equivalents’ for a public version of the survey, different from the professional version [21], and piloting the Delphi survey for accessibility [16, 18, 22,23,24].

Despite these positive advances in COS methodology, Young and Bagley [7] found that published reports did not always clearly distinguish between public participation and involvement. Furthermore, while published protocols broadly state which steps of the process the public are to be involved, they generally do not provide any in-depth detail on the procedures to be used and the anticipated benefits to be gained [16,17,18,19,20,21, 23]. As such, Gargon and colleagues [25] conclude that there is a need for greater methodological guidance in how to effectively involve the public in COS development, while Jones and colleagues [26] have called for further research about the impact of these forms of public involvement. An illustrative example of good reflective practice is shown by Bruce and colleagues [13]. Their COS study presented plain language descriptions of outcome domains for the public participants only, not the professional stakeholders. However, the consensus meeting identified some domain names which had been ambiguous in meaning to the professional Delphi participants indicating that in future studies the plain language descriptions should be available to all.


When designing the COMIT’ID study, no published procedures were found which described exactly how Public Research Partners have been involved in creating a long list of candidate outcome domains for a round 1 Delphi survey, nor evaluation of its impact. This methodology article addresses this gap in knowledge by reporting in detail our procedures to involve two Public Research Partners. The article also reflects on and evaluates the benefits of this form of public involvement. In so doing, the evidence contributes to the sharing of good practice in public involvement in developing COS. To further enhance the quality, transparency, and consistency of the PPI evidence base for public involvement in core outcome set development, reporting is guided by the GRIPP2-SF checklist [27].


The Study Management Team comprised three tinnitus researchers (HS, KF, DAH). Public involvement was delivered by a dedicated team of two Public Research Partners with lived experience of tinnitus (VC, BT) and a Patient and Public Involvement and Engagement Manager (PPI/E Manager, AH). All the Study Management Team was also part of the wider COMIT’ID Research Steering Group that included three physicians from France, Germany, and Portugal. Independent lay reviews were provided by 14 members of the British Tinnitus Association’s Users’ panel, again all with lived experience of tinnitus, and five clinical experts who were members of the British Tinnitus Association Professional Advisory Committee. To improve the appeal to healthcare users and to reduce attrition, the two Public Research Partners additionally commented on the feasibility of the Delphi survey design, and reviewed study documentation (advertisements, Information Sheets, video instructions for the survey). Because these procedures for public involvement are not the focus of the current article, they are not reported further. All planned public involvement activities were described in the project protocol (version 2, dated 13 March 2017) which was approved by the West Midlands - Solihull Research Ethics Committee (reference 17/WM/0095) on 21 March 2017.

The Study Management Team first sought to create a comprehensive list of all possible outcome domains relevant to tinnitus, regardless of intervention type. A common long list of outcome domains was therefore created for round 1 of all three Delphi surveys (sound, psychology and drug-based interventions). The researchers first identified 169 potential candidate outcome domains via: i) a systematic review of outcome domains reported in clinical trials of tinnitus interventions in adults (62 outcome domains) [8], ii) a thematic analysis of the items taken from 23 commonly reported patient-reported questionnaire instruments for tinnitus (64 outcome domains) [Fackrell, personal communication], and iii) a systematic review of dimensions of tinnitus-related complaints reported by patients and their significant others using questionnaire- and interview-based methods (43 outcome domains) [28]. The Study Management Team were able to reduce this list to 124 candidate outcome domains by consolidating domains that were literal duplicates of one another (see Fig. 1). Two members of the Study Management Team (HS, KF) and our two Public Research Partners (VC, BT) and PPI/E Manager (AH) then participated in two half-day workshops, with the initial objective to review the list of domains identified by the scoping exercise and their associated plain language descriptions; specifically to comment on the readability of the domain descriptions and on the suitability of the headings under which outcome domains were grouped. The team was motivated by the need to describe domains so that all participants, including patients, interpreted the meanings clearly and consistently across stakeholder groups. Also, that having English as an additional language would not act as a barrier to comprehension, given the international focus of the COMIT’ID study. Discussion focussed on the outcome domains themselves, not on their potential relevance to any sound-based, psychology-based, or pharmacology-based intervention.

Fig. 1
figure 1

Flow diagram illustrating the pre-Delphi stage that was completed with the involvement of healthcare users as Public Research Partners prior to round 1 of the modified Delphi survey. Taken from Fackrell et al. [5]

In preparation for these workshops, and to create a starting point for discussion, the Study Management Team compiled preliminary plain language descriptions taken from the systematic reviews and content analysis, and from searches of the Oxford English Dictionary [29]. The Public Research Partners were encouraged to explain what each outcome domain name meant to them and then suggest a way to describe it using language that would be accessible to other members of the public. A harmonised plain language description had to have approval from both Public Research Partners and the researchers, before moving the discussion to the next domain. Although domains were presented one by one, there was cross-referencing to previously discussed domains where needed. Each workshop was audio recorded for later reference. Handwritten notes to aid discussion were encouraged and the two researchers wrote down the plain language description as well as any strong opinions or concerns voiced by either Public Research Partner. Such insights led to a number of unforeseen decisions about the long list of outcome domains for the round 1 Delphi survey.

The public involvement work with the two Public Research Partners was then supplemented with independent reviews conducted by 14 members of the public with tinnitus recruited from the British Tinnitus Association Users’ Panel. These reviewers responded to an invitation emailed to 30 members of the panel. Reviewers were naïve to the details of the COMIT’ID study, but were nevertheless experienced Lay Reviewers. Here, outcome domain names, plain language descriptions and category labels that had been developed for each domain in the modified long list were presented to Lay Reviewers using an online survey platform (Survey Monkey), with the option for free text responses about each domain. For each plain language description, Lay Reviewers were asked to select whether it was “a clear and understandable definition (no changes needed)” or not. If not, then they were asked to explain why and suggest alternative words or phrasing. Lay Reviewers were also asked to review the outcome domain category labels. For each outcome domain, Lay Reviewers were asked to select whether it “fit in this category” or not. If not, then they were asked to explain why and suggest an alternative category.

This yielded further insights which informed the final version of the long list for round 1 of the Delphi survey.


The 124 outcome domains from the scoping process were presented to the Public Research Partners and PPI/E Manager for generating plain language descriptions, and these were then reviewed by expert Lay Reviewers. The initial impression of the Public Research Partners was that the original long list was so long it was not user friendly, and their concerns were that this would negatively impact on recruitment and on sustained respondent engagement across the three survey rounds. This insight prompted the Study Management Team to critically rethink the composition of the long list leading to decisions to modify it in a number of radical, but unforeseen, ways. Illustrative examples of changes made to the long list of candidate outcome domains as a direct result of public involvement are reported below.

Reducing the long list

The most notable unforeseen impact of the public involvement insights was to shorten the initial long list. It was reduced by 57 outcome domains leaving a final long list of 66 distinct, tinnitus-related domains. Overall, 36 domains were excluded, and 21 were pooled (i.e. consolidated) (see Fig. 1) either through direct recommendation by the Public Research Partners, or through Study Management Team discussion following the workshops.

The input of the two Public Research Partners had the greatest impact on the final content of the long list. Subsequent comments made by the Lay Reviewers led to only two further reductions in the number of candidate domains (see Fig. 1).

Excluding broad concepts captured by other outcome domain concepts

The Public Research Partners argued that 13 outcome domains were broad concepts whose meaning was already encapsulated by other domains in the long list (Fig. 1). In one example, the general domain ‘cognitive difficulties’ was deemed to be covered by the more specific domains: ‘concentration’, ‘tinnitus-related thoughts’, ‘confusion’, and ‘ability to ignore’. For example, on ‘tinnitus-related thoughts’, BT said “Thoughts about your tinnitus…Thoughts about tinnitus...’cause that basically is the cognitions”. In a second example, ‘health-related quality of life’ was deemed to be part of ‘impact on relationships’, ‘impact on individual activities’, ‘impact on social life’, ‘impact on work’, and ‘sexual difficulties’.

Such overlap in meanings was considered to risk causing confusion when rating the importance of individual outcome domains in the modified Delphi survey. To minimise this risk, the Public Research Partners recommended that such broad concepts should be removed from the long list and instead used as a category label for grouping similar domains together for presentation in the modified Delphi survey.

Consolidating equivalent outcome domains

The Public Research Partners considered that three fear-related concepts (‘Fear for health’, ‘Fear for quality of life’ and ‘Fear of tinnitus becoming worse’) could not easily be distinguished from one another. For example, the discussion of the domain ‘fear for quality of life’, cross-referenced back to the domain ‘fear for health’. BT said “That’s so close to that one. It’s the same question asked in a different way.” and VC said “When you look at it later, you’ll be able to see if it’s turned out similar to something else or one of them is more defined than another”. As a consequence, these candidate domains were consolidated into a single overarching health-related concept, named ‘fear’ (Fig. 1). The plain language description was expanded to include all concepts (including fear for health, now and in the future and fear of it getting worse).

Similar comments made by Public Research Partners and Lay Reviewers also prompted the consolidation of the domain ‘bothered’ with ‘annoyance’. For example, VC said “Do we need all of these?... Is annoying disruptive? As much as disruptive isn’t it?...Disturbing.” Three Lay Reviewers said “What's the difference between being 'annoyed' and being 'bothered'? In fact, what, specifically, does 'bothered' mean?” and “Questions 29 [tinnitus annoyance] and 30 [bothered] feel very similar to me, and difficult to distinguish between them clearly.” and “What's the difference between this, 'annoyed' and 'bothered' - and 'irritable' in the later aspect?”. The plain language description for ‘annoyance’ was expanded to include elements of bothered (i.e. knowing tinnitus is there and finding it a nuisance).

Lay Reviewers also considered ‘frustration’ to share important conceptual similarities with ‘upset’ and as a consequence, ‘frustration’ was consolidated with ‘upset’. Their feedback also prompted us to change the plain language description for ‘upset’ to more closely match the definition given in the Oxford English Dictionary [Oxford English Dictionary, 2017]. The Study Management Team was satisfied that this new description also encapsulated aspects of the associated concept ‘frustration’.

Consolidating outcome domains that define how the domain might be measured

A number of outcome domains were judged to relate more directly to how an outcome domain could be measured. For example, an ‘active task to distract or cope with tinnitus’ and ‘purposefully protecting or reducing the chance of potential problems’ were both considered different ways to measure the same health construct ‘coping’. This recommendation from the Public Research Partners resulted in ten outcome domains being consolidated in the final version of the long list under existing outcome domains ‘tinnitus loudness’, ‘tinnitus quality’, ‘coping’, and the category ‘body structures and functions’.

Modifying outcome domain names

Discussion with the Public Research Partners’ highlighted ambiguities in meaning and awkwardness of repeated language for some outcome domain names, particularly in the ‘health-related quality of life’ category (see Table 1).

Table 1 Illustrative examples of modifications to the outcome domain names for domains in the ‘health-related quality of life’ category

In one example, they spoke about how group-based activities present very different challenges compared with individual activities. As a result, the domain name was modified to emphasise the individual nature of the activities in question (Table 1). A transcript of part of the discussion about the outcome domain named ‘interference on activities’ is reported below:

VC: “An activity sounds physical doesn’t it, rather than…

BT: “So a social

VC: “Gathering

BT: “Gathering, because that could be anywhere

VC: “That could be restaurant, pub, party

VC: “If you’re asking people for an honest reaction to how much it interferes with their activities, there are two very different things. One is on a one-to-one erm social activity with somebody and the other is a group gathering or activity with people…It’s much easier to control the one-to-one…

In a second example, there was discussion about the term ‘interference’ and its repeated usage which led to its substitution with the term ‘impact’ (Table 1):

VC: “Isn’t it, rather than interfere…Interfere is quite a negative word isn’t it?

KF: “Well yeh, we try to use impact over interfere because impact can be… its more posi…, so you can have a positive impact and a negative impact…

VC: “Yeh

KF: “…Whereas interfere just comes across as...

VC: “Very negative

KF: “It’s negatively interfering with something

HS: “Effecting or impact would be quite neutral

Modifying the plain language descriptions

Table 2 reports three illustrative examples of how input from patients informed changes in our descriptions to make them more understandable and relevant. Lay Reviewer comments were especially informative for understanding how other people interpreted the outcome domains according to the plain language descriptions that had been co-created by the Public Research partners and the Study Management Team. For a number of domains, Lay Reviewers’ suggestions prompted us to add more examples to widen the range of possible experiences or scenarios. Inviting feedback from a wider group, with a more diverse range of lived experiences with tinnitus, enabled us to enhance the personal relevance of our descriptions.

Table 2 Three illustrative examples of how input from patients informed changes in our descriptions to make them more understandable and relevant to their experience of tinnitus

Potential impact on the round 1 Delphi survey

Although it is not possible to directly attribute study performance to the public involvement described above, the online Delphi survey was successful in recruiting and retaining participants. Evidence indicates that participants from a diverse range of backgrounds had positive experiences of taking part.

High participant recruitment

Recruitment exceeded expectations. Although recruitment was open for only a 14-week period (April–July 2017), it surpassed the target of 420 experts [5], with 719 participants consenting and 670 completing round 1 of the Delphi surveys. The COMIT’ID Delphi survey appealed equally to patients (n = 358) as well as to professionals (n = 312). This contrasts with a number of other COS studies which have recruited markedly lower numbers of patients compared to professionals (e.g. [14, 23, 30, 31]).

Positive participatory experience

Accessibility of the Delphi survey is illustrated by the following selection of feedback quotes given by patients:

SOUND00016: “It took a little while for me to understand the concept but once I got it; it was fine.

SOUND00078: “Very good questions. I did not know that what I have been thinking so long was even real; or just real for me.

TALKC00200: “Straight forward to complete.

DRUGC00051: “Easy to understand and follow. Ready for the next round.

Following the GRIPP2-SF checklist [27], consideration was also given to any negative participatory experiences. These were relatively few, but some participants expressed general difficulties in understanding some of the outcome domain concepts, with comments including the following:

SOUND00064: “I'm not sure I understand the subtle distinction between some categories – e.g. several mention behaviour; several mention coping and dealing with tinnitus. I've tried my best to separate them.

SOUND00147: “As a member of the public with tinnitus a number of the questions were difficult to understand and/or answer; either because the language used was too technical or because it was hard to see how it related to either tinnitus or the treatment thereof.

TALKC00188: “A little difficult to understand what I'm being asked. Questions need to be more clear.

Post-hoc analysis explored whether non-native English language might be a barrier to participation, but ruled out this possibility for the following reasons. First, 233 of the 719 participants declared that they did not speak English as a native language (47 of these were patients). Second, although there was an option to leave a comment against each domain, we received very few comments (n = 10) that were concerned with a lack of understanding individual domains. Only four of those were from people who did not have English as their first language. For example, one French healthcare professional (SOUND00190) said “For a hearing-impaired person this question may be ambiguous. One of his major concern is to understand speech when noise is present in his surrounding. Do what does “confusion” refers to? Does it relate to the understanding of the conversation or clear thoughts”. A full list of feedback comments gathered during the first round of the Delphi survey can be found in Additional file 1: Table S1.

High retention

Furthermore, there is no reason to believe that the length of the long list was off putting to patient participants. First, attrition was very low. Of the 670 completing round 1, 586 also completed round 2, giving a 87% overall retention rate. Retention was similar for patients (305/358, 85%) as it was for professionals (281/312, 90%). Second, in round 1 only one negative comment was received from a patient about the length of the survey (SOUND00294 said “Too long... I just want some help please”).

High retention of patients in the COMIT’ID Delphi survey contrasts with a number of other COS studies in which patients, more so than professionals, have withdrawn across rounds 1 and 2. In one extreme example, Al Wattar [30] reported 0% retention for patients (0/24) compared with 68% for professionals (35/51). In another example, the COS study [14] achieved a reasonable 78% retention for patients (25/32), but not as good as the 91% for professionals (149/163).


Patient involvement was an integral and transformative step of the study design process. Through the inclusion of Public Research Partners with lived experience of tinnitus and a PPI/E Manager as three out of the nine members of the Research Steering Group, the Study Management Team co-produced a final long list comprising 66 outcome domains describing distinct tinnitus-related complaints, with plain language domain names and concise but understandable descriptions for each.

To the best of our knowledge, no previous COS studies seem to report detailed public involvement methods for optimising the long list of candidate outcome domains. It is interesting to note that the recent COMET handbook v1 [Section, 9] discusses the value of qualitative research findings from patient interviews and focus groups for ensuring that the outcome domain names and explanations are understandable to patients, but does not specifically mention the role of patient involvement in this process. In this way, the procedures and insights described in this article not only made valuable contributions to the successful conduct of our Delphi survey, but also contribute to the broader methods literature by demonstrating how patient involvement can enhance the accessibility of the consensus processes.

This Discussion highlights seven general methodological features which the Research Steering Group feels made a substantial difference and which fit with general principles of good patient involvement. These recommendations for effective patient involvement in refining a long list of candidate outcome domains in a modified Delphi survey are briefly explained below.

  1. 1.

    Planning: Plan what PPI steps are likely to have most beneficial impact on the Core Outcome Set project and incorporate them into the ethical approval process.

    Although ethical approval is not mandatory for public involvement, setting out these steps in the study protocol that was approved by the Research Ethics Committee was a valuable element of our project planning (see also [32]). By specifying the roles of the Public Research Partners and other healthcare users, it was possible to fully inform lay members what was expected from them during the recruitment process. While VC was an experienced Public Research Partner with whom the Study Management Team had worked previously, BT was recruited to this role after responding to an advertisement and completing an informal telephone interview with HS. This approach would not have been possible without prior planning. Second, by clearly defining the public involvement roles it was possible to budget appropriately for the associated costs in the project. An additional grant was subsequently obtained from the British Tinnitus Association specifically to support the public involvement component of the COS development, and this application was led by the PPI/E Manager and with one of the Public Research partners (VC) as a co-applicant. Agreeing the roles of the lay members and appropriate budgeting are two recognised principles of good practice for successful healthcare user involvement in National Health Service research [33]. Moreover, involving the Public Research Partners in the grant application and as members of the Research Steering Group were opportunities to formally acknowledge the importance of their role.

  2. 2.

    Seeking relevant input: Appoint a small number of Public Research Partners with lived experience of the condition of interest, an eagerness to express their opinions, and whom can provide continuity over a number of face-to-face workshops. A PPI/E Manager can be helpful in providing personal support and mentorship.

    The two Public Research Partners shared a willingness to express their opinions and a sense of humour. These were two positive personal qualities that made the public involvement process enjoyable, not only for the individuals involved but also for the Study Management Team. It was helpful that the Public Research Partners lived locally and were retired because this made it possible for them to provide continuity over a number of face-to-face workshops. Personal access to internet and email were also beneficial for sharing documentation before and after the workshops, and as members of the Research Steering Group. The involvement of the PPI/E Manager was also useful in providing personal support and mentorship, whenever required. It can be reassuring to have a professional member of the Research Steering Group who has not been involved in designing the study, providing an impartial contact for healthcare users and a balance of perspectives. Again, provision of lay support meets one of the principles of good public involvement [33].

  3. 3.

    Being open-minded: Remain open minded, flexible, and curious about the lay perspective.

    In this study, public involvement was far from tokenistic. The Study Management Team remained open minded, flexible and curious about the lay perspective. They were not simply expecting the Public Research Partners to confirm the initial long list of candidate outcome domains. In fact, by respecting their knowledge and experience, the Study Management Team welcomed the challenge at the first workshop to reconsider the length of the preliminary long list. The final long list of candidate outcome domains was directly shaped in response to this challenge. It was bespoke according to the needs of the study as seen through the eyes of our Public Research Partners and Lay Reviewers.

  4. 4.

    Careful reviewing: Review the preliminary long list of candidate outcome domains in a face-to-face workshop(s), focusing discussion on the names, underlying theoretical constructs and plain language descriptions of each one.

    The face-to-face workshop format worked well in enabling facilitated discussions about the underlying theoretical constructs and plain language descriptions. To maintain a balance of membership, a workshop was convened with an equal number of researchers to Public Research Partners. Having a PPI/E Manager participate in the workshop was helpful in providing support to the Public Research Partners and prompting the researchers when technical terms required further explanation or when clarification was needed. A single workshop was insufficient. After 3½ hours of discussion, there were still domains on the preliminary long list which had not been considered. Participation was an intense (and tiring) experience and so a second workshop was planned to complete the work. In hindsight, the Study Management Team had perhaps been too ambitious to expect to complete discussion of all 124 domains in one meeting. This lesson has been taken on board when planning the agenda for the final face-to-face consensus meeting after the Delphi survey is completed. Meaningful discussion on all 66 outcome domains in the final long list is unlikely to be achieved in a one-day meeting.

  5. 5.

    Note taking: Take notes and audio record the workshop discussions for later reference.

    Handwritten notes were important, but an audio recording of the workshop discussions was invaluable for later reference to jog the memory of the researchers. Not all decisions about modifying the long list were made in the workshops themselves. The notes and audio recordings were therefore important in assisting the Study Management Team to reflect on the content of the discussions and to make informed decisions without any time pressures and also to reflect again on the discussion to make sure that none of the researchers were ‘putting words in the mouth’ of the Public Research Partners. The audio recording was particularly helpful to the member of the Study Management Team (DAH) who was not present in the workshops. It gave a retrospective yet accurate record of who said what. The Study Management Team also referred to the audio recording when preparing this research report.

  6. 6.

    Reflecting: Reflect on the discussion and recommendations as a Study Management Team to create an interim long list.

    Taking time outside the two Public Research Partner workshops gave an important opportunity for the Study Management Team to reflect on the discussion and recommendations without feeling any undue time pressures. Members of the Study Management Team who did attend the workshop also found it helpful to go over the main points of the conversation with their fellow member who had not been present in the workshop, so giving a new perspective.

  7. 7.

    Engaging wider feedback: Identify and engage a diverse group to review the interim list of outcome domain names and plain language descriptions, thus creating a final long list.

    Review of the interim long list by a reasonably broad mix of patients and clinical experts increased our confidence in the conceptual uniqueness of each domain and personal relevance of each plain language description. The experience in round 1 was that none of the feedback challenged our grouping or definitions of the candidate outcome domains. From this the conclusion is that reducing the overlap in the constructs corresponding to each outcome domain at the preparatory stage reduced the likelihood of Delphi participants recommending changes to the long list. Rather than querying the validity of the outcome domains themselves, participants have focused on scoring importance. In at least one preceding COS study, participant feedback was found to challenge investigators to exclude some overarching domains in between the Delphi rounds [31]. It is uncertain whether this earlier study incorporated patient involvement at the preparatory stage.

The study findings suggested that non-native English was not a barrier to participation in the English language Delphi survey. Nevertheless, including patient involvement with non-native English language speakers could have brought additional benefits by enabling a more representative group of people to comment on their understanding of the outcome domain names and descriptions. This limitation could have been overcome by engaging the European professional members of the Research Steering Group to lead some of the patient involvement activities in their own countries, either in the form of workshops or an online survey. Enlisting patient involvement from those with a range of English language abilities is just one aspect of diversity. Involving people with a range of ages, ethnicity and socio-economic status are other aspects to consider in future work. The Study Management Team did not act on the small number of negative participatory experiences that were received in the round 1 feedback, due to resource limitations and time pressures for opening round 2. However, future work could consider a role for patient involvement here, especially where comments pertain to the understanding of specific outcome domain names and descriptions.

The time spent developing a high-quality long list of candidate outcome domains as preparation before the Delphi process is anticipated to bring later benefits in the face-to-face consensus meetings. Several COS study reports show how considerable discussion time during the consensus meeting has been taken up with queries about consolidating outcome domains into larger categories, excluding some domains on the basis that they are more relevant to how the domain is measured or that they are specific to one form of intervention, or sometimes even redefining the outcome domains themselves. In one example, during a COS consensus meeting for otitis media with effusion in children with cleft palate, participants agreed to combine the outcome domain ‘consonant production (cleft-related speech patterns)’ with ‘consonant production’, and furthermore that both of these should directly inform subsequent decisions of how to measure ‘speech development’ [13]. In another example, during a COS consensus meeting for psoriatic arthritis, participants agreed to redefine the outcome domain ‘emotional well-being’ (feeling good about oneself) so that it additionally captured the outcome domains: ‘anxiety’, ‘depressive mood’, ‘embarrassment’, ‘frustration’, ‘self-worth’, and ‘stress’ [34].

For the COMIT’ID study, the Study Management Team has pre-empted many of these discussions in the preparatory phase, leaving greater opportunity for the consensus meeting to focus on prioritising what should be in the core set.


The end result of the public involvement process has been a rich insight into the healthcare user perspective for the design of a long list of candidate outcome domains for a modified Delphi survey. This article describes procedures and offers methodological guidance for working effectively with patients to develop a mutual understanding of (often complex) healthcare concepts, and to co-produce the final long list with a set of plain language descriptions. Although it is not possible to directly attribute study performance to public involvement, the online Delphi survey has been highly successful in terms of recruitment, retention and accessibility. This experience confirms the wide-ranging benefits of public involvement in reviewing the long list before an online Delphi survey is launched. Procedures recommended in this article go beyond core outcome set development because they embody general standards for good patient involvement that can be transferred into other research contexts.



Core Outcome Measures in Tinnitus International Delphi


Core outcome set


Cooperation in Science and Technology


Patient and Public Involvement and Engagement


European TINnitus NETwork


  1. Hall DA, Haider H, Kikidis D, Mielczarek M, Mazurek B, Szczepek AJ, et al. Towards a global consensus on outcome measures for clinical trials in tinnitus: report from the first international meeting of the COMiT initiative 14 November 2014, Amsterdam, The Netherlands. Trends Hear. 2015:1–7.

  2. Londero A, Hall DA. Call for an evidence-based consensus on outcome reporting in tinnitus intervention studies. Front Med (Lausanne). 2017;4:42.

    Article  Google Scholar 

  3. Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13:132.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Kirkham JJ, Gorst S, Altman DG, Blazeby JM, Clarke M, Devane D, et al. Core outcome set–STAndards for reporting: the COS-STAR statement. PLoS Med. 2016;13(10):e1002148.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Fackrell K, Smith H, Colley V, Thacker B, Horobin A, Haider HF, et al. Core outcome domains for early-phase clinical trials of sound-, psychology-, and pharmacology-based interventions to manage chronic subjective tinnitus in adults: the COMIT’ID study protocol for using a Delphi process and face-to-face meetings to establish consensus. Trials. 2017;18:388.

    Article  PubMed  PubMed Central  Google Scholar 

  6. INVOLVE. What is public involvement in research? 2016. Accessed 22 Apr 2017.

  7. Young B, Bagley H. Including patients in core outcome set development: issues to consider based on three workshops with around 100 international delegates. Res Involv Engagem. 2016;2:25.

    Article  Google Scholar 

  8. Hall DA, Haider H, Szczepek AJ, Lau P, Rabau S, Jones-Diette J, et al. Systematic review of outcome domains and instruments used in clinical trials of tinnitus treatments in adults. Trials. 2016;17:270.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Williamson PR, Altman DG, Bagley H, Barnes KL, Blazeby JM, Brookes ST, et al. The COMET handbook: version 1.0. Trials. 2017;18(Suppl 3):280.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Keeley T, Williamson P, Callery P, Jones LL, Mathers J, Jones J, et al. The use of qualitative methods to inform Delphi surveys in core outcome set development. Trials. 2016;17:230.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  11. Delphi plain language summary. In: Plain Language Summary. The COMET Initiative. 2017. Accessed 13 Apr 2017.

  12. Harman NL, Bruce IA, Kirkham JJ, Tierney S, Callery P, O’Brien K, et al. The importance of integration of stakeholder views in Core outcome set development: Otitis media with effusion in children with cleft palate. PLoS One. 2015;10(6):e0129514.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Bruce I, Harman N, Williamson P, Tierney S, Callery P, Mohiuddin S, et al. The management of Otitis media with effusion in children with cleft palate (mOMEnt): a feasibility study and economic evaluation. Health Technol Assess. 2015;19:68.

    Article  Google Scholar 

  14. van’t Hooft J, Duffy JMN, Daly M, Williamson PR, Meher S, Thom E, et al. A core outcome set for evaluation of interventions to prevent preterm birth. Obstet Gynecol. 2016;127(1):49–58.

    Article  Google Scholar 

  15. Wylde V, MacKichan F, Bruce J, Gooberman-Hill R. Assessment of chronic post-surgical pain after knee replacement: development of a core outcome set. Eur J Pain. 2015:611–20.

  16. Taylor J, Böhnke JR, Wright J, Kellar I, Alderson SL, Hughes T, et al. A core outcome set for evaluating self-management interventions in people with comorbid diabetes and severe mental illness: study protocol for a modified Delphi study and systematic review. Trials. 2017;18:70.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Duffy JMN, Van’t Hooft J, Galec C, Brown M, Grobman W, Fitzpatrick R, et al. A protocol for developing, disseminating, and implementing a core outcome set for pre-eclampsia. Pregnancy Hypertens. 2016;6(4):274–8.

    Article  PubMed  Google Scholar 

  18. Evangelidis N, Tong A, Manns B, Hemmelgarn B, Wheeler DC, Tugwell P, et al. Developing a set of Core outcomes for trials in Hemodialysis: an international Delphi survey. Am J Kidney Dis. 2017;

  19. Hirsch M, Duffy JMN, Barker C, Hummelshoj L, Johnson NP, Mol B, et al. Protocol for developing, disseminating and implementing a core outcome set for endometriosis. BMJ Open. 2016;6(12):e013998. ubMed: 28238554

    Article  PubMed  PubMed Central  Google Scholar 

  20. Tong A, Samuel S, Zappitelli M, Dart A, Furth S, Eddy A, et al. Standardised outcomes in nephrology—children and adolescents (SONG-kids): a protocol for establishing a core outcome set for children with chronic kidney disease. Trials. 2016;17:401.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Allin B, Bradnock T, Kenny S, Walker G, Knight M. NETS1HD: study protocol for development of a core outcome set for use in determining the overall success of Hirschsprung’s disease treatment. Trials. 2016;17:577.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Sinha IP, Gallagher R, Williamson PR, Smyth RL. Development of a core outcome set for clinical trials in childhood asthma: a survey of clinicians, parents, and young people. Trials. 2012;13:103.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Kaufman J, Ryan R, Lewin S, Bosch-Capblanch X, Glenton C, Cliff J, et al. Identification of preliminary core outcome domains for communication about childhood vaccination: an online Delphi survey. Vaccine. 2017; [Epub ahead of print]

  24. Thorlacius L, Ingram JR, Garg A, Villumsen B, Esmann S, Kirby JS, et al. Protocol for the development of a core domain set for hidradenitis suppurativa trial outcomes. BMJ Open. 2017;7:e014733.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Gargon E, Gurung B, Medley N, Altman DG, Blazeby JM, Clarke M, et al. Choosing important health outcomes for comparative effectiveness research: a systematic review. PLoS One. 2014;9(6):e99111.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Jones JE, Jones LL, Keeley TJH, Calvert MJ, Mathers J. A review of patient and carer participation and the use of qualitative research in the development of core outcome sets. PLoS One. 2017;12(3):e0172937.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Staniszewska S, Brett J, Simera I, Seers K, Mockford C, Goodlad S, et al. GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. Res Involv Engagem 2017;3:13.

  28. Haider H, Fackrell K, Kennedy V, Hall DA. Dimensions of tinnitus-related complaints reported by patients and their significant others: protocol for a systematic review. BMJ Open. 2016;6:e009171.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Oxford English Dictionary. Oxford University Press. Accessed 02 Sept 2017.

  30. Al Wattar BH, Tamilselvan K, Khan R, Kelso A, Sinha A, Pirie AM, et al. Development of a Core outcome set for epilepsy in pregnancy (E-CORE): a national multi-stakeholder modified Delphi consensus study. BJOG. 2017;124:661–7.

    Article  CAS  PubMed  Google Scholar 

  31. Chiarotto A, Deyo RA, Terwee CB, Boers M, Buchbinder R, Corbin TP, et al. Core outcome domains for clinical trials in non-specific low back pain. Eur Spine J. 24:1127–42.

  32. Morgan H, Thomson G, Crossland N, Dykes F, Hoddinott P. Combining PPI with qualitative research to engage ‘harder-to-reach’ populations: service user groups as co-applicants on a platform study for a trial. Res Involvem Engagem. 2016;2:7.

    Article  Google Scholar 

  33. Boote J, Barber R, Cooper C. Principles and indicators of successful consumer involvement in NHS research. Results of a Delphi study and subgroup analysis. Health Policy. 2006;75:280–97.

    Article  PubMed  Google Scholar 

  34. Orbai A-M, de Wit M, Mease P, Shea JA, Gossec L, Leung YY, et al. International patient and physician consensus on a psoriatic arthritis core outcome set for clinical trials. Ann Rheum Dis. 2017;76:673–80.

    Article  PubMed  Google Scholar 

Download references


We thank Heather Bagley and Iain Bruce for preparatory discussions about meaningful patient involvement. We thank Haula F. Haider (ENT Department of Hospital Cuf Infante Santo - Nova Medical School, Lisbon, Portugal), Alain Londero (Service ORL et CCF, Hôpital Européen G. Pompidou, Paris, France), and Birgit Mazurek (Tinnitus Center, Charite University Hospital, Berlin, Germany) for their input as members of the COMIT’ID Research Steering Group. Thanks to members of the British Tinnitus Association’s Users’ panel and members of the British Tinnitus Association Professional Advisory Committee for reviewing the resulting outcome domain names, plain language descriptions and category labels. The COMIT’ID initiative also acknowledges the support of the National Institute of Health Research Clinical Research Network (NIHR CRN) in recruitment.


The main source of funding was from the NIHR Nottingham Hearing Biomedical Research Unit and NIHR Nottingham Biomedical Research Centre. An additional grant of £4803 was awarded to AH, VC and KF from the British Tinnitus Association to support the public involvement work described in this article. TINNET is supported by a European Cooperation in Science and Technology (COST) Action (BM1306). None of the funders played an active role in the study design nor in the decision to submit this manuscript and do not have ultimate authority over any of these activities.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Author information

Authors and Affiliations




DAH conceived the study. HS, KF, AH and DAH analysed the public research partners’ data, and interpreted the impact of the public involvement. HS, KF and DAH took final decisions about modifying the long list. DAH wrote the manuscript, with contributions from HS and AH. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Deborah A. Hall.

Ethics declarations

Authors’ information

DAH is the project lead and HS and KF are researchers. AH is the PPI/E Manager for the hearing and deafness theme of the NIHR Nottingham Biomedical Research Centre. VC and BT have lived experience of tinnitus. VC is an experienced public research partner, while BT was recruited to this role after responding to an advertisement and completing an informal telephone interview with HS.

Ethics approval and consent to participate

The COMIT’ID study was given a favourable opinion by West Midlands - Solihull (Proportionate Review) Research Ethics Committee (ref: 17/WM/0095) on 21 March 2017. All Delphi survey participants gave informed consent using an online interface.

Consent for publication

Not applicable

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1: Table S1.

A full list of the general feedback comments received by participants completing round 1 of the online Delphi survey. (DOCX 26 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Smith, H., Horobin, A., Fackrell, K. et al. Defining and evaluating novel procedures for involving patients in Core Outcome Set research: creating a meaningful long list of candidate outcome domains. Res Involv Engagem 4, 8 (2018).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: