Combining PPI with qualitative research to engage ‘harder-to-reach’ populations: service user groups as co-applicants on a platform study for a trial
© Morgan et al. 2016
Received: 21 May 2015
Accepted: 13 February 2016
Published: 24 March 2016
Plain english summary
It is recommended that research studies are carried out with or by patients and the public through their involvement from the beginning and in as many stages as possible (known as PPI). Some studies formally invite patients and the public to participate in interviews and focused group discussions to collect views about topics (known as qualitative research). In our study on financial incentives for giving up smoking in pregnancy and breastfeeding, we combined both PPI and qualitative research to include the views of women with a range of experiences of smoking and breastfeeding.
We involved two mother and baby groups in disadvantaged areas of North East Scotland and North West England as research partners on our team. First, we asked members to comment on our research plans and documents, which is standard PPI. Second, we asked members to participate in voice recorded discussions, contributing to qualitative research data. These discussions revealed different views from those that we heard through research interviews. They allowed us to develop more relevant research tools and resources. Members also helped us to identify people outside the groups who we could interview.
Combining involvement and participation helped us to include the views of a wide range of women from ‘harder-to-reach’ groups who don’t usually take part in research. This was important because the research was intended for women who could benefit from incentives to stop smoking in pregnancy and breastfeed, often present in such groups. Positive continuing relationships and trust improved on involvement or participation alone.
Patient and public involvement (PPI) in all research studies is recommended from the earliest point and in as many stages as possible. Qualitative research is also recommended in the early stages of designing complex intervention trials. Combining both together might enable inclusion of ‘harder-to-reach’ perspectives from the target population(s), particularly when the research is intended for their benefit. However, the interface between PPI and qualitative research has received little attention.
In a multi-disciplinary, mixed methods study to inform the design of incentive trials for smoking cessation in pregnancy and breastfeeding, we combined PPI and qualitative research, with some overlap. Mother and baby groups from two geographically separate disadvantaged areas, with diverse experiences of the smoking and breastfeeding, but no training or previous involvement in research, were recruited as PPI research grant co-applicants. An iterative partnership approach facilitated involvement in research conduct and design across all project phases. Group PPI members were also invited to contribute to more formal qualitative data collection, as and when indicated by the research questions, and emerging analysis.
We engaged with ‘harder-to-reach’ women in mother and baby group settings, rather than in academic or home environments. These settings were relaxed and informal, which facilitated rapport-building, disclosures of unexpected information and maintained trust. Twenty-one women participated in standard PPI activities: feedback on study protocols and documents; piloting questionnaires and interview schedules. PPI members voiced some different perspectives from those captured within the qualitative dataset. Nineteen participated in focused qualitative research. Novel aspects were audio recorded PPI discussions, which contributed qualitative data; first, to interpret systematic review findings and construct intervention vignettes for use in the qualitative research; second, to assist with recruitment to improve sample diversity in the formal qualitative dataset; and third, to translate theory and findings presented in a researcher generated logic model into a lay tool. This had face validity for potential trial participants and used the metaphor of a ladder.
Combining and overlapping PPI and qualitative research added ‘harder-to-reach’ contributions, sample diversity, trust and engagement in creative approaches beyond what could be achieved through PPI or qualitative research alone.
KeywordsPublic involvement Qualitative research Complex interventions ‘harder-to-reach’ perspectives Feasibility studies Trial design Service user collaboration Participatory methods
Patient and public involvement (PPI) in research entails working with patients (people with a medical condition receiving health service treatment), public (residents of the country) and people who use services. The purpose is to enhance depth, credibility and applicability of findings, improve clarity of research reports and recommendations and ensure immediate links between practice-based evidence and evidence-based methodology [1–3]. It is recommended in medical and health services research, including clinical trials, especially when funding is through taxation. The importance of incorporating PPI into research is acknowledged internationally, e.g. Ottawa Charter for Health Promotion 1986 (World Health Organization) , and nationally, e.g. US National Institutes of Health , and is enshrined in UK statutes, for example the Health and Social Care Act 2001, within the Local Government and Public Involvement in Health Act 2007, and more recent reforms contained within the Health and Social Care Act 2012, as well as in National Institute for Health Research  and other guidance [7, 8]. Qualitative interviews, focus groups and observations are also recommended in The Medical Research Council’s guidance on the development and feasibility testing of complex interventions to incorporate wider views and perspectives into the study direction .
Both PPI and qualitative research  should begin at the earliest stage possible for maximum benefit to all stakeholders [10–12]. Qualitative research has rigorous quality appraisal standards  and reporting guidelines, with particular attention paid to the rationale for the sampling strategy and reporting the sample characteristics . While the descriptions of how PPI representatives are selected and their characteristics vary across studies, the recent GRIPP checklist  has outlined key issues that should be reported. However, there are still specific challenges when selecting PPI and qualitative research participants for prevention research topics where health inequalities are observed, for example lifestyle behaviours in pregnancy . The most disadvantaged or marginalised in society, who are often the main target population for such interventions, are the hardest to access and engage . ‘Harder-to-reach’ is often used to refer to individuals who have a low socioeconomic status (SES), are members of ethnic minorities, and who have a low level of literacy. However, it is a contested term as it can be perceived as pejorative and is usually underpinned by a number of preconceptions . There are numerous barriers that can prevent people who are labelled as ‘hard-to-reach’ from being involved in research, e.g. personal characteristics, where people live, communication issues  Therefore, the ways in which research teams approach PPI vary . In practice, early PPI with representatives of ‘harder-to-reach’ populations is not always easily achievable unless a formal, regular PPI arrangement already exists . Advertising for volunteers may not attract typical members of the target population(s) . Constraints on time and funding to proactively recruit appropriate PPI members can act as barriers, which can marginalise involvement in the crucial stages, i.e. the development of a research bid [22–25]. PPI rationale and selection procedures have historically lacked clear description. The GRIPP checklist has helped to address some of the key issues by highlighting what needs to be addressed: limited conceptualisation of PPI; poor quality of methods reporting; poor reporting of context and process; wide variability in the way impact is reported; little formal evaluation of the quality of involvement; limited focus on negative impacts; and little robust measurement of impact . GRIPP 2 will soon be published. However, it can still be difficult to assess whether and how the PPI integrates perspectives from members of the intended target population(s) and into what aspects or stages of the research . As a result, formal qualitative research tends to take precedence because methodologies and sampling are more systematically documented. PPI and qualitative research components are not proxies for one another and perform different, although often complementary, functions. They can, however, overlap and be combined, which might offer potential for including ‘harder-to-reach’ perspectives, though the interface between them has received little attention.
‘BIBS: Benefits of Incentives for Breastfeeding and Smoking cessation in pregnancy: a platform study for a trial’  was multi-disciplinary and used mixed methods: we used both PPI and qualitative approaches, sometimes overlapping, as well as quantitative approaches. A platform study is one that considers the range of quantitative and qualitative evidence available around populations, interventions, comparators and outcomes to inform the design of a clinical trial. BIBS therefore comprised evidence syntheses, primary qualitative research and surveys (including a Discrete Choice Experiment (DCE), which is a quantitative technique for eliciting preferences that can be used in the absence of revealed preference data. The method involves asking individuals to state their preference for hypothetical alternative scenarios, goods or services). We also aimed to understand and incorporate target population perspectives into all stages through PPI and by including representatives’ participation in the formal qualitative research component. We set out to foster a partnership approach , emphasising interactive communication, with what are locally considered by health and social care services to be ‘harder-to-reach’ women who are more likely to smoke in pregnancy and not breastfeed, and who are targeted for participation in community groups. These groups aim to engage mothers living in less privileged postcode areas who are encouraged by health or community staff to attend for health and social care support. While women were not specifically ‘recruited’ to attend, the outreach workers at the Children’s Centre and health-care staff (i.e. midwives, health visitors) actively encourage all mothers in the local community to access the group for social and emotional support. We worked with both groups as co-applicants early on in order to guarantee inclusion and engagement of populations with whom the target behaviours of smoking in pregnancy and not breastfeeding are typically associated [16, 26]. (See also Appendix 1) Service user involvement was therefore conceived as an iterative, loosely structured process that was emergent and sensitive to their views and preferences in respect of our research conduct, whilst also enabling us to access and include ‘harder-to-reach’ perspectives through formal qualitative research activities, including recruitment to the main sample. The aim of this paper is to describe and reflect on the PPI and its interaction with the formal qualitative research components undertaken in our study.
Our multi-disciplinary, mixed methods platform study to inform the design of incentive trials for smoking cessation in pregnancy and breastfeeding was commissioned by a UK government funding body that involves service user representatives in decision-making about research priorities . In the UK, these commissioned calls have tight turnarounds, with the deadline for outline application 8–12 weeks after advertisement, and a further 8 weeks for a full application if shortlisted. No pre-existing PPI arrangements were suitable for this study where the target population for financial incentives would be women who smoke in pregnancy and who do not breastfeed. As such, these women are a ‘harder-to-reach’ group and we adopted an innovative approach by actively engaging service users from two mother and baby groups as co-applicants. These groups were selected because they were located in the most disadvantaged postcode areas of the cities closest to the academic centres leading the PPI and qualitative research in the BIBS project. This helped to facilitate involvement as the researchers had fairly close proximity with the groups and hence could be more flexible with attending meetings.
Identification of service user groups
At the grant application stage, we approached maternity services, primary care and children’s centre managers to ask them to suggest two thriving, but diverse, mother and baby groups operating in areas of high deprivation in North East Scotland and North West England where smoking in pregnancy and formula milk feeding are prevalent. Such women infrequently attend health promoting services , and do not tend to participate in PPI and research initiatives; thus their perspectives are less often heard.  Smoking in pregnancy and not breastfeeding are also more likely for younger, less educated women living in more disadvantaged areas , and therefore involving perspectives from these populations, included within the memberships of the groups, was all the more important. Managers identified group liaison staff to negotiate and represent group involvement in the pre-funding study discussions: a health visitor (Group 1) and a mother and baby group coordinator (Group 2). The groups themselves were named as the co-applicants. The groups differed: Group 1 was user-led, successful in generating sustainability funding and provided a café and crèche as incentives to attend. Group 2 was based in a Local Authority Children’s Centre, where a previous peer support service delivering incentives to improve breastfeeding outcomes had been located .
The respective group representatives were sent drafts of the grant proposal to discuss with the groups, and group members provided feedback via the representatives during the research planning stages in 2010. They negotiated reimbursement for their group’s refreshments and/or crèche costs as part of the grant application (£960.00 each, spread over three or four payments spanning 20 months). Although the study was on incentives, this was not offered as a financial incentive for participation, but was reimbursement for use of premises, refreshments and crèche provision as a research cost.
Principles of successful consumer involvement in NHS research
The research will lead to benefits for consumers, in terms identified by the consumers themselves;
Consumers are involved in every stage of the research, from identifying the research area through to sharing the research findings;
Consumers’ expectations of being involved in the research are made clear to the researcher;
The roles of consumers are agreed between the researchers and consumers involved in the research;
Consumers have the opportunity to engage in research in the manner and at the level they wish, opting out of being involved in research at any time;
Researchers budget appropriately for the costs of consumer involvement in research;
Consumers are from sections of society and walks of life that are appropriate to the research;
Researchers respect the differing skills, knowledge and experience of consumers;
Consumers are offered training and personal support, to enable them to be involved in research;
Researchers ensure that they have the necessary skills to involve consumers in the research process;
Consumers are involved in decisions about how participants are both recruited and kept informed about the progress of the research;
Consumer involvement is described in research reports;
Research findings are available to consumers, in formats and in language they can easily understand
Our participatory research approach was at the interface between PPI and qualitative research and the principles do not account for ethical approvals which might be required where there is such overlap. Therefore, we considered it necessary to obtain ethics approval to gain each woman’s informed consent to actively participate in the study and to be able to use verbatim quotations in study publications. Because the groups had not been involved in research before, we wanted to engage on women’s own ‘turf and terms’, in addition to inviting women to project meetings at the University. From the start, we anticipated that we might want to audio record some of our discussions to include as more formal qualitative research. Recording and the potential to use quotes as data in reports distinguishes PPI from qualitative research, as any recording is potentially identifiable and rigorous data management procedures are required under the Data Protection Act 1998. We also wanted to collect socio-demographic data, where possible, and a history of participants’ infant feeding and smoking behaviours to ensure broad representation of the target population. This creates tensions around the optimum timing of ethics applications at the interface between PPI and qualitative research. We wanted external ethical review to ensure that our approach was consistent with the Helsinki declaration  and not over-burdening service users. When the study started (February 2012), service users assisted in developing the protocol and study information materials through third-party feedback to the group representatives whilst we applied for ethical approval. Approval was granted by the National Research Ethics Service, North of Scotland, on 10 May 2012 (Research Ethics Committee reference: 12/NS/0041; Protocol number: 2/024/12 ). Once ethical approval had been obtained, the group representatives introduced the researchers (HM to Group 1 and GT to Group 2) to the service users. The two researchers led the PPI and the formal qualitative recruitment, data collection and analysis for the study.
Co-applicant mother and baby group profiles
Location; distance from University
Purpose and structure
Group meetings attended by researcher
Total individual participants formally engaged
Meetings contributing to formal qualitative data collection (audio recorded)
North East Scotland; ~10 minute walk
One room within a larger family centre, including a separate crèche room. Sofas, play area adjacent, toys, kitchen area, dining area, unobtrusive site manager in room nearby
~7 mothers (2 mothers left the group and 2 new mothers joined) and their babies, 2 grandmothers – fairly regular group membership
Weekly, Wednesdays noon-2 pm (except during school holidays), plus some additional social/fundraising events
Set format: the café is facilitated by mothers who make and sell a cheap, healthy lunch costing £2 (subsidised through fundraising) to mothers and babies during the first hour. During the second hour, the babies go into the crèche and the mothers enjoy a coffee and a catch up, or participate in training activities/external speakers
Established through a partnership project between Aberdeen City Council, Homestart (http://www.home-start.org.uk/) and NHS Grampian. The café crèche is now independent and self-funding
n = 11 (+ n = 3 social events)
n = 9
Two focus groups (n = 5; n = 5); logic model session (n = 4)
North West England; 18 mile drive
One room within a larger community centre, with toys, books and soft play facilities and seating for parents. A café is also located in this room for parents to purchase drinks and food
16-20 families (discontinuous participation)
Weekly, Fridays 9 am-noon (except during school holidays)
Unstructured format. Mother and baby/toddler group in an informal setting where parents can drop in and out to interact with and receive support from Children’s Centre staff members and engage with peers. Children Centre staff member in attendance throughout the session
n = 4
n = 12
One focus group (n = 4); individual interviews (n = 2); logic model session (n = 4)
Settings and involvement
An unusual aspect of our study is the involvement of the research team with established community groups in their own environments during their usual meeting times. The research was not an ‘add on’ and no additional time commitment was requested. Negotiating this engagement was crucial to the success of the involvement we achieved from ‘harder-to-reach’ women, particularly as these women were caring for babies and older children.
At both sites, attendance at group meetings initially consisted of building a rapport with the members and observing in an unobtrusive manner to acknowledge the researchers’ roles as visitors to established groups . Participatory approaches were adopted for PPI to facilitate group-directed involvement . We wanted the groups to guide the development of our research partnerships in their usual group meetings, but to have flexibility to address specific research questions through audio recorded group discussions which would contribute qualitative data to the study. We worked to their timetables regarding convenient visiting times to either involve members or to formally collect data. This resulted in different patterns of engagement at each site.
Contacts were made through the health visitor group representative to begin with, the first visit having been advertised in advance with a flyer and through circulation of reminder information leaflets. However, the researcher (HM) was able to exchange contact details with group members at her second visit at which point the gatekeeper role shifted from the health visitor to the group’s treasurer (a mother) and secretary (a grandmother). They suggested appropriate weeks to attend for participatory visits. Informed written consent to contribute to research activities was gained from each member at the first visit and from new members on subsequent visits. Women were also requested to complete socio-demographic, smoking and breastfeeding experience characteristics forms. (See Additional file 1).
Communicating directly with members of Group 1 by email and Short Message Service (SMS) allowed for less formal communication, good rapport-building and thus intermittently ‘dropping in’ for lunch visits, as well as the more directed research sessions, eased by the close proximity of the group to the University (~10 min walk). In February/March 2013, the health visitor support for the group ceased following a change of personnel and service reorganisation; however, this did not adversely affect engaging service users and possibly strengthened a direct partnership between the researcher and the group. Health professional/community staff presence was observed as inhibiting the spontaneity of contributions. Group 1 used Facebook (www.facebook.com) as its primary communication mode; however, despite gaining ethical approval to do so, the research team decided against trying to engage with the group using this social media. Several concerns were discussed around maintaining confidentiality and objectivity for the researcher during the study, appropriate boundaries, best use of researcher time and a lack of organizational guidance on research use of social media. Using email/phone/SMS only, however, did not appear to prevent or diminish involvement.
To begin with, the researcher stood passively on the periphery of the meeting room, waiting to be invited in. Eventually, and over the course of the collaboration, she was encouraged to join women sitting comfortably on the sofas, or on the floor. She also more actively engaged when she was asked to console crying babies, make tea and wash dishes. During the meetings, women sat around a large coffee table or leaned in over the kitchen counter if they were making the tea/coffee. Meanwhile, their children were occupied in the crèche, which meant that the researcher could interact with women both more and less formally.
The group environment, cohesiveness and set structure at Group 1 suited focused activities to address research questions and collect data. Women in this group were familiar with engaging in courses (e.g. food preparation, child first aid) and had regularly welcomed external speakers in the second hour of their weekly two-hour meetings. Computing/projector facilities were not available, so a white board and printed interactive materials were spread out on a low table for developing vignettes and testing a ladder logic model both discussed below . A logic model is a tool used by funders, managers, and evaluators of programs to help understand how a program is likely to work). These uninterrupted, audio-recorded and focused activity discussions enabled service users to be positively engaged in the study in a more concrete and tangible way. However, some voices were dominant and there was a hierarchical structure and group membership rules that had seemed to have emerged over time. On one occasion, internal politics caused tension and prevented research activities taking place because members disputed rota duties (e.g. washing up and cleaning the carpets). This shows that the presence of the researcher did not affect normal group dynamics, which may have been the case had involvement taken place within a University setting or if a formal focus group had been arranged as is usual in qualitative research. Regularly taking the research to the groups, and thus reaching out to people in their communities to make involvement easier , enabled ‘harder-to-reach’ and more confident PPI than members of the research team have experienced when group representatives have been invited to the university setting. This methodological strategy avoids the selection bias that can occur if group members are asked to volunteer to participate in additional research activities outside usual group time. Previous experience suggests that only the more confident individuals accept such invitations and their views may differ from those who decline such invitations. Women involved in PPI could access support within their groups from the attached health visitors and/or social care professionals. Our approach contrasts with more ‘professionalised’ PPI, where cumulative experience and training can limit PPI representatives’ ability to offer a patient or lay perspective . Although there can be benefits to involving trained PPI representatives , it was important that representatives of our study’s target population maintained their own perspectives, rather than by training them in contributing PPI to research from academic, health services research or clinical perspectives We wanted as far as possible to seek maximum diversity and disconfirming perspectives which is appropriate in the early stages of trial design. Our focused activity discussions emerged as a strategy to meld PPI with qualitative research and thus to broaden our sample.
More formal meetings emerged as the most appropriate strategy for engagement with Group 2. Whilst an outline of the study had been provided to the group by the coordinator prior to the initial visit, the researcher (GT) initiated all direct contact with the women. The distance of travel to the group (18 miles by car) was a barrier to informal contacts and attendance was negotiated through the mother and baby group coordinator. When the researcher wanted to attend, email communication occurred at least two weeks in advance. During each visit, the researcher circulated and chatted to mothers informally prior to discussing the research. The use of a large room within the Children’s Centre meant that the researcher took a more opportunistic approach to involving group members spread across the space. Informed written consent to contribute to research activities was gained from each member of the group. The group’s drop-in format presented the researcher with the practical challenge of obtaining informed written consent on an ongoing basis, or re-establishing it for those who had consented previously, together with completing the characteristics forms. (See Additional file 1).
Standard PPI contributions
This involved contributions to decision-making about all stages of the research, including the design of information materials and what data should be collected, from which participants, and how. Group members also helped to pilot interview topic guides for the formal qualitative research, survey questionnaires and a DCE; collaborated in writing the lay summary; and dissemination of key findings of the study. These more standard contributions made by PPI to our study are detailed in Appendix 2 with the qualitative research being detailed in Appendix 3.
Qualitative data collection combined with PPI
After establishing PPI relationships and gauging an appropriate level of involvement with our work through negotiation, we gradually introduced specific research questions into more focused activity discussions with the groups. This melded PPI with qualitative research in an iterative manner and sessions were audio recorded with participants’ informed consent . We negotiated with group members that transcriptions of our audio recordings of group discussions would be included in formal qualitative data analysis and members were keen to have their views incorporated.
Study intervention vignettes, presented in further detail below, were co-developed in focused sessions for subsequent use in interviews and focus groups. This enabled the formal qualitative study to gain a diverse sample of participants’ perspectives of incentive interventions identified in systematic reviews and to assist with short-listing promising incentive strategies. By using vignettes, we aimed to include service user group members in the interpretation of systematic review findings (four evidence syntheses of the benefits of incentives for smoking cessation in pregnancy and breastfeeding, barriers and facilitators to maternal behaviour change and incentives for other lifestyle behaviours) [26, 39, 40] and to involve them as part of the research team.
A logic model was created from study findings to inform the design of incentive interventions as is recommended for public health interventions . The logic model was presented to the groups towards the end of the study using interactive materials, which were adapted to suit the different formats of the groups. These included intervention components and processes for the delivery of incentive trials and how they fit with the findings from a narrative review of the qualitative evidence about the barriers and facilitators to behaviour change and maintenance for smoking cessation in pregnancy and for breastfeeding in women’s everyday lives . These data contributed to the final version of the logic model and the assessment of its face validity.
In addition, members from both groups assisted with suggesting ‘harder-to-reach’ local candidates who might have different perspectives (e.g. teenagers, fathers) for recruitment to the main sample through affiliated NHS (National Health Service) staff and snowball sampling.
Audio recordings of formal data collection sessions with the mother and baby groups were transcribed as soon as possible to facilitate accurate and comprehensive data documentation. Although the noise levels (children playing, shouting, etc.) compromised the digital recordings of discussions, especially with Group 2, GT also kept handwritten notes and transcribed the session recordings herself as soon as possible to facilitate accurate and comprehensive data documentation. These were circulated among the wider research team and entered into Nvivo10 software (QSR International, Burlington, MA) with other primary qualitative study data and analysed thematically following a Framework approach . The qualitative research for this study is described in detail elsewhere .
Researcher reflexivity, where the researcher acknowledges and reflects on their effect(s) on the process and outcomes of research, is an important quality indicator in qualitative research and indeed any study. Our multi-disciplinary, multi-institution team and the selection of areas where health service attitudes to and experience of incentives were known to differ were a priori decisions to minimise bias of interpretation. Our research team included previous smokers, researchers with and without children, those with experiences of breast and formula milk feeding and male and female researchers. Researchers held different perspectives on incentive interventions for behaviour change, with four involved in incentive interventions (LB, DT, GT and PH). Differences and potential biases were discussed in regular team meetings. In addition, the researchers involved with the mother and baby groups (HM and GT) kept reflective diaries of their interactions and observations. Diary extracts were shared with the research team to facilitate consideration of influences that might inform group dynamics and involvement. Discussions occurred after every interaction and informed project development until study end in September 2013. An advance draft of this paper was also shared with a member of Group 1 for comment.
Twenty-one women across the two groups were involved in various PPI activities on more than one occasion over a period of time. Nineteen of these women were also engaged more formally in qualitative research: nine in North East Scotland and ten in North West England. (See Appendix 1). Women were provided with clear information (verbal and written) about each stage of their involvement and asked to sign a consent form (consent was an ongoing process as women engaged in the different activities, including use of quotations in study publications) .
Rapport-building, relationships with service users and unintended consequences
Both service user groups took evident pride in researchers coming into their territory. They were welcoming and hospitable, and offered refreshments and home bakes (Group 1) or computer facilities for directed sessions (Group 2). In addition, the researchers witnessed conversations that are seldom encountered in formal health or research settings. For example, some group members were adversarial towards health professionals; others described complex and disruptive domestic relationships, or expressed highly charged personal opinions towards health behaviours (smoking during pregnancy), which, on one occasion, resulted in someone leaving the room. In the presence of health service staff, such accounts would have been unlikely.
Stable membership meant there was a group culture/norm and that collaboration was through these terms in Group 1. This meant that members would not wish to challenge the group ‘norms’, although it did allow for informal conversations and the atmosphere was friendly and respectful. The Group 1 researcher (HM) was also invited to participate in social events. Of the six informal social events that took place during the course of this research, HM attended three to show goodwill (a Hallowe’en party, a Christmas fair and an end of term mums’ bingo night). Researcher participation can therefore be considered as ‘active’  as HM was able to gain insight into the cultural codes and rules for behaviour of the group. The requirements of a researcher to remain objective and maintain a distance, compared with a more anthropological approach to fieldwork, were discussed with the wider research team. Informal contacts and balancing communication traffic presented a need for the researcher to be mindful of balancing PPI activities and remaining objective as in most formal qualitative research with pressure for full group membership.
The management and composition of Group 2 meant that these opportunities were not available so undertaking qualitative research was more challenging due to structural and organisational factors. Additional challenges were faced when trying to engage parents and carers in meaningful and uninterrupted conversations when their infants/children were present and noise levels (children playing, mothers talking, babies crying, etc.) compromised some audio recordings. The need to move around the room to approach members was a barrier to group engagement. The frequently changing group membership also resulted in more researcher time spent introducing these formal research processes than in Group 1. Children’s Centre staff supervised children when research activities involved concentration to read and discuss materials and computer facilities were made available to facilitate involvement of group members when needed. The drop-in format at Group 2 meant that the group was sufficiently anonymous to allow for people to express their thoughts and feelings; however, these could be too freely expressed and, on occasion, was identified to have caused offence. Flexibility was key and it was necessary to adapt rather than replicate methods according to group context and membership, which resulted in different experiences.
For both groups, long school holidays coincided with crucial research stages, for example the final qualitative data analysis overlapped with the school summer holidays.
Our collbaorations may have led to unintended consequences. Some group members appeared to change, or express desire to change, lifestyle behaviours during the involvement process, including stopping smoking and eating more healthily, while others did not. One group member joined a local health services committee as a lay member and enrolled on a further education college course.
Relationships with wider research team
Whilst mother and baby group members were invited to attend research team meetings, and to meet with our advisory panel, they did not attend at either site. Therefore, engaging the mother and baby groups with the wider research team was problematic. The health visitor group facilitator for Group 1 attended one meeting (September 2012) and suggested that group members would have been very nervous, had any of them attended, because of the formality of the meeting. Her successor was invited to a subsequent meeting and expressed an interest, however, she did not attend. Travelling was an issue for members of Group 2; however, they did not accept an invitation to join meetings by telephone. Likewise, when research staff team members were invited to attend visits to the mother and baby groups, the idea was mostly rejected. Some made highly sceptical comments like “you’ll be lucky!” and “ahem, no! [laughing]” PH attended with HM at Group 1 on two occasions and NC attended with GT at Group 2 on one occasion, both to assist as observers of qualitative research activities.
Substantive contributions from combining PPI and qualitative research
PPI members voiced some different perspectives from those found in the emerging main qualitative dataset. In the PPI activities, the researchers perceived that women were more candid when describing the barriers to changing their behaviour, whereas in the interviews, there seemed to be an element of ‘what the researcher might want to hear’. Members also assisted with recruitment of ‘harder-to-reach’ women through snowball sampling (i.e. by referral of friends and acquaintances) to ensure further inclusion of disconfirming data (new data that challenged what we had already found).
The substantive contributions made by combining and overlapping PPI and qualitative research are summarised below.
Commenting on and interpreting the presentation of researcher constructed study vignettes derived from a systematic review prior to using them in qualitative interviews
Systematic reviews and study vignettes
The mother and baby groups contributed to interpreting systematic review findings by providing feedback on study vignettes derived from the evidence syntheses. Six vignettes were constructed by the research team from three smoking cessation and three breastfeeding studies. These studies were selected either because they had statistically significant effects or involved an unusual or innovative approach [43–48]. Different vignette structures were tried out, for example presenting and discussing an intervention chronologically and sequentially, revealing one bullet point at a time (the preferred option amongst the research team) compared to presenting the whole vignette. Various formats for the vignettes were also piloted (e.g. power point screenshot, written bullet points on one and/or multiple sheets, patient journey/map). Group discussions enabled us to identify that presenting the vignette as a whole and the power point screenshot were the most popular. Through discussion, this helped us to make study interventions more salient to facilitate assessment of the acceptability of components and processes, as well as to inform the development of a short-list of promising incentive strategies. This was particularly useful as very little detail around feasibility and acceptability of content and delivery processes was identified in the systematic reviews, even where included studies were classified as ‘qualitative’ or ‘mixed methods’ .
‘Ladder’ logic model
Service users engaged well with constructing ladders and this stimulated rich discussions about the choices they would make and the reasons why – what they would need and value within a trial –demonstrating face validity for the co-design of complex incentive interventions. Women found the ‘ladder’ idea easy to understand, relate to and use. More importantly, they found it acceptable as a model to consider how incentive behaviour change theory could be translated into their everyday lives. Some participants spontaneously talked about the implications for other health behaviours, in particular healthy eating, which they considered too expensive to do, suggesting the model may have wider relevance for addressing other health behaviours. However, others expressed entrenched resistance to change; for example, when completing a ladder for stopping smoking, one woman responded to breastfeeding saying “no, no, no, no, no!”
Summary of key similarities and differences between PPI and qualitative research
Both PPI and qualitative research aim to incorporate deeper understanding of the research problem and ensure greater relevance of the findings to society. Both were used to gather information to help in the design of an intervention and potential clinical trial.
PPI involves non-researchers and non-clinicians in research to inform study design and conduct. Qualitative research involves collecting data from participants to answer the research question(s).
Both PPI and qualitative research can include representatives of the target population of the study.
PPI might only include representatives of patients or the public in general rather than the target population and representatives might be trained in PPI. PPI representatives are usually fewer in number than researchers or research participants. Qualitative research might seek to include broader perspectives and disconfirming data from as diverse a sample as possible.
Both PPI and qualitative research (with consent) can collect data using traditional methods such as recorded discussions, interactive sessions, and activities. Any collection of data for research purposes, audio-recording or subsequent use of quotations requires research ethics committee approval (http://www.hra.nhs.uk/).
PPI is predominantly involvement in the tasks of research and is a two way exchange of knowledge that influences study design, whereas qualitative research is predominantly for advancing understanding and thus involves the researchers being informed by the participants. Qualitative research requires research ethics committee approval whereas PPI usually does not.
Both PPI and qualitative research can take place in a range of settings, including Universities or public spaces and either face-to-face, by telephone or using remote audio-visual technology.
PPI tends to involve inviting representatives to join research team meetings in academic settings, but can include researchers going out into the community. The setting for qualitative research takes into account participant preferences and where is best for the data collection.
Both PPI and qualitative research can involve single or serial interactions or meetings.
PPI is more likely to take place over an extended period and involve multiple meetings. Qualitative research is more likely to involve a one-time data collection session.
Both PPI and qualitative research might employ similar purposive sampling approaches to represent specific populations.
PPI is more likely to draw on established networks of people interested in contributing to research. Qualitative research designs vary based on the aims of the study, e.g. snowball, stratified, theoretical, purposive and convenience sampling .
Our approach shows that physically going to and engaging with PPI in ‘harder-to-reach’ group settings  – relevant for health behaviours where there are strong social gradients – is valuable. Going to the groups, on their terms, flexibly and gradually enabled us to build trusting rapports and to see issues through the participants’ eyes. This participatory approach also enabled less ‘professionalised’ PPI . This contrasts with trained PPI where there is potential for representatives to be influenced more by the academic community (e.g. by receiving training to understand research language so that they can contribute in research team meetings where academics outnumber PPI). In our study, PPI representatives outnumbered the researchers.
Combining PPI and qualitative research can add value to studies informing the design and delivery of complex interventions which target ‘harder-to-reach’ populations. This is because PPI can contribute additional information to research data when combined with formal data collection, for example if perspectives are being voiced that are different to those being heard through traditional qualitative data collection methods , as happened in our study. Qualitative interviewing is primarily a guided conversation where knowledge, views and experiences are provided by the participants in response to open questions and prompts by the interviewer. Often great care is taken in how the researcher contributes, with the research team carefully constructing an introduction and topic guide in order to avoid leading or priming a participant’s response. The main difference with PPI is that there is more two-way sharing of knowledge, views and experiences within a research team and this tends to be around the ‘doing of’ research. Power differentials are played out in different ways in that PPI can contribute more to informing what and how research is done. Our approach resulted in new perspectives from the target populations for potential smoking cessation and breastfeeding incentive intervention trials, which might not have been gained with more traditional methods of engagement with PPI or qualitative research alone. PPI can advise and help to refine research processes so that they assist in recruiting ‘harder-to-reach’ groups or translate research findings in a way that is meaningful for these groups . Their suggestions modified our approach, our language, our strategies to facilitate involvement of further relevant perspectives and translation of a logic model to a lay audience. Working with two groups in different areas taught us that there is no set formula or protocol to engaging service users through their groups and involvement benefits from being adapted to the particular context. In particular, seating arrangements, the structure of the room and drop in formats were not suitable for focused group activities.
However, in our study, the distinction between PPI and qualitative research was subtle as there was overlap and movement between PPI and qualitative research throughout. In practice, groups or individuals had multiple roles in that they were influencing decisions, making substantive contributions to research processes, generating data and providing feedback on the results. This created a need to negotiate boundaries between the researchers and the researched where the lines between objective observation and subjective participation required careful consideration.
Ethical approval was gained, although it is not required for PPI, and our study meets the principles of good PPI [31, 32]. We considered ethics committee approval necessary to meet international guidance  to obtain informed consent to audio record discussions to contribute to qualitative research data; use research team generated topic guides; use interactive session outputs in study publications and more generally for documenting, managing and reporting on both PPI and qualitative data . We consider that ethical approval for these types of activities is something that needs to be addressed in future guidelines for best practice in PPI. In addition, it was important to consider issues of burden and ability to opt out or remove data at any stage, particularly so individual members did not feel under any obligation given that fees were negotiated in advance for group involvement and that potential identification was possible. The groups were given the option of anonymity but chose to have their involvement acknowledged by group name in study publications. This raises issues for those considering similar models of PPI where anonymity may be more difficult to preserve.
Co-applicant PPI collaboration is a strength for any complex or contentious research, especially where the target population(s) is ‘harder-to-reach’. The use of diverse, iterative and flexible approaches allows adaptation and integration of PPI and qualitative research methods. This strengthened our results because new findings were included and these were accessed via the relationships with women in the groups rather than their familiarity with being involved in research. In addition, they were not only ‘research participants’, but were also helping us to frame our research study and to explore insights from other studies, which enabled us to have more open and frank conversations. Working with groups over an extended period of time throughout the research project built trust, which meant that our research went beyond either involvement or qualitative research. Another benefit of our approach was the sense of pride expressed by group members about being involved in the research and hosting researcher visits, which challenged traditional power relations.
A key strength of our approach is to combine the complementary features of PPI with qualitative research, so that they enrich the contributions that each make to the research questions and the study design. Flexible, sensitive and reflexive approaches were needed to achieve rich engagement from what are often considered ‘harder-to-reach’ groups. This is one of few primary research accounts of the issues encountered when doing this, in particular our consideration of ethical issues. Potential limitations of combining both are the blurred boundaries between PPI and qualitative research, and the challenges of pre-emptively describing the participatory approach in order to obtain ethical approval. From the perspectives of the women involved, they became more familiar with the research as their PPI matured. By incorporating informed consent and opt-out options throughout, over-burdening of representatives was prevented. Our separate qualitative sample ensured that we incorporated a broader range of views, and the mixed methods involved in the wider project allowed us to include the perspectives of a number of populations (e.g. the MORI survey of the general public ).
A limitation of combining PPI and qualitative research is the possibility of ‘group think’ .
We sought to minimise this, however: first by working with two groups and second by using other strategies within the research . Although group members contributed to the development work leading to the shortlist of incentive strategies asked about in the MORI survey of the general public , a separate sample of independent general public participants was sought to pilot this to minimise the ‘group think’ that may occur through repeated discussion of a topic. Our approach involved considerable researcher time. This will not add value to every project. Therefore, the benefits need to be weighed against resources available. The debate around how far to take PPI without compromising research outcomes is also pertinent . For example, PPI collaborators may not consider reflexivity and bias in the same way/s as trained researchers, which may limit the role they have and stages in which it is appropriate for them to be involved, and may influence the qualitative data that they contribute, depending on when it is collected in relation to the PPI. This is especially relevant for ensuring that ‘lay’ voices remain ‘lay’ and are not obscured by research training or over-professionalisation of involvement by experience or desensitisation. Undertaking qualitative research that overlaps with PPI requires re-definition of roles and relationships in this regard and reflexivity in reporting. In addition, the extent to which the collaboration might have primed individual motivation to behave differently is uncertain and could be a limitation. However, despite some exceptions, many PPI participants remained highly resistant to change in relation to either stopping smoking and/or breastfeeding. Involvement with the wider research team was usually dependent on single researcher observations rather than PPI within a wider team (e.g. at project meetings). This often applies to qualitative research data collection, however, this is a potential limitation to consider when comparing PPI with qualitative research.
We recommend that each research team for every study considers whether and how both PPI and qualitative research might contribute to or compromise research outcomes and if there are opportunities to combine the two for maximum benefit to all stakeholders. How each can contribute is likely to depend on the research question being addressed, the stage of the research and the design of the study. When ‘harder-to-reach’ service user perspectives are crucial to the research, funding bodies should consider how early involvement in the design and application stages of a grant can be achieved. However, for all PPI in future, greater attention needs to be paid to maximising opportunities for researchers to work more flexibly with the most relevant PPI representatives, who might not always be those who are trained for involvement in research. Consideration should also be given to PPI representatives’ social context(s) to ensure that these can be approached in reflexive and sensitive ways that support involvement and mutual benefits for the research and those who contribute PPI.
Combining and overlapping PPI and qualitative research added ‘harder-to-reach’ contributions from members of the target populations to trial design research beyond what could be achieved through PPI or qualitative research alone. Extensive involvement of ‘harder-to-reach’ service user groups as collaborators across a range of research activities in their own research settings rather than in an academic setting enabled less ‘professionalised’ PPI that benefited the progression of the study. Collection of formal qualitative data from PPI service users added disconfirming data.
There are, however, implications of our approach for other research teams. These include further consideration of: who, where and how to engage PPI representatives in research; the interface between PPI and qualitative research and their underpinning logic models; how researchers work flexibly with people and understand the context in reflexive and sensitive ways; the scope and remit of ethical approvals and the pros and cons of providing training for PPI representatives.
We would like to thank all members of the research team, especially our co-applicant collaborators: Mastrick Café Crèche, Aberdeen and Wendy Ratcliffe (Health Visitor); and St Cuthbert’s and Palatine Children’s Centre, Blackpool and Helen Cook (Manager). Extra and special thanks to Donna McPhee, Mastrick Café Crèche, who read and provided invaluable feedback on the final draft of this paper and with whom we continue to work to disseminate the findings of our collaboration and research. We would also like to thank our grant co-applicants: Professor Linda Bauld, Professor David Tappin, Professor Anne Ludbrook, Dr Shelley Farrar, Professor Falko Sniehotta, Graeme MacLennan and Professor Marion Campbell for their collaboration and input into the study design. Additional thanks to colleagues at the University of Aberdeen: Dr Sharon McCann, for helpful discussions around service user involvement in research, Kieran Rothnie and Fiona Stewart for their contributions to team discussions about service user involvement, Fiona Stewart and Cynthia Fraser for providing guidance with literature searching and reference management; and Lara Kemp for providing secretarial support.
Department of Health disclaimer
This report presents independent research commissioned by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, MRC, CCF, NETSCC, the HTA programme or the Department of Health.
Source of funding
This project was funded by the NIHR Health Technology Assessment Programme (10/31/02) and is published in full in Health Technology Assessment: Morgan H, Hoddinott P, Thomson G, Crossland N, Farrar S, Yi D, et al. Benefits of Incentives for Breastfeeding and Smoking cessation in pregnancy (BIBS): a mixed-methods study to inform trial design. Health Technol Assess 2015;19(30). See here: http://www.nets.nihr.ac.uk/projects/hta/103102.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Boote J, Baird W, Sutton A. Public involvement in the systematic review process in health and social care: a narrative review of case examples. Health Policy. 2011;102(2–3):105–16.View ArticlePubMedGoogle Scholar
- Braye S, Preston-Shoot M. Emerging from out of the shadows? Service user and carer involvement in systematic reviews. Evid Policy. 2005;1(2):173–94.View ArticleGoogle Scholar
- Smith E, Donovan S, Beresford P, et al. Getting ready for user involvement in a systematic review. Health Expect. 2009;12(2):197–208.PubMedGoogle Scholar
- The Ottawa Charter for Health Promotion [website on the Internet]. World Health Organisation 1986. Available from: URL: Available from: http://www.who.int/healthpromotion/conferences/previous/ottawa/en/index1.html. Accessed date 18 Feb 2016.
- Public involvement with NIH [website on the Internet]. National Institutes of Health 2011. Available from: URL: Available from: http://nih.gov/about-nih/what-we-do/get-involved-nih/public-involvement-nih. Accessed date 18 Feb 2016.
- INVOLVE [website on the Internet]. National Institute for Health Research 2013. Available from: URL: http://www.invo.org.uk/. Accessed date 18 Feb 2016.
- Patient and public involvement policy [document on the Internet]. National Institute for Health and Care Excellence 2013. Available from: URL: Available from: http://www.nice.org.uk/getinvolved/patientandpublicinvolvement/patientandpublicinvolvementpolicy/patient_and_public_involvement_policy.jsp. Accessed date 18 Feb 2016.
- School of Health and Related Research. Patient and public involvement in research [webpage on the Internet]. University of Sheffield 2014. Available from: URL: Available from: http://www.sheffield.ac.uk/scharr/ppi. Accessed date 18 Feb 2016.
- A Framework for development and evaluation of RCTs for Complex Interventions to Improve Health [document on the Internet]. Medical Research Council 2000. Available from: URL: Available from URL: http://mrc.ac.uk/documents/pdf/rcts-for-complex-interventions-to-improve-health. Accessed date 18 Feb 2016.
- Research Design Service North West. PPI bursary [webpage on the Internet]. NHS National Institute for Health Research 2014. Available from: URL: Available from: http://www.rds-nw.nihr.ac.uk/public-involvement/ppi-bursary/. Accessed date 18 Feb 2016.
- Public Involvement Impact Assessment Framework (PiiAF) [website on the Internet] 2014. Available from URL: http://piiaf.org.uk/. Accessed date 18 Feb 2016.
- Staniszewska S, Adebajo A, Barber R, et al. Developing the evidence base of patient and public involvement in health and social care research: the case for measuring impact. Int J Consum Stud. 2011;35(6):628–32.View ArticleGoogle Scholar
- Dixon Woods M, Shaw RL, Agarwal S, Smith JA. The problem of appraising qualitative research. BMJ Qual Saf. 2004;13(3):223–223Google Scholar
- Equator Network: Enhancing the QUAlity and Transparency Of health Research [website on the Internet]. Available from URL: http://www.equator-network.org/reporting-guidelines/. Accessed date 18 Feb 2016.
- Staniszewska S, Brett J, Mockford C, et al. The GRIPP checklist: Strengthening the quality of patient and public involvement reporting in research. Int J Technol Assess Health Care. 2011;27(4):391–9.View ArticlePubMedGoogle Scholar
- McAndrew F, Thompson J, Fellows L, Large A, Speed M, Renfrew MJ. Infant feeding survey 2010: summary [document on the Internet]. Health and Social Care Information Centre 2012 Available from URL: http://www.hscic.gov.uk/catalogue/PUB08694/ifs-uk-2010-sum.pdf. Accessed date 18 Feb 2016.
- Bonevski B, Randell M, Paul C, Chapman K, Twyman, L, Bryant J, Brozek I, Hughes C. Reaching the hard-to-reach: a systematic review of strategies for improving health and medical research with socially disadvantaged groups. BMC Med Res Methodol. 2014; 14(42) doi: doi:https://doi.org/10.1186/1471-2288-14-42.
- Freimuth VS, Mettger W. Is there a hard-to-reach audience? Public Health Rep. 1990;105(3):232–8. Available from URL: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1580016/.PubMedPubMed CentralGoogle Scholar
- Beresford P. Beyond the Usual Suspects: Towards Inclusive User Involvement. London: Shaping Our Lives; 2013. Available from URL: http://shapingourlives.org.uk/wp-content/uploads/2015/03/BTUS-Report.pdf.Google Scholar
- Oliver SR, Rees RW, Clarke-Jones L, et al. A multidimensional conceptual framework for analysing public involvement in health services research. Health Expect. 2008;11(1):72–84.View ArticlePubMedGoogle Scholar
- Collaborations for Leadership in Applied Health Research and Care. Patient and Public Involvement (PPI) [webpage on the Internet]. NHS National Institute for Health Research 2014. Available from: URL: Available from: http://clahrc-west.nihr.ac.uk/patient-and-public-involvement. Accessed date 18 Feb 2016.
- Boote J, Baird W, Beecroft C. Public involvement at the design stage of primary health research: A narrative review of case examples. Health Policy. 2009;95(1):10–23.View ArticlePubMedGoogle Scholar
- Staniszewska S, Jones N, Newburn M, et al. User involvement in the development of a research bid: barriers, enablers and impact. Health Expect. 2007;10(2):173–83.View ArticlePubMedGoogle Scholar
- Brett J, Staniszewska S. Mockford C, et al. The PIRICOM Study: A systematic review of the conceptualisation, measurement, impact and outcomes of patient and public involvement in health and social care research. UK Clinical Research Collaboration; 2010.Google Scholar
- Staley K. Exploring Impact: Public involvement in NHS, public health and social care research. Eastleigh: INVOLVE; 2009.Google Scholar
- Morgan H, Hoddinott P, Thomson G, Crossland N, Farrar S, Yi D, et al. Benefits of Incentives for Breastfeeding and Smoking cessation in pregnancy (BIBS): a mixed-methods study to inform trial design. Health Technol Assess. 2015;19(30):1–522.View ArticlePubMedGoogle Scholar
- Reed J, Weiner R, Cook G. Partnership research with older people - moving towards making the rhetoric a reality. J Clin Nurs. 2004;13(3a):3–10.View ArticlePubMedGoogle Scholar
- Evaluation, trials and studies: Public and patient involvement [webpage on the Internet]. NHS National Institute for Health Research 2014. Available from: URL: http://www.nets.nihr.ac.uk/ppi. Accessed date 18 Feb 2016.
- Thomson G, Dykes F, Hurley MA, Hoddinott P. Incentives as connectors: insights into a breastfeeding incentive intervention in a disadvantaged area of North-West England. BMC Pregnancy Childbirth. 2012;12:22. http://dx.doi.org/https://doi.org/10.1186/1471-2393-12-22.View ArticlePubMedPubMed CentralGoogle Scholar
- INVOLVE and NRES. Patient and public involvement in research and research ethics committee review. [ONLINE] [document on the Internet]. Available from URL: http://www.invo.org.uk/wp-content/uploads/2011/12/INVOLVENRESfinalStatement310309.pdf. 2009. Accessed date 18 Feb 2016.
- Telford R, Boote JD, Cooper CL. What does it mean to involve consumers successfully in NHS research? A consensus study. Health Expect. 2004;7(3):209–20. Available from URL: http://onlinelibrary.wiley.com/doi/10.1111/j.1369-7625.2004.00278.x/pdf.View ArticlePubMedGoogle Scholar
- Boote J, Barber R, Cooper C. Principles and indicators of successful consumer involvement in NHS research: Results of a Delphi study and subgroup analysis. Health Policy. 2006;75:280–97. Available from URL: http://www.researchgate.net/profile/Jonathan_Boote/publication/7841118_Principles_and_indicators. Accessed date 18 Feb 2016.
- Declaration of Helsinki [document on the Internet]. World Medical Assocation. Available from URL: http://www.wma.net/en/30publications/10policies/b3/index.html. Accessed date 18 Feb 2016
- BIBS: Benefits of Incentives for Breastfeeding and Smoking cessation in pregnancy: Research Protocol [document on the Internet]. NIHR-HTA. Available from URL: http://www.nets.nihr.ac.uk/__data/assets/pdf_file/0020/55370/PRO-10-31-02.pdf. Accessed date 18 Feb 2016.
- Spradley JP. Participant Observation. New York: Holt, Rinehart and Winston; 1980.Google Scholar
- Scott R, Ayelet K, Brian DH. Qualitative research methodologies: ethnography. BMJ. 2008;337:a1020.View ArticleGoogle Scholar
- Hoddinott P, Morgan H, Thomson G, Crossland N, Dombrowski SU. A ladder logic model to inform the design of incentive trials for women around childbirth. Int J Behav Med. 2014;21(Supp.1):139. doi:https://doi.org/10.1007/s12529-014-9418-2. 0465. [ONLINE].Google Scholar
- Dudley L, Gamble C, Allam A, Bell P, Buck D, Goodare H, et al. A little more conversation please? Qualitative researchers’ and patients’ interview accounts of training for patient and public involvement in clinical trials. Trials. 2014;16:190. Available from URL: http://www.trialsjournal.com/content/16/1/190. Accessed date 18 Feb 2016.
- Hoddinott P, Hislop J, Morgan H, Stewart F, Farrar S, Rothnie K, et al. Incentive interventions for smoking cessation in pregnancy: a mixed methods evidence synthesis. Lancet. 2013;380 Suppl 3:S48. http://dx.doi.org/https://doi.org/10.1016/S0140-6736(13)60404-3.Google Scholar
- Hall Moran V, Morgan H, Rothnie K, MacLennan G, Stewart F, Thomson G, et al. Incentives to Promote Breastfeeding: A Systematic Review. Pediatrics. 2015;135(3):e687–702.View ArticleGoogle Scholar
- Armstrong R, Waters E, Moore L, et al. Improving the reporting of public health intervention research: advancing TREND and CONSORT. J Public Health. 2008;30(1):103–9.View ArticleGoogle Scholar
- Ritchie J, Lewis J, Nicholls CM, Ormston R. Qualitative research practice: A guide for social science students and researchers. London, UK. Sage, 2013.Google Scholar
- Chamberlain LB, McMahon M, Philipp BL, Merewood A. Breast pump access in the inner city: A hospital-based initiative to provide breast pumps for low-income women. J Hum Lact. 2006;22(1):94–8.View ArticlePubMedGoogle Scholar
- Gulliver SB, Colby SM, Hayes K, Raffa SD. Tobacco cessation treatment for pregnant smokers: incorporating partners and incentives. Med Health Rhode Island. 2004;87(1):9–12.Google Scholar
- Heil SH, Higgins ST, Bernstein IM, et al. Effects of voucher-based incentives on abstinence from cigarette smoking and fetal growth among pregnant women. Addiction. 2008;103(6):1009–18.View ArticlePubMedPubMed CentralGoogle Scholar
- Pugh LC, Milligan RA. Nursing intervention to increase the duration of breastfeeding. Appl Nurs Res. 1998;11(4):190–4.View ArticlePubMedGoogle Scholar
- Volpe EM, Bear M. Enhancing breastfeeding initiation in adolescent mothers through the Breastfeeding Educated and Supported Teen (BEST) Club. J Hum Lact. 2000;16(3):196–200.View ArticlePubMedGoogle Scholar
- Walsh RA, Redman S, Brinsmead MW, Byrne JM, Melmeth A. A smoking cessation program at a public antenatal clinic. Am J Public Health. 1997;87(7):1201–4.View ArticlePubMedPubMed CentralGoogle Scholar
- Hoddinott P, Morgan H, MacLennan G, Sewel K, Thomson G, Bauld L, Yi D, Ludbrook A, Campbell MK. Public acceptability of financial incentives for smoking cessation in pregnancy and breast feeding: a survey of the British public. BMJ Open. 2014;4(7): http://dx.doi.org/https://doi.org/10.1136/bmjopen-2014-005524.
- Willis G. Cognitive interviewing: A ‘How To’ Guide. Reducing Survey Error through Research on the Cognitive and Decision Processes in Surveys [document on the Internet]. Research Triangle Institute 1999Available from: URL: Available from URL: http://www.hkr.se/PageFiles/35002/GordonWillis.pdf.. Accessed date 18 Feb 2016.