Skip to main content
Download PDF
Download ePub

Participatory evaluation of the process of co-producing resources for the public on data science and artificial intelligence

Research Involvement and Engagement volume 9, Article number: 67 (2023) Cite this article

Abstract

Background

The growth of data science and artificial intelligence offers novel healthcare applications and research possibilities. Patients should be able to make informed choices about using healthcare. Therefore, they must be provided with lay information about new technology. A team consisting of academic researchers, health professionals, and public contributors collaboratively co-designed and co-developed the new resource offering that information. In this paper, we evaluate this novel approach to co-production.

Methods

We used participatory evaluation to understand the co-production process. This consisted of creative approaches and reflexivity over three stages. Firstly, everyone had an opportunity to participate in three online training sessions. The first one focused on the aims of evaluation, the second on photovoice (that included practical training on using photos as metaphors), and the third on being reflective (recognising one’s biases and perspectives during analysis). During the second stage, using photovoice, everyone took photos that symbolised their experiences of being involved in the project. This included a session with a professional photographer. At the last stage, we met in person and, using data collected from photovoice, built the mandala as a representation of a joint experience of the project. This stage was supported by professional artists who summarised the mandala in the illustration.

Results

The mandala is the artistic presentation of the findings from the evaluation. It is a shared journey between everyone involved. We divided it into six related layers. Starting from inside layers present the following experiences (1) public contributors had space to build confidence in a new topic, (2) relationships between individuals and within the project, (3) working remotely during the COVID-19 pandemic, (4) motivation that influenced people to become involved in this particular piece of work, (5) requirements that co-production needs to be inclusive and accessible to everyone, (6) expectations towards data science and artificial intelligence that researchers should follow to establish public support.

Conclusions

The participatory evaluation suggests that co-production around data science and artificial intelligence can be a meaningful process that is co-owned by everyone involved.

Plain English Summary

Modern technology offers new treatment options for patients and novel avenues of research. However, there is limited available information in easily understandable language for the public explaining how technology relates to them and could influence their healthcare. The researchers, healthcare professionals and public members worked together collaboratively to address this problem by creating new materials for the public. Our paper explores that project through creative methods. Firstly, everyone involved was offered an opportunity to attend training sessions. Then, people took photos and described them to illustrate to others what is their experience of working together. Finally, we all met to use included photos as building blocks to present a shared experience in the project. Afterwards, the professional artist included it as one circular illustration with six interlinked layers. These layers present everyone’s experiences (from inside) (1) is about the opportunity to build confidence in a new topic, (2) relationships with others, (3) working remotely during the pandemic, (4) motivation that influenced people to become involved in this particular piece of work, (5) expectation that the project needs be inclusive and accessible, (6) ethical principles that researchers using new technology should follow. We showed that it is possible for researchers, healthcare professionals and members of the public to feel joint ownership of the project and that working together can be meaningful to everyone.

Peer Review reports

New technology underpinned by data science and artificial intelligence (AI) offers unique solutions for healthcare services and research. Data science and AI have an increasing influence on all our lives. The public is mostly supportive of reusing their data for health research but remains concerned about how this is being conducted [1,2,3]. They are often familiar with artificial intelligence but mostly thanks to examples from outside the healthcare settings [4]. In a data-rich healthcare system, the public should have access to information which explains how these technologies use data and AI. Access to this information can enable patients to ask questions about healthcare technologies and make informed decisions about using them. In some instances, patients might ask healthcare professionals about data science and AI. However, healthcare professionals might not have the time, knowledge or feel confident explaining artificial intelligence, or why data is collected and how reused if they are not directly responsible for it [5].

To help to address this emerging issue, our research team at Imperial College London, consisting of academic researchers and health professionals alongside 30 patients and members of the public (later referred to as public contributors) worked together on a public involvement project to co-design and co-develop an information resource in order to learn more about data science and AI in a healthcare setting. Public contributors were recruited from the patient involvement group for cancer research at Imperial College London, an Equality, Diversity & Inclusion group at Guy's & St Thomas's Hospital and an information governance committee at Patient Experience Research Centre. The breakdown was 60% women, 40% men, ages range 18–78, ethnicity White British, Eastern European, African, Caribbean, Asian. The project team (KG, JJG and MM) ensured that the involvement is accessible to public contributors by supporting them throughout the project. Involvement requires ongoing support, practical and monetary investments to be conducted meaningfully[6]. In this project, for example, public contributors who preferred hard-paper copies would receive one and could have sent the comments back on paper. The postage was reimbursed. This feedback was later incorporated by the project team to a digital version. Public contributors were recruited from several different patient and public involvement groups across medical specialities. The aim was to contribute resources that could increase digital health literacy to aid both health professionals and the public in making more informed and personal decisions in this increasingly digital world.

In our work, we perceive public involvement as a work "being carried out 'with' or 'by' members of the public rather than 'to', 'about' or 'for' them” [7]. We followed the UK Standards for Public Involvement to ensure the quality of involvement in our project [8]. In line with these principles, four public contributors are also co-authors of this paper.

Providing introductory information for public contributors around data science is not new; e.g. Teng and colleagues [9] provided members of the public with a booklet written by researchers on what linked data sets are and contentious issues around their use to provide a base for the discussion. Our project is novel because it aimed to co-develop materials with public contributors around data science and artificial intelligence.

Discussing data science and artificial intelligence with public contributors can be challenging because of the need to present appropriate information using lay language [10,11,12,13], as jargon and specialist terms can marginalise anyone unfamiliar with that vocabulary [14]. Public contributors might be apprehensive about joining involvement groups that focus on what they perceive as technical topics [15]. Involving those with a better understanding of the topic could overcome this barrier but such groups could have different views than a wider population [16]. Some researchers report that public contributors may not be interested in discussing data science [17] and can be more comfortable with involvement in qualitative rather than quantitative research (thanks to their lived experiences) [18]. To facilitate public involvement in complex topics additional training will be required for public contributors and researchers [19].

This paper reports on an evaluation of this novel approach to the development of materials on data science and artificial intelligence. We aimed to understand the co-production process through the following specific objectives:

  1. (1)

    To describe the experiences of working together on the project.

  2. (2)

    To explore how the process supports the personal development of public contributors.

  3. (3)

    To gain insights into the co-production process.

Methodology

We adopted a participatory evaluation approach where public contributors, researchers and evaluators were directly involved in planning, conducting and analysing the evaluation [20]. It has many similarities with public involvement as it rebalances the power between those involved. This approach was jointly agreed between everyone involved. Participatory evaluation has been successfully used to evaluate public involvement [21]. We checked our reporting of public involvement in this evaluation using the GRIPP2 short form [22] (see Additional file 1). Throughout the evaluation process, we used creative methods [23] to make involvement more easily understood and appreciated by public contributors [24,25,26]. In the previous research, Tierney and colleagues [24] produced artwork to present public contributors' views and found it to be a more accessible way to disseminate research. In this section, we present the participatory evaluation process as summarised in Fig. 1.

Fig. 1
figure 1

Summary of evaluation process

Full size image

The evaluation consisted of three stages. Firstly, PT delivered three training sessions on evaluation aims, photovoice and reflexivity. Secondly, utilizing photovoice public contributors captured their experience by choosing photos and using them to reflect and describe their experiences. Thirdly, using collected photos, paintings, and poetry, we built a joint experience of involvement as a ‘mandala’ turned a geometric configuration of symbols (or photos in our case) that represent the journey from the inner core to outside through different layers. Mandala was turned into a professional illustration. The evaluation process was embedded into already ongoing project sessions with evaluators joining existing meetings.

The evaluation was in a hybrid model, with activities taking place remotely and in person. This follows recent research showing that many public contributors prefer a mix of in-person and remote involvement [27]. All online sessions and the theoretical part of the in-person meeting with the photographer were recorded for those who could not attend. Meaningful remote public involvement is possible [28, 29]. However, it might require additional support for those who feel less comfortable with technology [27]; we provided additional instructions, for example, how to upload or comment on photos on the Padlet (an online notice board). We also offered one-to-one technical support for public contributors. This included assisting a public contributor to be one of the panellists during the Tweet Chat.

Stage 1: training sessions

The first training focused on the aim of the evaluation. During the second one, everyone has received training around photovoice. We discussed ethics, photovoice aims, brief instructions on taking photos and reassured everyone that they could use phone cameras as no professional equipment is needed. In contrast to some other photovoice projects [30, 31] we did not provide disposable cameras as people felt that they would have access to one on their phone or tablet. When introducing the concept, everyone looked at sample photos and was asked to choose one representing their experience and share it with others during the discussion. This exercise aimed to help people reflect on how to link photos to their experiences. This is followed by another piece of training on reflexivity. Being reflective has become recognised as an important feature in research [32, 33]. Understanding our biases helps us to see what we bring to the discussion. Reflection is an ongoing process that starts even before research begins and is never final and complete. As public contributors reflected on their experiences through photovoice and were later involved in the analysis, it was crucial for them to reflect on their biases.

Stage 2: photovoice

Photovoice, developed by Wang and Burris [34], is a participatory research method that utilises photos taken by participants. Photovoice is embedded in literature around feminism, community-based approaches and education for critical consciousness [34, 35]. Photos (taken and selected by people) accompanied by stories behind their meaning allow their authors to reflect more critically and present their experiences [36, 37]. Joining photos and words offers a more reflective space than when these modalities are presented in isolation. Authors of photos become more empowered than traditional study participants as they can decide the focus of the discussion rather than the researchers. It fosters trust with researchers and creates a sense of ownership among participants [38]. Photovoice is a popular method to empower seldom-heard groups [35, 39, 40] and has been previously used to involve communities around health issues [31].

Photovoice is a flexible method, and it can be successfully adapted to suit different contexts, groups and projects [41]. Following the principles of participatory evaluation, researchers, evaluation facilitators and public contributors were invited to participate in photovoice. Hence, we refer to everyone as participants. Everyone's voice was equal, allowing us to break power disbalances between evaluators, professionals and public contributors.

Photovoice does not require professional photographing skills. However, during discussions with public contributors, we recognised that some do not feel comfortable taking good-quality photos themselves. Therefore, we organised an in-person session with a professional photographer who provided tips about taking photos and then people had an opportunity to practise with the photographer on a one-to-one basis. This allowed people to build confidence when positively appraised by a professional photographer [42]. The session took place in a local charity centre in central London, thus ensuring enough room for everyone to move around and take photos. They were encouraged to bring some props to take a photo of that to express their experience of being involved in the project. One public contributor could not take photos themselves due to tremors in their hands so one of evaluators support them. The public contributor created a scene from their props, and the evaluator took a photo of it. Then, we discussed all photos taken during the session as a group. Although this was not compulsory, many of them were later selected for photovoice. The session's main aim was for people to improve their confidence when taking photos.

Everyone was invited to take photos and upload them online on Padlet. We kept it as a living document where photos and descriptions were available to see and comment on as the project progressed. This was similar to Fedorowicz and colleagues [43] use of Facebook for public involvement to allow more exchanges among public contributors. Many people used the opportunity to comment on each other's photos and encouraged others by praising their reflections or/and photos.

Stage 3: reflection meeting and dissemination

During three months period, over 60 photos (alongside paintings and poetry) were uploaded online. Photos could be reviewed individually or in a group. We organised a reflection event to look at all taken photos and choose these representing common experiences. This was achieved by building a mandala. It offers an opportunity for people to reflect and produce new data [44]. Photovoice offered individuals perspectives on the project. The mandala assisted in organising the data from the photovoice. We used it to put together everyone's experiences to represent a shared journey. This was a joint analysis during which we identified connections and relationships between different aspects of the process and built consensus around a shared experience. Mandala was built by public contributors, researchers and evaluators. We brought the graphic designer who turned our mandala into an illustration. The draft was distributed to everyone involved (not only those attending the in-person meeting) for feedback. Six people used this opportunity meaning that eleven people gave their feedback on the process, and the illustration was updated accordingly.

Photovoice is an innovative and approachable way to reach external audiences such as other public contributors, researchers or policymakers [30, 35]. Because of the ongoing Covid-19 pandemic, we decided to go for online dissemination. We utilised Tweet Chat as this offered an open space to discuss the project with registered Twitter users (and was available to read by anyone). Tweet Chat has been utilised successfully by other researchers [24]. During our Tweet Chat using #JourneyIntoAI [45], we shared the draft Mandala and the selection of photos and invited a public involvement expert and an experienced public contributor not related to our project to join us in the online conversation.

Ethical considerations

The Imperial Research Governance and Integrity Team (RGIT) was consulted about the proposed activities prior to starting. As this study is classified as evaluation and public involvement, all involved in the project are considered as collaborators not sources of data, and therefore no ethical approval is required [46]. However, we recognise the ethical considerations of using photovoice [42, 47]. Explicit consent for photo reuse was provided. Everyone's ownership of their photos remains in place, and we recognise everyone's contribution, for example, by co-authoring this paper with public contributors (everyone who took part in photovoice and attend one of face-to-face sessions could participate; public contributors decided themselves to which paper section they wanted to contribute). To avoid additional consent, we asked people not to take photos of people not involved in the project. We did not expect that sharing of included photos could impact anyone negatively. However, one public contributor requested assurance that their photos would not be identifiable. As meetings included working as a group we established ground rules to ensure everyone felt welcome and respected.

Results

We constructed the mandala (see Fig. 2) to illustrate everyone's experience of being involved in the project. These experiences are spread along six interconnected layers embodied as a connected journey. The chosen title, "Our Journey into A.I.", reflects the feeling of ownership among participants. The water and boats are present throughout as these "express the sense of adventure, sailing into new experiences as a team" (Public contributor 4). This emphasises that multiple bidirectional communication channels exist between all six layers of the mandala. This section discusses each layer of the mandala, starting from the inner core and moving towards to most outwards. We also use original commentaries of the photos collected during photovoice and used during the reflection meeting to provide more detailed descriptions. Each layer consists of related stories chosen by participants, but we would not consider them equal to themes (like could be found in thematic analysis). Participants recognised that issues discussed in inside layers could not have happened without those in more outward ones.

Fig. 2
figure 2

Mandala

Full size image

Layer 1

The project offered public contributors an opportunity to develop as individuals, and learn new things, for example, improved knowledge of data science and artificial intelligence (especially as the majority of public contributors had limited understanding of the topic) and building confidence to be involved in other future projects. Flowers in the centre symbolise growth and transition into something new (often unexpected) in how participants developed but also hope that they will keep blossoming afterwards personally and as public contributors. However, that development was a process as explained by the public contributor:

"Two pictures of the red flower show my increased confidence, understanding and knowledge of data science and artificial intelligence. I am showing my new interpersonal and communication skills by displaying more confidence" (Public Contributor 2).

These are accompanied by four things: a fallen tree, the chest, the cup and the wrapped gift that summarise participants' experiences on a personal level and worth:

"The drawing of this fallen tree in the park, shows a tree having unfruitful branches while other branches are alive and blossoming. Being part of this project tells me I am still able to contribute hopefully something of value. The chest speaks for itself as the project enables me to use life's experience and knowledge. The cup may seem odd but I am often told I am doing too much. I should rest more. I can feel like a precious porcelain cup that should be on display rather than still being used for tea. And the wrapped gift is an expression of my gratitude in being asked to participate in this project" (Public contributor 4).

Layer 2

This layer elicits participants' interactions and connections with aspects of data science, artificial intelligence, and other participants. The slogans “togetherness” and “going to into this journey together” recognise that this was a team work and the connection with other participants is described as a "bridge near the Thames":

"I chose this photo because of the connectivity that we all had. Despite being miles away from each other we were all still connected via Teams, emails etc. Just like a bridge connecting two places allowing a passage route. "(Public contributor 6).

Their joint experiences and work were explained with the references to human history as described by one of the public contributors using analogy to Alexander the Great through. This poem explains how they perceived project aims:

"We have moved forward since the days of Achilles and Alexander.

But

Our scientific community today is helping us wage war using the help of

Artificial intelligence.

Astonishingly, for the same needs and motives-

Enhancement of human life." (Public contributor 1).

The connections between the project topics, what participants found were unexpected as elicited by these two descriptions of photos:

"I love seeing patterns. Mathematical ones, those in nature [some tallying with Fibonacci numbers] and those designed by us humans e.g. the Kente cloth woven in Ghana. And so with Machine Learning. What patterns will be discovered with this data?" (Public contributor 4).

"We live in a world bombarded with data. As with ‘Alice in Wonderland’ one can desire at times to be free from data. Throw it away! I have become more aware of how data can be misused through ignorance but also by deliberately being messaged. There is need for integrity and WISDOM [hence the Pearls]. And the pennies? they represent the joy of those moments when in the confusion THE PENNY DROPS." (Public contributor 4).

Layer 3

That connection with other participants (as being one team) continues in this layer, although here it is mostly related to digital technology that takes a centre place here. The photo of a computer screen with faces symbolises working online as the project took place remotely. This is how participants saw each other during meetings. Participants recognised that sometimes it was challenging staying active through full two-hour meetings in the evening. More than one photo (or screenshot) of zoom were included. These were captured under different circumstances to record various emotional stages of the co-production:

“The breakthrough. …but we were never too far away from a good laugh.” (Public contributor 7, describing the photo with zoom participants smiling).

“The blood sugar drop. It wasn't always easy to keep going through a two hour meeting from 5 to 7 pm…” (Public contributor 7, describing the photo with zoom participants looking tired).

The computer with keyboard and papers refer to their experience of note taking and providing feedback to draft materials.

"This picture represents me having to read through the different chapters for the project. I was intrigued and interested to learn more about the case studies and the information from the different chapters. It makes me feel like I am making a difference with my opinions and input." (Public contributor 3).

Layer 4

This layer underpins why people got involved in this project in the first place. It includes an old photo of one of the researchers with their aunt, who was diagnosed with a mental illness. This is how they described their gratitude towards everyone's involvement:

"She was diagnosed late in life with bipolar disease. I remember how she struggled and I wonder what her life might have been like if we had the means to diagnose and treat bipolar disease back then. Research matters if we are going to improve how illness is diagnosed and managed. We need everyday people to work with researchers to develop new treatments and new ways of managing disease. That is why it is important that we are here together learning about an emerging field that will very likely revolutionise healthcare. That is why I am grateful, that for all the ways you could spend your time, you choose to spend it here, with us." (Researcher 1).

However, participants also pointed out that they might not have been so interested in it at the beginning, as explained by this public contributor:

"I chose this photo because despite it being a topic I am not particularly interested in, like the waterfall I was able to immerse myself in the calm and chaos that the project presented. I had found myself brainstorming so many ideas/thoughts that it felt chaotic at times but simultaneously I felt calm in being able to build upon the opinions and suggestions everyone else had as well." (Public contributor 6).

This process of involvement had a positive outcome for participants:

"[artificial intelligence] is like rocky mountain to climb, but worth it." (Public contributor 3).

This layer also includes a recognition of the role of researchers who supported public contributors. The woman in the photo is one of the researchers working on the project (KG). Public contributors wanted to point out that their experience (and outcomes included in previous layers) would not have been possible without the researchers’ help and support.

Layer 5

This layer explores principles around public involvement and their expectations towards what should take place to ensure that the co-production process is inclusive and accessible to everyone. Everyone needs to be engaged, but this also means public contributors have to be empowered to do it by building confidence. Remote working has challenges, and participants recognised that involvement must be accessible for people who have computer difficulties. That support needs to be ongoing, and people feel comfortable requesting it. During the reflection meeting, public contributors said that they felt these requirements were met. Ensuring the supporting environment led to improved confidence and pride as the project was progressing, as shown in the photo titled the “Japanese Puzzle Box and Pearls”:

"How do you open the box? It is not easy—it can take hours to fathom how. Grappling with AI, particularly machine learning, is not easy and yet in gaining some understanding it brings a sense of discovering pearls of knowledge" (Public contributor 4).

Layer 6

The most outmost layer shows what the participants perceived as underlying principles of data and artificial intelligence usage for research and healthcare services. They were: ethical and safe artificial intelligence, creating something that will last (as this should not be a one-off event or improvement), change should be a shared endeavour, and if more people are involved then more stories will be included (thus more meaningful change will take place). As this public contributor expressed in this poem around their red roses, when done in an appropriate way data science and artificial intelligence can make a real-life difference:

"The red rose tree, is about 60 years old.

The yellow rose tree, is about 40 years old.

They enhance my life in joy and serenity with their beauty.

They demand careful care to do this, in feeding and pruning and watering. Then the roses flourish and only bring happiness.

This reminds me, of the artificial intelligence, I am studying at the moment.

Like my roses, when artificial intelligence is given careful care, pruning the parts that could endanger, feeding it the right ideas, it will enhance health and life, in joy and serenity. "(Public contributor 1).

These principles were also applicable to what participants expected from this project. Without them being met, they would not feel comfortable being involved in it. This layer is connected to the previous one throughout the growing tree (having some roots in this layer) that "represents growth & expansion, growing in knowledge and learning new things" (Public contributor 5). Thus, showing the interconnectivity between public involvement principles and those of data science and artificial intelligence.

Discussion

This participatory evaluation of the co-production process using creative methods has illustrated everyone’s experience of being involved in the project to co-design and co-develop new materials around data science and AI. The evaluation revealed that the entire project was in line with high-quality co-production and public involvement standards: sharing power, up-skilling/learning new things, and providing inclusive and accessible space.

Public involvement requires planning and time [48], and creative approaches (such as photovoice and the mandala) might require additional resources and time but offer a novel way of presenting public contributors' experiences. Photovoice is often seen as an accessible method of participatory approaches; however, a significant challenge is the digital literacy needed to utilize photographic tools (i.e. smart phones, cameras, etc.). This study demonstrated both the accessibility of photovoice, the techniques and adequate support needed to circumvent the usual pitfalls of the common challenges of such a strategy.

The use of nonverbal communication, in this instance, photography and visual mediums, overcomes conventional barriers of language. Photos are known to enable people to make and share their experiences [49]. In this study, we could see how everyone communicated their feelings, thoughts and responses in a way that did not need to be translated. In public involvement, we often work with diverse groups who have mastery of separate languages. Although this is an amazing dynamic, it often means the communication between parties can be difficult to navigate; however, the use of photos and images allowed us to form new ways of communicating that saw past our expected barriers of language. Researchers using photovoice often utilise interview or focus group transcript data to answer research aims rather than photos [41]. Our approach was to put photos at the centre of the analysis and dissemination, hence the creation of the mandala.

The project was tackling a notoriously dense and inaccessible topic—data science and AI. However, this project presented a method of communicating these intense topics and subjects in a manner that was easy to ingest both for everyone. The final application of this methodology was the visualization of a mandala where the independent stories from each of our participants were sewn together into a greater narrative of our research. This method was particularly impactful as it took the place of a traditional findings and critical analysis section, and in its place, we had this visual and deeply emotional reflection on our shared perspectives on this work. Broomfield [50] used creative methods during her public involvement activities and found that discussion based on metaphors of things brought creativity in group members to describe the involvement group they were a part of, understand each other views and thus see different perspectives to identify group dynamics and relationships. This was also our experience when we built the mandala.

The evaluation revealed that their motivations for joining the project were important. These potentially also influenced to the point public contributors’ willingness to sustain their involvement throughout the project. This aligns with previous research with public contributors emphasising that motivation to make a difference is an important aspect of why public members join the research project as public contributors [51]. The public mostly supports clinical AI but has some reservations [4]. Our findings suggest new learnings around AI and data science targeted at researchers and the academic community—it is that extra care and ethical approaches should be taken while deploying data science and AI within the healthcare setting. Meaningful public involvement is required to ensure ethical data science and AI. This could be achieved by following established standards, for example, key principles for public involvement and engagement in data-intensive health research [52].

Public involvement, by its nature, is an ‘alternate’ form of research. The use of creativity allows for the established walls and rules of conventional academia to be broken down and manipulated into a shape where everyone is able to participate. When both researchers and members of the public are encouraged to be curious, imaginative and creative, we develop a holistic and deep understanding of what we are trying to explore. We shed the barriers of a convention that ultimately limit our ability to participate, engage and connect both with each other and the problems we seek to understand.

Creative approaches alongside reflexivity allowed for accessible, inclusive, and therefore more ethical involvement of public partners in the evaluation process around the complex and often abstract topics like data science and AI. Reflexivity offered additional time and space for deeper consideration that is needed for ethical involvement. Following Helen Kara [33] we perceive reflexivity as an opportunity for debiasing, taking a step back to consider what one knows about the process and how one is aware of it. Public contributors can have biases when expressing their views [53]. Thus, our evaluation process showed that reflexivity could address that problem. We embedded public contributors’ reflexivity throughout the evaluation process and when co-writing this paper. The following two section consists of reflections from two public contributors who are co-authors.

Reshma’s reflections

I had experience as a patient and public contributor in other branches of medicine, but when I started to participate with Imperial College as a public contributor on this project, I was not aware of the existence of the concept of AI. I had absolutely no background in science or mathematics.

I then discovered the prominent role artificial intelligence will play in medicine. I have had caring responsibilities towards my family and decided to understand and engage with this science to help my family and myself when we are unwell and, indeed, be useful and active participants in the health service. I can now help as a patient and public contributor in developing tools and strategies for the dissemination of AI in medicine to others like myself. I am now a part of two steering groups for artificial intelligence in dementia and a PPI member for artificial intelligence in respiratory care in intensive care medicine. I think it is imperative that for the patient and the health service to succeed, people understand, as far as possible and actively participate in using artificial intelligence in medicine.

Suzanne’s reflections

My introduction to Artificial Intelligence came through being involved in a project whereby an algorithm was developed using NHS data for medical diagnostic use. This present project has furthered my understanding while making me realise how little I know.

What did I enjoy most? Certainly, the creation of photos and paintings to express visually the understanding I had gained and the feelings experienced. I recall surveying my flat to choose objects to photograph or paint and being surprised at which items I selected and why. Focusing on the process was illuminating!

It is so easy to have an experience without spending time reflecting on it. In so doing one fails to recognise much of what one has learned and the impact it has had. Thus, Photovoice has been a very affirmative activity and I will continue using art forms for reflection and expressing emotions.

Strengths and limitations

Using a participatory approach allowed us to bring people to offer different perspectives and views and ensured the feeling of ownership among all involved, including public contributors. This aligns with the review of the literature on photovoice in health and public health that photovoice could use participatory analysis [41]. However, photovoice could also be used alongside further transcribed interviews or focus groups, thus allowing data triangulation. Due to the project timeline, this was not possible to achieve. Future research could combine conventional and creative methods to produce more nuanced findings [23].

Due to funding restrictions, the evaluation started when the project was already ongoing. We recognise that we managed to capture the ongoing experience of everyone during three months of photovoice, but for previous months, this was mostly reflective and was not apparent in the findings. Teodorowski and colleagues [54], explored the experiences of researchers and public involvement facilitators in involving and engaging seldom-heard communities in big data research; they recognise that there is no one right approach to working with all communities. Our evaluation suggests that researchers can use creative approaches as one of the effective ways of involving members of the public around data science and AI. We recommend that other researchers consider applying participatory evaluation to explore their co-production process. However, we recognise that participatory evaluation can take longer and require more resources than alternatives.

Conclusions

This paper reports the findings from the participatory evaluation of the co-production process with public contributors. By using creative approaches and embedding reflexivity, the findings suggest that working with public contributors on technical topics that might seem abstract and full of jargon for lay members of the public can be achieved meaningfully for both researchers and public contributors. The findings showed that the co-production process ensured that everyone involved had a true feeling of ownership over the project.

Availability of data and materials

Anonymised data from photovoice is available on request.

References

  1. Aitken M, de St. Jorre J, Pagliari C, Jepson R, Cunningham-Burley S. Public responses to the sharing and linkage of health data for research purposes: a systematic review and thematic synthesis of qualitative studies. BMC Medical Ethics. 2016;17(1):73.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Aitken M, Cunningham-Burley S, Pagliari C. Moving from trust to trustworthiness: experiences of public engagement in the Scottish Health Informatics Programme. Sci Publ Policy. 2016;43(5):713–23.

    Article  Google Scholar 

  3. Kalkman S, Van Delden J, Banerjee A, Tyl B, Mostert M, Van Thiel G. Patients’ and public views and attitudes towards the sharing of health data for research: a narrative review of the empirical evidence. J Med Eth. 2022;48(1):3–13.

    Article  Google Scholar 

  4. Young AT, Amara D, Bhattacharya A, Wei ML. Patient and general public attitudes towards clinical artificial intelligence: a mixed methods systematic review. Lancet Digit Health. 2021;3(9):e599–611.

    Article  CAS  PubMed  Google Scholar 

  5. Creary N, Amevenu D, Shivji J, H HC. What it means to be seen: Closing gaps in patient data for Black and South Asian Communities. London: Liberating Knowledge; 2022.

  6. Ocloo J, Matthews R. From tokenism to empowerment: progressing patient and public involvement in healthcare improvement. BMJ Qual Saf. 2016;25(8):626–32.

    Article  PubMed  PubMed Central  Google Scholar 

  7. NIHR. Briefing notes for researchers - public involvement in NHS, health and social care research 2021 [Available from: https://www.nihr.ac.uk/documents/briefing-notes-for-researchers-public-involvement-in-nhs-health-and-social-care-research/27371.

  8. NIHR. National Standards for Public Involvement. 2019.

  9. Teng J, Bentley C, Burgess MM, O’Doherty KC, Mcgrail KM. Sharing linked data sets for research: results from a deliberative public engagement event in British Columbia Canada. Int J Popul Data Sci. 2019. https://doi.org/10.23889/ijpds.v4i1.1103.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Jones KH, McNerney CL, Ford DV. Involving consumers in the work of a data linkage research unit. Int J Consum Stud. 2014;38(1):45–51.

    Article  Google Scholar 

  11. Deverka PA, Gilmore D, Richmond J, Smith Z, Mangrum R, Koenig BA, et al. Hopeful and concerned: public input on building a trustworthy medical information commons. J Law Med Eth. 2019;47(1):70–87.

    Article  Google Scholar 

  12. Ballantyne A, Stewart C. Big data and public-private partnerships in healthcare and research. Asian Bioeth Rev. 2019;11(3):315–26.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Evans EA, Delorme E, Cyr K, Goldstein DM. A qualitative study of big data and the opioid epidemic: recommendations for data governance. BMC Med Eth. 2020;21(1):1–13.

    Google Scholar 

  14. Steel R. Actively involving marginalized and vulnerble people in research. In: Lowes L, Hulatt I, editors. Involving service users in health and social care research. Abingdon: Routledge; 2005.

    Google Scholar 

  15. Jones KH, Heys S, Thompson R, Cross L, Ford D. Public involvement & engagement in the work of a data safe haven: a case study of the SAIL Databank. Int J Popul Data Sci. 2020;5(3):1371.

    PubMed  PubMed Central  Google Scholar 

  16. Kirkham EJ, Crompton CJ, Iveson MH, Beange I, McIntosh AM, Fletcher-Watson S. Co-development of a best practice checklist for mental health data science: a Delphi study. Front Psychiatry. 2021. https://doi.org/10.3389/fpsyt.2021.643914.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Mouton Dorey C, Baumann H, Biller-Andorno N. Patient data and patient rights: Swiss healthcare stakeholders’ ethical awareness regarding large patient data sets-a qualitative study. BMC Med Eth. 2018;19(1):20.

    Article  Google Scholar 

  18. Boylan AM, Locock L, Thomson R, Staniszewska S. “About sixty per cent I want to do it”: health researchers’ attitudes to, and experiences of, patient and public involvement (PPI)—A qualitative interview study. Health Expect. 2019;22(4):721–30.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Jewell A, Pritchard M, Barrett K, Green P, Markham S, Mckenzie S, et al. The Maudsley Biomedical Research Centre (BRC) data linkage service user and carer advisory group: creating and sustaining a successful patient and public involvement group to guide research in a complex area. Res Involv Engagem. 2019;5(1):1–10.

    Article  Google Scholar 

  20. Zukoski AP, Bosserman C. Essentials of participatory evaluation. In: Fetterman DM, Rodríguez-Campos L, Zukoski AP, editors. Collaborative, participatory, and empowerment evaluation: stakeholder involvement approaches. Guilford Publications; 2017. p. 48–56.

    Google Scholar 

  21. Vat LE, Warren M, Goold S, Davidge E, Porter N, Schuitmaker-Warnaar TJ, et al. Giving patients a voice: a participatory evaluation of patient engagement in Newfoundland and Labrador Health Research. Res Involv Engagem. 2020;6(1):1–14.

    Article  Google Scholar 

  22. Staniszewska S, Brett J, Simera I, Seers K, Mockford C, Goodlad S, et al. GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. BMJ. 2017. https://doi.org/10.1136/bmj.j3453.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Kara H. Creative research methods: a practical guide. Policy Press; 2020.

    Book  Google Scholar 

  24. Tierney S, Dawson S, Boylan A-M, Richards G, Park S, Turk A, et al. Broadening diversity through creative involvement to identify research priorities. Res Involv Engagem. 2021;7(1):1–10.

    Article  Google Scholar 

  25. Gilbertson A, Ridd MJ, Sutton E, Liddiard L, Clayton J, Roberts A, et al. Engaging with diverse audiences to raise awareness about childhood eczema: reflections from two community events. Res Involv Engagem. 2021;7(1):1–8.

    Article  Google Scholar 

  26. Broomfield K, Craig C, Smith S, Jones G, Judge S, Sage K. Creativity in public involvement: supporting authentic collaboration and inclusive research with seldom heard voices. Res Involv Engagem. 2021;7(1):1–14.

    Article  Google Scholar 

  27. Jones E, Frith L, Gabbay M, Tahir N, Hossain M, Goodall M, et al. Remote working in public involvement: findings from a mixed methods study. Res Involv Engagem. 2022;8(1):58.

    Article  PubMed  PubMed Central  Google Scholar 

  28. Adeyemi I, Sanders C, Ong BN, Howells K, Quinlivan L, Gorman L, et al. Challenges and adaptations to public involvement with marginalised groups during the COVID-19 pandemic: commentary with illustrative case studies in the context of patient safety research. Res Involv Engagem. 2022;8(1):13.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Lampa E, Sonnentheil B, Tökés A, Warner G. What has the COVID-19 pandemic taught us about conducting patient and public involvement remotely? Insights from a series of digital meeting observations. Res Involv Engagem. 2021;7(1):1–8.

    Article  Google Scholar 

  30. Nault Connors JD, Conley MJ, Lorenz LS. Use of photovoice to engage stakeholders in planning for patient-centered outcomes research. Res Involv Engagem. 2019;5(1):1–10.

    Article  Google Scholar 

  31. Mtuy TB, Mepukori J, Lankoi J, Lees S. Empowering maasai women behind the camera: photovoice as a tool for trachoma control. Res Involv Engagem. 2021;7(1):1–11.

    Article  Google Scholar 

  32. Braun V, Clarke V. Thematic analysis: a practical guide. London: Sage; 2021.

    Google Scholar 

  33. Kara H. Research ethics in the real world. Bristol: Policy Press; 2018.

    Book  Google Scholar 

  34. Wang C, Burris MA. Empowerment through photo novella: portraits of participation. Health Educ Q. 1994;21(2):171–86.

    Article  CAS  PubMed  Google Scholar 

  35. Wang CC, Cash JL, Powers LS. Who knows the streets as well as the homeless? Promoting personal and community action through photovoice. Health Promot Pract. 2000;1(1):81–9.

    Article  Google Scholar 

  36. Wang C, Burris MA. Photovoice: concept, methodology, and use for participatory needs assessment. Health Educ Behav. 1997;24(3):369–87.

    Article  CAS  PubMed  Google Scholar 

  37. Wang CC. Photovoice: a participatory action research strategy applied to women’s health. J Womens Health. 1999;8(2):185.

    Article  CAS  PubMed  Google Scholar 

  38. Castleden H, Garvin T. Modifying Photovoice for community-based participatory Indigenous research. Soc Sci Med. 2008;66(6):1393–405.

    Article  PubMed  Google Scholar 

  39. Watson M, Douglas F. It’s making us look disgusting…and it makes me feel like a mink…it makes me feel depressed!: using photovoice to help ‘see’ and understand the perspectives of disadvantaged young people about the neighbourhood determinants of their mental well-being. Int J Health Promot Educ. 2012;50(6):278–95.

    Article  Google Scholar 

  40. Duffy LR. Hidden heroines: lone mothers assessing community health using photovoice. Health Promot Pract. 2010;11(6):788–97.

    Article  PubMed  Google Scholar 

  41. Catalani C, Minkler M. Photovoice: a review of the literature in health and public health. Health Educ Behav. 2010;37(3):424–51.

    Article  PubMed  Google Scholar 

  42. Wang CC, Redwood-Jones YA. Photovoice ethics: perspectives from flint photovoice. Health Educ Behav. 2001;28(5):560–72.

    Article  CAS  PubMed  Google Scholar 

  43. Fedorowicz S, Riley V, Cowap L, Ellis NJ, Chambers R, Grogan S, et al. Using social media for patient and public involvement and engagement in health research: the process and impact of a closed Facebook group. Health Expect. 2022;25(6):2786–95.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Palmer VJ, Dowrick C, Gunn JM. Mandalas as a visual research method for understanding primary care for depression. Int J Soc Res Methodol. 2014;17(5):527–41.

    Article  Google Scholar 

  45. Twitter. Tweetchat: Our Journey Into AI 2022 [Available from: https://twitter.com/i/events/1590708227657834496.

  46. Close JB, Bornemann J, Piggin M, Jayacodi S, Luan LX, Carhart-Harris R, et al. Co-design of guidance for patient and public involvement in psychedelic research. Front Psychiatry. 2021;12:1297–9.

    Article  Google Scholar 

  47. Creighton G, Oliffe JL, Ferlatte O, Bottorff J, Broom A, Jenkins EK. Photovoice ethics: critical reflections from men’s mental health research. Qual Health Res. 2018;28(3):446–55.

    Article  PubMed  Google Scholar 

  48. Romsland GI, Milosavljevic KL, Andreassen TA. Facilitating non-tokenistic user involvement in research. Res Involv Engagem. 2019;5(1):1–12.

    Article  Google Scholar 

  49. Rogers C. Necessary connections: ‘feelings photographs’ in criminal justice research. Methodol Innov. 2020;13(2):205979912092525.

    Article  Google Scholar 

  50. Broomfield K. The unspoken voices project: creating accessible research through collaborative design. In: Volkmer A, Broomfield K, editors. Seldom heard voices in service user involvement. Croydon: J&R Press; 2022. p. 111–28.

    Google Scholar 

  51. Cook L, Rothstein P, Emeh L, Frumiento P, Kennedy D, McNicholas D, et al. In the physical to digital transition with friends—A story of performing inclusive research together no matter what life throws at you. Br J Learn Disabil. 2021;49(3):271–81.

    Article  PubMed  PubMed Central  Google Scholar 

  52. Aitken M, Tully M, Porteous C, Denegri S, Cunningham-Burley S, Banner N, et al. Consensus statement on public involvement and engagement with data-intensive health research. Int J Popul Data Sci. 2019. https://doi.org/10.23889/ijpds.v4i1.586.

    Article  PubMed  PubMed Central  Google Scholar 

  53. Rees L, Sherwood M, Shields N. Consumer engagement in doctoral research–What difference does it make? Spinal Cord. 2022;61:175–83.

    Article  PubMed  PubMed Central  Google Scholar 

  54. Teodorowski P, Rodgers SE, Fleming K, Tahir N, Ahmed S, Frith L. ‘To me, it’s ones and zeros, but in reality that one is death’: a qualitative study exploring researchers’ experience of involving and engaging seldom-heard communities in big data research. Health Expect. 2023;26:882–91.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We would like to thanks all public contributors involved in the project. Furthermore, we would like to acknowledge the involvement of artist Federica Ciotti and photographer Stephen D Lawrence.

Funding

PT is a NIHR Short Placement Award for Research Collaboration awardee supported by the National Institute for Health and Care Research (NIHR) and his research is funded by the National Institute for Health Research Applied Research Collaboration North West Coast. HW is a National Institute for Health and Care Research Senior Investigator, and acknowledges support from the NIHR Imperial Biomedical Research Centre (BRC) and the NIHR Applied Research Collaboration (ARC) North West London. MM acknowledges support from the National Institute for Health Research (NIHR) Imperial Biomedical Research Centre (BRC). We acknowledge the funders support for this project from: Imperial Societal Engagement Participatory Research Seed Fund Award, Imperial Experimental Cancer Medicine Centre, Cancer Research UK Convergence Science Centre, Northwest London NIHR Applied Research Collaboration, NIHR Imperial Biomedical Research Centre Surgery Cancer Theme, NIHR Imperial Biomedical Research Centre Patient Experience Research Centre. DC, KM and HW are supported in part by the Northwest London NIHR Applied Research Collaboration. Imperial College London is grateful for support from the Northwest London NIHR Applied Research Collaboration. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.

Author information

Authors and Affiliations

  1. University of Liverpool, Liverpool, UK

    Piotr Teodorowski

  2. Imperial Cancer Research UK Lead Nurse, Department of Surgery and Cancer, Imperial College London, London, UK

    Kelly Gleason

  3. Computational Oncology Group, Department of Surgery and Cancer, Faculty of Medicine, Imperial College London, London, UK

    Jonathan J. Gregory

  4. School of Primary Care and Public Health, Imperial College London, London, UK

    Martha Martin

  5. Imperial College London, London, UK

    Reshma Punjabi, Suzanne Steer, Serdar Savasir & Pournamy Vema

  6. School of Public Health, Imperial College London, London, UK

    Kabelo Murray, Helen Ward & Dorota Chapko

  7. NIHR Applied Research Collaboration Northwest London, Imperial College London, London, UK

    Kabelo Murray, Helen Ward & Dorota Chapko

  8. National Institute for Health Research Imperial Biomedical Research Centre, London, UK

    Helen Ward

Authors
  1. Piotr Teodorowski
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kelly Gleason
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jonathan J. Gregory
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Martha Martin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Reshma Punjabi
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Suzanne Steer
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Serdar Savasir
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Pournamy Vema
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Kabelo Murray
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Helen Ward
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Dorota Chapko
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

PT developed the evaluation plan and led on drafting this paper. DC acted as a supervisor to PT, and provided PPIE expertise and contributed to the conceptualisation of the project. MM provided background to the project, KM and DC developed discussion section. All authors contributed to drafting and editing. RP, SuS, SeS and PV were public contributors involved in the project. All authors reviewed and approved the final manuscript.

Corresponding author

Correspondence to Piotr Teodorowski.

Ethics declarations

Ethics approval and consent to participate

The Imperial Research Governance and Integrity Team (RGIT) was consulted about the proposed activities prior to starting. As this study is classified as evaluation and public involvement, all involved in the project are considered as collaborators not sources of data, and therefore no ethical approval is required.

Consent for publication

Not applicable.

Competing interests

None.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

GRIPP2 short form.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Teodorowski, P., Gleason, K., Gregory, J.J. et al. Participatory evaluation of the process of co-producing resources for the public on data science and artificial intelligence. Res Involv Engagem 9, 67 (2023). https://doi.org/10.1186/s40900-023-00480-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s40900-023-00480-z

Keywords

Research Involvement and Engagement

ISSN: 2056-7529

Contact us