The activities described in this paper were completed at a one-day symposium at a large academic hospital located in Toronto, Canada in September of 2018. Further details related to the symposium are discussed in the following sections.
Planning and development of symposium
The planning and development of the symposium was completed in close collaboration with four patients and family members. One of the authors of this paper (RM) is a patient representative who participated in all aspects of this research project including, but not limited to, brainstorming, logistics planning, findings interpretation and manuscript submission. In addition, a Patient and Family Advisory Council, which consisted of four members (two patients and two family members), provided substantial input throughout the planning stage at monthly meetings with the members of the research team. This took place over the course of 6 months prior to the event. After the event, there were two debrief meetings that took place.
The goal of the symposium was to reach a consensus on essential elements for the guidance document on how to best engage patients and family members in health IT initiatives, as well as identify potential knowledge translation strategies to disseminate these findings to healthcare organizations. The symposium began with a keynote speech on the topic of ‘Patient Engagement in Health Informatics Research’ followed by a panel talk representing patients, family members, and health professionals’ experiences on collaboratively engaging in health IT initiatives. To further ensure all participants had at least a baseline understanding of the symposium’s research activities, another presentation on the results of a literature review  and findings from focus groups with patients on the topic of patient engagement methods in health IT initiatives was delivered. The two focus groups, which were conducted prior to the symposium, surveyed the perspectives of patients and family members on patient engagement in healthcare in general, and strategies frequently used in these initiatives . In the afternoon, the Group Priority Sort technique was used to obtain structured feedback from the participants in small groups. The content of the Group Priority Sort was informed by previous focus groups with patients, and feedback from patients and family members. At the end of the day, participants were asked to fill in a paper survey to evaluate their experience participating in the symposium. Ethical approval was obtained from the Centre for Addiction and Mental Health Research Ethics Board (004/2018).
Participants and recruitment
To recruit participants with diverse backgrounds and perspectives, targeted email invitations were sent to health professionals, researchers, vendor representatives, patients, family members of patients, students, and knowledge users (e.g., informatics professionals employed at a healthcare organization in Ontario). Approximately 6–8 invitations were sent to each of these user groups. Recruitment continued through a ‘snowball’ technique where members of the Patient and Family Advisory Council at the main study site and invited recipients were encouraged to forward the invite to other individuals who may be interested in participating in the symposium. Given that a ‘snowball’ technique was used for recruitment, it is unknown how many participants were invited in total. Additionally, recruitment activities included posting of recruitment material on an organizational website and an organizational research portal, and advertising in a newsletter through the Office of Family Engagement. Eligibility for participation included being a member of one of the previously mentioned participant groups and having some prior knowledge of the topic. For those participants who were invited to the event via email, conflicting schedules and prior commitments were the most common reasons given for being unable to attend.
A total of 37 individuals attended the symposium. Specifically, the participants were comprised of 4 patients, 2 family members of patients, 9 graduate students, 2 health informatics researchers, 9 health professionals, 2 health IT vendor representatives, and 9 knowledge users (e.g., management staff, project analysts). Participants worked at, or had experience at, a variety of healthcare organizations from various urban and rural settings in Ontario, Canada.
Group Priority Sort
This process was used to reach consensus within small, diverse groups of 4–6 participants in a manner which was highly relevant given the diverse background and role (e.g., knowledge user, patients) of stakeholders in health IT initiatives and the subjective nature of this research. This method allows for rich discussions to take place to shape how consensus is gained. In the following sections, the steps that were taken to complete the group priority sort technique are described.
Step 1: Group rapid-sort
To begin the group rapid-sort process, participants were asked to divide into groups of 4–6. To ensure diversity within each group, a pseudo single-blind process was used where each participant was provided with a colored sticker based on their stakeholder type (e.g. patient, health professional). The participants were then instructed to form groups with a diverse set of sticker colors, unaware of the color’s purpose. A facilitator and a note-taker were randomly assigned to each group and the former presented the group with a deck of 30 pre-populated index cards. Each card displayed a unique item (potential content for the resource), which had been identified through a preceding literature review and focus groups. Some of the items were slightly modified by the research team to ensure they could be adequately understood and assessed at face value by all participants. Facilitators then asked the group of participants to individually rate the first item on a Likert scale from 1 to 5 (1 being the least important, and 5 being the most important) on how the given item should be prioritized as a topic to be included in the resource that would be developed. The participants were encouraged to give a rating quickly with little consideration in an effort to capture their instinctive judgement. Following the 30 items, participants were asked to brainstorm any items which they believed to be missing. Any new items were recorded and rated by the group. Participants’ ratings for all 30 items, in addition to those added by the group, were recorded by note-takers. The main goal of this step was to provide a baseline for the last phase of the group priority sort technique which is called ‘group-forced sort’ and requires the group to come to consensus on their priorities .
Step 2: Group brainstorming
In the same groups, participants were asked to brainstorm potential methods and outlets for knowledge translation and dissemination of the resource, as well as how the final product should be formatted and/or structured (e.g. how it should look, how it should be accessed). This portion of the group activity was included as an additional part of the symposium’s research and not part of the Group Priority Sort. Participants were asked to be as descriptive and detailed as possible when making these suggestions. Note-takers captured the discussion points on white-paper.
Step 3: Group forced-sort
The group forced-sort required consensus within each group. To facilitate this, all index cards from the ‘Group Rapid Sort’ phase were presented back to the group marked with the participant ratings for reference. The participants were then asked to sort all the items into a 5-point Likert scale as a group, and encouraged to discuss amongst themselves to do so. They were instructed that each Likert scale point, or ‘category’, had to contain at least 6 but not more than 7 (because participants could have added items to the original 30). Both the facilitator and note-taker (who are not participants) were actively present to aid the participants in thinking critically about their priorities while ensuring all opinions were being heard in achieving consensus. The main goal achieved through this activity was to identify the relative priority of each item compared to the rest of the items in the set through strict parameters . At the end of group forced-sort activities, groups presented back to the broader group to share their rankings as well as thought process and rationale for their rankings.
In order to gain insight into the experience, needs and perceptions of participants, a brief evaluation was conducted following the symposium. Participants completed a paper survey to provide feedback on the symposium by rating elements on a 7-point Likert scale (e.g., venue, speakers, refreshments, research activities). This evaluation was followed by a second electronic survey sent out 3 days after the symposium by email through an online survey platform, which asked participants to respond to open-ended questions on the research activities and obtain their overall impression of the symposium once they had a chance to reflect upon it. Quantitative data was analyzed using descriptive measures, and qualitative analysis on open-ended responses was conducted using content analysis .