Experiential evidence methods

Literature searching and crowdsourcing processes identified 23 methods for gathering experiential evidence. The methods were mapped to the typology and stratified by mode, context and structure. The table includes methods identified through the literature review and crowdsourcing activity. This is not an exhaustive list of methods.

In person relies on interaction that happens in one place at a certain time, virtual relies on interaction that happens in different places remotely at the same time, digital relies on interaction that happens online in different places and at different times.

Talk-generated methods

Talk generated methods for gathering evidence. The first column has the method and a description. The other columns allow a comparison of the features of the method.
Method Revealing issues Identifying allegiances and shared views Generating solutions Reaching accordVirtualIn-personDigitalIndividualGroups Highly structuredSemi-structured

Interviews 1-7 *+

Structured, semi-structured or unstructured conversation with individuals or small groups to gather perspectives, opinions and world views.

Reveals issuesDoes not identify alliegances and shared viewsGenerates solutionsDoes not reach accordNot virtualNot in-personNot digitalIndividualGroupsHighly structuredSemi-structured

Roundtable +

Small group discussions with well-informed individuals to exchange perspectives and debate an issue or topic.

Reveals issuesIdentifies allegiances and shared viewsGenerates solutionsDoes not reach accordVirtualIn-personNot digitalNot individualGroupsNot highly structuredSemi-structured

Focus groups 1-3, 5, 7*+

In-depth exploration of an issue or topic with 8 to 12 individuals, under the direction of a facilitator, for up to 2.5 hours.

Reveals issuesDoes not identify alliegances and shared viewsGenerates solutionsDoes not reach accordVirtualIn-personNot digitalNot individualGroupsNot highly structuredSemi-structured

Yarning circles +

A conversational process that involves the sharing of stories and the development of knowledge in a culturally prescribed and respectful way.

Reveals issuesIdentifies allegiances and shared viewsGenerates solutionsDoes not reach accordVirtualIn-personNot digitalNot individualGroupsNot highly structuredNot semi-structured

Workshops +

An interactive process to discuss among groups of various sizes.

Reveals issuesDoes not identify alliegances and shared viewsGenerates solutionsDoes not reach accordVirtualIn-personNot digitalNot individualGroupsNot highly structuredSemi-structured

Nominal group technique +

A structured approach to group brainstorming which combines information gathering, idea generation and consensus building on a specific topic.

Reveals issuesIdentifies allegiances and shared viewsGenerates solutionsReaches accordVirtualIn-personNot digitalNot individualGroupsNot highly structuredSemi-structured

User or working groups 2, 8, 9*+

A small group established to process a particular issue or achieve a specific task.

Reveals issuesDoes not identify alliegances and shared viewsGenerates solutionsDoes not reach accordVirtualIn-personNot digitalNot individualGroupsNot highly structuredSemi-structured

Dialogues +

Small group sessions, usually over one day, where individuals receive information and deliberate on a topic. A formal vote, ranking exercise or pre- and post- surveys assess the deliberation outcome.

Does not reveal issuesIdentifies allegiances and shared viewsGenerates solutionsReaches accordVirtualIn-personNot digitalNot individualGroupsHighly structuredNot semi-structured

Experts or representatives 4, 9, 10*

Health professionals and consumer representatives on advisory councils, committees and participating in research or improvement initiatives and projects.

Reveals issuesDoes not identify alliegances and shared viewsGenerates solutionsDoes not reach accordVirtualIn-personNot digitalIndividualNot groupsNot highly structuredNot semi-structured

Consensus conference 8, 11*+

A panel of experts evaluate the scientific literature related to an issue or topic, hear testimonies and then formulate recommendations.

Does not reveal issuesIdentifies allegiances and shared viewsGenerates solutionsReaches accordNot virtualIn-personNot digitalNot individualGroupsHighly structuredNot semi-structured

Q-method 12*

Investigates the perspectives of individuals who represent different views on a topic through ranking and sorting statements.

Does not reveal issuesIdentifies allegiances and shared viewsDoes not generate solutionsReaches accordNot virtualIn-personDigitalNot individualGroupsHighly structuredNot semi-structured

Citizen juries or mini publics 8, 11*+

A small representative group of 12 to 24 individuals, deliberate on an issue or topic (generally one clearly framed question), over the period of 2 to 7 days. The groups are presented with evidence-based expert views.

Does not reveal issuesIdentifies allegiances and shared viewsGenerates solutionsReaches accordNot virtualIn-personNot digitalNot individualGroupsHighly structuredNot semi-structured

Citizen panel 11*+

A demographically representative group used to assess public preference or opinion.

Reveals issuesIdentifies allegiances and shared viewsDoes not generate solutionsReaches accordNot virtualIn-personNot digitalNot individualGroupsNot highly structuredSemi-structured

Narratives or storytelling 3, 13-15*+

An account of an individual’s experience and includes detail, characters and some plot. Communicated in verbal, print, audio or video formats.

Reveals issuesDoes not identify alliegances and shared viewsDoes not generate solutionsDoes not reach accordVirtualIn-personNot digitalIndividualGroupsNot highly structuredSemi-structured

Vignettes or case studies 2*+

Self-reported verbal, print, audio or video illustrations, including key elements of a scenario, experience or setting.

Reveals issuesDoes not identify alliegances and shared viewsGenerates solutionsDoes not reach accordVirtualIn-personDigitalIndividualGroupsHighly structuredSemi-structured

Observation 1, 5, 16*+

Gathering information by observing people, places and/or processes either directly or indirectly.

Reveals issuesDoes not identify alliegances and shared viewsDoes not generate solutionsDoes not reach accordNot virtualIn-personNot digitalIndividualGroupsNot highly structuredSemi-structured

Text-generated methods

Text generated methods for gathering evidence. The first column has the method and a description. The other columns allow a comparison of the features of the method.
Method Revealing issues Identifying allegiances and shared views Generating solutions Reaching accordVirtualIn-personDigitalIndividualGroups Highly structuredSemi-structured

Checklists / rating scales 6*

A checklist or rating scale presents a list of events or characteristics on which a person can indicate, by a check mark or rating on a scale, whether the event or characteristic of interest did occur or is present.

Reveals issuesIdentifies allegiances and shared viewsDoes not generate solutionsDoes not reach accordNot virtualNot in-personDigitalIndividualNot groupsHighly structuredNot semi-structured

Polling 8*+

Measures informed opinion on a particular issue, through aggregated individual opinions, rather than consensus.

Reveals issuesIdentifies allegiances and shared viewsDoes not generate solutionsDoes not reach accordNot virtualNot in-personDigitalNot individualGroupsHighly structuredNot semi-structured

Delphi +

Iterative multi-stage process designed to combine opinion into group consensus, through multiple rounds of opinion sharing, discussion and aggregation of quantitative data.

Does not reveal issuesIdentifies allegiances and shared viewsDoes not generate solutionsReaches accordVirtualIn-personDigitalNot individualGroupsHighly structuredNot semi-structured

Social media content 17, 18*+

Written posts, comments and microblogs on social media platforms and networking sites.

Reveals issuesDoes not identify alliegances and shared viewsGenerates solutionsDoes not reach accordVirtualNot in-personDigitalIndividualNot groupsNot highly structuredNot semi-structured

Questionnaire / survey 2, 4, 6, 8, 19*+

Specific set of open* or closed written questions to gather information from a large number of individuals.

Does not reveal issuesIdentifies allegiances and shared viewsDoes not generate solutionsDoes not reach accordVirtualIn-personDigitalIndividualNot groupsHighly structuredNot semi-structured

Visual or digital-generated methods

Visual and or digital generated methods for gathering evidence. The first column has the method and a description. The other columns allow a comparison of the features of the method.
Method Revealing issues Identifying allegiances and shared views Generating solutions Reaching accordVirtualIn-personDigitalIndividualGroups Highly structuredSemi-structured

Crowdsourcing 20-22*

A process of having a large group, including experts and non-experts, share ideas or solve issues through activities, such as open challenges, contests, hackathons and innovation tournaments.

Reveals issuesIdentifies allegiances and shared viewsGenerates solutionsReaches accordVirtualIn-personDigitalIndividualGroupsNot highly structuredSemi-structured

Creative arts 3, 5, 7, 23*+

Art genres, such as performance, writing, painting, photography, collage and installation art, as a method or as technical, communication or aesthetic element.

Reveals issuesDoes not identify alliegances and shared viewsDoes not generate solutionsReaches accordVirtualIn-personDigitalIndividualNot groupsNot highly structuredNot semi-structured

Notes

  1. * Peer reviewed literature
  2. + Crowdsourcing

References

  1. Ottrey E, Jong J, Porter J. Ethnography in nutrition and dietetics research: a systematic review. J Acad Nutr Diet. 2018 Oct;118(10):1903-42.e10. DOI: 10.1016/j.jand.2018.06.002
  2. Mockford C, Staniszewska S, Griffiths F, Herron-Marx S. The impact of patient and public involvement on UK NHS health care: a systematic review. Int J Qual Health Care. 2012 Feb;24(1):28-38. DOI: 10.1093/intqhc/mzr066.
  3. Haijes HA, van Thiel GJMW. Participatory methods in pediatric participatory research: a systematic review. Pediatr Res. 2016 May;79(5):676-83. DOI: 10.1038/pr.2015.279
  4. Jones EL, Williams-Yesson BA, Hackett RC, et al. Quality of reporting on patient and public involvement within surgical research: a systematic review. Ann Surg. 2015 Feb;261(2):243-50. DOI: 10.1097/SLA.0000000000000768
  5. Tugault-Lafleur CN, Black JL, Barr SI. A systematic review of methods to assess children's diets in the school context. Adv Nutr. 2017 Jan 17;8(1):63-79. DOI: 10.3945/an.116.013144
  6. Allen EN, Chandler CIR, Mandimika N, et al. Eliciting adverse effects data from participants in clinical trials. Cochrane Database Syst Rev. 2018 Jan 16;1(1):MR000039. DOI: 10.1002/14651858.MR000039.pub2.
  7. Bradbury-Jones C, Isham L, Taylor J. The complexities and contradictions in participatory research with vulnerable children and young people: A qualitative systematic review. Social Sci Med. 2018 Oct;215:80-91. DOI: 10.1016/j.socscimed.2018.08.038
  8. Dalton J, Chambers D, Harden M, et al. Service user engagement in health service reconfiguration: a rapid evidence synthesis. J Health Serv Res Policy. 2016 Jul;21(3):195-205. DOI: 10.1177/1355819615623305
  9. Dawson S, Campbell SM, Giles SJ, et al. Black and minority ethnic group involvement in health and social care research: A systematic review. Health Expect. 2018 Feb;21(1):3-22. DOI: 10.1111/hex.12597
  10. Concannon TW, Fuster M, Saunders T, et al. A systematic review of stakeholder engagement in comparative effectiveness and patient-centered outcomes research. J Gen Intern Med. 2014 Dec;29(12):1692-701. DOI: 10.1007/s11606-014-2878-x
  11. Street J, Duszynski K, Krawczyk S, Braunack-Mayer A. The use of citizens' juries in health policy decision-making: a systematic review. Social Sci Med. 2014 May;109:1-9. DOI: 10.1016/j.socscimed.2014.03.005
  12. Dziopa F, Ahern K. A systematic literature review of the applications of Q-technique and its methodology. Methodology (Gott). 2011;7(2):39-55. DOI: 10.1027/1614-2241/a000021
  13. O'Brien MR, Clark D. Use of unsolicited first-person written illness narratives in research: Systematic review. J Adv Nurs. 2010 Aug;66(8):1671-82. DOI: 10.1111/j.1365-2648.2010.05349.x
  14. Miloyan B, McFarlane K, Vasquez-Echeverria A. The adapted autobiographical interview: a systematic review and proposal for conduct and reporting. Behav Brain Res. 2019 Sep 16;370:111881. DOI: 10.1016/j.bbr.2019.03.050
  15. Fadlallah R, El-Jardali F, Nomier M, et al. Using narratives to impact health policy-making: a systematic review. Health Res Policy Syst. Mar 5;17(1):26. DOI: 10.1186/s12961-019-0423-4
  16. Bradbury-Jones C, Isham L, Taylor J. The complexities and contradictions in participatory research with vulnerable children and young people: A qualitative systematic review. Soc Sci Med. 2018 Oct;215:80-91. DOI: 10.1016/j.socscimed.2018.08.038
  17. Koskan A, Klasko L, Davis SN, et al. Use and taxonomy of social media in cancer-related research: a systematic review. Am J Public Health. 2014 Jul;104(7):e20-37. DOI: 10.2105/AJPH.2014.301980
  18. Mairs K, McNeil H, McLeod J, et al. Online strategies to facilitate health-related knowledge transfer: a systematic search and review. Health Info Libr J. 2013 Dec;30(4):261-77. DOI: 10.1111/hir.12048
Back to top