PROJECT 2 EXAMPLE: Feedback X Prevalence Using Dermatology Stimuli

NCT05244122 | Completed Results

• 2 other identifiers: 2007P000646-AR01EY017001
• Study Type: interventional
• Target: 1,121
• Locations: 1 country
• Sites: 1

Brief Summary

Imagine that a dermatologist spends the morning seeing patients who have been referred for suspicion of skin cancer. Many of them do, in fact, have skin lesions that require treatment. For this set of patients, disease 'prevalence' would be high. Suppose that the next task is to spend the afternoon giving annual screening exams to members of the general population. Here disease prevalence will be low. Would the morning's work influence decisions about patients in the afternoon? It is known from other contexts that recent history can influence current decisions and that target prevalence has an impact on decisions. In this study, decisions were decisions about skin lesions from individuals with varying degrees of expertise, using an online, medical imaging labelling app (DiagnosUs). This allowed examination of the effects of feedback history and prevalence in a single study. Blocks of trials could be of low or high prevalence, with or without feedback. Over 300,000 individual judgements were collected. (taken from Wolfe, J. M. (2022). How one block of trials influences the next: Persistent effects of disease prevalence and feedback on decisions about images of skin lesions in a large online study. . Cognitive Research: Principles and Implications (CRPI), 7, 10. doi: https://doi.org/10.1186/s41235-022-00362-0

Conditions

Decision Making

Keywords

dermatology; signal detection theory; prevalence effects; base rate effects; online study; BESH

Outcome Measures

Primary Outcomes (2)

• Change in D' Between Pairs of Blocks.

  D' (d-prime) is defined as z-transform of the true positive rate - z-transform of false positive rate. True positive is when you say that a real melanoma is a melanoma. False positive is when you say that a nevis is a melanoma. A correction of 0.5 error is added to avoid calculation problems when z=0 or z=1. D' of zero indicates no ability to discriminate. D' \> zero indicates some ability to discriminate. The change of interest is the D' for Block 2 when it follows Block 1 compared to the D' for Block 2 averaged across all conditions.

  Participants could be in the study for as little as two blocks in one day up to 24 blocks collected over 6 days.

• Change in Criterion Between Pairs of Blocks.

  Criterion, c, corresponds to the position of the midpoint between the z-transformed probabilities of hits (correct yes responses) and false alarms (incorrect yes responses). It is calculated as -\[z(p(h))+z(p(FA))\]/2. The criterion, c, z-score quantifies the distance away from being unbiased in units of standard deviations. A Z-score of 0 is said to be unbiased. Negative values for c indicate a more relaxed criterion for saying yes. Positive numbers indicate a more strict criterion for saying yes.

  Participants could be in the study for as little as two blocks in one day up to 24 blocks collected over 6 days.

Study Arms (1)

Feedback X Prevalence Using Dermatology Stimuli

EXPERIMENTAL

In this experiment, observers (Os) completed blocks of 80 trials. On each trial, they saw an image of a spot on the skin. They classified this as a melanoma (cancer) or a nevis (benign). Blocks could be of low prevalence (20% cancer cases, 16 images) or high prevalence (50%, 40 images). Os either did received trial by trial "Feedback" about their performance accuracy, or they did not. Thus, there were four types of block. Low prevalence, No Feedback Low prevalence, Feedback High prevalence, No Feedback High prevalence, Feedback Each of these four types of block was made available to Os on each of 6 days. Os could elect to view each of the four blocks each day. Our particular interest was in the effect of performing one block on performance on an immediately subsequent block.

Behavioral: Feedback; Behavioral: Prevalence

Interventions

Feedback BEHAVIORAL

presence or absence of trial by trial feedback

Feedback X Prevalence Using Dermatology Stimuli

Prevalence BEHAVIORAL

In some blocks, skin cancer "target" images were present on 50% of trials (high prevalence). In other blocks, disease prevalence was 20%.

Feedback X Prevalence Using Dermatology Stimuli

Eligibility Criteria

• Age: 18 Years+
• Sex: all
• Healthy Volunteers: Yes
• Age Groups: Adult (18-64), Older Adult (65+)

You may qualify if:

• All welcome to enroll on line

You may not qualify if:

• Under 18 yrs

Sponsors & Collaborators

• Brigham and Women's Hospital: lead

Study Sites (1)

Visual Attention Lab, Brigham and Women's Hospital

Boston, Massachusetts, 02215, United States

Location

Related Publications (1)

• Wolfe JM. How one block of trials influences the next: persistent effects of disease prevalence and feedback on decisions about images of skin lesions in a large online study. Cogn Res Princ Implic. 2022 Feb 2;7(1):10. doi: 10.1186/s41235-022-00362-0.

  PMID: 35107667 RESULT

MeSH Terms

Interventions

Prevalence

Intervention Hierarchy (Ancestors)

Morbidity; Vital Statistics; Data Collection; Epidemiologic Methods; Investigative Techniques; Demography; Population Characteristics; Epidemiologic Measurements; Public Health; Environment and Public Health

Limitations and Caveats

Convenience sample of 803 participants is unbalanced. Online study lacks control over the display and ambient lighting. Other statistical analyses could be performed

Results Point of Contact

• Title: Jeremy Wolfe, Principle Investigator
• Organization: BrighamHospital

jwolfe@bwh.harvard.edu

617-851-1166

Study Officials

• Jeremy M Wolfe, PhD

  Brigham and Women's Hospital

  PRINCIPAL INVESTIGATOR

Publication Agreements

• PI is Sponsor Employee: Yes

Study Design

• Study Type: interventional
• Phase: not applicable
• Allocation: NA
• Masking: NONE
• Masking Details: Observers were naive about the hypothesis but could have figured out if a specific condition did or did not have feedback, for example. Once the data were collected, investigators could determine what conditions were tested on which observers
• Purpose: BASIC SCIENCE
• Intervention Model: SINGLE GROUP
• Sponsor Type: OTHER
• Responsible Party: PRINCIPAL INVESTIGATOR
• PI Title: Professor

Model Details: We tested \~800 observers on line in a study where they could participate in multiple conditions. We were looking for the effect of condition 1 on condition N+1

Study Record Dates

• First Submitted: February 8, 2022
• First Posted: February 17, 2022
• Study Start: June 22, 2021
• Primary Completion: June 27, 2021
• Study Completion: June 27, 2021
• Last Updated: October 26, 2022
• Results First Posted: October 26, 2022

Record last verified: 2022-10

Data Sharing

• IPD Sharing: Will share
• Shared Documents: STUDY PROTOCOL, SAP, ANALYTIC CODE
• Time Frame: Available currently and indefinitely.
• Access Criteria: OSF is publicly available. Other requests to jwolfe@bwh.harvard.edu

Locations

US (1)