Specialist Review of Harm Ratings for Clinical Responses Acronym: NOHARM

NCT07823062 | Not Yet Recruiting

• 1 other identifier: 85610
• Study Type: observational
• Target: 80
• Locations: 0 countries
• Sites: N/A

Brief Summary

The purpose of this study is to determine how closely potential-harm ratings assigned using the NOHARM rubric agree with independent ratings from specialty-matched physicians. The study uses reconstructed clinical consultation cases and previously generated responses written by artificial intelligence systems or physicians. The materials contain no patient identifiers or protected health information. Board-certified attending physicians from 10 medical specialties will review assigned case-response pairs through a secure website. For each response, they will indicate whether it contains an error, describe the error when applicable, and rate the greatest potential patient harm as none, mild, moderate, or severe. Reviewers will not be shown the response source, existing rubric rating, or other reviewers' ratings. Two specialists will initially review each response; a third may review it if the first two ratings differ. The study includes a development stage, possible rubric revision, and a separate validation stage.

Conditions

Patient Safety; Medical Errors

Keywords

Artificial Intelligence; Clinical Response Evaluation; Harm Assessment; Clinical Benchmark; Rubric Validation

Outcome Measures

Primary Outcomes (2)

• Inverse-Probability-Weighted Confirmation of Rubric-Severe Classifications

  The inverse-probability-weighted proportion of responses classified as severe by the NOHARM rubric that are also classified as severe by specialist consensus: P(specialist severe \| rubric severe). This measure will be calculated separately for the development and validation stages. Values range from 0 to 1; higher values indicate greater confirmation. The prespecified threshold is 0.70.

  At completion of the development and validation stages, up to 2 months

• Inverse-Probability-Weighted Capture of Specialist-Severe Classifications

  The inverse-probability-weighted proportion of responses classified as severe by specialist consensus that are also classified as severe by the NOHARM rubric: P(rubric severe \| specialist severe). This measure will be calculated separately for the development and validation stages. Values range from 0 to 1; higher values indicate greater capture. The prespecified threshold is 0.70. Both co-primary measures must meet this threshold.

  At completion of the development and validation stages, up to 2 months

Study Arms (1)

Specialist Physician Reviewers

Licensed or registered attending- or consultant-level physicians practicing in one of the represented medical specialties in the United States, Canada, United Kingdom, or Australia. Participants will independently review and rate reconstructed clinical case-response pairs during the development stage, validation stage, or both. Physicians involved in rubric revision will not provide validation-stage ratings.

Eligibility Criteria

• Age: 18 Years+
• Sex: all
• Healthy Volunteers: Yes
• Age Groups: Adult (18-64), Older Adult (65+)
• Sampling Method: Non-Probability Sample

Study Population

Licensed or registered attending- or consultant-level physicians practicing in one of the medical specialties represented in the NOHARM benchmark in the United States, Canada, United Kingdom, or Australia. Participants will be recruited through professional networks, institutional communications, and collaborating clinical communities and will complete specialty-matched rating activities remotely.

You may qualify if:

• Licensed or registered physician in good standing
• Attending- or consultant-level physician actively practicing in allergy and immunology, cardiology, dermatology, endocrinology, gastroenterology, hematology, infectious disease, nephrology, neurology, or pulmonology
• Practicing in the United States, Canada, United Kingdom, or Australia
• Assigned clinical materials match the physician's specialty
• Able to read and respond in English
• Access to a computer with a stable internet connection
• Willing and able to complete at least one remote specialist-rating engagement of approximately three hours

You may not qualify if:

• Does not meet the professional, career-level, specialty, geographic, or credential-verification requirements above
• Prior involvement with assigned cases, responses, or rubric information that could compromise independent rating
• Unable to complete the remote rating activities in English
• Unable or unwilling to follow the independent-rating procedures
• For validation-stage assignments, participation in the rubric-revision process

Sponsors & Collaborators

• Stanford University: lead

Central Study Contacts

Jonathan H Chen, MD, PHD

CONTACT

650-725-3655; jonc101@stanford.edu

Study Design

• Study Type: observational
• Observational Model: COHORT
• Time Perspective: PROSPECTIVE
• Sponsor Type: OTHER
• Responsible Party: PRINCIPAL INVESTIGATOR
• PI Title: Associate Professor of Medicine

Study Record Dates

• First Submitted: September 10, 2026
• First Posted: September 16, 2026
• Study Start: September 21, 2026
• Primary Completion (Estimated): October 25, 2026
• Study Completion (Estimated): October 30, 2026
• Last Updated: September 16, 2026

Record last verified: 2026-09

Data Sharing

• IPD Sharing: Will share
• Shared Documents: STUDY PROTOCOL, SAP, ANALYTIC CODE