Assessing ChatGPT vs. evidence-based online responses for polycystic ovary syndrome self-management and education: an international cross-sectional blinded survey of healthcare professionals

Sandro Graca; Alexander Dallaway; Folashade Alloh; Harpal S.  Randeva; Chris Kite; Ioannis Kyrou

doi:10.3389/fdgth.2025.1700018

Assessing ChatGPT vs. evidence-based online responses for polycystic ovary syndrome self-management and education: an international cross-sectional blinded survey of healthcare professionals

Sandro Graca
, Alexander Dallaway
, Folashade Alloh
, Harpal S. Randeva
, Chris Kite
, Ioannis Kyrou

University of Wolverhampton
Warwickshire Institute for the Study of Diabetes, Endocrinology and Metabolism
University Hospitals Coventry and Warwickshire NHS Trust
University of Warwick
University of Chester
Warwick Medical School
Aston Medical School
University of Derby

Research output: Contribution to journal › Article › peer-review

2 Downloads (Pure)

Abstract

Artificial intelligence (AI)-powered large language models, such as ChatGPT, are increasingly used by the public for health information. The reliability of such novel AI-tools in providing credible polycystic ovary syndrome (PCOS) information/advice requires investigation. Healthcare professionals involved in PCOS care ( n = 43 from 14 countries) used a 5-point Likert scale to evaluate ChatGPT-generated responses to frequently asked questions about PCOS against the corresponding patient-orientated, evidence-based recommendations/responses available online. ChatGPT responses were rated significantly higher than the evidence-based responses for 11 of the 12 study questions, with moderate to large effect sizes ( r r b = −0.46 to −1.00; all p -values <0.05), with ChatGPT answers being rated on average 0.824 units higher. Scoring agreement varied (poor to fair), with seven questions showing statistically fair agreement (κ = 0.24–0.37, p < 0.05). Readability analyses found no statistically significant differences between ChatGPT and evidence-based responses. However, using ChatGPT for simplifying the responses resulted in significant improvement. ChatGPT holds potential as a complementary patient self-education tool in PCOS, capable of interactive engagement and simplifying medical language. Further research is needed to identity optimal integration of AI tools and validate their clinical applicability for PCOS self-education/management.

Original language	English
Article number	1700018
Number of pages	12
Journal	Frontiers in Digital Health
Volume	7
Early online date	31 Mar 2026
DOIs	https://doi.org/10.3389/fdgth.2025.1700018
Publication status	E-pub ahead of print - 31 Mar 2026

Bibliographical note

© 2026 Graca, Dallaway, Alloh, Randeva, Kite and Kyrou. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

Funding

The author(s) declare that financial support was received for the research and/or publication of this article. This study was completed as part of a funded PhD project (S.G.) through the University of Wolverhampton. The PhD is jointly funded by the University of Wolverhampton and the University Hospitals Coventry and Warwickshire NHS Trust. The funders had no participation in the design or writing of this study.

Funders
University of Wolverhampton
University Hospitals Coventry and Warwickshire NHS Trust

Keywords

AI
ChatGPT
PCOS
artificial intelligence
large language models
polycystic ovary syndrome

Access to Document

10.3389/fdgth.2025.1700018Licence: CC BY

Kyrou2026VoRFinal published version, 1.06 MBLicence: CC BY

Cite this

Graca, S., Dallaway, A., Alloh, F., Randeva, H. S., Kite, C., & Kyrou, I. (2026). Assessing ChatGPT vs. evidence-based online responses for polycystic ovary syndrome self-management and education: an international cross-sectional blinded survey of healthcare professionals. Frontiers in Digital Health, 7, Article 1700018. Advance online publication. https://doi.org/10.3389/fdgth.2025.1700018

Assessing ChatGPT vs. evidence-based online responses for polycystic ovary syndrome self-management and education: an international cross-sectional blinded survey of healthcare professionals. / Graca, Sandro; Dallaway, Alexander; Alloh, Folashade et al.
In: Frontiers in Digital Health, Vol. 7, 1700018, 31.03.2026.

Research output: Contribution to journal › Article › peer-review

@article{2ae8dc57bf6543e08a67dde168280800,

title = "Assessing ChatGPT vs. evidence-based online responses for polycystic ovary syndrome self-management and education: an international cross-sectional blinded survey of healthcare professionals",

abstract = "Artificial intelligence (AI)-powered large language models, such as ChatGPT, are increasingly used by the public for health information. The reliability of such novel AI-tools in providing credible polycystic ovary syndrome (PCOS) information/advice requires investigation. Healthcare professionals involved in PCOS care ( n = 43 from 14 countries) used a 5-point Likert scale to evaluate ChatGPT-generated responses to frequently asked questions about PCOS against the corresponding patient-orientated, evidence-based recommendations/responses available online. ChatGPT responses were rated significantly higher than the evidence-based responses for 11 of the 12 study questions, with moderate to large effect sizes ( r r b = −0.46 to −1.00; all p -values <0.05), with ChatGPT answers being rated on average 0.824 units higher. Scoring agreement varied (poor to fair), with seven questions showing statistically fair agreement (κ = 0.24–0.37, p < 0.05). Readability analyses found no statistically significant differences between ChatGPT and evidence-based responses. However, using ChatGPT for simplifying the responses resulted in significant improvement. ChatGPT holds potential as a complementary patient self-education tool in PCOS, capable of interactive engagement and simplifying medical language. Further research is needed to identity optimal integration of AI tools and validate their clinical applicability for PCOS self-education/management.",

keywords = "AI, ChatGPT, PCOS, artificial intelligence, large language models, polycystic ovary syndrome",

author = "Sandro Graca and Alexander Dallaway and Folashade Alloh and Randeva, \{Harpal S.\} and Chris Kite and Ioannis Kyrou",

note = "{\textcopyright} 2026 Graca, Dallaway, Alloh, Randeva, Kite and Kyrou. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.",

year = "2026",

month = mar,

day = "31",

doi = "10.3389/fdgth.2025.1700018",

language = "English",

volume = "7",

journal = "Frontiers in Digital Health",

issn = "2673-253X",

publisher = "Frontiers Media S.A.",

}

TY - JOUR

T1 - Assessing ChatGPT vs. evidence-based online responses for polycystic ovary syndrome self-management and education

T2 - an international cross-sectional blinded survey of healthcare professionals

AU - Graca, Sandro

AU - Dallaway, Alexander

AU - Alloh, Folashade

AU - Randeva, Harpal S.

AU - Kite, Chris

AU - Kyrou, Ioannis

N1 - © 2026 Graca, Dallaway, Alloh, Randeva, Kite and Kyrou. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

PY - 2026/3/31

Y1 - 2026/3/31

N2 - Artificial intelligence (AI)-powered large language models, such as ChatGPT, are increasingly used by the public for health information. The reliability of such novel AI-tools in providing credible polycystic ovary syndrome (PCOS) information/advice requires investigation. Healthcare professionals involved in PCOS care ( n = 43 from 14 countries) used a 5-point Likert scale to evaluate ChatGPT-generated responses to frequently asked questions about PCOS against the corresponding patient-orientated, evidence-based recommendations/responses available online. ChatGPT responses were rated significantly higher than the evidence-based responses for 11 of the 12 study questions, with moderate to large effect sizes ( r r b = −0.46 to −1.00; all p -values <0.05), with ChatGPT answers being rated on average 0.824 units higher. Scoring agreement varied (poor to fair), with seven questions showing statistically fair agreement (κ = 0.24–0.37, p < 0.05). Readability analyses found no statistically significant differences between ChatGPT and evidence-based responses. However, using ChatGPT for simplifying the responses resulted in significant improvement. ChatGPT holds potential as a complementary patient self-education tool in PCOS, capable of interactive engagement and simplifying medical language. Further research is needed to identity optimal integration of AI tools and validate their clinical applicability for PCOS self-education/management.

AB - Artificial intelligence (AI)-powered large language models, such as ChatGPT, are increasingly used by the public for health information. The reliability of such novel AI-tools in providing credible polycystic ovary syndrome (PCOS) information/advice requires investigation. Healthcare professionals involved in PCOS care ( n = 43 from 14 countries) used a 5-point Likert scale to evaluate ChatGPT-generated responses to frequently asked questions about PCOS against the corresponding patient-orientated, evidence-based recommendations/responses available online. ChatGPT responses were rated significantly higher than the evidence-based responses for 11 of the 12 study questions, with moderate to large effect sizes ( r r b = −0.46 to −1.00; all p -values <0.05), with ChatGPT answers being rated on average 0.824 units higher. Scoring agreement varied (poor to fair), with seven questions showing statistically fair agreement (κ = 0.24–0.37, p < 0.05). Readability analyses found no statistically significant differences between ChatGPT and evidence-based responses. However, using ChatGPT for simplifying the responses resulted in significant improvement. ChatGPT holds potential as a complementary patient self-education tool in PCOS, capable of interactive engagement and simplifying medical language. Further research is needed to identity optimal integration of AI tools and validate their clinical applicability for PCOS self-education/management.

KW - AI

KW - ChatGPT

KW - PCOS

KW - artificial intelligence

KW - large language models

KW - polycystic ovary syndrome

UR - https://www.scopus.com/pages/publications/105041183135

U2 - 10.3389/fdgth.2025.1700018

DO - 10.3389/fdgth.2025.1700018

M3 - Article

SN - 2673-253X

VL - 7

JO - Frontiers in Digital Health

JF - Frontiers in Digital Health

M1 - 1700018

ER -

Assessing ChatGPT vs. evidence-based online responses for polycystic ovary syndrome self-management and education: an international cross-sectional blinded survey of healthcare professionals

Abstract

Bibliographical note

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this