Artificial intelligence (AI), particularly large language models (LLMs), has advanced rapidly in recent years, enabling novel applications across diverse areas of medicine [1, 2, 3]. These models can generate contextually relevant, human-like responses to complex clinical queries, and their accessibility, along with their ability to synthesize medical knowledge, highlights potential roles in medical education, residency training, and clinical decision-making [4, 5].

In obstetrics and gynecology, residency training demands mastery of a broad knowledge base spanning endocrinology, gynecology, obstetrics, and oncology. Traditional learning methods such as textbooks, lectures, and supervised clinical experience, remain essential but are often limited by time constraints and variability in clinical exposure [6]. AI-driven tools may supplement training by providing on-demand access to information, thereby supporting both didactic learning and clinical reasoning [7]. Despite this promise, the reliability and accuracy of LLMs in specialized medical fields remain uncertain. Prior studies evaluating LLM performance in general medicine and other specialties have reported mixed findings compared with physicians [8]. However, evidence in obstetrics and gynecology remains limited [9]. Moreover, potential differences between freely accessible and subscription-based AI models have not been systematically examined.

This study aimed to address these gaps by systematically evaluating and comparing the accuracy of several widely used LLMs with that of obstetrics and gynecology resident physicians across specialty-specific domains [10]. In doing so, we sought to clarify the strengths and limitations of current general-purpose LLMs in supporting medical education and to assess whether these models could serve as reliable adjuncts in residency training and clinical decision-making [11].

25 obstetrics and gynecology residents completed the 40-item multiple-choice examination spanning gynecology, obstetrics, endocrinology, and oncology. Internal consistency was high (Cronbach’s alpha = 0.82). Five LLMs were evaluated on the same item set. Overall accuracy varied across groups (Table 1). Residents achieved a mean score of 20.6 out of 40 (51.5% correct). Among the AI models, ChatGPT (GPT-4o; subscription-based) achieved the highest overall accuracy (36/40; 90.0%), followed by DeepSeek (35/40; 87.5%) and Gemini (35/40; 87.5%). ChatGPT (GPT-4o Mini; free) performed similarly to residents (19/40; 47.5%), whereas Microsoft Copilot (free) had the lowest overall accuracy (15/40; 37.5%). Corresponding 95% confidence intervals (CIs) are shown in Table 1.

Fig. 1 summarizes overall accuracy (percent correct) across residents and model configurations. Overall performance fell into three tiers. ChatGPT (GPT-4o; subscription-based) achieved the highest accuracy numerically (36/40; 90.0%), followed closely by DeepSeek and Gemini (35/40; 87.5% each). Residents and ChatGPT (GPT-4o Mini; free) showed intermediate, similar accuracy, whereas Microsoft Copilot (free) performed worst across the 40-item examination.

Fig. 1.

Overall accuracy across residents and LLM configurations. Bars show the percentage of correctly answered items out of 40. Residents’ accuracy reflects pooled performance across 25 residents (25 $\times$ 40 = 1000 total responses), whereas LLM accuracy reflects the proportion correct across the same 40-item set for each model configuration.

At the item level, the resident-majority was correct on 18/40 items (45.0%). In paired item-level comparisons using the exact McNemar test (Table 2), ChatGPT (GPT-4o; subscription-based) outperformed the resident-majority (36/40 vs. 18/40; b = 21, c = 3; p 0.001). DeepSeek and Gemini also outperformed residents (each 35/40 vs. 18/40; b = 19, c = 2; p 0.001 for both). In contrast, ChatGPT (GPT-4o Mini; free) showed no significant difference (19/40 vs. 18/40; b = 9, c = 8; p = 1.000), and Microsoft Copilot (free) did not differ significantly (15/40 vs. 18/40; b = 6, c = 9; p = 0.607).

Domain-specific analyses are summarized in Table 3. Given the small number of items in some subdomains (particularly endocrinology, n = 3), these domain-specific results are descriptive and provided for reference only; they should not be interpreted as statistically precise estimates. In endocrinology (3 items), residents attained 32.0% accuracy, whereas ChatGPT (GPT-4o; subscription-based), DeepSeek, and Gemini answered all items correctly (100.0% each). Both ChatGPT (GPT-4o Mini; free) and Microsoft Copilot (free) showed accuracies comparable to residents (33.3%). In gynecology (18 items), residents scored 51.1% on average, whereas ChatGPT (GPT-4o; 83.3%), DeepSeek (100.0%), and Gemini (94.4%) achieved substantially higher accuracy. ChatGPT (GPT-4o Mini) performed similarly to residents (55.6%), while Microsoft Copilot remained lower (38.9%). In obstetrics (9 items), residents achieved 64.0% accuracy, lower than ChatGPT (GPT-4o; 100.0%), DeepSeek (77.8%), and Gemini (88.9%). ChatGPT (GPT-4o Mini; 66.7%) and Microsoft Copilot (55.6%) demonstrated intermediate performance. In oncology (10 items), residents scored 47.2%, whereas ChatGPT (GPT-4o), DeepSeek, and Gemini reached 90.0%, 70.0%, and 70.0% accuracy, respectively. ChatGPT (GPT-4o Mini) and Microsoft Copilot performed worst in this domain (both 20.0%). CIs were wide in domains with few items, particularly endocrinology, reflecting limited precision for these estimates.

Qualitative review (data not shown) noted that some incorrect model outputs were internally inconsistent or ambiguous, particularly for items with nuanced clinical framing. Among the reviewed resident errors (sampled), responses more often reflected incomplete or partially correct reasoning rather than consistent selection of a single incorrect option.

This study directly compared several LLMs with obstetrics and gynecology residents using a specialty-specific multiple-choice examination, providing domain-focused insights in this field. The primary finding was that an advanced subscription-based model, ChatGPT (GPT-4o; subscription-based), along with two free models, DeepSeek and Gemini, achieved substantially higher overall accuracy than residents. In paired item-level comparisons using an exact McNemar framework based on resident-majority item correctness, ChatGPT (GPT-4o; subscription-based), DeepSeek, and Gemini also demonstrated significantly higher item-level accuracy than the resident group (all p 0.001). In contrast, ChatGPT (GPT-4o Mini; free) performed similarly to residents and did not differ significantly in paired comparisons, while Microsoft Copilot showed lower overall accuracy but did not differ significantly from residents in the paired item-level analysis. Differences among the top-performing models were small (one item) and should not be interpreted as definitive head-to-head superiority.

Our findings are consistent with reports from other medical specialties in which state-of-the-art LLMs have achieved near-expert or superior performance on formal knowledge assessments [12]. The present study extends the literature by focusing on obstetrics and gynecology, a field that integrates cognitive, procedural, and guideline-driven decision-making and often relies on case-based reasoning. The high accuracy of ChatGPT (GPT-4o; subscription-based), DeepSeek, and Gemini across the 40-item examination, particularly in gynecology and oncology, suggests potential value as educational and decision-support adjuncts in women’s health. At the same time, the similar performance of ChatGPT (GPT-4o Mini; free) to residents, along with the lower overall accuracy observed for Microsoft Copilot, illustrates that performance is not uniform across commonly available LLM interfaces.

Our findings complement prior obstetrics and gynecology-focused LLM studies, including work using reflective writing tasks and vignette-based scenarios. Those studies primarily evaluated performance on narrative responses or simulated clinical cases under specific constraints, whereas we employed a standardized multiple-choice examination mapped to the national residency curriculum. By directly comparing several free and subscription-based models with residents on the same specialty-specific examination, our study provides a multi-model, exam-based perspective that aligns with formal assessment settings while remaining distinct from vignette- and portfolio-based evaluations. Our multiple-choice, curriculum-mapped examination design complements prior vignette-based studies by assessing core knowledge performance under standardized inputs.

From a clinical and educational perspective, these results suggest several possible applications. High-performing models may serve as adjunctive study aids by providing immediate feedback on practice questions, generating explanations, and supporting guideline review. However, the observed variability among models, including differences between subscription-based and free systems, underscores that recommendations for LLM use should be model-specific and supported by ongoing validation [13, 14, 15, 16]. Careful model selection will therefore be essential if training programs consider integrating LLMs into curricula or clinical decision-support workflows. Our findings should be interpreted as supporting adjunct use rather than replacement. LLMs may complement trainees by providing explanations and highlighting guideline-based concepts, but clinically actionable outputs must be verified against authoritative references and interpreted under faculty or clinician supervision. In clinical settings, LLM use should adhere to local governance policies, with verification of high-stakes recommendations documented and accountability clearly retained by the supervising clinician.

When an LLM output references a study, guideline, or specific evidence, learners should treat the response as a preliminary aid and verify key claims in the primary source. If the model’s recommendation conflicts with established references or supervisor instruction, the appropriate course of action is to prioritize authoritative guidance, cross-check the cited source, and consult faculty or senior clinicians. This verification step is particularly important for clinically actionable content, as unverified outputs should not guide patient care. In practice, training programs should implement clear guardrails for LLM use, including reference standards and escalation pathways when outputs are uncertain or conflict with established guidance. Ongoing validation should be coordinated at the institutional level by residency program leadership and clinical governance structures, with guidance informed by specialty societies as models evolve.

The use of an exact McNemar test reflects the paired structure of the data at the item level. Each examination item was answered by both the resident cohort and by each model, enabling item-level comparisons that partially control for item difficulty. Nevertheless, this approach summarizes resident performance using a group-level majority-vote indicator and does not fully account for within-resident and within-item correlation structures. More complex approaches, such as mixed-effects models, could be considered in future studies, particularly in larger, multi-institutional cohorts.

A descriptive review of incorrect model outputs suggested that some responses were internally inconsistent or ambiguous, particularly for items with nuanced clinical framing. Because this review was not a formal safety assessment and did not quantify “clinically unsafe” outputs using predefined criteria and independent raters, conclusions regarding comparative safety across models cannot be drawn from this study. Future studies should classify incorrect outputs using predefined, safety-focused error categories and independent clinician raters. More broadly, concerns have been raised that LLMs may provide incorrect answers with high confidence, reinforcing the need for critical appraisal and oversight [14, 17]. These observations support the broader view that LLM outputs should be interpreted critically and used with appropriate clinician oversight, particularly when applied outside educational contexts [17].

This study has several strengths, including a specialty-specific examination developed by domain experts, high internal consistency, direct comparisons across multiple LLMs and residents, and transparent item-level paired analyses.

This study provides empirical data on the performance of general-purpose LLMs compared with obstetrics and gynecology residents on standardized clinical questions. By characterizing accuracy and the nature of incorrect outputs, it informs educators and health system leaders about the realistic capabilities and limitations of current models. These findings can guide the safe integration of AI tools into postgraduate training and clinical decision-support workflows, while emphasizing the continued need for clinician oversight.

Advanced LLM configurations, including ChatGPT (GPT-4o; subscription-based), DeepSeek, and Gemini, achieved higher accuracy than residents on this specialty-specific multiple-choice examination and outperformed the resident-majority indicator in paired item-level comparisons. These findings support the potential role of LLMs as supplementary educational tools and adjuncts for structured information support in women’s health. Importantly, performance was not uniform across models. ChatGPT (GPT-4o Mini; free) did not differ significantly from residents, and Microsoft Copilot showed lower overall accuracy but did not differ significantly from residents in paired item-level comparisons. Taken together, these results support the use of LLMs as adjuncts rather than replacements for human expertise and emphasize the importance of careful model selection, ongoing validation, and responsible integration with clinician oversight [17].

The datasets generated and analyzed during this study are available from the corresponding author upon reasonable request. Due to institutional privacy policies, de-identified datasets are not publicly available.

ÖFÖ and CD conceptualized the study and designed the question set. EŞ, SD, and NUD contributed to data collection and analysis. MÖ and İM made substantial contributions to the interpretation of data, academic supervision, and critical revision of the manuscript for important intellectual content. All authors contributed to editorial changes in the manuscript. All authors read and approved the final manuscript. All authors have participated sufficiently in the work and agreed to be accountable for all aspects of the work.

The study was conducted in accordance with the Declaration of Helsinki and approved by the Akdeniz University Faculty of Medicine Clinical Research Ethics Committee (approval number: TBAEK-863). Written informed consent was obtained from all participants prior to inclusion in the study.

The authors thank all resident physicians from Akdeniz University Faculty of Medicine, Department of Obstetrics and Gynecology, for their participation in this study. We also appreciate the contributions of the department’s academic staff in reviewing the question bank.

This research received no external funding.

The authors declare that they have no conflicts of interest in the research. All figures and tables are original and were created by the authors for this manuscript; no permissions were required.

During the preparation of this work the authors used ChatGPT for language editing (spelling/grammar) and clarity improvement. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.