AI Clin Med

AI in Clinical Medicine, ISSN 2819-7437 online, Open Access

Article copyright, the authors; Journal compilation copyright, AI Clin Med and Elmer Press Inc

Journal website https://aicm.elmerpub.com

Original Article

Volume 1, 2025, e12

Evaluating ChatGPT-5o as a Clinical Decision-Support Tool in Inflammatory Bowel Disease: A Pilot Study of Guideline Adherence and Clinical Agreement

Figure 2. Cohen’s Kappa agreement levels across different categories. The color scale represents the level of agreement: dark blue (1.0): perfect agreement (e.g., anti-IL-23, antibiotics, diagnostic workup, symptom management, surgical consultation, continuous monitoring). Medium blue (≈ 0.6 - 0.8): substantial agreement. Light blue (≈ 0.3 - 0.5): moderate to fair agreement, indicating some variability in alignment.

**Table 1.** Interpretation of Agreement Levels
Agreement level	Interpretation
Perfect (Kappa = 1.000)	Complete alignment, no variation between recommendations.
Substantial (Kappa ≈ 0.6 - 0.8)	High agreement with some minor differences, reflecting consistent recommendations.
Moderate to fair (Kappa ≈ 0.3 - 0.5)	Notable variability, indicating different interpretations or thresholds in decision-making.
Non-significant	Agreement level is not statistically meaningful; recommendations may vary substantially.

Table 1. Interpretation of Agreement Levels

Agreement level

Interpretation

Perfect (Kappa = 1.000)

Complete alignment, no variation between recommendations.

Substantial (Kappa ≈ 0.6 - 0.8)

High agreement with some minor differences, reflecting consistent recommendations.

Moderate to fair (Kappa ≈ 0.3 - 0.5)

Notable variability, indicating different interpretations or thresholds in decision-making.

Non-significant

Agreement level is not statistically meaningful; recommendations may vary substantially.

**Table 2.** Summary of Cohen’s Kappa Agreement for Each Category
Category	Human vs. guidelines Kappa	Human vs. ChatGPT Kappa	ChatGPT vs. guidelines Kappa	Agreement level	Significance
5-ASA: 5-aminosalicylic acid; IL: interleukin; TNF: tumor necrosis factor.
5-ASA	0.689	0.689	0.604	Substantial	Yes
Steroids	0.406	0.578	0.771	Moderate to substantial	Yes
Anti-TNF	0.289	0.345	0.685	Fair to substantial	Yes
Anti-integrins	0.486	0.336	0.771	Fair to substantial	Yes
Anti-IL-23	1.000	1.000	1.000	Perfect	Yes
Thiopurines	0.441	0.313	0.771	Fair to substantial	No
Antibiotics	0.642	1.000	1.000	Substantial to perfect	Yes
Diagnostic workup	1.000	1.000	1.000	Perfect	Yes
Symptom management	-	1.000	1.000	Perfect	Yes
Surgical consult	-	1.000	1.000	Perfect	Yes
Continuous monitoring	-	1.000	1.000	Perfect	Yes

Table 2. Summary of Cohen’s Kappa Agreement for Each Category