Guideline Adherence - Help

Overview

The Guideline Adherence module provides real-time monitoring of evidence-based clinical guideline bundles. It generates GUIDELINE_DEVIATION alerts when bundle elements are not completed within their time windows, and tracks aggregate compliance metrics for quality improvement and Joint Commission (JC) reporting.

Why Separate from Antibiotic Indications?

Aspect	Antibiotic Indications	Guideline Adherence
Question	"Is this order justified?"	"Did we follow the full pathway?"
Timing	Real-time (per order)	Real-time monitoring with time windows
Scope	Antibiotic orders only	Full care bundle (cultures, labs, imaging, consults, antibiotics)
Alert Trigger	No documented indication	Bundle element not completed within time window
Output	ASP alert for review	ASP alert + compliance metrics dashboard
Audience	Pharmacist, ordering provider	ASP team, QI, JC surveyors

How It Works - Three-Mode Monitoring Pipeline

The guideline monitoring system operates in three distinct modes that work together:

Workflow Steps

Available Guidelines

Guideline	Elements	Key Metrics
Pediatric Sepsis	6	Blood culture, lactate, ABX ≤1h, fluids, reassessment
Pediatric CAP	6	CXR, SpO2, empiric choice, duration ≤7d
Febrile Neutropenia	6	Cultures, ABX ≤1h, risk stratification
Surgical Prophylaxis	5	Agent selection, timing ≤60min, duration ≤24h
Pediatric UTI	7	UA, culture, empiric choice, imaging
SSTI/Cellulitis	6	Margins marked, MRSA coverage, I&D if needed
Febrile Infant (AAP 2021)	14	UA, blood cx, inflammatory markers, LP (age-stratified), HSV risk, safe discharge
Neonatal HSV (CCHMC 2024)	11	CSF/blood PCR, surface cultures, LFTs, acyclovir ≤1h, ID consult, treatment duration
C. diff Testing (CCHMC 2024)	8	Diagnostic stewardship: age, symptoms, laxatives, risk factors, appropriateness score

Sepsis Bundle Details (CMS SEP-1)

The Pediatric Sepsis Bundle is based on Surviving Sepsis Campaign guidelines:

Element	Time Window	Data Source	Required
Blood culture obtained	1 hour	Lab orders (LOINC 600-7)	Yes
Lactate measured	3 hours	Lab results (LOINC 2524-7)	Yes
Antibiotics within 1h	1 hour	Medication Administration	Yes
Fluid bolus (if shock)	1 hour	Medication Administration	Conditional
Repeat lactate if >2	6 hours	Lab results	Conditional
48h reassessment	72 hours	Clinical notes	Yes

Febrile Infant Bundle Details (AAP 2021)

The Febrile Infant Bundle implements the AAP Clinical Practice Guideline for well-appearing febrile infants 8-60 days old:

Age Group	LP Required	Antibiotics	Disposition
8-21 days	Yes (all)	Yes - parenteral	Admit all
22-28 days, IMs abnormal	Yes	Yes - parenteral	Admit
22-28 days, IMs normal	Consider	Consider	Home with close f/u acceptable
29-60 days, IMs abnormal	Consider	Yes	Varies
29-60 days, IMs normal	Not required	If UTI only	Home observation acceptable

Inflammatory Marker Thresholds (Abnormal):

• Procalcitonin > 0.5 ng/mL
• ANC > 4,000/μL
• CRP > 2.0 mg/dL

HSV Risk Integration (CCHMC Enhancement): For infants 0-28 days, HSV risk factors are automatically detected:

• Maternal HSV history
• Scalp electrode use
• Vesicular rash
• Seizures
• CSF pleocytosis
• Elevated LFTs (>3x ULN)

If risk factors present → Acyclovir administration is tracked as a required element.

Safe Discharge Checklist (CCHMC Enhancement): For infants being discharged home, tracks 5 safety elements:

1. Follow-up within 24h arranged
2. Working phone number documented
3. Reliable transportation confirmed
4. Parent education completed
5. Return precautions verbalized

Neonatal HSV Bundle Details (CCHMC 2024)

Comprehensive monitoring for suspected HSV in neonates ≤21 days. Triggered by HSV diagnosis, acyclovir order, or HSV PCR order.

Element	Time Window	Required	Data Source
CSF HSV PCR	4 hours	Yes	Lab orders (LOINC 16955-7)
Surface cultures (SEM)	4 hours	Yes	Lab orders
Blood HSV PCR	4 hours	Yes	Lab orders (LOINC 49986-3)
LFTs obtained	4 hours	Yes	Lab results (AST, ALT)
Acyclovir started	1 hour	Yes	Medication Administration
Acyclovir 60mg/kg/day Q8H	24 hours	Yes	Medication orders
ID consult	24 hours	Yes	Consult orders
Ophthalmology (if ocular)	48 hours	Conditional	Consult orders
Neuroimaging (if CNS)	48 hours	Conditional	Imaging orders
Treatment duration met	End of therapy	Yes	Medication orders
Suppressive therapy follow-up	Discharge	Yes	Clinical notes

Classification-Based Treatment Duration:

• SEM (Skin, Eye, Mouth): 14 days
• CNS disease: 21 days
• Disseminated: 21 days

C. diff Testing Appropriateness Bundle (CCHMC 2024)

Diagnostic stewardship bundle to ensure C. diff testing criteria are met before ordering. Triggered when a C. diff test is ordered.

Appropriateness Criteria

Criterion	Requirement
Age	≥3 years (or exception documented)
Symptoms	≥3 liquid stools in 24 hours
No laxatives	Not given within 48 hours
No enteral contrast	Not given within 48 hours
No tube feed changes	Not changed within 48 hours
No active GI bleed	Documented absence
Risk factor present	Antibiotics, hospitalization, PPI, gastrostomy, or chronic disease
Symptom duration	≥48 hours if low-risk patient

Appropriateness Classification

• Appropriate: All criteria met
• Potentially Inappropriate: 1-2 concerns
• Inappropriate: 3+ concerns or major exclusion

NLP Clinical Note Analysis

The system uses LLM-based Natural Language Processing to extract clinical information from unstructured notes that cannot be reliably captured with simple keyword matching.

Clinical Impression Extraction (Febrile Infant)

Critical for risk stratification in the Febrile Infant guideline. The LLM analyzes clinical notes to determine:

Assessment	Indicators	Clinical Action
Well-Appearing	Alert, playful, active, feeding well, good eye contact, pink, well-perfused	Follow standard age-stratified pathway
Ill-Appearing	Lethargic, irritable, poor feeding, mottled, pale, decreased activity, fussy	Higher risk stratification, consider HSV workup
Toxic-Appearing	Obtunded, unresponsive, septic appearance, shock, poor perfusion, inconsolable	Full sepsis workup, aggressive resuscitation

GI Symptom Extraction (C. diff Testing)

Extracts precise symptom information for C. diff testing appropriateness:

• Stool count: Exact number of stools in past 24 hours
• Stool consistency: Liquid, watery, loose, soft, or formed
• Symptom duration: When diarrhea started (converted to hours)
• Associated symptoms: Abdominal pain, cramping, fever, bloody stool

NLP Configuration

The NLP module uses a local LLM via Ollama for HIPAA-compliant processing:

# Check LLM availability
python -c "from guideline_src.nlp.clinical_impression import check_llm_availability; print(check_llm_availability())"

# Test clinical impression extraction
python -m guideline_src.nlp.clinical_impression

# Test GI symptom extraction
python -m guideline_src.nlp.gi_symptoms

Fallback: If LLM is unavailable, the system falls back to keyword-based matching with reduced accuracy. NLP extraction adds HIGH/MEDIUM confidence levels, while keyword matching is tagged with LOW/MEDIUM confidence.

Running the Monitor

Mode 1: Trigger Monitoring

Polls FHIR for new patients matching bundle trigger criteria and creates episodes.

cd /home/david/projects/aegis/guideline-adherence

# Poll once for new triggers
python -m guideline_src.runner --trigger --once

# Poll continuously (every 60 seconds)
python -m guideline_src.runner --trigger --daemon --interval 60

# Show monitoring status
python -m guideline_src.runner --trigger --status

# Use real FHIR connection
python -m guideline_src.runner --trigger --daemon --use-fhir

Mode 2: Episode Monitoring

Checks active episodes for compliance and creates alerts for overdue elements.

# Check all active episodes once
python -m guideline_src.runner --episodes --once

# Run as daemon (every 5 minutes)
python -m guideline_src.runner --episodes --daemon --interval 5

# Show episode status (active episodes, pending elements, alerts)
python -m guideline_src.runner --episodes --status

# Verbose output (shows all element checks)
python -m guideline_src.runner --episodes --once --verbose

Mode 3: Adherence Checking

Direct FHIR queries to verify element completion and calculate adherence.

# Check all active episodes once
python -m guideline_src.runner --once

# Run for specific bundle only
python -m guideline_src.runner --once --bundle sepsis_peds_2024

# Dry run (no alerts created)
python -m guideline_src.runner --once --dry-run --verbose

# Run as daemon (every 15 minutes)
python -m guideline_src.runner --daemon --interval 15

Utility Commands

# List available bundles
python -m guideline_src.runner --list-bundles

# Create demo patients for testing
python demo_patients.py

Cron Setup (Production)

# Trigger monitoring - poll for new patients every minute
* * * * * cd /home/david/projects/aegis/guideline-adherence && \
    python -m guideline_src.runner --trigger --once --use-fhir \
    >> /var/log/aegis/trigger-monitor.log 2>&1

# Episode monitoring - check for overdue elements every 5 minutes
*/5 * * * * cd /home/david/projects/aegis/guideline-adherence && \
    python -m guideline_src.runner --episodes --once \
    >> /var/log/aegis/episode-monitor.log 2>&1

# Adherence checking - full compliance check every 15 minutes
*/15 * * * * cd /home/david/projects/aegis/guideline-adherence && \
    python -m guideline_src.runner --once \
    >> /var/log/aegis/guideline-adherence.log 2>&1

Alert Content

GUIDELINE_DEVIATION alerts appear in ASP Alerts and include:

• Bundle Info: Which guideline bundle triggered the alert
• Element Details: Specific element that was not met
• Time Window: Required completion timeframe and when it expired
• Recommendation: Suggested action for ASP team
• Overall Adherence: Current compliance percentage for this episode

LLM Review Workflow

The system uses an HAI-style automated review workflow with LLM-based analysis and human oversight.

Automatic Analysis

When a guideline episode is created, the system automatically:

1. Retrieves clinical notes from FHIR (DocumentReference resources)
2. Runs LLM analysis using tiered extraction (fast 7B triage → optional 70B full analysis)
3. Determines overall guideline adherence (appropriate, deviation, or pending)
4. Saves assessment with confidence level and supporting evidence

No manual "Run LLM" button required - analysis happens on episode creation and can be re-run periodically.

Episode Review Interface

The episode detail page provides an HAI-style two-column layout:

Left Column (Information)	Right Column (Actions)
Patient information (MRN, age, location)	Reviewer name input
Bundle information (trigger, time)	Decision buttons (Appropriate / Deviation / Needs Info)
LLM assessment (determination, confidence, reasoning)	Override reason dropdown (when applicable)
Bundle elements (met/not_met/pending status)	Deviation type dropdown (for deviations)
Review history	Notes and submit button

Override Detection

The system automatically detects when human judgment differs from automated assessment:

• Direct Override: LLM says "appropriate" but human says "deviation" (or vice versa)
• System Override: LLM says "pending" but there are not_met elements, and human says "appropriate"

When an override is detected, the system shows an Override Reason dropdown to capture why the human disagrees. This data is logged for training improvement.

Override Reason Categories

Category	Description
Extraction Error	LLM misread clinical documentation
Element Detection Error	LLM missed or incorrectly detected an element
Timing Error	LLM got timing/sequence wrong
Clinical Judgment	Clinical context not captured in notes
Documentation Gap	Key documentation not available
Rule Interpretation	Different interpretation of guideline
Patient Specific	Patient-specific factors justify deviation

Training Data Collection

The system captures training data for future LLM fine-tuning, following the same pattern as HAI Detection.

What Gets Logged

• Clinical Appearance Extractions: Every LLM extraction is logged to clinical_appearance_{YYYY_MM}.jsonl
• Guideline Reviews: Human review decisions are logged to guideline_reviews_{YYYY_MM}.jsonl
• Overrides: Discrepancies between LLM and human are flagged for training priority

Training Stats Dashboard

Access via: /guideline-adherence/training/stats

The training stats page shows:

• Total Extractions: Number of LLM extractions performed
• Human Reviewed: Count and percentage of extractions reviewed by humans
• Correction Rate: Percentage of reviewed cases where human corrected the LLM
• Triage Performance: Fast path rate (7B model sufficient) vs escalated (needed 70B)
• Response Times: Average triage and full extraction latency
• Appearance Distribution: Well/Ill/Toxic breakdown
• Override Reasons: Distribution of why humans disagreed with LLM
• Training Readiness: Whether enough data exists for fine-tuning (target: 100+ reviewed)

Export for Fine-Tuning

cd /home/david/projects/aegis/guideline-adherence

# Export clinical appearance training data
python -c "
from guideline_src.nlp.training_collector import get_training_collector
collector = get_training_collector()
count = collector.export_training_data('training_export.jsonl', reviewed_only=True)
print(f'Exported {count} training examples')
"

# Export guideline review data
python -c "
from guideline_src.nlp.training_collector import get_guideline_review_collector
collector = get_guideline_review_collector()
count = collector.export_training_data('guideline_reviews_export.jsonl')
print(f'Exported {count} guideline reviews')
"

Dashboard Pages

Active Episodes

URL: /guideline-adherence/active

Shows episodes currently being monitored (not yet reviewed). Features:

• Filter by bundle type
• Patient info, bundle, trigger time
• Adherence percentage (color-coded: green ≥90%, yellow ≥70%, red <70%)
• LLM Analysis status (determination + confidence, or "Not Analyzed")
• Review and Details action buttons

Note: Reviewed episodes are automatically removed from this list.

History

URL: /guideline-adherence/history

Shows all episodes (both active and completed). Features:

• Filter by bundle type and status (all/active/completed)
• Same columns as Active Episodes plus review status
• Useful for auditing past cases

Compliance Metrics

URL: /guideline-adherence/metrics

• Filter by bundle and time period (7/30/90/365 days)
• Episode counts (total, active, completed)
• Element-level compliance rates grouped by bundle
• Visual compliance bars (color-coded by rate)

Episode Detail / Review

URL: /guideline-adherence/episode/{id}/detail

HAI-style review interface with:

• Full patient and bundle information
• LLM assessment with reasoning and evidence
• Element-by-element status with values and notes
• Review decision workflow with override detection
• Success state after submission

Training Stats

URL: /guideline-adherence/training/stats

Analytics for LLM training data collection (see Training Data Collection section above).

Running the Assessment Monitor

The assessment monitor runs LLM analysis on episodes. Use the run_monitor.py script:

cd /home/david/projects/aegis/guideline-adherence

# Run assessment on all unprocessed episodes (one-time)
python run_monitor.py --assess

# Run full monitor (trigger detection + assessment)
python run_monitor.py --once

# Run as daemon with periodic reassessment
python run_monitor.py --interval 60

# Options:
#   --assess   Only run LLM assessment on pending episodes
#   --once     Run once and exit
#   --interval Run continuously with N second interval

Demo Data with FHIR Notes

Create demo patients with clinical notes stored in FHIR for LLM parsing:

cd /home/david/projects/aegis/guideline-adherence

# Create demo patients with FHIR clinical notes
python demo_patients.py --persist --fhir --clear

# Then run assessment
python run_monitor.py --assess

Joint Commission (JC) Compliance

This module supports MM.09.01.01 EP 18-19 compliance by:

• Implementing evidence-based guidelines for infectious disease treatment
• Real-time monitoring of antimicrobial stewardship protocols
• Generating aggregate compliance metrics for quality improvement
• Providing drill-down capability to individual episodes
• Documenting interventions through the alert resolution workflow

References

• Surviving Sepsis Campaign. International Guidelines for Management of Sepsis and Septic Shock. 2021.
• AAP Clinical Practice Guideline. Evaluation and Management of Well-Appearing Febrile Infants 8 to 60 Days Old. Pediatrics. August 2021.
• Kimberlin DW. Neonatal herpes simplex infection. Clin Microbiol Rev. 2004.
• IDSA/SHEA. Clinical Practice Guidelines for Clostridium difficile Infection. 2018.
• CCHMC Pocket Docs. Bugs & Drugs Guidelines, Neonatal HSV Algorithm, C. diff Testing Algorithm. 2024.
• Bradley JS, et al. The Management of Community-Acquired Pneumonia in Infants and Children Older Than 3 Months of Age: PIDS and IDSA. Clin Infect Dis. 2011.
• The Joint Commission. MM.09.01.01 - Antimicrobial Stewardship Standard. Effective January 1, 2023.
• CMS Severe Sepsis and Septic Shock Early Management Bundle (SEP-1).