Consolidated Review: “A Multimodal Sleep Foundation Model for Disease Prediction” (SleepFM)

Paper: Thapa et al., Nature Medicine, 2026
DOI: 10.1038/s41591-025-04133-4
Code: https://github.com/zou-group/sleepfm-clinical

Executive Summary

This consolidated review synthesizes seven independent expert assessments of SleepFM, a self-supervised foundation model trained on >585,000 hours of polysomnography (PSG) data from ~65,000 participants. The reviews exhibit remarkable consensus on both strengths and weaknesses, with all reviewers agreeing this represents a meaningful engineering contribution while expressing serious concerns about the clinical claims.

Verdict: Strong technical contribution with overstated clinical implications. The paper advances the state of the art in PSG representation learning, but disease prediction claims require substantial qualification.

Part 1: Validated Strengths (Consensus Points)

1.1 Dataset Scale and Curation (All Reviewers Agree: Major Strength)

The dataset is unprecedented for sleep AI research:

  • 585,000+ hours from 65,000+ participants
  • 5-25× larger than prior supervised sleep models
  • Multiple cohorts: Stanford Sleep Clinic (SSC), BioSerenity, MESA, MrOS, with SHHS held out for transfer learning
  • Multi-ethnic, wide age range (1-100 years)

This scale is not merely a “big data flex”—it enables robust representations that prior smaller-scale efforts could not achieve. The reviewers universally acknowledge this as a legitimate contribution.

1.2 Channel-Agnostic Architecture (All Reviewers Agree: Practical Innovation)

Real-world PSG recordings vary wildly in channel availability and montage. The model’s solution is elegant:

Channel heterogeneity → Attention-based pooling within modality → Permutation-invariant aggregation

Code verification (from models.py):

class AttentionPooling(nn.Module):
    def __init__(self, input_dim, num_heads=1, dropout=0.1):
        self.transformer_layer = nn.TransformerEncoderLayer(
            d_model=input_dim, 
            nhead=num_heads,
            dropout=dropout,
            batch_first=True
        )

This design makes cross-site deployment realistic—a genuine practical contribution that addresses real clinical deployment barriers.

1.3 Leave-One-Out Contrastive Learning (Reviewers Agree: Sensible Design)

LOO-CL aligns each modality’s embedding to the average of all other modalities:

Code verification (from pretrain.py):

if mode == "leave_one_out":
    for i in range(num_modalities):
        other_emb = torch.stack([emb[j] for j in list(range(i)) + list(range(i + 1, num_modalities))]).sum(0) / (num_modalities - 1)
        logits = torch.matmul(emb[i], other_emb.transpose(0, 1)) * torch.exp(temperature)

Reviewer Assessment: While not wholly novel (similar to multiview contrastive methods), this is a sensible design choice for multimodal fusion that handles missing modalities gracefully.

1.4 Model Efficiency (All Reviewers Note Positively)

  • 4.44M parameters for pretraining
  • ~0.91M learnable parameters for fine-tuning
  • 15 hours on single A100 GPU for one epoch on 432,000 hours of data

This is unusually tractable for “foundation model” work and makes replication realistic.

1.5 Reproducibility and Openness (All Reviewers Commend)

  • Full code release with pretrained weights
  • Multiple checkpoints (base model, sleep staging, disease prediction)
  • Stanford Sleep Dataset publicly available (via bdsp.io)
  • Demo notebook for MESA dataset

Part 2: Critical Weaknesses (Consensus Points)

2.1 Incident Disease Definition is Critically Flawed (6/7 Reviewers: Major Concern)

The Problem: Positive cases are defined as patients whose first phecode instance occurred >7 days after PSG.

Why This Matters:

  • For chronic diseases (hypertension, diabetes, CKD, dementia, Parkinson’s), “first code in this EHR” ≠ “new onset”
  • Often reflects “first appearance in this institution’s coding stream” after referral
  • The model may predict “already sick but not yet coded here” rather than “future disease onset”

Reviewer Consensus: A 7-day washout is not a serious incident definition for EHR phenotyping. Multiple reviewers independently calculated that this inflates performance significantly.

What Would Be Needed:

  • 6-12 month minimum washout
  • Exclude anyone with any pre-PSG evidence of the condition
  • Sensitivity analyses showing performance vs. minimum time-to-event

2.2 Selection Bias: The Elephant in the Room (7/7 Reviewers: Major Concern)

The cohort consists entirely of patients referred for clinical sleep studies. These are individuals with:

  • Suspected sleep disorders (OSA, insomnia)
  • Existing cardiovascular disease
  • Neurodegenerative symptoms
  • Obesity and metabolic syndrome
  • Psychiatric conditions

Consequence: High predictive performance for heart failure or dementia may reflect the model learning “this is a sick patient with disrupted sleep” rather than independent physiological risk signals.

One reviewer’s succinct summary: “The model may simply be an expensive way to diagnose Sleep Apnea and Atrial Fibrillation, which are well-known risk factors for these diseases.”

GLM 4.7’s framing: The model learns to predict disease based on “sick sleep”—how well would this perform on a healthy population?

2.3 Mini-Batch Cox Loss Implementation (3/6 Reviewers: Technical Red Flag)

The Problem Identified: The paper states that because computing Cox loss over all patients in one batch is infeasible, they compute it in batches of 32, “with patients sorted by event time in descending order to ensure correct computation of the partial likelihood.”

Code verification (from finetune_diagnosis_coxph.py):

def cox_ph_loss(hazards, event_times, is_event):
    # Sort event times and get corresponding indices
    event_times, sorted_idx = event_times.sort(dim=0, descending=True)
    hazards = hazards.gather(0, sorted_idx)
    is_event = is_event.gather(0, sorted_idx)
    log_cumulative_hazard = torch.logcumsumexp(hazards.float(), dim=0)
    # ...

Technical Issue: CoxPH requires risk sets that include all individuals still at risk at each event time. Sorting within minibatches does not recreate full risk sets—it optimizes a different objective. This is not a known valid approximation.

Reviewer Verdict: “This is not a nit. If implemented as literally described, the training objective is mis-specified.”

2.4 Baselines Are Not Strong Enough (7/7 Reviewers: Major Concern)

Current baselines:

  1. Demographics-only MLP (age, sex, BMI, race/ethnicity)
  2. End-to-end PSG model (same architecture, no pretraining)

GLM 4.7’s characterization: “This is a strawman.”

Missing critical baselines:

  • Standard PSG-derived clinical features: AHI, ODI, mean/min SpO₂, %time <90% SpO₂, arousal index, sleep efficiency, WASO, REM%, N3%, HRV metrics, PLMI
  • Classical feature engineering: Spectral power (delta, theta, alpha), heart rate variability, respiratory variability, entropy measures—a Random Forest or XGBoost on hand-crafted features often beats deep learning on tabular medical data
  • EHR comorbidity baseline: Problem list, Charlson/Elixhauser score, prior diagnoses, medications, labs
  • Combined baseline: EHR + conventional PSG metrics
  • Established risk scores: For outcomes like CV disease (Framingham, ASCVD), dementia (CAIDE), etc.
  • General time-series foundation models: TimeGPT, Chronos, Lag-Llama (GLM 4.7 contribution)

Why This Matters: Without these comparisons, “SleepFM predicts cancer/dementia” may just be capturing well-known correlates (obesity/OSA severity, hypoxemia, autonomic dysfunction) in a more flexible way. By omitting classical feature baselines, the authors hide the possibility that simple, interpretable spectral features might perform comparably.

2.5 AUROC Computation Under Censoring is Under-Specified (5/7 Reviewers)

The paper reports AUROC at a 6-year horizon alongside Harrell’s C-index. But for time-to-event data with censoring, “plain AUROC” can be biased depending on how censored-before-horizon individuals are handled.

The paper does not spell out:

  • Treatment of patients with insufficient follow-up
  • Whether inverse probability of censoring weights are used
  • Time-dependent AUC methods

Reviewer Note: “AUROC values are a major part of the headline claims. If AUROC computation mishandles censoring, ‘130 diseases with AUROC ≥ 0.75’ can be partly an artifact.”

2.6 Limited External Validation for Phenome-Wide Claims (6/7 Reviewers)

SHHS transfer learning only covers 6 cardiovascular outcomes due to label availability. The broad “130 diseases” claim remains effectively validated in one institution (Stanford) plus a temporal split.

This is insufficient for generalizable disease risk prediction claims across clinical settings.

2.7 Calibration is Ignored (GLM 4.7: Novel Concern)

The paper focuses almost exclusively on discrimination metrics (AUROC, C-Index) but never reports calibration (reliability curves).

Why This Matters:

  • If the model predicts a patient has 90% risk of death, do they actually die 90% of the time?
  • Deep learning models are notoriously overconfident
  • In clinical settings, an uncalibrated model is dangerous even with high AUROC
  • Calibration is essential for risk communication and clinical decision-making

What Would Be Needed: Reliability diagrams, Brier scores, or calibration-in-the-large metrics for key outcomes.

2.8 Temporal Drift Undermines “Foundation” Claims (GLM 4.7: Novel Framing)

The model performance drops on the “Temporal Test Set” (2020+ data). The authors acknowledge this, but GLM 4.7 frames it as a critical failure mode for any “Foundation Model”:

The Core Problem: If the physics of sleep hasn’t changed between 2015 and 2020, why does the model degrade?

Implication: The model is learning sensor-specific artifacts or clinic-specific biases rather than invariant physiological laws. If the model requires retraining every few years as hardware changes, its utility as a universal “foundation” is severely limited.

2.9 “Foundation Model” Terminology is Overstated (GLM 4.7: Novel Critique)

In the AI community (GPT, BERT), a foundation model implies zero-shot or few-shot capabilities—you give it a prompt or minimal data and it works.

SleepFM Reality: Requires full supervised fine-tuning for every new task and cohort.

GLM 4.7’s Verdict: “This is not a foundation model; it is a pre-trained backbone. Calling it a foundation model is marketing hype that overpromises on the model’s flexibility.”

Fair Counterpoint: The term “foundation model” is used loosely in medical AI; SleepFM is consistent with how others use the term in this domain. But the critique highlights that expectations should be calibrated accordingly.

2.10 The “Black Box” Problem (5/7 Reviewers)

For a Nature Medicine paper, interpretability is weak. The ablation study shows EEG helps with mental health and ECG helps with heart disease—this is trivial and matches established physiology.

What We Don’t Learn:

  • What micro-features in EEG predict Parkinson’s?
  • Is it REM sleep behavior disorder? Loss of spindle density?
  • Which respiratory patterns drive cardiovascular risk beyond known OSA severity?

Without these insights, the model remains a “curious oracle rather than a clinical tool” (GLM 4.7).

Part 3: Reviewer Verdicts on the Reviews

Review Quality Assessment

Review Strengths Weaknesses Overall Quality
5.2Pro1 Deep technical analysis of CoxPH issue, thorough methods critique Very long, some redundancy Excellent - Identifies the most critical technical flaw
5.2Pro2 Balanced, excellent missed connections section Occasionally verbose Excellent - Best structured critique
O4.5 Comprehensive, good clinical perspective Some redundancy with others Very Good - Solid synthesis
G3P Accessible, good “cynic’s view” framing Less technical depth Good - Useful high-level perspective
G4Expert Strong on missed citations, extensions Slightly more generous than warranted Very Good - Best extensions section
G4.1T Concise, well-calibrated Less detail on methods issues Good - Most balanced tone
GLM 4.7 Novel calibration critique, sharp temporal drift framing, “foundation model” definition pushback Misses Cox loss issue, no code verification Good - Fresh angles, less technical depth
K2T Most critical, thorough code analysis Possibly overcritical in places Very Good - Best code verification

Points Where Reviews Diverge

  1. Severity of Selection Bias: Some reviewers see this as a limitation to acknowledge; others view it as potentially invalidating the clinical claims entirely.

  2. LOO-CL Novelty: Some reviewers (G3P, K2T) view it as essentially “an engineering choice, not a fundamental ML breakthrough.” Others (G4Expert) give more credit for the multimodal fusion design.

  3. Clinical Utility: Ranges from “potentially useful for opportunistic screening” to “clinical utility is unproven and possibly nonexistent.”

Part 4: Verified Technical Points from Code Review

Confirmed Issues

  1. Mini-batch Cox Loss: The implementation confirms reviewers’ concerns—the loss is computed within 32-sample batches with sorting, which is not equivalent to full-cohort Cox partial likelihood.

  2. 7-Day Washout: Not explicitly visible in released code, but referenced in paper methods.

  3. Demographics Baseline Uses Only 4 Features: Confirmed in DiagnosisFinetuneDemoOnlyEmbed:

self.demo_embedding = nn.Sequential(
    nn.Linear(4, embed_dim // 4),  # 4 input features
    ...
)

Architecture Verification

The architecture matches paper descriptions:

  • Tokenizer: 6 convolutional layers (1→4→8→16→32→64→128)
  • AttentionPooling: Transformer layer for permutation-invariant aggregation
  • Temporal modeling: Bidirectional LSTM with hidden_size=embed_dim//2
  • Fine-tuning head concatenates PSG embeddings with demographic features

Part 5: Missed Connections and Extensions

Older Work Not Adequately Cited (Consensus)

  1. Multiview/Multimodal Representation Learning:

    • Deep CCA / generalized CCA
    • Contrastive multiview coding (CMC)
    • VICReg/Barlow Twins style non-contrastive alignment

  2. Permutation-Invariant Architectures:

    • Deep Sets / Set Transformer (the channel pooling is essentially a set-function)

  3. Survival Modeling Literature:

    • DeepSurv, DeepHit, discrete-time survival models
    • These handle minibatching naturally

  4. Sleep-Specific Physiology:

    • CAP (Cyclic Alternating Pattern) analysis
    • Cardiopulmonary coupling
    • Sleep-specific HRV metrics

  5. Time-Series Foundation Models (GLM 4.7):

    • TimeGPT, Chronos, Lag-Llama
    • How does a general-purpose time-series transformer compare to SleepFM when trained on this data?
    • By building a custom architecture, the authors may have missed architectural efficiencies from the broader time-series community

  6. “Sleep Age” / Biological Age Gap (GLM 4.7):

    • The paper cites Brink-Kjaer et al. (2022) on “Brain Age” but misses the opportunity to integrate age acceleration as a mechanistic covariate
    • If the model predicts mortality, is the mechanism simply that the patient’s sleep “looks old”?
    • Computing “Sleep Age Gap” (predicted age vs. chronological age) could strengthen mechanistic arguments

Promising Extensions (Reviewer Suggestions)

  1. Wearable Transfer Learning: Distill SleepFM to work with Apple Watch/Fitbit/Oura data. GLM 4.7 notes the channel-agnostic architecture is uniquely positioned for reduced montages (e.g., just frontal EEG + PPG)
  2. Treatment Response Prediction: Does CPAP effectiveness correlate with embeddings?
  3. Hierarchical Phenome Prediction: Leverage phecode ontology for multi-task learning
  4. Causal Discovery: Apply PCMCI or similar to learned embeddings. GLM 4.7: propensity score matching on sleep embeddings to see if “fixing” the sleep signal predicts better outcomes
  5. Integration with EHR Foundation Models: Combine sleep embeddings with clinical note embeddings

Part 6: Final Consolidated Assessment

What the Paper Actually Demonstrates

  1. ✅ Self-supervised pretraining at large scale produces useful PSG representations
  2. ✅ Channel-agnostic architecture enables cross-site deployment
  3. ✅ Pretraining improves performance over supervised baselines on the same data
  4. ✅ Transfer learning to SHHS (held-out cohort) works for cardiovascular outcomes
  5. ✅ Sleep staging performance is competitive with specialized models

What the Paper Does NOT Demonstrate

  1. ❌ That sleep alone “predicts” disease (confounding by existing pathology)
  2. ❌ That predictions generalize to population cohorts (only sleep clinic patients)
  3. ❌ Clinical utility or actionability of risk predictions
  4. ❌ Incremental value over standard PSG clinical features
  5. ❌ Correct survival modeling (mini-batch Cox issue)
  6. ❌ Calibration of risk predictions (GLM 4.7)
  7. ❌ Temporal stability expected of a true “foundation” (GLM 4.7)
  8. ❌ Zero/few-shot capabilities implied by “foundation model” terminology (GLM 4.7)

The headline claim—”from one night of sleep, SleepFM accurately predicts 130 conditions”—should be interpreted as:

“Among patients referred to sleep clinics, multimodal PSG embeddings show strong association with subsequent EHR codes for 130 conditions, likely reflecting baseline disease burden and diagnostic trajectories rather than independent sleep biomarkers.”

Score Summary (Based on Reviewer Consensus)

Dimension Score Notes
Technical Innovation 7/10 Solid engineering, sensible design choices
Dataset/Scale 9/10 Genuinely unprecedented for the field
Methodological Rigor 5/10 Major issues with incident definition, survival modeling, baselines
Clinical Significance 4/10 Overstated claims, utility undemonstrated
Reproducibility 8/10 Excellent code/data release
Overall Impact 6/10 Important step for field, but claims need major qualification

Bottom Line

For ML/engineering venues: This is a strong contribution demonstrating that foundation model approaches work for multimodal PSG at scale.

For clinical interpretation: The disease prediction claims are not as clean as headlines suggest. The combination of weak baselines, permissive incident definitions, referral bias, and potential survival modeling issues means the clinical value proposition remains under-supported.

The lasting contribution will likely be SleepFM as a reusable encoder for sleep researchers, not the specific disease prediction claims.

Appendix: Summary of Review Scores

Reviewer Technical Clinical Methodology Overall
GPT-5.2 Pro run 1 7 4 4 Major revision needed
GPT-5.2 Pro run 2 7 4 5 Major revision needed
Opus 4.5 7 4 5 Strong accept (methods); Revise (clinical)
Gemini 3 Pro 8 5 6 Strong Accept (resource); Weak Accept (methods)
Grok 4 Expert 7 5 6 High-quality with limitations
Grok 4.1 Thinking 7 5 6 Good contribution, not transformative
GLM 4.7 8 5 5 8/10 (Data/Engineering); 5/10 (Novelty/Insight)
Kimi K2 Thinking 7 4 5 Borderline accept with major revisions

Consensus: The reviews converge on the same conclusion through independent analysis—impressive engineering achievement with overclaimed clinical implications.

Appendix B: GLM 4.7 Review Summary

Unique Contributions:

  1. Calibration critique (reliability curves missing)
  2. Temporal drift as “foundation model” failure mode
  3. Comparison to general time-series foundation models (TimeGPT, Chronos, Lag-Llama)
  4. “Sleep Age Gap” as mechanistic covariate suggestion
  5. Sharp critique of “foundation model” terminology

Missing from GLM 4.7:

  1. Mini-batch Cox loss implementation issue (most critical technical flaw)
  2. Incident disease definition problem (7-day washout)
  3. Code verification
GLM 4.7’s Verdict: “8/10 for Data/Engineering 5/10 for Novelty/Insight. This is a solid piece of engineering… However, it is not the ‘GPT of Sleep.’ It is a highly optimized pre-training pipeline that improves predictions by a few percentage points over simple demographics.”

Memorable Quote: “It is a tool that tells us what will happen with high accuracy, but fails to tell us why, leaving the clinician—and the scientist—still in the dark.”