Problem
Arrhythmia classifiers on the MIT-BIH dataset routinely report 98%+ accuracy. Two things inflate that: most beats are normal, so a model that says "normal" every time scores highly; and if beats from the same patient appear in both training and test sets, the model learns the patient, not the arrhythmia.
Method
Beats are segmented from the raw WFDB signals and split at the patient level using the standard DS1/DS2 protocol, so no patient leaks across the split. A logistic-regression baseline, then a random forest, then a small PyTorch network, all logged in MLflow. Evaluation is imbalance-aware: per-class precision and recall, confusion matrices and ROC-AUC alongside accuracy, plus a majority-class baseline to compare against.
Evaluation
The random forest scores 92.3% accuracy against a trivial always-normal baseline of 89.0%, and on the two rarest arrhythmia classes it is nearly blind: 1% recall on supraventricular beats, 0% on fusion beats. A logistic regression with balanced class weights scores only 57.9% accuracy, yet catches 16% of supraventricular, 56% of ventricular and 68% of fusion beats. The model with the worse headline number is the one that finds the arrhythmias. Knowing when a metric lies is the finding.
The decision it drives
Which metric to report to a clinician, and why patient-level splitting is not optional. The same discipline transfers to any imbalanced business problem: churn, fraud, conversion.