Research

Try it — flip one bit in a live float32 weight ↓

• Cross-Scale Model Resilience for LLM Inference Under Hardware Faults

  2025

  Manuscript under review Co-author

  TRUE (Trustworthy and Responsible) AI Lab, Oregon State University

  Problem

  Bit flips in GPU memory can silently corrupt LLM inference, and a model at another scale is a cheap reference for noticing when they have.

  Built

  Robustness experiments and benchmarks supporting a speculative-decoding defense, where draft and target models cross-check each other.

  Result

  Accuracy under critical faults restored from 0% to 96.9–99.4%, at 0.1–0.4% latency overhead. Manuscript under review.

  Hardware faults can silently corrupt LLM inference, and numerical checks are a poor proxy for whether the output is still correct. The fault model is specific: random double-bit flips in weight tensors. Single-bit errors are corrected transparently by on-die GPU ECC, so double-bit errors — detected but not corrected — are the realistic residual threat.

  This work proposes a speculative-decoding-based defense. A smaller draft model and the target model cross-check each other, and the divergence between them flags corruption at the output level rather than in the weights. The signal is the maximum of the target’s normalized negative log-likelihood under the draft and the normalized Jensen–Shannon divergence between the two distributions, averaged over a sliding 10-token window, alarming at the 99.5th percentile of a fault-free calibration run. Verification is the detector; recovery reloads only the weight tensors whose hashes changed, then regenerates.

  What it achieves. Accuracy under critical faults goes from 0% to 96.9–99.4%, at 0.1–0.4% latency overhead in fault-free execution, with detection at or above 97.5% true-positive for a false-positive rate at or below 0.6%. Hash-based isolation with selective reload recovers 17.3–112.2× faster than a full model reload — 0.1 to 0.3 seconds. Across the sampled fault population, 88.6% of faults are benign and 11.4% critical.

  Evaluated on 8B–32B parameter target models — Llama-3-8B, Falcon3-10B and Qwen3-32B, paired with 1B–4B drafts — across GSM8K, MMLU and APPS, against four published baselines: Ranger, FT2, Structural Coding and LM-Fix.

  My contribution: experiments, benchmarks and the accuracy / fault-tolerance trade-off analysis, run on the lab’s 8× A40 Slurm cluster.

• Which defense actually holds, and why the survivors survive

  2026

  Unpublished — internal study Experiments and analysis

  TRUE (Trustworthy and Responsible) AI Lab, Oregon State University

  Problem

  Published fault-tolerance defenses are each evaluated on their own terms, so nobody knows which one to reach for.

  Built

  A 42-cell comparison: 7 defense configurations against 2 bit-flip attack classes across 3 benchmarks.

  Result

  Every fault the defense detected, it also corrected — so the failures that remain are detection failures, not repair failures.

  Published defenses against bit-flip attacks are each evaluated on their own benchmark, under their own fault model, so there is no honest answer to which one should I use. The two evaluation tracks were team priorities; I ran the comparison: 7 defense configurations × 2 attack classes (damage and stealth) × 3 benchmarks (GSM8K, DROP, TriviaQA), swept across flip counts from 0 to 50 — 42 evaluation cells, on Llama-3-8B.

  The result that changed what the team works on. Every fault the defense detected, it also corrected. Not one case, across any attack or any benchmark, was detected-but-uncorrected. That means the repair step is not where the losses are — every answer that comes back wrong is a detection failure. The early-warning signal is the whole problem, and the recovery machinery is already good enough.

  Where it goes blind. The stealth attack bites hardest at low flip counts and recovers as flips accumulate, which is the opposite of the intuition. In that low-flip regime accuracy collapses while detection stays near zero. The distinctive failure is output that is fluent, confident, and wrong — text that reads perfectly normally, produces no disagreement between draft and target, and therefore raises no alarm. That is the target for a better signal.

  Getting there meant recovering legacy fault records from a superseded cluster path and verifying that the recovered flip records reproduce bit-for-bit through the new pipeline, so old results stayed comparable without re-spending GPU time; finding and fixing two correctness bugs in the shared evaluation runner; and correcting the cohort methodology — conditioning failure analysis on clean-correct samples, which retracted a preliminary cross-attack result that turned out to be entirely baseline model error.

Code for this work is lab-owned and private until publication.