Blog post · November 19, 2026 · aviznetworks.com
Aviz Networks outlines GPU failures in Llama-3 training
- About Aviz Networks
- Also names AMD
- Arista Networks
- ECC
- It
- Meta Platforms
- Nvidia
- ServiceNow
- Read the original
Summary
Meta's Llama-3 405B training run recorded interruptions caused by hardware, with 78% tied to GPUs, memory, and network components. The findings matter to enterprise IT and AI infrastructure leaders because similar failure risks apply to any large AI cluster.
Research Overview
Meta reported that 78% of unexpected interruptions during a 54-day Llama-3 training run came from GPUs, memory, network switches, and other infrastructure components rather than software or scaling choices. Faulty GPUs accounted for 30% of interruptions, GPU HBM3 memory problems for 17%, and network switches and cables for 8%. Host maintenance and software bugs were listed among the other causes.
GPU Health Indicators
The post describes common GPU issues, including uneven utilization where some GPUs run at 100% while others stay below 20%. It also lists correctable single-bit and uncorrectable double-bit ECC memory errors, thermal and power anomalies that cause throttling, and failed health checks that signal hardware degradation. Without continuous monitoring, these signals may be missed.
ONES 4.0 Monitoring and Alerts
According to the post, ONES 4.0 provides monitoring across GPUs, CPUs, NICs, memory, SSDs, power, and network fabric, with support for NVIDIA Spectrum-X, AMD accelerators, SONiC-based networks, and Arista EOS. It runs proactive health checks that raise ECC error alerts, thermal throttling warnings, and GPU degradation flags. A dashboard displays GPU inventory and color-coded health status, and alerts can be routed to ITSM tools such as ServiceNow, Zendesk, or Slack.
Error Detection Example
The post includes an nvidia-smi query showing one A100 GPU with four retired pages from correctable single-bit errors and another with one retired page from an uncorrectable double-bit error. ONES assigns these events severity levels, labeling the single-bit case as a warning and the double-bit case as critical.
Threshold and Rule Configuration
Administrators can define healthy and unhealthy ranges, such as yellow for utilization above 60% and red above 80%. Alert rules can be scoped by workload, GPU type, or cluster, with examples including a Slack warning when a GPU sustains more than 75% utilization for five minutes and a critical alert above 85%.
The post concludes that hardware failures drove most Llama-3 interruptions and that continuous GPU monitoring can reduce wasted compute and downtime. For enterprise decision-makers, the findings apply to AI clusters regardless of vendor mix. This Blog Summary is a fact-based summary of the vendor blog.
Our summary. The original keeps its own byline.