SONiC Reports ECMP Failure After Adding a Third Spine Through SAI

During lab validation of a SONiC-based fabric, ECMP forwarding worked with two spines but failed after adding a third spine, coinciding with a next-hop group replacement in SAI that did not program the new group correctly.

Research Overview

The issue was observed in a controlled topology where IPv4 routes were advertised from spines to leafs via BGP EVPN, with traffic forwarding checked as the fabric scale increased.

The failure appeared when the topology moved from a two-spine setup to three spines, where forwarding consistency broke and a crash was observed after orchestrator activity around next-hop groups.

Key Findings

In SONiC, ECMP is implemented using next-hop groups, and when topology changes, SONiC updates groups to include the additional equal-cost paths.

When moving from two to three spines, SONiC attempted to replace the existing two-next-hop group with a new group containing three next hops, but the SAI layer returned a group programming failure.

Technical Breakdown

Logs from swss and orchagent showed next-hop groups being created with multiple members, followed by an attempted remove-and-recreate sequence during the scale-up event to add the third spine.

During the transition, the remove operation reported SAI_STATUS_OBJECT_IN_USE, and the group removal step failed with an orchagent error, after which the orchestrator exited.

Root Cause and Scope

The blog attributes the root cause to an SAI-level implementation issue in handling next-hop group updates during the 2-to-3 ECMP transition.

The behavior was reported as platform-specific, seen only on the tested platform while operating in the spine role, and described as not reproducible in the same way on the smaller two-spine setup with the same software version.

Operational Impact

The immediate impact was that ECMP forwarding stopped working after the third spine was added, despite the earlier two-spine configuration functioning correctly in the lab.

The failure path included next-hop group programming failures and orchestrator instability, tied to how next-hop group replacement was executed through SAI during scale-up.

Leadership Perspective

The blog highlights that incremental scale testing can expose faults that only occur during transitional states rather than at a steady-state smaller configuration.

It also frames SAI behavior as a dependency for multi-vendor interoperability in SONiC deployments, since inconsistencies at that abstraction boundary can propagate into dataplane issues.

The blog describes an ECMP scale-up failure in a SONiC lab where SONiC replaced a two-path next-hop group with a three-path group, and an SAI update error prevented proper programming and led to group removal failures and orchestration crashes, underscoring the value of transitional scale validation for enterprise networking teams. Blog Signals brief is a fact-based summary of the vendor blog.