SONiC Details SAI Next-Hop Group Failure When Adding a Third Spine

In SONiC lab validation, ECMP forwarding worked with two spines but stopped functioning after a third spine was added, coinciding with a failure to replace next-hop groups through the SAI layer. The case matters because enterprise fabrics rely on predictable scale-up behavior for BGP EVPN forwarding.

Research Overview

The report describes a debugging effort focused on why Equal-Cost Multi-Path routing broke after expanding a data center fabric from two to three spines. It attributes the fault to how the Switch Abstraction Interface handles next-hop group replacement during ECMP scale-up.

The environment used a SONiC-based fabric where IPv4 routes were advertised from spines to leafs via BGP EVPN. The symptoms included forwarding inconsistency and crashes during the topology change.

Key Findings

The blog states that ECMP routing based on next-hop groups functioned correctly in a two-spine setup. When the topology added a third spine, SONiC attempted to replace the existing next-hop group with a new group containing three next hops.

According to the described logs, the orchestration component attempted to remove the old next-hop group and create the updated group, but the group-creation step failed at the SAI layer. The reported failure included a SAI remove status of SAI_STATUS_OBJECT_IN_USE and a status tied to SAI_API_NEXT_HOP_GROUP.

Technical Breakdown

The blog explains that in SONiC, ECMP is implemented through next-hop groups, where each group represents the set of equal-cost forwarding paths. During topology changes, SONiC updates these groups so that the ECMP set reflects the new set of available paths.

During the two-to-three spine transition, the operational sequence included deleting the existing two-path group and programming a replacement group with an additional member. The blog attributes the breakdown to a platform-specific SAI implementation issue affecting allocation and programming of the new group entry.

Operational Impact

In the described lab, the control-plane debugging did not resolve the issue, so the investigation moved to dataplane programming and switch-side logs from swss and syncd. The failing condition occurred when the fabric added the third spine, creating the need to program a 3-path ECMP next-hop group.

The blog notes that the issue was observed only on a specific platform when operating in the spine role, while the same ECMP logic and test sequence worked in the smaller two-spine setup. It also states that the problem was rare and hard to reproduce because it surfaced only under a particular scale-up condition.

Conclusion

The blog’s overall takeaway is that ECMP next-hop group updates can fail at the SAI layer during transitional scale-up steps, producing group programming failures and orchestration crashes even when smaller topologies work. This Blog Signals brief is a fact-based summary of the vendor blog.

Full content from aviznetworks.com.