SONiC network migration report outlines L2 leaf-spine with MCLAG
The vendor describes a real-world data center network migration from a Layer-2 design that depended on Spanning Tree Protocol (STP) to a Layer-2 leaf-spine fabric built on SONiC Network Optimization Suite (NOS), with MCLAG for server uplinks. The report outlines the target architecture, migration approach, MCLAG behavior, and operational outcomes relevant to teams managing scaling and change in production networks.
Research Overview
The document covers the shift from a traditional Layer-2 network that used STP, extensive VLANs, trunking between aggregation and access, and single-homed or active-standby server connections. It describes observed issues in that environment, including reconvergence delays, blocked redundant paths, operational complexity, and scalability limits from Virtual Local Area Network (LAN) (VLAN) and broadcast domain growth.
To address those limitations, the author implemented a Layer-2 leaf-spine fabric using SONiC Network Operating System (OS) with Multi-Chassis Link Aggregation for server connectivity. The goal was a loop-free forwarding design without STP in the data plane while maintaining Layer-2 adjacency for existing workloads.
Key Findings
Post-migration, the report states that STP events were removed from operations and bandwidth utilization improved through active-active links. It also reports faster failure convergence and simplified troubleshooting and visibility compared with the prior design.
The paper also records challenges encountered during adoption, including an initial learning curve tied to SONiC’s architecture and service model. It highlights the need for strict configuration consistency across MCLAG peers and notes that clear runbooks for failure scenarios were required.
Technical Breakdown
The target design is a Layer-2 leaf-spine (L2LS) topology where spine switches provide a non-blocking Layer-2 fabric between leaf switches and leaf switches connect servers while forming MCLAG pairs at the access layer. Server connectivity is described as dual-homed to two leaf switches using port-channels, intended to maintain full bandwidth utilization while keeping traffic at Layer 2.
For routing and security, the document places the default gateway and Layer-3 termination on a firewall, while SONiC leaf-spine functions as a Layer-2 switching domain. It states that server VLANs are extended across the SONiC leaf switches using MCLAG, and that the firewall continues to provide inter-VLAN routing, Network Address Translation (NAT), inspection, and security policy enforcement, with the intent of keeping routing logic and security policies unchanged during migration.
Operational Impact and Migration Strategy
The migration approach avoids a “big bang” cutover and instead uses a parallel build model. It describes racking and cabling the new SONiC spine and leaf switches alongside the existing legacy gear, then creating a temporary high-bandwidth Layer-2 bridge between the legacy core and the new SONiC spines to extend the L2 domain for Virtual Machine (VM) migration.
Host migration is described as rack-by-rack movement of cables from legacy access to SONiC leaf switches along with updating server-side bonding to Link Aggregation Control Protocol (LACP) (802.3ad). After each batch, the report says connectivity and application health were verified, followed by severing the bridge link and powering down the legacy hardware once all hosts were moved.
Overall, the document presents the operational effects of moving from an STP-dependent Layer-2 network to a SONiC-based leaf-spine Layer-2 fabric with MCLAG, while keeping Layer-3 termination on an external firewall. This Blog Signals brief is a fact-based summary of the vendor blog.