Aviz networks details packet deduplication solution for data center monitoring

Network observability in data centers can be hindered by redundant network traffic from sources such as TAPs and mirrored configurations, leading to processing overhead and reduced monitoring efficiency. This blog outlines a deduplication approach that filters, identifies duplicates, and optimizes traffic distribution to enhance analytics accuracy and network performance.

Research Overview

The issue addressed involves the extensive bandwidth consumption and processing caused by duplicate packets in enterprise network infrastructures. Redundant data from mirrored traffic elevates storage and computational demands, complicating effective network monitoring and security analytics.

The blog's proposed method involves systematic filtering of irrelevant traffic, followed by real-time duplication detection and removing redundant packets before forwarding data to analytics systems.

Technical Breakdown

The architecture includes several components: data sources capturing traffic via physical TAPs and SPAN/ERSPAN; a filtering layer using an Open Packet Broker Network Operating System (OS) to limit forwarded data to relevant traffic; a core fabric implemented with high-capacity switches; and an Aviz Service Node performing deduplication.

This service node applies a DPDK-accelerated engine to inspect packets within time windows ranging from 2 to 8 milliseconds. It assesses packets based on configurable parameters such as packet source types (full packet or routed packet), anchor points within packets, and comparison offsets. Duplicate packets detected within these constraints are dropped, and unique traffic is reintegrated into the core fabric.

Product Update

The deduplication process leverages specialized hardware and software configurations to handle real-time packet analysis at data center scale. It incorporates adjustable settings for packet inspection granularity and duplicate detection window size, affording customization based on network environments.

Following deduplication, the traffic is load-balanced and distributed to various analytics tools for monitoring, security evaluation, and performance assessment, which streamlines resource consumption by eliminating redundant data.

Operational Impact

The approach reduces processing overhead from duplicate packets by 30 to 50 percent, increasing the efficiency of network monitoring operations. It also supports improved analytics responsiveness by supplying cleaner input data.

By lowering bandwidth and storage requirements through effective filtering and deduplication, it curtails operational costs and optimization of computational resources. Security monitoring benefits from enhanced accuracy in traffic visibility, supporting compliance with regulatory data retention policies.

Eliminating duplicate traffic at the core fabric level helps maintain scalability and performance in high-throughput data center networks by ensuring only unique packets are analyzed and stored.

Overall, this deduplication method contributes to maintaining network infrastructure performance and security by managing data volume and flow intelligently.

This Blog Signals brief summarizes the vendor's presentation of network traffic deduplication technology that integrates filtering, real-time analysis, and distribution to optimize data center monitoring and analytics.