Skip to main content

Apache Streampipes

Apache StreamPipes is an open-source framework for modeling, processing, and analyzing industrial and Internet of Things (IoT) data streams (stream processing / industrial IoT analytics).

  • Graphical composition of streaming analytics pipelines, including filters, aggregations, pattern detection, and enrichment (stream processing).
  • Connectors and adapters for integrating data from industrial sensors, IoT devices, and enterprise systems (data integration).
  • Extensible processing element model for custom functions, algorithms, and domain-specific analytics (developer framework).
  • Tooling for live data exploration, monitoring, and visualization of streaming data (observability / analytics UI).
  • Deployment options for on-premises (on-prem) and cloud-native environments, including container-based setups (platform deployment).

More About Apache Streampipes

Apache StreamPipes is an open-source framework from The Apache Software Foundation focused on the design, execution, and management of pipelines for continuous processing of industrial and IoT data streams (stream processing / industrial IoT analytics). It addresses use cases where sensor data, machine telemetry, or other event streams need to be collected, transformed, and analyzed in near real time without requiring custom-built integration stacks for each scenario.

The core of Apache StreamPipes is a browser-based pipeline editor that allows users to compose streaming analytics pipelines from configurable building blocks (low-code stream processing). Users can define ingestion from various data sources, apply filters, aggregations, event pattern detection, and enrichment steps, and route results to storage systems or dashboards. This model supports use cases such as condition monitoring, production quality analysis, and alerting on machine states in industrial environments.

Apache StreamPipes provides a connector layer with adapters to integrate data from industrial protocols, IoT platforms, message brokers, and enterprise systems (data integration). Through this layer, data from sensors, PLCs, field devices, and existing Operational technology (OT) or IT systems can be brought into a unified streaming environment. Connectors are extensible so that organizations can implement custom adapters for proprietary or domain-specific sources.

The framework defines an extensible processing element concept that allows developers to contribute custom functions, analytics operators, and algorithms (developer framework). These elements can encapsulate domain logic, Machine Learning (ML) models, or specialized calculations and are exposed through the graphical pipeline editor for reuse by non-developer users. This extension mechanism supports organization-specific analytics libraries while keeping operational users within a visual design environment.

For operations and monitoring, StreamPipes includes tools for live data exploration, visualization, and pipeline monitoring (observability / analytics UI). Users can inspect data streams, validate pipeline behavior, and create views or dashboards to observe streaming metrics and events. This supports continuous improvement of analytics pipelines and quicker diagnosis when data characteristics or equipment behavior change.

Apache StreamPipes is designed to run in on-prem industrial networks as well as cloud or hybrid environments (platform deployment). It supports container-based deployment, aligning with Kubernetes and similar orchestration approaches, and integrates with other Apache projects through standard protocols and interfaces where applicable. Within an enterprise taxonomy, Apache StreamPipes fits into categories such as stream processing, industrial IoT analytics, low-code data engineering, and data integration for OT environments.