Skip to main content

Apache Submarine 0.8.0

Apache Submarine 0.8.0 is an open-source platform for end-to-end Machine Learning (ML) (ML platform) that provides unified training, model management, and serving on Kubernetes-based and Hadoop/YARN-based infrastructure.

  • Unified ML workbench for authoring, running, and managing ML experiments (ML platform).
  • Support for distributed training workloads on Kubernetes and YARN clusters (ML training orchestration).
  • Model tracking and lifecycle management through experiment, metric, and artifact management (ML model management).
  • Notebook, job, and environment management integrated with containerized runtimes (developer productivity tooling).
  • Representational State Transfer (REST) APIs, web console, and integration hooks for embedding Submarine into existing data and ML stacks (platform integration).

More About Apache Submarine 0.8.0

  • Apache Submarine 0.8.0 is an open-source ML platform (ML platform) from The Apache Software Foundation designed to support end-to-end workflows, from data scientists developing models to operators deploying and managing them on shared compute infrastructure. It focuses on running ML workloads on Kubernetes and Hadoop/YARN clusters, giving organizations a single system to handle training, tracking, and serving across heterogeneous environments.
  • The project targets the problem space of managing distributed ML experiments and production workloads (ML operations) in environments where cluster resources are shared across teams and applications. It provides abstractions for jobs, environments, and workspaces so users can define training or inference tasks without managing low-level container orchestration or YARN application details. This helps align ML workflows with established big data and container platforms already present in many enterprises.
  • Core capabilities in Submarine 0.8.0 include job and experiment management (ML orchestration), notebook management (developer tooling), and environment management (runtime management). Users can submit ML jobs that describe resources, images, commands, and parameters, and Submarine maps these to Kubernetes or YARN workloads. The experiment and tracking components store metrics and artifacts associated with runs, while workspaces and environments manage code, dependencies, and container images.
  • The platform exposes a web console and REST APIs (API platform) that allow interaction with Submarine resources such as users, workspaces, experiments, models, and environments. Notebook integration (interactive computing) enables data scientists to launch and manage notebooks that run inside the same infrastructure as training jobs, using predefined or custom Docker images. This supports reproducible development and execution by aligning interactive analysis and batch training under one configuration model.
  • From an architectural perspective, Apache Submarine operates as a control plane (cluster management) that connects to back-end resource managers like Kubernetes and Hadoop YARN. It leverages container images for runtime isolation (containerization) and can integrate with storage backends for model artifacts and logs. The design allows organizations to plug Submarine into existing big data clusters or Kubernetes-based environments, aligning ML workflows with existing scheduling, security, and monitoring practices.
  • In enterprise and institutional environments, Submarine can be positioned as a shared ML platform (ML operations) for teams that use deep learning or other ML frameworks running on containerized infrastructure. It can support workloads that require Graphics Processing Unit (GPU) or Central Processing Unit (CPU) resources, batch or interactive patterns, and different frameworks, as long as they are packaged into suitable images. Its role in a directory taxonomy aligns with categories such as ML platforms, ML experiment tracking, distributed training orchestration, and Kubernetes/YARN-based ML infrastructure tooling.