Apache ORC
Apache ORC (Optimized Row Columnar) is a columnar storage format for Hadoop-based and other big data processing systems, designed for efficient storage, compression, and query execution on large datasets (data storage / analytics).
- Columnar file format for structured data storage in Hadoop ecosystems (data storage).
- Supports predicate pushdown, column pruning, and lazy decompression for query performance (data analytics optimization).
- Uses lightweight and heavyweight compression schemes and encoding strategies to reduce storage footprint (data compression).
- Provides rich type support, including nested types like structs, lists, maps, and unions (data modeling).
- Integrates with Apache projects and processing engines that read and write ORC files (big data ecosystem interoperability).
More About Apache ORC
Apache ORC (Optimized Row Columnar) is a self-describing, type-aware columnar storage format designed for large-scale analytical workloads in Hadoop and similar distributed data processing environments (data storage / analytics). ORC addresses the need to store large volumes of tabular and semi-structured data in a format that minimizes disk usage and improves scan and query performance when compared to row-based formats.
The ORC format organizes data by column rather than by row, which allows analytical engines to read only the columns required for a given query (data analytics optimization). This approach supports column pruning and predicate pushdown, where filter conditions can be evaluated using column-level statistics before full data blocks are read. As a result, systems can skip large portions of data that do not match query predicates, lowering I/O and Central Processing Unit (CPU) usage for analytical queries.
ORC files are divided into stripes, each stripe containing a set of rows stored in a columnar layout along with indexes and statistics (data storage internals). Within stripes, ORC uses encoding techniques such as run-length encoding, dictionary encoding, and bit packing, combined with compression codecs, to reduce storage footprint (data compression). The file footer and metadata regions store schema information, column statistics, and indexes, which enable readers to understand the file structure without external schema definitions (schema management).
The format supports a range of primitive and complex data types, including integers, floating point numbers, decimals, strings, timestamps, binary, as well as structs, lists, maps, and unions (data modeling). This type richness allows ORC to represent nested data structures commonly used in big data applications. In addition, ORC maintains column-level and stripe-level statistics such as minimum, maximum, and count, which query engines can use for optimization (query optimization).
In enterprise environments, ORC is used as a storage format in data lakes, warehouse-style workloads, and large-scale batch processing pipelines (enterprise data architecture). It is commonly read and written by processing engines within the Apache ecosystem that support ORC as a native or pluggable format (big data ecosystem interoperability). Because ORC is self-describing, enterprises can evolve schemas over time while maintaining access to historical data, as readers can interpret file-level schema information (schema evolution).
From a directory and taxonomy standpoint, Apache ORC is categorized as a columnar data storage format and file specification for analytical processing, typically deployed in Hadoop-compatible file systems and cloud object storage (data storage / analytics). Its role is to provide an efficient on-disk representation that underpins query performance and storage efficiency for large datasets in distributed processing frameworks.