Types on Parquet

Geospatial Type

Mon, 01 Jan 0001 00:00:00 +0000

Geospatial Definitions

This document contains the specification of geospatial types and statistics.

Background

The Geometry and Geography class hierarchy and its Well-Known Text (WKT) and Well-Known Binary (WKB) serializations (ISO variant supporting XY, XYZ, XYM, XYZM) are defined by OpenGIS Implementation Specification for Geographic information - Simple feature access - Part 1: Common architecture, from OGC (Open Geospatial Consortium).

The version of the OGC standard first used here is 1.2.1, but future versions may also be used if the WKB representation remains wire-compatible.

Logical Types

Mon, 01 Jan 0001 00:00:00 +0000

Parquet Logical Type Definitions

Logical types are used to extend the types that parquet can be used to store, by specifying how the primitive types should be interpreted. This keeps the set of primitive types to a minimum and reuses parquet’s efficient encodings. For example, strings are stored with the primitive type BYTE_ARRAY with a STRING annotation.

This file contains the specification for all logical types.

Metadata

The parquet format’s LogicalType stores the type annotation. The annotation may require additional metadata fields, as well as rules for those fields.

Variant Shredding

Mon, 01 Jan 0001 00:00:00 +0000

Variant Shredding

The Variant type is designed to store and process semi-structured data efficiently, even with heterogeneous values. Query engines encode each Variant value in a self-describing format, and store it as a group containing value and metadata binary fields in Parquet. Since data is often partially homogeneous, it can be beneficial to extract certain fields into separate Parquet columns to further improve performance. This process is called shredding.

Shredding enables the use of Parquet’s columnar representation for more compact data encoding, column statistics for data skipping, and partial projections.

Variant Type

Mon, 01 Jan 0001 00:00:00 +0000

Variant Binary Encoding

A Variant represents a type that contains one of:

Primitive: A type and corresponding value (e.g. INT, STRING)
Array: An ordered list of Variant values
Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys.

A Variant is encoded with 2 binary values, the value and the metadata.

There are a fixed number of allowed primitive types, provided in the table below. These represent a commonly supported subset of the logical types allowed by the Parquet format.