Developer Guide on Parquet

Sub-Projects

Mon, 01 Jan 0001 00:00:00 +0000

The parquet-format project contains format specifications and Thrift definitions of metadata required to properly read Parquet files.

The parquet-java project is a Java library to read and write Parquet files. It consists of multiple sub-modules, which implement the core components of reading and writing a nested, column-oriented data stream, to and from the Parquet format, along with Hadoop Input/Output Formats, Pig loaders, and other Java-based utilities for interacting with Parquet.

The parquet-cpp project is a C++ library to read-write Parquet files. It is part of the Apache Arrow C++ implementation, with bindings to Python, R, Ruby and C/GLib.

Building Parquet

Mon, 01 Jan 0001 00:00:00 +0000

Building Java resources can be build using mvn package. The current stable version should always be available from Maven Central.

C++ thrift resources can be generated via make.

Thrift can be also code-genned into any other thrift-supported language.

Contributing to Parquet-Java

Mon, 01 Jan 0001 00:00:00 +0000

Pull Requests

We prefer to receive contributions in the form of GitHub pull requests. Please send pull requests against the github.com/apache/parquet-java repository. If you’ve previously forked Parquet from its old location, you will need to add a remote or update your origin remote to https://github.com/apache/parquet-java.git. Here are a few tips to get your contribution in:

Break your work into small, single-purpose patches if possible. It’s much harder to merge in a large change with a lot of disjoint features.
Create an Issue on the Parquet-Java issues.
Submit the patch as a GitHub pull request against the master branch. For a tutorial, see the GitHub guides on forking a repo and sending a pull request. Prefix your pull request name with the Issue GH-2935: (ex: https://github.com/apache/parquet-java/pull/2951).
Make sure that your code passes the unit tests. You can run the tests with mvn test in the root directory.
Add new unit tests for your code.
All Pull Requests are tested automatically on GitHub Actions.

If you’d like to report a bug but don’t have time to fix it, you can still raise an issue, or email the mailing list (dev@parquet.apache.org).

Releasing Parquet-Java

Mon, 01 Jan 0001 00:00:00 +0000

Setup

N.B. The mechanics of releasing parquet-format is the same (e.g. setting up keys, branching, votes, etc)

You will need:

PGP code signing keys, published in KEYS.
Permission to stage artifacts in Nexus.

Make sure you have permission to deploy Parquet artifacts to Nexus by pushing a snapshot:

mvn deploy

If you have problems, read the publishing Maven artifacts documentation.

Release process

Parquet uses the maven-release-plugin to tag a release and push binary artifacts to staging in Nexus. Once maven completes the release, the official source tarball is built from the tag.