Repository: arrow Updated Branches: refs/heads/master 7f20f6e73 -> 38efabea9
ARROW-844: [Format] Update README documents in format/ Added a section reflecting specification maturity and stability. Author: Wes McKinney <wes.mckin...@twosigma.com> Closes #556 from wesm/ARROW-844 and squashes the following commits: 03dbb71 [Wes McKinney] Update README documents in format/ Project: http://git-wip-us.apache.org/repos/asf/arrow/repo Commit: http://git-wip-us.apache.org/repos/asf/arrow/commit/38efabea Tree: http://git-wip-us.apache.org/repos/asf/arrow/tree/38efabea Diff: http://git-wip-us.apache.org/repos/asf/arrow/diff/38efabea Branch: refs/heads/master Commit: 38efabea9bbc8d6386f96a635a95c53ba70e6149 Parents: 7f20f6e Author: Wes McKinney <wes.mckin...@twosigma.com> Authored: Tue Apr 18 11:43:13 2017 -0400 Committer: Wes McKinney <wes.mckin...@twosigma.com> Committed: Tue Apr 18 11:43:13 2017 -0400 ---------------------------------------------------------------------- format/README.md | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/arrow/blob/38efabea/format/README.md ---------------------------------------------------------------------- diff --git a/format/README.md b/format/README.md index 048badb..3aa8fdd 100644 --- a/format/README.md +++ b/format/README.md @@ -14,16 +14,14 @@ ## Arrow specification documents -> **Work-in-progress specification documents**. These are discussion documents -> created by the Arrow developers during late 2015 and in no way represents a -> finalized specification. - Currently, the Arrow specification consists of these pieces: - Metadata specification (see Metadata.md) - Physical memory layout specification (see Layout.md) -- Metadata serialized representation (see Message.fbs) +- Logical Types, Schemas, and Record Batch Metadata (see Schema.fbs) +- Encapsulated Messages (see Message.fbs) - Mechanics of messaging between Arrow systems (IPC, RPC, etc.) (see IPC.md) +- Tensor (Multi-dimensional array) Metadata (see Tensor.fbs) The metadata currently uses Google's [flatbuffers library][1] for serializing a couple related pieces of information: @@ -35,4 +33,16 @@ couple related pieces of information: schema, and enable a system to send and receive Arrow row batches in a form that can be precisely disassembled or reconstructed. +## Arrow Format Maturity and Stability + +We have made significant progress hardening the Arrow in-memory format and +Flatbuffer metadata since the project started in February 2016. We have +integration tests which verify binary compatibility between the Java and C++ +implementations, for example. + +Major versions may still include breaking changes to the memory format or +metadata, so it is recommended to use the same released version of all +libraries in your applications for maximum compatibility. Data stored in the +Arrow IPC formats should not be used for long term storage. + [1]: http://github.com/google/flatbuffers