[ https://issues.apache.org/jira/browse/ARROW-300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15667895#comment-15667895 ]
Wes McKinney commented on ARROW-300: ------------------------------------ One issue with doing compression only at the transport level is if people use the Arrow memory layout and metadata to create file formats for storing larger amounts of data. For example, I would like to deprecate the Feather metadata https://github.com/wesm/feather/blob/master/cpp/src/feather/metadata.fbs and use only the Arrow metadata. Unless you support column/buffer-level compression, then it would be expensive to read only a subset of the file. You could argue that such data should be stored as Parquet instead, but it does offer a flexibility that's really appealing (particularly since random access on memory-mapped Arrow-like data would be possible). > [Format] Add buffer compression option to IPC file format > --------------------------------------------------------- > > Key: ARROW-300 > URL: https://issues.apache.org/jira/browse/ARROW-300 > Project: Apache Arrow > Issue Type: New Feature > Components: Format > Reporter: Wes McKinney > > It may be useful if data is to be sent over the wire to compress the data > buffers themselves as their being written in the file layout. > I would propose that we keep this extremely simple with a global buffer > compression setting in the file Footer. Probably only two compressors worth > supporting out of the box would be zlib (higher compression ratios) and lz4 > (better performance). > What does everyone think? -- This message was sent by Atlassian JIRA (v6.3.4#6332)