[ https://issues.apache.org/jira/browse/ARROW-786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101836#comment-16101836 ]
Jacques Nadeau commented on ARROW-786: -------------------------------------- The current format of the java implementation is an embedded sign bit. GCC/Clang/Intel support __int128 which I believe on x86-64 machines is represented with the sign bit embedded (?). I remember talking to [~nongli] about this years ago and (if I recall correctly), we chose the Parquet representation based on his experiments with GCC or Clang/LLVM. (Unfortunately, I'm unable to find the thread.) The current Java implementation supports a 16-bit wide, sign-bit embedded twos-complement big-endian representation that is the same as the Parquet description here: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L81 > [Format] In-memory format for 128-bit Decimals, handling of sign bit > -------------------------------------------------------------------- > > Key: ARROW-786 > URL: https://issues.apache.org/jira/browse/ARROW-786 > Project: Apache Arrow > Issue Type: Improvement > Components: Format > Reporter: Wes McKinney > Fix For: 0.6.0 > > > cc [~cpcloud] > We found in ARROW-655 that we needed to add an extra bit for signedness for > decimals stored as 128-bit values to be able to use the Boost multiprecision > libraries. This makes Decimal128 not fit completely neatly as a 16-byte fixed > size binary value, and more of a {{struct<sign_bitmap: boolean, data: > fixed_size_binary(16)>}}. What is the current formata in the Java > implementation? We will need to document the memory layout for decimals that > maximizes compatibility across languages and eventually implement integration > tests for IPC. -- This message was sent by Atlassian JIRA (v6.4.14#64029)