Parquet float/double values can hold any IEEE floating point value - https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L413. So there's no reason you can't write NaN to the files. If a reader isn't handling NaN values correctly, that seems like an issue with that reader, although I think you're correct in that you're more likely to hit reader bugs with NaN than NULL. (I may be telling you something you already know, but thought I'd start with that).
I don't think the Parquet format is opinionated about what NULL vs NaN means, although I'd assume that NULL means that the data simply wasn't present, and NaN means that it was the result of a floating point calculation that resulted in NaN. The rep/definition level encoding is fairly complex because of the handling of nested types and the various ways of encoding the sequence of levels. The way I'd think about it is: - If you don't have any complex/nested types, rep levels aren't needed and the logical def levels degenerate into 1=not null, 0 = null. - The RLE encoding has a bit-width implied by the max def level value - if the max-level is 1, 1 bit is needed per value. If it is 0, 0 bits are needed per value. - If the field is required, the max def level is 0, therefore all values are 0, therefore the def levels can be "decoded" from nothing and the def levels can be omitted for the page. - If the field is nullable, the bit width is 0, therefore each def level is logically a bit. However, RLE encoding is applied to the sequence of 1/0 levels - https://github.com/apache/parquet-format/blob/master/Encodings.md The last point is where I think your understanding might diverge from the implementation - the encoded def levels are not simply a bit vector, it's a more complex hybrid RLE/bit-packed encoding. If you use one of the existing Parquet libraries it will handle all this for you - it's a headache to get it all right from scratch. - Tim On Mon, May 13, 2019 at 8:43 AM Brian Bowman <[email protected]> wrote: > All, > > I’m working to integrate the historic usage of SAS missing values for IEEE > doubles into our SAS Viya Parquet integration. SAS writes a NAN to > represent floating-point doubles that are “missing,” i.e. NULL in more > general data management terms. > > Of course SAS’ goal is to create .parquet files that are universally > readable. Therefore, it appears that the SAS Parquet writer(s) will NOT be > able to write the usual NAN to represent “missing,” because doing so will > cause a floating point exception for other readers. > > Based on the Parquet doc at: > https://parquet.apache.org/documentation/latest/ and by examining code, I > understand that Parquet NULL values are indicated by setting 0x000 at the > definition level vector offset corresponding to each NULL column offset > value. > > Conversely, It appears that the per-column, per page definition level data > is never written when required is not specified for the column schema. > > Is my understanding and Parquet terminology correct here? > > Thanks, > > Brian >
