mattcasters commented on PR #9211: URL: https://github.com/apache/hop/pull/9211#issuecomment-6585073278
@hansva, thanks for the review. Parquet input now opens the filename with Beam FileSystems and a seekable InputFile, so a local path or a file://, gs://, s3:// or hdfs:// URI works. The file:// case is covered in BeamParquetFileIOTest. Timestamps are written as microseconds through ParquetWriteSupport.epochValue. Decimal BigNumbers are the unscaled bytes of the scaled BigDecimal. The reader returns Date, Timestamp, BigDecimal and byte[] for those columns. The round trip includes Date, Timestamp (including one microsecond before the epoch), BigNumber with a length, and Binary, and it checks the stored Parquet values as well as the Hop types. JSON and UUID fields are refused on Beam, and that is on the output docs page. The empty-input Count and ParDo are named with the transform name. A null row is only built for a bounded input in the global window, and the input docs page says so. Rows passed on increment written. The parquet dependency comment now says both sides are Parquet 1.18.1 and that the Beam loader sees two identical copies. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
