wgtmac commented on code in PR #1177:
URL: https://github.com/apache/parquet-mr/pull/1177#discussion_r1501398970
##########
parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java:
##########
@@ -2255,27 +2269,47 @@ public static
org.apache.parquet.internal.column.columnindex.ColumnIndex fromPar
parquetColumnIndex.getNull_pages(),
parquetColumnIndex.getNull_counts(),
parquetColumnIndex.getMin_values(),
- parquetColumnIndex.getMax_values());
+ parquetColumnIndex.getMax_values(),
+ parquetColumnIndex.getRepetition_level_histograms(),
+ parquetColumnIndex.getDefinition_level_histograms());
}
public static OffsetIndex toParquetOffsetIndex(
org.apache.parquet.internal.column.columnindex.OffsetIndex offsetIndex) {
List<PageLocation> pageLocations = new
ArrayList<>(offsetIndex.getPageCount());
+ List<Long> unencodedByteArrayDataTypes = new
ArrayList<>(offsetIndex.getPageCount());
for (int i = 0, n = offsetIndex.getPageCount(); i < n; ++i) {
pageLocations.add(new PageLocation(
offsetIndex.getOffset(i), offsetIndex.getCompressedPageSize(i),
offsetIndex.getFirstRowIndex(i)));
+ Optional<Long> unencodedByteArrayDataType =
offsetIndex.getUnencodedByteArrayDataBytes(i);
+ if (unencodedByteArrayDataType.isPresent() &&
unencodedByteArrayDataTypes.size() == i) {
+ unencodedByteArrayDataTypes.add(unencodedByteArrayDataType.get());
+ }
+ }
+ OffsetIndex parquetOffsetIndex = new OffsetIndex(pageLocations);
+ if (unencodedByteArrayDataTypes.size() == pageLocations.size()) {
Review Comment:
This method converts an internal OffsetIndex to a generated thrift
OffsetIndex object which does not provide a builder. Therefore we need to check
if the unencodedByteArrayDataTypes list is malformed since it may be used in
the rewriter where the OffsetIndex was created somewhere else (e.g. from other
implementations).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
