mapleFU commented on code in PR #1044:
URL: https://github.com/apache/parquet-mr/pull/1044#discussion_r1271472674
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1627,23 +1627,36 @@ public ColumnChunkPageReader
readAllPages(BlockCipher.Decryptor headerBlockDecry
break;
case DATA_PAGE_V2:
DataPageHeaderV2 dataHeaderV2 =
pageHeader.getData_page_header_v2();
- int dataSize = compressedPageSize -
dataHeaderV2.getRepetition_levels_byte_length() -
dataHeaderV2.getDefinition_levels_byte_length();
- pagesInChunk.add(
- new DataPageV2(
- dataHeaderV2.getNum_rows(),
- dataHeaderV2.getNum_nulls(),
- dataHeaderV2.getNum_values(),
-
this.readAsBytesInput(dataHeaderV2.getRepetition_levels_byte_length()),
-
this.readAsBytesInput(dataHeaderV2.getDefinition_levels_byte_length()),
- converter.getEncoding(dataHeaderV2.getEncoding()),
- this.readAsBytesInput(dataSize),
- uncompressedPageSize,
- converter.fromParquetStatistics(
- getFileMetaData().getCreatedBy(),
- dataHeaderV2.getStatistics(),
- type),
- dataHeaderV2.isIs_compressed()
- ));
+ int dataSize = compressedPageSize -
dataHeaderV2.getRepetition_levels_byte_length() -
+ dataHeaderV2.getDefinition_levels_byte_length();
+ final BytesInput repetitionLevels =
this.readAsBytesInput(dataHeaderV2.getRepetition_levels_byte_length());
+ final BytesInput definitionLevels =
this.readAsBytesInput(dataHeaderV2.getDefinition_levels_byte_length());
+ final BytesInput values = this.readAsBytesInput(dataSize);
+ if (options.usePageChecksumVerification() &&
pageHeader.isSetCrc()) {
+ pageBytes = BytesInput.concat(repetitionLevels,
definitionLevels, values);
Review Comment:
Does this concat three parts and get crc? Can it accumulated crc and avoid
concat these buffers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@parquet.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
