[
https://issues.apache.org/jira/browse/PARQUET-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269328#comment-17269328
]
ASF GitHub Bot commented on PARQUET-1580:
-----------------------------------------
wangyum commented on a change in pull request #647:
URL: https://github.com/apache/parquet-mr/pull/647#discussion_r561914480
##########
File path:
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ColumnChunkPageWriteStore.java
##########
@@ -119,14 +127,28 @@ public void writePage(BytesInput bytes,
+ compressedSize);
}
tempOutputStream.reset();
- parquetMetadataConverter.writeDataPageV1Header(
+ if (pageWriteChecksumEnabled) {
+ crc.reset();
+ crc.update(compressedBytes.toByteArray());
+ parquetMetadataConverter.writeDataPageV1Header(
(int)uncompressedSize,
(int)compressedSize,
valueCount,
rlEncoding,
dlEncoding,
valuesEncoding,
+ (int) crc.getValue(),
Review comment:
@gszadovszky Do we need to consider int overflow? For example:
```
crc.getValue() = 4169965210
(int) crc.getValue() = -125002086
(int) (crc.getValue() & 0x7FFFFFFF) = 2022481562
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Page-level CRC checksum verification for DataPageV1
> ---------------------------------------------------
>
> Key: PARQUET-1580
> URL: https://issues.apache.org/jira/browse/PARQUET-1580
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-mr
> Reporter: Boudewijn Braams
> Assignee: Boudewijn Braams
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> Now that the specification with respect to page level checksums has been
> clarified (see https://jira.apache.org/jira/browse/PARQUET-1539), we can
> actually implement it. What needs to be done:
> * Implement calculating and writing checksums on write path
> * Implement checksum validation on read path
> * Make writing out checksums and validation on reads optional (default off)
> * Implement tests
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
