kszucs commented on code in PR #55639:
URL: https://github.com/apache/arrow/pull/55639#discussion_r4597155480


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -326,13 +326,11 @@ class ContentDefinedChunker::Impl {
         }
 
         if (rep_level == 0 && gearhash.NeedNewChunk()) {
-          // if we are at a record boundary and need a new chunk, we create a 
new chunk
-          auto levels_to_write = offset - prev_offset;
-          if (levels_to_write > 0) {
-            chunks.push_back({prev_offset, prev_value_offset, 
levels_to_write});
-            prev_offset = offset;
-            prev_value_offset = value_offset;
-          }
+          // if we are at a record boundary and need a new chunk, we create a 
new chunk,
+          // at offset 0 it is empty and closes the page of the previous call
+          chunks.push_back({prev_offset, prev_value_offset, offset - 
prev_offset});
+          prev_offset = offset;
+          prev_value_offset = value_offset;

Review Comment:
   Only possible at offset 0, since `offset` advances every iteration and 
`prev_offset = offset` after each push, same as the flat branches. It never 
becomes an empty page: the column writer only closes a page when values are 
buffered, which the tests now assert. Documented in `ValidateChunks`.



##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -396,6 +396,7 @@ ParquetInfo GetColumnParquetInfo(const 
std::shared_ptr<Buffer>& data, int column
     while (auto page = page_reader->NextPage()) {
       if (page->type() == PageType::DATA_PAGE || page->type() == 
PageType::DATA_PAGE_V2) {
         auto data_page = static_cast<DataPage*>(page.get());
+        EXPECT_GT(data_page->num_values(), 0) << "empty data page";

Review Comment:
   GTest already reports the failing parametrized test, keeping it simple.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to