mikepapadim commented on PR #3734:
URL: https://github.com/apache/parquet-java/pull/3734#issuecomment-6880604268

   I ran this PR as a writer for Apache Fluss lake tiering, which is one of the 
downstream users named in the description. I used head 1856d5b8, built against 
parquet 1.18.2 on JDK 21, and the 10 tests pass. The data was a TPC-H 
lineitem-like table: 19 nullable columns (INT32, INT64, DOUBLE, DATE, STRING, 
TIMESTAMP) in 16384-row batches, written with ZSTD. For comparison I wrote the 
same rows with `ExampleParquetWriter`.
   
   **Page sizing.** With dictionary off on both sides, this PR's file is 26.8 
MB against 21.7 MB from `ExampleParquetWriter` (128 MB of Arrow input). The gap 
is in columns where encoding doesn't matter: a date column is 1.94 vs 0.70 MB, 
and a constant INT32 column is 30.7 vs 2.7 KB. The cause is in `writeBatch`:
   
   ```java
   int typeLen = col.getPrimitiveType().getTypeLength();
   estimatedRowBytes += typeLen > 0 ? typeLen : VAR_WIDTH_ESTIMATE_BYTES;
   ...
   int rowsPerPage = Math.max(1, TARGET_PAGE_SIZE_BYTES / 
Math.max(estimatedRowBytes, 1));
   ```
   
   `getTypeLength()` is 0 for every type except FIXED_LEN_BYTE_ARRAY, so each 
column is estimated at 32 bytes. Because the estimate is summed over all 
columns, the 1 MB target is spread across the whole row. With 19 columns that 
gives 1724 rows a page: about 7 KB INT32 pages and 590 pages a column chunk, 
where `ExampleParquetWriter` writes 49. Each page is compressed on its own, so 
ZSTD gets very little context.
   
   I tried sizing by the widest column instead, using primitive widths (BOOLEAN 
1, INT32/FLOAT 4, INT64/DOUBLE 8, INT96 12, FLBA its length, binary 32), and 
taking the max instead of the sum:
   
   | Arrow input | as is | per-column page sizing |
   |---:|---:|---:|
   | 128 MB | 700 ms, 26.8 MB | 457 ms, 21.8 MB |
   | 512 MB | 2568 ms, 107.5 MB | 1733 ms, 87.5 MB |
   
   With the change, the output matches `ExampleParquetWriter` with dictionary 
off (21.79 vs 21.74 MB), and the write is about a third faster. Pages are still 
capped at one batch, because pages don't span `writeBatch` calls. A real fix 
should probably also estimate each column's bytes separately (or cut pages per 
column by bytes), so that wide string columns and narrow INT32 columns both get 
sensible pages. I have this as a small commit with two tests: 
`testPageSizeDoesNotShrinkWithColumnCount`, plus an assertion on the page count 
in `testLargeBatchProducesMultiplePages`, whose comment expects about 2 pages 
but gets 16 today. It's up as a PR against your branch: 
https://github.com/qzyu999/parquet-java/pull/1
   
   **Dictionary.** After that change, the remaining gap to a dictionary-encoded 
file (16.9 MB from `ExampleParquetWriter`) comes from low-cardinality strings 
and repeated doubles. For example, a 4-value string column is 798 KB plain vs 
30 KB with dictionary. So dictionary encoding, which is already listed under 
follow-ups, matters for size.
   
   For reference, on the same machine (i9-13900K), the row-object path through 
Iceberg's `GenericRecord` writer takes 7.3 s for the 512 MB.
   
   **JDK note.** On JDK 17+, the writer needs 
`--add-opens=java.base/java.nio=ALL-UNNAMED`, because `ArrowBuf.nioBuffer()` 
goes through `MemoryUtil.directBuffer`. It might be worth documenting that next 
to the writer.
   
   **Formatting.** Running `mvn spotless:apply` in `parquet-arrow` reformats 8 
of the new main files and the test. CI hasn't run on this PR yet, so that will 
probably be the first failure once the workflows are approved.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to