hansva opened a new issue, #9474:
URL: https://github.com/apache/hop/issues/9474

   ### What needs to happen?
   
   Two transforms compress with a fixed level that cannot be changed:
   
   - **Serialize to file** (`CubeOutput`, `plugins/transforms/cube`) opens a 
plain `GZIPOutputStream`, so it always writes at level 6. After #9439 made the 
row serialization faster, the compression is the main cost of the transform.
   - **Zip file** (`ZipFile`, `plugins/transforms/zipfile`) is hard-coded to 
`Deflater.BEST_COMPRESSION`, level 9. The *Zip file* action already has a level 
option; the transform has none.
   
   ### Proposed change
   
   Add a compression level option to both transforms, in the dialog and in the 
metadata.
   
   - Reuse `GzipCompressionLevel` from the engine 
(`org.apache.hop.core.compress.gzip`), which Sort rows and the Blocking 
transform already use (#9344). It gives the nine levels with a code and a 
description for the combo.
   - **Existing pipelines do not change.** A transform saved without a level 
keeps the level it always used: 6 for Serialize to file, 9 for Zip file.
   - For Zip file this means the "not stored" default differs from 
`GzipCompressionLevel.NOT_STORED_DEFAULT` (level 6), so the transform needs its 
own fallback to level 9.
   - The files stay readable by every Hop version: each level produces a normal 
GZIP or ZIP stream. De-serialize from file needs no change.
   
   To decide in review: the level of a newly created transform. Sort rows and 
the Blocking transform use level 1 for new transforms because their files are 
temporary. The files of these two transforms are kept, so level 6 for both is 
the conservative choice.
   
   ### Expected gain
   
   **Serialize to file**, measured with a prototype on 2 million order rows 
(medians of 5 runs, `apache/hop:Development`, JDK 21):
   
   | Level | Rows/s | Gain | File size |
   |---|---|---|---|
   | 6 (today) | 228,000 | | 72.8 MB |
   | 3 | 487,000 | 2.1x | 79.3 MB (+9%) |
   | 1 | 705,000 | 3.1x | 90.0 MB (+24%) |
   
   **Zip file**, compression alone, timed on two kinds of content:
   
   | Content | Level 9 (today) | Level 6 | Level 1 | Size at level 6 against 
level 9 |
   |---|---|---|---|---|
   | 208 MB CSV | 10.2 s | 5.7 s | 1.3 s | +1% |
   | 277 MB serialized rows | 99 s | 7.5 s | 1.8 s | +5% |
   
   How much of the Zip file transform's run time this is depends on the file 
sizes: with many small files, opening and closing the files costs more than the 
compression. That share is not measured.
   
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: Transforms


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to