hansva opened a new issue, #9474: URL: https://github.com/apache/hop/issues/9474
### What needs to happen? Two transforms compress with a fixed level that cannot be changed: - **Serialize to file** (`CubeOutput`, `plugins/transforms/cube`) opens a plain `GZIPOutputStream`, so it always writes at level 6. After #9439 made the row serialization faster, the compression is the main cost of the transform. - **Zip file** (`ZipFile`, `plugins/transforms/zipfile`) is hard-coded to `Deflater.BEST_COMPRESSION`, level 9. The *Zip file* action already has a level option; the transform has none. ### Proposed change Add a compression level option to both transforms, in the dialog and in the metadata. - Reuse `GzipCompressionLevel` from the engine (`org.apache.hop.core.compress.gzip`), which Sort rows and the Blocking transform already use (#9344). It gives the nine levels with a code and a description for the combo. - **Existing pipelines do not change.** A transform saved without a level keeps the level it always used: 6 for Serialize to file, 9 for Zip file. - For Zip file this means the "not stored" default differs from `GzipCompressionLevel.NOT_STORED_DEFAULT` (level 6), so the transform needs its own fallback to level 9. - The files stay readable by every Hop version: each level produces a normal GZIP or ZIP stream. De-serialize from file needs no change. To decide in review: the level of a newly created transform. Sort rows and the Blocking transform use level 1 for new transforms because their files are temporary. The files of these two transforms are kept, so level 6 for both is the conservative choice. ### Expected gain **Serialize to file**, measured with a prototype on 2 million order rows (medians of 5 runs, `apache/hop:Development`, JDK 21): | Level | Rows/s | Gain | File size | |---|---|---|---| | 6 (today) | 228,000 | | 72.8 MB | | 3 | 487,000 | 2.1x | 79.3 MB (+9%) | | 1 | 705,000 | 3.1x | 90.0 MB (+24%) | **Zip file**, compression alone, timed on two kinds of content: | Content | Level 9 (today) | Level 6 | Level 1 | Size at level 6 against level 9 | |---|---|---|---|---| | 208 MB CSV | 10.2 s | 5.7 s | 1.3 s | +1% | | 277 MB serialized rows | 99 s | 7.5 s | 1.8 s | +5% | How much of the Zip file transform's run time this is depends on the file sizes: with many small files, opening and closing the files costs more than the compression. That share is not measured. ### Issue Priority Priority: 2 ### Issue Component Component: Transforms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
