[
https://issues.apache.org/jira/browse/PARQUET-2075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17654842#comment-17654842
]
ASF GitHub Bot commented on PARQUET-2075:
-----------------------------------------
wgtmac commented on PR #1014:
URL: https://github.com/apache/parquet-mr/pull/1014#issuecomment-1371885483
> If you can add more unit tests, particularly the combinations of prune,
mask, trans-compression etc, it would be better.
I have added some test cases in the `ParquetRewriterTest` to mix config of
prune, nullify and trans-compression.
There are two outstanding issues as below:
- Masking and encrypting same column is not supported yet.
- Fix `createdBy` of rewritten file to keep both original info and new
rewriter info.
I am inclined to fix them in separate JIRAs because this patch is simply a
refactoring patch to unify different rewriters without changing any behavior,
and it is large enough.
@ggershinsky @shangxinli Please review again when you have time. Thanks!
> Unified Rewriter Tool
> -----------------------
>
> Key: PARQUET-2075
> URL: https://issues.apache.org/jira/browse/PARQUET-2075
> Project: Parquet
> Issue Type: New Feature
> Reporter: Xinli Shang
> Assignee: Gang Wu
> Priority: Major
>
> During the discussion of PARQUET-2071, we came up with the idea of a
> universal tool to translate the existing file to a different state while
> skipping some level steps like encoding/decoding, to gain speed. For example,
> only decompress pages and then compress directly. For PARQUET-2071, we only
> decrypt and then encrypt directly. This will be useful for the existing data
> to onboard Parquet features like column encryption, zstd etc.
> We already have tools like trans-compression, column pruning etc. We will
> consolidate all these tools with this universal tool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
