JingsongLi opened a new pull request, #10685: URL: https://github.com/apache/paimon/pull/10685
Physical compaction and clustering rewrites both need to preserve stored BLOB references and avoid applying ingestion shuffle/sort options to their output. These overrides were configured locally for Spark DE clustering and separately for BLOB compaction. Centralize the four overrides in `CompactOptions.REWRITE_OPTIONS`, apply them at the Spark/Flink compact entry points, and reuse them in direct DE compaction tasks. Remove the dedicated Spark DE clustering table/relation setup. Partition selection still resolves against the original table before applying runtime overrides, preserving `partition.default-name` behavior. Compaction-specific file size and row limits remain local to those tasks. Validation: 52 core, 16 Flink and 12 Spark cases passed locally with normal static checks enabled. Coverage includes ordinary PK/append/sort compaction, partition filtering, BLOB compaction, inline descriptors/views/vectors, clustering, DV materialization and output row limits. Independent review completed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
