ehurheap opened a new pull request, #19828: URL: https://github.com/apache/hudi/pull/19828
feat(cli): pass system properties through to archiver HoodieWriteConfig The ARCHIVE command in SparkMain accepts only positional args (minCommits, maxCommits, commitsRetained, enableMetadata, basePath) and builds HoodieWriteConfig from just those. Any hoodie.* property set via spark-submit — whether on SparkConf or on driver JVM opts — was silently ignored, so tuning e.g. hoodie.commits.archival.batch.size at run time was not possible. Merge System.getProperties() into the builder so that `-Dhoodie.*=...` in spark.driver.extraJavaOptions reaches the archiver config. No effect when no such properties are set. Immediate motivation: US events archiver hits the JVM 2GB byte-array ceiling inside HoodieAvroDataBlock.serializeRecords when batching completed clean/rollback/commit metadata for a 6.7M-partition table. Setting hoodie.commits.archival.batch.size=1 is the cheapest experiment to determine whether batching alone is the problem or a single instant's serialized metadata exceeds 2GB on its own — but requires this passthrough to take effect. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
