[GitHub] [hudi] luffyd commented on issue #1866: [SUPPORT]Clean up does not seem to happen on MOR table
luffyd commented on issue #1866: URL: https://github.com/apache/hudi/issues/1866#issuecomment-668667895 Please resolve this, Cleans are happening fine. I also added, I think it comes at the expense of timeline feature. We will relax it later ` val compactionConfig = HoodieCompactionConfig.newBuilder() .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_FILE_VERSIONS) .retainFileVersions(1) .build() val writerConfig = HoodieWriteConfig.newBuilder() .withCompactionConfig(compactionConfig) .withPath(getHudiPath(spark)) .build() val writeClient = new HoodieWriteClient(spark.sparkContext, writerConfig) // Run cleaner val cleanStats = writeClient.clean() ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] luffyd commented on issue #1866: [SUPPORT]Clean up does not seem to happen on MOR table
luffyd commented on issue #1866: URL: https://github.com/apache/hudi/issues/1866#issuecomment-663741729 Ok thanks, I will be running "clean commands" from hudi cli so that my tests progress faster for streaming. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] luffyd commented on issue #1866: [SUPPORT]Clean up does not seem to happen on MOR table
luffyd commented on issue #1866: URL: https://github.com/apache/hudi/issues/1866#issuecomment-663648589 Thanks saitsh, I have inline turned on by default, Now I see cleans did happen! Is there a possibility that commits get archived before clean job is resulting in a noop. I will continue to monitor. Also can you confirm If I can run a clean job in a separate spark job concurrently while streaming write is happening, guess it should be fine as compaction runs have that ability This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
