Re: [I] [SUPPORT] Metadata table not cleaned / compacted, log files growing rapidly [hudi]
danny0405 commented on issue #8567: URL: https://github.com/apache/hudi/issues/8567#issuecomment-2058029478 > 2a0969c9972ef746d377dbddd278ef13bf3d299d For mor table, it should be fine if it is the upsert semantics. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT] Metadata table not cleaned / compacted, log files growing rapidly [hudi]
Qiuzhuang commented on issue #8567: URL: https://github.com/apache/hudi/issues/8567#issuecomment-2057270718 > if you have any pending/inflight in data table timeline, metadata table compaction will stalled until that gets to completion. may be there is some lingering pending operation (clustering or something) which we need to chase and fix them if its just left hanging. can you post the contents of ".hoodie". please do sort the out out based on last mod time. Yes - checking .hoodie folder, we find pending `.hoodie/20240412125709660.commit.requested`. As a result, spark streaming writer can even not write data. Is it safe to remove this pending commit to fix this this issue? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT] Metadata table not cleaned / compacted, log files growing rapidly [hudi]
danny0405 commented on issue #8567: URL: https://github.com/apache/hudi/issues/8567#issuecomment-2033341535 There might be no good solutions for 0..x release, recently we have merged a PR that address this issue on master branch: https://github.com/apache/hudi/pull/10874, it would be involved in 1.0.0 GA release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT] Metadata table not cleaned / compacted, log files growing rapidly [hudi]
Qiuzhuang commented on issue #8567: URL: https://github.com/apache/hudi/issues/8567#issuecomment-2032292630 I also run into this issue with hud 0.14.0. Can we have fsck/repair for those cases? - I finally corrupt this table:) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
