kazdy commented on issue #8261: URL: https://github.com/apache/hudi/issues/8261#issuecomment-1478254161
in 0.12.1 there was a bug related to hive sync, I also observed increasing processing time, which was mostly spent on reading all files under .hoodie/archived directory you can disable hive sync to confirm if this is the issue EMR 6.10 comes with 0.12.2 and this is fixed in this version afaik switching to non global index could also help, depending on the workload you can also consider async table services if this is streaming job, but you'd need to rewrite your job and not use forEachBatch() as it only works with inline services -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
