Jason-liujc commented on issue #7653: URL: https://github.com/apache/hudi/issues/7653#issuecomment-1687260803
We are encountering the same issue. After using DynamoDB as the lock table, we still see this error: `java.util.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes` What I observed: 1. I have 4 EMR Spark clusters that writes to the same table. One by one, it fails with the above error. When I look at the DynamoDB lock history, I see locks constantly getting created and released. 2. The DynamoDB lock is not at file level, but on the table level. So two Hudi jobs might try to write to the same files and one of them failure. It seems if there are a couple of concurrent jobs running at the same time writing to the same files, it'll go into some sort of failure storm, which might fail everything unless you set a really really high retry threshold. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
