nsivabalan commented on issue #6474: URL: https://github.com/apache/hudi/issues/6474#issuecomment-1224816176
yeah. From what I see, the cleaner waits for the lock (which was acquired to apply `20220822020402958` to metadata table", but after retrying, before giving up, looks like the cleaner unlocks which should not happen. We did made a fix in 0.11.1 to avoid non owner releasing the lock [here](https://github.com/apache/hudi/pull/5255), but looks like there could be more to be looked into. ``` 02:06:31 : acquiring lock by 20220822020402958__deltacommit__INFLIGHT in MDT. 02:06:46 : clean is attempted in data table. (async cleaner) 02:06:48: clean tries to acquire lock. 22/08/22 02:06:48 INFO org.apache.hudi.client.transaction.TransactionManager: Transaction starting for Optional.empty with latest completed transaction instant Optional.empty 22/08/22 02:06:48 INFO org.apache.hudi.client.transaction.lock.LockManager: LockProvider org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider 02:07:47: after checking for compaction, new delta commit started in MDT. 02:07:50: we see the deltacommit state is moved to completed. 02:08:22 : new delta commit starts. regular writer. 02:08:59: tries to acquire lock. 02:11:10: tries to acquire lock. 02:13:21: tries to acquire lock. 02:15:32:tries to acquire lock. 02:17:43:tries to acquire lock. 02:19:53:tries to acquire lock. 02:22:04:tries to acquire lock. 02:24:15:tries to acquire lock. 02:26:25:tries to acquire lock. 02:28:36:tries to acquire lock. 02:30:47: INFO org.apache.hudi.client.transaction.TransactionManager: Transaction ending with transaction owner Optional.empty 22/08/22 02:30:47 INFO org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider: RELEASING lock at DynamoDb table = HudiLocker, partition key = process 22/08/22 02:30:47 INFO org.apache.hudi.client.transaction.TransactionManager: Transaction ended with transaction owner Optional.empty 25 mins so far from the time clean tried to acquire lock. clean fails since it could not acquire the lock. 22/08/22 02:31:00 : original owner who acquired the lock is releasing it now. 22/08/22 02:31:00 INFO org.apache.hudi.client.transaction.TransactionManager: Transaction ending with transaction owner Option{val=[==>20220822020402958__deltacommit__INFLIGHT]} 22/08/22 02:31:00 INFO org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider: RELEASING lock at DynamoDb table = HudiLocker, partition key = process 22/08/22 02:31:00 INFO org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider: RELEASED lock at DynamoDb table = HudiLocker, partition key = process 22/08/22 02:31:00 INFO org.apache.hudi.client.transaction.TransactionManager: Transaction ended with transaction owner Option{val=[==>20220822020402958__deltacommit__INFLIGHT]} ``` I might need to spend some more time to put in a fix for this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
