yanxiaole opened a new pull request #29392:
URL: https://github.com/apache/spark/pull/29392
### What changes were proposed in this pull request?
Synchronize listing iteration manually
### Why are the changes needed?
It seems to be a race condition in `FSHistoryProvider` that iterate the
listing with high concurrency:
```java
val stale = listing.view(classOf[LogInfo])
.index("lastProcessed")
.last(newLastScanTime - 1)
.asScala
.toList
```
`toList` will iterate items in the listing, if one of listing entries was
deleted by some clean thread, `checkForLogs` will throw out a
`java.util.NoSuchElementException` and abort current execution.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
The race contention is hard to reproduce, but the current logic seems
causing the issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
