yanxiaole edited a comment on pull request #29392: URL: https://github.com/apache/spark/pull/29392#issuecomment-670993081
It happens when data structure getting modified, mainly `delete` causes the problem. Take `LevelDB` implementation for example, in its iterator's `next` function it will first check `hasNext` then call `get`, if the entry was deleted between two calls then it will throw out a `NoSuchElementException` which will interrupt outside workflow. In our cluster, it either terminates the `cleanLogs` thread or interrupts `checkForLogs` which introduces inconsistency between log and app listing. https://github.com/apache/spark/blob/2ec9b866285fc059cae6816033babca64b4da7ec/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java#L125 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
