Github user rustagi commented on the issue:
https://github.com/apache/spark/pull/11205
Sorry haven't been able to confirm this patch becaus have not seen issue in
production for quite some time.
It was much more persistent with 2.0 than 2.1
Not sure of cause
Github user rustagi commented on the issue:
https://github.com/apache/spark/pull/11205
I can confirm that removing speculation & setting maxtaskfailure to 1
eliminates this problem. Will try the patch & confirm
---
If your project is set up for it, you can reply to th
Github user rustagi commented on the issue:
https://github.com/apache/spark/pull/11205
I am seeing this issue quite frequently. Not sure what is causing it but
frequently we will get a onTaskEnd event after a stage has ended. This will
cause the numRunningTasks to become negative
