jerryshao opened a new pull request #25552: [SPARK-28849][CORE] Add number of transferTo calls to avoid infinite loop in some occasional cases URL: https://github.com/apache/spark/pull/25552 This patch propose a way to detect the infinite loop issue in `transferTo` and make it fail fast. The detailed issue is shown in https://issues.apache.org/jira/browse/SPARK-28849. Here propose a new undocumented configuration and a way to track the number of `transferTo` calls, if the number is larger than the specified value, it will jump out of the loop and throw an exception. Typically this will not be happened, and user don't need to set this configuration explicitly, in our environment it happens occasionally and cause the task hung infinitely.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
