pan3793 commented on code in PR #6473:
URL: https://github.com/apache/kyuubi/pull/6473#discussion_r1642652411
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecuteStatement.scala:
##########
@@ -104,6 +104,7 @@ class ExecuteStatement(
onError(cancel = true)
} finally {
shutdownTimeoutMonitor()
+ if (!spark.sparkContext.isStopped)
spark.sparkContext.cancelJobGroup(statementId)
Review Comment:
thought, this should be fine in most cases, but still a corner case that may
fail incremental collect queries.
when incremental collect is enabled,
1. trigger the job to calculate the first partition of the final stage
result RDD
2. when the above job is finished, change the operation state from RUNNING
to FINISHED
3.1 can job group
3.2 the client starts to consume the first partition result, after consuming
all records, triggers a new job to calculate the second partition of result RDD
3.1 and 3.2 could happen in parallel, but in most cases, 3.1 completes
quickly while 3.2 takes a few seconds, so the whole process is fine. while
there is a chance that the result of the first partition is small and the
client consumes the result quickly, and the scheduler is under high pressure
that jobGroup cancellation takes a few seconds, then such a code could fail the
query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
