zhengchenyu commented on code in PR #4899:
URL: https://github.com/apache/hive/pull/4899#discussion_r2324689270
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java:
##########
@@ -293,6 +293,13 @@ protected void initializeAndRunProcessor(Map<String,
LogicalInput> inputs,
rproc.run();
perfLogger.perfLogEnd(CLASS_NAME, PerfLogger.TEZ_RUN_PROCESSOR);
+
+ // Try to call canCommit to AM. If there is no other speculative attempt
execute canCommit, then continue.
+ // If there are other speculative attempt execute canCommit first, then
wait until the attempt is killed
+ // or the committed task fails.
+ while (!this.processorContext.canCommit()) {
Review Comment:
If canCommit returns false, it means another task attempt has committed. And
this task attempt is likely to be killed. Therefore, we can delay it for a
relatively long time for example, 500ms. In my opinion, backoff is not
necessary, because in most case, this task attempt will be killed!
However, we need to avoid low-probability events: the rare case where a task
attempt fails after committed, so we can't set an excessively long delay.
Thus, choosing the sleep time is crucial. It should account for
low-probability events. For example, if the time from task attempt commit to
task attempt failure is `t1`. Here the sleep time should be related to `t1`,
`0.5 * t1` seems reasonable. Perhaps a simple sleep time of 500ms is a good
choice.
This is just my personal opinion. please correct me if I'm wrong.
@abstractdog @deniskuzZ
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
