[jira] [Commented] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[ https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989448#comment-15989448 ] zhihai xu commented on HIVE-16456: -- Thanks [~xuefuz]! I created a Review Request for my patch at the following RB link: https://reviews.apache.org/r/58856/ > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. > - > > Key: HIVE-16456 > URL: https://issues.apache.org/jira/browse/HIVE-16456 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-16456.000.patch > > > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. If the InterruptedException happened in RemoteSparkJobMonitor and > LocalSparkJobMonitor, it will be better to kill the job. Also there is a race > condition between submit the spark job and query/operation cancellation, it > will be better to check driverContext.isShutdown right after submit the spark > job. This will guarantee the job being killed no matter when shutdown is > called. It is similar as HIVE-15997. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[ https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991134#comment-15991134 ] Chao Sun commented on HIVE-16456: - +1 > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. > - > > Key: HIVE-16456 > URL: https://issues.apache.org/jira/browse/HIVE-16456 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-16456.000.patch > > > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. If the InterruptedException happened in RemoteSparkJobMonitor and > LocalSparkJobMonitor, it will be better to kill the job. Also there is a race > condition between submit the spark job and query/operation cancellation, it > will be better to check driverContext.isShutdown right after submit the spark > job. This will guarantee the job being killed no matter when shutdown is > called. It is similar as HIVE-15997. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[
https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15992120#comment-15992120
]
Rui Li commented on HIVE-16456:
---
Thanks [~zxu] for working on this. One question is could you explain in what
situation will InterruptedException happen in the monitors? I think one case is
the sleep during check intervals. In other cases however, e.g.
{{sparkJobStatus.getState()}}, the InterruptedException may be wrapped and
thrown as a HiveException, which your patch doesn't handle.
And a minor improvement is we can check {{if (jobRef != null && !jobKilled)}}
before we enter the synchronized block right?
{code}
private void killJob() {
boolean needToKillJob = false;
synchronized(this) {
if (jobRef != null && !jobKilled) {
jobKilled = true;
needToKillJob = true;
}
}
{code}
> Kill spark job when InterruptedException happens or driverContext.isShutdown
> is true.
> -
>
> Key: HIVE-16456
> URL: https://issues.apache.org/jira/browse/HIVE-16456
> Project: Hive
> Issue Type: Improvement
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-16456.000.patch
>
>
> Kill spark job when InterruptedException happens or driverContext.isShutdown
> is true. If the InterruptedException happened in RemoteSparkJobMonitor and
> LocalSparkJobMonitor, it will be better to kill the job. Also there is a race
> condition between submit the spark job and query/operation cancellation, it
> will be better to check driverContext.isShutdown right after submit the spark
> job. This will guarantee the job being killed no matter when shutdown is
> called. It is similar as HIVE-15997.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[ https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15997060#comment-15997060 ] Xuefu Zhang commented on HIVE-16456: [~zxu], any thoughts on [~lirui]'s question above? Also, the patch probably doesn't apply after HIVE-16552. You might need to rebase. Thanks. > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. > - > > Key: HIVE-16456 > URL: https://issues.apache.org/jira/browse/HIVE-16456 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-16456.000.patch > > > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. If the InterruptedException happened in RemoteSparkJobMonitor and > LocalSparkJobMonitor, it will be better to kill the job. Also there is a race > condition between submit the spark job and query/operation cancellation, it > will be better to check driverContext.isShutdown right after submit the spark > job. This will guarantee the job being killed no matter when shutdown is > called. It is similar as HIVE-15997. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[ https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15997347#comment-15997347 ] zhihai xu commented on HIVE-16456: -- Thanks [~xuefuz] and [~lirui]! Yes, Rui's suggestion is good, I uploaded a new patch HIVE-16456.001.patch which addressed rui's comment. Please review it! thanks > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. > - > > Key: HIVE-16456 > URL: https://issues.apache.org/jira/browse/HIVE-16456 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-16456.000.patch, HIVE-16456.001.patch > > > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. If the InterruptedException happened in RemoteSparkJobMonitor and > LocalSparkJobMonitor, it will be better to kill the job. Also there is a race > condition between submit the spark job and query/operation cancellation, it > will be better to check driverContext.isShutdown right after submit the spark > job. This will guarantee the job being killed no matter when shutdown is > called. It is similar as HIVE-15997. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[ https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15997349#comment-15997349 ] zhihai xu commented on HIVE-16456: -- I also updated the new patch at the review board: https://reviews.apache.org/r/58856/diff/1-2/ > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. > - > > Key: HIVE-16456 > URL: https://issues.apache.org/jira/browse/HIVE-16456 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-16456.000.patch, HIVE-16456.001.patch > > > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. If the InterruptedException happened in RemoteSparkJobMonitor and > LocalSparkJobMonitor, it will be better to kill the job. Also there is a race > condition between submit the spark job and query/operation cancellation, it > will be better to check driverContext.isShutdown right after submit the spark > job. This will guarantee the job being killed no matter when shutdown is > called. It is similar as HIVE-15997. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[ https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998027#comment-15998027 ] Rui Li commented on HIVE-16456: --- +1 pending test > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. > - > > Key: HIVE-16456 > URL: https://issues.apache.org/jira/browse/HIVE-16456 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-16456.000.patch, HIVE-16456.001.patch > > > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. If the InterruptedException happened in RemoteSparkJobMonitor and > LocalSparkJobMonitor, it will be better to kill the job. Also there is a race > condition between submit the spark job and query/operation cancellation, it > will be better to check driverContext.isShutdown right after submit the spark > job. This will guarantee the job being killed no matter when shutdown is > called. It is similar as HIVE-15997. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[ https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998910#comment-15998910 ] Xuefu Zhang commented on HIVE-16456: [~zxu], I had one comment on RB. Thanks. > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. > - > > Key: HIVE-16456 > URL: https://issues.apache.org/jira/browse/HIVE-16456 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-16456.000.patch, HIVE-16456.001.patch > > > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. If the InterruptedException happened in RemoteSparkJobMonitor and > LocalSparkJobMonitor, it will be better to kill the job. Also there is a race > condition between submit the spark job and query/operation cancellation, it > will be better to check driverContext.isShutdown right after submit the spark > job. This will guarantee the job being killed no matter when shutdown is > called. It is similar as HIVE-15997. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[
https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999003#comment-15999003
]
Hive QA commented on HIVE-16456:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866483/HIVE-16456.001.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10652 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30]
(batchId=148)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5053/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5053/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5053/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12866483 - PreCommit-HIVE-Build
> Kill spark job when InterruptedException happens or driverContext.isShutdown
> is true.
> -
>
> Key: HIVE-16456
> URL: https://issues.apache.org/jira/browse/HIVE-16456
> Project: Hive
> Issue Type: Improvement
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-16456.000.patch, HIVE-16456.001.patch
>
>
> Kill spark job when InterruptedException happens or driverContext.isShutdown
> is true. If the InterruptedException happened in RemoteSparkJobMonitor and
> LocalSparkJobMonitor, it will be better to kill the job. Also there is a race
> condition between submit the spark job and query/operation cancellation, it
> will be better to check driverContext.isShutdown right after submit the spark
> job. This will guarantee the job being killed no matter when shutdown is
> called. It is similar as HIVE-15997.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[ https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999318#comment-15999318 ] zhihai xu commented on HIVE-16456: -- thanks [~xuefuz] for the review, I replied your comment on RB. > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. > - > > Key: HIVE-16456 > URL: https://issues.apache.org/jira/browse/HIVE-16456 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-16456.000.patch, HIVE-16456.001.patch > > > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. If the InterruptedException happened in RemoteSparkJobMonitor and > LocalSparkJobMonitor, it will be better to kill the job. Also there is a race > condition between submit the spark job and query/operation cancellation, it > will be better to check driverContext.isShutdown right after submit the spark > job. This will guarantee the job being killed no matter when shutdown is > called. It is similar as HIVE-15997. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[ https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1695#comment-1695 ] Xuefu Zhang commented on HIVE-16456: [~zxu], just as a FYI, I don't see your response on review board. > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. > - > > Key: HIVE-16456 > URL: https://issues.apache.org/jira/browse/HIVE-16456 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-16456.000.patch, HIVE-16456.001.patch > > > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. If the InterruptedException happened in RemoteSparkJobMonitor and > LocalSparkJobMonitor, it will be better to kill the job. Also there is a race > condition between submit the spark job and query/operation cancellation, it > will be better to check driverContext.isShutdown right after submit the spark > job. This will guarantee the job being killed no matter when shutdown is > called. It is similar as HIVE-15997. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[ https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000289#comment-16000289 ] zhihai xu commented on HIVE-16456: -- [~xuefuz]], sorry, I forget to publish my comment, I just published it. If you can't see it, please let me know. > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. > - > > Key: HIVE-16456 > URL: https://issues.apache.org/jira/browse/HIVE-16456 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-16456.000.patch, HIVE-16456.001.patch > > > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. If the InterruptedException happened in RemoteSparkJobMonitor and > LocalSparkJobMonitor, it will be better to kill the job. Also there is a race > condition between submit the spark job and query/operation cancellation, it > will be better to check driverContext.isShutdown right after submit the spark > job. This will guarantee the job being killed no matter when shutdown is > called. It is similar as HIVE-15997. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[ https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16001868#comment-16001868 ] Xuefu Zhang commented on HIVE-16456: [~zxu] See it now. Thanks for the explanation. +1 from me also. > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. > - > > Key: HIVE-16456 > URL: https://issues.apache.org/jira/browse/HIVE-16456 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-16456.000.patch, HIVE-16456.001.patch > > > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. If the InterruptedException happened in RemoteSparkJobMonitor and > LocalSparkJobMonitor, it will be better to kill the job. Also there is a race > condition between submit the spark job and query/operation cancellation, it > will be better to check driverContext.isShutdown right after submit the spark > job. This will guarantee the job being killed no matter when shutdown is > called. It is similar as HIVE-15997. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[
https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15969698#comment-15969698
]
Hive QA commented on HIVE-16456:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863512/HIVE-16456.000.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10578 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_order_null]
(batchId=27)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4701/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4701/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4701/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863512 - PreCommit-HIVE-Build
> Kill spark job when InterruptedException happens or driverContext.isShutdown
> is true.
> -
>
> Key: HIVE-16456
> URL: https://issues.apache.org/jira/browse/HIVE-16456
> Project: Hive
> Issue Type: Improvement
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-16456.000.patch
>
>
> Kill spark job when InterruptedException happens or driverContext.isShutdown
> is true. If the InterruptedException happened in RemoteSparkJobMonitor and
> LocalSparkJobMonitor, it will be better to kill the job. Also there is a race
> condition between submit the spark job and query/operation cancellation, it
> will be better to check driverContext.isShutdown right after submit the spark
> job. This will guarantee the job being killed no matter when shutdown is
> called. It is similar as HIVE-15997.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[ https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987607#comment-15987607 ] Xuefu Zhang commented on HIVE-16456: Hi [~zxu], Thanks for working on this. Could you please provide a RB link for your patch? Thanks. [~lirui], please review as well. Thanks. > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. > - > > Key: HIVE-16456 > URL: https://issues.apache.org/jira/browse/HIVE-16456 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-16456.000.patch > > > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. If the InterruptedException happened in RemoteSparkJobMonitor and > LocalSparkJobMonitor, it will be better to kill the job. Also there is a race > condition between submit the spark job and query/operation cancellation, it > will be better to check driverContext.isShutdown right after submit the spark > job. This will guarantee the job being killed no matter when shutdown is > called. It is similar as HIVE-15997. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
