[jira] [Commented] (HIVE-10249) ACID: show locks should show who the lock is waiting for
[ https://issues.apache.org/jira/browse/HIVE-10249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15237636#comment-15237636 ] Wei Zheng commented on HIVE-10249: -- HIVE-11793 will include the fix for the missing column issue > ACID: show locks should show who the lock is waiting for > > > Key: HIVE-10249 > URL: https://issues.apache.org/jira/browse/HIVE-10249 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-10249.2.patch, HIVE-10249.3.patch, HIVE-10249.patch > > > instead of just showing state WAITING, we should include what the lock is > waiting for. It will make diagnostics easier. > It would also be useful to add QueryPlan.getQueryId() so it's easy to see > which query the lock belongs to. > # need to store this in HIVE_LOCKS (additional field); this has a perf hit to > do another update on failed attempt and to clear filed on successful attempt. > (Actually on success, we update anyway). How exactly would this be > displayed? Each lock can block but we acquire all parts of external lock at > once. Since we stop at first one that blocked, we’d only update that one… > # This needs a matching Thrift change to pass to client: ShowLocksResponse > # Perhaps we can start updating this info after lock was in W state for some > time to reduce perf hit. > # This is mostly useful for “Why is my query stuck” -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10249) ACID: show locks should show who the lock is waiting for
[ https://issues.apache.org/jira/browse/HIVE-10249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15228792#comment-15228792 ] Wei Zheng commented on HIVE-10249: -- [~ekoifman] I noticed in patch 3, dbtxnmgr_showlocks.q.out missed the last column Hostname.. > ACID: show locks should show who the lock is waiting for > > > Key: HIVE-10249 > URL: https://issues.apache.org/jira/browse/HIVE-10249 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-10249.2.patch, HIVE-10249.3.patch, HIVE-10249.patch > > > instead of just showing state WAITING, we should include what the lock is > waiting for. It will make diagnostics easier. > It would also be useful to add QueryPlan.getQueryId() so it's easy to see > which query the lock belongs to. > # need to store this in HIVE_LOCKS (additional field); this has a perf hit to > do another update on failed attempt and to clear filed on successful attempt. > (Actually on success, we update anyway). How exactly would this be > displayed? Each lock can block but we acquire all parts of external lock at > once. Since we stop at first one that blocked, we’d only update that one… > # This needs a matching Thrift change to pass to client: ShowLocksResponse > # Perhaps we can start updating this info after lock was in W state for some > time to reduce perf hit. > # This is mostly useful for “Why is my query stuck” -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10249) ACID: show locks should show who the lock is waiting for
[
https://issues.apache.org/jira/browse/HIVE-10249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217833#comment-15217833
]
Hive QA commented on HIVE-10249:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12795686/HIVE-10249.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 9891 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-parallel_join0.q-union_remove_9.q-smb_mapjoin_21.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7410/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7410/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7410/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12795686 - PreCommit-HIVE-TRUNK-Build
> ACID: show locks should show who the lock is waiting for
>
>
> Key: HIVE-10249
> URL: https://issues.apache.org/jira/browse/HIVE-10249
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-10249.2.patch, HIVE-10249.3.patch, HIVE-10249.patch
>
>
> instead of just showing state WAITING, we should include what the lock is
> waiting for. It will make diagnostics easier.
> It would also be useful to add QueryPlan.getQueryId() so it's easy to see
> which query the lock belongs to.
> # need to store this in HIVE_LOCKS (additional field); this has a perf hit to
> do another update on failed attempt and to clear filed on successful attempt.
> (Actually on success, we update anyway). How exactly would this be
> displayed? Each lock can block but we acquire all parts of external lock at
> once. Since we stop at first one that blocked, we’d only update that one…
> # This needs a matching Thrift change to pass to client: ShowLocksResponse
> # Perhaps we can start updating this info after lock was in W state for some
> time to reduce perf hit.
> # This is mostly useful for “Why is my query stuck”
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10249) ACID: show locks should show who the lock is waiting for
[
https://issues.apache.org/jira/browse/HIVE-10249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211761#comment-15211761
]
Hive QA commented on HIVE-10249:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12795138/HIVE-10249.2.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9864 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dbtxnmgr_showlocks
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7363/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7363/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7363/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12795138 - PreCommit-HIVE-TRUNK-Build
> ACID: show locks should show who the lock is waiting for
>
>
> Key: HIVE-10249
> URL: https://issues.apache.org/jira/browse/HIVE-10249
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-10249.2.patch, HIVE-10249.patch
>
>
> instead of just showing state WAITING, we should include what the lock is
> waiting for. It will make diagnostics easier.
> It would also be useful to add QueryPlan.getQueryId() so it's easy to see
> which query the lock belongs to.
> # need to store this in HIVE_LOCKS (additional field); this has a perf hit to
> do another update on failed attempt and to clear filed on successful attempt.
> (Actually on success, we update anyway). How exactly would this be
> displayed? Each lock can block but we acquire all parts of external lock at
> once. Since we stop at first one that blocked, we’d only update that one…
> # This needs a matching Thrift change to pass to client: ShowLocksResponse
> # Perhaps we can start updating this info after lock was in W state for some
> time to reduce perf hit.
> # This is mostly useful for “Why is my query stuck”
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10249) ACID: show locks should show who the lock is waiting for
[ https://issues.apache.org/jira/browse/HIVE-10249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209791#comment-15209791 ] Wei Zheng commented on HIVE-10249: -- Thanks Eugene. +1 pending tests. > ACID: show locks should show who the lock is waiting for > > > Key: HIVE-10249 > URL: https://issues.apache.org/jira/browse/HIVE-10249 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-10249.2.patch, HIVE-10249.patch > > > instead of just showing state WAITING, we should include what the lock is > waiting for. It will make diagnostics easier. > It would also be useful to add QueryPlan.getQueryId() so it's easy to see > which query the lock belongs to. > # need to store this in HIVE_LOCKS (additional field); this has a perf hit to > do another update on failed attempt and to clear filed on successful attempt. > (Actually on success, we update anyway). How exactly would this be > displayed? Each lock can block but we acquire all parts of external lock at > once. Since we stop at first one that blocked, we’d only update that one… > # This needs a matching Thrift change to pass to client: ShowLocksResponse > # Perhaps we can start updating this info after lock was in W state for some > time to reduce perf hit. > # This is mostly useful for “Why is my query stuck” -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10249) ACID: show locks should show who the lock is waiting for
[
https://issues.apache.org/jira/browse/HIVE-10249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209325#comment-15209325
]
Wei Zheng commented on HIVE-10249:
--
In TxnHandler.checkLock, there's a typo {code}shouldNeverHappen(info.txnId,
info.extLockId, info.extLockId);{code} where the 3rd param should be intLockId.
> ACID: show locks should show who the lock is waiting for
>
>
> Key: HIVE-10249
> URL: https://issues.apache.org/jira/browse/HIVE-10249
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-10249.patch
>
>
> instead of just showing state WAITING, we should include what the lock is
> waiting for. It will make diagnostics easier.
> It would also be useful to add QueryPlan.getQueryId() so it's easy to see
> which query the lock belongs to.
> # need to store this in HIVE_LOCKS (additional field); this has a perf hit to
> do another update on failed attempt and to clear filed on successful attempt.
> (Actually on success, we update anyway). How exactly would this be
> displayed? Each lock can block but we acquire all parts of external lock at
> once. Since we stop at first one that blocked, we’d only update that one…
> # This needs a matching Thrift change to pass to client: ShowLocksResponse
> # Perhaps we can start updating this info after lock was in W state for some
> time to reduce perf hit.
> # This is mostly useful for “Why is my query stuck”
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10249) ACID: show locks should show who the lock is waiting for
[ https://issues.apache.org/jira/browse/HIVE-10249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208827#comment-15208827 ] Eugene Koifman commented on HIVE-10249: --- [~wzheng] could you review please > ACID: show locks should show who the lock is waiting for > > > Key: HIVE-10249 > URL: https://issues.apache.org/jira/browse/HIVE-10249 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-10249.patch > > > instead of just showing state WAITING, we should include what the lock is > waiting for. It will make diagnostics easier. > It would also be useful to add QueryPlan.getQueryId() so it's easy to see > which query the lock belongs to. > # need to store this in HIVE_LOCKS (additional field); this has a perf hit to > do another update on failed attempt and to clear filed on successful attempt. > (Actually on success, we update anyway). How exactly would this be > displayed? Each lock can block but we acquire all parts of external lock at > once. Since we stop at first one that blocked, we’d only update that one… > # This needs a matching Thrift change to pass to client: ShowLocksResponse > # Perhaps we can start updating this info after lock was in W state for some > time to reduce perf hit. > # This is mostly useful for “Why is my query stuck” -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10249) ACID: show locks should show who the lock is waiting for
[ https://issues.apache.org/jira/browse/HIVE-10249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15087875#comment-15087875 ] Alan Gates commented on HIVE-10249: --- I don't think the perf hit is that bad. When we first discover that the lock has to wait we have to do an update to indicate what it has to wait on. We only have to change the record again if the blocking record changes. > ACID: show locks should show who the lock is waiting for > > > Key: HIVE-10249 > URL: https://issues.apache.org/jira/browse/HIVE-10249 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > instead of just showing state WAITING, we should include what the lock is > waiting for. It will make diagnostics easier. > It would also be useful to add QueryPlan.getQueryId() so it's easy to see > which query the lock belongs to. > # need to store this in HIVE_LOCKS (additional field); this has a perf hit to > do another update on failed attempt and to clear filed on successful attempt. > (Actually on success, we update anyway). How exactly would this be > displayed? Each lock can block but we acquire all parts of external lock at > once. Since we stop at first one that blocked, we’d only update that one… > # This needs a matching Thrift change to pass to client: ShowLocksResponse > # Perhaps we can start updating this info after lock was in W state for some > time to reduce perf hit. > # This is mostly useful for “Why is my query stuck” -- This message was sent by Atlassian JIRA (v6.3.4#6332)
