[jira] [Commented] (HIVE-13959) MoveTask should only release its query associated locks
[ https://issues.apache.org/jira/browse/HIVE-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327753#comment-15327753 ] Yongzhi Chen commented on HIVE-13959: - After talked with [~ctang.ma], getLocks return all the locks related to the path. It seems a consistent behavior in EmbeddedLockManager and ZooKeeperHiveLockManager. So the first patch is a good fix. +1 for the fix HIVE-13959.patch > MoveTask should only release its query associated locks > --- > > Key: HIVE-13959 > URL: https://issues.apache.org/jira/browse/HIVE-13959 > Project: Hive > Issue Type: Bug > Components: Locking >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-13959.1.patch, HIVE-13959.patch, HIVE-13959.patch > > > releaseLocks in MoveTask releases all locks under a HiveLockObject pathNames. > But some of locks under this pathNames might be for other queries and should > not be released. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13959) MoveTask should only release its query associated locks
[ https://issues.apache.org/jira/browse/HIVE-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327551#comment-15327551 ] Chaoyu Tang commented on HIVE-13959: The failed tests seem not related to the patch. > MoveTask should only release its query associated locks > --- > > Key: HIVE-13959 > URL: https://issues.apache.org/jira/browse/HIVE-13959 > Project: Hive > Issue Type: Bug > Components: Locking >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-13959.1.patch, HIVE-13959.patch, HIVE-13959.patch > > > releaseLocks in MoveTask releases all locks under a HiveLockObject pathNames. > But some of locks under this pathNames might be for other queries and should > not be released. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13959) MoveTask should only release its query associated locks
[
https://issues.apache.org/jira/browse/HIVE-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327506#comment-15327506
]
Hive QA commented on HIVE-13959:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12809813/HIVE-13959.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10224 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/106/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/106/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-106/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12809813 - PreCommit-HIVE-MASTER-Build
> MoveTask should only release its query associated locks
> ---
>
> Key: HIVE-13959
> URL: https://issues.apache.org/jira/browse/HIVE-13959
> Project: Hive
> Issue Type: Bug
> Components: Locking
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-13959.1.patch, HIVE-13959.patch, HIVE-13959.patch
>
>
> releaseLocks in MoveTask releases all locks under a HiveLockObject pathNames.
> But some of locks under this pathNames might be for other queries and should
> not be released.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13959) MoveTask should only release its query associated locks
[ https://issues.apache.org/jira/browse/HIVE-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15320673#comment-15320673 ] Yongzhi Chen commented on HIVE-13959: - It seems that the ZookeeperHiveLockManager may have some issue, it does not store and use HiveLockObjectData See EmbeddedLockManager.java https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/EmbeddedLockManager.java#L403 > MoveTask should only release its query associated locks > --- > > Key: HIVE-13959 > URL: https://issues.apache.org/jira/browse/HIVE-13959 > Project: Hive > Issue Type: Bug > Components: Locking >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-13959.patch, HIVE-13959.patch > > > releaseLocks in MoveTask releases all locks under a HiveLockObject pathNames. > But some of locks under this pathNames might be for other queries and should > not be released. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13959) MoveTask should only release its query associated locks
[
https://issues.apache.org/jira/browse/HIVE-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15319761#comment-15319761
]
Chaoyu Tang commented on HIVE-13959:
[~ychena] Thanks for the review. For your analysis and questions, please see
below:
Yes -- one WriteEntity map to one List
Yes -- These list of HiveLockObj are all created during acquireLocks related to
the query.
Yes -- In the releaseLocks code, lockObj.getObj() return HiveLockObject
The problem is here: List locks = lockMgr.getLocks(lockObj.getObj(),
false, true); it returns all locks under the pathName which might not related
to this MoveTask query:
{code}
The getLocks method in ZookeeperHiveLockManager:
private static List getLocks(HiveConf conf,
HiveLockObject key, String parent, boolean verifyTablePartition, boolean
fetchData)
throws LockException {
List locks = new ArrayList();
List children;
boolean recurse = true;
String commonParent;
try {
if (key != null) {
commonParent = "/" + parent + "/" + key.getName();
children = curatorFramework.getChildren().forPath(commonParent);
/* ==> this call returns all locks under commonParent, say
db/cdhpart/LOCK-SHARE-00, db/cdhpart/LOCK-SHARE-01 for pathNames
db/cdhpart */
recurse = false;
}
else {
commonParent = "/" + parent;
children = curatorFramework.getChildren().forPath(commonParent);
}
} catch (Exception e) {
// no locks present
return locks;
}
{code}
For an example, if we run query1 in one session "insert overwrite table cdhpart
partition (level1= 'l1', level2='l2', level3 = 'l3', level4) select key, value,
level4 from cdhsrc;" and query2 in the other session concurrently "select *
from cdhpart where level1 = 'l1'"
query1 and query2 both have its own znode (lock) under pathNames (db/cdhpart/)
say LOCK-SHARE-00, LOCK-SHARE-01 respectively, the getLocks for
HiveLockObject key with its getName() value db/cdhpart/ will return both
LOCK-SHARE-00, LOCK-SHARE-01. But LOCK-SHARE-01 is not in the
ctx.getHiveLocks(), the lock list for the query1, so
ctx.getHiveLocks().remove() returns false because the HiveLockObjectData.equals
always return false due to the different queryStr/queryId, therefore
lockMgr.unlock(lock) should not be called to unlock the LOCK-SHARE-01 for
query2.
> MoveTask should only release its query associated locks
> ---
>
> Key: HIVE-13959
> URL: https://issues.apache.org/jira/browse/HIVE-13959
> Project: Hive
> Issue Type: Bug
> Components: Locking
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-13959.patch, HIVE-13959.patch
>
>
> releaseLocks in MoveTask releases all locks under a HiveLockObject pathNames.
> But some of locks under this pathNames might be for other queries and should
> not be released.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13959) MoveTask should only release its query associated locks
[
https://issues.apache.org/jira/browse/HIVE-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15319545#comment-15319545
]
Hive QA commented on HIVE-13959:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12808724/HIVE-13959.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10223 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_table_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_func1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testPermFunc
org.apache.hive.jdbc.TestJdbcWithMiniMr.testPermFunc
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/36/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/36/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-36/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12808724 - PreCommit-HIVE-MASTER-Build
> MoveTask should only release its query associated locks
> ---
>
> Key: HIVE-13959
> URL: https://issues.apache.org/jira/browse/HIVE-13959
> Project: Hive
> Issue Type: Bug
> Components: Locking
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-13959.patch, HIVE-13959.patch
>
>
> releaseLocks in MoveTask releases all locks under a HiveLockObject pathNames.
> But some of locks under this pathNames might be for other queries and should
> not be released.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13959) MoveTask should only release its query associated locks
[ https://issues.apache.org/jira/browse/HIVE-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15319241#comment-15319241 ] Yongzhi Chen commented on HIVE-13959: - [~ctang.ma] I do not quite understand how MoveTask releases all locks under a HiveLockObject pathNames. It looks for me as following: One LoadTableDesc object map to one WriteEntity; one WriteEntity map to one List. These list of HiveLockObj are all created during acquireLocks related to the query. In the releaseLocks code, lockObj.getObj() return HiveLockObject List locks = lockMgr.getLocks(lockObj.getObj(), false, true); HiveLockObject equals method need both pathNames are same and HiveLockObjectData are same. HiveLockObjectData equals method need query_id, lock_mode, lock_time... are all same. So I think the locks return by getLocks should all related to the query. Therefore, MoveTask has no chance to release other query's locks. Do I miss something? > MoveTask should only release its query associated locks > --- > > Key: HIVE-13959 > URL: https://issues.apache.org/jira/browse/HIVE-13959 > Project: Hive > Issue Type: Bug > Components: Locking >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-13959.patch, HIVE-13959.patch > > > releaseLocks in MoveTask releases all locks under a HiveLockObject pathNames. > But some of locks under this pathNames might be for other queries and should > not be released. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13959) MoveTask should only release its query associated locks
[ https://issues.apache.org/jira/browse/HIVE-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15319023#comment-15319023 ] Chaoyu Tang commented on HIVE-13959: The TestSparkCliDriver test failures seem not related to this patch, they might be due to infra issue (java.lang.IllegalStateException: Error trying to obtain executor info: java.lang.IllegalStateException: RPC channel is closed.). All other failures are aged and also not related to this patch. Kick off another run of precommit build test. > MoveTask should only release its query associated locks > --- > > Key: HIVE-13959 > URL: https://issues.apache.org/jira/browse/HIVE-13959 > Project: Hive > Issue Type: Bug > Components: Locking >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-13959.patch > > > releaseLocks in MoveTask releases all locks under a HiveLockObject pathNames. > But some of locks under this pathNames might be for other queries and should > not be released. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13959) MoveTask should only release its query associated locks
[
https://issues.apache.org/jira/browse/HIVE-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15318990#comment-15318990
]
Hive QA commented on HIVE-13959:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12808546/HIVE-13959.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 10223 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_table_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_func1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby4_map_skew
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_bigdata
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_resolution
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_sort_1_23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_literals
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_merging
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapreduce1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_skewjoinopt2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_statsfs
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_19
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_view
org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testPermFunc
org.apache.hive.jdbc.TestJdbcWithMiniMr.testPermFunc
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/31/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/31/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-31/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12808546 - PreCommit-HIVE-MASTER-Build
> MoveTask should only release its query associated locks
> ---
>
> Key: HIVE-13959
> URL: https://issues.apache.org/jira/browse/HIVE-13959
> Project: Hive
> Issue Type: Bug
> Components: Locking
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-13959.patch
>
>
> releaseLocks in MoveTask releases all locks under a HiveLockObject pathNames.
> But some of locks under this pathNames might be for other queries and should
> not be released.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
