[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[ https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553072#comment-16553072 ] Eugene Koifman commented on HIVE-17683: --- +1 > Annotate Query Plan with locking information > > > Key: HIVE-17683 > URL: https://issues.apache.org/jira/browse/HIVE-17683 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Igor Kryvenko >Priority: Critical > Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch, > HIVE-17683.03.patch, HIVE-17683.04.patch, HIVE-17683.05.patch, > HIVE-17683.06.patch > > > Explore if it's possible to add info about what locks will be asked for to > the query plan. > Lock acquisition (for Acid Lock Manager) is done in > DbTxnManager.acquireLocks() which is called once the query starts running. > Would need to refactor that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551600#comment-16551600
]
Igor Kryvenko commented on HIVE-17683:
--
[~ekoifman] Hi, Eugene. Yeah, {{getLocks}} had the bug, but I've already fixed
it in the latest patch.
Please, review it, when you will have time for it.
Thanks, Igor.
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch,
> HIVE-17683.03.patch, HIVE-17683.04.patch, HIVE-17683.05.patch,
> HIVE-17683.06.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551504#comment-16551504
]
Hive QA commented on HIVE-17683:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932403/HIVE-17683.06.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14681 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12748/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12748/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12748/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932403 - PreCommit-HIVE-Build
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch,
> HIVE-17683.03.patch, HIVE-17683.04.patch, HIVE-17683.05.patch,
> HIVE-17683.06.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551495#comment-16551495
]
Hive QA commented on HIVE-17683:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
52s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 5 new + 277 unchanged - 35
fixed = 282 total (was 312) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12748/dev-support/hive-personality.sh
|
| git revision | master / e569ef0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12748/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12748/yetus/patch-mvninstall-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12748/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12748/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: ql standalone-metastore/metastore-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12748/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch,
> HIVE-17683.03.patch, HIVE-17683.04.patch, HIVE-17683.05.patch,
> HIVE-17683.06.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550296#comment-16550296
]
Hive QA commented on HIVE-17683:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932262/HIVE-17683.05.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12719/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12719/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12719/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-07-20 06:34:56.177
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-12719/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-07-20 06:34:56.180
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 851c8ab HIVE-17896: TopNKey: Create a standalone vectorizable
TopNKey operator (Teddy Choi, reviewed by Jesus Camacho Rodriguez)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 851c8ab HIVE-17896: TopNKey: Create a standalone vectorizable
TopNKey operator (Teddy Choi, reviewed by Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-07-20 06:34:57.158
+ rm -rf ../yetus_PreCommit-HIVE-Build-12719
+ mkdir ../yetus_PreCommit-HIVE-Build-12719
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-12719
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-12719/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockComponentBuilder.java:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java:
does not exist in index
error: src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java: does not exist
in index
error: src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java: does not exist in
index
error: src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java: does not
exist in index
error: src/java/org/apache/hadoop/hive/ql/metadata/Table.java: does not exist
in index
error: src/java/org/apache/hadoop/hive/ql/parse/ExplainConfiguration.java: does
not exist in index
error: src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java:
does not exist in index
error: src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g: does not exist in
index
error: src/java/org/apache/hadoop/hive/ql/plan/ExplainWork.java: does not exist
in index
error:
metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockComponentBuilder.java:
does not exist in index
error:
metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java:
does not exist in index
error: java/org/apache/hadoop/hive/ql/exec/ExplainTask.java: does not exist in
index
error: java/org/apache/hadoop/hive/ql/io/AcidUtils.java: does not exist in index
error: java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java: does not exist
in index
error: java/org/apache/hadoop/hive/ql/metadata/Table.java: does not exist in
index
error: java/org/apache/hadoop/hive/ql/parse/ExplainConfiguration.java: does not
exist in index
error: java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java: does
not exist in index
error: java/org/apache/hadoop/hive/ql/parse/HiveParser.g: does not exist in
index
error:
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548382#comment-16548382
]
Eugene Koifman commented on HIVE-17683:
---
[~ikryvenko],
this looks good in general. One question:
Did you mean to add support for 'formatted' option? In patch 4, adding
'formatted' produces
{noformat}
{"LOCK INFORMATION:":"[]"}
{noformat}
{{ExplainTask.getLocks()}} has a bug
I think, {[explain locks drop table test_explain_locks}} produces
Read/WriteEntity because this table doesn't exist at the time this command runs.
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch,
> HIVE-17683.03.patch, HIVE-17683.04.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[ https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546710#comment-16546710 ] Igor Kryvenko commented on HIVE-17683: -- [~ekoifman] Hi Eugene. I did changes which you requested in your review, please review it when you will have time for it. Seems like failing of tests is not related again. Thanks, Igor. > Annotate Query Plan with locking information > > > Key: HIVE-17683 > URL: https://issues.apache.org/jira/browse/HIVE-17683 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Igor Kryvenko >Priority: Critical > Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch, > HIVE-17683.03.patch, HIVE-17683.04.patch > > > Explore if it's possible to add info about what locks will be asked for to > the query plan. > Lock acquisition (for Acid Lock Manager) is done in > DbTxnManager.acquireLocks() which is called once the query starts running. > Would need to refactor that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546665#comment-16546665
]
Hive QA commented on HIVE-17683:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931907/HIVE-17683.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14648 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=106)
[update_orig_table.q,explainanalyze_3.q,orc_vectorization_ppd.q,vector_non_string_partition.q,multi_count_distinct.q,explainanalyze_1.q,explainuser_3.q,explainanalyze_4.q,hybridgrace_hashjoin_1.q,delete_orig_table.q,hybridgrace_hashjoin_2.q,tez-tag.q,orc_merge12.q,explainanalyze_5.q,vector_join_part_col_char.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12654/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12654/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12654/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12931907 - PreCommit-HIVE-Build
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch,
> HIVE-17683.03.patch, HIVE-17683.04.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546588#comment-16546588
]
Hive QA commented on HIVE-17683:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
17s{color} | {color:blue} standalone-metastore/metastore-common in master has
218 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
25s{color} | {color:blue} ql in master has 2273 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 5 new + 276 unchanged - 35
fixed = 281 total (was 311) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12654/dev-support/hive-personality.sh
|
| git revision | master / dceeefb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12654/yetus/diff-checkstyle-ql.txt
|
| modules | C: standalone-metastore/metastore-common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12654/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch,
> HIVE-17683.03.patch, HIVE-17683.04.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546207#comment-16546207
]
Igor Kryvenko commented on HIVE-17683:
--
Seems like failure is not related
{code}
{code}
I will attach the same patch to check this.
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch,
> HIVE-17683.03.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545993#comment-16545993
]
Hive QA commented on HIVE-17683:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931818/HIVE-17683.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14611 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.beeline.TestBeeLineWithArgs.org.apache.hive.beeline.TestBeeLineWithArgs
(batchId=244)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12647/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12647/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12647/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12931818 - PreCommit-HIVE-Build
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch,
> HIVE-17683.03.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545983#comment-16545983
]
Hive QA commented on HIVE-17683:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
5s{color} | {color:blue} standalone-metastore/metastore-common in master has
218 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 2273 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 5 new + 276 unchanged - 35
fixed = 281 total (was 311) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12647/dev-support/hive-personality.sh
|
| git revision | master / 85a3dd7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12647/yetus/diff-checkstyle-ql.txt
|
| modules | C: standalone-metastore/metastore-common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12647/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch,
> HIVE-17683.03.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544622#comment-16544622
]
Igor Kryvenko commented on HIVE-17683:
--
[~ekoifman] Hi, Eugene.
Yeah, I can refactor it.
Actually, I was a little bit confused too why {{drop table}} doesn't acquire
any locks, but it really produces no {{readEntity}} and {{writeEntity}}.
Also, I will add switching between JSON and default output in the following
patch.
Thanks for the review.
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542380#comment-16542380
]
Eugene Koifman commented on HIVE-17683:
---
[~ikryvenko], sorry, it took a while to get back to this.
Your implementation creates ExplainTask.getJsonLocks() which duplicates a lot
of the logic in DbTxnManger.acquireLocks(). This is problematic because they
have to be kept in sync.
Could you refactor it so that they share code?
For example, create a {{LockRequest makeLockRequest(List,
List)}} and use it in both places?
Also, the refactoring in acquireLocks() lost
{noformat}
default:
throw new IllegalArgumentException(String
.format("Lock type [%s] for Database.Table [%s.%s] is unknown", lockType,
t.getDbName(),
t.getTableName()
));{noformat}
This may change how errors are surfaced - not sure it's a good idea.
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[ https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516021#comment-16516021 ] Eugene Koifman commented on HIVE-17683: --- sorry for the delay. Will try get to this (and your other patch) next week. > Annotate Query Plan with locking information > > > Key: HIVE-17683 > URL: https://issues.apache.org/jira/browse/HIVE-17683 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Igor Kryvenko >Priority: Critical > Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch > > > Explore if it's possible to add info about what locks will be asked for to > the query plan. > Lock acquisition (for Acid Lock Manager) is done in > DbTxnManager.acquireLocks() which is called once the query starts running. > Would need to refactor that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[ https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505144#comment-16505144 ] Igor Kryvenko commented on HIVE-17683: -- I do the same. I just copied this command from build's logs. > Annotate Query Plan with locking information > > > Key: HIVE-17683 > URL: https://issues.apache.org/jira/browse/HIVE-17683 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Igor Kryvenko >Priority: Critical > Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch > > > Explore if it's possible to add info about what locks will be asked for to > the query plan. > Lock acquisition (for Acid Lock Manager) is done in > DbTxnManager.acquireLocks() which is called once the query starts running. > Would need to refactor that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505142#comment-16505142
]
Eugene Koifman commented on HIVE-17683:
---
I usually run {{mvn clean install -DskipTests}} from hive root and then the
same from itests/.
I don't know what itests profile does
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[ https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505141#comment-16505141 ] Igor Kryvenko commented on HIVE-17683: -- Have I need to do something with it, or just wait for your review? Anyway, thanks for the help. > Annotate Query Plan with locking information > > > Key: HIVE-17683 > URL: https://issues.apache.org/jira/browse/HIVE-17683 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Igor Kryvenko >Priority: Critical > Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch > > > Explore if it's possible to add info about what locks will be asked for to > the query plan. > Lock acquisition (for Acid Lock Manager) is done in > DbTxnManager.acquireLocks() which is called once the query starts running. > Would need to refactor that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[ https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505140#comment-16505140 ] Eugene Koifman commented on HIVE-17683: --- This is very strange. The build bot ran and passed all tests based on https://issues.apache.org/jira/browse/HIVE-17683?focusedCommentId=16505064=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16505064. In any case, I've not had a chance to read the patch yet. > Annotate Query Plan with locking information > > > Key: HIVE-17683 > URL: https://issues.apache.org/jira/browse/HIVE-17683 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Igor Kryvenko >Priority: Critical > Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch > > > Explore if it's possible to add info about what locks will be asked for to > the query plan. > Lock acquisition (for Acid Lock Manager) is done in > DbTxnManager.acquireLocks() which is called once the query starts running. > Would need to refactor that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505134#comment-16505134
]
Igor Kryvenko commented on HIVE-17683:
--
[~ekoifman] Yes, you are right it compiles metastore at first. I just tried
{{mvn -DskipTests -Pitests -fae clean install -DskipTests=true
-Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true}} locally
and it worked fine. Have you any suggestions about the cause of it?
Thanks, Igor.
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505119#comment-16505119
]
Igor Kryvenko commented on HIVE-17683:
--
{code}
[ERROR]
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11593/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java:[521,22]
cannot find symbol
symbol: method setLock(org.apache.hadoop.hive.metastore.api.LockType)
location: variable compBuilder of type
org.apache.hadoop.hive.metastore.LockComponentBuilder
{code}
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[ https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505089#comment-16505089 ] Eugene Koifman commented on HIVE-17683: --- the build/test of patch 2 passed based on the above comment. ql module depends on the metastore, so I don't think it's compiled before metastore. > Annotate Query Plan with locking information > > > Key: HIVE-17683 > URL: https://issues.apache.org/jira/browse/HIVE-17683 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Igor Kryvenko >Priority: Critical > Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch > > > Explore if it's possible to add info about what locks will be asked for to > the query plan. > Lock acquisition (for Acid Lock Manager) is done in > DbTxnManager.acquireLocks() which is called once the query starts running. > Would need to refactor that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505079#comment-16505079
]
Igor Kryvenko commented on HIVE-17683:
--
[~ekoifman] Hi. Can you help, why {{mvninstall}} phase failed? As I understand
it compiles {{ql}} module at first then {{metastore}} module, that's why it
can't find new method in {{LockComponent}}. How can I change the order of
building?
Thanks, Igor.
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505064#comment-16505064
]
Hive QA commented on HIVE-17683:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926597/HIVE-17683.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14482 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11593/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11593/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11593/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926597 - PreCommit-HIVE-Build
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505022#comment-16505022
]
Hive QA commented on HIVE-17683:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
27s{color} | {color:blue} ql in master has 2284 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
37s{color} | {color:blue} standalone-metastore in master has 216 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
42s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 4 new + 323 unchanged - 0
fixed = 327 total (was 323) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11593/dev-support/hive-personality.sh
|
| git revision | master / cfd5734 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11593/yetus/patch-mvninstall-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11593/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11593/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501297#comment-16501297
]
Hive QA commented on HIVE-17683:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926112/HIVE-17683.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14463 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_decimal_date]
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_columnPruning]
(batchId=84)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11516/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11516/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11516/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926112 - PreCommit-HIVE-Build
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501283#comment-16501283
]
Hive QA commented on HIVE-17683:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
24s{color} | {color:blue} ql in master has 2277 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
32s{color} | {color:blue} standalone-metastore in master has 214 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
43s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 10 new + 323 unchanged - 0
fixed = 333 total (was 323) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
35s{color} | {color:red} ql generated 1 new + 2277 unchanged - 0 fixed = 2278
total (was 2277) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Null pointer dereference of t in
org.apache.hadoop.hive.ql.exec.ExplainTask.getJsonLocks(ExplainWork)
Dereferenced at ExplainTask.java:in
org.apache.hadoop.hive.ql.exec.ExplainTask.getJsonLocks(ExplainWork)
Dereferenced at ExplainTask.java:[line 345] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11516/dev-support/hive-personality.sh
|
| git revision | master / f567a82 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11516/yetus/patch-mvninstall-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11516/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11516/yetus/whitespace-eol.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11516/yetus/new-findbugs-ql.html
|
| modules | C: ql standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11516/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter:
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[ https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494354#comment-16494354 ] Igor Kryvenko commented on HIVE-17683: -- [~ekoifman] Thanks for the response. Okay, I will start working on "explain locks select a from foo" in order of this JIRA. > Annotate Query Plan with locking information > > > Key: HIVE-17683 > URL: https://issues.apache.org/jira/browse/HIVE-17683 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Igor Kryvenko >Priority: Critical > > Explore if it's possible to add info about what locks will be asked for to > the query plan. > Lock acquisition (for Acid Lock Manager) is done in > DbTxnManager.acquireLocks() which is called once the query starts running. > Would need to refactor that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494227#comment-16494227
]
Eugene Koifman commented on HIVE-17683:
---
I agree that only Sink/Scan need to be annotated, but I think a single
TableScan may read multiple Partitions of a table. So {{select * from T where
p in (1,2)}} where 'p' is a partition column, I think should get 2 Shared
partition locks. So it doesn't quite map 1-1. Also, we don't have this yet,
but I'd like to add in the future Intention locks, where a an S lock on a
partition would cause an IS lock on parent table and DB objects. Also, perhaps
down the road we'll add support for finer grained locking, like locking
individual buckets of a bucketed table - buckets are not represented in the QP
at all.
So I think to keep things flexible and simple, I'd suggest adding support for
"explain locks select a from foo", that will simply dump the locks that will be
acquired as a table with 2 columns (resource name like cat.db.table.part, lock
Type). This would be very useful to users. I'm not even sure we need to
attach this output to the Plan output.
I think HIVE-18948 may be a lot of work.
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491971#comment-16491971
]
Igor Kryvenko commented on HIVE-17683:
--
[~ekoifman] Hi. I investigated this issue. Seems we really need to annotate
only TableScan and FileSink operators, because only them generate write and
read entities. But problem is that we can't add lock information to them
because we acquire locks after setting explain work, and in the {{aquireLocks
method}} have query plan with explain work but not with origin plan, which has
no inputs and outputs, accordingly we can't add any lock information to
existing operators.
So, I see a couple of solution.
1. As you said in the comment above, just use {{show locks}} and add a result
to query plan.
2. Move {{acquireLocks}} method before analyzing query, as mentioned in
HIVE-18948
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[ https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491176#comment-16491176 ] Eugene Koifman commented on HIVE-17683: --- Hmm - very good question. We don't acquire locks on anything that would correspond to an intermediate node in the plan like Join or group by. Lock manager can lock database/table/partition objects but FileSink may write multiple partitions (dynamic partition insert) and a Scan can read many partitions. And I don't know if Database is represented in the QueryPlan at all... I think you need experiment with this a bit and see what locks you get for various query plans to see if there is a reasonable way to represent lock info as part of the plan. It may turn out one doesn't map very well to the other. In that case maybe you should consider writing something like "show locks " or "explain locks " that just dumps LockRequest structure. cc [~alangates] > Annotate Query Plan with locking information > > > Key: HIVE-17683 > URL: https://issues.apache.org/jira/browse/HIVE-17683 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Igor Kryvenko >Priority: Critical > > Explore if it's possible to add info about what locks will be asked for to > the query plan. > Lock acquisition (for Acid Lock Manager) is done in > DbTxnManager.acquireLocks() which is called once the query starts running. > Would need to refactor that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[ https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491150#comment-16491150 ] Igor Kryvenko commented on HIVE-17683: -- [~ekoifman] Thanks for the useful tips. Do we need to annotate only TableScan and FileSink operators? > Annotate Query Plan with locking information > > > Key: HIVE-17683 > URL: https://issues.apache.org/jira/browse/HIVE-17683 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Igor Kryvenko >Priority: Critical > > Explore if it's possible to add info about what locks will be asked for to > the query plan. > Lock acquisition (for Acid Lock Manager) is done in > DbTxnManager.acquireLocks() which is called once the query starts running. > Would need to refactor that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490856#comment-16490856
]
Eugene Koifman commented on HIVE-17683:
---
Great.
Here are a some pointers that may be useful:
Once {{rqstBuilder.build()}} is done at the end of
{{DbTxnManager.acquireLocks}} but before it is passed {{lockMgr}}, you know
everything about the locks that will be asked for, but haven't yet asked for
them.
Then if you walk the QueryPlan to find all TableScan and FileSink, you should
be able to annotate their Desc object with lock info so that it shows up in
EXPLAIN output.
{{QueryPlanPostProcessor}} may be useful - it's an example of how to walk
physical plan to find \{{FileSinkDesc}}
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[ https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490253#comment-16490253 ] Igor Kryvenko commented on HIVE-17683: -- [~ekoifman] Okay. I will start working on it. Thanks, for the proposition. > Annotate Query Plan with locking information > > > Key: HIVE-17683 > URL: https://issues.apache.org/jira/browse/HIVE-17683 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > > Explore if it's possible to add info about what locks will be asked for to > the query plan. > Lock acquisition (for Acid Lock Manager) is done in > DbTxnManager.acquireLocks() which is called once the query starts running. > Would need to refactor that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[ https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490127#comment-16490127 ] Eugene Koifman commented on HIVE-17683: --- [~vbeshka], this would be very useful if you are interested in working on this > Annotate Query Plan with locking information > > > Key: HIVE-17683 > URL: https://issues.apache.org/jira/browse/HIVE-17683 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > > Explore if it's possible to add info about what locks will be asked for to > the query plan. > Lock acquisition (for Acid Lock Manager) is done in > DbTxnManager.acquireLocks() which is called once the query starts running. > Would need to refactor that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
