[jira] [Comment Edited] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546207#comment-16546207
]
Igor Kryvenko edited comment on HIVE-17683 at 7/17/18 9:26 AM:
---
Seems like failure is not related
{code:java}
{code}
I will attach the same patch to check this.
Locally it also works fine:
{code}
[INFO] Running org.apache.hive.beeline.TestBeeLineWithArgs
[WARNING] Tests run: 54, Failures: 0, Errors: 0, Skipped: 2, Time elapsed:
237.004 s - in org.apache.hive.beeline.TestBeeLineWithArgs
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 54, Failures: 0, Errors: 0, Skipped: 2
[INFO]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 05:00 min
[INFO] Finished at: 2018-07-17T12:25:43+03:00
[INFO] Final Memory: 136M/2524M
[INFO]
{code}
was (Author: ikryvenko):
Seems like failure is not related
{code}
{code}
I will attach the same patch to check this.
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch,
> HIVE-17683.03.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542380#comment-16542380
]
Eugene Koifman edited comment on HIVE-17683 at 7/13/18 12:54 AM:
-
[~ikryvenko], sorry, it took a while to get back to this.
Your implementation creates ExplainTask.getJsonLocks() which duplicates a lot
of the logic in DbTxnManger.acquireLocks(). This is problematic because they
have to be kept in sync.
Could you refactor it so that they share code?
For example, create a {{LockRequest makeLockRequest(List,
List)}} and use it in both places?
Also, the refactoring in acquireLocks() lost
{noformat}
default:
throw new IllegalArgumentException(String
.format("Lock type [%s] for Database.Table [%s.%s] is unknown", lockType,
t.getDbName(),
t.getTableName()
));{noformat}
This may change how errors are surfaced - not sure it's a good idea.
Don't know if it's related to your changes but in explain_locks.q.out
{{explain locks drop table test_explain_locks}}
doesn't acquire any locks - this is odd - I'd expect X lock on the table for a
drop command.
Why did you chose to output the data as JSON?
was (Author: ekoifman):
[~ikryvenko], sorry, it took a while to get back to this.
Your implementation creates ExplainTask.getJsonLocks() which duplicates a lot
of the logic in DbTxnManger.acquireLocks(). This is problematic because they
have to be kept in sync.
Could you refactor it so that they share code?
For example, create a {{LockRequest makeLockRequest(List,
List)}} and use it in both places?
Also, the refactoring in acquireLocks() lost
{noformat}
default:
throw new IllegalArgumentException(String
.format("Lock type [%s] for Database.Table [%s.%s] is unknown", lockType,
t.getDbName(),
t.getTableName()
));{noformat}
This may change how errors are surfaced - not sure it's a good idea.
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505119#comment-16505119
]
Igor Kryvenko edited comment on HIVE-17683 at 6/7/18 7:06 PM:
--
{code}
[ERROR]
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11593/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java:[521,22]
cannot find symbol
symbol: method setLock(org.apache.hadoop.hive.metastore.api.LockType)
location: variable compBuilder of type
org.apache.hadoop.hive.metastore.LockComponentBuilder
{code}
It can't find added method.
Is the last patch good for you?
was (Author: ikryvenko):
{code}
[ERROR]
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11593/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java:[521,22]
cannot find symbol
symbol: method setLock(org.apache.hadoop.hive.metastore.api.LockType)
location: variable compBuilder of type
org.apache.hadoop.hive.metastore.LockComponentBuilder
{code}
It can't find added method.
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505119#comment-16505119
]
Igor Kryvenko edited comment on HIVE-17683 at 6/7/18 7:06 PM:
--
{code}
[ERROR]
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11593/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java:[521,22]
cannot find symbol
symbol: method setLock(org.apache.hadoop.hive.metastore.api.LockType)
location: variable compBuilder of type
org.apache.hadoop.hive.metastore.LockComponentBuilder
{code}
It can't find added method.
was (Author: ikryvenko):
{code}
[ERROR]
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11593/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java:[521,22]
cannot find symbol
symbol: method setLock(org.apache.hadoop.hive.metastore.api.LockType)
location: variable compBuilder of type
org.apache.hadoop.hive.metastore.LockComponentBuilder
{code}
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
