[jira] [Commented] (HIVE-12254) Improve logging with yarn/hdfs
[
https://issues.apache.org/jira/browse/HIVE-12254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636489#comment-16636489
]
Hive QA commented on HIVE-12254:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
18s{color} | {color:blue} shims/common in master has 6 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
19s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} cli in master has 13 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
49s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
42s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
16s{color} | {color:red} cli in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14193/dev-support/hive-personality.sh
|
| git revision | master / c44853c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14193/yetus/patch-mvninstall-cli.txt
|
| modules | C: shims/common shims/0.23 cli ql service U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14193/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve logging with yarn/hdfs
> --
>
> Key: HIVE-12254
> URL: https://issues.apache.org/jira/browse/HIVE-12254
> Project: Hive
> Issue Type: Bug
> Components: Shims
>Affects Versions: 1.2.1
>Reporter: Vikram Dixit K
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-12254.1.patch, HIVE-12254.2.patch,
> HIVE-12254.3.patch, HIVE-12254.4.patch, HIVE-12254.5.patch
>
>
> In extension to HIVE-12249, adding info for Yarn/HDFS as well. Both
> HIVE-12249 and HDFS-9184 are required (and upgraded
[jira] [Commented] (HIVE-20536) Add Surrogate Keys function to Hive
[
https://issues.apache.org/jira/browse/HIVE-20536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636480#comment-16636480
]
Ashutosh Chauhan commented on HIVE-20536:
-
[~mgergely] [~vgarg] Actually merge statement already supports default values
via HIVE-19129
So, from Miklos's example following should already work:
{code}
MERGE INTO t1
...
WHEN NOT MATCHED THEN INSERT (a, b) VALUES (t2.a, t2.b, default);
{code}
Miklos, can you add test for that?
> Add Surrogate Keys function to Hive
> ---
>
> Key: HIVE-20536
> URL: https://issues.apache.org/jira/browse/HIVE-20536
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20536.01.patch, HIVE-20536.02.patch,
> HIVE-20536.03.patch, HIVE-20536.04.patch, HIVE-20536.05.patch,
> HIVE-20536.06.patch, HIVE-20536.07.patch
>
>
> Surrogate keys is an ability to generate and use unique integers for each row
> in a table. If we have that ability then in conjunction with default clause
> we can get surrogate keys functionality. Consider following ddl:
> create table t1 (a string, b bigint default unique_long());
> We already have default clause wherein you can specify a function to provide
> values. So, what we need is udf which can generate unique longs for each row
> across queries for a table.
> Idea is to use write_id . This is a column in metastore table TXN_COMPONENTS
> whose value is determined at compile time to be used during query execution.
> Each query execution generates a new write_id. So, we can seed udf with this
> value during compilation.
> Then we statically allocate ranges for each task from which it can draw next
> long. So, lets say 64-bit write_id we divy up such that last 24 bits belong
> to original usage of it that is txns. Next 16 bits are used for task_attempts
> and last 24 bits to generate new long for each row. This implies we can allow
> 17M txns, 65K tasks and 17M rows in a task. If you hit any of those limits we
> can fail the query.
> Implementation wise: serialize write_id in initialize() of udf. Then during
> execute() we find out what task_attempt current task is and use it along with
> write_id() to get starting long and give a new value on each invocation of
> execute().
> Here we are assuming write_id can be determined at compile time, which should
> be the case but we need to figure out how to get handle to it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[ https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-20644: -- Attachment: HIVE-20644.03 Status: Patch Available (was: Open) This .03 patch is rebased version of .02 and is exactly same as .02. Re-attaching it to fire ptest run. > Avoid exposing sensitive infomation through a Hive Runtime exception > > > Key: HIVE-20644 > URL: https://issues.apache.org/jira/browse/HIVE-20644 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Minor > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-20644.01, HIVE-20644.02, HIVE-20644.03 > > > The HiveException raised from the following methods is exposing the datarow > the caused the run time exception. > # ReduceRecordSource::GroupIterator::next() - around line 372 > # MapOperator::process() - around line 567 > # ExecReducer::reduce() - around line 243 > In all the cases, a string representation of the row is constructed on the > fly and is included in > the error message. > VectorMapOperator::process() - around line 973 raises the same exception but > it's not exposing the row since the row contents are not included in the > error message. > While trying to reproduce above error, I also found that the arguments to a > UDF get exposed in log messages from FunctionRegistry::invoke() around line > 1114. This too can cause sensitive information to be leaked through error > message. > This way some sensitive information is leaked to a user through exception > message. That information may not be available to the user otherwise. Hence > it's a kind of security breach or violation of access control. > The contents of the row or the arguments to a function may be useful for > debugging and hence it's worth to add those to logs. Hence proposal here to > log a separate message with log level DEBUG or INFO containing the string > representation of the row. Users can configure their logging so that > DEBUG/INFO messages do not go to the client but at the same time are > available in the hive server logs for debugging. The actual exception message > will not contain any sensitive data like row data or argument data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[ https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-20644: -- Status: Open (was: Patch Available) > Avoid exposing sensitive infomation through a Hive Runtime exception > > > Key: HIVE-20644 > URL: https://issues.apache.org/jira/browse/HIVE-20644 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Minor > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-20644.01, HIVE-20644.02, HIVE-20644.03 > > > The HiveException raised from the following methods is exposing the datarow > the caused the run time exception. > # ReduceRecordSource::GroupIterator::next() - around line 372 > # MapOperator::process() - around line 567 > # ExecReducer::reduce() - around line 243 > In all the cases, a string representation of the row is constructed on the > fly and is included in > the error message. > VectorMapOperator::process() - around line 973 raises the same exception but > it's not exposing the row since the row contents are not included in the > error message. > While trying to reproduce above error, I also found that the arguments to a > UDF get exposed in log messages from FunctionRegistry::invoke() around line > 1114. This too can cause sensitive information to be leaked through error > message. > This way some sensitive information is leaked to a user through exception > message. That information may not be available to the user otherwise. Hence > it's a kind of security breach or violation of access control. > The contents of the row or the arguments to a function may be useful for > debugging and hence it's worth to add those to logs. Hence proposal here to > log a separate message with log level DEBUG or INFO containing the string > representation of the row. Users can configure their logging so that > DEBUG/INFO messages do not go to the client but at the same time are > available in the hive server logs for debugging. The actual exception message > will not contain any sensitive data like row data or argument data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[ https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-20644: -- Attachment: HIVE-20644.03 Status: Patch Available (was: Open) This patch is rebased version of .02 and is exactly same as .02. Re-attaching it to fire ptest run. > Avoid exposing sensitive infomation through a Hive Runtime exception > > > Key: HIVE-20644 > URL: https://issues.apache.org/jira/browse/HIVE-20644 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Minor > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-20644.01, HIVE-20644.02 > > > The HiveException raised from the following methods is exposing the datarow > the caused the run time exception. > # ReduceRecordSource::GroupIterator::next() - around line 372 > # MapOperator::process() - around line 567 > # ExecReducer::reduce() - around line 243 > In all the cases, a string representation of the row is constructed on the > fly and is included in > the error message. > VectorMapOperator::process() - around line 973 raises the same exception but > it's not exposing the row since the row contents are not included in the > error message. > While trying to reproduce above error, I also found that the arguments to a > UDF get exposed in log messages from FunctionRegistry::invoke() around line > 1114. This too can cause sensitive information to be leaked through error > message. > This way some sensitive information is leaked to a user through exception > message. That information may not be available to the user otherwise. Hence > it's a kind of security breach or violation of access control. > The contents of the row or the arguments to a function may be useful for > debugging and hence it's worth to add those to logs. Hence proposal here to > log a separate message with log level DEBUG or INFO containing the string > representation of the row. Users can configure their logging so that > DEBUG/INFO messages do not go to the client but at the same time are > available in the hive server logs for debugging. The actual exception message > will not contain any sensitive data like row data or argument data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[ https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-20644: -- Attachment: (was: HIVE-20644.03) > Avoid exposing sensitive infomation through a Hive Runtime exception > > > Key: HIVE-20644 > URL: https://issues.apache.org/jira/browse/HIVE-20644 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Minor > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-20644.01, HIVE-20644.02 > > > The HiveException raised from the following methods is exposing the datarow > the caused the run time exception. > # ReduceRecordSource::GroupIterator::next() - around line 372 > # MapOperator::process() - around line 567 > # ExecReducer::reduce() - around line 243 > In all the cases, a string representation of the row is constructed on the > fly and is included in > the error message. > VectorMapOperator::process() - around line 973 raises the same exception but > it's not exposing the row since the row contents are not included in the > error message. > While trying to reproduce above error, I also found that the arguments to a > UDF get exposed in log messages from FunctionRegistry::invoke() around line > 1114. This too can cause sensitive information to be leaked through error > message. > This way some sensitive information is leaked to a user through exception > message. That information may not be available to the user otherwise. Hence > it's a kind of security breach or violation of access control. > The contents of the row or the arguments to a function may be useful for > debugging and hence it's worth to add those to logs. Hence proposal here to > log a separate message with log level DEBUG or INFO containing the string > representation of the row. Users can configure their logging so that > DEBUG/INFO messages do not go to the client but at the same time are > available in the hive server logs for debugging. The actual exception message > will not contain any sensitive data like row data or argument data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[ https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-20644: -- Comment: was deleted (was: This patch is rebased version of .02 and is exactly same as .02. Re-attaching it to fire ptest run.) > Avoid exposing sensitive infomation through a Hive Runtime exception > > > Key: HIVE-20644 > URL: https://issues.apache.org/jira/browse/HIVE-20644 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Minor > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-20644.01, HIVE-20644.02 > > > The HiveException raised from the following methods is exposing the datarow > the caused the run time exception. > # ReduceRecordSource::GroupIterator::next() - around line 372 > # MapOperator::process() - around line 567 > # ExecReducer::reduce() - around line 243 > In all the cases, a string representation of the row is constructed on the > fly and is included in > the error message. > VectorMapOperator::process() - around line 973 raises the same exception but > it's not exposing the row since the row contents are not included in the > error message. > While trying to reproduce above error, I also found that the arguments to a > UDF get exposed in log messages from FunctionRegistry::invoke() around line > 1114. This too can cause sensitive information to be leaked through error > message. > This way some sensitive information is leaked to a user through exception > message. That information may not be available to the user otherwise. Hence > it's a kind of security breach or violation of access control. > The contents of the row or the arguments to a function may be useful for > debugging and hence it's worth to add those to logs. Hence proposal here to > log a separate message with log level DEBUG or INFO containing the string > representation of the row. Users can configure their logging so that > DEBUG/INFO messages do not go to the client but at the same time are > available in the hive server logs for debugging. The actual exception message > will not contain any sensitive data like row data or argument data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[ https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636471#comment-16636471 ] Ashutosh Bapat edited comment on HIVE-20644 at 10/3/18 5:22 AM: Testcase org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] has been failing for some time and the failure is unrelated to this patch. The testcase has now been disabled, so need to fire fresh ptest run with my patch. Hence cancelling so that I can re-attach the patch. was (Author: ashutosh.bapat): Testcase > Avoid exposing sensitive infomation through a Hive Runtime exception > > > Key: HIVE-20644 > URL: https://issues.apache.org/jira/browse/HIVE-20644 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Minor > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-20644.01, HIVE-20644.02 > > > The HiveException raised from the following methods is exposing the datarow > the caused the run time exception. > # ReduceRecordSource::GroupIterator::next() - around line 372 > # MapOperator::process() - around line 567 > # ExecReducer::reduce() - around line 243 > In all the cases, a string representation of the row is constructed on the > fly and is included in > the error message. > VectorMapOperator::process() - around line 973 raises the same exception but > it's not exposing the row since the row contents are not included in the > error message. > While trying to reproduce above error, I also found that the arguments to a > UDF get exposed in log messages from FunctionRegistry::invoke() around line > 1114. This too can cause sensitive information to be leaked through error > message. > This way some sensitive information is leaked to a user through exception > message. That information may not be available to the user otherwise. Hence > it's a kind of security breach or violation of access control. > The contents of the row or the arguments to a function may be useful for > debugging and hence it's worth to add those to logs. Hence proposal here to > log a separate message with log level DEBUG or INFO containing the string > representation of the row. Users can configure their logging so that > DEBUG/INFO messages do not go to the client but at the same time are > available in the hive server logs for debugging. The actual exception message > will not contain any sensitive data like row data or argument data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[ https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-20644: -- Status: Open (was: Patch Available) Testcase > Avoid exposing sensitive infomation through a Hive Runtime exception > > > Key: HIVE-20644 > URL: https://issues.apache.org/jira/browse/HIVE-20644 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Minor > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-20644.01, HIVE-20644.02 > > > The HiveException raised from the following methods is exposing the datarow > the caused the run time exception. > # ReduceRecordSource::GroupIterator::next() - around line 372 > # MapOperator::process() - around line 567 > # ExecReducer::reduce() - around line 243 > In all the cases, a string representation of the row is constructed on the > fly and is included in > the error message. > VectorMapOperator::process() - around line 973 raises the same exception but > it's not exposing the row since the row contents are not included in the > error message. > While trying to reproduce above error, I also found that the arguments to a > UDF get exposed in log messages from FunctionRegistry::invoke() around line > 1114. This too can cause sensitive information to be leaked through error > message. > This way some sensitive information is leaked to a user through exception > message. That information may not be available to the user otherwise. Hence > it's a kind of security breach or violation of access control. > The contents of the row or the arguments to a function may be useful for > debugging and hence it's worth to add those to logs. Hence proposal here to > log a separate message with log level DEBUG or INFO containing the string > representation of the row. Users can configure their logging so that > DEBUG/INFO messages do not go to the client but at the same time are > available in the hive server logs for debugging. The actual exception message > will not contain any sensitive data like row data or argument data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20610) TestDbNotificationListener should not use /tmp directory
[
https://issues.apache.org/jira/browse/HIVE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-20610:
Attachment: HIVE-20610.3.patch
> TestDbNotificationListener should not use /tmp directory
>
>
> Key: HIVE-20610
> URL: https://issues.apache.org/jira/browse/HIVE-20610
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0, 4.0.0
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20610.1.patch, HIVE-20610.2.patch,
> HIVE-20610.3.patch
>
>
> Using /tmp directory creates exceptions for tests like dropTable :
> {code:java}
> 2018-09-19T06:42:04,818 INFO [main] metastore.HiveMetaStore: 0: drop_table :
> tbl=hive.default.droptbl
> 2018-09-19T06:42:04,819 INFO [main] HiveMetaStore.audit: ugi=hiveptest
> ip=unknown-ip-addr cmd=drop_table : tbl=hive.default.droptbl
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.ICE-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.XIM-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.X11-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/hsperfdata_root]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.font-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.Test-unix]: it still exists.
> 2018-09-19T06:42:05,072 ERROR [main] utils.FileUtils: Failed to delete
> file:/tmp
> 2018-09-19T06:42:05,072 ERROR [main] utils.MetaStoreUtils: Got exception:
> org.apache.hadoop.hive.metastore.api.MetaException Unable to delete
> directory: file:/tmp
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to delete
> directory: file:/tmp
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreFsImpl.deleteDir(HiveMetaStoreFsImpl.java:45)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:365)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:353)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.deleteTableData(HiveMetaStore.java:2562)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:2523)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:2685)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_102]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_102]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy33.drop_table_with_environment_context(Unknown
> Source) [?:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.drop_table_with_environment_context(HiveMetaStoreClient.java:3204)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1492)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1432)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropTable(TestDbNotificationListener.java:522)
> [test-classes/:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]{code}
>
>
--
This message was sent by
[jira] [Commented] (HIVE-20535) Add new configuration to set the size of the global compile lock
[
https://issues.apache.org/jira/browse/HIVE-20535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636432#comment-16636432
]
Hive QA commented on HIVE-20535:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942143/HIVE-20535.23.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15009 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14192/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14192/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14192/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942143 - PreCommit-HIVE-Build
> Add new configuration to set the size of the global compile lock
>
>
> Key: HIVE-20535
> URL: https://issues.apache.org/jira/browse/HIVE-20535
> Project: Hive
> Issue Type: Task
> Components: HiveServer2
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20535.1.patch, HIVE-20535.10.patch,
> HIVE-20535.11.patch, HIVE-20535.12.patch, HIVE-20535.13.patch,
> HIVE-20535.14.patch, HIVE-20535.15.patch, HIVE-20535.16.patch,
> HIVE-20535.17.patch, HIVE-20535.18.patch, HIVE-20535.19.patch,
> HIVE-20535.2.patch, HIVE-20535.20.patch, HIVE-20535.21.patch,
> HIVE-20535.22.patch, HIVE-20535.23.patch, HIVE-20535.3.patch,
> HIVE-20535.4.patch, HIVE-20535.5.patch, HIVE-20535.6.patch,
> HIVE-20535.8.patch, HIVE-20535.9.patch
>
>
> When removing the compile lock, it is quite risky to remove it entirely.
> It would be good to provide a pool size for the concurrent compilation, so
> the administrator can limit the load
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20627) Concurrent async queries intermittently fails with LockException and cause memory leak.
[
https://issues.apache.org/jira/browse/HIVE-20627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636420#comment-16636420
]
Sankar Hariappan commented on HIVE-20627:
-
[~pvary],
You are suggesting that, we shall maintain an Atomic reference counter in the
Hive object. We increment the counter when somebody sets it in thread local
hiveDb and decrement it when somebody releases it. Only when the counter is
down to 0, we should close the connection.
Couple of cases to release the thread local hiveDb:
* When synchronous query execution in master thread re-creates Hive object due
to config change. We also, need to update the sessionHive object in the current
session as we releases it from thread local hiveDb of master thread.
* When async thread exits after completing execution or due to exception.
If the session is getting closed, it is guaranteed that the reference counter
is down to 0 as we forcefully close all the async operations and so we can
close the connection there.
I think, this will work fine. I will create a Jira to incorporate this fix.
Please let me know if I missed anything.
cc [~daijy], [~thejas]
> Concurrent async queries intermittently fails with LockException and cause
> memory leak.
> ---
>
> Key: HIVE-20627
> URL: https://issues.apache.org/jira/browse/HIVE-20627
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Transactions
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20627.01.patch
>
>
> When multiple async queries are executed from same session, it leads to
> multiple async query execution DAGs share the same Hive object which is set
> by caller for all threads. In case of loading dynamic partitions, it creates
> MoveTask which re-creates the Hive object and closes the shared Hive object
> which causes metastore connection issues for other async execution thread who
> still access it. This is also seen if ReplDumpTask and ReplLoadTask are part
> of the DAG.
> *Call Stack:*
> {code:java}
> 2018-09-16T04:38:04,280 ERROR [load-dynamic-partitions-7]: metadata.Hive
> (Hive.java:call(2436)) - Exception when loading partition with parameters
> partPath=hdfs://mycluster/warehouse/tablespace/managed/hive/tbl_3bcvvdubni/.hive-staging_hive_2018-09-16_04-35-50_708_7776079613819042057-1147/-ext-1/age=55,
> table=tbl_3bcvvdubni, partSpec={age=55}, loadFileType=KEEP_EXISTING,
> listBucketingLevel=0, isAcid=true, hasFollowingStatsTask=true
> org.apache.hadoop.hive.ql.lockmgr.LockException: Error communicating with the
> metastore
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.getValidWriteIds(DbTxnManager.java:714)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getTableValidWriteIdListWithTxnList(AcidUtils.java:1791)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getTableSnapshot(AcidUtils.java:1756)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getTableSnapshot(AcidUtils.java:1714)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1976)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:2415)
> [hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:2406)
> [hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_171]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_171]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_171]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
> Caused by: org.apache.thrift.protocol.TProtocolException: Required field
> 'validTxnList' is unset!
> Struct:GetValidWriteIdsRequest(fullTableNames:[default.tbl_3bcvvdubni],
> validTxnList:null)
> at
> org.apache.hadoop.hive.metastore.api.GetValidWriteIdsRequest.validate(GetValidWriteIdsRequest.java:396)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_valid_write_ids_args.validate(ThriftHiveMetastore.java)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_valid_write_ids_args$get_valid_write_ids_argsStandardScheme.write(ThriftHiveMetastore.java)
>
[jira] [Commented] (HIVE-20535) Add new configuration to set the size of the global compile lock
[
https://issues.apache.org/jira/browse/HIVE-20535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636411#comment-16636411
]
Hive QA commented on HIVE-20535:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
57s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 2 new + 142 unchanged - 6
fixed = 144 total (was 148) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14192/dev-support/hive-personality.sh
|
| git revision | master / c44853c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14192/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14192/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add new configuration to set the size of the global compile lock
>
>
> Key: HIVE-20535
> URL: https://issues.apache.org/jira/browse/HIVE-20535
> Project: Hive
> Issue Type: Task
> Components: HiveServer2
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20535.1.patch, HIVE-20535.10.patch,
> HIVE-20535.11.patch, HIVE-20535.12.patch, HIVE-20535.13.patch,
> HIVE-20535.14.patch, HIVE-20535.15.patch, HIVE-20535.16.patch,
> HIVE-20535.17.patch, HIVE-20535.18.patch, HIVE-20535.19.patch,
> HIVE-20535.2.patch, HIVE-20535.20.patch, HIVE-20535.21.patch,
> HIVE-20535.22.patch, HIVE-20535.23.patch, HIVE-20535.3.patch,
> HIVE-20535.4.patch, HIVE-20535.5.patch, HIVE-20535.6.patch,
> HIVE-20535.8.patch, HIVE-20535.9.patch
>
>
> When removing the compile lock, it is quite risky to remove it entirely.
> It would be good to provide a pool size for the concurrent compilation, so
> the
[jira] [Commented] (HIVE-20545) Ability to exclude potentially large parameters in HMS Notifications
[ https://issues.apache.org/jira/browse/HIVE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636399#comment-16636399 ] Andrew Sherman commented on HIVE-20545: --- +1 LGTM > Ability to exclude potentially large parameters in HMS Notifications > > > Key: HIVE-20545 > URL: https://issues.apache.org/jira/browse/HIVE-20545 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.1.0, 4.0.0 >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > Attachments: HIVE-20545.1.patch, HIVE-20545.2.patch, > HIVE-20545.3.branch-3.patch, HIVE-20545.3.patch, HIVE-20545.4.patch, > HIVE-20545.6.patch, HIVE-20545.7.patch > > > Clients can add large-sized parameters in Table/Partition objects. So we need > to enable adding regex patterns through HiveConf to match parameters to be > filtered from table and partition objects before serialization in HMS > notifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20623) Shared work: Extend sharing of map-join cache entries in LLAP
[
https://issues.apache.org/jira/browse/HIVE-20623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20623:
---
Attachment: HIVE-20623.04.patch
> Shared work: Extend sharing of map-join cache entries in LLAP
> -
>
> Key: HIVE-20623
> URL: https://issues.apache.org/jira/browse/HIVE-20623
> Project: Hive
> Issue Type: Improvement
> Components: llap, Logical Optimizer
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20623.01.patch, HIVE-20623.02.patch,
> HIVE-20623.02.patch, HIVE-20623.02.patch, HIVE-20623.03.patch,
> HIVE-20623.03.patch, HIVE-20623.03.patch, HIVE-20623.04.patch,
> HIVE-20623.04.patch, HIVE-20623.04.patch, HIVE-20623.patch,
> hash-shared-work.json.txt, hash-shared-work.svg
>
>
> For a query like this
> {code}
> with all_sales as (
> select ss_customer_sk as customer_sk, ss_ext_list_price-ss_ext_discount_amt
> as ext_price from store_sales
> UNION ALL
> select ws_bill_customer_sk as customer_sk,
> ws_ext_list_price-ws_ext_discount_amt as ext_price from web_sales
> UNION ALL
> select cs_bill_customer_sk as customer_sk, cs_ext_sales_price -
> cs_ext_discount_amt as ext_price from catalog_sales)
> select sum(ext_price) total_price, c_customer_id from all_sales, customer
> where customer_sk = c_customer_sk
> group by c_customer_id
> order by total_price desc
> limit 100;
> {code}
> The hashtable used for all 3 joins are identical, which is loaded 3x times in
> the same LLAP instance because they are named.
> {code}
> cacheKey = "HASH_MAP_" + this.getOperatorId() + "_container";
> {code}
> in the cache.
> If those are identical in nature (i.e vectorization, hashtable type etc),
> then the duplication is just wasted CPU, memory and network - using the cache
> name for hashtables which will be identical in layout would be extremely
> useful.
> In cases where the join is pushed through a UNION, those are identical.
> This optimization can only be done without concern for accidental delays when
> the same upstream task is generating all of these hashtables, which is what
> is achieved by the shared scan optimizer already.
> In case the shared work is not present, this has potential downsides - in
> case two customer broadcasts were sourced from "Map 1" and "Map 2", the Map 1
> builder will block the other task from reading from Map 2, even though Map 2
> might have started after, but finished ahead of Map 1.
> So this specific optimization can always be considered for cases where the
> shared work unifies the operator tree and the parents of all the RS entries
> involved are same (& the RS layout is the same).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20623) Shared work: Extend sharing of map-join cache entries in LLAP
[
https://issues.apache.org/jira/browse/HIVE-20623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636392#comment-16636392
]
Hive QA commented on HIVE-20623:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942140/HIVE-20623.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15007 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=195)
[druidmini_masking.q,druidmini_test1.q,druidkafkamini_basic.q,druidmini_joins.q,druid_timestamptz.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14191/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14191/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14191/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942140 - PreCommit-HIVE-Build
> Shared work: Extend sharing of map-join cache entries in LLAP
> -
>
> Key: HIVE-20623
> URL: https://issues.apache.org/jira/browse/HIVE-20623
> Project: Hive
> Issue Type: Improvement
> Components: llap, Logical Optimizer
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20623.01.patch, HIVE-20623.02.patch,
> HIVE-20623.02.patch, HIVE-20623.02.patch, HIVE-20623.03.patch,
> HIVE-20623.03.patch, HIVE-20623.03.patch, HIVE-20623.04.patch,
> HIVE-20623.04.patch, HIVE-20623.patch, hash-shared-work.json.txt,
> hash-shared-work.svg
>
>
> For a query like this
> {code}
> with all_sales as (
> select ss_customer_sk as customer_sk, ss_ext_list_price-ss_ext_discount_amt
> as ext_price from store_sales
> UNION ALL
> select ws_bill_customer_sk as customer_sk,
> ws_ext_list_price-ws_ext_discount_amt as ext_price from web_sales
> UNION ALL
> select cs_bill_customer_sk as customer_sk, cs_ext_sales_price -
> cs_ext_discount_amt as ext_price from catalog_sales)
> select sum(ext_price) total_price, c_customer_id from all_sales, customer
> where customer_sk = c_customer_sk
> group by c_customer_id
> order by total_price desc
> limit 100;
> {code}
> The hashtable used for all 3 joins are identical, which is loaded 3x times in
> the same LLAP instance because they are named.
> {code}
> cacheKey = "HASH_MAP_" + this.getOperatorId() + "_container";
> {code}
> in the cache.
> If those are identical in nature (i.e vectorization, hashtable type etc),
> then the duplication is just wasted CPU, memory and network - using the cache
> name for hashtables which will be identical in layout would be extremely
> useful.
> In cases where the join is pushed through a UNION, those are identical.
> This optimization can only be done without concern for accidental delays when
> the same upstream task is generating all of these hashtables, which is what
> is achieved by the shared scan optimizer already.
> In case the shared work is not present, this has potential downsides - in
> case two customer broadcasts were sourced from "Map 1" and "Map 2", the Map 1
> builder will block the other task from reading from Map 2, even though Map 2
> might have started after, but finished ahead of Map 1.
> So this specific optimization can always be considered for cases where the
> shared work unifies the operator tree and the parents of all the RS entries
> involved are same (& the RS layout is the same).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636348#comment-16636348
]
Jesus Camacho Rodriguez edited comment on HIVE-20652 at 10/3/18 3:58 AM:
-
Patch includes:
- Fixes for join and union pushdown from Calcite to verify the datasource is
the same.
- Several improvements and refactoring (e.g. for constants defined in multiple
places).
- HIVE-20669.
- HIVE-20677.
was (Author: jcamachorodriguez):
Patch includes:
- HIVE-20677.
- Fixes for join and union pushdown from Calcite to verify the datasource is
the same.
- Several improvements and refactoring (e.g. for constants defined in multiple
places).
> JdbcStorageHandler push join of two different datasource to jdbc driver
> ---
>
> Key: HIVE-20652
> URL: https://issues.apache.org/jira/browse/HIVE-20652
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20652.patch, external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 JOIN ext_auth2 ON ext_auth1.ikey = ext_auth2.ikey
> {code}
> Error message:
> {code}
> 2018-09-28T00:36:23,860 DEBUG [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT *
> FROM (SELECT *
> FROM "SIMPLE_DERBY_TABLE1"
> WHERE "ikey" IS NOT NULL) AS "t"
> INNER JOIN (SELECT *
> FROM "SIMPLE_DERBY_TABLE2"
> WHERE "ikey" IS NOT NULL) AS "t0" ON "t"."ikey" = "t0"."ikey" {LIMIT 1}]
> 2018-09-28T00:36:23,864 ERROR [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Error while trying to get column names.
> java.sql.SQLSyntaxErrorException: Table/View 'SIMPLE_DERBY_TABLE2' does not
> exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:74)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:78)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:295)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:277)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20652:
---
Attachment: HIVE-20652.patch
> JdbcStorageHandler push join of two different datasource to jdbc driver
> ---
>
> Key: HIVE-20652
> URL: https://issues.apache.org/jira/browse/HIVE-20652
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20652.patch, external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 JOIN ext_auth2 ON ext_auth1.ikey = ext_auth2.ikey
> {code}
> Error message:
> {code}
> 2018-09-28T00:36:23,860 DEBUG [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT *
> FROM (SELECT *
> FROM "SIMPLE_DERBY_TABLE1"
> WHERE "ikey" IS NOT NULL) AS "t"
> INNER JOIN (SELECT *
> FROM "SIMPLE_DERBY_TABLE2"
> WHERE "ikey" IS NOT NULL) AS "t0" ON "t"."ikey" = "t0"."ikey" {LIMIT 1}]
> 2018-09-28T00:36:23,864 ERROR [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Error while trying to get column names.
> java.sql.SQLSyntaxErrorException: Table/View 'SIMPLE_DERBY_TABLE2' does not
> exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:74)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:78)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:295)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:277)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genTablePlan(SemanticAnalyzer.java:11100)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11468)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:525)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20652:
---
Attachment: (was: HIVE-20652.patch)
> JdbcStorageHandler push join of two different datasource to jdbc driver
> ---
>
> Key: HIVE-20652
> URL: https://issues.apache.org/jira/browse/HIVE-20652
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20652.patch, external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 JOIN ext_auth2 ON ext_auth1.ikey = ext_auth2.ikey
> {code}
> Error message:
> {code}
> 2018-09-28T00:36:23,860 DEBUG [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT *
> FROM (SELECT *
> FROM "SIMPLE_DERBY_TABLE1"
> WHERE "ikey" IS NOT NULL) AS "t"
> INNER JOIN (SELECT *
> FROM "SIMPLE_DERBY_TABLE2"
> WHERE "ikey" IS NOT NULL) AS "t0" ON "t"."ikey" = "t0"."ikey" {LIMIT 1}]
> 2018-09-28T00:36:23,864 ERROR [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Error while trying to get column names.
> java.sql.SQLSyntaxErrorException: Table/View 'SIMPLE_DERBY_TABLE2' does not
> exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:74)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:78)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:295)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:277)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genTablePlan(SemanticAnalyzer.java:11100)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11468)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:525)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
[jira] [Commented] (HIVE-20545) Ability to exclude potentially large parameters in HMS Notifications
[ https://issues.apache.org/jira/browse/HIVE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636390#comment-16636390 ] Bharathkrishna Guruvayoor Murali commented on HIVE-20545: - Tests ran locally without failures. [~asherman] , can you please push this if it looks good. > Ability to exclude potentially large parameters in HMS Notifications > > > Key: HIVE-20545 > URL: https://issues.apache.org/jira/browse/HIVE-20545 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.1.0, 4.0.0 >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > Attachments: HIVE-20545.1.patch, HIVE-20545.2.patch, > HIVE-20545.3.branch-3.patch, HIVE-20545.3.patch, HIVE-20545.4.patch, > HIVE-20545.6.patch, HIVE-20545.7.patch > > > Clients can add large-sized parameters in Table/Partition objects. So we need > to enable adding regex patterns through HiveConf to match parameters to be > filtered from table and partition objects before serialization in HMS > notifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20646) Partition filter condition is not pushed down to metastore query if it has IS NOT NULL.
[ https://issues.apache.org/jira/browse/HIVE-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20646: Affects Version/s: 3.1.0 > Partition filter condition is not pushed down to metastore query if it has IS > NOT NULL. > --- > > Key: HIVE-20646 > URL: https://issues.apache.org/jira/browse/HIVE-20646 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Standalone Metastore >Affects Versions: 3.1.0, 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: Metastore, Performance, pull-request-available > Attachments: HIVE-20646.01.patch, HIVE-20646.02.patch > > > If the partition filter condition has "is not null" then the filter query > isn't getting pushed to the SQL query in RDMBS. > This slows down metastore api calls for getting list of partitions with > filter condition. > This condition gets added by optimizer in many cases so this is affecting > many queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20646) Partition filter condition is not pushed down to metastore query if it has IS NOT NULL.
[ https://issues.apache.org/jira/browse/HIVE-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20646: Status: Patch Available (was: Open) > Partition filter condition is not pushed down to metastore query if it has IS > NOT NULL. > --- > > Key: HIVE-20646 > URL: https://issues.apache.org/jira/browse/HIVE-20646 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: Metastore, Performance, pull-request-available > Attachments: HIVE-20646.01.patch, HIVE-20646.02.patch > > > If the partition filter condition has "is not null" then the filter query > isn't getting pushed to the SQL query in RDMBS. > This slows down metastore api calls for getting list of partitions with > filter condition. > This condition gets added by optimizer in many cases so this is affecting > many queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20646) Partition filter condition is not pushed down to metastore query if it has IS NOT NULL.
[ https://issues.apache.org/jira/browse/HIVE-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20646: Attachment: HIVE-20646.02.patch > Partition filter condition is not pushed down to metastore query if it has IS > NOT NULL. > --- > > Key: HIVE-20646 > URL: https://issues.apache.org/jira/browse/HIVE-20646 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: Metastore, Performance, pull-request-available > Attachments: HIVE-20646.01.patch, HIVE-20646.02.patch > > > If the partition filter condition has "is not null" then the filter query > isn't getting pushed to the SQL query in RDMBS. > This slows down metastore api calls for getting list of partitions with > filter condition. > This condition gets added by optimizer in many cases so this is affecting > many queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20646) Partition filter condition is not pushed down to metastore query if it has IS NOT NULL.
[ https://issues.apache.org/jira/browse/HIVE-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20646: Attachment: (was: HIVE-20646.02.patch) > Partition filter condition is not pushed down to metastore query if it has IS > NOT NULL. > --- > > Key: HIVE-20646 > URL: https://issues.apache.org/jira/browse/HIVE-20646 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: Metastore, Performance, pull-request-available > Attachments: HIVE-20646.01.patch, HIVE-20646.02.patch > > > If the partition filter condition has "is not null" then the filter query > isn't getting pushed to the SQL query in RDMBS. > This slows down metastore api calls for getting list of partitions with > filter condition. > This condition gets added by optimizer in many cases so this is affecting > many queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20646) Partition filter condition is not pushed down to metastore query if it has IS NOT NULL.
[ https://issues.apache.org/jira/browse/HIVE-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20646: Status: Open (was: Patch Available) > Partition filter condition is not pushed down to metastore query if it has IS > NOT NULL. > --- > > Key: HIVE-20646 > URL: https://issues.apache.org/jira/browse/HIVE-20646 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: Metastore, Performance, pull-request-available > Attachments: HIVE-20646.01.patch, HIVE-20646.02.patch > > > If the partition filter condition has "is not null" then the filter query > isn't getting pushed to the SQL query in RDMBS. > This slows down metastore api calls for getting list of partitions with > filter condition. > This condition gets added by optimizer in many cases so this is affecting > many queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20623) Shared work: Extend sharing of map-join cache entries in LLAP
[
https://issues.apache.org/jira/browse/HIVE-20623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636377#comment-16636377
]
Hive QA commented on HIVE-20623:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
52s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14191/dev-support/hive-personality.sh
|
| git revision | master / c44853c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14191/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Shared work: Extend sharing of map-join cache entries in LLAP
> -
>
> Key: HIVE-20623
> URL: https://issues.apache.org/jira/browse/HIVE-20623
> Project: Hive
> Issue Type: Improvement
> Components: llap, Logical Optimizer
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20623.01.patch, HIVE-20623.02.patch,
> HIVE-20623.02.patch, HIVE-20623.02.patch, HIVE-20623.03.patch,
> HIVE-20623.03.patch, HIVE-20623.03.patch, HIVE-20623.04.patch,
> HIVE-20623.04.patch, HIVE-20623.patch, hash-shared-work.json.txt,
> hash-shared-work.svg
>
>
> For a query like this
> {code}
> with all_sales as (
> select ss_customer_sk as customer_sk, ss_ext_list_price-ss_ext_discount_amt
> as ext_price from store_sales
> UNION ALL
> select ws_bill_customer_sk as customer_sk,
> ws_ext_list_price-ws_ext_discount_amt as ext_price from web_sales
> UNION ALL
> select cs_bill_customer_sk as customer_sk, cs_ext_sales_price -
> cs_ext_discount_amt as ext_price from catalog_sales)
> select sum(ext_price) total_price, c_customer_id from all_sales, customer
> where
[jira] [Commented] (HIVE-20563) Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type are different
[
https://issues.apache.org/jira/browse/HIVE-20563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636374#comment-16636374
]
Ashutosh Chauhan commented on HIVE-20563:
-
[~gopalv] [~teddy.choi] Can you please review?
> Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type
> are different
> ---
>
> Key: HIVE-20563
> URL: https://issues.apache.org/jira/browse/HIVE-20563
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Matt McCline
>Priority: Major
> Attachments: HIVE-20563.01.patch, HIVE-20563.02.patch,
> HIVE-20563.03.patch, HIVE-20563.04.patch, HIVE-20563.05.patch,
> HIVE-20563.06.patch
>
>
> With the following stacktrace:
> {code}
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
> [hadoop-mapreduce-client-common-3.1.0.jar:?]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:973)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:154)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating
> cstring1
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:149)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:965)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:938)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
>
[jira] [Commented] (HIVE-20563) Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type are different
[
https://issues.apache.org/jira/browse/HIVE-20563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636365#comment-16636365
]
Hive QA commented on HIVE-20563:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942136/HIVE-20563.06.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15010 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testTokenAuth (batchId=266)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14189/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14189/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14189/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942136 - PreCommit-HIVE-Build
> Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type
> are different
> ---
>
> Key: HIVE-20563
> URL: https://issues.apache.org/jira/browse/HIVE-20563
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Matt McCline
>Priority: Major
> Attachments: HIVE-20563.01.patch, HIVE-20563.02.patch,
> HIVE-20563.03.patch, HIVE-20563.04.patch, HIVE-20563.05.patch,
> HIVE-20563.06.patch
>
>
> With the following stacktrace:
> {code}
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
> [hadoop-mapreduce-client-common-3.1.0.jar:?]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:973)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:154)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
[jira] [Assigned] (HIVE-20678) HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure compatibility
[ https://issues.apache.org/jira/browse/HIVE-20678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alice Fan reassigned HIVE-20678: > HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure > compatibility > > > Key: HIVE-20678 > URL: https://issues.apache.org/jira/browse/HIVE-20678 > Project: Hive > Issue Type: Bug > Components: HBase Handler, Hive, Spark >Affects Versions: 4.0.0 >Reporter: Alice Fan >Assignee: Alice Fan >Priority: Major > > This issue was caused by same reason described in > https://issues.apache.org/jira/browse/HIVE-11166. As the original Jira seems > to be abandoned, so create this one to resolve the issue. > HiveHBaseTableOutputFormat does not implemented HiveOutputFormat, it will > break the compatibility in spark's API when it is using hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20608) Incorrect handling of sql command args in hive service leading to misleading error messages
[
https://issues.apache.org/jira/browse/HIVE-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Soumabrata Chakraborty updated HIVE-20608:
--
Status: In Progress (was: Patch Available)
Working on adding test case
> Incorrect handling of sql command args in hive service leading to misleading
> error messages
> ---
>
> Key: HIVE-20608
> URL: https://issues.apache.org/jira/browse/HIVE-20608
> Project: Hive
> Issue Type: Bug
>Reporter: Soumabrata Chakraborty
>Assignee: Soumabrata Chakraborty
>Priority: Major
> Attachments: HIVE-20608.patch
>
>
> *Steps to reproduce:*
> (1) Connect to HiveServer2 using JDBC driver (not via Beeline)
> (2) Execute a set command with a space before set – e.g. " set
> hive.exec.dynamic.partiton=true"
> (3) The error that is returned says:
> {code:java}
> Caused by: org.apache.hive.service.cli.HiveSQLException: Error while
> processing statement: Cannot modify set hive.exec.dynamic.partition at
> runtime. It is not in list of params that are allowed to be modified at
> runtime
> {code}
> (4) However on removing the space before the set command - it works fine.
>
> *Analysis:*
> Looks like an issue with
> [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java]
>
> In the runInternal() method, the untrimmed sql statement is split by space
> (\\s) but the substring operation to get the command args is done on the
> trimmed sql statement. This causes the issue.
> {code:java}
> String command = getStatement().trim();
> String[] tokens = statement.split("\\s");
> String commandArgs = command.substring(tokens[0].length()).trim();
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20608) Incorrect handling of sql command args in hive service leading to misleading error messages
[
https://issues.apache.org/jira/browse/HIVE-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636356#comment-16636356
]
Soumabrata Chakraborty commented on HIVE-20608:
---
Ok - I'll add a test case.
> Incorrect handling of sql command args in hive service leading to misleading
> error messages
> ---
>
> Key: HIVE-20608
> URL: https://issues.apache.org/jira/browse/HIVE-20608
> Project: Hive
> Issue Type: Bug
>Reporter: Soumabrata Chakraborty
>Assignee: Soumabrata Chakraborty
>Priority: Major
> Attachments: HIVE-20608.patch
>
>
> *Steps to reproduce:*
> (1) Connect to HiveServer2 using JDBC driver (not via Beeline)
> (2) Execute a set command with a space before set – e.g. " set
> hive.exec.dynamic.partiton=true"
> (3) The error that is returned says:
> {code:java}
> Caused by: org.apache.hive.service.cli.HiveSQLException: Error while
> processing statement: Cannot modify set hive.exec.dynamic.partition at
> runtime. It is not in list of params that are allowed to be modified at
> runtime
> {code}
> (4) However on removing the space before the set command - it works fine.
>
> *Analysis:*
> Looks like an issue with
> [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java]
>
> In the runInternal() method, the untrimmed sql statement is split by space
> (\\s) but the substring operation to get the command args is done on the
> trimmed sql statement. This causes the issue.
> {code:java}
> String command = getStatement().trim();
> String[] tokens = statement.split("\\s");
> String commandArgs = command.substring(tokens[0].length()).trim();
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20652:
---
Attachment: HIVE-20652.patch
> JdbcStorageHandler push join of two different datasource to jdbc driver
> ---
>
> Key: HIVE-20652
> URL: https://issues.apache.org/jira/browse/HIVE-20652
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20652.patch, external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 JOIN ext_auth2 ON ext_auth1.ikey = ext_auth2.ikey
> {code}
> Error message:
> {code}
> 2018-09-28T00:36:23,860 DEBUG [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT *
> FROM (SELECT *
> FROM "SIMPLE_DERBY_TABLE1"
> WHERE "ikey" IS NOT NULL) AS "t"
> INNER JOIN (SELECT *
> FROM "SIMPLE_DERBY_TABLE2"
> WHERE "ikey" IS NOT NULL) AS "t0" ON "t"."ikey" = "t0"."ikey" {LIMIT 1}]
> 2018-09-28T00:36:23,864 ERROR [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Error while trying to get column names.
> java.sql.SQLSyntaxErrorException: Table/View 'SIMPLE_DERBY_TABLE2' does not
> exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:74)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:78)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:295)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:277)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genTablePlan(SemanticAnalyzer.java:11100)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11468)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:525)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20652:
---
Attachment: (was: HIVE-20652.patch)
> JdbcStorageHandler push join of two different datasource to jdbc driver
> ---
>
> Key: HIVE-20652
> URL: https://issues.apache.org/jira/browse/HIVE-20652
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20652.patch, external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 JOIN ext_auth2 ON ext_auth1.ikey = ext_auth2.ikey
> {code}
> Error message:
> {code}
> 2018-09-28T00:36:23,860 DEBUG [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT *
> FROM (SELECT *
> FROM "SIMPLE_DERBY_TABLE1"
> WHERE "ikey" IS NOT NULL) AS "t"
> INNER JOIN (SELECT *
> FROM "SIMPLE_DERBY_TABLE2"
> WHERE "ikey" IS NOT NULL) AS "t0" ON "t"."ikey" = "t0"."ikey" {LIMIT 1}]
> 2018-09-28T00:36:23,864 ERROR [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Error while trying to get column names.
> java.sql.SQLSyntaxErrorException: Table/View 'SIMPLE_DERBY_TABLE2' does not
> exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:74)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:78)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:295)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:277)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genTablePlan(SemanticAnalyzer.java:11100)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11468)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:525)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
[jira] [Commented] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636348#comment-16636348
]
Jesus Camacho Rodriguez commented on HIVE-20652:
Patch includes:
- HIVE-20677.
- Fixes for join and union pushdown from Calcite to verify the datasource is
the same.
- Several improvements and refactoring (e.g. for constants defined in multiple
places).
> JdbcStorageHandler push join of two different datasource to jdbc driver
> ---
>
> Key: HIVE-20652
> URL: https://issues.apache.org/jira/browse/HIVE-20652
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20652.patch, external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 JOIN ext_auth2 ON ext_auth1.ikey = ext_auth2.ikey
> {code}
> Error message:
> {code}
> 2018-09-28T00:36:23,860 DEBUG [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT *
> FROM (SELECT *
> FROM "SIMPLE_DERBY_TABLE1"
> WHERE "ikey" IS NOT NULL) AS "t"
> INNER JOIN (SELECT *
> FROM "SIMPLE_DERBY_TABLE2"
> WHERE "ikey" IS NOT NULL) AS "t0" ON "t"."ikey" = "t0"."ikey" {LIMIT 1}]
> 2018-09-28T00:36:23,864 ERROR [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Error while trying to get column names.
> java.sql.SQLSyntaxErrorException: Table/View 'SIMPLE_DERBY_TABLE2' does not
> exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:74)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:78)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:295)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:277)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genTablePlan(SemanticAnalyzer.java:11100)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11468)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
>
[jira] [Updated] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20652:
---
Attachment: HIVE-20652.patch
> JdbcStorageHandler push join of two different datasource to jdbc driver
> ---
>
> Key: HIVE-20652
> URL: https://issues.apache.org/jira/browse/HIVE-20652
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20652.patch, external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 JOIN ext_auth2 ON ext_auth1.ikey = ext_auth2.ikey
> {code}
> Error message:
> {code}
> 2018-09-28T00:36:23,860 DEBUG [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT *
> FROM (SELECT *
> FROM "SIMPLE_DERBY_TABLE1"
> WHERE "ikey" IS NOT NULL) AS "t"
> INNER JOIN (SELECT *
> FROM "SIMPLE_DERBY_TABLE2"
> WHERE "ikey" IS NOT NULL) AS "t0" ON "t"."ikey" = "t0"."ikey" {LIMIT 1}]
> 2018-09-28T00:36:23,864 ERROR [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Error while trying to get column names.
> java.sql.SQLSyntaxErrorException: Table/View 'SIMPLE_DERBY_TABLE2' does not
> exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:74)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:78)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:295)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:277)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genTablePlan(SemanticAnalyzer.java:11100)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11468)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:525)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20652:
---
Status: Patch Available (was: In Progress)
> JdbcStorageHandler push join of two different datasource to jdbc driver
> ---
>
> Key: HIVE-20652
> URL: https://issues.apache.org/jira/browse/HIVE-20652
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 JOIN ext_auth2 ON ext_auth1.ikey = ext_auth2.ikey
> {code}
> Error message:
> {code}
> 2018-09-28T00:36:23,860 DEBUG [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT *
> FROM (SELECT *
> FROM "SIMPLE_DERBY_TABLE1"
> WHERE "ikey" IS NOT NULL) AS "t"
> INNER JOIN (SELECT *
> FROM "SIMPLE_DERBY_TABLE2"
> WHERE "ikey" IS NOT NULL) AS "t0" ON "t"."ikey" = "t0"."ikey" {LIMIT 1}]
> 2018-09-28T00:36:23,864 ERROR [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Error while trying to get column names.
> java.sql.SQLSyntaxErrorException: Table/View 'SIMPLE_DERBY_TABLE2' does not
> exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:74)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:78)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:295)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:277)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genTablePlan(SemanticAnalyzer.java:11100)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11468)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:525)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
>
[jira] [Work started] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-20652 started by Jesus Camacho Rodriguez.
--
> JdbcStorageHandler push join of two different datasource to jdbc driver
> ---
>
> Key: HIVE-20652
> URL: https://issues.apache.org/jira/browse/HIVE-20652
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 JOIN ext_auth2 ON ext_auth1.ikey = ext_auth2.ikey
> {code}
> Error message:
> {code}
> 2018-09-28T00:36:23,860 DEBUG [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT *
> FROM (SELECT *
> FROM "SIMPLE_DERBY_TABLE1"
> WHERE "ikey" IS NOT NULL) AS "t"
> INNER JOIN (SELECT *
> FROM "SIMPLE_DERBY_TABLE2"
> WHERE "ikey" IS NOT NULL) AS "t0" ON "t"."ikey" = "t0"."ikey" {LIMIT 1}]
> 2018-09-28T00:36:23,864 ERROR [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Error while trying to get column names.
> java.sql.SQLSyntaxErrorException: Table/View 'SIMPLE_DERBY_TABLE2' does not
> exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:74)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:78)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:295)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:277)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genTablePlan(SemanticAnalyzer.java:11100)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11468)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:525)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
>
[jira] [Commented] (HIVE-20563) Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type are different
[
https://issues.apache.org/jira/browse/HIVE-20563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636339#comment-16636339
]
Hive QA commented on HIVE-20563:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
48s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 3 new + 782 unchanged - 0
fixed = 785 total (was 782) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14189/dev-support/hive-personality.sh
|
| git revision | master / c44853c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14189/yetus/diff-checkstyle-ql.txt
|
| modules | C: itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14189/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type
> are different
> ---
>
> Key: HIVE-20563
> URL: https://issues.apache.org/jira/browse/HIVE-20563
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Matt McCline
>Priority: Major
> Attachments: HIVE-20563.01.patch, HIVE-20563.02.patch,
> HIVE-20563.03.patch, HIVE-20563.04.patch, HIVE-20563.05.patch,
> HIVE-20563.06.patch
>
>
> With the following stacktrace:
> {code}
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
> [hadoop-mapreduce-client-common-3.1.0.jar:?]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
[jira] [Assigned] (HIVE-20677) JDBC storage handler ordering problem - single split flag
[ https://issues.apache.org/jira/browse/HIVE-20677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-20677: -- > JDBC storage handler ordering problem - single split flag > - > > Key: HIVE-20677 > URL: https://issues.apache.org/jira/browse/HIVE-20677 > Project: Hive > Issue Type: Bug > Components: StorageHandler >Reporter: Gunther Hagleitner >Assignee: Jesus Camacho Rodriguez >Priority: Major > > When Calcite is pushing queries into the JDBC handler splitting the query via > offset/limit can cause issues (RDBMs is not guaranteed to return the data in > the same order every time.) > For these cases we want to: > a) Add a "do not split" flag to the jdbc handler. In that mode jdbc handler > will skip the count and offset/limit processing but just run the query in a > single node. Flag default will be false. > b) Have calcite automatically set this flag. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20545) Ability to exclude potentially large parameters in HMS Notifications
[
https://issues.apache.org/jira/browse/HIVE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636330#comment-16636330
]
Hive QA commented on HIVE-20545:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942132/HIVE-20545.7.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15010 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testMergeOnTezEdges
(batchId=300)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery (batchId=252)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14188/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14188/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14188/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942132 - PreCommit-HIVE-Build
> Ability to exclude potentially large parameters in HMS Notifications
>
>
> Key: HIVE-20545
> URL: https://issues.apache.org/jira/browse/HIVE-20545
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20545.1.patch, HIVE-20545.2.patch,
> HIVE-20545.3.branch-3.patch, HIVE-20545.3.patch, HIVE-20545.4.patch,
> HIVE-20545.6.patch, HIVE-20545.7.patch
>
>
> Clients can add large-sized parameters in Table/Partition objects. So we need
> to enable adding regex patterns through HiveConf to match parameters to be
> filtered from table and partition objects before serialization in HMS
> notifications.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[ https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636304#comment-16636304 ] Andrew Sherman commented on HIVE-20306: --- +1 LGTM with small spelling fixes from RB > Implement projection spec for fetching only requested fields from partitions > > > Key: HIVE-20306 > URL: https://issues.apache.org/jira/browse/HIVE-20306 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch, > HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch, > HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch, > HIVE-20306.10.patch, HIVE-20306.11.patch, HIVE-20306.12.patch, > HIVE-20306.13.patch, HIVE-20306.14.patch, HIVE-20306.15.patch, > HIVE-20306.16.patch, HIVE-20306.17.patch, HIVE-20306.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20545) Ability to exclude potentially large parameters in HMS Notifications
[
https://issues.apache.org/jira/browse/HIVE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636300#comment-16636300
]
Hive QA commented on HIVE-20545:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
23s{color} | {color:blue} standalone-metastore/metastore-common in master has
28 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
4s{color} | {color:blue} standalone-metastore/metastore-server in master has
182 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
14s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 182 unchanged - 0 fixed = 183 total (was 182) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Write to static field
org.apache.hadoop.hive.metastore.messaging.json.JSONMessageFactory.paramsFilter
from instance method
org.apache.hadoop.hive.metastore.messaging.json.JSONMessageFactory.init() At
JSONMessageFactory.java:from instance method
org.apache.hadoop.hive.metastore.messaging.json.JSONMessageFactory.init() At
JSONMessageFactory.java:[line 113] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14188/dev-support/hive-personality.sh
|
| git revision | master / c44853c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14188/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14188/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Ability to exclude potentially large parameters in HMS Notifications
>
>
> Key: HIVE-20545
> URL: https://issues.apache.org/jira/browse/HIVE-20545
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Bharathkrishna Guruvayoor Murali
>
[jira] [Updated] (HIVE-17043) Remove non unique columns from group by keys if not referenced later
[ https://issues.apache.org/jira/browse/HIVE-17043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17043: --- Status: Patch Available (was: Open) > Remove non unique columns from group by keys if not referenced later > > > Key: HIVE-17043 > URL: https://issues.apache.org/jira/browse/HIVE-17043 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer >Affects Versions: 3.0.0 >Reporter: Ashutosh Chauhan >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-17043.1.patch, HIVE-17043.2.patch, > HIVE-17043.3.patch, HIVE-17043.4.patch, HIVE-17043.5.patch, > HIVE-17043.6.patch, HIVE-17043.7.patch, HIVE-17043.8.patch > > > Group by keys may be a mix of unique (or primary) keys and regular columns. > In such cases presence of regular column won't alter cardinality of groups. > So, if regular columns are not referenced later, they can be dropped from > group by keys. Depending on operator tree may result in those columns not > being read at all from disk in best case. In worst case, we will avoid > shuffling and sorting regular columns from mapper to reducer, which still > could be substantial CPU and network savings. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17043) Remove non unique columns from group by keys if not referenced later
[ https://issues.apache.org/jira/browse/HIVE-17043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17043: --- Attachment: HIVE-17043.8.patch > Remove non unique columns from group by keys if not referenced later > > > Key: HIVE-17043 > URL: https://issues.apache.org/jira/browse/HIVE-17043 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer >Affects Versions: 3.0.0 >Reporter: Ashutosh Chauhan >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-17043.1.patch, HIVE-17043.2.patch, > HIVE-17043.3.patch, HIVE-17043.4.patch, HIVE-17043.5.patch, > HIVE-17043.6.patch, HIVE-17043.7.patch, HIVE-17043.8.patch > > > Group by keys may be a mix of unique (or primary) keys and regular columns. > In such cases presence of regular column won't alter cardinality of groups. > So, if regular columns are not referenced later, they can be dropped from > group by keys. Depending on operator tree may result in those columns not > being read at all from disk in best case. In worst case, we will avoid > shuffling and sorting regular columns from mapper to reducer, which still > could be substantial CPU and network savings. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17043) Remove non unique columns from group by keys if not referenced later
[ https://issues.apache.org/jira/browse/HIVE-17043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17043: --- Status: Open (was: Patch Available) > Remove non unique columns from group by keys if not referenced later > > > Key: HIVE-17043 > URL: https://issues.apache.org/jira/browse/HIVE-17043 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer >Affects Versions: 3.0.0 >Reporter: Ashutosh Chauhan >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-17043.1.patch, HIVE-17043.2.patch, > HIVE-17043.3.patch, HIVE-17043.4.patch, HIVE-17043.5.patch, > HIVE-17043.6.patch, HIVE-17043.7.patch, HIVE-17043.8.patch > > > Group by keys may be a mix of unique (or primary) keys and regular columns. > In such cases presence of regular column won't alter cardinality of groups. > So, if regular columns are not referenced later, they can be dropped from > group by keys. Depending on operator tree may result in those columns not > being read at all from disk in best case. In worst case, we will avoid > shuffling and sorting regular columns from mapper to reducer, which still > could be substantial CPU and network savings. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20676) HiveServer2: PrivilegeSynchronizer is not set to daemon status
[ https://issues.apache.org/jira/browse/HIVE-20676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-20676: Status: Patch Available (was: Open) > HiveServer2: PrivilegeSynchronizer is not set to daemon status > -- > > Key: HIVE-20676 > URL: https://issues.apache.org/jira/browse/HIVE-20676 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-20676.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[ https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-20306: --- Attachment: (was: HIVE-20306.17.patch) > Implement projection spec for fetching only requested fields from partitions > > > Key: HIVE-20306 > URL: https://issues.apache.org/jira/browse/HIVE-20306 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch, > HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch, > HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch, > HIVE-20306.10.patch, HIVE-20306.11.patch, HIVE-20306.12.patch, > HIVE-20306.13.patch, HIVE-20306.14.patch, HIVE-20306.15.patch, > HIVE-20306.16.patch, HIVE-20306.17.patch, HIVE-20306.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[ https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-20306: --- Attachment: HIVE-20306.17.patch > Implement projection spec for fetching only requested fields from partitions > > > Key: HIVE-20306 > URL: https://issues.apache.org/jira/browse/HIVE-20306 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch, > HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch, > HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch, > HIVE-20306.10.patch, HIVE-20306.11.patch, HIVE-20306.12.patch, > HIVE-20306.13.patch, HIVE-20306.14.patch, HIVE-20306.15.patch, > HIVE-20306.16.patch, HIVE-20306.17.patch, HIVE-20306.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20676) HiveServer2: PrivilegeSynchronizer is not set to daemon status
[ https://issues.apache.org/jira/browse/HIVE-20676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636284#comment-16636284 ] Thejas M Nair commented on HIVE-20676: -- +1 pending tests cc [~daijy] > HiveServer2: PrivilegeSynchronizer is not set to daemon status > -- > > Key: HIVE-20676 > URL: https://issues.apache.org/jira/browse/HIVE-20676 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-20676.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20676) HiveServer2: PrivilegeSynchronizer is not set to daemon status
[ https://issues.apache.org/jira/browse/HIVE-20676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-20676: Attachment: HIVE-20676.1.patch > HiveServer2: PrivilegeSynchronizer is not set to daemon status > -- > > Key: HIVE-20676 > URL: https://issues.apache.org/jira/browse/HIVE-20676 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-20676.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[ https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-20306: --- Attachment: HIVE-20306.17.patch > Implement projection spec for fetching only requested fields from partitions > > > Key: HIVE-20306 > URL: https://issues.apache.org/jira/browse/HIVE-20306 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch, > HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch, > HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch, > HIVE-20306.10.patch, HIVE-20306.11.patch, HIVE-20306.12.patch, > HIVE-20306.13.patch, HIVE-20306.14.patch, HIVE-20306.15.patch, > HIVE-20306.16.patch, HIVE-20306.17.patch, HIVE-20306.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[ https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar reassigned HIVE-20306: -- Assignee: Vihang Karajgaonkar (was: Alexander Kolbasov) > Implement projection spec for fetching only requested fields from partitions > > > Key: HIVE-20306 > URL: https://issues.apache.org/jira/browse/HIVE-20306 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch, > HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch, > HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch, > HIVE-20306.10.patch, HIVE-20306.11.patch, HIVE-20306.12.patch, > HIVE-20306.13.patch, HIVE-20306.14.patch, HIVE-20306.15.patch, > HIVE-20306.16.patch, HIVE-20306.17.patch, HIVE-20306.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[ https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636271#comment-16636271 ] Vihang Karajgaonkar commented on HIVE-20306: Thank you [~akolb] for taking the patch forward. The test failures are unrelated. Reattaching the patch which include's [~asherman]'s suggested changes. > Implement projection spec for fetching only requested fields from partitions > > > Key: HIVE-20306 > URL: https://issues.apache.org/jira/browse/HIVE-20306 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch, > HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch, > HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch, > HIVE-20306.10.patch, HIVE-20306.11.patch, HIVE-20306.12.patch, > HIVE-20306.13.patch, HIVE-20306.14.patch, HIVE-20306.15.patch, > HIVE-20306.16.patch, HIVE-20306.17.patch, HIVE-20306.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20575) Fix flaky connection metric tests
[
https://issues.apache.org/jira/browse/HIVE-20575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636267#comment-16636267
]
Hive QA commented on HIVE-20575:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942113/HIVE-20575.11.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15007 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14187/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14187/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14187/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942113 - PreCommit-HIVE-Build
> Fix flaky connection metric tests
> -
>
> Key: HIVE-20575
> URL: https://issues.apache.org/jira/browse/HIVE-20575
> Project: Hive
> Issue Type: Test
> Components: Hive, Test
>Affects Versions: 4.0.0
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20575.01.patch, HIVE-20575.02.patch,
> HIVE-20575.03.patch, HIVE-20575.04.patch, HIVE-20575.05.patch,
> HIVE-20575.06.patch, HIVE-20575.07.patch, HIVE-20575.08.patch,
> HIVE-20575.09.patch, HIVE-20575.10.patch, HIVE-20575.11.patch
>
>
> TestHs2ConnectionMetricsHttp.testOpenConnectionMetrics() is flaky. We need to
> fix it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20435) Failed Dynamic Partition Insert into insert only table may loose transaction metadata
[
https://issues.apache.org/jira/browse/HIVE-20435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-20435:
--
Summary: Failed Dynamic Partition Insert into insert only table may loose
transaction metadata (was: Failed Dynamic Partition Insert into insert only
table may loose transaction metadat)
> Failed Dynamic Partition Insert into insert only table may loose transaction
> metadata
> -
>
> Key: HIVE-20435
> URL: https://issues.apache.org/jira/browse/HIVE-20435
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
>
> {{TxnHandler.enqueueLockWithRetry()}} has an optimization where it doesn't
> writ to {{TXN_COMPONENTS}} if the write is a dynamic partition insert because
> it expects to write to this table from {{addDynamicPartitions()}}.
> For insert-only, transactional tables, we create the target dir and start
> writing to it before {{addDynamicPartitions()}} is called. So if a txn is
> aborted, we may have a delta dir in the partition but no corresponding entry
> in {{TXN_COMPONENTS}}. This means {{TxnStore.cleanEmptyAbortedTxns()}} may
> clean up {{TXNS}} entry for the aborted transaction before Compactor removes
> this delta dir, at which point it looks like committed data.
> Full CRUD are currently immune to this since they rely on "move" operation in
> MoveTask but longer term they should follow the same model as insert-only
> tables.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20671) Hive Streaming has a broken dependency on metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636255#comment-16636255
]
Jaume M commented on HIVE-20671:
[~akolb], could you reproduce this on master?
> Hive Streaming has a broken dependency on metastore-server
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20671.02.patch, HIVE-20671.03.patch,
> HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20660) Group by statistics estimation could be improved by bounding the total number of rows to source table
[
https://issues.apache.org/jira/browse/HIVE-20660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20660:
---
Status: Patch Available (was: Open)
> Group by statistics estimation could be improved by bounding the total number
> of rows to source table
> -
>
> Key: HIVE-20660
> URL: https://issues.apache.org/jira/browse/HIVE-20660
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20660.1.patch, HIVE-20660.2.patch
>
>
> Currently the stats for group by is estimated by taking product of NDVs of
> all the keys and bounding it by the number of rows of its input. This bound
> could be improved by using the source table instead of immediate input, the
> insight in this case is that cardinality/ndvs of a table can not go beyond
> the original (outer joins will only add NULLs thereby increasing the
> cardinality by 1).
> Note that the assumption here is that group by keys all belong to the same
> source table/input.
> This will improve the estimation in situations where group by is executed
> after joins wherein Hive could end up estimating the number of rows.
> *Reproducer*
> {code:sql}
> set hive.stats.fetch.column.stats=true;
> create table t1(i int, j int);
> alter table t1 update statistics set('numRows'='1',
> 'rawDataSize'='18000');
> alter table t1 update statistics for column i
> set('numDVs'='2500','numNulls'='50','highValue'='1000','lowValue'='0');
> alter table t1 update statistics for column j
> set('numDVs'='500','numNulls'='30','highValue'='100','lowValue'='50');
> create table t2(i2 int, j2 int);
> alter table t2 update statistics set('numRows'='1',
> 'rawDataSize'='1');
> alter table t2 update statistics for column i2
> set('numDVs'='1000','numNulls'='0','highValue'='8000','lowValue'='0');
> alter table t2 update statistics for column j2
> set('numDVs'='10','numNulls'='0','highValue'='800','lowValue'='-1');
> explain select count (1) from t1,t2
> where t1.j=t2.i2
> group by t1.i, t1.j;
> {code}
> {code:sql}
> Reducer 2
> Reduce Operator Tree:
> Merge Join Operator
> condition map:
> Inner Join 0 to 1
> keys:
> 0 _col1 (type: int)
> 1 _col0 (type: int)
> outputColumnNames: _col0, _col1
> Statistics: Num rows: 99700 Data size: 797288 Basic stats:
> COMPLETE Column stats: COMPLETE
> Group By Operator
> aggregations: count()
> keys: _col0 (type: int), _col1 (type: int)
> mode: hash
> outputColumnNames: _col0, _col1, _col2
> Statistics: Num rows: 49850 Data size: 797448 Basic stats:
> COMPLETE Column stats: COMPLETE <==
> Reduce Output Operator
> key expressions: _col0 (type: int), _col1 (type: int)
> sort order: ++
> Map-reduce partition columns: _col0 (type: int), _col1
> (type: int)
> Statistics: Num rows: 49850 Data size: 797448 Basic
> stats: COMPLETE Column stats: COMPLETE
> value expressions: _col2 (type: bigint)
> .
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20660) Group by statistics estimation could be improved by bounding the total number of rows to source table
[
https://issues.apache.org/jira/browse/HIVE-20660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20660:
---
Status: Open (was: Patch Available)
> Group by statistics estimation could be improved by bounding the total number
> of rows to source table
> -
>
> Key: HIVE-20660
> URL: https://issues.apache.org/jira/browse/HIVE-20660
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20660.1.patch, HIVE-20660.2.patch
>
>
> Currently the stats for group by is estimated by taking product of NDVs of
> all the keys and bounding it by the number of rows of its input. This bound
> could be improved by using the source table instead of immediate input, the
> insight in this case is that cardinality/ndvs of a table can not go beyond
> the original (outer joins will only add NULLs thereby increasing the
> cardinality by 1).
> Note that the assumption here is that group by keys all belong to the same
> source table/input.
> This will improve the estimation in situations where group by is executed
> after joins wherein Hive could end up estimating the number of rows.
> *Reproducer*
> {code:sql}
> set hive.stats.fetch.column.stats=true;
> create table t1(i int, j int);
> alter table t1 update statistics set('numRows'='1',
> 'rawDataSize'='18000');
> alter table t1 update statistics for column i
> set('numDVs'='2500','numNulls'='50','highValue'='1000','lowValue'='0');
> alter table t1 update statistics for column j
> set('numDVs'='500','numNulls'='30','highValue'='100','lowValue'='50');
> create table t2(i2 int, j2 int);
> alter table t2 update statistics set('numRows'='1',
> 'rawDataSize'='1');
> alter table t2 update statistics for column i2
> set('numDVs'='1000','numNulls'='0','highValue'='8000','lowValue'='0');
> alter table t2 update statistics for column j2
> set('numDVs'='10','numNulls'='0','highValue'='800','lowValue'='-1');
> explain select count (1) from t1,t2
> where t1.j=t2.i2
> group by t1.i, t1.j;
> {code}
> {code:sql}
> Reducer 2
> Reduce Operator Tree:
> Merge Join Operator
> condition map:
> Inner Join 0 to 1
> keys:
> 0 _col1 (type: int)
> 1 _col0 (type: int)
> outputColumnNames: _col0, _col1
> Statistics: Num rows: 99700 Data size: 797288 Basic stats:
> COMPLETE Column stats: COMPLETE
> Group By Operator
> aggregations: count()
> keys: _col0 (type: int), _col1 (type: int)
> mode: hash
> outputColumnNames: _col0, _col1, _col2
> Statistics: Num rows: 49850 Data size: 797448 Basic stats:
> COMPLETE Column stats: COMPLETE <==
> Reduce Output Operator
> key expressions: _col0 (type: int), _col1 (type: int)
> sort order: ++
> Map-reduce partition columns: _col0 (type: int), _col1
> (type: int)
> Statistics: Num rows: 49850 Data size: 797448 Basic
> stats: COMPLETE Column stats: COMPLETE
> value expressions: _col2 (type: bigint)
> .
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20660) Group by statistics estimation could be improved by bounding the total number of rows to source table
[
https://issues.apache.org/jira/browse/HIVE-20660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20660:
---
Attachment: HIVE-20660.2.patch
> Group by statistics estimation could be improved by bounding the total number
> of rows to source table
> -
>
> Key: HIVE-20660
> URL: https://issues.apache.org/jira/browse/HIVE-20660
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20660.1.patch, HIVE-20660.2.patch
>
>
> Currently the stats for group by is estimated by taking product of NDVs of
> all the keys and bounding it by the number of rows of its input. This bound
> could be improved by using the source table instead of immediate input, the
> insight in this case is that cardinality/ndvs of a table can not go beyond
> the original (outer joins will only add NULLs thereby increasing the
> cardinality by 1).
> Note that the assumption here is that group by keys all belong to the same
> source table/input.
> This will improve the estimation in situations where group by is executed
> after joins wherein Hive could end up estimating the number of rows.
> *Reproducer*
> {code:sql}
> set hive.stats.fetch.column.stats=true;
> create table t1(i int, j int);
> alter table t1 update statistics set('numRows'='1',
> 'rawDataSize'='18000');
> alter table t1 update statistics for column i
> set('numDVs'='2500','numNulls'='50','highValue'='1000','lowValue'='0');
> alter table t1 update statistics for column j
> set('numDVs'='500','numNulls'='30','highValue'='100','lowValue'='50');
> create table t2(i2 int, j2 int);
> alter table t2 update statistics set('numRows'='1',
> 'rawDataSize'='1');
> alter table t2 update statistics for column i2
> set('numDVs'='1000','numNulls'='0','highValue'='8000','lowValue'='0');
> alter table t2 update statistics for column j2
> set('numDVs'='10','numNulls'='0','highValue'='800','lowValue'='-1');
> explain select count (1) from t1,t2
> where t1.j=t2.i2
> group by t1.i, t1.j;
> {code}
> {code:sql}
> Reducer 2
> Reduce Operator Tree:
> Merge Join Operator
> condition map:
> Inner Join 0 to 1
> keys:
> 0 _col1 (type: int)
> 1 _col0 (type: int)
> outputColumnNames: _col0, _col1
> Statistics: Num rows: 99700 Data size: 797288 Basic stats:
> COMPLETE Column stats: COMPLETE
> Group By Operator
> aggregations: count()
> keys: _col0 (type: int), _col1 (type: int)
> mode: hash
> outputColumnNames: _col0, _col1, _col2
> Statistics: Num rows: 49850 Data size: 797448 Basic stats:
> COMPLETE Column stats: COMPLETE <==
> Reduce Output Operator
> key expressions: _col0 (type: int), _col1 (type: int)
> sort order: ++
> Map-reduce partition columns: _col0 (type: int), _col1
> (type: int)
> Statistics: Num rows: 49850 Data size: 797448 Basic
> stats: COMPLETE Column stats: COMPLETE
> value expressions: _col2 (type: bigint)
> .
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20675) Log pollution from PrivilegeSynchronizer if zk is not configured
[
https://issues.apache.org/jira/browse/HIVE-20675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-20675:
--
Attachment: HIVE-20675.1.patch
> Log pollution from PrivilegeSynchronizer if zk is not configured
>
>
> Key: HIVE-20675
> URL: https://issues.apache.org/jira/browse/HIVE-20675
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Minor
> Attachments: HIVE-20675.1.patch
>
>
> Shall stop PrivilegeSynchronizer if "hive.zookeeper.quorum" is not
> configured. Note "hive.privilege.synchronizer" is on by default.
> {code}
> 2018-10-02T16:04:12,488 WARN [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing
> socket connection and attempting reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> ~[?:1.8.0_91]
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> ~[?:1.8.0_91]
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
> ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
> ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20675) Log pollution from PrivilegeSynchronizer if zk is not configured
[
https://issues.apache.org/jira/browse/HIVE-20675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-20675:
--
Status: Patch Available (was: Open)
> Log pollution from PrivilegeSynchronizer if zk is not configured
>
>
> Key: HIVE-20675
> URL: https://issues.apache.org/jira/browse/HIVE-20675
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Minor
> Attachments: HIVE-20675.1.patch
>
>
> Shall stop PrivilegeSynchronizer if "hive.zookeeper.quorum" is not
> configured. Note "hive.privilege.synchronizer" is on by default.
> {code}
> 2018-10-02T16:04:12,488 WARN [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing
> socket connection and attempting reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> ~[?:1.8.0_91]
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> ~[?:1.8.0_91]
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
> ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
> ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20676) HiveServer2: PrivilegeSynchronizer is not set to daemon status
[ https://issues.apache.org/jira/browse/HIVE-20676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta reassigned HIVE-20676: --- Assignee: Vaibhav Gumashta > HiveServer2: PrivilegeSynchronizer is not set to daemon status > -- > > Key: HIVE-20676 > URL: https://issues.apache.org/jira/browse/HIVE-20676 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20675) Log pollution from PrivilegeSynchronizer if zk is not configured
[
https://issues.apache.org/jira/browse/HIVE-20675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai reassigned HIVE-20675:
-
> Log pollution from PrivilegeSynchronizer if zk is not configured
>
>
> Key: HIVE-20675
> URL: https://issues.apache.org/jira/browse/HIVE-20675
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Minor
>
> Shall stop PrivilegeSynchronizer if "hive.zookeeper.quorum" is not
> configured. Note "hive.privilege.synchronizer" is on by default.
> {code}
> 2018-10-02T16:04:12,488 WARN [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing
> socket connection and attempting reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> ~[?:1.8.0_91]
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> ~[?:1.8.0_91]
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
> ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
> ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20674) TestJdbcWithMiniLlapArrow.testKillQuery fail frequently
[
https://issues.apache.org/jira/browse/HIVE-20674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-20674:
--
Status: Patch Available (was: Open)
> TestJdbcWithMiniLlapArrow.testKillQuery fail frequently
> ---
>
> Key: HIVE-20674
> URL: https://issues.apache.org/jira/browse/HIVE-20674
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20674.1.patch
>
>
> Intermittent but frequently:
> {code}
> java.sql.SQLException: org.apache.thrift.transport.TTransportException
> at
> org.apache.hive.jdbc.HiveStatement.closeStatementIfNeeded(HiveStatement.java:222)
> at
> org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:227)
> at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:243)
> at
> org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery(TestJdbcWithMiniLlapArrow.java:349)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
> Caused by: org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
> at
> org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453)
> at
> org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_CloseOperation(TCLIService.java:521)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.CloseOperation(TCLIService.java:508)
> at
[jira] [Updated] (HIVE-20674) TestJdbcWithMiniLlapArrow.testKillQuery fail frequently
[
https://issues.apache.org/jira/browse/HIVE-20674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-20674:
--
Attachment: HIVE-20674.1.patch
> TestJdbcWithMiniLlapArrow.testKillQuery fail frequently
> ---
>
> Key: HIVE-20674
> URL: https://issues.apache.org/jira/browse/HIVE-20674
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20674.1.patch
>
>
> Intermittent but frequently:
> {code}
> java.sql.SQLException: org.apache.thrift.transport.TTransportException
> at
> org.apache.hive.jdbc.HiveStatement.closeStatementIfNeeded(HiveStatement.java:222)
> at
> org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:227)
> at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:243)
> at
> org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery(TestJdbcWithMiniLlapArrow.java:349)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
> Caused by: org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
> at
> org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453)
> at
> org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_CloseOperation(TCLIService.java:521)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.CloseOperation(TCLIService.java:508)
> at
[jira] [Updated] (HIVE-20674) TestJdbcWithMiniLlapArrow.testKillQuery fail frequently
[
https://issues.apache.org/jira/browse/HIVE-20674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-20674:
--
Description:
Intermittent but frequently:
{code}
java.sql.SQLException: org.apache.thrift.transport.TTransportException
at
org.apache.hive.jdbc.HiveStatement.closeStatementIfNeeded(HiveStatement.java:222)
at
org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:227)
at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:243)
at
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery(TestJdbcWithMiniLlapArrow.java:349)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
Caused by: org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
at
org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453)
at
org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
at
org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at
org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_CloseOperation(TCLIService.java:521)
at
org.apache.hive.service.rpc.thrift.TCLIService$Client.CloseOperation(TCLIService.java:508)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1669)
at com.sun.proxy.$Proxy131.CloseOperation(Unknown Source)
at
[jira] [Assigned] (HIVE-20674) TestJdbcWithMiniLlapArrow.testKillQuery fail frequently
[ https://issues.apache.org/jira/browse/HIVE-20674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai reassigned HIVE-20674: - > TestJdbcWithMiniLlapArrow.testKillQuery fail frequently > --- > > Key: HIVE-20674 > URL: https://issues.apache.org/jira/browse/HIVE-20674 > Project: Hive > Issue Type: Bug >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20674) TestJdbcWithMiniLlapArrow.testKillQuery fail frequently
[
https://issues.apache.org/jira/browse/HIVE-20674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-20674:
--
Description:
{code}
java.sql.SQLException: org.apache.thrift.transport.TTransportException
at
org.apache.hive.jdbc.HiveStatement.closeStatementIfNeeded(HiveStatement.java:222)
at
org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:227)
at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:243)
at
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery(TestJdbcWithMiniLlapArrow.java:349)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
Caused by: org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
at
org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453)
at
org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
at
org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at
org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_CloseOperation(TCLIService.java:521)
at
org.apache.hive.service.rpc.thrift.TCLIService$Client.CloseOperation(TCLIService.java:508)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1669)
at com.sun.proxy.$Proxy131.CloseOperation(Unknown Source)
at
[jira] [Commented] (HIVE-20671) Hive Streaming has a broken dependency on metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636240#comment-16636240
]
Alexander Kolbasov commented on HIVE-20671:
---
The 03 patch also removes test dependency on metastore-server from
hive-streaming.
> Hive Streaming has a broken dependency on metastore-server
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20671.02.patch, HIVE-20671.03.patch,
> HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20671) Hive Streaming has a broken dependency on metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated HIVE-20671:
--
Attachment: HIVE-20671.03.patch
> Hive Streaming has a broken dependency on metastore-server
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20671.02.patch, HIVE-20671.03.patch,
> HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20671) Hive Streaming has a broken dependency on metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636236#comment-16636236
]
Alexander Kolbasov commented on HIVE-20671:
---
I tried running
{code:java}
mvn -DskipTests -Pitests clean test-compile
{code}
and it works for me. Are you seeing the same error?
> Hive Streaming has a broken dependency on metastore-server
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20671.02.patch, HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20575) Fix flaky connection metric tests
[
https://issues.apache.org/jira/browse/HIVE-20575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636231#comment-16636231
]
Hive QA commented on HIVE-20575:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14187/dev-support/hive-personality.sh
|
| git revision | master / 499539f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: itests/hive-unit U: itests/hive-unit |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14187/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix flaky connection metric tests
> -
>
> Key: HIVE-20575
> URL: https://issues.apache.org/jira/browse/HIVE-20575
> Project: Hive
> Issue Type: Test
> Components: Hive, Test
>Affects Versions: 4.0.0
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20575.01.patch, HIVE-20575.02.patch,
> HIVE-20575.03.patch, HIVE-20575.04.patch, HIVE-20575.05.patch,
> HIVE-20575.06.patch, HIVE-20575.07.patch, HIVE-20575.08.patch,
> HIVE-20575.09.patch, HIVE-20575.10.patch, HIVE-20575.11.patch
>
>
> TestHs2ConnectionMetricsHttp.testOpenConnectionMetrics() is flaky. We need to
> fix it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20609) Create SSD cache dir if it doesnt exist already
[ https://issues.apache.org/jira/browse/HIVE-20609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-20609: Resolution: Fixed Fix Version/s: (was: 3.0.1) 4.0.0 Status: Resolved (was: Patch Available) Pushed to master. Failure is known fragile test. Thanks Gergely! > Create SSD cache dir if it doesnt exist already > --- > > Key: HIVE-20609 > URL: https://issues.apache.org/jira/browse/HIVE-20609 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.0.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20609.01.patch, HIVE-20609.02.patch, > HIVE-20609.03.patch, HIVE-20609.04.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18945) Support "analyze table T"
[
https://issues.apache.org/jira/browse/HIVE-18945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636227#comment-16636227
]
Ashutosh Chauhan commented on HIVE-18945:
-
I beg to differ. {{analyze table T}} syntax will cause more confusion then
necessary. We have made {{analyze T for cols}} to gather basic stats too so
user just need to run one command. Having 2 syntax which achieves same thing is
not useful. Further, in longer run we may do more than collect stats in analyze
statement, e.g., some DBs has syntax like {{analyze table T validate}} which
performs bunch of validation on Table. On Hive, one such validation could be
making sure data files correspond to metadata w.r.t file format. If we ever
introduce such syntax in future, it will be puzzling for user that shortcut of
{{analyze T}} does stats collection or validation. So, IMHO we shall not add
this syntax.
> Support "analyze table T"
> -
>
> Key: HIVE-18945
> URL: https://issues.apache.org/jira/browse/HIVE-18945
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
>
> I think it would be good to have it behave the same as
> {code}
> analyze table T compute statistics for columns
> {code}
> this could help people who not yet know the different analyze commands ; to
> run the probably most appropriate one
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20647) HadoopVer was ignored in QTestUtil
[
https://issues.apache.org/jira/browse/HIVE-20647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636214#comment-16636214
]
Hive QA commented on HIVE-20647:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942117/HIVE-20647.7.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14186/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14186/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14186/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12942117/HIVE-20647.7.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942117 - PreCommit-HIVE-Build
> HadoopVer was ignored in QTestUtil
> --
>
> Key: HIVE-20647
> URL: https://issues.apache.org/jira/browse/HIVE-20647
> Project: Hive
> Issue Type: Bug
> Components: Test
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20647.1.patch, HIVE-20647.2.patch,
> HIVE-20647.3.patch, HIVE-20647.4.patch, HIVE-20647.5.patch,
> HIVE-20647.6.patch, HIVE-20647.7.patch
>
>
> * Removed obsolete hadoopVer from QTestUtil
> * Cleaned up QTestUtil, QTestArgumentsBuilder
> * Refactored AccumuloTestSetup, AccumuloQTestUtil due to findbugs violation:
> (UR_UNINIT_READ_CALLED_FROM_SUPER_CONSTRUCTOR: Uninitialized read of field
> method called from constructor of superclass)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20647) HadoopVer was ignored in QTestUtil
[
https://issues.apache.org/jira/browse/HIVE-20647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636213#comment-16636213
]
Hive QA commented on HIVE-20647:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942117/HIVE-20647.7.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15006 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.spark.TestSparkSessionTimeout.testMultiSparkSessionTimeout
(batchId=246)
org.apache.hive.jdbc.TestJdbcWithMiniHS2ErasureCoding.testExplainErasureCoding
(batchId=255)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14185/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14185/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14185/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942117 - PreCommit-HIVE-Build
> HadoopVer was ignored in QTestUtil
> --
>
> Key: HIVE-20647
> URL: https://issues.apache.org/jira/browse/HIVE-20647
> Project: Hive
> Issue Type: Bug
> Components: Test
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20647.1.patch, HIVE-20647.2.patch,
> HIVE-20647.3.patch, HIVE-20647.4.patch, HIVE-20647.5.patch,
> HIVE-20647.6.patch, HIVE-20647.7.patch
>
>
> * Removed obsolete hadoopVer from QTestUtil
> * Cleaned up QTestUtil, QTestArgumentsBuilder
> * Refactored AccumuloTestSetup, AccumuloQTestUtil due to findbugs violation:
> (UR_UNINIT_READ_CALLED_FROM_SUPER_CONSTRUCTOR: Uninitialized read of field
> method called from constructor of superclass)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20671) Hive Streaming has a broken dependency on metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636211#comment-16636211
]
Jaume M commented on HIVE-20671:
I still get a similar error with your patch [~akolb]
> Hive Streaming has a broken dependency on metastore-server
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20671.02.patch, HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20673) Vectorized map join fails with 'Unexpected column vector type STRUCT'
[
https://issues.apache.org/jira/browse/HIVE-20673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajkumar Singh updated HIVE-20673:
--
Summary: Vectorized map join fails with 'Unexpected column vector type
STRUCT' (was: Vectorized map join fails with Unexpected column vector type
STRUCT.)
> Vectorized map join fails with 'Unexpected column vector type STRUCT'
> -
>
> Key: HIVE-20673
> URL: https://issues.apache.org/jira/browse/HIVE-20673
> Project: Hive
> Issue Type: Bug
> Components: Hive, Transactions, Vectorization
>Affects Versions: 3.1.0
> Environment: hive-3, java-8
>Reporter: Rajkumar Singh
>Priority: Major
>
> update query on ACID table fails with the following exception.
>
> UPDATE census_clus SET name = 'updated name' where ssn=100 and EXISTS
> (select distinct ssn from census where ssn=census_clus.ssn);
> {code}
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:354)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 16 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected
> column vector type STRUCT
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorCopyRow.init(VectorCopyRow.java:302)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator.initializeOp(VectorMapJoinCommonOperator.java:419)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.initializeOp(VectorMapJoinGenerateResultOperator.java:115)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:572)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:524)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:335)
> {code}
> STEPS TO REPRODUCE
> {code}
> create table census(
> ssn int,
> name string,
> city string,
> email string)
> row format delimited
> fields terminated by ',';
> insert into census values(100,"raj","san jose","email");
> create table census_clus(
> ssn int,
> name string,
> city string,
> email string)
> clustered by (ssn) into 4 buckets stored as orc TBLPROPERTIES
> ('transactional'='true');
> insert into table census_clus select * from census;
> UPDATE census_clus SET name = 'updated name' where ssn=100 and EXISTS
> (select distinct ssn from census where ssn=census_clus.ssn);
> {code}
> looking at the exception it seems the join operator getting typeInfo
> incorrectly while doing join, _col6 seems to be of struct type.
> {code}
> 2018-10-02 22:22:23,392 [INFO] [TezChild] |exec.CommonJoinOperator|: JOIN
> struct<_col2:string,_col3:string,_col6:struct>
> totalsz = 3
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20673) Vectorized map join fails with Unexpected column vector type STRUCT.
[
https://issues.apache.org/jira/browse/HIVE-20673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajkumar Singh updated HIVE-20673:
--
Summary: Vectorized map join fails with Unexpected column vector type
STRUCT. (was: vectorized map join fail with Unexpected column vector type
STRUCT.)
> Vectorized map join fails with Unexpected column vector type STRUCT.
>
>
> Key: HIVE-20673
> URL: https://issues.apache.org/jira/browse/HIVE-20673
> Project: Hive
> Issue Type: Bug
> Components: Hive, Transactions, Vectorization
>Affects Versions: 3.1.0
> Environment: hive-3, java-8
>Reporter: Rajkumar Singh
>Priority: Major
>
> update query on ACID table fails with the following exception.
>
> UPDATE census_clus SET name = 'updated name' where ssn=100 and EXISTS
> (select distinct ssn from census where ssn=census_clus.ssn);
> {code}
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:354)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 16 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected
> column vector type STRUCT
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorCopyRow.init(VectorCopyRow.java:302)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator.initializeOp(VectorMapJoinCommonOperator.java:419)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.initializeOp(VectorMapJoinGenerateResultOperator.java:115)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:572)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:524)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:335)
> {code}
> STEPS TO REPRODUCE
> {code}
> create table census(
> ssn int,
> name string,
> city string,
> email string)
> row format delimited
> fields terminated by ',';
> insert into census values(100,"raj","san jose","email");
> create table census_clus(
> ssn int,
> name string,
> city string,
> email string)
> clustered by (ssn) into 4 buckets stored as orc TBLPROPERTIES
> ('transactional'='true');
> insert into table census_clus select * from census;
> UPDATE census_clus SET name = 'updated name' where ssn=100 and EXISTS
> (select distinct ssn from census where ssn=census_clus.ssn);
> {code}
> looking at the exception it seems the join operator getting typeInfo
> incorrectly while doing join, _col6 seems to be of struct type.
> {code}
> 2018-10-02 22:22:23,392 [INFO] [TezChild] |exec.CommonJoinOperator|: JOIN
> struct<_col2:string,_col3:string,_col6:struct>
> totalsz = 3
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20671) Hive Streaming has a broken dependency on metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636203#comment-16636203
]
Alexander Kolbasov commented on HIVE-20671:
---
Looks like it isn't needed and we can remove it. Since this is a test-only
dependency it doesn't matter so much if it is there or not.
> Hive Streaming has a broken dependency on metastore-server
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20671.02.patch, HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20671) Hive Streaming has a broken dependency on metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636196#comment-16636196
]
Jaume M commented on HIVE-20671:
Do we need the {{hive-standalone-metastore-server}} test dependency in the
hive-streaming pom?
> Hive Streaming has a broken dependency on metastore-server
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20671.02.patch, HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20671) Hive Streaming has a broken dependency on metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636195#comment-16636195
]
Alexander Kolbasov commented on HIVE-20671:
---
Transfered to me.
> Hive Streaming has a broken dependency on metastore-server
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20671.02.patch, HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20671) Hive Streaming has a broken dependency on metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated HIVE-20671:
--
Attachment: HIVE-20671.02.patch
> Hive Streaming has a broken dependency on metastore-server
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20671.02.patch, HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20671) Hive Streaming has a broken dependency on metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov reassigned HIVE-20671:
-
Assignee: Alexander Kolbasov (was: Jaume M)
> Hive Streaming has a broken dependency on metastore-server
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20671) Hive Streaming has a broken dependency on metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636193#comment-16636193
]
Jaume M commented on HIVE-20671:
I agree it's a better solution [~akolb], feel free to assign the ticket to
yourself.
> Hive Streaming has a broken dependency on metastore-server
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20647) HadoopVer was ignored in QTestUtil
[
https://issues.apache.org/jira/browse/HIVE-20647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636189#comment-16636189
]
Hive QA commented on HIVE-20647:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} itests/util in master has 52 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} itests/util: The patch generated 0 new + 148
unchanged - 17 fixed = 148 total (was 165) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} itests/util generated 0 new + 48 unchanged - 4 fixed
= 48 total (was 52) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} hive-unit in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14185/dev-support/hive-personality.sh
|
| git revision | master / 499539f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: itests/util itests/hive-unit U: itests |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14185/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HadoopVer was ignored in QTestUtil
> --
>
> Key: HIVE-20647
> URL: https://issues.apache.org/jira/browse/HIVE-20647
> Project: Hive
> Issue Type: Bug
> Components: Test
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20647.1.patch, HIVE-20647.2.patch,
> HIVE-20647.3.patch, HIVE-20647.4.patch, HIVE-20647.5.patch,
> HIVE-20647.6.patch, HIVE-20647.7.patch
>
>
> * Removed obsolete hadoopVer from QTestUtil
> * Cleaned up QTestUtil, QTestArgumentsBuilder
> * Refactored AccumuloTestSetup, AccumuloQTestUtil due to findbugs violation:
> (UR_UNINIT_READ_CALLED_FROM_SUPER_CONSTRUCTOR: Uninitialized read of field
[jira] [Commented] (HIVE-20610) TestDbNotificationListener should not use /tmp directory
[
https://issues.apache.org/jira/browse/HIVE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636183#comment-16636183
]
Andrew Sherman commented on HIVE-20610:
---
+1 LGTM pending test results
> TestDbNotificationListener should not use /tmp directory
>
>
> Key: HIVE-20610
> URL: https://issues.apache.org/jira/browse/HIVE-20610
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0, 4.0.0
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20610.1.patch, HIVE-20610.2.patch
>
>
> Using /tmp directory creates exceptions for tests like dropTable :
> {code:java}
> 2018-09-19T06:42:04,818 INFO [main] metastore.HiveMetaStore: 0: drop_table :
> tbl=hive.default.droptbl
> 2018-09-19T06:42:04,819 INFO [main] HiveMetaStore.audit: ugi=hiveptest
> ip=unknown-ip-addr cmd=drop_table : tbl=hive.default.droptbl
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.ICE-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.XIM-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.X11-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/hsperfdata_root]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.font-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.Test-unix]: it still exists.
> 2018-09-19T06:42:05,072 ERROR [main] utils.FileUtils: Failed to delete
> file:/tmp
> 2018-09-19T06:42:05,072 ERROR [main] utils.MetaStoreUtils: Got exception:
> org.apache.hadoop.hive.metastore.api.MetaException Unable to delete
> directory: file:/tmp
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to delete
> directory: file:/tmp
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreFsImpl.deleteDir(HiveMetaStoreFsImpl.java:45)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:365)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:353)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.deleteTableData(HiveMetaStore.java:2562)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:2523)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:2685)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_102]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_102]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy33.drop_table_with_environment_context(Unknown
> Source) [?:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.drop_table_with_environment_context(HiveMetaStoreClient.java:3204)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1492)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1432)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropTable(TestDbNotificationListener.java:522)
> [test-classes/:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]{code}
>
>
--
This message was sent by Atlassian JIRA
[jira] [Comment Edited] (HIVE-20671) Hive Streaming has a broken dependency on metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636182#comment-16636182
]
Alexander Kolbasov edited comment on HIVE-20671 at 10/2/18 10:07 PM:
-
I tried this and it now compiles fine. Let me know if you need my patch.
was (Author: akolb):
I tried this and it now compiles fine.
> Hive Streaming has a broken dependency on metastore-server
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20671) Hive Streaming has a broken dependency on metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636182#comment-16636182
]
Alexander Kolbasov commented on HIVE-20671:
---
I tried this and it now compiles fine.
> Hive Streaming has a broken dependency on metastore-server
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20671) Hive Streaming has a broken dependency on metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated HIVE-20671:
--
Summary: Hive Streaming has a broken dependency on metastore-server (was:
Change pom dependency scope of hive-standalone-metastore-server to compile)
> Hive Streaming has a broken dependency on metastore-server
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20671) Change pom dependency scope of hive-standalone-metastore-server to compile
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636179#comment-16636179
]
Alexander Kolbasov commented on HIVE-20671:
---
I think it is better to move LockComponentBuilder to metastore-common
> Change pom dependency scope of hive-standalone-metastore-server to compile
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20610) TestDbNotificationListener should not use /tmp directory
[
https://issues.apache.org/jira/browse/HIVE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636173#comment-16636173
]
Bharathkrishna Guruvayoor Murali commented on HIVE-20610:
-
Adding patch version 2 addressing review comments.
> TestDbNotificationListener should not use /tmp directory
>
>
> Key: HIVE-20610
> URL: https://issues.apache.org/jira/browse/HIVE-20610
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0, 4.0.0
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20610.1.patch, HIVE-20610.2.patch
>
>
> Using /tmp directory creates exceptions for tests like dropTable :
> {code:java}
> 2018-09-19T06:42:04,818 INFO [main] metastore.HiveMetaStore: 0: drop_table :
> tbl=hive.default.droptbl
> 2018-09-19T06:42:04,819 INFO [main] HiveMetaStore.audit: ugi=hiveptest
> ip=unknown-ip-addr cmd=drop_table : tbl=hive.default.droptbl
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.ICE-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.XIM-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.X11-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/hsperfdata_root]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.font-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.Test-unix]: it still exists.
> 2018-09-19T06:42:05,072 ERROR [main] utils.FileUtils: Failed to delete
> file:/tmp
> 2018-09-19T06:42:05,072 ERROR [main] utils.MetaStoreUtils: Got exception:
> org.apache.hadoop.hive.metastore.api.MetaException Unable to delete
> directory: file:/tmp
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to delete
> directory: file:/tmp
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreFsImpl.deleteDir(HiveMetaStoreFsImpl.java:45)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:365)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:353)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.deleteTableData(HiveMetaStore.java:2562)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:2523)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:2685)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_102]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_102]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy33.drop_table_with_environment_context(Unknown
> Source) [?:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.drop_table_with_environment_context(HiveMetaStoreClient.java:3204)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1492)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1432)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropTable(TestDbNotificationListener.java:522)
> [test-classes/:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]{code}
>
>
[jira] [Updated] (HIVE-20610) TestDbNotificationListener should not use /tmp directory
[
https://issues.apache.org/jira/browse/HIVE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-20610:
Attachment: HIVE-20610.2.patch
> TestDbNotificationListener should not use /tmp directory
>
>
> Key: HIVE-20610
> URL: https://issues.apache.org/jira/browse/HIVE-20610
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0, 4.0.0
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20610.1.patch, HIVE-20610.2.patch
>
>
> Using /tmp directory creates exceptions for tests like dropTable :
> {code:java}
> 2018-09-19T06:42:04,818 INFO [main] metastore.HiveMetaStore: 0: drop_table :
> tbl=hive.default.droptbl
> 2018-09-19T06:42:04,819 INFO [main] HiveMetaStore.audit: ugi=hiveptest
> ip=unknown-ip-addr cmd=drop_table : tbl=hive.default.droptbl
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.ICE-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.XIM-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.X11-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/hsperfdata_root]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.font-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.Test-unix]: it still exists.
> 2018-09-19T06:42:05,072 ERROR [main] utils.FileUtils: Failed to delete
> file:/tmp
> 2018-09-19T06:42:05,072 ERROR [main] utils.MetaStoreUtils: Got exception:
> org.apache.hadoop.hive.metastore.api.MetaException Unable to delete
> directory: file:/tmp
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to delete
> directory: file:/tmp
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreFsImpl.deleteDir(HiveMetaStoreFsImpl.java:45)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:365)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:353)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.deleteTableData(HiveMetaStore.java:2562)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:2523)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:2685)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_102]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_102]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy33.drop_table_with_environment_context(Unknown
> Source) [?:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.drop_table_with_environment_context(HiveMetaStoreClient.java:3204)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1492)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1432)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropTable(TestDbNotificationListener.java:522)
> [test-classes/:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]{code}
>
>
--
This message was sent by Atlassian JIRA
[jira] [Commented] (HIVE-20590) Allow merge statement to have column schema
[
https://issues.apache.org/jira/browse/HIVE-20590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636170#comment-16636170
]
Hive QA commented on HIVE-20590:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942109/HIVE-20590.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15006 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery (batchId=252)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14184/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14184/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14184/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942109 - PreCommit-HIVE-Build
> Allow merge statement to have column schema
> ---
>
> Key: HIVE-20590
> URL: https://issues.apache.org/jira/browse/HIVE-20590
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Vineet Garg
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-20590.01.patch
>
>
> Currently MERGE statement doesn't let user specify column schema with INSERT
> statements, therefore DEFAULT constraint are not applicable with it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20672) Logging thread in LlapTaskSchedulerService should report every fixed interval
[ https://issues.apache.org/jira/browse/HIVE-20672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636155#comment-16636155 ] Jaume M commented on HIVE-20672: Currently looks like the thread reports only once: https://github.com/apache/hive/blob/499539fb62b4aa8908d517816f27a0e4997555a7/llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java#L735 > Logging thread in LlapTaskSchedulerService should report every fixed interval > - > > Key: HIVE-20672 > URL: https://issues.apache.org/jira/browse/HIVE-20672 > Project: Hive > Issue Type: Bug >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20672.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20672) Logging thread in LlapTaskSchedulerService should report every fixed interval
[ https://issues.apache.org/jira/browse/HIVE-20672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-20672: --- Attachment: HIVE-20672.1.patch Status: Patch Available (was: Open) > Logging thread in LlapTaskSchedulerService should report every fixed interval > - > > Key: HIVE-20672 > URL: https://issues.apache.org/jira/browse/HIVE-20672 > Project: Hive > Issue Type: Bug >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20672.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20672) Logging thread in LlapTaskSchedulerService should report every fixed interval
[ https://issues.apache.org/jira/browse/HIVE-20672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M reassigned HIVE-20672: -- Assignee: Jaume M > Logging thread in LlapTaskSchedulerService should report every fixed interval > - > > Key: HIVE-20672 > URL: https://issues.apache.org/jira/browse/HIVE-20672 > Project: Hive > Issue Type: Bug >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20590) Allow merge statement to have column schema
[
https://issues.apache.org/jira/browse/HIVE-20590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636126#comment-16636126
]
Hive QA commented on HIVE-20590:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
1s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 12 new + 150 unchanged - 8
fixed = 162 total (was 158) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14184/dev-support/hive-personality.sh
|
| git revision | master / 499539f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14184/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14184/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Allow merge statement to have column schema
> ---
>
> Key: HIVE-20590
> URL: https://issues.apache.org/jira/browse/HIVE-20590
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Vineet Garg
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-20590.01.patch
>
>
> Currently MERGE statement doesn't let user specify column schema with INSERT
> statements, therefore DEFAULT constraint are not applicable with it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20255) Review LevelOrderWalker.java
[
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636100#comment-16636100
]
BELUGA BEHR commented on HIVE-20255:
[~ngangam] Are you able to assist me with this? :)
> Review LevelOrderWalker.java
>
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.1.patch, HIVE-20255.2.patch,
> HIVE-20255.3.patch, HIVE-20255.4.patch, HIVE-20255.5.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
> if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some
> real work, so do not throw away the work after checking for null. Simply
> call once and store the results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17300) WebUI query plan graphs
[
https://issues.apache.org/jira/browse/HIVE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636088#comment-16636088
]
Hive QA commented on HIVE-17300:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942094/HIVE-17300.10.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15008 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14183/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14183/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14183/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942094 - PreCommit-HIVE-Build
> WebUI query plan graphs
> ---
>
> Key: HIVE-17300
> URL: https://issues.apache.org/jira/browse/HIVE-17300
> Project: Hive
> Issue Type: Sub-task
> Components: Web UI
>Affects Versions: 4.0.0
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: beginner, features, patch
> Attachments: HIVE-17300.10.patch, HIVE-17300.10.patch,
> HIVE-17300.10.patch, HIVE-17300.3.patch, HIVE-17300.4.patch,
> HIVE-17300.5.patch, HIVE-17300.6.patch, HIVE-17300.7.patch,
> HIVE-17300.7.patch, HIVE-17300.8.patch, HIVE-17300.8.patch,
> HIVE-17300.8.patch, HIVE-17300.8.patch, HIVE-17300.9.patch, HIVE-17300.patch,
> complete_success.png, full_mapred_stats.png, graph_with_mapred_stats.png,
> last_stage_error.png, last_stage_running.png, non_mapred_task_selected.png
>
>
> Hi all,
> I’m working on a feature of the Hive WebUI Query Plan tab that would provide
> the option to display the query plan as a nice graph (scroll down for
> screenshots). If you click on one of the graph’s stages, the plan for that
> stage appears as text below.
> Stages are color-coded if they have a status (Success, Error, Running), and
> the rest are grayed out. Coloring is based on status already available in the
> WebUI, under the Stages tab.
> There is an additional option to display stats for MapReduce tasks. This
> includes the job’s ID, tracking URL (where the logs are found), and mapper
> and reducer numbers/progress, among other info.
> The library I’m using for the graph is called vis.js (http://visjs.org/). It
> has an Apache license, and the only necessary file to be included from this
> library is about 700 KB.
> I tried to keep server-side changes minimal, and graph generation is taken
> care of by the client. Plans with more than a given number of stages
> (default: 25) won't be displayed in order to preserve resources.
> I’d love to hear any and all input from the community about this feature: do
> you think it’s useful, and is there anything important I’m missing?
> Thanks,
> Karen Coppage
> Review request: https://reviews.apache.org/r/61663/
> Any input is welcome!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
