[jira] [Updated] (HIVE-20878) Hive Runner for Unit tests with Hive JDBC standanlone jar issue with log4j slf4j
[
https://issues.apache.org/jira/browse/HIVE-20878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carsten Steckel updated HIVE-20878:
---
Description:
I have an standalone java application using the hive-jdbc-standalone.jar to
create and drop databases, tables, indexes, views in a hive db via jdbc
connection. I want to unit test the executed DDL operations via hive runner.
The hive jdbc standalone jar brings a lot of dependencies (and shades them),
but that causes issues with using application context where there is logging
infrastructure configure and in place.
{noformat}
java.lang.IncompatibleClassChangeError: Class
org.apache.logging.slf4j.Log4jLoggerFactory does not implement the requested
interface org.apache.hive.org.slf4j.ILoggerFactory
at
org.apache.hive.org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285){noformat}
How to properly setup dependencies or exclusions? Should't a library like
hive-jdbc leave logging to the "surrounging" application context? Why the
dependency to logging?
was:
I have an standalone java application using the hive-jdbc-standalone.jar to
create and drop databases, tables, indexes, views in a hive db via jdbc
connection. I want to unit test the executed DDL operations via hive runner.
The hive jdbc standalone jar brings a lot of dependencies (and shades them),
but that causes issues with using application context where there is logging
infrastructure configure and in place.
{noformat}
java.lang.IncompatibleClassChangeError: Class
org.apache.logging.slf4j.Log4jLoggerFactory does not implement the requested
interface org.apache.hive.org.slf4j.ILoggerFactory
at
org.apache.hive.org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285){noformat}
How to properly setup dependencies or exclusions? Should't a library like
hive-jdbc leave logging to the "surrounging" application context? Why the
dependency to logging?
Maybe related to HIVE-20877
> Hive Runner for Unit tests with Hive JDBC standanlone jar issue with log4j
> slf4j
>
>
> Key: HIVE-20878
> URL: https://issues.apache.org/jira/browse/HIVE-20878
> Project: Hive
> Issue Type: Bug
> Components: Hive, JDBC
>Affects Versions: 3.1.0, 3.1.1
> Environment: hive 3.1.1 and hive 3.1.0
> backend hadoop 2.9.1
> hive runner https://github.com/klarna/HiveRunner
>Reporter: Carsten Steckel
>Priority: Major
>
> I have an standalone java application using the hive-jdbc-standalone.jar to
> create and drop databases, tables, indexes, views in a hive db via jdbc
> connection. I want to unit test the executed DDL operations via hive runner.
> The hive jdbc standalone jar brings a lot of dependencies (and shades them),
> but that causes issues with using application context where there is logging
> infrastructure configure and in place.
> {noformat}
> java.lang.IncompatibleClassChangeError: Class
> org.apache.logging.slf4j.Log4jLoggerFactory does not implement the requested
> interface org.apache.hive.org.slf4j.ILoggerFactory
> at
> org.apache.hive.org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285){noformat}
>
> How to properly setup dependencies or exclusions? Should't a library like
> hive-jdbc leave logging to the "surrounging" application context? Why the
> dependency to logging?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20886) Fix NPE: GenericUDFLower
[
https://issues.apache.org/jira/browse/HIVE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-20886:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Thanks [~gopalv]. Committed to master.
> Fix NPE: GenericUDFLower
>
>
> Key: HIVE-20886
> URL: https://issues.apache.org/jira/browse/HIVE-20886
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 2.0.0
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Trivial
> Fix For: 4.0.0
>
> Attachments: HIVE-20886.1.patch
>
>
> {noformat}
> create table if not exists test1(uuid array);
> select lower(uuid) from test1;
> Error: Error while compiling statement: FAILED: NullPointerException null
> (state=42000,code=4)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20734) Beeline: When beeline-site.xml is and hive CLI redirects to beeline, it should use the system username/dummy password instead of prompting for one
[
https://issues.apache.org/jira/browse/HIVE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680986#comment-16680986
]
Hive QA commented on HIVE-20734:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947344/HIVE-20734.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15527 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14827/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14827/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14827/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947344 - PreCommit-HIVE-Build
> Beeline: When beeline-site.xml is and hive CLI redirects to beeline, it
> should use the system username/dummy password instead of prompting for one
> --
>
> Key: HIVE-20734
> URL: https://issues.apache.org/jira/browse/HIVE-20734
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.1.0
>Reporter: Vaibhav Gumashta
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-20734.1.patch, HIVE-20734.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20881) Constant propagation oversimplifies projections
[
https://issues.apache.org/jira/browse/HIVE-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680958#comment-16680958
]
Ashutosh Chauhan edited comment on HIVE-20881 at 11/9/18 6:55 AM:
--
I see. If Calcite is already handling FIL appropriately. +1
was (Author: ashutoshc):
I see. +1
> Constant propagation oversimplifies projections
> ---
>
> Key: HIVE-20881
> URL: https://issues.apache.org/jira/browse/HIVE-20881
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20881.01.patch
>
>
> {code:java}
> create table cx2(bool1 boolean);
> insert into cx2 values (true),(false),(null);
> set hive.cbo.enable=true;
> select bool1 IS TRUE OR (cast(NULL as boolean) AND bool1 IS NOT TRUE AND
> bool1 IS NOT FALSE) from cx2;
> ++
> | _c0 |
> ++
> | true |
> | false |
> | NULL |
> ++
> set hive.cbo.enable=false;
> select bool1 IS TRUE OR (cast(NULL as boolean) AND bool1 IS NOT TRUE AND
> bool1 IS NOT FALSE) from cx2;
> +---+
> | _c0 |
> +---+
> | true |
> | NULL |
> | NULL |
> +---+
> {code}
> from explain it seems the expression was simplified to: {{(_col0 is true or
> null)}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20881) Constant propagation oversimplifies projections
[
https://issues.apache.org/jira/browse/HIVE-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680958#comment-16680958
]
Ashutosh Chauhan commented on HIVE-20881:
-
I see. +1
> Constant propagation oversimplifies projections
> ---
>
> Key: HIVE-20881
> URL: https://issues.apache.org/jira/browse/HIVE-20881
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20881.01.patch
>
>
> {code:java}
> create table cx2(bool1 boolean);
> insert into cx2 values (true),(false),(null);
> set hive.cbo.enable=true;
> select bool1 IS TRUE OR (cast(NULL as boolean) AND bool1 IS NOT TRUE AND
> bool1 IS NOT FALSE) from cx2;
> ++
> | _c0 |
> ++
> | true |
> | false |
> | NULL |
> ++
> set hive.cbo.enable=false;
> select bool1 IS TRUE OR (cast(NULL as boolean) AND bool1 IS NOT TRUE AND
> bool1 IS NOT FALSE) from cx2;
> +---+
> | _c0 |
> +---+
> | true |
> | NULL |
> | NULL |
> +---+
> {code}
> from explain it seems the expression was simplified to: {{(_col0 is true or
> null)}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20896) CachedStore fail to cache stats in multiple code paths
[ https://issues.apache.org/jira/browse/HIVE-20896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20896: -- Attachment: HIVE-20896.0.patch > CachedStore fail to cache stats in multiple code paths > -- > > Key: HIVE-20896 > URL: https://issues.apache.org/jira/browse/HIVE-20896 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20896.0.patch > > > Bunch of issues discovered in CachedStore to keep up column statistics: > 1. The criteria for partition/non-partition is wrong > (table.isSetPartitionKeys() is always true) > 2. In update(), partition column stats are removed when populate table basic > stats > 3. Dirty flags are true right after prewarm(), so the first update() does not > do anything > 4. Could invoke cacheLock without holding the lock, which results a freeze in > update() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20893) BloomK Filter probing method is not thread safe
[ https://issues.apache.org/jira/browse/HIVE-20893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V reassigned HIVE-20893: -- Assignee: (was: Gopal V) > BloomK Filter probing method is not thread safe > --- > > Key: HIVE-20893 > URL: https://issues.apache.org/jira/browse/HIVE-20893 > Project: Hive > Issue Type: Bug > Components: storage-api >Reporter: slim bouguerra >Priority: Major > Attachments: HIVE-20893.WIP.patch > > > As far i can tell this is not an issue for Hive yet (most of the usage of > probing seems to be done by one thread at a time) but it is an issue of other > users like Druid as per the following > issue.[https://github.com/apache/incubator-druid/issues/6546] > The fix is proposed by the author of > [https://github.com/apache/incubator-druid/pull/6584] is to make couple of > local fields as ThreadLocals. > Idea looks good to me and doesn't have any perf drawbacks. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20896) CachedStore fail to cache stats in multiple code paths
[ https://issues.apache.org/jira/browse/HIVE-20896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680947#comment-16680947 ] Daniel Dai commented on HIVE-20896: --- Initial patch. Will follow up with tests. > CachedStore fail to cache stats in multiple code paths > -- > > Key: HIVE-20896 > URL: https://issues.apache.org/jira/browse/HIVE-20896 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20896.0.patch > > > Bunch of issues discovered in CachedStore to keep up column statistics: > 1. The criteria for partition/non-partition is wrong > (table.isSetPartitionKeys() is always true) > 2. In update(), partition column stats are removed when populate table basic > stats > 3. Dirty flags are true right after prewarm(), so the first update() does not > do anything > 4. Could invoke cacheLock without holding the lock, which results a freeze in > update() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20896) CachedStore fail to cache stats in multiple code paths
[ https://issues.apache.org/jira/browse/HIVE-20896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20896: -- Status: Patch Available (was: Open) > CachedStore fail to cache stats in multiple code paths > -- > > Key: HIVE-20896 > URL: https://issues.apache.org/jira/browse/HIVE-20896 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20896.0.patch > > > Bunch of issues discovered in CachedStore to keep up column statistics: > 1. The criteria for partition/non-partition is wrong > (table.isSetPartitionKeys() is always true) > 2. In update(), partition column stats are removed when populate table basic > stats > 3. Dirty flags are true right after prewarm(), so the first update() does not > do anything > 4. Could invoke cacheLock without holding the lock, which results a freeze in > update() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20881) Constant propagation oversimplifies projections
[
https://issues.apache.org/jira/browse/HIVE-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680940#comment-16680940
]
Zoltan Haindrich commented on HIVE-20881:
-
Yes; but Calcite is already prepared to do that kinda of simplifications.
The caveat with translating NULLs to false is that beyond a not or an IS
operator it may not do that even for FIL:
{code}
( (col1 and null) is null )
( NOT ( col1 and null) )
{code}
It will need some more logic to ensure that it doesn't get break the above
cases - but I would like to wait for a ptest result and see how much difference
it would make - since I think Calcite should also have been taken care of these
cases.
> Constant propagation oversimplifies projections
> ---
>
> Key: HIVE-20881
> URL: https://issues.apache.org/jira/browse/HIVE-20881
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20881.01.patch
>
>
> {code:java}
> create table cx2(bool1 boolean);
> insert into cx2 values (true),(false),(null);
> set hive.cbo.enable=true;
> select bool1 IS TRUE OR (cast(NULL as boolean) AND bool1 IS NOT TRUE AND
> bool1 IS NOT FALSE) from cx2;
> ++
> | _c0 |
> ++
> | true |
> | false |
> | NULL |
> ++
> set hive.cbo.enable=false;
> select bool1 IS TRUE OR (cast(NULL as boolean) AND bool1 IS NOT TRUE AND
> bool1 IS NOT FALSE) from cx2;
> +---+
> | _c0 |
> +---+
> | true |
> | NULL |
> | NULL |
> +---+
> {code}
> from explain it seems the expression was simplified to: {{(_col0 is true or
> null)}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20893) BloomK Filter probing method is not thread safe
[ https://issues.apache.org/jira/browse/HIVE-20893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-20893: --- Attachment: HIVE-20893.WIP.patch > BloomK Filter probing method is not thread safe > --- > > Key: HIVE-20893 > URL: https://issues.apache.org/jira/browse/HIVE-20893 > Project: Hive > Issue Type: Bug > Components: storage-api >Reporter: slim bouguerra >Assignee: Gopal V >Priority: Major > Attachments: HIVE-20893.WIP.patch > > > As far i can tell this is not an issue for Hive yet (most of the usage of > probing seems to be done by one thread at a time) but it is an issue of other > users like Druid as per the following > issue.[https://github.com/apache/incubator-druid/issues/6546] > The fix is proposed by the author of > [https://github.com/apache/incubator-druid/pull/6584] is to make couple of > local fields as ThreadLocals. > Idea looks good to me and doesn't have any perf drawbacks. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20893) BloomK Filter probing method is not thread safe
[ https://issues.apache.org/jira/browse/HIVE-20893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680942#comment-16680942 ] Gopal V commented on HIVE-20893: [~cwylie]: I think I munged up that change - attached a proper patch, instead of a comment > BloomK Filter probing method is not thread safe > --- > > Key: HIVE-20893 > URL: https://issues.apache.org/jira/browse/HIVE-20893 > Project: Hive > Issue Type: Bug > Components: storage-api >Reporter: slim bouguerra >Priority: Major > Attachments: HIVE-20893.WIP.patch > > > As far i can tell this is not an issue for Hive yet (most of the usage of > probing seems to be done by one thread at a time) but it is an issue of other > users like Druid as per the following > issue.[https://github.com/apache/incubator-druid/issues/6546] > The fix is proposed by the author of > [https://github.com/apache/incubator-druid/pull/6584] is to make couple of > local fields as ThreadLocals. > Idea looks good to me and doesn't have any perf drawbacks. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20893) BloomK Filter probing method is not thread safe
[ https://issues.apache.org/jira/browse/HIVE-20893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V reassigned HIVE-20893: -- Assignee: Gopal V > BloomK Filter probing method is not thread safe > --- > > Key: HIVE-20893 > URL: https://issues.apache.org/jira/browse/HIVE-20893 > Project: Hive > Issue Type: Bug > Components: storage-api >Reporter: slim bouguerra >Assignee: Gopal V >Priority: Major > Attachments: HIVE-20893.WIP.patch > > > As far i can tell this is not an issue for Hive yet (most of the usage of > probing seems to be done by one thread at a time) but it is an issue of other > users like Druid as per the following > issue.[https://github.com/apache/incubator-druid/issues/6546] > The fix is proposed by the author of > [https://github.com/apache/incubator-druid/pull/6584] is to make couple of > local fields as ThreadLocals. > Idea looks good to me and doesn't have any perf drawbacks. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20896) CachedStore fail to cache stats in multiple code paths
[ https://issues.apache.org/jira/browse/HIVE-20896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai reassigned HIVE-20896: - > CachedStore fail to cache stats in multiple code paths > -- > > Key: HIVE-20896 > URL: https://issues.apache.org/jira/browse/HIVE-20896 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > > Bunch of issues discovered in CachedStore to keep up column statistics: > 1. The criteria for partition/non-partition is wrong > (table.isSetPartitionKeys() is always true) > 2. In update(), partition column stats are removed when populate table basic > stats > 3. Dirty flags are true right after prewarm(), so the first update() does not > do anything > 4. Could invoke cacheLock without holding the lock, which results a freeze in > update() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20734) Beeline: When beeline-site.xml is and hive CLI redirects to beeline, it should use the system username/dummy password instead of prompting for one
[
https://issues.apache.org/jira/browse/HIVE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680930#comment-16680930
]
Hive QA commented on HIVE-20734:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14827/dev-support/hive-personality.sh
|
| git revision | master / 5aac805 |
| modules | C: . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14827/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Beeline: When beeline-site.xml is and hive CLI redirects to beeline, it
> should use the system username/dummy password instead of prompting for one
> --
>
> Key: HIVE-20734
> URL: https://issues.apache.org/jira/browse/HIVE-20734
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.1.0
>Reporter: Vaibhav Gumashta
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-20734.1.patch, HIVE-20734.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20826) Enhance HiveSemiJoin rule to convert join + group by on left side to Left Semi Join
[
https://issues.apache.org/jira/browse/HIVE-20826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680926#comment-16680926
]
Ashutosh Chauhan commented on HIVE-20826:
-
+1 Patch looks good. Can you also update RB with updated golden files too.
> Enhance HiveSemiJoin rule to convert join + group by on left side to Left
> Semi Join
> ---
>
> Key: HIVE-20826
> URL: https://issues.apache.org/jira/browse/HIVE-20826
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20826.1.patch, HIVE-20826.2.patch,
> HIVE-20826.3.patch
>
>
> Currently HiveSemiJoin rule looks for pattern where group by is on right side.
> We can convert joins which have group by on left side (assuming group by keys
> are same as join keys and none of the columns are being projected from left
> side) to LEFT SEMI JOIN by swapping the inputs. e.g. queries such as:
> {code:sql}
> explain select pp.p_partkey from (select distinct p_name from part) p join
> part pp on pp.p_name = p.p_name;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20873) Use Murmur hash for VectorHashKeyWrapperTwoLong to reduce hash collision
[
https://issues.apache.org/jira/browse/HIVE-20873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680922#comment-16680922
]
Hive QA commented on HIVE-20873:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947335/HIVE-20873.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15531 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamptz_2]
(batchId=85)
org.apache.hive.jdbc.TestJdbcDriver2.testSelectExecAsync2 (batchId=259)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14825/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14825/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14825/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947335 - PreCommit-HIVE-Build
> Use Murmur hash for VectorHashKeyWrapperTwoLong to reduce hash collision
>
>
> Key: HIVE-20873
> URL: https://issues.apache.org/jira/browse/HIVE-20873
> Project: Hive
> Issue Type: Improvement
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20873.1.patch, HIVE-20873.2.patch,
> HIVE-20873.3.patch
>
>
> VectorHashKeyWrapperTwoLong is implemented with few bit shift operators and
> XOR operators for short computation time, but more hash collision. Group by
> operations become very slow on large data sets. It needs Murmur hash or a
> better hash function for less hash collision.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20886) Fix NPE: GenericUDFLower
[
https://issues.apache.org/jira/browse/HIVE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680917#comment-16680917
]
Gopal V commented on HIVE-20886:
LGTM - +1
> Fix NPE: GenericUDFLower
>
>
> Key: HIVE-20886
> URL: https://issues.apache.org/jira/browse/HIVE-20886
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 2.0.0
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Trivial
> Attachments: HIVE-20886.1.patch
>
>
> {noformat}
> create table if not exists test1(uuid array);
> select lower(uuid) from test1;
> Error: Error while compiling statement: FAILED: NullPointerException null
> (state=42000,code=4)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20845) Fix TestJdbcWithDBTokenStoreNoDoAs flakiness
[ https://issues.apache.org/jira/browse/HIVE-20845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-20845: -- Attachment: HIVE-20845.2.patch > Fix TestJdbcWithDBTokenStoreNoDoAs flakiness > > > Key: HIVE-20845 > URL: https://issues.apache.org/jira/browse/HIVE-20845 > Project: Hive > Issue Type: Test >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-20845.2.patch, HIVE-20845.patch > > > Previously did a dirty fix for TestJdbcWithDBTokenStoreNoDoAs and > TestJdbcWithDBTokenStore > Found out the issue is that we do not wait enough for HS2 to come up. > Need to fix in MiniHS2.waitForStartup() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20873) Use Murmur hash for VectorHashKeyWrapperTwoLong to reduce hash collision
[
https://issues.apache.org/jira/browse/HIVE-20873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680894#comment-16680894
]
Hive QA commented on HIVE-20873:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
20s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
51s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14825/dev-support/hive-personality.sh
|
| git revision | master / 5aac805 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: storage-api common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14825/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Use Murmur hash for VectorHashKeyWrapperTwoLong to reduce hash collision
>
>
> Key: HIVE-20873
> URL: https://issues.apache.org/jira/browse/HIVE-20873
> Project: Hive
> Issue Type: Improvement
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20873.1.patch, HIVE-20873.2.patch,
> HIVE-20873.3.patch
>
>
> VectorHashKeyWrapperTwoLong is implemented with few bit shift operators and
> XOR operators for short computation time, but more hash collision. Group by
> operations become very slow on large data sets. It needs Murmur hash or a
> better hash function for less hash collision.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20880) Update default value for hive.stats.filter.in.min.ratio
[ https://issues.apache.org/jira/browse/HIVE-20880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-20880: Status: Patch Available (was: Open) > Update default value for hive.stats.filter.in.min.ratio > --- > > Key: HIVE-20880 > URL: https://issues.apache.org/jira/browse/HIVE-20880 > Project: Hive > Issue Type: Improvement > Components: Configuration >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Major > Attachments: HIVE-20880.2.patch, HIVE-20880.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20880) Update default value for hive.stats.filter.in.min.ratio
[ https://issues.apache.org/jira/browse/HIVE-20880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-20880: Status: Open (was: Patch Available) > Update default value for hive.stats.filter.in.min.ratio > --- > > Key: HIVE-20880 > URL: https://issues.apache.org/jira/browse/HIVE-20880 > Project: Hive > Issue Type: Improvement > Components: Configuration >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Major > Attachments: HIVE-20880.2.patch, HIVE-20880.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20880) Update default value for hive.stats.filter.in.min.ratio
[ https://issues.apache.org/jira/browse/HIVE-20880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-20880: Attachment: HIVE-20880.2.patch > Update default value for hive.stats.filter.in.min.ratio > --- > > Key: HIVE-20880 > URL: https://issues.apache.org/jira/browse/HIVE-20880 > Project: Hive > Issue Type: Improvement > Components: Configuration >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Major > Attachments: HIVE-20880.2.patch, HIVE-20880.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20886) Fix NPE: GenericUDFLower
[
https://issues.apache.org/jira/browse/HIVE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680872#comment-16680872
]
Hive QA commented on HIVE-20886:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947332/HIVE-20886.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15527 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14824/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14824/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14824/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947332 - PreCommit-HIVE-Build
> Fix NPE: GenericUDFLower
>
>
> Key: HIVE-20886
> URL: https://issues.apache.org/jira/browse/HIVE-20886
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 2.0.0
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Trivial
> Attachments: HIVE-20886.1.patch
>
>
> {noformat}
> create table if not exists test1(uuid array);
> select lower(uuid) from test1;
> Error: Error while compiling statement: FAILED: NullPointerException null
> (state=42000,code=4)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20886) Fix NPE: GenericUDFLower
[
https://issues.apache.org/jira/browse/HIVE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680809#comment-16680809
]
Hive QA commented on HIVE-20886:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14824/dev-support/hive-personality.sh
|
| git revision | master / 5aac805 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14824/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix NPE: GenericUDFLower
>
>
> Key: HIVE-20886
> URL: https://issues.apache.org/jira/browse/HIVE-20886
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 2.0.0
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Trivial
> Attachments: HIVE-20886.1.patch
>
>
> {noformat}
> create table if not exists test1(uuid array);
> select lower(uuid) from test1;
> Error: Error while compiling statement: FAILED: NullPointerException null
> (state=42000,code=4)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20823) Make Compactor run in a transaction
[
https://issues.apache.org/jira/browse/HIVE-20823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680785#comment-16680785
]
Hive QA commented on HIVE-20823:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947316/HIVE-20823.07.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14823/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14823/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14823/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12947316/HIVE-20823.07.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947316 - PreCommit-HIVE-Build
> Make Compactor run in a transaction
> ---
>
> Key: HIVE-20823
> URL: https://issues.apache.org/jira/browse/HIVE-20823
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-20823.01.patch, HIVE-20823.03.patch,
> HIVE-20823.04.patch, HIVE-20823.05.patch, HIVE-20823.07.patch
>
>
> Have compactor open a transaction and run the job in that transaction.
> # make compactor produced base/delta include this txn id in the folder name,
> e.g. base_7_c17 where 17 is the txnid.
> # add {{CQ_TXN_ID bigint}} to COMPACTION_QUEUE and COMPLETED_COMPACTIONS to
> record this txn id
> # make sure {{AcidUtils.getAcidState()}} pays attention to this transaction
> on read and ignores this dir if this txn id is not committed in the current
> snapshot
> ## this means not only validWriteIdList but ValidTxnIdList should be passed
> along in config (if it isn't yet)
> # once this is done, {{CompactorMR.createCompactorMarker()}} can be
> eliminated and {{AcidUtils.isValidBase}} modified accordingly
> # modify Cleaner so that it doesn't clean old files until new file is visible
> to all readers
> #
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20823) Make Compactor run in a transaction
[
https://issues.apache.org/jira/browse/HIVE-20823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680783#comment-16680783
]
Hive QA commented on HIVE-20823:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947316/HIVE-20823.07.patch
{color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15521 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestCatalogNonDefaultClient.tablesList
(batchId=221)
org.apache.hadoop.hive.ql.TestTxnCommands3.testCleaner2 (batchId=320)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14822/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14822/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14822/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947316 - PreCommit-HIVE-Build
> Make Compactor run in a transaction
> ---
>
> Key: HIVE-20823
> URL: https://issues.apache.org/jira/browse/HIVE-20823
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-20823.01.patch, HIVE-20823.03.patch,
> HIVE-20823.04.patch, HIVE-20823.05.patch, HIVE-20823.07.patch
>
>
> Have compactor open a transaction and run the job in that transaction.
> # make compactor produced base/delta include this txn id in the folder name,
> e.g. base_7_c17 where 17 is the txnid.
> # add {{CQ_TXN_ID bigint}} to COMPACTION_QUEUE and COMPLETED_COMPACTIONS to
> record this txn id
> # make sure {{AcidUtils.getAcidState()}} pays attention to this transaction
> on read and ignores this dir if this txn id is not committed in the current
> snapshot
> ## this means not only validWriteIdList but ValidTxnIdList should be passed
> along in config (if it isn't yet)
> # once this is done, {{CompactorMR.createCompactorMarker()}} can be
> eliminated and {{AcidUtils.isValidBase}} modified accordingly
> # modify Cleaner so that it doesn't clean old files until new file is visible
> to all readers
> #
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20823) Make Compactor run in a transaction
[
https://issues.apache.org/jira/browse/HIVE-20823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680761#comment-16680761
]
Hive QA commented on HIVE-20823:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
16s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} hcatalog/streaming in master has 11 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
45s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
7s{color} | {color:blue} standalone-metastore/metastore-server in master has
185 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
54s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} storage-api: The patch generated 0 new + 7 unchanged
- 1 fixed = 7 total (was 8) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch streaming passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
53s{color} | {color:red} ql: The patch generated 47 new + 1769 unchanged - 13
fixed = 1816 total (was 1782) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
6s{color} | {color:green} The patch metastore-server passed checkstyle {color}
|
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} streaming: The patch generated 2 new + 455 unchanged -
1 fixed = 457 total (was 456) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
56s{color} | {color:red} ql generated 1 new + 2314 unchanged - 1 fixed = 2315
total (was 2315) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
13s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 184 unchanged - 1 fixed = 185 total (was 185) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} storage-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} standalone-metastore_metastore-common generated 0
new + 16 unchanged - 1 fixed = 16 total (was 17) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} streaming in
[jira] [Commented] (HIVE-20826) Enhance HiveSemiJoin rule to convert join + group by on left side to Left Semi Join
[
https://issues.apache.org/jira/browse/HIVE-20826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680716#comment-16680716
]
Hive QA commented on HIVE-20826:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947307/HIVE-20826.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14821/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14821/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14821/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12947307/HIVE-20826.3.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947307 - PreCommit-HIVE-Build
> Enhance HiveSemiJoin rule to convert join + group by on left side to Left
> Semi Join
> ---
>
> Key: HIVE-20826
> URL: https://issues.apache.org/jira/browse/HIVE-20826
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20826.1.patch, HIVE-20826.2.patch,
> HIVE-20826.3.patch
>
>
> Currently HiveSemiJoin rule looks for pattern where group by is on right side.
> We can convert joins which have group by on left side (assuming group by keys
> are same as join keys and none of the columns are being projected from left
> side) to LEFT SEMI JOIN by swapping the inputs. e.g. queries such as:
> {code:sql}
> explain select pp.p_partkey from (select distinct p_name from part) p join
> part pp on pp.p_name = p.p_name;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20845) Fix TestJdbcWithDBTokenStoreNoDoAs flakiness
[
https://issues.apache.org/jira/browse/HIVE-20845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680713#comment-16680713
]
Hive QA commented on HIVE-20845:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947304/HIVE-20845.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14820/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14820/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14820/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-11-09 01:41:49.365
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-14820/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-11-09 01:41:49.367
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 5aac805 HIVE-20782: Clean unused code to improve redability
(Slim B, reviewed by Teddy Choi)
+ git clean -f -d
Removing ${project.basedir}/
Removing itests/${project.basedir}/
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 5aac805 HIVE-20782: Clean unused code to improve redability
(Slim B, reviewed by Teddy Choi)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-11-09 01:41:49.990
+ rm -rf ../yetus_PreCommit-HIVE-Build-14820
+ mkdir ../yetus_PreCommit-HIVE-Build-14820
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-14820
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-14820/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: git apply -p0
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc8902627240627946349.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc8902627240627946349.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
(process-resource-bundles) on project hive-shims-0.23: Execution
process-resource-bundles of goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed.
ConcurrentModificationException -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hive-shims-0.23
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-14820
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947304 - PreCommit-HIVE-Build
> Fix TestJdbcWithDBTokenStoreNoDoAs flakiness
>
[jira] [Commented] (HIVE-20797) Print Number of Locks Acquired
[
https://issues.apache.org/jira/browse/HIVE-20797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680712#comment-16680712
]
Hive QA commented on HIVE-20797:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947289/HIVE-20797.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15527 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_2]
(batchId=191)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14819/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14819/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14819/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947289 - PreCommit-HIVE-Build
> Print Number of Locks Acquired
> --
>
> Key: HIVE-20797
> URL: https://issues.apache.org/jira/browse/HIVE-20797
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, Locking
>Affects Versions: 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Labels: newbie, noob
> Attachments: HIVE-20797.1.patch
>
>
> The number of locks acquired by a query can greatly influence the performance
> and stability of the system, especially for ZK locks. Please add INFO level
> logging with the number of locks each query obtains.
> Log here:
> https://github.com/apache/hive/blob/3963c729fabf90009cb67d277d40fe5913936358/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L1670-L1672
> {quote}
> A list of acquired locks will be stored in the
> org.apache.hadoop.hive.ql.Context object and can be retrieved via
> org.apache.hadoop.hive.ql.Context#getHiveLocks.
> {quote}
> https://github.com/apache/hive/blob/758ff449099065a84c46d63f9418201c8a6731b1/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManager.java#L115-L127
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20797) Print Number of Locks Acquired
[
https://issues.apache.org/jira/browse/HIVE-20797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680669#comment-16680669
]
Hive QA commented on HIVE-20797:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
39s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14819/dev-support/hive-personality.sh
|
| git revision | master / 5aac805 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14819/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Print Number of Locks Acquired
> --
>
> Key: HIVE-20797
> URL: https://issues.apache.org/jira/browse/HIVE-20797
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, Locking
>Affects Versions: 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Labels: newbie, noob
> Attachments: HIVE-20797.1.patch
>
>
> The number of locks acquired by a query can greatly influence the performance
> and stability of the system, especially for ZK locks. Please add INFO level
> logging with the number of locks each query obtains.
> Log here:
> https://github.com/apache/hive/blob/3963c729fabf90009cb67d277d40fe5913936358/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L1670-L1672
> {quote}
> A list of acquired locks will be stored in the
> org.apache.hadoop.hive.ql.Context object and can be retrieved via
> org.apache.hadoop.hive.ql.Context#getHiveLocks.
> {quote}
> https://github.com/apache/hive/blob/758ff449099065a84c46d63f9418201c8a6731b1/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManager.java#L115-L127
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-20740:
---
Attachment: HIVE-20740.11.patch
> Remove global lock in ObjectStore.setConf method
>
>
> Key: HIVE-20740
> URL: https://issues.apache.org/jira/browse/HIVE-20740
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch,
> HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch,
> HIVE-20740.08.patch, HIVE-20740.09.patch, HIVE-20740.10.patch,
> HIVE-20740.11.patch
>
>
> The ObjectStore#setConf method has a global lock which can block other
> clients in concurrent workloads.
> {code}
> @Override
> @SuppressWarnings("nls")
> public void setConf(Configuration conf) {
> // Although an instance of ObjectStore is accessed by one thread, there
> may
> // be many threads with ObjectStore instances. So the static variables
> // pmf and prop need to be protected with locks.
> pmfPropLock.lock();
> try {
> isInitialized = false;
> this.conf = conf;
> this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf,
> ConfVars.HIVE_TXN_STATS_ENABLED);
> configureSSL(conf);
> Properties propsFromConf = getDataSourceProps(conf);
> boolean propsChanged = !propsFromConf.equals(prop);
> if (propsChanged) {
> if (pmf != null){
> clearOutPmfClassLoaderCache(pmf);
> if (!forTwoMetastoreTesting) {
> // close the underlying connection pool to avoid leaks
> pmf.close();
> }
> }
> pmf = null;
> prop = null;
> }
> assert(!isActiveTransaction());
> shutdown();
> // Always want to re-create pm as we don't know if it were created by
> the
> // most recent instance of the pmf
> pm = null;
> directSql = null;
> expressionProxy = null;
> openTrasactionCalls = 0;
> currentTransaction = null;
> transactionStatus = TXN_STATUS.NO_STATE;
> initialize(propsFromConf);
> String partitionValidationRegex =
> MetastoreConf.getVar(this.conf,
> ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
> if (partitionValidationRegex != null &&
> !partitionValidationRegex.isEmpty()) {
> partitionValidationPattern =
> Pattern.compile(partitionValidationRegex);
> } else {
> partitionValidationPattern = null;
> }
> // Note, if metrics have not been initialized this will return null,
> which means we aren't
> // using metrics. Thus we should always check whether this is non-null
> before using.
> MetricRegistry registry = Metrics.getRegistry();
> if (registry != null) {
> directSqlErrors =
> Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS);
> }
> this.batchSize = MetastoreConf.getIntVar(conf,
> ConfVars.RAWSTORE_PARTITION_BATCH_SIZE);
> if (!isInitialized) {
> throw new RuntimeException(
> "Unable to create persistence manager. Check dss.log for details");
> } else {
> LOG.debug("Initialized ObjectStore");
> }
> } finally {
> pmfPropLock.unlock();
> }
> }
> {code}
> The {{pmfPropLock}} is a static object and it disallows any other new
> connection to HMS which is trying to instantiate ObjectStore. We should
> either remove the lock or reduce the scope of the lock so that it is held for
> a very small amount of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18415) Lower "Updating Partition Stats" Logging Level
[
https://issues.apache.org/jira/browse/HIVE-18415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680634#comment-16680634
]
Hive QA commented on HIVE-18415:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947287/HIVE-18415.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15527 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14818/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14818/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14818/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947287 - PreCommit-HIVE-Build
> Lower "Updating Partition Stats" Logging Level
> --
>
> Key: HIVE-18415
> URL: https://issues.apache.org/jira/browse/HIVE-18415
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 1.2.2, 2.2.0, 3.0.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-18415.1.patch
>
>
> {code:title=org.apache.hadoop.hive.metastore.utils.MetaStoreUtils}
> LOG.warn("Updating partition stats fast for: " + part.getTableName());
> ...
> LOG.warn("Updated size to " + params.get(StatsSetupConst.TOTAL_SIZE));
> {code}
> This logging produces many lines of WARN log messages in my log file and it's
> not clear to me what the issue is here. Why is this a warning and how should
> I respond to address this warning?
> DEBUG is probably more appropriate for a utility class. Please lower.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20892) Benchmark XXhash for 64 bit hashing function instead of Murmum hash
[ https://issues.apache.org/jira/browse/HIVE-20892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680595#comment-16680595 ] slim bouguerra commented on HIVE-20892: --- forgot to add that we need to look at 32bit hashes since that is what Hive uses for Joins and Grouping. > Benchmark XXhash for 64 bit hashing function instead of Murmum hash > --- > > Key: HIVE-20892 > URL: https://issues.apache.org/jira/browse/HIVE-20892 > Project: Hive > Issue Type: Sub-task >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > > https://cyan4973.github.io/xxHash/ > FYI this is used by lot of other MPP systems ... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20892) Benchmark XXhash for 64 bit hashing function instead of Murmum hash
[ https://issues.apache.org/jira/browse/HIVE-20892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680592#comment-16680592 ] slim bouguerra commented on HIVE-20892: --- [~prasanth_j] thanks that is what planing to do, seems like you have done most of the work, maybe worth re-run it with newer JVMs and on something else than laptop? Also am curious about the impact of the distribution over actual data like TPC-H > Benchmark XXhash for 64 bit hashing function instead of Murmum hash > --- > > Key: HIVE-20892 > URL: https://issues.apache.org/jira/browse/HIVE-20892 > Project: Hive > Issue Type: Sub-task >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > > https://cyan4973.github.io/xxHash/ > FYI this is used by lot of other MPP systems ... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20888) TxnHandler: sort() called on immutable lists
[
https://issues.apache.org/jira/browse/HIVE-20888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680581#comment-16680581
]
Eugene Koifman commented on HIVE-20888:
---
cc [~ikryvenko]
> TxnHandler: sort() called on immutable lists
>
>
> Key: HIVE-20888
> URL: https://issues.apache.org/jira/browse/HIVE-20888
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Gopal V
>Priority: Major
>
> {code}
> } else {
> assert (!rqst.isSetSrcTxnToWriteIdList());
> assert (rqst.isSetTxnIds());
> txnIds = rqst.getTxnIds();
> }
> Collections.sort(txnIds); //easier to read logs and for assumption
> done in replication flow
> {code}
> when the input comes from
> {code}
> @Override
> public long allocateTableWriteId(long txnId, String dbName, String
> tableName) throws TException {
> return allocateTableWriteIdsBatch(Collections.singletonList(txnId),
> dbName, tableName).get(0).getWriteId();
> }
> {code}
> {code}
> java.lang.UnsupportedOperationException: null
> at java.util.AbstractList.set(AbstractList.java:132) ~[?:1.8.0]
> at java.util.AbstractList$ListItr.set(AbstractList.java:426) ~[?:1.8.0]
> at java.util.Collections.sort(Collections.java:170) ~[?:1.8.0]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.allocateTableWriteIds(TxnHandler.java:1523)
> ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.allocate_table_write_ids(HiveMetaStore.java:7349)
> ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18415) Lower "Updating Partition Stats" Logging Level
[
https://issues.apache.org/jira/browse/HIVE-18415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680564#comment-16680564
]
Hive QA commented on HIVE-18415:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
6s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
0s{color} | {color:blue} standalone-metastore/metastore-server in master has
185 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14818/dev-support/hive-personality.sh
|
| git revision | master / 5aac805 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14818/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Lower "Updating Partition Stats" Logging Level
> --
>
> Key: HIVE-18415
> URL: https://issues.apache.org/jira/browse/HIVE-18415
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 1.2.2, 2.2.0, 3.0.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-18415.1.patch
>
>
> {code:title=org.apache.hadoop.hive.metastore.utils.MetaStoreUtils}
> LOG.warn("Updating partition stats fast for: " + part.getTableName());
> ...
> LOG.warn("Updated size to " + params.get(StatsSetupConst.TOTAL_SIZE));
> {code}
> This logging produces many lines of WARN log messages in my log file and it's
> not clear to me what the issue is here. Why is this a warning and how should
> I respond to address this warning?
> DEBUG is probably more appropriate for a utility class. Please lower.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20881) Constant propagation oversimplifies projections
[
https://issues.apache.org/jira/browse/HIVE-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680561#comment-16680561
]
Ashutosh Chauhan edited comment on HIVE-20881 at 11/8/18 11:08 PM:
---
For Sel we need to return NULL in such cases. However, for Filter operator we
can still fold it to false, correct? But with this change we will return NULL
in all cases. Can we return false for FIL ?
was (Author: ashutoshc):
For Sel we need to return in such cases. However, for Filter operator we can
still fold it to false, correct? But with this change we will return NULL in
all cases. Can we return false for FIL ?
> Constant propagation oversimplifies projections
> ---
>
> Key: HIVE-20881
> URL: https://issues.apache.org/jira/browse/HIVE-20881
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20881.01.patch
>
>
> {code:java}
> create table cx2(bool1 boolean);
> insert into cx2 values (true),(false),(null);
> set hive.cbo.enable=true;
> select bool1 IS TRUE OR (cast(NULL as boolean) AND bool1 IS NOT TRUE AND
> bool1 IS NOT FALSE) from cx2;
> ++
> | _c0 |
> ++
> | true |
> | false |
> | NULL |
> ++
> set hive.cbo.enable=false;
> select bool1 IS TRUE OR (cast(NULL as boolean) AND bool1 IS NOT TRUE AND
> bool1 IS NOT FALSE) from cx2;
> +---+
> | _c0 |
> +---+
> | true |
> | NULL |
> | NULL |
> +---+
> {code}
> from explain it seems the expression was simplified to: {{(_col0 is true or
> null)}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20881) Constant propagation oversimplifies projections
[
https://issues.apache.org/jira/browse/HIVE-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680561#comment-16680561
]
Ashutosh Chauhan commented on HIVE-20881:
-
For Sel we need to return in such cases. However, for Filter operator we can
still fold it to false, correct? But with this change we will return NULL in
all cases. Can we return false for FIL ?
> Constant propagation oversimplifies projections
> ---
>
> Key: HIVE-20881
> URL: https://issues.apache.org/jira/browse/HIVE-20881
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20881.01.patch
>
>
> {code:java}
> create table cx2(bool1 boolean);
> insert into cx2 values (true),(false),(null);
> set hive.cbo.enable=true;
> select bool1 IS TRUE OR (cast(NULL as boolean) AND bool1 IS NOT TRUE AND
> bool1 IS NOT FALSE) from cx2;
> ++
> | _c0 |
> ++
> | true |
> | false |
> | NULL |
> ++
> set hive.cbo.enable=false;
> select bool1 IS TRUE OR (cast(NULL as boolean) AND bool1 IS NOT TRUE AND
> bool1 IS NOT FALSE) from cx2;
> +---+
> | _c0 |
> +---+
> | true |
> | NULL |
> | NULL |
> +---+
> {code}
> from explain it seems the expression was simplified to: {{(_col0 is true or
> null)}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20888) TxnHandler: sort() called on immutable lists
[
https://issues.apache.org/jira/browse/HIVE-20888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-20888:
--
Component/s: Transactions
> TxnHandler: sort() called on immutable lists
>
>
> Key: HIVE-20888
> URL: https://issues.apache.org/jira/browse/HIVE-20888
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Gopal V
>Priority: Major
>
> {code}
> } else {
> assert (!rqst.isSetSrcTxnToWriteIdList());
> assert (rqst.isSetTxnIds());
> txnIds = rqst.getTxnIds();
> }
> Collections.sort(txnIds); //easier to read logs and for assumption
> done in replication flow
> {code}
> when the input comes from
> {code}
> @Override
> public long allocateTableWriteId(long txnId, String dbName, String
> tableName) throws TException {
> return allocateTableWriteIdsBatch(Collections.singletonList(txnId),
> dbName, tableName).get(0).getWriteId();
> }
> {code}
> {code}
> java.lang.UnsupportedOperationException: null
> at java.util.AbstractList.set(AbstractList.java:132) ~[?:1.8.0]
> at java.util.AbstractList$ListItr.set(AbstractList.java:426) ~[?:1.8.0]
> at java.util.Collections.sort(Collections.java:170) ~[?:1.8.0]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.allocateTableWriteIds(TxnHandler.java:1523)
> ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.allocate_table_write_ids(HiveMetaStore.java:7349)
> ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20831) Add Session ID to Operation Logging
[
https://issues.apache.org/jira/browse/HIVE-20831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680538#comment-16680538
]
Hive QA commented on HIVE-20831:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947285/HIVE-20831.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15527 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14817/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14817/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14817/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947285 - PreCommit-HIVE-Build
> Add Session ID to Operation Logging
> ---
>
> Key: HIVE-20831
> URL: https://issues.apache.org/jira/browse/HIVE-20831
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Labels: newbie, noob
> Attachments: HIVE-20831.1.patch
>
>
> {code:java|title=OperationManager.java}
> LOG.info("Adding operation: " + operation.getHandle());
> {code}
> Please add additional logging to explicitly state which Hive session this
> operation is being added to.
> https://github.com/apache/hive/blob/3963c729fabf90009cb67d277d40fe5913936358/service/src/java/org/apache/hive/service/cli/operation/OperationManager.java#L201
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20893) BloomK Filter probing method is not thread safe
[ https://issues.apache.org/jira/browse/HIVE-20893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680539#comment-16680539 ] Clint Wylie commented on HIVE-20893: Unfortunately the suggested fix is failing most tests when I run them locally. > BloomK Filter probing method is not thread safe > --- > > Key: HIVE-20893 > URL: https://issues.apache.org/jira/browse/HIVE-20893 > Project: Hive > Issue Type: Bug > Components: storage-api >Reporter: slim bouguerra >Priority: Major > > As far i can tell this is not an issue for Hive yet (most of the usage of > probing seems to be done by one thread at a time) but it is an issue of other > users like Druid as per the following > issue.[https://github.com/apache/incubator-druid/issues/6546] > The fix is proposed by the author of > [https://github.com/apache/incubator-druid/pull/6584] is to make couple of > local fields as ThreadLocals. > Idea looks good to me and doesn't have any perf drawbacks. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20893) BloomK Filter probing method is not thread safe
[ https://issues.apache.org/jira/browse/HIVE-20893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680524#comment-16680524 ] Clint Wylie commented on HIVE-20893: Thanks for the explanation! I'll see what I can do about getting perfasm output. > BloomK Filter probing method is not thread safe > --- > > Key: HIVE-20893 > URL: https://issues.apache.org/jira/browse/HIVE-20893 > Project: Hive > Issue Type: Bug > Components: storage-api >Reporter: slim bouguerra >Priority: Major > > As far i can tell this is not an issue for Hive yet (most of the usage of > probing seems to be done by one thread at a time) but it is an issue of other > users like Druid as per the following > issue.[https://github.com/apache/incubator-druid/issues/6546] > The fix is proposed by the author of > [https://github.com/apache/incubator-druid/pull/6584] is to make couple of > local fields as ThreadLocals. > Idea looks good to me and doesn't have any perf drawbacks. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20887) Tests: openjdk 8 has a bug that prevents surefire from working
[
https://issues.apache.org/jira/browse/HIVE-20887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680518#comment-16680518
]
Kevin Risden commented on HIVE-20887:
-
There is a debian openjdk8 packaging bug that is recent:
[https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=911925] it ended up in
docker images as well for openjdk8 image.
> Tests: openjdk 8 has a bug that prevents surefire from working
> --
>
> Key: HIVE-20887
> URL: https://issues.apache.org/jira/browse/HIVE-20887
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
>
> It looks like the problem is
> https://bugs.openjdk.java.net/browse/JDK-8030046. It looks like:
> {code:bash}
> [ERROR] Caused by:
> org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM
> terminated without properly saying goodbye. VM crash or System.exit called?
> {code}
> The surefire-reports/*.dumpstream looks like:
> {code:bash}
> Error: Could not find or load main class
> org.apache.maven.surefire.booter.ForkedBooter
> {code}
> and we can work around the problem by changing the surefire configuration:
> {code:bash}
> + false
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20484) Disable Block Cache By Default With HBase SerDe
[
https://issues.apache.org/jira/browse/HIVE-20484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680520#comment-16680520
]
Prasanth Jayachandran commented on HIVE-20484:
--
+1
> Disable Block Cache By Default With HBase SerDe
> ---
>
> Key: HIVE-20484
> URL: https://issues.apache.org/jira/browse/HIVE-20484
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler
>Affects Versions: 1.2.3, 2.4.0, 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-20484.1.patch, HIVE-20484.2.patch
>
>
> {quote}
> Scan instances can be set to use the block cache in the RegionServer via the
> setCacheBlocks method. For input Scans to MapReduce jobs, this should be
> false.
> https://hbase.apache.org/book.html#perf.hbase.client.blockcache
> {quote}
> However, from the Hive code, we can see that this is not the case.
> {code}
> public static final String HBASE_SCAN_CACHEBLOCKS = "hbase.scan.cacheblock";
> ...
> String scanCacheBlocks =
> tableProperties.getProperty(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS);
> if (scanCacheBlocks != null) {
> jobProperties.put(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS, scanCacheBlocks);
> }
> ...
> String scanCacheBlocks = jobConf.get(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS);
> if (scanCacheBlocks != null) {
> scan.setCacheBlocks(Boolean.parseBoolean(scanCacheBlocks));
> }
> {code}
> In the Hive code, we can see that if {{hbase.scan.cacheblock}} is not
> specified in the {{SERDEPROPERTIES}} then {{setCacheBlocks}} is not called
> and the default value of the HBase {{Scan}} class is used.
> {code:java|title=Scan.java}
> /**
>* Set whether blocks should be cached for this Scan.
>*
>* This is true by default. When true, default settings of the table and
>* family are used (this will never override caching blocks if the block
>* cache is disabled for that family or entirely).
>*
>* @param cacheBlocks if false, default settings are overridden and blocks
>* will not be cached
>*/
> public Scan setCacheBlocks(boolean cacheBlocks) {
> this.cacheBlocks = cacheBlocks;
> return this;
> }
> {code}
> Hive is doing full scans of the table with MapReduce/Spark and therefore,
> according to the HBase docs, the default behavior here should be that blocks
> are not cached. Hive should set this value to "false" by default unless the
> table {{SERDEPROPERTIES}} override this.
> {code:sql}
> -- Commands for HBase
> -- create 'test', 't'
> CREATE EXTERNAL TABLE test(value map, row_key string)
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
> "hbase.columns.mapping" = "t:,:key",
> "hbase.scan.cacheblock" = "false"
> );
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20161) Do Not Print StackTraces to STDERR in ParseDriver
[
https://issues.apache.org/jira/browse/HIVE-20161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680513#comment-16680513
]
Prasanth Jayachandran commented on HIVE-20161:
--
+1
> Do Not Print StackTraces to STDERR in ParseDriver
> -
>
> Key: HIVE-20161
> URL: https://issues.apache.org/jira/browse/HIVE-20161
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Labels: newbie, noob
> Attachments: HIVE-20161.1.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java
> {code}
> // Do not print stack trace to STDERR - remove this, just throw the
> HiveException
> } catch (Exception e) {
> e.printStackTrace();
> throw new HiveException(e);
> }
> ...
> // Do not log and throw. log *or* throw. In this case, just throw. Remove
> logging.
> // Remove explicit 'return' call. No need for it.
> try {
> skewJoinKeyContext.endGroup();
> } catch (IOException e) {
> LOG.error(e.getMessage(), e);
> throw new HiveException(e);
> }
> return;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20160) Do Not Print StackTraces to STDERR in OperatorFactory
[
https://issues.apache.org/jira/browse/HIVE-20160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680514#comment-16680514
]
Prasanth Jayachandran commented on HIVE-20160:
--
+1
> Do Not Print StackTraces to STDERR in OperatorFactory
> -
>
> Key: HIVE-20160
> URL: https://issues.apache.org/jira/browse/HIVE-20160
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Labels: newbie, noob
> Attachments: HIVE-20160.1.patch
>
>
> https://github.com/apache/hive/blob/ac6b2a3fb195916e22b2e5f465add2ffbcdc7430/ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java#L158
> {code}
> } catch (Exception e) {
> e.printStackTrace();
> throw new HiveException(...
> {code}
> Do not print the stack trace. The error is being wrapped in a HiveException.
> Allow the code catching this exception to print the error to a logger
> instead of dumping it here to STDERR. There are several instances of this in
> the class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20223) SmallTableCache.java SLF4J Parameterized Logging
[
https://issues.apache.org/jira/browse/HIVE-20223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680512#comment-16680512
]
Prasanth Jayachandran commented on HIVE-20223:
--
+1
> SmallTableCache.java SLF4J Parameterized Logging
>
>
> Key: HIVE-20223
> URL: https://issues.apache.org/jira/browse/HIVE-20223
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Labels: newbie, noob
> Attachments: HIVE-20223.1.patch
>
>
> {code:java|title=org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java}
> if (LOG.isDebugEnabled()) {
> LOG.debug("Cleaned up small table cache for query " + queryId);
> }
> if (tableContainerMap.putIfAbsent(path, tableContainer) == null &&
> LOG.isDebugEnabled()) {
> LOG.debug("Cached small table file " + path + " for query " + queryId);
> }
> if (tableContainer != null && LOG.isDebugEnabled()) {
> LOG.debug("Loaded small table file " + path + " from cache for query "
> + queryId);
> }
> {code}
>
> Remove {{isDebugEnabled}} and replace with parameterized logging.
> https://www.slf4j.org/faq.html#logging_performance
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20893) BloomK Filter probing method is not thread safe
[
https://issues.apache.org/jira/browse/HIVE-20893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680486#comment-16680486
]
Gopal V commented on HIVE-20893:
The masks object was rough place-holder for an eventual Long8 type to be
allocated locally (the jdk value types), to stand in for the __m512i variables
Talked to [~bslim] about this and I'm thinking an even simpler solution than
unrolling might work out just fine.
{code}
- masks[wordOffset] |= (1L << bitPos);
+ final long bloomWord = bitSet.data[wordOffset + i];
+ if (0 == (bloomWord & (1L << bitPos))) return false;
+ }
+ return true;
{code}
And re-run your benchmark?
That makes it similar to the original bloom implementation and better than the
ThreadLocal at rejecting rows.
If you do have a linux box with libhsdis, then the output of JMH -prof perfasm
would be super useful to look into.
> BloomK Filter probing method is not thread safe
> ---
>
> Key: HIVE-20893
> URL: https://issues.apache.org/jira/browse/HIVE-20893
> Project: Hive
> Issue Type: Bug
> Components: storage-api
>Reporter: slim bouguerra
>Priority: Major
>
> As far i can tell this is not an issue for Hive yet (most of the usage of
> probing seems to be done by one thread at a time) but it is an issue of other
> users like Druid as per the following
> issue.[https://github.com/apache/incubator-druid/issues/6546]
> The fix is proposed by the author of
> [https://github.com/apache/incubator-druid/pull/6584] is to make couple of
> local fields as ThreadLocals.
> Idea looks good to me and doesn't have any perf drawbacks.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20665) Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-20665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20665:
---
Attachment: HIVE-20665.1.patch
> Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
>
>
> Key: HIVE-20665
> URL: https://issues.apache.org/jira/browse/HIVE-20665
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.2, 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-20665.1.patch, HIVE-20665.1.patch
>
>
> When parallel tasks are enabled in Hive, all of the resulting queries share
> the same Hive configuration. This is problematic as each query will modify
> the same {{HiveConf}} object with things like query ID and query text. This
> will overwrite each other and cause {{ConcurrentModificationException}}
> issues.
> {code:java|title=SQLOperation.java}
> public Object run() throws HiveSQLException {
> Hive.set(parentHive, false);
> // TODO: can this result in cross-thread reuse of session state?
> SessionState.setCurrentSessionState(parentSessionState);
> PerfLogger.setPerfLogger(SessionState.getPerfLogger());
> LogUtils.registerLoggingContext(queryState.getConf());
> try {
> if (asyncPrepare) {
> prepare(queryState);
> }
> runQuery();
> } catch (HiveSQLException e) {
> // ...
> {code}
> [Code
> Here|https://github.com/apache/hive/blob/6e27a5315a44c55ef3b178e7212c9068de322d01/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L308-L319]
> From this code it can be seen that for every thread launched, it is all
> calling {{setCurrentSessionState}}.
> {code:java|title=SessionStates.java}
> /**
>* Sets the given session state in the thread local var for sessions.
>*/
> public static void setCurrentSessionState(SessionState startSs) {
> tss.get().attach(startSs);
> }
> // SessionState is not available in runtime and Hive.get().getConf() is not
> safe to call
> private static class SessionStates {
> private SessionState state;
> private HiveConf conf;
> private void attach(SessionState state) {
> this.state = state;
> attach(state.getConf());
> }
> private void attach(HiveConf conf) {
> // -- SHALLOW COPY HERE, ALL THREADS SHARING SAME REFERENCE -- //
> this.conf = conf;
> ClassLoader classLoader = conf.getClassLoader();
> if (classLoader != null) {
> Thread.currentThread().setContextClassLoader(classLoader);
> }
> }
> }
> {code}
> [Code
> Here|https://github.com/apache/hive/blob/7795c0a7dc59941671f8845d78b16d9e5ddc9ea3/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L540-L556]
> Ensure that all threads get their own copy of the {{HiveConf}} object to use
> and modify.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20782) Cleaning some unused code
[
https://issues.apache.org/jira/browse/HIVE-20782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-20782:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Cleaning some unused code
> -
>
> Key: HIVE-20782
> URL: https://issues.apache.org/jira/browse/HIVE-20782
> Project: Hive
> Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-20782.2.patch, HIVE-20782.2.patch,
> HIVE-20782.3.patch, HIVE-20782.3.patch, HIVE-20782.3.patch, HIVE-20782.patch
>
>
> Am making my way into the vectorize code and trying understand the APIs. Ran
> into this unused one, i guess it is not used anymore.
> [~ashutoshc] maybe can explain as you are the main contributor to this file
> {code}
> a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedSerde.java{code}
> ?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20847) Review of NullScan Code
[
https://issues.apache.org/jira/browse/HIVE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20847:
---
Status: Open (was: Patch Available)
> Review of NullScan Code
> ---
>
> Key: HIVE-20847
> URL: https://issues.apache.org/jira/browse/HIVE-20847
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20847.1.patch, HIVE-20847.1.patch,
> HIVE-20847.1.patch
>
>
> What got me looking at this class was the verboseness of some of the logging.
> I would like to request that we DEBUG the logging since this level of detail
> means nothing to a cluster admin.
> Also... this {{contains}} call would be better applied onto a {{HashSet}}
> instead of an {{ArrayList}}.
> {code:java|title=NullScanTaskDispatcher.java}
> private void processAlias(MapWork work, Path path, ArrayList
> aliasesAffected, ArrayList aliases) {
> // the aliases that are allowed to map to a null scan.
> ArrayList allowed = new ArrayList();
> for (String alias : aliasesAffected) {
> if (aliases.contains(alias)) {
> allowed.add(alias);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20847) Review of NullScan Code
[
https://issues.apache.org/jira/browse/HIVE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20847:
---
Attachment: HIVE-20847.1.patch
> Review of NullScan Code
> ---
>
> Key: HIVE-20847
> URL: https://issues.apache.org/jira/browse/HIVE-20847
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20847.1.patch, HIVE-20847.1.patch,
> HIVE-20847.1.patch
>
>
> What got me looking at this class was the verboseness of some of the logging.
> I would like to request that we DEBUG the logging since this level of detail
> means nothing to a cluster admin.
> Also... this {{contains}} call would be better applied onto a {{HashSet}}
> instead of an {{ArrayList}}.
> {code:java|title=NullScanTaskDispatcher.java}
> private void processAlias(MapWork work, Path path, ArrayList
> aliasesAffected, ArrayList aliases) {
> // the aliases that are allowed to map to a null scan.
> ArrayList allowed = new ArrayList();
> for (String alias : aliasesAffected) {
> if (aliases.contains(alias)) {
> allowed.add(alias);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20847) Review of NullScan Code
[
https://issues.apache.org/jira/browse/HIVE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20847:
---
Status: Patch Available (was: Open)
> Review of NullScan Code
> ---
>
> Key: HIVE-20847
> URL: https://issues.apache.org/jira/browse/HIVE-20847
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20847.1.patch, HIVE-20847.1.patch,
> HIVE-20847.1.patch
>
>
> What got me looking at this class was the verboseness of some of the logging.
> I would like to request that we DEBUG the logging since this level of detail
> means nothing to a cluster admin.
> Also... this {{contains}} call would be better applied onto a {{HashSet}}
> instead of an {{ArrayList}}.
> {code:java|title=NullScanTaskDispatcher.java}
> private void processAlias(MapWork work, Path path, ArrayList
> aliasesAffected, ArrayList aliases) {
> // the aliases that are allowed to map to a null scan.
> ArrayList allowed = new ArrayList();
> for (String alias : aliasesAffected) {
> if (aliases.contains(alias)) {
> allowed.add(alias);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20813) udf to_epoch_milli need to support timestamp without time zone as well
[
https://issues.apache.org/jira/browse/HIVE-20813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-20813:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> udf to_epoch_milli need to support timestamp without time zone as well
> --
>
> Key: HIVE-20813
> URL: https://issues.apache.org/jira/browse/HIVE-20813
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-20813.patch, HIVE-20813.patch, HIVE-20813.patch,
> HIVE-20813.patch
>
>
> Currently the following query will fail with a cast exception (tries to cast
> timestamp to timestamp with local timezone).
> {code}
> select to_epoch_milli(current_timestamp)
> {code}
> As a simple fix we need to add support for timestamp object inspector.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18902) Lower Logging Level for Cleaning Up "local RawStore"
[
https://issues.apache.org/jira/browse/HIVE-18902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-18902:
---
Attachment: HIVE-18902.5.patch
> Lower Logging Level for Cleaning Up "local RawStore"
>
>
> Key: HIVE-18902
> URL: https://issues.apache.org/jira/browse/HIVE-18902
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: BELUGA BEHR
>Assignee: Bohdan Chupika
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18902.01.patch, HIVE-18902.02.patch,
> HIVE-18902.03.patch, HIVE-18902.04.patch, HIVE-18902.5.patch
>
>
> [https://github.com/apache/hive/blob/7c22d74c8d0eb0650adf6e84e0536127c103e46c/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L7756-L7768]
>
> {code:java}
> private static void cleanupRawStore() {
> try {
> RawStore rs = HMSHandler.getRawStore();
> if (rs != null) {
> HMSHandler.logInfo("Cleaning up thread local RawStore...");
> rs.shutdown();
> }
> } finally {
> HMSHandler handler = HMSHandler.threadLocalHMSHandler.get();
> if (handler != null) {
> handler.notifyMetaListenersOnShutDown();
> }
> HMSHandler.threadLocalHMSHandler.remove();
> HMSHandler.threadLocalConf.remove();
> HMSHandler.threadLocalModifiedConfig.remove();
> HMSHandler.removeRawStore();
> HMSHandler.logInfo("Done cleaning up thread local RawStore");
> }
> }
> {code}
> {code}
> 2018-03-03 17:21:49,832 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-21]: 19:
> Cleaning up thread local RawStore...
> 2018-03-03 17:21:49,834 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-21]: 19: Done
> cleaning up thread local RawStore
> {code}
> Not very helpful logging. Please change logging levels to _debug_ or even
> _trace_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20360) QTest: ignore driver/qtest exclusions if -Dqfile param is set
[
https://issues.apache.org/jira/browse/HIVE-20360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679339#comment-16679339
]
Hive QA commented on HIVE-20360:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947248/HIVE-20360.04.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 669 failed/errored test(s), 15688 tests
executed
*Failed tests:*
{noformat}
TestHBaseCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=105)
[hbase_handler_snapshot.q,hbase_binary_binary.q,hbase_binary_external_table_queries.q,hbase_timestamp.q,hbase_binary_storage_queries.q]
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=172)
[vectorized_parquet.q,vector_udf_string_to_boolean.q,vector_case_when_2.q,strict_managed_tables_sysdb.q,vector_decimal_expressions.q,union_fast_stats.q,cbo_subq_not_in.q,resourceplan.q,identity_project_remove_skip.q,windowing.q,delete_tmp_table.q,bucket2.q,vector_aggregate_without_gby.q,schema_evol_text_nonvec_part_all_primitive.q,vector_windowing_streaming.q,materialized_view_create_rewrite_dummy.q,tez_smb_reduce_side.q,orc_create.q,vector_distinct_2.q,vector_groupby_rollup1.q,vectorization_13.q,mapjoin_mapjoin.q,update_all_partitioned.q,vector_decimal_precision.q,dynpart_sort_optimization.q,external_jdbc_table2.q,tez_join_hash.q,multi_insert_lateral_view.q,non_native_window_udf.q,schema_evol_orc_vec_part_all_complex.q]
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=178)
[smb_mapjoin_15.q,insert_values_partitioned.q,vector_orc_merge_incompat_schema.q,selectDistinctStar.q,strict_managed_tables1.q,bucket4.q,vectorized_distinct_gby.q,vector_groupby_mapjoin.q,insert_values_dynamic_partitioned.q,autoColumnStats_10.q,tez_smb_1.q,tez_vector_dynpart_hashjoin_1.q,filter_join_breaktask2.q,union_rowcounts.q,mm_conversions.q,multiMapJoin2.q,filter_join_breaktask.q,column_table_stats.q,update_access_time_non_current_db.q,schema_evol_orc_nonvec_part.q,dynamic_semijoin_reduction_2.q,alter_merge_2_orc.q,bucket_num_reducers2.q,union4.q,auto_sortmerge_join_8.q,llap_text.q,dynamic_semijoin_reduction_sw2.q,change_allowincompatible_vectorization_false_date.q,dynamic_semijoin_reduction_3.q,vector_decimal_udf2.q]
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=182)
[vector_udf_adaptor_1.q,schema_evol_text_vec_part_llap_io.q,join_is_not_distinct_from.q,tez_schema_evolution.q,vector_multi_insert.q,input16_cc.q,materialized_view_rewrite_no_join_opt.q,temp_table.q,cross_product_check_2.q,cte_1.q,partition_pruning.q,vectorization_17.q,bucket_groupby.q,vector_nullsafe_join.q,vector_decimal_udf.q,materialized_view_rewrite_3.q,schema_evol_text_nonvec_part_all_primitive_llap_io.q,bucketpruning1.q,vector_groupby6.q,vector_char_varchar_1.q,smb_mapjoin_5.q,vector_char_4.q,auto_sortmerge_join_5.q,auto_join0.q,cbo_limit.q,vector_udf_inline.q,vector_windowing.q,schema_evol_orc_nonvec_part_all_primitive.q,auto_sortmerge_join_11.q,orc_merge_incompat_writer_version.q]
TestMiniTezCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=109)
[explainanalyze_4.q,acid_vectorization_original_tez.q,orc_merge12.q,tez_union_with_udf.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=98)
[jira] [Updated] (HIVE-20868) SMB Join fails intermittently when TezDummyOperator has child op in getFinalOp in MapRecordProcessor
[
https://issues.apache.org/jira/browse/HIVE-20868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deepak Jaiswal updated HIVE-20868:
--
Attachment: HIVE-20868.2.patch
> SMB Join fails intermittently when TezDummyOperator has child op in
> getFinalOp in MapRecordProcessor
>
>
> Key: HIVE-20868
> URL: https://issues.apache.org/jira/browse/HIVE-20868
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-20868.1.patch, HIVE-20868.2.patch
>
>
> In MapRecordProcessor::getFinalOp() due to external cause(not known), the
> TezDummyStoreOperator may have MergeJoin Op as child intermittently. Due to
> this, the fetchDone remains set to true for the DummyOp which was set by
> previous task. Ideally, fetchDone should be reset for each task. This
> eventually leads to the join op skip rows from that dummy op resulting in
> wrong results.
> Good init order
> {code}
> 2018-11-01 21:42:33,677 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp child Ops = TS[3] (core)
> 2018-11-01 21:42:33,677 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp child Ops = FIL[24]
> 2018-11-01 21:42:33,677 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp child Ops = SEL[5]
> 2018-11-01 21:42:33,677 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp child Ops = DUMMY_STORE[45]
> 2018-11-01 21:42:33,677 [INFO] [TezChild] |tez.MapRecordProcessor|: Iterating
> children of dummy op DUMMY_STORE[45]
> 2018-11-01 21:42:33,677 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp returns DUMMY_STORE[45]
> 2018-11-01 21:42:33,677 [INFO] [TezChild] |tez.MapRecordProcessor|:
> InitProcessor : setting fetchDone to false
> {code}
> Bad init order
> {code}
> 2018-11-01 21:42:33,304 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp child Ops = TS[3] (core)
> 2018-11-01 21:42:33,304 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp child Ops = FIL[24]
> 2018-11-01 21:42:33,304 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp child Ops = SEL[5]
> 2018-11-01 21:42:33,304 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp child Ops = DUMMY_STORE[45]
> 2018-11-01 21:42:33,304 [INFO] [TezChild] |tez.MapRecordProcessor|:
> Iterating children of dummy op DUMMY_STORE[45]
> 2018-11-01 21:42:33,304 [INFO] [TezChild] |tez.MapRecordProcessor|: Child of
> Dummy Op MERGEJOIN[44]
> 2018-11-01 21:42:33,304 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp child Ops = MERGEJOIN[44]
> 2018-11-01 21:42:33,304 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp child Ops = SEL[13]
> 2018-11-01 21:42:33,304 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp child Ops = RS[14]
> 2018-11-01 21:42:33,304 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp returns RS[14]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20831) Add Session ID to Operation Logging
[
https://issues.apache.org/jira/browse/HIVE-20831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680456#comment-16680456
]
Hive QA commented on HIVE-20831:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14817/dev-support/hive-personality.sh
|
| git revision | master / 5aac805 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: service U: service |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14817/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add Session ID to Operation Logging
> ---
>
> Key: HIVE-20831
> URL: https://issues.apache.org/jira/browse/HIVE-20831
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Labels: newbie, noob
> Attachments: HIVE-20831.1.patch
>
>
> {code:java|title=OperationManager.java}
> LOG.info("Adding operation: " + operation.getHandle());
> {code}
> Please add additional logging to explicitly state which Hive session this
> operation is being added to.
> https://github.com/apache/hive/blob/3963c729fabf90009cb67d277d40fe5913936358/service/src/java/org/apache/hive/service/cli/operation/OperationManager.java#L201
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20813) udf to_epoch_milli need to support timestamp without time zone as well
[
https://issues.apache.org/jira/browse/HIVE-20813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679483#comment-16679483
]
Hive QA commented on HIVE-20813:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
45s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14806/dev-support/hive-personality.sh
|
| git revision | master / 840dd43 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14806/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> udf to_epoch_milli need to support timestamp without time zone as well
> --
>
> Key: HIVE-20813
> URL: https://issues.apache.org/jira/browse/HIVE-20813
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-20813.patch, HIVE-20813.patch, HIVE-20813.patch,
> HIVE-20813.patch
>
>
> Currently the following query will fail with a cast exception (tries to cast
> timestamp to timestamp with local timezone).
> {code}
> select to_epoch_milli(current_timestamp)
> {code}
> As a simple fix we need to add support for timestamp object inspector.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-20801: --- Status: Patch Available (was: Open) > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679734#comment-16679734
]
Hive QA commented on HIVE-20512:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
43s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} ql: The patch generated 0 new + 4 unchanged - 5
fixed = 4 total (was 9) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
47s{color} | {color:red} ql generated 1 new + 2315 unchanged - 0 fixed = 2316
total (was 2315) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Increment of volatile field
org.apache.hadoop.hive.ql.exec.spark.SparkRecordHandler.rowNumber in
org.apache.hadoop.hive.ql.exec.spark.SparkRecordHandler.incrementRowNumber()
At SparkRecordHandler.java:in
org.apache.hadoop.hive.ql.exec.spark.SparkRecordHandler.incrementRowNumber()
At SparkRecordHandler.java:[line 113] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14809/dev-support/hive-personality.sh
|
| git revision | master / ad597b0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14809/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14809/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch, HIVE-20512.2.patch,
> HIVE-20512.3.patch, HIVE-20512.4.patch, HIVE-20512.5.patch,
> HIVE-20512.6.patch, HIVE-20512.7.patch, HIVE-20512.8.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but
> we should improve the methodology for how frequently we log this info.
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if (currentThreshold >= 100) {
> return
[jira] [Updated] (HIVE-20886) Fix NPE: GenericUDFLower
[
https://issues.apache.org/jira/browse/HIVE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-20886:
Attachment: HIVE-20886.1.patch
> Fix NPE: GenericUDFLower
>
>
> Key: HIVE-20886
> URL: https://issues.apache.org/jira/browse/HIVE-20886
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Trivial
> Attachments: HIVE-20886.1.patch
>
>
> {noformat}
> create table if not exists test1(uuid array);
> select lower(uuid) from test1;
> Error: Error while compiling statement: FAILED: NullPointerException null
> (state=42000,code=4)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20734) Beeline: When beeline-site.xml is and hive CLI redirects to beeline, it should use the system username/dummy password instead of prompting for one
[ https://issues.apache.org/jira/browse/HIVE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20734: --- Status: Patch Available (was: Open) > Beeline: When beeline-site.xml is and hive CLI redirects to beeline, it > should use the system username/dummy password instead of prompting for one > -- > > Key: HIVE-20734 > URL: https://issues.apache.org/jira/browse/HIVE-20734 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: mahesh kumar behera >Priority: Major > Attachments: HIVE-20734.1.patch, HIVE-20734.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19925) NPE in SparkTask#printConsoleMetrics
[
https://issues.apache.org/jira/browse/HIVE-19925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali reassigned HIVE-19925:
---
Assignee: (was: Bharathkrishna Guruvayoor Murali)
> NPE in SparkTask#printConsoleMetrics
>
>
> Key: HIVE-19925
> URL: https://issues.apache.org/jira/browse/HIVE-19925
> Project: Hive
> Issue Type: Sub-task
>Reporter: Bharathkrishna Guruvayoor Murali
>Priority: Major
>
> When running a join query with HOS, as :
> {code:java}
> SELECT a.id FROM sample a JOIN sample b ON (a.id=b.id);{code}
> Got the following exception :
> {code:java}
> Error while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask. java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.printConsoleMetrics(SparkTask.java:229)
> at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:166)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2678)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2330)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2001)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1701)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1695)
> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:87)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:315)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:328)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) (state=08S01,code=1)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20826) Enhance HiveSemiJoin rule to convert join + group by on left side to Left Semi Join
[
https://issues.apache.org/jira/browse/HIVE-20826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680434#comment-16680434
]
Hive QA commented on HIVE-20826:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947307/HIVE-20826.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 15527 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_limit]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_join_ptp]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_in]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_views]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_mapjoin_reduce]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=190)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_in]
(batchId=139)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_views]
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_mapjoin_reduce]
(batchId=147)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14816/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14816/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14816/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947307 - PreCommit-HIVE-Build
> Enhance HiveSemiJoin rule to convert join + group by on left side to Left
> Semi Join
> ---
>
> Key: HIVE-20826
> URL: https://issues.apache.org/jira/browse/HIVE-20826
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20826.1.patch, HIVE-20826.2.patch,
> HIVE-20826.3.patch
>
>
> Currently HiveSemiJoin rule looks for pattern where group by is on right side.
> We can convert joins which have group by on left side (assuming group by keys
> are same as join keys and none of the columns are being projected from left
> side) to LEFT SEMI JOIN by swapping the inputs. e.g. queries such as:
> {code:sql}
> explain select pp.p_partkey from (select distinct p_name from part) p join
> part pp on pp.p_name = p.p_name;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18902) Lower Logging Level for Cleaning Up "local RawStore"
[
https://issues.apache.org/jira/browse/HIVE-18902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-18902:
---
Status: Patch Available (was: Open)
Rebased patch
> Lower Logging Level for Cleaning Up "local RawStore"
>
>
> Key: HIVE-18902
> URL: https://issues.apache.org/jira/browse/HIVE-18902
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: BELUGA BEHR
>Assignee: Bohdan Chupika
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18902.01.patch, HIVE-18902.02.patch,
> HIVE-18902.03.patch, HIVE-18902.04.patch, HIVE-18902.5.patch
>
>
> [https://github.com/apache/hive/blob/7c22d74c8d0eb0650adf6e84e0536127c103e46c/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L7756-L7768]
>
> {code:java}
> private static void cleanupRawStore() {
> try {
> RawStore rs = HMSHandler.getRawStore();
> if (rs != null) {
> HMSHandler.logInfo("Cleaning up thread local RawStore...");
> rs.shutdown();
> }
> } finally {
> HMSHandler handler = HMSHandler.threadLocalHMSHandler.get();
> if (handler != null) {
> handler.notifyMetaListenersOnShutDown();
> }
> HMSHandler.threadLocalHMSHandler.remove();
> HMSHandler.threadLocalConf.remove();
> HMSHandler.threadLocalModifiedConfig.remove();
> HMSHandler.removeRawStore();
> HMSHandler.logInfo("Done cleaning up thread local RawStore");
> }
> }
> {code}
> {code}
> 2018-03-03 17:21:49,832 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-21]: 19:
> Cleaning up thread local RawStore...
> 2018-03-03 17:21:49,834 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-21]: 19: Done
> cleaning up thread local RawStore
> {code}
> Not very helpful logging. Please change logging levels to _debug_ or even
> _trace_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20892) Benchmark XXhash for 64 bit hashing function instead of Murmum hash
[ https://issues.apache.org/jira/browse/HIVE-20892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680441#comment-16680441 ] Prasanth Jayachandran edited comment on HIVE-20892 at 11/8/18 9:40 PM: --- [https://github.com/prasanthj/hasher] has perf comparison for some of the non-cryptographic hashing algorithms. Murmur2 is slightly better in terms of perf than Murmur3 but for this reason [https://github.com/apache/hive/blob/master/storage-api/src/java/org/apache/hive/common/util/BloomFilter.java#L37-L40] Murmur3 is chosen for bloomfilter and HLL in Hive. was (Author: prasanth_j): [https://github.com/prasanthj/hasher] Murmur2 is slightly better in terms of perf than Murmur3 but for this reason [https://github.com/apache/hive/blob/master/storage-api/src/java/org/apache/hive/common/util/BloomFilter.java#L37-L40] Murmur3 is chosen for bloomfilter and HLL in Hive. > Benchmark XXhash for 64 bit hashing function instead of Murmum hash > --- > > Key: HIVE-20892 > URL: https://issues.apache.org/jira/browse/HIVE-20892 > Project: Hive > Issue Type: Sub-task >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > > https://cyan4973.github.io/xxHash/ > FYI this is used by lot of other MPP systems ... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18902) Lower Logging Level for Cleaning Up "local RawStore"
[
https://issues.apache.org/jira/browse/HIVE-18902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-18902:
---
Status: Open (was: Patch Available)
> Lower Logging Level for Cleaning Up "local RawStore"
>
>
> Key: HIVE-18902
> URL: https://issues.apache.org/jira/browse/HIVE-18902
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: BELUGA BEHR
>Assignee: Bohdan Chupika
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18902.01.patch, HIVE-18902.02.patch,
> HIVE-18902.03.patch, HIVE-18902.04.patch, HIVE-18902.5.patch
>
>
> [https://github.com/apache/hive/blob/7c22d74c8d0eb0650adf6e84e0536127c103e46c/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L7756-L7768]
>
> {code:java}
> private static void cleanupRawStore() {
> try {
> RawStore rs = HMSHandler.getRawStore();
> if (rs != null) {
> HMSHandler.logInfo("Cleaning up thread local RawStore...");
> rs.shutdown();
> }
> } finally {
> HMSHandler handler = HMSHandler.threadLocalHMSHandler.get();
> if (handler != null) {
> handler.notifyMetaListenersOnShutDown();
> }
> HMSHandler.threadLocalHMSHandler.remove();
> HMSHandler.threadLocalConf.remove();
> HMSHandler.threadLocalModifiedConfig.remove();
> HMSHandler.removeRawStore();
> HMSHandler.logInfo("Done cleaning up thread local RawStore");
> }
> }
> {code}
> {code}
> 2018-03-03 17:21:49,832 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-21]: 19:
> Cleaning up thread local RawStore...
> 2018-03-03 17:21:49,834 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-21]: 19: Done
> cleaning up thread local RawStore
> {code}
> Not very helpful logging. Please change logging levels to _debug_ or even
> _trace_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20831) Add Session ID to Operation Logging
[
https://issues.apache.org/jira/browse/HIVE-20831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680510#comment-16680510
]
Prasanth Jayachandran commented on HIVE-20831:
--
+1
> Add Session ID to Operation Logging
> ---
>
> Key: HIVE-20831
> URL: https://issues.apache.org/jira/browse/HIVE-20831
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Labels: newbie, noob
> Attachments: HIVE-20831.1.patch
>
>
> {code:java|title=OperationManager.java}
> LOG.info("Adding operation: " + operation.getHandle());
> {code}
> Please add additional logging to explicitly state which Hive session this
> operation is being added to.
> https://github.com/apache/hive/blob/3963c729fabf90009cb67d277d40fe5913936358/service/src/java/org/apache/hive/service/cli/operation/OperationManager.java#L201
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19354) from_utc_timestamp returns incorrect results for datetime values with timezone
[
https://issues.apache.org/jira/browse/HIVE-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680429#comment-16680429
]
Bharathkrishna Guruvayoor Murali commented on HIVE-19354:
-
Unassigning this Jira as I am not planning to work on it any time soon. I have
attached a patch, but there could be a better way to do this.
> from_utc_timestamp returns incorrect results for datetime values with timezone
> --
>
> Key: HIVE-19354
> URL: https://issues.apache.org/jira/browse/HIVE-19354
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Bruce Robbins
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19354.01.patch
>
>
> On the master branch, from_utc_timestamp returns incorrect results for
> datetime strings that contain a timezone:
> {noformat}
> hive> select from_utc_timestamp('2000-10-10 00:00:00+00:00',
> 'America/Los_Angeles');
> OK
> 2000-10-09 10:00:00
> Time taken: 0.294 seconds, Fetched: 1 row(s)
> hive> select from_utc_timestamp('2000-10-10 00:00:00', 'America/Los_Angeles');
> OK
> 2000-10-09 17:00:00
> Time taken: 0.121 seconds, Fetched: 1 row(s)
> hive>
> {noformat}
> Both inputs are 2000-10-10 00:00:00 in UTC time, but I got two different
> results.
> In version 2.3.3, from_utc_timestamp doesn't accept timezones in its input
> strings, so it does not have this bug:
> {noformat}
> hive> select from_utc_timestamp('2000-10-10 00:00:00+00:00',
> 'America/Los_Angeles');
> OK
> NULL
> Time taken: 5.152 seconds, Fetched: 1 row(s)
> hive> select from_utc_timestamp('2000-10-10 00:00:00', 'America/Los_Angeles');
> OK
> 2000-10-09 17:00:00
> Time taken: 0.069 seconds, Fetched: 1 row(s)
> hive>
> {noformat}
> Since the function is expecting a UTC datetime value, it probably should
> continue to reject input that contains a timezone component.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-8446) Failing to delete data after dropping a table or database should result in error
[ https://issues.apache.org/jira/browse/HIVE-8446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali reassigned HIVE-8446: -- Assignee: (was: Bharathkrishna Guruvayoor Murali) > Failing to delete data after dropping a table or database should result in > error > > > Key: HIVE-8446 > URL: https://issues.apache.org/jira/browse/HIVE-8446 > Project: Hive > Issue Type: Bug >Reporter: Brock Noland >Priority: Major > Attachments: HIVE-8446.1.patch > > > Currently if we drop a table and it fails to delete the data, the command > completes successfully. We should instead return an error to the user. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20665) Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-20665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20665:
---
Status: Open (was: Patch Available)
> Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
>
>
> Key: HIVE-20665
> URL: https://issues.apache.org/jira/browse/HIVE-20665
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0, 2.3.2, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-20665.1.patch, HIVE-20665.1.patch
>
>
> When parallel tasks are enabled in Hive, all of the resulting queries share
> the same Hive configuration. This is problematic as each query will modify
> the same {{HiveConf}} object with things like query ID and query text. This
> will overwrite each other and cause {{ConcurrentModificationException}}
> issues.
> {code:java|title=SQLOperation.java}
> public Object run() throws HiveSQLException {
> Hive.set(parentHive, false);
> // TODO: can this result in cross-thread reuse of session state?
> SessionState.setCurrentSessionState(parentSessionState);
> PerfLogger.setPerfLogger(SessionState.getPerfLogger());
> LogUtils.registerLoggingContext(queryState.getConf());
> try {
> if (asyncPrepare) {
> prepare(queryState);
> }
> runQuery();
> } catch (HiveSQLException e) {
> // ...
> {code}
> [Code
> Here|https://github.com/apache/hive/blob/6e27a5315a44c55ef3b178e7212c9068de322d01/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L308-L319]
> From this code it can be seen that for every thread launched, it is all
> calling {{setCurrentSessionState}}.
> {code:java|title=SessionStates.java}
> /**
>* Sets the given session state in the thread local var for sessions.
>*/
> public static void setCurrentSessionState(SessionState startSs) {
> tss.get().attach(startSs);
> }
> // SessionState is not available in runtime and Hive.get().getConf() is not
> safe to call
> private static class SessionStates {
> private SessionState state;
> private HiveConf conf;
> private void attach(SessionState state) {
> this.state = state;
> attach(state.getConf());
> }
> private void attach(HiveConf conf) {
> // -- SHALLOW COPY HERE, ALL THREADS SHARING SAME REFERENCE -- //
> this.conf = conf;
> ClassLoader classLoader = conf.getClassLoader();
> if (classLoader != null) {
> Thread.currentThread().setContextClassLoader(classLoader);
> }
> }
> }
> {code}
> [Code
> Here|https://github.com/apache/hive/blob/7795c0a7dc59941671f8845d78b16d9e5ddc9ea3/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L540-L556]
> Ensure that all threads get their own copy of the {{HiveConf}} object to use
> and modify.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20838) Timestamps with timezone are set to null when using the streaming API
[
https://issues.apache.org/jira/browse/HIVE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679697#comment-16679697
]
Hive QA commented on HIVE-20838:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947268/HIVE-20838.8.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15526 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14808/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14808/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14808/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947268 - PreCommit-HIVE-Build
> Timestamps with timezone are set to null when using the streaming API
> -
>
> Key: HIVE-20838
> URL: https://issues.apache.org/jira/browse/HIVE-20838
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20838.1.patch, HIVE-20838.2.patch,
> HIVE-20838.3.patch, HIVE-20838.3.patch, HIVE-20838.4.patch,
> HIVE-20838.5.patch, HIVE-20838.6.patch, HIVE-20838.7.patch, HIVE-20838.8.patch
>
>
> For example:
> {code}
> beeline> create table default.timest (a TIMESTAMP) stored as orc " +
> "TBLPROPERTIES('transactional'='true')
> # And then:
> connection.write("2018-10-19 10:35:00 America/Los_Angeles".getBytes());
> {code}
> inserts NULL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20850) Push case conditional from projections to dimension tables if possible
[
https://issues.apache.org/jira/browse/HIVE-20850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-20850:
Attachment: HIVE-20850.04.patch
> Push case conditional from projections to dimension tables if possible
> --
>
> Key: HIVE-20850
> URL: https://issues.apache.org/jira/browse/HIVE-20850
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20850.01.patch, HIVE-20850.01.patch,
> HIVE-20850.01.patch, HIVE-20850.02.patch, HIVE-20850.03.patch,
> HIVE-20850.04.patch
>
>
> noticed by [~gopalv]: If there is a project which could be only evaluated
> after the join; but the condition references only a single column from a
> small dimension table; hive will end up evaluating the same thing over and
> over again...
> {code}
> explain
> select s_store_name, s_store_id,
> sum(case when (d_day_name='Sunday') then ss_sales_price else null
> end) sun_sales,
> sum(case when (d_day_name='Monday') then ss_sales_price else null
> end) mon_sales,
> sum(case when (d_day_name='Tuesday') then ss_sales_price else null
> end) tue_sales,
> sum(case when (d_day_name='Wednesday') then ss_sales_price else null
> end) wed_sales,
> sum(case when (d_day_name='Thursday') then ss_sales_price else null
> end) thu_sales,
> sum(case when (d_day_name='Friday') then ss_sales_price else null
> end) fri_sales,
> sum(case when (d_day_name='Saturday') then ss_sales_price else null
> end) sat_sales
> from date_dim, store_sales, store
> where d_date_sk = ss_sold_date_sk and
>s_store_sk = ss_store_sk and
>s_gmt_offset = -6 and
>d_year = 1998
> group by s_store_name, s_store_id
> order by s_store_name,
> s_store_id,sun_sales,mon_sales,tue_sales,wed_sales,thu_sales,fri_sales,sat_sales
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20665) Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-20665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20665:
---
Status: Patch Available (was: Open)
> Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
>
>
> Key: HIVE-20665
> URL: https://issues.apache.org/jira/browse/HIVE-20665
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0, 2.3.2, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-20665.1.patch, HIVE-20665.1.patch
>
>
> When parallel tasks are enabled in Hive, all of the resulting queries share
> the same Hive configuration. This is problematic as each query will modify
> the same {{HiveConf}} object with things like query ID and query text. This
> will overwrite each other and cause {{ConcurrentModificationException}}
> issues.
> {code:java|title=SQLOperation.java}
> public Object run() throws HiveSQLException {
> Hive.set(parentHive, false);
> // TODO: can this result in cross-thread reuse of session state?
> SessionState.setCurrentSessionState(parentSessionState);
> PerfLogger.setPerfLogger(SessionState.getPerfLogger());
> LogUtils.registerLoggingContext(queryState.getConf());
> try {
> if (asyncPrepare) {
> prepare(queryState);
> }
> runQuery();
> } catch (HiveSQLException e) {
> // ...
> {code}
> [Code
> Here|https://github.com/apache/hive/blob/6e27a5315a44c55ef3b178e7212c9068de322d01/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L308-L319]
> From this code it can be seen that for every thread launched, it is all
> calling {{setCurrentSessionState}}.
> {code:java|title=SessionStates.java}
> /**
>* Sets the given session state in the thread local var for sessions.
>*/
> public static void setCurrentSessionState(SessionState startSs) {
> tss.get().attach(startSs);
> }
> // SessionState is not available in runtime and Hive.get().getConf() is not
> safe to call
> private static class SessionStates {
> private SessionState state;
> private HiveConf conf;
> private void attach(SessionState state) {
> this.state = state;
> attach(state.getConf());
> }
> private void attach(HiveConf conf) {
> // -- SHALLOW COPY HERE, ALL THREADS SHARING SAME REFERENCE -- //
> this.conf = conf;
> ClassLoader classLoader = conf.getClassLoader();
> if (classLoader != null) {
> Thread.currentThread().setContextClassLoader(classLoader);
> }
> }
> }
> {code}
> [Code
> Here|https://github.com/apache/hive/blob/7795c0a7dc59941671f8845d78b16d9e5ddc9ea3/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L540-L556]
> Ensure that all threads get their own copy of the {{HiveConf}} object to use
> and modify.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20891) Call alter_partition in batch when dynamically loading partitions
[ https://issues.apache.org/jira/browse/HIVE-20891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20891: - Attachment: HIVE-20891.01.patch > Call alter_partition in batch when dynamically loading partitions > - > > Key: HIVE-20891 > URL: https://issues.apache.org/jira/browse/HIVE-20891 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20891.01.patch > > > When dynamically loading partitions, the setStatsPropAndAlterPartition() is > called for each partition one by one, resulting in unnecessary calls to the > metastore client. This whole logic can be changed to just one call. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20223) SmallTableCache.java SLF4J Parameterized Logging
[
https://issues.apache.org/jira/browse/HIVE-20223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679970#comment-16679970
]
Hive QA commented on HIVE-20223:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947275/HIVE-20223.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15526 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14811/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14811/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14811/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12947275 - PreCommit-HIVE-Build
> SmallTableCache.java SLF4J Parameterized Logging
>
>
> Key: HIVE-20223
> URL: https://issues.apache.org/jira/browse/HIVE-20223
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Labels: newbie, noob
> Attachments: HIVE-20223.1.patch
>
>
> {code:java|title=org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java}
> if (LOG.isDebugEnabled()) {
> LOG.debug("Cleaned up small table cache for query " + queryId);
> }
> if (tableContainerMap.putIfAbsent(path, tableContainer) == null &&
> LOG.isDebugEnabled()) {
> LOG.debug("Cached small table file " + path + " for query " + queryId);
> }
> if (tableContainer != null && LOG.isDebugEnabled()) {
> LOG.debug("Loaded small table file " + path + " from cache for query "
> + queryId);
> }
> {code}
>
> Remove {{isDebugEnabled}} and replace with parameterized logging.
> https://www.slf4j.org/faq.html#logging_performance
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16839) Unbalanced calls to openTransaction/commitTransaction when alter the same partition concurrently
[
https://issues.apache.org/jira/browse/HIVE-16839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680476#comment-16680476
]
Vihang Karajgaonkar commented on HIVE-16839:
yeah, we haven't had a release on branch-1 and below for a long time so I won't
count on it. You might be able to manually port the patch on your installation
since it is a simple patch.
> Unbalanced calls to openTransaction/commitTransaction when alter the same
> partition concurrently
>
>
> Key: HIVE-16839
> URL: https://issues.apache.org/jira/browse/HIVE-16839
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.1, 1.1.0, 3.0.0, 2.3.4
>Reporter: Nemon Lou
>Assignee: Guang Yang
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-16839.01.patch, HIVE-16839.02.patch,
> HIVE-16839.03.patch
>
>
> SQL to reproduce:
> prepare:
> {noformat}
> hdfs dfs -mkdir -p
> /hzsrc/external/writing_dc/ltgsm/16e7a9b2-21a1-3f4f-8061-bc3395281627
> 1,create external table tb_ltgsm_external (id int) PARTITIONED by (cp
> string,ld string);
> {noformat}
> open one beeline run these two sql many times
> {noformat} 2,ALTER TABLE tb_ltgsm_external ADD IF NOT EXISTS PARTITION
> (cp=2017060513,ld=2017060610);
> 3,ALTER TABLE tb_ltgsm_external PARTITION (cp=2017060513,ld=2017060610) SET
> LOCATION
> 'hdfs://hacluster/hzsrc/external/writing_dc/ltgsm/16e7a9b2-21a1-3f4f-8061-bc3395281627';
> {noformat}
> open another beeline to run this sql many times at the same time.
> {noformat}
> 4,ALTER TABLE tb_ltgsm_external DROP PARTITION (cp=2017060513,ld=2017060610);
> {noformat}
> MetaStore logs:
> {noformat}
> 2017-06-06 21:58:34,213 | ERROR | pool-6-thread-197 | Retrying HMSHandler
> after 2000 ms (attempt 1 of 10) with error:
> javax.jdo.JDOObjectNotFoundException: No such database row
> FailedObject:49[OID]org.apache.hadoop.hive.metastore.model.MStorageDescriptor
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:475)
> at
> org.datanucleus.api.jdo.JDOAdapter.getApiExceptionForNucleusException(JDOAdapter.java:1158)
> at
> org.datanucleus.state.JDOStateManager.isLoaded(JDOStateManager.java:3231)
> at
> org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoGetcd(MStorageDescriptor.java)
> at
> org.apache.hadoop.hive.metastore.model.MStorageDescriptor.getCD(MStorageDescriptor.java:184)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1282)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1299)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToPart(ObjectStore.java:1680)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartition(ObjectStore.java:1586)
> at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98)
> at com.sun.proxy.$Proxy0.getPartition(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:538)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions(HiveMetaStore.java:3317)
> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102)
> at com.sun.proxy.$Proxy12.alter_partitions(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:9963)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:9947)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1673)
> at
>
[jira] [Updated] (HIVE-19647) use bitvectors in IN operators
[ https://issues.apache.org/jira/browse/HIVE-19647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-19647: Attachment: HIVE-19647.03.patch > use bitvectors in IN operators > -- > > Key: HIVE-19647 > URL: https://issues.apache.org/jira/browse/HIVE-19647 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-19647.01.patch, HIVE-19647.02.patch, > HIVE-19647.03.patch, partial.patch > > > for strings there is no easy way to filter things out ; even min/max is null; > but the bitvectors could be used if they are available -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20836) Fix TestJdbcDriver2.testYarnATSGuid flakiness
[
https://issues.apache.org/jira/browse/HIVE-20836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-20836:
--
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master.
Thanks [~kuczoram] for the review!
> Fix TestJdbcDriver2.testYarnATSGuid flakiness
> -
>
> Key: HIVE-20836
> URL: https://issues.apache.org/jira/browse/HIVE-20836
> Project: Hive
> Issue Type: Test
> Components: JDBC
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20836.2.patch, HIVE-20836.3.patch, HIVE-20836.patch
>
>
> Seen flakiness in internal test.
> {code:java}
> Error Message
> Failed to set the YARN ATS Guid
> Stacktrace
> java.lang.AssertionError: Failed to set the YARN ATS Guid
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hive.jdbc.TestJdbcDriver2.testYarnATSGuid(TestJdbcDriver2.java:2434){code}
> The query finished too fast, and the GUID thread did not try to check the
> value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20734) Beeline: When beeline-site.xml is and hive CLI redirects to beeline, it should use the system username/dummy password instead of prompting for one
[ https://issues.apache.org/jira/browse/HIVE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20734: --- Status: Open (was: Patch Available) > Beeline: When beeline-site.xml is and hive CLI redirects to beeline, it > should use the system username/dummy password instead of prompting for one > -- > > Key: HIVE-20734 > URL: https://issues.apache.org/jira/browse/HIVE-20734 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: mahesh kumar behera >Priority: Major > Attachments: HIVE-20734.1.patch, HIVE-20734.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20893) BloomK Filter probing method is not thread safe
[ https://issues.apache.org/jira/browse/HIVE-20893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680454#comment-16680454 ] Clint Wylie commented on HIVE-20893: [~gopalv]: I'm throwing together a PR for this, but it's non obvious to me how to properly unroll the masks loop, what did you have in mind for the refactor? > BloomK Filter probing method is not thread safe > --- > > Key: HIVE-20893 > URL: https://issues.apache.org/jira/browse/HIVE-20893 > Project: Hive > Issue Type: Bug > Components: storage-api >Reporter: slim bouguerra >Priority: Major > > As far i can tell this is not an issue for Hive yet (most of the usage of > probing seems to be done by one thread at a time) but it is an issue of other > users like Druid as per the following > issue.[https://github.com/apache/incubator-druid/issues/6546] > The fix is proposed by the author of > [https://github.com/apache/incubator-druid/pull/6584] is to make couple of > local fields as ThreadLocals. > Idea looks good to me and doesn't have any perf drawbacks. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20826) Enhance HiveSemiJoin rule to convert join + group by on left side to Left Semi Join
[
https://issues.apache.org/jira/browse/HIVE-20826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679109#comment-16679109
]
Vineet Garg commented on HIVE-20826:
[~ashutoshc] Review is at https://reviews.apache.org/r/69294/
> Enhance HiveSemiJoin rule to convert join + group by on left side to Left
> Semi Join
> ---
>
> Key: HIVE-20826
> URL: https://issues.apache.org/jira/browse/HIVE-20826
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20826.1.patch, HIVE-20826.2.patch,
> HIVE-20826.3.patch
>
>
> Currently HiveSemiJoin rule looks for pattern where group by is on right side.
> We can convert joins which have group by on left side (assuming group by keys
> are same as join keys and none of the columns are being projected from left
> side) to LEFT SEMI JOIN by swapping the inputs. e.g. queries such as:
> {code:sql}
> explain select pp.p_partkey from (select distinct p_name from part) p join
> part pp on pp.p_name = p.p_name;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20886) Fix NPE: GenericUDFLower
[
https://issues.apache.org/jira/browse/HIVE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-20886:
Status: Patch Available (was: Open)
> Fix NPE: GenericUDFLower
>
>
> Key: HIVE-20886
> URL: https://issues.apache.org/jira/browse/HIVE-20886
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Trivial
> Attachments: HIVE-20886.1.patch
>
>
> {noformat}
> create table if not exists test1(uuid array);
> select lower(uuid) from test1;
> Error: Error while compiling statement: FAILED: NullPointerException null
> (state=42000,code=4)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20892) Benchmark XXhash for 64 bit hashing function instead of Murmum hash
[ https://issues.apache.org/jira/browse/HIVE-20892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680441#comment-16680441 ] Prasanth Jayachandran commented on HIVE-20892: -- [https://github.com/prasanthj/hasher] Murmur2 is slightly better in terms of perf than Murmur3 but for this reason [https://github.com/apache/hive/blob/master/storage-api/src/java/org/apache/hive/common/util/BloomFilter.java#L37-L40] Murmur3 is chosen for bloomfilter and HLL in Hive. > Benchmark XXhash for 64 bit hashing function instead of Murmum hash > --- > > Key: HIVE-20892 > URL: https://issues.apache.org/jira/browse/HIVE-20892 > Project: Hive > Issue Type: Sub-task >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > > https://cyan4973.github.io/xxHash/ > FYI this is used by lot of other MPP systems ... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20871) There are some syntax definition wrong in hqlsql.g4
[
https://issues.apache.org/jira/browse/HIVE-20871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
flying updated HIVE-20871:
--
Description:
hqlsql.g4 in version 3.x:
h2. 1. Any comment in the last line of sql text can not be identified.
sql:
{code:java}
// You will get an error if try parsing it.
select a from b; // I am comment
{code}
cause:
{code:java}
// line 1917
L_S_COMMENT : ('--' | '//') .*? '\r'? '\n' -> channel(HIDDEN) ; //
Single line comment
{code}
maybe the code can fix to:
{code:java}
L_S_COMMENT : ('--' | '//') .*? (('\r'? '\n') | EOF) -> channel(HIDDEN) ;
// Single line comment
// The EOF can be recognized as the end of the single line comment
{code}
h2. 2. "limit" syntax definition may be mistake.
two sqls:
{code:java}
select a from b limit 10; // just limit 10 record. this sql can be parsed
normally.
select a from b limit 10, 10; // pagnation, 10 record per page. you will get an
error if try parsing it.
{code}
cause:
{code:java}
// line 900
select_options_item :
T_LIMIT expr
| T_WITH (T_RR | T_RS | T_CS | T_UR) (T_USE T_AND T_KEEP (T_EXCLUSIVE |
T_UPDATE | T_SHARE) T_LOCKS)?
;
{code}
maybe the code can fix to:
{code:java}
// line 900
select_options_item :
T_LIMIT expr (T_COMMA expr)?
| T_WITH (T_RR | T_RS | T_CS | T_UR) (T_USE T_AND T_KEEP (T_EXCLUSIVE |
T_UPDATE | T_SHARE) T_LOCKS)?
;
// add optional '(T_COMMA expr)?' after 'T_LIMIT expr'.{code}
h2. 3. The word 'type' can not be recognized as a column name.
sql:
{code:java}
select type from someTable;
// 'type' is a column. you will get an error if try parsing it. but you can
execute the sql in hive query engine normally.
{code}
Cause the lexer 'T_TYPE' is not added to 'non_reserved_words'.
// Sorry for not good at English. Talk is cheap, just show you code. :)
was:
hqlsql.g4 in version 3.x:
h2. 1. Any comment in the last line of sql text can not be identified.
sql:
{code:java}
// You will get an error if try parsing it.
select a from b; // I am comment
{code}
cause:
{code:java}
// line 1917
L_S_COMMENT : ('--' | '//') .*? '\r'? '\n' -> channel(HIDDEN) ; //
Single line comment
{code}
maybe the code can fix to:
{code:java}
L_S_COMMENT : ('--' | '//') .*? (('\r'? '\n') | EOF) -> channel(HIDDEN) ;
// Single line comment
// The EOF can be recognized as the end of the single line comment
{code}
h2. 2. "limit" syntax definition may be mistake.
two sqls:
{code:java}
select a from b limit 10; // just limit 10 record. this sql can be parsed
normally.
select a from b limit 10, 10; // pagnation, 10 record per page. you will get an
error if try parsing it.
{code}
cause:
{code:java}
// line 900
select_options_item :
T_LIMIT expr
| T_WITH (T_RR | T_RS | T_CS | T_UR) (T_USE T_AND T_KEEP (T_EXCLUSIVE |
T_UPDATE | T_SHARE) T_LOCKS)?
;
{code}
maybe the code can fix to:
{code:java}
// line 900
select_options_item :
T_LIMIT expr (T_COMMA expr)?
| T_WITH (T_RR | T_RS | T_CS | T_UR) (T_USE T_AND T_KEEP (T_EXCLUSIVE |
T_UPDATE | T_SHARE) T_LOCKS)?
;
// add optional '(T_COMMA expr)?' after 'T_LIMIT expr'.{code}
h2. 3. The word 'type' can not be recognized as a column name.
sql:
{code:java}
select type from someTable;
// 'type' is a column. you will get an error if try parsing it. but you can
execute the sql in hive query engine normally.
{code}
Cause the lexer 'T_TYPE' is not added to 'non_reserved_words'.
// Sorry for good at English. Talk is cheap, just show you code. :)
> There are some syntax definition wrong in hqlsql.g4
> ---
>
> Key: HIVE-20871
> URL: https://issues.apache.org/jira/browse/HIVE-20871
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 3.1.1
>Reporter: flying
>Priority: Major
>
> hqlsql.g4 in version 3.x:
> h2. 1. Any comment in the last line of sql text can not be identified.
> sql:
> {code:java}
> // You will get an error if try parsing it.
> select a from b; // I am comment
> {code}
> cause:
> {code:java}
> // line 1917
> L_S_COMMENT : ('--' | '//') .*? '\r'? '\n' -> channel(HIDDEN) ; //
> Single line comment
> {code}
> maybe the code can fix to:
> {code:java}
> L_S_COMMENT : ('--' | '//') .*? (('\r'? '\n') | EOF) -> channel(HIDDEN) ;
> // Single line comment
> // The EOF can be recognized as the end of the single line comment
> {code}
> h2. 2. "limit" syntax definition may be mistake.
> two sqls:
> {code:java}
> select a from b limit 10; // just limit 10 record. this sql can be parsed
> normally.
> select a from b limit 10, 10; // pagnation, 10 record per page. you will get
> an error if try parsing it.
> {code}
> cause:
> {code:java}
> // line 900
> select_options_item :
> T_LIMIT expr
> | T_WITH (T_RR | T_RS | T_CS | T_UR) (T_USE T_AND T_KEEP (T_EXCLUSIVE |
> T_UPDATE | T_SHARE)
[jira] [Assigned] (HIVE-13157) MetaStoreEventListener.onAlter triggered for INSERT and SELECT
[ https://issues.apache.org/jira/browse/HIVE-13157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali reassigned HIVE-13157: --- Assignee: (was: Bharathkrishna Guruvayoor Murali) > MetaStoreEventListener.onAlter triggered for INSERT and SELECT > -- > > Key: HIVE-13157 > URL: https://issues.apache.org/jira/browse/HIVE-13157 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 4.0.0 >Reporter: Eugen Stoianovici >Priority: Critical > > The event onAlter from > org.apache.hadoop.hive.metastore.MetaStoreEventListener is triggered when > INSERT or SELECT statements are executed on the target table. > Furthermore, the value of transient_lastDdl is updated in table properties > for INSERT statements. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19354) from_utc_timestamp returns incorrect results for datetime values with timezone
[
https://issues.apache.org/jira/browse/HIVE-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali reassigned HIVE-19354:
---
Assignee: (was: Bharathkrishna Guruvayoor Murali)
> from_utc_timestamp returns incorrect results for datetime values with timezone
> --
>
> Key: HIVE-19354
> URL: https://issues.apache.org/jira/browse/HIVE-19354
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Bruce Robbins
>Priority: Major
> Attachments: HIVE-19354.01.patch
>
>
> On the master branch, from_utc_timestamp returns incorrect results for
> datetime strings that contain a timezone:
> {noformat}
> hive> select from_utc_timestamp('2000-10-10 00:00:00+00:00',
> 'America/Los_Angeles');
> OK
> 2000-10-09 10:00:00
> Time taken: 0.294 seconds, Fetched: 1 row(s)
> hive> select from_utc_timestamp('2000-10-10 00:00:00', 'America/Los_Angeles');
> OK
> 2000-10-09 17:00:00
> Time taken: 0.121 seconds, Fetched: 1 row(s)
> hive>
> {noformat}
> Both inputs are 2000-10-10 00:00:00 in UTC time, but I got two different
> results.
> In version 2.3.3, from_utc_timestamp doesn't accept timezones in its input
> strings, so it does not have this bug:
> {noformat}
> hive> select from_utc_timestamp('2000-10-10 00:00:00+00:00',
> 'America/Los_Angeles');
> OK
> NULL
> Time taken: 5.152 seconds, Fetched: 1 row(s)
> hive> select from_utc_timestamp('2000-10-10 00:00:00', 'America/Los_Angeles');
> OK
> 2000-10-09 17:00:00
> Time taken: 0.069 seconds, Fetched: 1 row(s)
> hive>
> {noformat}
> Since the function is expecting a UTC datetime value, it probably should
> continue to reject input that contains a timezone component.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19925) NPE in SparkTask#printConsoleMetrics
[
https://issues.apache.org/jira/browse/HIVE-19925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680421#comment-16680421
]
Bharathkrishna Guruvayoor Murali commented on HIVE-19925:
-
Unassigning this Jira as I am not planning to look into this soon.
> NPE in SparkTask#printConsoleMetrics
>
>
> Key: HIVE-19925
> URL: https://issues.apache.org/jira/browse/HIVE-19925
> Project: Hive
> Issue Type: Sub-task
>Reporter: Bharathkrishna Guruvayoor Murali
>Priority: Major
>
> When running a join query with HOS, as :
> {code:java}
> SELECT a.id FROM sample a JOIN sample b ON (a.id=b.id);{code}
> Got the following exception :
> {code:java}
> Error while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask. java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.printConsoleMetrics(SparkTask.java:229)
> at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:166)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2678)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2330)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2001)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1701)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1695)
> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:87)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:315)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:328)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) (state=08S01,code=1)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20868) SMB Join fails intermittently when TezDummyOperator has child op in getFinalOp in MapRecordProcessor
[
https://issues.apache.org/jira/browse/HIVE-20868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679234#comment-16679234
]
Deepak Jaiswal commented on HIVE-20868:
---
[~gopalv] can you please review the v2 patch?
> SMB Join fails intermittently when TezDummyOperator has child op in
> getFinalOp in MapRecordProcessor
>
>
> Key: HIVE-20868
> URL: https://issues.apache.org/jira/browse/HIVE-20868
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-20868.1.patch, HIVE-20868.2.patch
>
>
> In MapRecordProcessor::getFinalOp() due to external cause(not known), the
> TezDummyStoreOperator may have MergeJoin Op as child intermittently. Due to
> this, the fetchDone remains set to true for the DummyOp which was set by
> previous task. Ideally, fetchDone should be reset for each task. This
> eventually leads to the join op skip rows from that dummy op resulting in
> wrong results.
> Good init order
> {code}
> 2018-11-01 21:42:33,677 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp child Ops = TS[3] (core)
> 2018-11-01 21:42:33,677 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp child Ops = FIL[24]
> 2018-11-01 21:42:33,677 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp child Ops = SEL[5]
> 2018-11-01 21:42:33,677 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp child Ops = DUMMY_STORE[45]
> 2018-11-01 21:42:33,677 [INFO] [TezChild] |tez.MapRecordProcessor|: Iterating
> children of dummy op DUMMY_STORE[45]
> 2018-11-01 21:42:33,677 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp returns DUMMY_STORE[45]
> 2018-11-01 21:42:33,677 [INFO] [TezChild] |tez.MapRecordProcessor|:
> InitProcessor : setting fetchDone to false
> {code}
> Bad init order
> {code}
> 2018-11-01 21:42:33,304 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp child Ops = TS[3] (core)
> 2018-11-01 21:42:33,304 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp child Ops = FIL[24]
> 2018-11-01 21:42:33,304 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp child Ops = SEL[5]
> 2018-11-01 21:42:33,304 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp child Ops = DUMMY_STORE[45]
> 2018-11-01 21:42:33,304 [INFO] [TezChild] |tez.MapRecordProcessor|:
> Iterating children of dummy op DUMMY_STORE[45]
> 2018-11-01 21:42:33,304 [INFO] [TezChild] |tez.MapRecordProcessor|: Child of
> Dummy Op MERGEJOIN[44]
> 2018-11-01 21:42:33,304 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp child Ops = MERGEJOIN[44]
> 2018-11-01 21:42:33,304 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp child Ops = SEL[13]
> 2018-11-01 21:42:33,304 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp child Ops = RS[14]
> 2018-11-01 21:42:33,304 [INFO] [TezChild] |tez.MapRecordProcessor|:
> getFinalOp returns RS[14]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-18772) Make Acid Cleaner use MIN_HISTORY_LEVEL
[ https://issues.apache.org/jira/browse/HIVE-18772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman resolved HIVE-18772. --- Resolution: Duplicate Fix Version/s: 4.0.0 Release Note: n/a this is addressed in HIVE-20823 > Make Acid Cleaner use MIN_HISTORY_LEVEL > --- > > Key: HIVE-18772 > URL: https://issues.apache.org/jira/browse/HIVE-18772 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-18772.01.patch, HIVE-18772.02.patch, > HIVE-18772.02.patch, HIVE-18772.03.patch, HIVE-18772.04.patch > > > Instead of using Lock Manager state as it currently does. > This will eliminate possible race conditions > See this > [comment|https://issues.apache.org/jira/browse/HIVE-18192?focusedCommentId=16338208=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16338208] > Suppose A is the set of all ValidTxnList across all active readers. Each > ValidTxnList has minOpenTxnId. > MIN_HISTORY_LEVEL allows us to determine X = min(minOpenTxnId) across all > currently active readers > This means that no active transaction in the system sees any txn with txnid < > X as open. > This means if construct ValidTxnIdList with HWM=X-1 and use that in > getAcidState(), any files determined by this call as 'obsolete', will be seen > as obsolete by any existing/future reader, i.e. can be physically deleted. > This is also necessary for multi-statement transactions where relying on the > state of Lock Manager is not sufficient. For example > Suppose txn 17 starts at t1 and sees txnid 13 with writeID 13 open. > 13 commits (via it's parent txn) at t2 > t1. (17 is still running). > Compaction runs at t3 >t2 to produce base_14 (or delta_10_14 for example) on > Table1/Part1 (17 is still running) > Now delta_13 may be cleaned since it can be seen as obsolete and there may be > no locks on it, i.e. no one is reading it. > Now at t4 > t3 17 may (multi stmt txn) needs to read Table1/Part1. It cannot > use base_14 is that may have absorbed delete events from delete_delta_14. > Another Use Case > There is delta_1_1 and delta_2_2 on disk both created by committed txns. > T5 starts reading these. At the same time compactor creates delta_1_2. > Now Cleaner sees delta_1_1 and delta_1_2 as obsolete and may remove them > while the read is still in progress. This is because Compactor itself is not > running in a txn and the files that > it produces are visible immediately. If it ran in a txn, the new files would > only be visible once > this txn is visible to others (including the Cleaner). > Using MIN_HISTORY_LEVEL solves this. > See description of HIVE-18747 for more details on MIN_HISTORY_LEVEL -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-20801: --- Labels: Branch3Candidate TODOC (was: Branch3Candidate) > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20881) Constant propagation oversimplifies projections
[
https://issues.apache.org/jira/browse/HIVE-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-20881:
Attachment: HIVE-20881.01.patch
> Constant propagation oversimplifies projections
> ---
>
> Key: HIVE-20881
> URL: https://issues.apache.org/jira/browse/HIVE-20881
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20881.01.patch
>
>
> {code:java}
> create table cx2(bool1 boolean);
> insert into cx2 values (true),(false),(null);
> set hive.cbo.enable=true;
> select bool1 IS TRUE OR (cast(NULL as boolean) AND bool1 IS NOT TRUE AND
> bool1 IS NOT FALSE) from cx2;
> ++
> | _c0 |
> ++
> | true |
> | false |
> | NULL |
> ++
> set hive.cbo.enable=false;
> select bool1 IS TRUE OR (cast(NULL as boolean) AND bool1 IS NOT TRUE AND
> bool1 IS NOT FALSE) from cx2;
> +---+
> | _c0 |
> +---+
> | true |
> | NULL |
> | NULL |
> +---+
> {code}
> from explain it seems the expression was simplified to: {{(_col0 is true or
> null)}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20889) Support timestamp-micros in AvroSerDe
[ https://issues.apache.org/jira/browse/HIVE-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinisha updated HIVE-20889: --- Affects Version/s: 1.1.0 > Support timestamp-micros in AvroSerDe > - > > Key: HIVE-20889 > URL: https://issues.apache.org/jira/browse/HIVE-20889 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 1.1.0 >Reporter: vinisha >Priority: Major > > This change only supports timestamp-millis. Avro 1.8.2 also supports > timestamp-micros. > [https://avro.apache.org/docs/1.8.2/spec.html#Timestamp+%28microsecond+precision%29] > timestamp-micros should also be supported in hive AvroSerde because hive > timestamps support nano second level precision. > [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-TimestampstimestampTimestamps] > One possibility is to support avro timestamp-millis and avro timestamp-micros > in serialization. Avro Deserializer can map hive timestamp to > timestamp-micros. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20845) Fix TestJdbcWithDBTokenStoreNoDoAs flakiness
[
https://issues.apache.org/jira/browse/HIVE-20845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679096#comment-16679096
]
Peter Vary commented on HIVE-20845:
---
The patch contains 2 changes:
* Fix for {{TestJdbcWithDBTokenStore}} - Several more configuration changes
done by MiniHS2 init (USE_THRIFT_SASL, KERBEROS_PRINCIPAL,
KERBEROS_KEYTAB_FILE) should be set in System properties, so the DriverManager
can use them. This fixes the long outstanding issue, that 2 HMS instances is
used here
* The MiniHS2 {{waitForStartup}} should wait not only for the possibility to
open the session, but for the possibility to open the transport as well. This
could be done by using {{DriverManager.getConnection}} instead of
{{hs2Client.openSession}}. The {{DriverManager.getConnection}} will open a
HiveConnection which calls {{openTransport}} and {{openSession}} as well.
Running tests to see what will happen with the HA passive tests :)
> Fix TestJdbcWithDBTokenStoreNoDoAs flakiness
>
>
> Key: HIVE-20845
> URL: https://issues.apache.org/jira/browse/HIVE-20845
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-20845.patch
>
>
> Previously did a dirty fix for TestJdbcWithDBTokenStoreNoDoAs and
> TestJdbcWithDBTokenStore
> Found out the issue is that we do not wait enough for HS2 to come up.
> Need to fix in MiniHS2.waitForStartup()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20730) Do delete event filtering even if hive.acid.index is not there
[
https://issues.apache.org/jira/browse/HIVE-20730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Saurabh Seth updated HIVE-20730:
Status: In Progress (was: Patch Available)
> Do delete event filtering even if hive.acid.index is not there
> --
>
> Key: HIVE-20730
> URL: https://issues.apache.org/jira/browse/HIVE-20730
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Saurabh Seth
>Priority: Major
> Attachments: HIVE-20730.patch
>
>
> since HIVE-16812 {{VectorizedOrcAcidRowBatchReader}} filters delete events
> based on min/max ROW__ID in the split which relies on {{hive.acid.index}} to
> be in the ORC footer.
> There is no way to generate {{hive.acid.index}} from a plain query as in
> HIVE-20699 and so we need to make sure that we generate a SARG into
> delete_delta/bucket_x based on stripe stats even the index is missing
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20734) Beeline: When beeline-site.xml is and hive CLI redirects to beeline, it should use the system username/dummy password instead of prompting for one
[ https://issues.apache.org/jira/browse/HIVE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera reassigned HIVE-20734: -- Assignee: mahesh kumar behera (was: Vaibhav Gumashta) > Beeline: When beeline-site.xml is and hive CLI redirects to beeline, it > should use the system username/dummy password instead of prompting for one > -- > > Key: HIVE-20734 > URL: https://issues.apache.org/jira/browse/HIVE-20734 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: mahesh kumar behera >Priority: Major > Attachments: HIVE-20734.1.patch, HIVE-20734.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20823) Make Compactor run in a transaction
[
https://issues.apache.org/jira/browse/HIVE-20823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679133#comment-16679133
]
Eugene Koifman commented on HIVE-20823:
---
patch 5 bunch more test fixes and some real (Replication) fixes
> Make Compactor run in a transaction
> ---
>
> Key: HIVE-20823
> URL: https://issues.apache.org/jira/browse/HIVE-20823
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-20823.01.patch, HIVE-20823.03.patch,
> HIVE-20823.04.patch, HIVE-20823.05.patch
>
>
> Have compactor open a transaction and run the job in that transaction.
> # make compactor produced base/delta include this txn id in the folder name,
> e.g. base_7_c17 where 17 is the txnid.
> # add {{CQ_TXN_ID bigint}} to COMPACTION_QUEUE and COMPLETED_COMPACTIONS to
> record this txn id
> # make sure {{AcidUtils.getAcidState()}} pays attention to this transaction
> on read and ignores this dir if this txn id is not committed in the current
> snapshot
> ## this means not only validWriteIdList but ValidTxnIdList should be passed
> along in config (if it isn't yet)
> # once this is done, {{CompactorMR.createCompactorMarker()}} can be
> eliminated and {{AcidUtils.isValidBase}} modified accordingly
> # modify Cleaner so that it doesn't clean old files until new file is visible
> to all readers
> #
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
