[jira] [Commented] (HIVE-19592) CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481253#comment-16481253 ] Sahil Takiar commented on HIVE-19592: - I've tried to look into this before, but could never find a root cause. As far as I know nothing is cleaning up /tmp/ on the ptest machines. Maybe some other unit test is clearing the /tmp/ directory, or maybe ptest is cleaning it up. Regardless, we have seen lots of issues with writing tmp files to /tmp/, the fix has always been to configure the test to not write to /tmp/. > CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19592 > URL: https://issues.apache.org/jira/browse/HIVE-19592 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19592.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19579) remove HBase transitive dependency that drags in some snapshot
[
https://issues.apache.org/jira/browse/HIVE-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481248#comment-16481248
]
Sahil Takiar commented on HIVE-19579:
-
hmmm think I have a fix for {{TestSparkPerfCliDriver}}, its the same as
HIVE-19562, I just missed a few {{hive-site.xml}} files
> remove HBase transitive dependency that drags in some snapshot
> --
>
> Key: HIVE-19579
> URL: https://issues.apache.org/jira/browse/HIVE-19579
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19579.01.patch, HIVE-19579.02.patch,
> HIVE-19579.patch
>
>
> This:
> {noformat}
> [INFO] org.apache.hive:hive-llap-server:jar:3.1.0-SNAPSHOT
> [INFO] \- org.apache.hbase:hbase-server:jar:2.0.0-alpha4:compile
> [INFO]\- org.glassfish.web:javax.servlet.jsp:jar:2.3.2:compile
> [INFO] \- org.glassfish:javax.el:jar:3.0.1-b10:compile (version
> selected from constraint [3.0.0,))
> {noformat}
> occasionally results in bizarre build failures like this:
> {noformat}
> Failed to read artifact descriptor for
> org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT: Could not transfer artifact
> org.glassfish:javax.el:pom:3.0.1-b06-SNAPSHOT from/to jvnet-nexus-snapshots
> (https://maven.java.net/content/repositories/snapshots): Failed to transfer
> file:
> https://maven.java.net/content/repositories/snapshots/org/glassfish/javax.el/3.0.1-b06-SNAPSHOT/javax.el-3.0.1-b06-SNAPSHOT.pom.
> Return code is: 402 , ReasonPhrase:Payment Required.
> {noformat}
> cc [~enis] [~stack] if you guys want to take a look at HBase side
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19592) CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481234#comment-16481234 ] Vihang Karajgaonkar commented on HIVE-19592: As far as I know Ptest doesn't do any cleanup in /tmp/hadoop/mapred directory. cc [~stakiar] in case he knows anything different. > CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19592 > URL: https://issues.apache.org/jira/browse/HIVE-19592 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19592.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19586) Optimize Count(distinct X) pushdown based on the storage capabilities
[
https://issues.apache.org/jira/browse/HIVE-19586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481232#comment-16481232
]
Ashutosh Chauhan commented on HIVE-19586:
-
+1
> Optimize Count(distinct X) pushdown based on the storage capabilities
> --
>
> Key: HIVE-19586
> URL: https://issues.apache.org/jira/browse/HIVE-19586
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration, Logical Optimizer
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19586.2.patch, HIVE-19586.3.patch, HIVE-19586.patch
>
>
> h1. Goal
> Provide a way to rewrite queries with combination of COUNT(Distinct) and
> Aggregates like SUM as a series of Group By.
> This can be useful to push down to Druid queries like
> {code}
> select count(DISTINCT interval_marker), count (distinct dim), sum(num_l)
> FROM druid_test_table GROUP BY `__time`, `zone` ;
> {code}
> In general this can be useful to be used in cases where storage handlers can
> not perform count (distinct column)
> h1. How to do it.
> Use the Calcite rule {code}
> org.apache.calcite.rel.rules.AggregateExpandDistinctAggregatesRule{code} that
> breaks down Count distinct to a single Group by with Grouping sets or
> multiple series of Group by that might be linked with Joins if multiple
> counts are present.
> FYI today Hive does have a similar rule {code}
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveExpandDistinctAggregatesRule{code},
> but it only provides a rewrite to Grouping sets based plan.
> I am planing to use the actual Calcite rule, [~ashutoshc] any concerns or
> caveats to be aware of?
> h2. Concerns/questions
> Need to have a way to switch between Grouping sets or Simple chained group by
> based on the plan cost. For instance for Druid based scan it makes always
> sense (at least today) to push down a series of Group by and stitch result
> sets in Hive later (as oppose to scan everything).
> But this might be not true for other storage handler that can handle Grouping
> sets it is better to push down the Grouping sets as one table scan.
> Am still unsure how i can lean on the cost optimizer to select the best plan,
> [~ashutoshc]/[~jcamachorodriguez] any inputs?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19604) Incorrect Handling of Boolean in DruidSerde
[ https://issues.apache.org/jira/browse/HIVE-19604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481222#comment-16481222 ] Ashutosh Chauhan commented on HIVE-19604: - Do we need instanceof check? These are results returned from Druid and they will always be int. I think its safe to assume type is int since its unlikely that Druid will change that. > Incorrect Handling of Boolean in DruidSerde > --- > > Key: HIVE-19604 > URL: https://issues.apache.org/jira/browse/HIVE-19604 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19604.patch > > > Results of boolean expressions from Druid are expressed in the form of > numeric 1 or 0. > When reading the results in DruidSerde both 1 and 0 are translated to String > and then we call Boolean.valueOf(stringForm), this leads to the boolean being > read always as false. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19612) Add option to mask lineage in q files
[
https://issues.apache.org/jira/browse/HIVE-19612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481221#comment-16481221
]
Jesus Camacho Rodriguez commented on HIVE-19612:
[~sershe], can you take a look? Thanks
> Add option to mask lineage in q files
> -
>
> Key: HIVE-19612
> URL: https://issues.apache.org/jira/browse/HIVE-19612
> Project: Hive
> Issue Type: Test
> Components: Test, Testing Infrastructure
>Affects Versions: 3.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19612.patch
>
>
> Similar to {{HIVE-19572}}, with {{-- MASK_LINEAGE}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19612) Add option to mask lineage in q files
[
https://issues.apache.org/jira/browse/HIVE-19612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19612:
---
Status: Open (was: Patch Available)
> Add option to mask lineage in q files
> -
>
> Key: HIVE-19612
> URL: https://issues.apache.org/jira/browse/HIVE-19612
> Project: Hive
> Issue Type: Test
> Components: Test, Testing Infrastructure
>Affects Versions: 3.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19612.patch
>
>
> Similar to {{HIVE-19572}}, with {{-- MASK_LINEAGE}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19612) Add option to mask lineage in q files
[
https://issues.apache.org/jira/browse/HIVE-19612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19612:
---
Status: Patch Available (was: In Progress)
> Add option to mask lineage in q files
> -
>
> Key: HIVE-19612
> URL: https://issues.apache.org/jira/browse/HIVE-19612
> Project: Hive
> Issue Type: Test
> Components: Test, Testing Infrastructure
>Affects Versions: 3.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19612.patch
>
>
> Similar to {{HIVE-19572}}, with {{-- MASK_LINEAGE}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-19612) Add option to mask lineage in q files
[
https://issues.apache.org/jira/browse/HIVE-19612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-19612 started by Jesus Camacho Rodriguez.
--
> Add option to mask lineage in q files
> -
>
> Key: HIVE-19612
> URL: https://issues.apache.org/jira/browse/HIVE-19612
> Project: Hive
> Issue Type: Test
> Components: Test, Testing Infrastructure
>Affects Versions: 3.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19612.patch
>
>
> Similar to {{HIVE-19572}}, with {{-- MASK_LINEAGE}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19612) Add option to mask lineage in q files
[
https://issues.apache.org/jira/browse/HIVE-19612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19612:
---
Attachment: HIVE-19612.patch
> Add option to mask lineage in q files
> -
>
> Key: HIVE-19612
> URL: https://issues.apache.org/jira/browse/HIVE-19612
> Project: Hive
> Issue Type: Test
> Components: Test, Testing Infrastructure
>Affects Versions: 3.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19612.patch
>
>
> Similar to {{HIVE-19572}}, with {{-- MASK_LINEAGE}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[ https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18079: - Attachment: HIVE-18079.17.patch > Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator > bit-size > > > Key: HIVE-18079 > URL: https://issues.apache.org/jira/browse/HIVE-18079 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Statistics >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18079-branch-3.patch, HIVE-18079-branch-3.patch, > HIVE-18079-branch-3.patch, HIVE-18079.1.patch, HIVE-18079.10.patch, > HIVE-18079.11.patch, HIVE-18079.12.patch, HIVE-18079.13.patch, > HIVE-18079.14.patch, HIVE-18079.15.patch, HIVE-18079.15.patch, > HIVE-18079.15.patch, HIVE-18079.16.patch, HIVE-18079.17.patch, > HIVE-18079.17.patch, HIVE-18079.2.patch, HIVE-18079.4.patch, > HIVE-18079.5.patch, HIVE-18079.6.patch, HIVE-18079.7.patch, > HIVE-18079.8.patch, HIVE-18079.9.patch > > > HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its > mathematical hash distribution & construction. > Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second > scan over the data-set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[ https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481220#comment-16481220 ] Prasanth Jayachandran commented on HIVE-18079: -- Yet another rebase > Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator > bit-size > > > Key: HIVE-18079 > URL: https://issues.apache.org/jira/browse/HIVE-18079 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Statistics >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18079-branch-3.patch, HIVE-18079-branch-3.patch, > HIVE-18079-branch-3.patch, HIVE-18079.1.patch, HIVE-18079.10.patch, > HIVE-18079.11.patch, HIVE-18079.12.patch, HIVE-18079.13.patch, > HIVE-18079.14.patch, HIVE-18079.15.patch, HIVE-18079.15.patch, > HIVE-18079.15.patch, HIVE-18079.16.patch, HIVE-18079.17.patch, > HIVE-18079.17.patch, HIVE-18079.2.patch, HIVE-18079.4.patch, > HIVE-18079.5.patch, HIVE-18079.6.patch, HIVE-18079.7.patch, > HIVE-18079.8.patch, HIVE-18079.9.patch > > > HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its > mathematical hash distribution & construction. > Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second > scan over the data-set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19579) remove HBase transitive dependency that drags in some snapshot
[
https://issues.apache.org/jira/browse/HIVE-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481216#comment-16481216
]
Hive QA commented on HIVE-19579:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923997/HIVE-19579.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14408 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query39]
(batchId=256)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11059/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11059/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11059/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923997 - PreCommit-HIVE-Build
> remove HBase transitive dependency that drags in some snapshot
> --
>
> Key: HIVE-19579
> URL: https://issues.apache.org/jira/browse/HIVE-19579
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19579.01.patch, HIVE-19579.02.patch,
> HIVE-19579.patch
>
>
> This:
> {noformat}
> [INFO] org.apache.hive:hive-llap-server:jar:3.1.0-SNAPSHOT
> [INFO] \- org.apache.hbase:hbase-server:jar:2.0.0-alpha4:compile
> [INFO]\- org.glassfish.web:javax.servlet.jsp:jar:2.3.2:compile
> [INFO] \- org.glassfish:javax.el:jar:3.0.1-b10:compile (version
> selected from constraint [3.0.0,))
> {noformat}
> occasionally results in bizarre build failures like this:
> {noformat}
> Failed to read artifact descriptor for
> org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT: Could not transfer artifact
> org.glassfish:javax.el:pom:3.0.1-b06-SNAPSHOT from/to jvnet-nexus-snapshots
> (https://maven.java.net/content/repositories/snapshots): Failed to transfer
> file:
> https://maven.java.net/content/repositories/snapshots/org/glassfish/javax.el/3.0.1-b06-SNAPSHOT/javax.el-3.0.1-b06-SNAPSHOT.pom.
> Return code is: 402 , ReasonPhrase:Payment Required.
> {noformat}
> cc [~enis] [~stack] if you guys want to take a look at HBase side
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-19612) Add option to mask lineage in q files
[
https://issues.apache.org/jira/browse/HIVE-19612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-19612 started by Jesus Camacho Rodriguez.
--
> Add option to mask lineage in q files
> -
>
> Key: HIVE-19612
> URL: https://issues.apache.org/jira/browse/HIVE-19612
> Project: Hive
> Issue Type: Test
> Components: Test, Testing Infrastructure
>Affects Versions: 3.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Similar to {{HIVE-19572}}, with {{-- MASK_LINEAGE}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19609) pointless callstacks in the logs as usual
[ https://issues.apache.org/jira/browse/HIVE-19609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481208#comment-16481208 ] Thejas M Nair commented on HIVE-19609: -- +1 Thanks! This is a patch I wished for many many times, finally granted by Sergey! :) > pointless callstacks in the logs as usual > - > > Key: HIVE-19609 > URL: https://issues.apache.org/jira/browse/HIVE-19609 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19609.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19612) Add option to mask lineage in q files
[
https://issues.apache.org/jira/browse/HIVE-19612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19612:
---
Description: Similar to {{HIVE-19572}}, with {{-- MASK_LINEAGE}}.
> Add option to mask lineage in q files
> -
>
> Key: HIVE-19612
> URL: https://issues.apache.org/jira/browse/HIVE-19612
> Project: Hive
> Issue Type: Test
> Components: Test, Testing Infrastructure
>Affects Versions: 3.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Similar to {{HIVE-19572}}, with {{-- MASK_LINEAGE}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19612) Add option to mask lineage in q files
[ https://issues.apache.org/jira/browse/HIVE-19612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-19612: -- > Add option to mask lineage in q files > - > > Key: HIVE-19612 > URL: https://issues.apache.org/jira/browse/HIVE-19612 > Project: Hive > Issue Type: Test > Components: Test, Testing Infrastructure >Affects Versions: 3.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19579) remove HBase transitive dependency that drags in some snapshot
[
https://issues.apache.org/jira/browse/HIVE-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481201#comment-16481201
]
Hive QA commented on HIVE-19579:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
6s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
11s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11059/dev-support/hive-personality.sh
|
| git revision | master / 99d94a9 |
| Default Java | 1.8.0_111 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11059/yetus/whitespace-eol.txt
|
| modules | C: . hbase-handler itests/hcatalog-unit itests/hive-minikdc
itests/hive-unit itests/hive-unit-hadoop2 itests/qtest itests/qtest-accumulo
itests/qtest-spark itests/util llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11059/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> remove HBase transitive dependency that drags in some snapshot
> --
>
> Key: HIVE-19579
> URL: https://issues.apache.org/jira/browse/HIVE-19579
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19579.01.patch, HIVE-19579.02.patch,
> HIVE-19579.patch
>
>
> This:
> {noformat}
> [INFO] org.apache.hive:hive-llap-server:jar:3.1.0-SNAPSHOT
> [INFO] \- org.apache.hbase:hbase-server:jar:2.0.0-alpha4:compile
> [INFO]\- org.glassfish.web:javax.servlet.jsp:jar:2.3.2:compile
> [INFO] \- org.glassfish:javax.el:jar:3.0.1-b10:compile (version
> selected from constraint [3.0.0,))
> {noformat}
> occasionally results in bizarre build failures like this:
> {noformat}
> Failed to read artifact descriptor for
> org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT: Could not transfer artifact
> org.glassfish:javax.el:pom:3.0.1-b06-SNAPSHOT from/to jvnet-nexus-snapshots
> (https://maven.java.net/content/repositories/snapshots): Failed to transfer
> file:
> https://maven.java.net/content/repositories/snapshots/org/glassfish/javax.el/3.0.1-b06-SNAPSHOT/javax.el-3.0.1-b06-SNAPSHOT.pom.
> Return code is: 402 , ReasonPhrase:Payment Required.
> {noformat}
> cc [~enis] [~stack] if you guys want to take a look at HBase side
--
This message was sent by Atlassian JIRA
(v7.6.3
[jira] [Commented] (HIVE-19603) Decrease batch size of TestMinimrCliDriver
[
https://issues.apache.org/jira/browse/HIVE-19603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481197#comment-16481197
]
Jesus Camacho Rodriguez commented on HIVE-19603:
Fantastic, thanks [~stakiar]!
> Decrease batch size of TestMinimrCliDriver
> --
>
> Key: HIVE-19603
> URL: https://issues.apache.org/jira/browse/HIVE-19603
> Project: Hive
> Issue Type: Test
> Components: Tests
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 3.1.0
>
>
> We have seen a lot of flakiness with the {{TestMinimrCliDriver}} - it keeps
> on timing out. I checked a recent Hive QA run and running the following tests
> locally takes my machine 1 hour:
> {code}
> mvn -B test -Dtest.groups= -Dtest=TestMinimrCliDriver
> -Dminimr.query.files=infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,scriptfile1.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q
> {code}
> On ptest, the timeout is 40 minutes. I suggest we decrease the batch size
> from 10 to 5.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-19603) Decrease batch size of TestMinimrCliDriver
[
https://issues.apache.org/jira/browse/HIVE-19603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar resolved HIVE-19603.
-
Resolution: Fixed
Fix Version/s: 3.1.0
> Decrease batch size of TestMinimrCliDriver
> --
>
> Key: HIVE-19603
> URL: https://issues.apache.org/jira/browse/HIVE-19603
> Project: Hive
> Issue Type: Test
> Components: Tests
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 3.1.0
>
>
> We have seen a lot of flakiness with the {{TestMinimrCliDriver}} - it keeps
> on timing out. I checked a recent Hive QA run and running the following tests
> locally takes my machine 1 hour:
> {code}
> mvn -B test -Dtest.groups= -Dtest=TestMinimrCliDriver
> -Dminimr.query.files=infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,scriptfile1.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q
> {code}
> On ptest, the timeout is 40 minutes. I suggest we decrease the batch size
> from 10 to 5.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19603) Decrease batch size of TestMinimrCliDriver
[
https://issues.apache.org/jira/browse/HIVE-19603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481186#comment-16481186
]
Sahil Takiar commented on HIVE-19603:
-
Haven't seen it fail recently. Marking this as resolved.
> Decrease batch size of TestMinimrCliDriver
> --
>
> Key: HIVE-19603
> URL: https://issues.apache.org/jira/browse/HIVE-19603
> Project: Hive
> Issue Type: Test
> Components: Tests
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 3.1.0
>
>
> We have seen a lot of flakiness with the {{TestMinimrCliDriver}} - it keeps
> on timing out. I checked a recent Hive QA run and running the following tests
> locally takes my machine 1 hour:
> {code}
> mvn -B test -Dtest.groups= -Dtest=TestMinimrCliDriver
> -Dminimr.query.files=infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,scriptfile1.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q
> {code}
> On ptest, the timeout is 40 minutes. I suggest we decrease the batch size
> from 10 to 5.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19594) CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
[
https://issues.apache.org/jira/browse/HIVE-19594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481183#comment-16481183
]
Sahil Takiar commented on HIVE-19594:
-
I've seen a lot of issues like this when tests write to {{/tmp/}} folder. Files
randomly disappear for some reason and the test fails with FNF. Best solution
is to configure things to never write to {{/tmp/}}.
> CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on
> HiveQA
>
>
> Key: HIVE-19594
> URL: https://issues.apache.org/jira/browse/HIVE-19594
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19594.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19611) reenable BHIF test on SparkonYarn if needed
[ https://issues.apache.org/jira/browse/HIVE-19611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481185#comment-16481185 ] Sahil Takiar commented on HIVE-19611: - Thanks for the follow up JIRA [~sershe], will take a look. > reenable BHIF test on SparkonYarn if needed > --- > > Key: HIVE-19611 > URL: https://issues.apache.org/jira/browse/HIVE-19611 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sahil Takiar >Priority: Major > > See HIVE-19608. Fails occasionally, looks like it's caused by an OOM in some > task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19611) reenable BHIF test on SparkonYarn if needed
[ https://issues.apache.org/jira/browse/HIVE-19611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-19611: --- > reenable BHIF test on SparkonYarn if needed > --- > > Key: HIVE-19611 > URL: https://issues.apache.org/jira/browse/HIVE-19611 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sahil Takiar >Priority: Major > > See HIVE-19608. Fails occasionally, looks like it's caused by an OOM in some > task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19610) reeenable union_stats test if needed
[ https://issues.apache.org/jira/browse/HIVE-19610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-19610: --- > reeenable union_stats test if needed > > > Key: HIVE-19610 > URL: https://issues.apache.org/jira/browse/HIVE-19610 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Ashutosh Chauhan >Priority: Major > > There's some flaky stats diff. See HIVE-19608 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19608) disable flaky tests 2
[
https://issues.apache.org/jira/browse/HIVE-19608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-19608:
Resolution: Fixed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
Committed to master. Thanks for the review!
> disable flaky tests 2
> -
>
> Key: HIVE-19608
> URL: https://issues.apache.org/jira/browse/HIVE-19608
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19608.patch
>
>
> union_stats
> {noformat}
> java.lang.AssertionError:
> Client Execution succeeded but contained differences (error code = 1) after
> executing union_stats.q
> 362a363
> > COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> 364a366,367
> > numRows 1000
> > rawDataSize 10624
> {noformat}
> Every few runs
> bucketizedinputformat for MiniSparkOnYarn; OOMs occasionally, example
> https://issues.apache.org/jira/browse/HIVE-19596
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19595) disable webui in MiniHS2
[ https://issues.apache.org/jira/browse/HIVE-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481171#comment-16481171 ] Sergey Shelukhin commented on HIVE-19595: - I'll wait for QA run for this one, in case some tests rely on it... will commit if only known flaky tests fail. > disable webui in MiniHS2 > > > Key: HIVE-19595 > URL: https://issues.apache.org/jira/browse/HIVE-19595 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19595.01.patch, HIVE-19595.patch > > > Causes tests flakiness if they same port is used. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19579) remove HBase transitive dependency that drags in some snapshot
[
https://issues.apache.org/jira/browse/HIVE-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-19579:
Attachment: HIVE-19579.02.patch
> remove HBase transitive dependency that drags in some snapshot
> --
>
> Key: HIVE-19579
> URL: https://issues.apache.org/jira/browse/HIVE-19579
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19579.01.patch, HIVE-19579.02.patch,
> HIVE-19579.patch
>
>
> This:
> {noformat}
> [INFO] org.apache.hive:hive-llap-server:jar:3.1.0-SNAPSHOT
> [INFO] \- org.apache.hbase:hbase-server:jar:2.0.0-alpha4:compile
> [INFO]\- org.glassfish.web:javax.servlet.jsp:jar:2.3.2:compile
> [INFO] \- org.glassfish:javax.el:jar:3.0.1-b10:compile (version
> selected from constraint [3.0.0,))
> {noformat}
> occasionally results in bizarre build failures like this:
> {noformat}
> Failed to read artifact descriptor for
> org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT: Could not transfer artifact
> org.glassfish:javax.el:pom:3.0.1-b06-SNAPSHOT from/to jvnet-nexus-snapshots
> (https://maven.java.net/content/repositories/snapshots): Failed to transfer
> file:
> https://maven.java.net/content/repositories/snapshots/org/glassfish/javax.el/3.0.1-b06-SNAPSHOT/javax.el-3.0.1-b06-SNAPSHOT.pom.
> Return code is: 402 , ReasonPhrase:Payment Required.
> {noformat}
> cc [~enis] [~stack] if you guys want to take a look at HBase side
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19579) remove HBase transitive dependency that drags in some snapshot
[
https://issues.apache.org/jira/browse/HIVE-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481168#comment-16481168
]
Sergey Shelukhin commented on HIVE-19579:
-
After looking at flaky tests today, all of these are known flaky.. guess we'll
keep running HiveQA and disabling tests in parallel until this can be committed
:)
> remove HBase transitive dependency that drags in some snapshot
> --
>
> Key: HIVE-19579
> URL: https://issues.apache.org/jira/browse/HIVE-19579
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19579.01.patch, HIVE-19579.02.patch,
> HIVE-19579.patch
>
>
> This:
> {noformat}
> [INFO] org.apache.hive:hive-llap-server:jar:3.1.0-SNAPSHOT
> [INFO] \- org.apache.hbase:hbase-server:jar:2.0.0-alpha4:compile
> [INFO]\- org.glassfish.web:javax.servlet.jsp:jar:2.3.2:compile
> [INFO] \- org.glassfish:javax.el:jar:3.0.1-b10:compile (version
> selected from constraint [3.0.0,))
> {noformat}
> occasionally results in bizarre build failures like this:
> {noformat}
> Failed to read artifact descriptor for
> org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT: Could not transfer artifact
> org.glassfish:javax.el:pom:3.0.1-b06-SNAPSHOT from/to jvnet-nexus-snapshots
> (https://maven.java.net/content/repositories/snapshots): Failed to transfer
> file:
> https://maven.java.net/content/repositories/snapshots/org/glassfish/javax.el/3.0.1-b06-SNAPSHOT/javax.el-3.0.1-b06-SNAPSHOT.pom.
> Return code is: 402 , ReasonPhrase:Payment Required.
> {noformat}
> cc [~enis] [~stack] if you guys want to take a look at HBase side
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19595) disable webui in MiniHS2
[ https://issues.apache.org/jira/browse/HIVE-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481165#comment-16481165 ] Jesus Camacho Rodriguez commented on HIVE-19595: I think so, please go ahead and commit that one too. +1 > disable webui in MiniHS2 > > > Key: HIVE-19595 > URL: https://issues.apache.org/jira/browse/HIVE-19595 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19595.01.patch, HIVE-19595.patch > > > Causes tests flakiness if they same port is used. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19608) disable flaky tests 2
[
https://issues.apache.org/jira/browse/HIVE-19608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481167#comment-16481167
]
Jesus Camacho Rodriguez commented on HIVE-19608:
LGTM +1
> disable flaky tests 2
> -
>
> Key: HIVE-19608
> URL: https://issues.apache.org/jira/browse/HIVE-19608
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19608.patch
>
>
> union_stats
> {noformat}
> java.lang.AssertionError:
> Client Execution succeeded but contained differences (error code = 1) after
> executing union_stats.q
> 362a363
> > COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> 364a366,367
> > numRows 1000
> > rawDataSize 10624
> {noformat}
> Every few runs
> bucketizedinputformat for MiniSparkOnYarn; OOMs occasionally, example
> https://issues.apache.org/jira/browse/HIVE-19596
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19609) pointless callstacks in the logs as usual
[ https://issues.apache.org/jira/browse/HIVE-19609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19609: Status: Patch Available (was: Open) > pointless callstacks in the logs as usual > - > > Key: HIVE-19609 > URL: https://issues.apache.org/jira/browse/HIVE-19609 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19609.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19609) pointless callstacks in the logs as usual
[ https://issues.apache.org/jira/browse/HIVE-19609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19609: Attachment: HIVE-19609.patch > pointless callstacks in the logs as usual > - > > Key: HIVE-19609 > URL: https://issues.apache.org/jira/browse/HIVE-19609 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19609.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19609) pointless callstacks in the logs as usual
[ https://issues.apache.org/jira/browse/HIVE-19609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-19609: --- > pointless callstacks in the logs as usual > - > > Key: HIVE-19609 > URL: https://issues.apache.org/jira/browse/HIVE-19609 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19609.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19609) pointless callstacks in the logs as usual
[ https://issues.apache.org/jira/browse/HIVE-19609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481163#comment-16481163 ] Sergey Shelukhin commented on HIVE-19609: - [~thejas] can you take a look? trivial patch > pointless callstacks in the logs as usual > - > > Key: HIVE-19609 > URL: https://issues.apache.org/jira/browse/HIVE-19609 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19609.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19589) NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481161#comment-16481161 ] Sergey Shelukhin commented on HIVE-19589: - Yeah, checked the code, it's just logged. I will clean it up, no need to log the callstack > NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19589 > URL: https://issues.apache.org/jira/browse/HIVE-19589 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19589.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19608) disable flaky tests 2
[
https://issues.apache.org/jira/browse/HIVE-19608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-19608:
Status: Patch Available (was: Open)
Should be ok to commit w/o QA since this just removes tests.
I'll file follow-ups to fix.
> disable flaky tests 2
> -
>
> Key: HIVE-19608
> URL: https://issues.apache.org/jira/browse/HIVE-19608
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19608.patch
>
>
> union_stats
> {noformat}
> java.lang.AssertionError:
> Client Execution succeeded but contained differences (error code = 1) after
> executing union_stats.q
> 362a363
> > COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> 364a366,367
> > numRows 1000
> > rawDataSize 10624
> {noformat}
> Every few runs
> bucketizedinputformat for MiniSparkOnYarn; OOMs occasionally, example
> https://issues.apache.org/jira/browse/HIVE-19596
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19589) NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481156#comment-16481156 ] Sergey Shelukhin commented on HIVE-19589: - I think no stats exception is a red herring, I see it in logs all over the place but it happens after real failure. I filed some jiras/patches for some other issues. Some tests need stats or lineage masking, 2 are flaky q files, many failures are due to tmp dir issues. > NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19589 > URL: https://issues.apache.org/jira/browse/HIVE-19589 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19589.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-19596) NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19596: Comment: was deleted (was: Actually I'd add it to another patch) > NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19596 > URL: https://issues.apache.org/jira/browse/HIVE-19596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19596.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19608) disable flaky tests 2
[
https://issues.apache.org/jira/browse/HIVE-19608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-19608:
Attachment: HIVE-19608.patch
> disable flaky tests 2
> -
>
> Key: HIVE-19608
> URL: https://issues.apache.org/jira/browse/HIVE-19608
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19608.patch
>
>
> union_stats
> {noformat}
> java.lang.AssertionError:
> Client Execution succeeded but contained differences (error code = 1) after
> executing union_stats.q
> 362a363
> > COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> 364a366,367
> > numRows 1000
> > rawDataSize 10624
> {noformat}
> Every few runs
> bucketizedinputformat for MiniSparkOnYarn; OOMs occasionally, example
> https://issues.apache.org/jira/browse/HIVE-19596
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19608) disable flaky tests 2
[
https://issues.apache.org/jira/browse/HIVE-19608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-19608:
Description:
union_stats
{noformat}
java.lang.AssertionError:
Client Execution succeeded but contained differences (error code = 1) after
executing union_stats.q
362a363
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
364a366,367
> numRows 1000
> rawDataSize 10624
{noformat}
Every few runs
bucketizedinputformat for MiniSparkOnYarn; OOMs occasionally, example
https://issues.apache.org/jira/browse/HIVE-19596
was:
union_stats
{noformat}
java.lang.AssertionError:
Client Execution succeeded but contained differences (error code = 1) after
executing union_stats.q
362a363
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
364a366,367
> numRows 1000
> rawDataSize 10624
{noformat}
Every few runs
> disable flaky tests 2
> -
>
> Key: HIVE-19608
> URL: https://issues.apache.org/jira/browse/HIVE-19608
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19608.patch
>
>
> union_stats
> {noformat}
> java.lang.AssertionError:
> Client Execution succeeded but contained differences (error code = 1) after
> executing union_stats.q
> 362a363
> > COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> 364a366,367
> > numRows 1000
> > rawDataSize 10624
> {noformat}
> Every few runs
> bucketizedinputformat for MiniSparkOnYarn; OOMs occasionally, example
> https://issues.apache.org/jira/browse/HIVE-19596
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19596) NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19596: Description: (was: Fails routinely due to some OOM) > NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19596 > URL: https://issues.apache.org/jira/browse/HIVE-19596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19596.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19608) disable flaky tests 2
[
https://issues.apache.org/jira/browse/HIVE-19608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-19608:
Attachment: (was: HIVE-19608.patch)
> disable flaky tests 2
> -
>
> Key: HIVE-19608
> URL: https://issues.apache.org/jira/browse/HIVE-19608
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19608.patch
>
>
> union_stats
> {noformat}
> java.lang.AssertionError:
> Client Execution succeeded but contained differences (error code = 1) after
> executing union_stats.q
> 362a363
> > COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> 364a366,367
> > numRows 1000
> > rawDataSize 10624
> {noformat}
> Every few runs
> bucketizedinputformat for MiniSparkOnYarn; OOMs occasionally, example
> https://issues.apache.org/jira/browse/HIVE-19596
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (HIVE-19596) NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reopened HIVE-19596: - > NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19596 > URL: https://issues.apache.org/jira/browse/HIVE-19596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19596.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19596) NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481153#comment-16481153 ] Sergey Shelukhin commented on HIVE-19596: - MiniMr is addressed elsehwhere, so is trigger test. bucketizedhiveinputformat is flaky and will be disabled (seen it before) > NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19596 > URL: https://issues.apache.org/jira/browse/HIVE-19596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19596.patch > > > Fails routinely due to some OOM -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19596) NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19596: Summary: NOOP jira to see which tests are flaky on HiveQA (was: disable bucketizedhiveinputformat for MiniSparkonYarn) > NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19596 > URL: https://issues.apache.org/jira/browse/HIVE-19596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19596.patch > > > Fails routinely due to some OOM -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19596) disable bucketizedhiveinputformat for MiniSparkonYarn
[ https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19596: Resolution: Duplicate Status: Resolved (was: Patch Available) Actually I'd add it to another patch > disable bucketizedhiveinputformat for MiniSparkonYarn > - > > Key: HIVE-19596 > URL: https://issues.apache.org/jira/browse/HIVE-19596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19596.patch > > > Fails routinely due to some OOM -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19596) disable bucketizedhiveinputformat for MiniSparkonYarn
[ https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19596: Description: Fails routinely due to some OOM > disable bucketizedhiveinputformat for MiniSparkonYarn > - > > Key: HIVE-19596 > URL: https://issues.apache.org/jira/browse/HIVE-19596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19596.patch > > > Fails routinely due to some OOM -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19596) disable bucketizedhiveinputformat for MiniSparkonYarn
[ https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19596: Summary: disable bucketizedhiveinputformat for MiniSparkonYarn (was: CLONE - CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA) > disable bucketizedhiveinputformat for MiniSparkonYarn > - > > Key: HIVE-19596 > URL: https://issues.apache.org/jira/browse/HIVE-19596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19596.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19463) TezTask - getting groups may fail (PartialGroupNameException in some tests)
[
https://issues.apache.org/jira/browse/HIVE-19463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-19463:
Attachment: HIVE-19463.05.patch
> TezTask - getting groups may fail (PartialGroupNameException in some tests)
> ---
>
> Key: HIVE-19463
> URL: https://issues.apache.org/jira/browse/HIVE-19463
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19463.01.patch, HIVE-19463.02.patch,
> HIVE-19463.03.patch, HIVE-19463.03.patch, HIVE-19463.04.patch,
> HIVE-19463.05.patch, HIVE-19463.patch
>
>
> {noformat}
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping$PartialGroupNameException:
> The user name 'hive_test_user' is not found. id: hive_test_user: no such user
> id: hive_test_user: no such user
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.resolvePartialGroupNames(ShellBasedUnixGroupsMapping.java:294)
> ~[hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:207)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:97)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:51)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.security.Groups$GroupCacheLoader.fetchGroupList(Groups.java:384)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:319)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:269)
> [hadoop-common-3.1.0.jar:?]
> at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3542)
> [guava-19.0.jar:?]
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2323)
> [guava-19.0.jar:?]
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2286)
> [guava-19.0.jar:?]
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
> [guava-19.0.jar:?]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3953)
> [guava-19.0.jar:?]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3957)
> [guava-19.0.jar:?]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4875)
> [guava-19.0.jar:?]
> at org.apache.hadoop.security.Groups.getGroups(Groups.java:227)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.security.UserGroupInformation.getGroups(UserGroupInformation.java:1540)
> [hadoop-common-3.1.0.jar:?]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:163)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19516) TestNegative merge_negative_5 and mm_concatenate are causing timeouts
[
https://issues.apache.org/jira/browse/HIVE-19516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481150#comment-16481150
]
Sergey Shelukhin commented on HIVE-19516:
-
[~jcamachorodriguez] should this be ok to commit? union_stats is going to be
disabled in a separate jira.
> TestNegative merge_negative_5 and mm_concatenate are causing timeouts
> -
>
> Key: HIVE-19516
> URL: https://issues.apache.org/jira/browse/HIVE-19516
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Vineet Garg
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19516.01.patch, HIVE-19516.02.patch,
> HIVE-19516.03.patch, HIVE-19516.patch
>
>
> I haven't tried to reproduce this in isolation but it is reproducible if you
> run in batch on local system
> {noformat}
> mvn -B test -Dtest.groups= -Dtest=TestNegativeCliDriver
> -Dqfile=udf_invalid.q,authorization_uri_export.q,default_constraint_complex_default_value.q,druid_datasource2.q,check_constraint_max_length.q,view_update.q,default_partition_name.q,authorization_public_create.q,load_wrong_fileformat_rc_seq.q,default_constraint_invalid_type.q,altern1.q,describe_xpath1.q,drop_view_failure2.q,temp_table_rename.q,invalid_select_column_with_subquery.q,udf_trunc_error1.q,insert_view_failure.q,dbtxnmgr_nodbunlock.q,authorization_show_columns.q,cte_recursion.q,merge_constraint_notnull.q,clusterbyorderby.q,orc_type_promotion2.q,ctas_noperm_loc.q,udf_min.q,udf_instr_wrong_args_len.q,invalid_create_tbl2.q,part_col_complex_type.q,authorization_drop_db_empty.q,smb_mapjoin_14.q,subquery_scalar_multi_rows.q,alter_partition_coltype_2columns.q,subquery_corr_in_agg.q,insert_overwrite_notnull_constraint.q,authorization_show_grant_otheruser_wtab.q,regex_col_groupby.q,ptf_negative_DuplicateWindowAlias.q,exim_22_export_authfail.q,authorization_insert_noinspriv.q,udf_likeany_wrong1.q,groupby_key.q,ambiguous_col.q,groupby3_multi_distinct.q,authorization_alter_drop_ptn.q,invalid_cast_from_binary_5.q,show_create_table_does_not_exist.q,invalid_select_column.q,exim_20_managed_location_over_existing.q,interval_3.q,authorization_compile.q,join35.q,udf_concat_ws_wrong3.q,create_or_replace_view8.q,create_external_with_notnull_constraint.q,split_sample_out_of_range.q,materialized_view_no_transactional_rewrite.q,authorization_show_grant_otherrole.q,create_with_constraints_duplicate_name.q,invalid_stddev_samp_syntax.q,authorization_view_disable_cbo_7.q,autolocal1.q,avro_non_nullable_union.q,load_orc_negative_part.q,drop_view_failure1.q,columnstats_partlvl_invalid_values_autogather.q,exim_13_nonnative_import.q,alter_table_wrong_regex.q,udf_next_day_error_2.q,authorization_select.q,udf_trunc_error2.q,authorization_view_7.q,udf_format_number_wrong5.q,touch2.q,orc_type_promotion1.q,lateral_view_alias.q,show_tables_bad_db1.q,unset_table_property.q,alter_non_native.q,nvl_mismatch_type.q,load_orc_negative3.q,authorization_create_role_no_admin.q,invalid_distinct1.q,authorization_grant_server.q,orc_type_promotion3_acid.q,hms_using_serde_alter_table_update_columns.q,show_tables_bad1.q,macro_unused_parameter.q,drop_invalid_constraint3.q,drop_partition_filter_failure.q,char_pad_convert_fail3.q,exim_23_import_exist_authfail.q,drop_invalid_constraint4.q,authorization_create_macro1.q,archive1.q,subquery_multiple_cols_in_select.q,change_hive_hdfs_session_path.q,udf_trunc_error3.q,invalid_variance_syntax.q,authorization_truncate_2.q,invalid_avg_syntax.q,invalid_select_column_with_tablename.q,mm_truncate_cols.q,groupby_grouping_sets1.q,druid_location.q,groupby2_multi_distinct.q,authorization_sba_drop_table.q,dynamic_partitions_with_whitelist.q,compare_string_bigint_2.q,udf_greatest_error_2.q,authorization_view_6.q,show_tablestatus.q,duplicate_alias_in_transform_schema.q,create_with_fk_uk_same_tab.q,udtf_not_supported3.q,alter_table_constraint_invalid_fk_col2.q,udtf_not_supported1.q,dbtxnmgr_notableunlock.q,ptf_negative_InvalidValueBoundary.q,alter_table_constraint_duplicate_pk.q,udf_printf_wrong4.q,create_view_failure9.q,udf_elt_wrong_type.q,selectDistinctStarNeg_1.q,invalid_mapjoin1.q,load_stored_as_dirs.q,input1.q,udf_sort_array_wrong1.q,invalid_distinct2.q,invalid_select_fn.q,authorization_role_grant_otherrole.q,archive4.q,load_nonpart_authfail.q,recursive_view.q,authorization_view_disable_cbo_1.q,desc_failure4.q,create_not_acid.q,udf_sort_array_wrong3.q,char_pad_convert_fail0.q,udf_map_values_arg_type.q,alter_view_failure6_2.q,alter_partition_change_col_nonexist.q,update_non_acid_table.q,authorization_view_disable_cbo_5.q,ct_noperm_loc.q,interval_1.q,authorization_show_grant_otheruser_all.q,authorization_view_2.q,show_tables_bad2.q,groupby_rollup2.q,truncate_column_seqfile.q,create_view_failure5.q,authorization_create_v
[jira] [Updated] (HIVE-19608) disable flaky tests 2
[
https://issues.apache.org/jira/browse/HIVE-19608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-19608:
Attachment: HIVE-19608.patch
> disable flaky tests 2
> -
>
> Key: HIVE-19608
> URL: https://issues.apache.org/jira/browse/HIVE-19608
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19608.patch
>
>
> union_stats
> {noformat}
> java.lang.AssertionError:
> Client Execution succeeded but contained differences (error code = 1) after
> executing union_stats.q
> 362a363
> > COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> 364a366,367
> > numRows 1000
> > rawDataSize 10624
> {noformat}
> Every few runs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19608) disable flaky tests 2
[
https://issues.apache.org/jira/browse/HIVE-19608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-19608:
Attachment: (was: HIVE-19608.patch)
> disable flaky tests 2
> -
>
> Key: HIVE-19608
> URL: https://issues.apache.org/jira/browse/HIVE-19608
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19608.patch
>
>
> union_stats
> {noformat}
> java.lang.AssertionError:
> Client Execution succeeded but contained differences (error code = 1) after
> executing union_stats.q
> 362a363
> > COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> 364a366,367
> > numRows 1000
> > rawDataSize 10624
> {noformat}
> Every few runs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19608) disable flaky tests 2
[
https://issues.apache.org/jira/browse/HIVE-19608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481146#comment-16481146
]
Sergey Shelukhin commented on HIVE-19608:
-
cc [~jcamachorodriguez]
I might disable more if I see any. I think this should be ok to commit without
HiveQA since it just removes tests.
> disable flaky tests 2
> -
>
> Key: HIVE-19608
> URL: https://issues.apache.org/jira/browse/HIVE-19608
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19608.patch
>
>
> union_stats
> {noformat}
> java.lang.AssertionError:
> Client Execution succeeded but contained differences (error code = 1) after
> executing union_stats.q
> 362a363
> > COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> 364a366,367
> > numRows 1000
> > rawDataSize 10624
> {noformat}
> Every few runs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19608) disable flaky tests 2
[
https://issues.apache.org/jira/browse/HIVE-19608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-19608:
Attachment: HIVE-19608.patch
> disable flaky tests 2
> -
>
> Key: HIVE-19608
> URL: https://issues.apache.org/jira/browse/HIVE-19608
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19608.patch
>
>
> union_stats
> {noformat}
> java.lang.AssertionError:
> Client Execution succeeded but contained differences (error code = 1) after
> executing union_stats.q
> 362a363
> > COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> 364a366,367
> > numRows 1000
> > rawDataSize 10624
> {noformat}
> Every few runs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19608) disable flaky tests 2
[
https://issues.apache.org/jira/browse/HIVE-19608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-19608:
---
> disable flaky tests 2
> -
>
> Key: HIVE-19608
> URL: https://issues.apache.org/jira/browse/HIVE-19608
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>
> union_stats
> {noformat}
> java.lang.AssertionError:
> Client Execution succeeded but contained differences (error code = 1) after
> executing union_stats.q
> 362a363
> > COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> 364a366,367
> > numRows 1000
> > rawDataSize 10624
> {noformat}
> Every few runs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19516) TestNegative merge_negative_5 and mm_concatenate are causing timeouts
[
https://issues.apache.org/jira/browse/HIVE-19516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481135#comment-16481135
]
Hive QA commented on HIVE-19516:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923995/HIVE-19516.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14408 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_stats]
(batchId=160)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11058/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11058/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11058/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923995 - PreCommit-HIVE-Build
> TestNegative merge_negative_5 and mm_concatenate are causing timeouts
> -
>
> Key: HIVE-19516
> URL: https://issues.apache.org/jira/browse/HIVE-19516
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Vineet Garg
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19516.01.patch, HIVE-19516.02.patch,
> HIVE-19516.03.patch, HIVE-19516.patch
>
>
> I haven't tried to reproduce this in isolation but it is reproducible if you
> run in batch on local system
> {noformat}
> mvn -B test -Dtest.groups= -Dtest=TestNegativeCliDriver
> -Dqfile=udf_invalid.q,authorization_uri_export.q,default_constraint_complex_default_value.q,druid_datasource2.q,check_constraint_max_length.q,view_update.q,default_partition_name.q,authorization_public_create.q,load_wrong_fileformat_rc_seq.q,default_constraint_invalid_type.q,altern1.q,describe_xpath1.q,drop_view_failure2.q,temp_table_rename.q,invalid_select_column_with_subquery.q,udf_trunc_error1.q,insert_view_failure.q,dbtxnmgr_nodbunlock.q,authorization_show_columns.q,cte_recursion.q,merge_constraint_notnull.q,clusterbyorderby.q,orc_type_promotion2.q,ctas_noperm_loc.q,udf_min.q,udf_instr_wrong_args_len.q,invalid_create_tbl2.q,part_col_complex_type.q,authorization_drop_db_empty.q,smb_mapjoin_14.q,subquery_scalar_multi_rows.q,alter_partition_coltype_2columns.q,subquery_corr_in_agg.q,insert_overwrite_notnull_constraint.q,authorization_show_grant_otheruser_wtab.q,regex_col_groupby.q,ptf_negative_DuplicateWindowAlias.q,exim_22_export_authfail.q,authorization_insert_noinspriv.q,udf_likeany_wrong1.q,groupby_key.q,ambiguous_col.q,groupby3_multi_distinct.q,authorization_alter_drop_ptn.q,invalid_cast_from_binary_5.q,show_create_table_does_not_exist.q,invalid_select_column.q,exim_20_managed_location_over_existing.q,interval_3.q,authorization_compile.q,join35.q,udf_concat_ws_wrong3.q,create_or_replace_view8.q,create_external_with_notnull_constraint.q,split_sample_out_of_range.q,materialized_view_no_transactional_rewrite.q,authorization_show_grant_otherrole.q,create_with_constraints_duplicate_name.q,invalid_stddev_samp_syntax.q,authorization_view_disable_cbo_7.q,autolocal1.q,avro_non_nullable_union.q,load_orc_negative_part.q,drop_view_failure1.q,columnstats_partlvl_invalid_values_autogather.q,exim_13_nonnative_import.q,alter_table_wrong_regex.q,udf_next_day_error_2.q,authorization_select.q,udf_trunc_error2.q,authorization_view_7.q,udf_format_number_wrong5.q,touch2.q,orc_type_promotion1.q,lateral_view_alias.q,show_tables_bad_db1.q,unset_table_property.q,alter_non_native.q,nvl_mismatch_type.q,load_orc_negative3.q,authorization_create_role_no_admin.q,invalid_distinct1.q,authorization_grant_server.q,orc_type_promotion3_acid.q,hms_using_serde_alter_table_update_columns.q,show_tables_bad1.q,macro_unused_parameter.q,drop_invalid_constraint3.q,drop_partition_filter_failure.q,char_pad_convert_fail3.q,exim_23_import_exist_authfail.q,drop_invalid_constraint4.q,authorization_create_macro1.q,archive1.q,subquery_multiple_cols_in_select.q,change_hive_hdfs_session_path.q,udf_trunc_error3.q,invalid_variance_syntax.q,authorization_truncate_2.q,invalid_avg_syntax.q,invalid_select_column_with_tablename.q,mm_truncate_cols.q,groupby_grouping_sets1.q,druid_location.q,groupby2_multi_distinct.q,authorization_sba_drop_table.q,dynamic_partitions_with_whitelist.q,compare_string_bigint_2.q,udf_greatest_error_2.q,authorization_view_6.q,show_tablestatus.q,duplicate_alias_in_trans
[jira] [Resolved] (HIVE-19590) mask stats in llap_smb
[ https://issues.apache.org/jira/browse/HIVE-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez resolved HIVE-19590. Resolution: Fixed Fix Version/s: 3.1.0 > mask stats in llap_smb > -- > > Key: HIVE-19590 > URL: https://issues.apache.org/jira/browse/HIVE-19590 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19590.01.patch, HIVE-19590.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18866) Semijoin: Implement a Long -> Hash64 vector fast-path
[ https://issues.apache.org/jira/browse/HIVE-18866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18866: Attachment: HIVE-18866.02.patch > Semijoin: Implement a Long -> Hash64 vector fast-path > - > > Key: HIVE-18866 > URL: https://issues.apache.org/jira/browse/HIVE-18866 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Reporter: Gopal V >Assignee: Sergey Shelukhin >Priority: Major > Labels: performance > Attachments: 0001-hash64-WIP.patch, HIVE-18866.01.patch, > HIVE-18866.02.patch, HIVE-18866.patch, perf-hash64-long.png > > > A significant amount of CPU is wasted with JMM restrictions on byte[] arrays. > To transform from one Long -> another Long, this goes into a byte[] array, > which shows up as a hotspot. > !perf-hash64-long.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19595) disable webui in MiniHS2
[ https://issues.apache.org/jira/browse/HIVE-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19595: Description: Causes tests flakiness if they same port is used. > disable webui in MiniHS2 > > > Key: HIVE-19595 > URL: https://issues.apache.org/jira/browse/HIVE-19595 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19595.01.patch, HIVE-19595.patch > > > Causes tests flakiness if they same port is used. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19595) disable webui in MiniHS2
[ https://issues.apache.org/jira/browse/HIVE-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481123#comment-16481123 ] Sergey Shelukhin commented on HIVE-19595: - [~thejas] [~jcamachorodriguez] does this make sense? > disable webui in MiniHS2 > > > Key: HIVE-19595 > URL: https://issues.apache.org/jira/browse/HIVE-19595 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19595.01.patch, HIVE-19595.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19595) disable webui in MiniHS2
[ https://issues.apache.org/jira/browse/HIVE-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19595: Attachment: HIVE-19595.01.patch > disable webui in MiniHS2 > > > Key: HIVE-19595 > URL: https://issues.apache.org/jira/browse/HIVE-19595 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19595.01.patch, HIVE-19595.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19595) disable webui in MiniHS2
[ https://issues.apache.org/jira/browse/HIVE-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19595: Status: Open (was: Patch Available) > disable webui in MiniHS2 > > > Key: HIVE-19595 > URL: https://issues.apache.org/jira/browse/HIVE-19595 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19595.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19595) CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481115#comment-16481115 ] Sergey Shelukhin commented on HIVE-19595: - One address in use due to WebUI... should be disabled in MiniHS2. TestTriggersTezSessionPoolManager has some metastore communication errors, no idea why. > CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are > flaky on HiveQA > > > Key: HIVE-19595 > URL: https://issues.apache.org/jira/browse/HIVE-19595 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19595.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19595) disable webui in MiniHS2
[ https://issues.apache.org/jira/browse/HIVE-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19595: Summary: disable webui in MiniHS2 (was: CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA) > disable webui in MiniHS2 > > > Key: HIVE-19595 > URL: https://issues.apache.org/jira/browse/HIVE-19595 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19595.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18866) Semijoin: Implement a Long -> Hash64 vector fast-path
[ https://issues.apache.org/jira/browse/HIVE-18866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481105#comment-16481105 ] Sergey Shelukhin commented on HIVE-18866: - HCat test: Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/tmp/temp1540619121/tmp-2080326801 union_stats looks flaky w.r.t. these two lines. [~jcamachorodriguez] should be disable it? > Semijoin: Implement a Long -> Hash64 vector fast-path > - > > Key: HIVE-18866 > URL: https://issues.apache.org/jira/browse/HIVE-18866 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Reporter: Gopal V >Assignee: Sergey Shelukhin >Priority: Major > Labels: performance > Attachments: 0001-hash64-WIP.patch, HIVE-18866.01.patch, > HIVE-18866.patch, perf-hash64-long.png > > > A significant amount of CPU is wasted with JMM restrictions on byte[] arrays. > To transform from one Long -> another Long, this goes into a byte[] array, > which shows up as a hotspot. > !perf-hash64-long.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18748) Rename table impacts the ACID behavior as table names are not updated in meta-tables.
[ https://issues.apache.org/jira/browse/HIVE-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481106#comment-16481106 ] Eugene Koifman commented on HIVE-18748: --- same patch again - looks like pom has been updated > Rename table impacts the ACID behavior as table names are not updated in > meta-tables. > - > > Key: HIVE-18748 > URL: https://issues.apache.org/jira/browse/HIVE-18748 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Eugene Koifman >Priority: Critical > Labels: ACID, DDL > Attachments: HIVE-18748.01-branch-3.patch, > HIVE-18748.02-branch-3.patch, HIVE-18748.02.patch, > HIVE-18748.03-branch-3.patch, HIVE-18748.03.patch, HIVE-18748.04.patch, > HIVE-18748.05.patch > > > ACID implementation uses metatables such as TXN_COMPONENTS, > COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPCTION_QUEUE etc to > manage ACID operations. > Per table write ID implementation (HIVE-18192) introduces couple of > metatables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID to manage write ids > allocated per table. > Now, when we rename any tables, it is necessary to update the corresponding > table names in these metatables as well. Otherwise, ACID table operations > won't work properly. > Since, this change is significant and have other side-effects, we propose to > disable rename tables on ACID tables until a fix is figured out. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18748) Rename table impacts the ACID behavior as table names are not updated in meta-tables.
[ https://issues.apache.org/jira/browse/HIVE-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18748: -- Attachment: HIVE-18748.03-branch-3.patch > Rename table impacts the ACID behavior as table names are not updated in > meta-tables. > - > > Key: HIVE-18748 > URL: https://issues.apache.org/jira/browse/HIVE-18748 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Eugene Koifman >Priority: Critical > Labels: ACID, DDL > Attachments: HIVE-18748.01-branch-3.patch, > HIVE-18748.02-branch-3.patch, HIVE-18748.02.patch, > HIVE-18748.03-branch-3.patch, HIVE-18748.03.patch, HIVE-18748.04.patch, > HIVE-18748.05.patch > > > ACID implementation uses metatables such as TXN_COMPONENTS, > COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPCTION_QUEUE etc to > manage ACID operations. > Per table write ID implementation (HIVE-18192) introduces couple of > metatables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID to manage write ids > allocated per table. > Now, when we rename any tables, it is necessary to update the corresponding > table names in these metatables as well. Otherwise, ACID table operations > won't work properly. > Since, this change is significant and have other side-effects, we propose to > disable rename tables on ACID tables until a fix is figured out. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19516) TestNegative merge_negative_5 and mm_concatenate are causing timeouts
[
https://issues.apache.org/jira/browse/HIVE-19516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481103#comment-16481103
]
Hive QA commented on HIVE-19516:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/util in master has 55 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
51s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11058/dev-support/hive-personality.sh
|
| git revision | master / 0420bde |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: itests/util ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11058/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TestNegative merge_negative_5 and mm_concatenate are causing timeouts
> -
>
> Key: HIVE-19516
> URL: https://issues.apache.org/jira/browse/HIVE-19516
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Vineet Garg
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19516.01.patch, HIVE-19516.02.patch,
> HIVE-19516.03.patch, HIVE-19516.patch
>
>
> I haven't tried to reproduce this in isolation but it is reproducible if you
> run in batch on local system
> {noformat}
> mvn -B test -Dtest.groups= -Dtest=TestNegativeCliDriver
> -Dqfile=udf_invalid.q,authorization_uri_export.q,default_constraint_complex_default_value.q,druid_datasource2.q,check_constraint_max_length.q,view_update.q,default_partition_name.q,authorization_public_create.q,load_wrong_fileformat_rc_seq.q,default_constraint_invalid_type.q,altern1.q,describe_xpath1.q,drop_view_failure2.q,temp_table_rename.q,invalid_select_column_with_subquery.q,udf_trunc_error1.q,insert_view_failure.q,dbtxnmgr_nodbunlock.q,authorization_show_columns.q,cte_recursi
[jira] [Updated] (HIVE-19586) Optimize Count(distinct X) pushdown based on the storage capabilities
[
https://issues.apache.org/jira/browse/HIVE-19586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-19586:
--
Attachment: HIVE-19586.3.patch
> Optimize Count(distinct X) pushdown based on the storage capabilities
> --
>
> Key: HIVE-19586
> URL: https://issues.apache.org/jira/browse/HIVE-19586
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration, Logical Optimizer
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19586.2.patch, HIVE-19586.3.patch, HIVE-19586.patch
>
>
> h1. Goal
> Provide a way to rewrite queries with combination of COUNT(Distinct) and
> Aggregates like SUM as a series of Group By.
> This can be useful to push down to Druid queries like
> {code}
> select count(DISTINCT interval_marker), count (distinct dim), sum(num_l)
> FROM druid_test_table GROUP BY `__time`, `zone` ;
> {code}
> In general this can be useful to be used in cases where storage handlers can
> not perform count (distinct column)
> h1. How to do it.
> Use the Calcite rule {code}
> org.apache.calcite.rel.rules.AggregateExpandDistinctAggregatesRule{code} that
> breaks down Count distinct to a single Group by with Grouping sets or
> multiple series of Group by that might be linked with Joins if multiple
> counts are present.
> FYI today Hive does have a similar rule {code}
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveExpandDistinctAggregatesRule{code},
> but it only provides a rewrite to Grouping sets based plan.
> I am planing to use the actual Calcite rule, [~ashutoshc] any concerns or
> caveats to be aware of?
> h2. Concerns/questions
> Need to have a way to switch between Grouping sets or Simple chained group by
> based on the plan cost. For instance for Druid based scan it makes always
> sense (at least today) to push down a series of Group by and stitch result
> sets in Hive later (as oppose to scan everything).
> But this might be not true for other storage handler that can handle Grouping
> sets it is better to push down the Grouping sets as one table scan.
> Am still unsure how i can lean on the cost optimizer to select the best plan,
> [~ashutoshc]/[~jcamachorodriguez] any inputs?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19590) mask stats in llap_smb
[ https://issues.apache.org/jira/browse/HIVE-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-19590: --- Assignee: Jesus Camacho Rodriguez (was: Sergey Shelukhin) > mask stats in llap_smb > -- > > Key: HIVE-19590 > URL: https://issues.apache.org/jira/browse/HIVE-19590 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-19590.01.patch, HIVE-19590.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19590) mask stats in llap_smb
[ https://issues.apache.org/jira/browse/HIVE-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19590: Summary: mask stats in llap_smb (was: CLONE - NOOP jira to see which tests are flaky on HiveQA) > mask stats in llap_smb > -- > > Key: HIVE-19590 > URL: https://issues.apache.org/jira/browse/HIVE-19590 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19590.01.patch, HIVE-19590.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19590) mask stats in llap_smb
[ https://issues.apache.org/jira/browse/HIVE-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481087#comment-16481087 ] Sergey Shelukhin commented on HIVE-19590: - +1. It cannot possibly make tests any more broken than they already are so maybe it should just be pushed :) > mask stats in llap_smb > -- > > Key: HIVE-19590 > URL: https://issues.apache.org/jira/browse/HIVE-19590 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-19590.01.patch, HIVE-19590.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19590) CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481085#comment-16481085 ] Jesus Camacho Rodriguez commented on HIVE-19590: [~sershe], uploaded a patch fixing the issue. > CLONE - NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19590 > URL: https://issues.apache.org/jira/browse/HIVE-19590 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19590.01.patch, HIVE-19590.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19590) CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-19590: --- Attachment: HIVE-19590.01.patch > CLONE - NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19590 > URL: https://issues.apache.org/jira/browse/HIVE-19590 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-19590.01.patch, HIVE-19590.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19590) CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-19590: --- Status: Open (was: Patch Available) > CLONE - NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19590 > URL: https://issues.apache.org/jira/browse/HIVE-19590 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19590.01.patch, HIVE-19590.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-19590) CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-19590 started by Jesus Camacho Rodriguez. -- > CLONE - NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19590 > URL: https://issues.apache.org/jira/browse/HIVE-19590 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-19590.01.patch, HIVE-19590.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19590) CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-19590: -- Assignee: Sergey Shelukhin (was: Jesus Camacho Rodriguez) > CLONE - NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19590 > URL: https://issues.apache.org/jira/browse/HIVE-19590 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19590.01.patch, HIVE-19590.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19590) CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-19590: -- Assignee: Jesus Camacho Rodriguez (was: Sergey Shelukhin) > CLONE - NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19590 > URL: https://issues.apache.org/jira/browse/HIVE-19590 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-19590.01.patch, HIVE-19590.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19604) Incorrect Handling of Boolean in DruidSerde
[ https://issues.apache.org/jira/browse/HIVE-19604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481076#comment-16481076 ] slim bouguerra commented on HIVE-19604: --- Can you please add a test case as part of MiniDruid where we index a Boolean column and query it back ? > Incorrect Handling of Boolean in DruidSerde > --- > > Key: HIVE-19604 > URL: https://issues.apache.org/jira/browse/HIVE-19604 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19604.patch > > > Results of boolean expressions from Druid are expressed in the form of > numeric 1 or 0. > When reading the results in DruidSerde both 1 and 0 are translated to String > and then we call Boolean.valueOf(stringForm), this leads to the boolean being > read always as false. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19589) NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481074#comment-16481074 ] Sergey Shelukhin commented on HIVE-19589: - Actually it's not even a change in lineage, it's a change in ordering of lines for lineage it looks like. Masking lineage makes sense to me. Should I file a JIRA? > NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19589 > URL: https://issues.apache.org/jira/browse/HIVE-19589 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19589.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19589) NOOP jira to see which tests are flaky on HiveQA
[
https://issues.apache.org/jira/browse/HIVE-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481071#comment-16481071
]
Jesus Camacho Rodriguez commented on HIVE-19589:
For {{mm_all}} I see some changes in LINEAGE. [~sershe], should we add option
to mask lineage too (as we can do with stats)?
> NOOP jira to see which tests are flaky on HiveQA
>
>
> Key: HIVE-19589
> URL: https://issues.apache.org/jira/browse/HIVE-19589
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19589.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19592) CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
[
https://issues.apache.org/jira/browse/HIVE-19592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481067#comment-16481067
]
Jesus Camacho Rodriguez commented on HIVE-19592:
Timeout of {{TestMinimrCliDriver}} likely solved by HIVE-19603.
> CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
>
>
> Key: HIVE-19592
> URL: https://issues.apache.org/jira/browse/HIVE-19592
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19592.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19592) CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
[
https://issues.apache.org/jira/browse/HIVE-19592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481069#comment-16481069
]
Sergey Shelukhin commented on HIVE-19592:
-
MiniMR timed out, there's JIRA for this.
The other test failed like others:
{noformat}
2018-05-18T08:05:47,583 WARN [Thread-2372] mapred.LocalJobRunner:
job_local801116008_0040
java.io.FileNotFoundException: File
file:/tmp/hadoop/mapred/staging/hiveptest801116008/.staging/job_local801116008_0040/job.splitmetainfo
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
~[hadoop-common-3.1.0.jar:?]
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
~[hadoop-common-3.1.0.jar:?]
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
~[hadoop-common-3.1.0.jar:?]
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
~[hadoop-common-3.1.0.jar:?]
at
org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:51)
~[hadoop-mapreduce-client-core-3.1.0.jar:?]
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:538)
[hadoop-mapreduce-client-common-3.1.0.jar:?
{noformat}
[~jcamachorodriguez] [~vihangk1] do you know if anything is cleaning tmp dir
during tests on ptest machines? We get a lot of random failures like this from
various tests.
> CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
>
>
> Key: HIVE-19592
> URL: https://issues.apache.org/jira/browse/HIVE-19592
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19592.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18866) Semijoin: Implement a Long -> Hash64 vector fast-path
[
https://issues.apache.org/jira/browse/HIVE-18866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481066#comment-16481066
]
Hive QA commented on HIVE-18866:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923993/HIVE-18866.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11057/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11057/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11057/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12923993/HIVE-18866.01.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923993 - PreCommit-HIVE-Build
> Semijoin: Implement a Long -> Hash64 vector fast-path
> -
>
> Key: HIVE-18866
> URL: https://issues.apache.org/jira/browse/HIVE-18866
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
>Priority: Major
> Labels: performance
> Attachments: 0001-hash64-WIP.patch, HIVE-18866.01.patch,
> HIVE-18866.patch, perf-hash64-long.png
>
>
> A significant amount of CPU is wasted with JMM restrictions on byte[] arrays.
> To transform from one Long -> another Long, this goes into a byte[] array,
> which shows up as a hotspot.
> !perf-hash64-long.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19589) NOOP jira to see which tests are flaky on HiveQA
[
https://issues.apache.org/jira/browse/HIVE-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481064#comment-16481064
]
Hive QA commented on HIVE-19589:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923988/HIVE-19589.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 14408 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_stats]
(batchId=160)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testExternalNoAutoPurge
(batchId=234)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testPartitionedNoAutoPurge
(batchId=234)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testTruncateUnsetAutoPurge
(batchId=234)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitions
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerVertexRawInputSplitsNoKill
(batchId=241)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11056/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11056/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11056/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923988 - PreCommit-HIVE-Build
> NOOP jira to see which tests are flaky on HiveQA
>
>
> Key: HIVE-19589
> URL: https://issues.apache.org/jira/browse/HIVE-19589
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19589.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19590) CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481060#comment-16481060 ] Sergey Shelukhin commented on HIVE-19590: - Will be handled by MASK_STATS > CLONE - NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19590 > URL: https://issues.apache.org/jira/browse/HIVE-19590 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19590.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19594) CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
[
https://issues.apache.org/jira/browse/HIVE-19594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481045#comment-16481045
]
Sergey Shelukhin commented on HIVE-19594:
-
{noformat}
2018-05-18T10:24:44,991 WARN [Thread-3915] mapred.LocalJobRunner:
job_local632888732_0106
java.io.FileNotFoundException: File
file:/tmp/hadoop/mapred/staging/hiveptest632888732/.staging/job_local632888732_0106/job.splitmetainfo
does not exist
{noformat}
I see lots of *HCatFileStorer tests fail like this. Looks like some ptest
issue.
Can this test perhaps be changed to not use shared tmp directory and instead
use the test directory?
Or we can just disable it and wait for a fix from people who care about HCat ;)
cc [~jcamachorodriguez] [~thejas]
> CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on
> HiveQA
>
>
> Key: HIVE-19594
> URL: https://issues.apache.org/jira/browse/HIVE-19594
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19594.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19589) NOOP jira to see which tests are flaky on HiveQA
[
https://issues.apache.org/jira/browse/HIVE-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481021#comment-16481021
]
Jesus Camacho Rodriguez commented on HIVE-19589:
An initial list of tests that I have detected flaky for these runs and other
runs:
{noformat}
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag[4]
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteDecimal
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.org.apache.hive.jdbc.TestTriggersTezSessionPoolManager
org.apache.hive.jdbc.TestTriggersWorkloadManager.org.apache.hive.jdbc.TestTriggersWorkloadManager
org.apache.hadoop.hive.ql.TestTxnCommands.testQuotedIdentifier2
org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testSnapshotIsolationWithAbortedTxnOnMmTable
org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testMergeOnTezEdges
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testToAcidConversion02
org.apache.hadoop.hive.ql.plan.mapping.TestReOptimization.testStatCachingMetaStore
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testAutoPurgeUnset
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testPartitionedNoAutoPurge
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testTruncateUnsetAutoPurge
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[multi_insert_move_tasks_share_dependencies]
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_smb]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_stats]
{noformat}
A few notes:
- For {{union_stats}}, there is work going on in HIVE-19326. We can disable it
till it gets in.
- For {{TestReoptimization}}, HIVE-19574. We can also disable that one till
that gets fixed.
- {{TestAutoPurgeTables}} has been failing quite a lot, we added the retry
annotation, but it does not seem to help. Candidate to disable for the time
being?
- Same for triggers tests, usually with following error:
{noformat}
Error Message
java.net.BindException: Address already in use
{noformat}
- For some tests, mocking objects could fix a great deal. For instance,
{{testQuotedIdentifier2}} failed with the following stacktrace:
{code}
org.apache.hadoop.hive.metastore.api.NoSuchObjectException: Column stats
doesn't exist for db=default temp table=acidTbl2
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.deleteTempTableColumnStatsForTable(SessionHiveMetaStoreClient.java:795)
~[classes/:?]
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.drop_table_with_environment_context(SessionHiveMetaStoreClient.java:135)
[classes/:?]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1268)
[hive-standalone-metastore-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1222)
[hive-standalone-metastore-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1203)
[hive-standalone-metastore-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_102]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
[hive-standalone-metastore-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at com.sun.proxy.$Proxy39.dropTable(Unknown Source) [?:?]
at org.apache.hadoop.hive.ql.metadata.Hive.dropTable(Hive.java:1033)
[classes/:?]
at org.apache.hadoop.hive.ql.metadata.Hive.dropTable(Hive.java:970)
[classes/:?]
at org.apache.hadoop.hive.ql.exec.DDLTask.dropTable(DDLTask.java:4736)
[classes/:?]
at
org.apache.hadoop.hive.ql.exec.DDLTask.dropTableOrPartitions(DDLTask.java:4590)
[classes/:?]
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:438)
[classes/:?]
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
[classes/:?]
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
[classes/:?]
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2482)
[classes/:?]
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2149)
[classes/:?]
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1826)
[classes/:?]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1569) [classes/:?]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1558) [classes/:?]
[jira] [Commented] (HIVE-19588) Several invocation of file listing when creating VectorizedOrcAcidRowBatchReader
[ https://issues.apache.org/jira/browse/HIVE-19588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481017#comment-16481017 ] Eugene Koifman commented on HIVE-19588: --- +1 > Several invocation of file listing when creating > VectorizedOrcAcidRowBatchReader > > > Key: HIVE-19588 > URL: https://issues.apache.org/jira/browse/HIVE-19588 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0 >Reporter: Nita Dembla >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19588.1.patch, HIVE-19588.2.patch, Screen Shot > 2018-05-16 at 2.23.25 PM.png > > > Looks like we are doing file listing several times when creating one instance > of VectorizedOrcAcidRowBatchReader > AcidUtils.parseBaseOrDeltaBucketFilename() does full file listing (when > there are files with bucket_* prefix) just to get a single file out of a path > to figure out if it has ACID schema (as part of HIVE-18190). > There is full file listing where we populate > 1) ColumnizedDeleteEventRegistry > 2) SortMergedDeleteEventRegistry > 3) Twice in computeOffsetAndBucket() > > Attaching profiles which [~gopalv] took while debugging. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19589) NOOP jira to see which tests are flaky on HiveQA
[
https://issues.apache.org/jira/browse/HIVE-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16480999#comment-16480999
]
Hive QA commented on HIVE-19589:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11056/dev-support/hive-personality.sh
|
| git revision | master / 0420bde |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11056/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> NOOP jira to see which tests are flaky on HiveQA
>
>
> Key: HIVE-19589
> URL: https://issues.apache.org/jira/browse/HIVE-19589
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19589.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19586) Optimize Count(distinct X) pushdown based on the storage capabilities
[
https://issues.apache.org/jira/browse/HIVE-19586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-19586:
Target Version/s: 3.1.0
Fix Version/s: (was: 3.0.0)
> Optimize Count(distinct X) pushdown based on the storage capabilities
> --
>
> Key: HIVE-19586
> URL: https://issues.apache.org/jira/browse/HIVE-19586
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration, Logical Optimizer
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19586.2.patch, HIVE-19586.patch
>
>
> h1. Goal
> Provide a way to rewrite queries with combination of COUNT(Distinct) and
> Aggregates like SUM as a series of Group By.
> This can be useful to push down to Druid queries like
> {code}
> select count(DISTINCT interval_marker), count (distinct dim), sum(num_l)
> FROM druid_test_table GROUP BY `__time`, `zone` ;
> {code}
> In general this can be useful to be used in cases where storage handlers can
> not perform count (distinct column)
> h1. How to do it.
> Use the Calcite rule {code}
> org.apache.calcite.rel.rules.AggregateExpandDistinctAggregatesRule{code} that
> breaks down Count distinct to a single Group by with Grouping sets or
> multiple series of Group by that might be linked with Joins if multiple
> counts are present.
> FYI today Hive does have a similar rule {code}
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveExpandDistinctAggregatesRule{code},
> but it only provides a rewrite to Grouping sets based plan.
> I am planing to use the actual Calcite rule, [~ashutoshc] any concerns or
> caveats to be aware of?
> h2. Concerns/questions
> Need to have a way to switch between Grouping sets or Simple chained group by
> based on the plan cost. For instance for Druid based scan it makes always
> sense (at least today) to push down a series of Group by and stitch result
> sets in Hive later (as oppose to scan everything).
> But this might be not true for other storage handler that can handle Grouping
> sets it is better to push down the Grouping sets as one table scan.
> Am still unsure how i can lean on the cost optimizer to select the best plan,
> [~ashutoshc]/[~jcamachorodriguez] any inputs?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19604) Incorrect Handling of Boolean in DruidSerde
[ https://issues.apache.org/jira/browse/HIVE-19604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-19604: Fix Version/s: 3.0.0 Status: Patch Available (was: Open) > Incorrect Handling of Boolean in DruidSerde > --- > > Key: HIVE-19604 > URL: https://issues.apache.org/jira/browse/HIVE-19604 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19604.patch > > > Results of boolean expressions from Druid are expressed in the form of > numeric 1 or 0. > When reading the results in DruidSerde both 1 and 0 are translated to String > and then we call Boolean.valueOf(stringForm), this leads to the boolean being > read always as false. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19604) Incorrect Handling of Boolean in DruidSerde
[ https://issues.apache.org/jira/browse/HIVE-19604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16480989#comment-16480989 ] Nishant Bangarwa commented on HIVE-19604: - +cc [~ashutoshc] please review. > Incorrect Handling of Boolean in DruidSerde > --- > > Key: HIVE-19604 > URL: https://issues.apache.org/jira/browse/HIVE-19604 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-19604.patch > > > Results of boolean expressions from Druid are expressed in the form of > numeric 1 or 0. > When reading the results in DruidSerde both 1 and 0 are translated to String > and then we call Boolean.valueOf(stringForm), this leads to the boolean being > read always as false. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19604) Incorrect Handling of Boolean in DruidSerde
[ https://issues.apache.org/jira/browse/HIVE-19604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-19604: Attachment: HIVE-19604.patch > Incorrect Handling of Boolean in DruidSerde > --- > > Key: HIVE-19604 > URL: https://issues.apache.org/jira/browse/HIVE-19604 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-19604.patch > > > Results of boolean expressions from Druid are expressed in the form of > numeric 1 or 0. > When reading the results in DruidSerde both 1 and 0 are translated to String > and then we call Boolean.valueOf(stringForm), this leads to the boolean being > read always as false. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19604) Incorrect Handling of Boolean in DruidSerde
[ https://issues.apache.org/jira/browse/HIVE-19604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa reassigned HIVE-19604: --- > Incorrect Handling of Boolean in DruidSerde > --- > > Key: HIVE-19604 > URL: https://issues.apache.org/jira/browse/HIVE-19604 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > > Results of boolean expressions from Druid are expressed in the form of > numeric 1 or 0. > When reading the results in DruidSerde both 1 and 0 are translated to String > and then we call Boolean.valueOf(stringForm), this leads to the boolean being > read always as false. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19594) CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
[
https://issues.apache.org/jira/browse/HIVE-19594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16480982#comment-16480982
]
Hive QA commented on HIVE-19594:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923984/HIVE-19594.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14408 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteDecimal
(batchId=197)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11055/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11055/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11055/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923984 - PreCommit-HIVE-Build
> CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on
> HiveQA
>
>
> Key: HIVE-19594
> URL: https://issues.apache.org/jira/browse/HIVE-19594
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19594.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19508) SparkJobMonitor getReport doesn't print stage progress in order
[
https://issues.apache.org/jira/browse/HIVE-19508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali reassigned HIVE-19508:
---
Assignee: Bharathkrishna Guruvayoor Murali
> SparkJobMonitor getReport doesn't print stage progress in order
> ---
>
> Key: HIVE-19508
> URL: https://issues.apache.org/jira/browse/HIVE-19508
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
>
> You can end up with a progress output like this:
> {code}
> Stage-10_0: 0/29 Stage-11_0: 0/44Stage-12_0: 0/11
> Stage-13_0: 0/1 Stage-8_0: 258(+76)/468 Stage-9_0: 0/165
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19559) SparkClientImpl shouldn't name redirector thread "RemoteDriver"
[
https://issues.apache.org/jira/browse/HIVE-19559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali reassigned HIVE-19559:
---
Assignee: Bharathkrishna Guruvayoor Murali
> SparkClientImpl shouldn't name redirector thread "RemoteDriver"
> ---
>
> Key: HIVE-19559
> URL: https://issues.apache.org/jira/browse/HIVE-19559
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
>
> The thread responsible for re-directing the stdout / stderr of the
> spark-submit process are named {{RemoteDriver...}} which is misleading
> because these threads are not re-directing output from the {{RemoteDriver}},
> just from the spark-submit stdout / stderr.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19440) Make StorageBasedAuthorizer work with information schema
[ https://issues.apache.org/jira/browse/HIVE-19440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-19440: -- Attachment: HIVE-19440.6.patch > Make StorageBasedAuthorizer work with information schema > > > Key: HIVE-19440 > URL: https://issues.apache.org/jira/browse/HIVE-19440 > Project: Hive > Issue Type: Improvement >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Blocker > Fix For: 3.1.0 > > Attachments: HIVE-19440.1.patch, HIVE-19440.2.patch, > HIVE-19440.3.patch, HIVE-19440.4.patch, HIVE-19440.5.patch, HIVE-19440.6.patch > > > With HIVE-19161, Hive information schema works with external authorizer (such > as ranger). However, we also need to make StorageBasedAuthorizer > synchronization work as it is also widely use. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
