[jira] [Commented] (HIVE-19500) Prevent multiple selectivity estimations for the same variable in conjuctions
[
https://issues.apache.org/jira/browse/HIVE-19500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16476948#comment-16476948
]
Hive QA commented on HIVE-19500:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
53s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10982/dev-support/hive-personality.sh
|
| git revision | master / 38c757c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10982/yetus/whitespace-tabs.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10982/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Prevent multiple selectivity estimations for the same variable in conjuctions
> -
>
> Key: HIVE-19500
> URL: https://issues.apache.org/jira/browse/HIVE-19500
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19500.01.patch, HIVE-19500.02.patch
>
>
> see HIVE-19097 for problem description
> for filters like: {{(d_year in (2001,2002) and d_year = 2001)}} the current
> estimation is around {{(1/NDV)**2}} (iff column stats are available)
> actually the source of the problem was a small typo in HIVE-17465
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Status: Patch Available (was: In Progress) > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Attachment: HIVE-19308.2.patch > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[ https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-18079: Assignee: Gopal V (was: Prasanth Jayachandran) > Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator > bit-size > > > Key: HIVE-18079 > URL: https://issues.apache.org/jira/browse/HIVE-18079 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Statistics >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-18079.1.patch, HIVE-18079.10.patch, > HIVE-18079.11.patch, HIVE-18079.12.patch, HIVE-18079.13.patch, > HIVE-18079.14.patch, HIVE-18079.2.patch, HIVE-18079.4.patch, > HIVE-18079.5.patch, HIVE-18079.6.patch, HIVE-18079.7.patch, > HIVE-18079.8.patch, HIVE-18079.9.patch > > > HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its > mathematical hash distribution & construction. > Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second > scan over the data-set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[ https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-18079: Assignee: Prasanth Jayachandran (was: Gopal V) > Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator > bit-size > > > Key: HIVE-18079 > URL: https://issues.apache.org/jira/browse/HIVE-18079 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Statistics >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18079.1.patch, HIVE-18079.10.patch, > HIVE-18079.11.patch, HIVE-18079.12.patch, HIVE-18079.13.patch, > HIVE-18079.14.patch, HIVE-18079.2.patch, HIVE-18079.4.patch, > HIVE-18079.5.patch, HIVE-18079.6.patch, HIVE-18079.7.patch, > HIVE-18079.8.patch, HIVE-18079.9.patch > > > HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its > mathematical hash distribution & construction. > Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second > scan over the data-set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[ https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18079: - Attachment: HIVE-18079.14.patch > Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator > bit-size > > > Key: HIVE-18079 > URL: https://issues.apache.org/jira/browse/HIVE-18079 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Statistics >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18079.1.patch, HIVE-18079.10.patch, > HIVE-18079.11.patch, HIVE-18079.12.patch, HIVE-18079.13.patch, > HIVE-18079.14.patch, HIVE-18079.2.patch, HIVE-18079.4.patch, > HIVE-18079.5.patch, HIVE-18079.6.patch, HIVE-18079.7.patch, > HIVE-18079.8.patch, HIVE-18079.9.patch > > > HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its > mathematical hash distribution & construction. > Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second > scan over the data-set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[ https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16476962#comment-16476962 ] Prasanth Jayachandran commented on HIVE-18079: -- tez_vector_dynpart_hashjoin_1.q diff is handled in HIVE-19555. Removed from this patch. > Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator > bit-size > > > Key: HIVE-18079 > URL: https://issues.apache.org/jira/browse/HIVE-18079 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Statistics >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-18079.1.patch, HIVE-18079.10.patch, > HIVE-18079.11.patch, HIVE-18079.12.patch, HIVE-18079.13.patch, > HIVE-18079.14.patch, HIVE-18079.2.patch, HIVE-18079.4.patch, > HIVE-18079.5.patch, HIVE-18079.6.patch, HIVE-18079.7.patch, > HIVE-18079.8.patch, HIVE-18079.9.patch > > > HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its > mathematical hash distribution & construction. > Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second > scan over the data-set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-12342) Set default value of hive.optimize.index.filter to true
[ https://issues.apache.org/jira/browse/HIVE-12342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Kryvenko updated HIVE-12342: - Attachment: HIVE-12342.13.patch > Set default value of hive.optimize.index.filter to true > --- > > Key: HIVE-12342 > URL: https://issues.apache.org/jira/browse/HIVE-12342 > Project: Hive > Issue Type: Task > Components: Configuration >Reporter: Ashutosh Chauhan >Assignee: Igor Kryvenko >Priority: Major > Attachments: HIVE-12342.05.patch, HIVE-12342.06.patch, > HIVE-12342.07.patch, HIVE-12342.08.patch, HIVE-12342.09.patch, > HIVE-12342.1.patch, HIVE-12342.10.patch, HIVE-12342.11.patch, > HIVE-12342.12.patch, HIVE-12342.13.patch, HIVE-12342.2.patch, > HIVE-12342.3.patch, HIVE-12342.4.patch, HIVE-12342.patch > > > This configuration governs ppd for storage layer. When applicable, it will > always help. It should be on by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19500) Prevent multiple selectivity estimations for the same variable in conjuctions
[
https://issues.apache.org/jira/browse/HIVE-19500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16476982#comment-16476982
]
Hive QA commented on HIVE-19500:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923508/HIVE-19500.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 14406 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_multi]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_stats]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_cast]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_mapjoin_reduce]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_short_regress]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_nested_mapjoin]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=183)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
(batchId=240)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitions
(batchId=240)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
(batchId=240)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=240)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
(batchId=240)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=240)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomReadOps
(batchId=240)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=240)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=240)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerVertexRawInputSplitsNoKill
(batchId=240)
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob (batchId=304)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10982/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10982/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10982/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923508 - PreCommit-HIVE-Build
> Prevent multiple selectivity estimations for the same variable in conjuctions
> -
>
> Key: HIVE-19500
> URL: https://issues.apache.org/jira/browse/HIVE-19500
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19500.01.patch, HIVE-19500.02.patch
>
>
> see HIVE-19097 for problem description
> for filters like: {{(d_year in (2001,2002) and d_year = 2001)}} the current
> estimation is around {{(1/NDV)**2}} (iff column stats are available)
> actually the source of the problem was a small typo in HIVE-17465
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19557) stats: filters for dates are not taking advantage of min/max values
[
https://issues.apache.org/jira/browse/HIVE-19557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16476983#comment-16476983
]
Hive QA commented on HIVE-19557:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923503/HIVE-19557.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10983/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10983/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10983/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-05-16 07:43:50.688
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-10983/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-05-16 07:43:50.691
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 38c757c HIVE-19495: Arrow SerDe itest failure (Teddy Choi,
reviewed by Matt McCline)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 38c757c HIVE-19495: Arrow SerDe itest failure (Teddy Choi,
reviewed by Matt McCline)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-05-16 07:43:52.006
+ rm -rf ../yetus_PreCommit-HIVE-Build-10983
+ mkdir ../yetus_PreCommit-HIVE-Build-10983
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-10983
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-10983/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java:106
error: repository lacks the necessary blob to fall back on 3-way merge.
error:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java:
patch does not apply
error:
src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java:
does not exist in index
error:
java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java:
does not exist in index
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923503 - PreCommit-HIVE-Build
> stats: filters for dates are not taking advantage of min/max values
> ---
>
> Key: HIVE-19557
> URL: https://issues.apache.org/jira/browse/HIVE-19557
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19557.01.patch
>
>
> in StatsRulesProcFactory
> [https://github.com/apache/hive/blob/ab189f54047bbf6beeeaf8d0dcfd5fbe92e465fb/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java#L754|dates
> are assumed to be an integer]; however this is currently not true - and the
> resulting exception is handled as a default case... for N/3
> {code}
> set hive.explain.user=true;
> create table d1(d date);
> -- tblproperties('transactional'='false');
> insert into d1 values
> ('2010-10-01'),
> ('2010-10-02'),
> ('2010-10-03'),
> ('2010-10-04'),
> ('2010-10-05'),
> ('2010-10-06'),
> ('2010-10-07'),
> ('2010-10-08'),
> ('2010-10-09'),
> ('2010-10-10');
> analyze table d1 compute statistics for columns;
> desc format
[jira] [Updated] (HIVE-19557) stats: filters for dates are not taking advantage of min/max values
[
https://issues.apache.org/jira/browse/HIVE-19557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-19557:
Attachment: HIVE-19557.02.patch
> stats: filters for dates are not taking advantage of min/max values
> ---
>
> Key: HIVE-19557
> URL: https://issues.apache.org/jira/browse/HIVE-19557
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19557.01.patch, HIVE-19557.02.patch
>
>
> in StatsRulesProcFactory
> [https://github.com/apache/hive/blob/ab189f54047bbf6beeeaf8d0dcfd5fbe92e465fb/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java#L754|dates
> are assumed to be an integer]; however this is currently not true - and the
> resulting exception is handled as a default case... for N/3
> {code}
> set hive.explain.user=true;
> create table d1(d date);
> -- tblproperties('transactional'='false');
> insert into d1 values
> ('2010-10-01'),
> ('2010-10-02'),
> ('2010-10-03'),
> ('2010-10-04'),
> ('2010-10-05'),
> ('2010-10-06'),
> ('2010-10-07'),
> ('2010-10-08'),
> ('2010-10-09'),
> ('2010-10-10');
> analyze table d1 compute statistics for columns;
> desc formatted d1;
> desc formatted d1 d;
> explain
> select 'stats: FIL ~0 read',count(1) from d1 where d < '2010-03-01';
> explain
> select 'stats: FIL estimate some read',count(1) from d1 where d <
> '2010-10-03';
> explain
> select 'stats: FIL estimate all read',count(1) from d1 where d < '2010-11-03';
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19326) stats auto gather: incorrect aggregation during UNION queries (may lead to incorrect results)
[
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477018#comment-16477018
]
Hive QA commented on HIVE-19326:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/util in master has 55 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
48s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch util passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} ql: The patch generated 0 new + 142 unchanged - 16
fixed = 142 total (was 158) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 6 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10984/dev-support/hive-personality.sh
|
| git revision | master / 38c757c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10984/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10984/yetus/whitespace-tabs.txt
|
| modules | C: itests itests/util ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10984/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> stats auto gather: incorrect aggregation during UNION queries (may lead to
> incorrect results)
> -
>
> Key: HIVE-19326
> URL: https://issues.apache.org/jira/browse/HIVE-19326
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Sergey Shelukhin
>Assignee: Zoltan Haindrich
>Priority: Critical
> Attachments: HIVE-19326.01wip01.patch, HIVE-19326.02.patch,
> HIVE-19326.03.patch,
[jira] [Commented] (HIVE-18866) Semijoin: Implement a Long -> Hash64 vector fast-path
[ https://issues.apache.org/jira/browse/HIVE-18866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477098#comment-16477098 ] Laszlo Bodor commented on HIVE-18866: - [~gopalv] : would you mind if I assign this task to me? I was working on HIVE-19501, but it becomes unnecessary with this patch. > Semijoin: Implement a Long -> Hash64 vector fast-path > - > > Key: HIVE-18866 > URL: https://issues.apache.org/jira/browse/HIVE-18866 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Reporter: Gopal V >Priority: Major > Labels: performance > Attachments: 0001-hash64-WIP.patch, perf-hash64-long.png > > > A significant amount of CPU is wasted with JMM restrictions on byte[] arrays. > To transform from one Long -> another Long, this goes into a byte[] array, > which shows up as a hotspot. > !perf-hash64-long.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19326) stats auto gather: incorrect aggregation during UNION queries (may lead to incorrect results)
[
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477106#comment-16477106
]
Hive QA commented on HIVE-19326:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923516/HIVE-19326.05.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 14400 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=245)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=245)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_stats] (batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[varchar_2] (batchId=68)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_single_sourced_multi_insert]
(batchId=103)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_4]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=167)
org.apache.hadoop.hive.metastore.client.TestRuntimeStats.testCleanup[Embedded]
(batchId=209)
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery
(batchId=243)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10984/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10984/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10984/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923516 - PreCommit-HIVE-Build
> stats auto gather: incorrect aggregation during UNION queries (may lead to
> incorrect results)
> -
>
> Key: HIVE-19326
> URL: https://issues.apache.org/jira/browse/HIVE-19326
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Sergey Shelukhin
>Assignee: Zoltan Haindrich
>Priority: Critical
> Attachments: HIVE-19326.01wip01.patch, HIVE-19326.02.patch,
> HIVE-19326.03.patch, HIVE-19326.04.patch, HIVE-19326.05.patch
>
>
> Found when investigating the results change after converting tables to MM,
> turns out the MM result is correct but the current one is not.
> The test ends like so:
> {noformat}
> desc formatted small_alltypesorc_a;
> ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
> desc formatted small_alltypesorc_a;
> insert into table small_alltypesorc_a select * from small_alltypesorc1a;
> desc formatted small_alltypesorc_a;
> {noformat}
> The results from the descs in the golden file are:
> {noformat}
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 5
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 15
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles2
> numRows 20
> {noformat}
> Note the result change after analyze - the original nomRows is inaccurate,
> but BASIC_STATS is set to true.
> I am assuming with metadata only optimization this can produce incorrect
> results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19562) Flaky test: TestMiniSparkOnYarn FileNotFoundException in spark-submit
[
https://issues.apache.org/jira/browse/HIVE-19562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477150#comment-16477150
]
Peter Vary commented on HIVE-19562:
---
+1 pending tests
> Flaky test: TestMiniSparkOnYarn FileNotFoundException in spark-submit
> -
>
> Key: HIVE-19562
> URL: https://issues.apache.org/jira/browse/HIVE-19562
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19562.1.patch
>
>
> Seeing sporadic failures during test setup. Specifically, when spark-submit
> runs this error (or a similar error) gets thrown:
> {code}
> 2018-05-15T10:55:02,112 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: Exception in thread "main"
> java.io.FileNotFoundException: File
> file:/tmp/spark-56e217f7-b8a5-4c63-9a6b-d737a64f2820/__spark_libs__7371510645900072447.zip
> does not exist
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:867)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:365)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:316)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:356)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.yarn.Client.org$apache$spark$deploy$yarn$Client$$distribute$1(Client.scala:478)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:565)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:863)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:169)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.yarn.Client.run(Client.scala:1146)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1518)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient:
[jira] [Commented] (HIVE-19460) Improve stats estimations for NOT IN operator
[
https://issues.apache.org/jira/browse/HIVE-19460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477163#comment-16477163
]
Hive QA commented on HIVE-19460:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
53s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 20 new + 107 unchanged - 0
fixed = 127 total (was 107) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
4s{color} | {color:red} ql generated 11 new + 2320 unchanged - 0 fixed = 2331
total (was 2320) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Found reliance on default encoding in
org.apache.hadoop.hive.ql.optimizer.stats.annotation.HiveMurmur3Adapter.murmur3(Object):in
org.apache.hadoop.hive.ql.optimizer.stats.annotation.HiveMurmur3Adapter.murmur3(Object):
String.getBytes() At HiveMurmur3Adapter.java:[line 68] |
| | Redundant nullcheck of colRange, which is known to be non-null in
org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$FilterStatsRule.pruneImprobableValues(ColStatistics,
Set) Redundant null check at StatsRulesProcFactory.java:is known to be
non-null in
org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$FilterStatsRule.pruneImprobableValues(ColStatistics,
Set) Redundant null check at StatsRulesProcFactory.java:[line 688] |
| | Boxing/unboxing to parse a primitive
org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$FilterStatsRule$RangeOps.intersect(ExprNodeDesc)
At
StatsRulesProcFactory.java:org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$FilterStatsRule$RangeOps.intersect(ExprNodeDesc)
At StatsRulesProcFactory.java:[line 639] |
| | Boxing/unboxing to parse a primitive
org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$FilterStatsRule$RangeOps.intersect(ExprNodeDesc)
At
StatsRulesProcFactory.java:org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$FilterStatsRule$RangeOps.intersect(ExprNodeDesc)
At StatsRulesProcFactory.java:[line 645] |
| |
org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$FilterStatsRule$RangeOps.intersect(ExprNodeDesc)
invokes inefficient new Byte(String) constructor; use Byte.valueOf(String)
instead At StatsRulesProcFactory.java:constructor; use Byt
[jira] [Commented] (HIVE-19460) Improve stats estimations for NOT IN operator
[
https://issues.apache.org/jira/browse/HIVE-19460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477204#comment-16477204
]
Hive QA commented on HIVE-19460:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923515/HIVE-19460.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 14407 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[explainuser_2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multi_column_in]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multi_column_in_single]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_predicate_pushdown]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[parquet_predicate_pushdown]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_between_in]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_struct_in]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_0]
(batchId=175)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testAutoPurgeUnset (batchId=233)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testNoAutoPurge (batchId=233)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testPartitionedNoAutoPurge
(batchId=233)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10985/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10985/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10985/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923515 - PreCommit-HIVE-Build
> Improve stats estimations for NOT IN operator
> -
>
> Key: HIVE-19460
> URL: https://issues.apache.org/jira/browse/HIVE-19460
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19460.01wip01.patch, HIVE-19460.01wip02.patch,
> HIVE-19460.01wip03.patch, HIVE-19460.02.patch, HIVE-19460.03.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[
https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477225#comment-16477225
]
Hive QA commented on HIVE-18079:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
52s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
54s{color} | {color:blue} standalone-metastore in master has 215 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} standalone-metastore: The patch generated 12 new + 38
unchanged - 1 fixed = 50 total (was 39) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10986/dev-support/hive-personality.sh
|
| git revision | master / 38c757c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10986/yetus/diff-checkstyle-standalone-metastore.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10986/yetus/whitespace-eol.txt
|
| modules | C: ql standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10986/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator
> bit-size
>
>
> Key: HIVE-18079
> URL: https://issues.apache.org/jira/browse/HIVE-18079
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore, Statistics
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-18079.1.patch, HIVE-18079.10.patch,
> HIVE-18079.11.patch, HIVE-18079.12.patch, HIVE-18079.13.patch,
> HIVE-18079.14.patch, HIVE-18079.2.patch, HIVE-18079.4.patch,
> HIVE-18079.5.patch, HIVE-18079.6.patch, HIVE-18079.7.patch,
> HIVE-18079.8.patch, HIVE-18079.9.patch
>
>
> HyperLogLog can merg
[jira] [Commented] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[
https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477243#comment-16477243
]
Hive QA commented on HIVE-18079:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923605/HIVE-18079.14.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 14423 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,scriptfile1.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[smb_cache]
(batchId=160)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testAutoPurgeUnset (batchId=233)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testNoAutoPurge (batchId=233)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testPartitionedExternalNoAutoPurge
(batchId=233)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10986/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10986/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10986/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923605 - PreCommit-HIVE-Build
> Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator
> bit-size
>
>
> Key: HIVE-18079
> URL: https://issues.apache.org/jira/browse/HIVE-18079
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore, Statistics
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-18079.1.patch, HIVE-18079.10.patch,
> HIVE-18079.11.patch, HIVE-18079.12.patch, HIVE-18079.13.patch,
> HIVE-18079.14.patch, HIVE-18079.2.patch, HIVE-18079.4.patch,
> HIVE-18079.5.patch, HIVE-18079.6.patch, HIVE-18079.7.patch,
> HIVE-18079.8.patch, HIVE-18079.9.patch
>
>
> HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its
> mathematical hash distribution & construction.
> Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second
> scan over the data-set.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19166) TestMiniLlapLocalCliDriver sysdb failure
[
https://issues.apache.org/jira/browse/HIVE-19166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477283#comment-16477283
]
Hive QA commented on HIVE-19166:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/util in master has 55 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
46s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10987/dev-support/hive-personality.sh
|
| git revision | master / 38c757c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: itests/util ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10987/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TestMiniLlapLocalCliDriver sysdb failure
>
>
> Key: HIVE-19166
> URL: https://issues.apache.org/jira/browse/HIVE-19166
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Vineet Garg
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19166.04.patch, HIVE-19166.05.patch,
> HIVE-19166.06.patch, HIVE-19166.09.patch, HIVE-19166.1.patch,
> HIVE-19166.2.patch, HIVE-19166.3.patch
>
>
> Broken by HIVE-18715
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19527) Preparing for 2.4 development
[ https://issues.apache.org/jira/browse/HIVE-19527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477307#comment-16477307 ] Peter Vary commented on HIVE-19527: --- +1 pending tests > Preparing for 2.4 development > - > > Key: HIVE-19527 > URL: https://issues.apache.org/jira/browse/HIVE-19527 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña >Priority: Major > Attachments: HIVE-19527.1.patch, HIVE-19527.2.patch > > > branch-2 is now being used for 2.4 development. > The build files will need to reflect this change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19570) Multiple inserts using "Group by" generates incorrect results
[
https://issues.apache.org/jira/browse/HIVE-19570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-19570:
Description:
Repro steps:
{code}
drop database if exists ax1 cascade;
create database ax1;
use ax1;
CREATE TABLE
tmp1 (
v1 string , v2 string , v3 string )
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
;
INSERT INTO tmp1
VALUES
('a', 'b', 'c1')
, ('a', 'b', 'c2')
, ('d', 'e', 'f')
, ('g', 'h', 'i')
;
CREATE TABLE
tmp_grouped_by_one_col ( v1 string , cnt__v2 int , cnt__v3 int )
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
;
CREATE TABLE
tmp_grouped_by_two_col ( v1 string , v2 string , cnt__v3 int )
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
;
CREATE TABLE
tmp_grouped_by_all_col ( v1 string , v2 string , v3 string )
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
;
FROM tmp1
INSERT INTO tmp_grouped_by_one_col
SELECT v1, count(distinct v2), count(distinct v3)
GROUP BY v1
INSERT INTO tmp_grouped_by_all_col
SELECT v1, v2, v3
GROUP BY v1, v2, v3
;
select 'tmp_grouped_by_one_col',count(*) from tmp_grouped_by_one_col
union all
select 'tmp_grouped_by_two_col',count(*) from tmp_grouped_by_two_col
union all
select 'tmp_grouped_by_all_col',count(*) from tmp_grouped_by_all_col;
select * from tmp_grouped_by_all_col;
{code}
tmp_grouped_by_all_col table should have 4 reocrds but it loads 7 records into
the table.
{code}
++++--+
| tmp_grouped_by_all_col.v1 | tmp_grouped_by_all_col.v2 |
tmp_grouped_by_all_col.v3 |
++++--+
| a | b | b
|
| a | c1 | c1
|
| a | c2 | c2
|
| d | e | e
|
| d | f | f
|
| g | h | h
|
| g | i | i
|
++++--+
{code}
was:
Repro steps:
drop database if exists ax1 cascade;
create database ax1;
use ax1;
CREATE TABLE
tmp1 (
v1 string , v2 string , v3 string )
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
;
INSERT INTO tmp1
VALUES
('a', 'b', 'c1')
, ('a', 'b', 'c2')
, ('d', 'e', 'f')
, ('g', 'h', 'i')
;
CREATE TABLE
tmp_grouped_by_one_col ( v1 string , cnt__v2 int , cnt__v3 int )
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
;
CREATE TABLE
tmp_grouped_by_two_col ( v1 string , v2 string , cnt__v3 int )
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
;
CREATE TABLE
tmp_grouped_by_all_col ( v1 string , v2 string , v3 string )
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
;
FROM tmp1
INSERT INTO tmp_grouped_by_one_col
SELECT v1, count(distinct v2), count(distinct v3)
GROUP BY v1
INSERT INTO tmp_grouped_by_all_col
SELECT v1, v2, v3
GROUP BY v1, v2, v3
;
select 'tmp_grouped_by_one_col',count(*) from tmp_grouped_by_one_col
union all
select 'tmp_grouped_by_two_col',count(*) from tmp_grouped_by_two_col
union all
select 'tmp_grouped_by_all_col',count(*) from tmp_grouped_by_all_col;
select * from tmp_grouped_by_all_col;
tmp_grouped_by_all_col table should have 4 reocrds but it loads 7 records into
the table.
++++--+
| tmp_grouped_by_all_col.v1 | tmp_grouped_by_all_col.v2 |
tmp_grouped_by_all_col.v3 |
++++--+
| a | b | b
|
| a | c1 | c1
|
| a | c2 | c2
|
| d | e | e
|
| d | f | f
|
| g | h | h
|
| g | i | i
|
++++--+
> Multiple inserts u
[jira] [Commented] (HIVE-19166) TestMiniLlapLocalCliDriver sysdb failure
[
https://issues.apache.org/jira/browse/HIVE-19166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477325#comment-16477325
]
Hive QA commented on HIVE-19166:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923506/HIVE-19166.09.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14406 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hive.hcatalog.mapreduce.TestHCatMultiOutputFormat.org.apache.hive.hcatalog.mapreduce.TestHCatMultiOutputFormat
(batchId=203)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10987/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10987/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10987/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923506 - PreCommit-HIVE-Build
> TestMiniLlapLocalCliDriver sysdb failure
>
>
> Key: HIVE-19166
> URL: https://issues.apache.org/jira/browse/HIVE-19166
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Vineet Garg
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19166.04.patch, HIVE-19166.05.patch,
> HIVE-19166.06.patch, HIVE-19166.09.patch, HIVE-19166.1.patch,
> HIVE-19166.2.patch, HIVE-19166.3.patch
>
>
> Broken by HIVE-18715
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-19540) Enable TestMiniLlapLocalCliDriver#sysdb.q
[ https://issues.apache.org/jira/browse/HIVE-19540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich resolved HIVE-19540. - Resolution: Fixed will be enabled as part of HIVE-19166 > Enable TestMiniLlapLocalCliDriver#sysdb.q > - > > Key: HIVE-19540 > URL: https://issues.apache.org/jira/browse/HIVE-19540 > Project: Hive > Issue Type: Test > Components: Test >Affects Versions: 3.1.0 >Reporter: Jesus Camacho Rodriguez >Priority: Critical > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-19543) Enable TestMiniLlapLocalCliDriver#union_fast_stats.q
[ https://issues.apache.org/jira/browse/HIVE-19543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich resolved HIVE-19543. - Resolution: Fixed will be enabled as part of HIVE-19326 > Enable TestMiniLlapLocalCliDriver#union_fast_stats.q > > > Key: HIVE-19543 > URL: https://issues.apache.org/jira/browse/HIVE-19543 > Project: Hive > Issue Type: Test > Components: Test >Affects Versions: 3.1.0 >Reporter: Jesus Camacho Rodriguez >Priority: Critical > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19490) Locking on Insert into for non native and managed tables.
[ https://issues.apache.org/jira/browse/HIVE-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-19490: -- Attachment: HIVE-19490.6.patch > Locking on Insert into for non native and managed tables. > - > > Key: HIVE-19490 > URL: https://issues.apache.org/jira/browse/HIVE-19490 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Labels: druid, locking > Attachments: HIVE-19490.2.patch, HIVE-19490.3.patch, > HIVE-19490.4.patch, HIVE-19490.5.patch, HIVE-19490.6.patch, HIVE-19490.patch > > > Current state of the art: > Managed non native table like Druid Tables, will need to get a Lock on Insert > into or insert Over write. The nature of this lock is set to Exclusive by > default for any non native table. > This implies that Inserts into Druid table will Lock any read query as well > during the execution of the insert into. IMO this lock (on insert into) is > not needed since the insert statement is appending data and the state of > loading it is managed partially by Hive Storage handler hook and part of it > by Druid. > What i am proposing is to relax the lock level to shared for all non native > tables on insert into operations and keep it as Exclusive Write for insert > Overwrite for now. > > Any feedback is welcome. > cc [~ekoifman] / [~ashutoshc] / [~jdere] / [~hagleitn] > Also am not sure what is the best way to unit test this currently am using > debugger to check if locks are what i except, please let me know if there is > a better way to do this. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19571) Ability to run multiple pre-commit jobs on a ptest server
[ https://issues.apache.org/jira/browse/HIVE-19571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar reassigned HIVE-19571: --- > Ability to run multiple pre-commit jobs on a ptest server > - > > Key: HIVE-19571 > URL: https://issues.apache.org/jira/browse/HIVE-19571 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > > I've been taking a look at the Disk, Network, and CPU usage of the GCE > instances we run ptest on, and it doesn't look like we are fully utilizing > the machines. The resource usage is very up and down. > During each ptest execution, there is a large chunk of time (~20 min) where > its just the Jenkins job that is doing any work (checking out github repos, > building code, figuring out test batches, etc.). During this time, the ptest > nodes are mostly idle - the CPU and Disk I/O are almost zero. > Even when ptest is running, I think some of resources are under-utilized. > Network and disk resource spike at the beginning of the job, probably because > ptest is distributing resources to each machine, each slave is downloading > jars, etc. However, after that, when the actual tests run, there is almost 0 > network activity (which makes sense since tests runs on a single node). For > disk usage, there is activity, but not nearly as high as when the setup phase > was occuring. CPU usage fluctuates between 40-80%. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[
https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477401#comment-16477401
]
Hive QA commented on HIVE-19308:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} llap-client in master has 26 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
54s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
30s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
16s{color} | {color:red} llap-ext-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
31s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 31s{color}
| {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} itests/hive-unit: The patch generated 11 new + 51
unchanged - 1 fixed = 62 total (was 52) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} llap-client: The patch generated 10 new + 35 unchanged
- 18 fixed = 45 total (was 53) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} llap-ext-client: The patch generated 4 new + 39
unchanged - 1 fixed = 43 total (was 40) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 1 new + 4 unchanged - 1 fixed
= 5 total (was 5) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
30s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
6s{color} | {color:red} ql generated 1 new + 2320 unchanged - 0 fixed = 2321
total (was 2320) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | org.apache.hadoop.hive.ql.io.arrow.ArrowWrapperWritable defines
compareTo(Object) and uses Object.equals() At
ArrowWrapperWritable.java:Object.equals() At ArrowWrapperWritable.java:[line
55] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
[jira] [Commented] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[
https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477431#comment-16477431
]
Hive QA commented on HIVE-18079:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923605/HIVE-18079.14.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10989/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10989/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10989/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12923605/HIVE-18079.14.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923605 - PreCommit-HIVE-Build
> Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator
> bit-size
>
>
> Key: HIVE-18079
> URL: https://issues.apache.org/jira/browse/HIVE-18079
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore, Statistics
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-18079.1.patch, HIVE-18079.10.patch,
> HIVE-18079.11.patch, HIVE-18079.12.patch, HIVE-18079.13.patch,
> HIVE-18079.14.patch, HIVE-18079.2.patch, HIVE-18079.4.patch,
> HIVE-18079.5.patch, HIVE-18079.6.patch, HIVE-18079.7.patch,
> HIVE-18079.8.patch, HIVE-18079.9.patch
>
>
> HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its
> mathematical hash distribution & construction.
> Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second
> scan over the data-set.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[
https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477430#comment-16477430
]
Hive QA commented on HIVE-19308:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923601/HIVE-19308.2.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 14411 tests
executed
*Failed tests:*
{noformat}
TestJdbcWithMiniLlap - did not produce a TEST-*.xml file (likely timed out)
(batchId=240)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[explainuser_2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_4]
(batchId=164)
org.apache.hadoop.hive.llap.TestLlapOutputFormat.testValues (batchId=264)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testNonAsciiStrings (batchId=240)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10988/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10988/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10988/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923601 - PreCommit-HIVE-Build
> Provide an Arrow stream reader for external LLAP clients
> -
>
> Key: HIVE-19308
> URL: https://issues.apache.org/jira/browse/HIVE-19308
> Project: Hive
> Issue Type: Task
> Components: llap
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch
>
>
> This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream
> and produces Arrow batches for an external client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19462) Fix mapping for char_length function to enable pushdown to Druid.
[ https://issues.apache.org/jira/browse/HIVE-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477468#comment-16477468 ] slim bouguerra commented on HIVE-19462: --- [~ashutoshc] can you please take a look at this. > Fix mapping for char_length function to enable pushdown to Druid. > -- > > Key: HIVE-19462 > URL: https://issues.apache.org/jira/browse/HIVE-19462 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Attachments: HIVE-19462.patch > > > currently char_length is not push down to Druid because of missing mapping > form/to calcite > This patch will add this mapping. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19567) Fix flakiness in TestTriggers
[ https://issues.apache.org/jira/browse/HIVE-19567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-19567: --- Resolution: Fixed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) Pushed to master. > Fix flakiness in TestTriggers > - > > Key: HIVE-19567 > URL: https://issues.apache.org/jira/browse/HIVE-19567 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19567.1.patch > > > Identified another flakiness in TestTriggersMoveWorkloadManager which can > cause intermittent test failures. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-12342) Set default value of hive.optimize.index.filter to true
[
https://issues.apache.org/jira/browse/HIVE-12342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477497#comment-16477497
]
Hive QA commented on HIVE-12342:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
34s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} contrib in master has 13 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
48s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10990/dev-support/hive-personality.sh
|
| git revision | master / 38c757c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common accumulo-handler contrib hbase-handler ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10990/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Set default value of hive.optimize.index.filter to true
> ---
>
> Key: HIVE-12342
> URL: https://issues.apache.org/jira/browse/HIVE-12342
> Project: Hive
> Issue Type: Task
> Components: Configuration
>Reporter: Ashutosh Chauhan
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-12342.05.patch, HIVE-12342.06.patch,
> HIVE-12342.07.patch, HIVE-12342.08.patch, HIVE-12342.09.patch,
> HIVE-12342.1.patch, HIVE-12342.10.patch, HIVE-12342.11.patch,
> HIVE-12342.12.patch, HIVE-12342.13.patch, HIVE-12342.2.patch,
> HIVE-12342.3.patch,
[jira] [Commented] (HIVE-19462) Fix mapping for char_length function to enable pushdown to Druid.
[ https://issues.apache.org/jira/browse/HIVE-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477503#comment-16477503 ] Ashutosh Chauhan commented on HIVE-19462: - +1 > Fix mapping for char_length function to enable pushdown to Druid. > -- > > Key: HIVE-19462 > URL: https://issues.apache.org/jira/browse/HIVE-19462 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Attachments: HIVE-19462.patch > > > currently char_length is not push down to Druid because of missing mapping > form/to calcite > This patch will add this mapping. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-12342) Set default value of hive.optimize.index.filter to true
[
https://issues.apache.org/jira/browse/HIVE-12342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477526#comment-16477526
]
Hive QA commented on HIVE-12342:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923606/HIVE-12342.13.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 80 failed/errored test(s), 14405 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partition_wise_fileformat2]
(batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_20]
(batchId=76)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union33] (batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union] (batchId=4)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez_empty]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketmapjoin1]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketmapjoin2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketmapjoin3]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_smb]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part1]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part2]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_rebuild_dummy]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=169)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query10]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query11]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query13]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query16]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query17]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query18]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query19]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query24]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query25]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query26]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query27]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query29]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query31]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query33]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query34]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query36]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query38]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query40]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query46]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query47]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query48]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query4]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query50]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query53]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query54]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query55]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query56]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query57]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query58]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query59]
(batchId=253)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query5]
(batchId=253)
org.apache.hadoop.hive.cl
[jira] [Updated] (HIVE-12342) Set default value of hive.optimize.index.filter to true
[ https://issues.apache.org/jira/browse/HIVE-12342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Kryvenko updated HIVE-12342: - Attachment: HIVE-12342.14.patch > Set default value of hive.optimize.index.filter to true > --- > > Key: HIVE-12342 > URL: https://issues.apache.org/jira/browse/HIVE-12342 > Project: Hive > Issue Type: Task > Components: Configuration >Reporter: Ashutosh Chauhan >Assignee: Igor Kryvenko >Priority: Major > Attachments: HIVE-12342.05.patch, HIVE-12342.06.patch, > HIVE-12342.07.patch, HIVE-12342.08.patch, HIVE-12342.09.patch, > HIVE-12342.1.patch, HIVE-12342.10.patch, HIVE-12342.11.patch, > HIVE-12342.12.patch, HIVE-12342.13.patch, HIVE-12342.14.patch, > HIVE-12342.2.patch, HIVE-12342.3.patch, HIVE-12342.4.patch, HIVE-12342.patch > > > This configuration governs ppd for storage layer. When applicable, it will > always help. It should be on by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19572) Add option to mask data size in q files
[
https://issues.apache.org/jira/browse/HIVE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez reassigned HIVE-19572:
--
> Add option to mask data size in q files
> ---
>
> Key: HIVE-19572
> URL: https://issues.apache.org/jira/browse/HIVE-19572
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Affects Versions: 3.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Some tests are flaky because of minimal data size differences, e.g., one
> byte. However, many times we do not actually care about these differences.
> One example is {{default_constraint.q}}.
> Patch adds the possibility to mask printing of data size stats selectively on
> q files by adding the {{-- MASK_DATA_SIZE}} option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19572) Add option to mask data size in q files
[
https://issues.apache.org/jira/browse/HIVE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19572:
---
Status: Patch Available (was: In Progress)
> Add option to mask data size in q files
> ---
>
> Key: HIVE-19572
> URL: https://issues.apache.org/jira/browse/HIVE-19572
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Affects Versions: 3.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19572.patch
>
>
> Some tests are flaky because of minimal data size differences, e.g., one
> byte. However, many times we do not actually care about these differences.
> One example is {{default_constraint.q}}.
> Patch adds the possibility to mask printing of data size stats selectively on
> q files by adding the {{-- MASK_DATA_SIZE}} option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19572) Add option to mask data size in q files
[
https://issues.apache.org/jira/browse/HIVE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19572:
---
Attachment: HIVE-19572.patch
> Add option to mask data size in q files
> ---
>
> Key: HIVE-19572
> URL: https://issues.apache.org/jira/browse/HIVE-19572
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Affects Versions: 3.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19572.patch
>
>
> Some tests are flaky because of minimal data size differences, e.g., one
> byte. However, many times we do not actually care about these differences.
> One example is {{default_constraint.q}}.
> Patch adds the possibility to mask printing of data size stats selectively on
> q files by adding the {{-- MASK_DATA_SIZE}} option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-19572) Add option to mask data size in q files
[
https://issues.apache.org/jira/browse/HIVE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-19572 started by Jesus Camacho Rodriguez.
--
> Add option to mask data size in q files
> ---
>
> Key: HIVE-19572
> URL: https://issues.apache.org/jira/browse/HIVE-19572
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Affects Versions: 3.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19572.patch
>
>
> Some tests are flaky because of minimal data size differences, e.g., one
> byte. However, many times we do not actually care about these differences.
> One example is {{default_constraint.q}}.
> Patch adds the possibility to mask printing of data size stats selectively on
> q files by adding the {{-- MASK_DATA_SIZE}} option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19572) Add option to mask data size in q files
[
https://issues.apache.org/jira/browse/HIVE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477604#comment-16477604
]
Jesus Camacho Rodriguez commented on HIVE-19572:
[~prasanth_j], could you review? Thanks
> Add option to mask data size in q files
> ---
>
> Key: HIVE-19572
> URL: https://issues.apache.org/jira/browse/HIVE-19572
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Affects Versions: 3.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19572.patch
>
>
> Some tests are flaky because of minimal data size differences, e.g., one
> byte. However, many times we do not actually care about these differences.
> One example is {{default_constraint.q}}.
> Patch adds the possibility to mask printing of data size stats selectively on
> q files by adding the {{-- MASK_DATA_SIZE}} option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19557) stats: filters for dates are not taking advantage of min/max values
[
https://issues.apache.org/jira/browse/HIVE-19557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477621#comment-16477621
]
Hive QA commented on HIVE-19557:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
49s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 1 new + 39 unchanged - 0 fixed
= 40 total (was 39) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 10 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
7s{color} | {color:red} ql generated 1 new + 2320 unchanged - 0 fixed = 2321
total (was 2320) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Comparison of String objects using == or != in
org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$FilterStatsRule.evaluateComparator(Statistics,
ExprNodeGenericFuncDesc, long) At StatsRulesProcFactory.java:== or != in
org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$FilterStatsRule.evaluateComparator(Statistics,
ExprNodeGenericFuncDesc, long) At StatsRulesProcFactory.java:[line 757] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10991/dev-support/hive-personality.sh
|
| git revision | master / bd9264f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10991/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10991/yetus/whitespace-tabs.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10991/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10991/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> stats: filters for dates are not taking advantage of min/max values
> ---
>
> Key: HIVE-19557
> URL: https://issues.apache.org/jira/browse/HIVE-19557
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19557.01.patch, HIVE-19557.02.patch
>
>
> in StatsRulesProcFactory
> [https://github.com/apache/hive/blob/ab189f54047bbf6beeeaf8d0dcfd5fbe92e465fb/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java#L754|dates
> are assumed to be an integer]; however this is currently not true - and the
> resul
[jira] [Assigned] (HIVE-19573) Fix flaky TestMiniLlapLocalCliDriver#explainuser_4.q
[ https://issues.apache.org/jira/browse/HIVE-19573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-19573: -- > Fix flaky TestMiniLlapLocalCliDriver#explainuser_4.q > > > Key: HIVE-19573 > URL: https://issues.apache.org/jira/browse/HIVE-19573 > Project: Hive > Issue Type: Test > Components: Test >Affects Versions: 3.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-19573) Fix flaky TestMiniLlapLocalCliDriver#explainuser_4.q
[ https://issues.apache.org/jira/browse/HIVE-19573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-19573 started by Jesus Camacho Rodriguez. -- > Fix flaky TestMiniLlapLocalCliDriver#explainuser_4.q > > > Key: HIVE-19573 > URL: https://issues.apache.org/jira/browse/HIVE-19573 > Project: Hive > Issue Type: Test > Components: Test >Affects Versions: 3.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19555) Enable TestMiniLlapLocalCliDriver#tez_dynpart_hashjoin_1.q and TestMiniLlapLocalCliDriver#tez_vector_dynpart_hashjoin_1.q
[ https://issues.apache.org/jira/browse/HIVE-19555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477631#comment-16477631 ] Jesus Camacho Rodriguez commented on HIVE-19555: +1 > Enable TestMiniLlapLocalCliDriver#tez_dynpart_hashjoin_1.q and > TestMiniLlapLocalCliDriver#tez_vector_dynpart_hashjoin_1.q > - > > Key: HIVE-19555 > URL: https://issues.apache.org/jira/browse/HIVE-19555 > Project: Hive > Issue Type: Test > Components: Test >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jason Dere >Priority: Critical > Fix For: 3.1.0 > > Attachments: HIVE-19555.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19555) Enable TestMiniLlapLocalCliDriver#tez_dynpart_hashjoin_1.q and TestMiniLlapLocalCliDriver#tez_vector_dynpart_hashjoin_1.q
[ https://issues.apache.org/jira/browse/HIVE-19555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-19555: --- Resolution: Fixed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) Pushed to master, thanks [~jdere]! > Enable TestMiniLlapLocalCliDriver#tez_dynpart_hashjoin_1.q and > TestMiniLlapLocalCliDriver#tez_vector_dynpart_hashjoin_1.q > - > > Key: HIVE-19555 > URL: https://issues.apache.org/jira/browse/HIVE-19555 > Project: Hive > Issue Type: Test > Components: Test >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jason Dere >Priority: Critical > Fix For: 3.1.0 > > Attachments: HIVE-19555.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19573) Fix flaky TestMiniLlapLocalCliDriver#explainuser_4.q
[ https://issues.apache.org/jira/browse/HIVE-19573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-19573: --- Fix Version/s: 3.1.0 Status: Patch Available (was: In Progress) Pushed to master. > Fix flaky TestMiniLlapLocalCliDriver#explainuser_4.q > > > Key: HIVE-19573 > URL: https://issues.apache.org/jira/browse/HIVE-19573 > Project: Hive > Issue Type: Test > Components: Test >Affects Versions: 3.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Fix For: 3.1.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19573) Fix flaky TestMiniLlapLocalCliDriver#explainuser_4.q
[ https://issues.apache.org/jira/browse/HIVE-19573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-19573: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Fix flaky TestMiniLlapLocalCliDriver#explainuser_4.q > > > Key: HIVE-19573 > URL: https://issues.apache.org/jira/browse/HIVE-19573 > Project: Hive > Issue Type: Test > Components: Test >Affects Versions: 3.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Fix For: 3.1.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19574) TestReoptimization ensure that runtimestats cache is loaded
[ https://issues.apache.org/jira/browse/HIVE-19574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-19574: --- > TestReoptimization ensure that runtimestats cache is loaded > --- > > Key: HIVE-19574 > URL: https://issues.apache.org/jira/browse/HIVE-19574 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > > there is a slight chance that the runtimestats are not loaded when the check > is done -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19557) stats: filters for dates are not taking advantage of min/max values
[
https://issues.apache.org/jira/browse/HIVE-19557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477703#comment-16477703
]
Hive QA commented on HIVE-19557:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923609/HIVE-19557.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 14396 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,scriptfile1.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_cross_product_check_2]
(batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[d1] (batchId=6)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_4]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_between_in]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=183)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10991/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10991/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10991/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923609 - PreCommit-HIVE-Build
> stats: filters for dates are not taking advantage of min/max values
> ---
>
> Key: HIVE-19557
> URL: https://issues.apache.org/jira/browse/HIVE-19557
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19557.01.patch, HIVE-19557.02.patch
>
>
> in StatsRulesProcFactory
> [https://github.com/apache/hive/blob/ab189f54047bbf6beeeaf8d0dcfd5fbe92e465fb/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java#L754|dates
> are assumed to be an integer]; however this is currently not true - and the
> resulting exception is handled as a default case... for N/3
> {code}
> set hive.explain.user=true;
> create table d1(d date);
> -- tblproperties('transactional'='false');
> insert into d1 values
> ('2010-10-01'),
> ('2010-10-02'),
> ('2010-10-03'),
> ('2010-10-04'),
> ('2010-10-05'),
> ('2010-10-06'),
> ('2010-10-07'),
> ('2010-10-08'),
> ('2010-10-09'),
> ('2010-10-10');
> analyze table d1 compute statistics for columns;
> desc formatted d1;
> desc formatted d1 d;
> explain
> select 'stats: FIL ~0 read',count(1) from d1 where d < '2010-03-01';
> explain
> select 'stats: FIL estimate some read',count(1) from d1 where d <
> '2010-10-03';
> explain
> select 'stats: FIL estimate all read',count(1) from d1 where d < '2010-11-03';
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-6980) Drop table by using direct sql
[
https://issues.apache.org/jira/browse/HIVE-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477751#comment-16477751
]
Hive QA commented on HIVE-6980:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
55s{color} | {color:blue} standalone-metastore in master has 215 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore: The patch generated 6 new + 675
unchanged - 5 fixed = 681 total (was 680) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
8s{color} | {color:red} standalone-metastore generated 1 new + 214 unchanged -
1 fixed = 215 total (was 215) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore |
| | org.apache.hadoop.hive.metastore.ObjectStore$12.run(List) concatenates
strings using + in a loop At ObjectStore.java:in a loop At
ObjectStore.java:[line 8163] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10992/dev-support/hive-personality.sh
|
| git revision | master / ce0f7b0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10992/yetus/diff-checkstyle-standalone-metastore.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10992/yetus/new-findbugs-standalone-metastore.html
|
| modules | C: standalone-metastore U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10992/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Drop table by using direct sql
> --
>
> Key: HIVE-6980
> URL: https://issues.apache.org/jira/browse/HIVE-6980
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 0.12.0
>Reporter: Selina Zhang
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-6980.2.patch, HIVE-6980.3.patch, HIVE-6980.4.patch,
> HIVE-6980.patch, drop_table_after.png, drop_table_before.png
>
>
> Dropping table which has lots of partitions is slow. Even after applying the
> patch of HIVE-6265, the drop table still takes hours (100K+ partitions).
> The fixes come with two parts:
> 1. use directSQL to query the partitions protect mode;
> the current implementation needs to transfer the Partition object to client
> and check the protect mode for each partition. I'd like to move this part of
> logic to metastore. The check will be done by direct sql (if direct sql is
> disabled, execut
[jira] [Commented] (HIVE-19496) Check untar folder
[ https://issues.apache.org/jira/browse/HIVE-19496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477766#comment-16477766 ] Aihua Xu commented on HIVE-19496: - Also pushed to branch-3. > Check untar folder > -- > > Key: HIVE-19496 > URL: https://issues.apache.org/jira/browse/HIVE-19496 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Major > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19496.1.patch, HIVE-19496.2.patch > > > We need to check if the file is under untar folder. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19496) Check untar folder
[ https://issues.apache.org/jira/browse/HIVE-19496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-19496: Fix Version/s: 3.0.0 > Check untar folder > -- > > Key: HIVE-19496 > URL: https://issues.apache.org/jira/browse/HIVE-19496 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Major > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19496.1.patch, HIVE-19496.2.patch > > > We need to check if the file is under untar folder. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19557) stats: filters for dates are not taking advantage of min/max values
[
https://issues.apache.org/jira/browse/HIVE-19557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1640#comment-1640
]
Ashutosh Chauhan commented on HIVE-19557:
-
+1
> stats: filters for dates are not taking advantage of min/max values
> ---
>
> Key: HIVE-19557
> URL: https://issues.apache.org/jira/browse/HIVE-19557
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19557.01.patch, HIVE-19557.02.patch
>
>
> in StatsRulesProcFactory
> [https://github.com/apache/hive/blob/ab189f54047bbf6beeeaf8d0dcfd5fbe92e465fb/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java#L754|dates
> are assumed to be an integer]; however this is currently not true - and the
> resulting exception is handled as a default case... for N/3
> {code}
> set hive.explain.user=true;
> create table d1(d date);
> -- tblproperties('transactional'='false');
> insert into d1 values
> ('2010-10-01'),
> ('2010-10-02'),
> ('2010-10-03'),
> ('2010-10-04'),
> ('2010-10-05'),
> ('2010-10-06'),
> ('2010-10-07'),
> ('2010-10-08'),
> ('2010-10-09'),
> ('2010-10-10');
> analyze table d1 compute statistics for columns;
> desc formatted d1;
> desc formatted d1 d;
> explain
> select 'stats: FIL ~0 read',count(1) from d1 where d < '2010-03-01';
> explain
> select 'stats: FIL estimate some read',count(1) from d1 where d <
> '2010-10-03';
> explain
> select 'stats: FIL estimate all read',count(1) from d1 where d < '2010-11-03';
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-6980) Drop table by using direct sql
[
https://issues.apache.org/jira/browse/HIVE-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477810#comment-16477810
]
Hive QA commented on HIVE-6980:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923219/HIVE-6980.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14377 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=157)
[filter_union.q,bucket3.q,mm_bhif.q,materialized_view_create_rewrite_4.q,schema_evol_text_nonvec_table.q,mrr.q,orc_ppd_schema_evol_2b.q,orc_analyze.q,cbo_rp_udf_udaf_stats_opt.q,subquery_views.q,vector_interval_1.q,tez_join_result_complex.q,partition_shared_scan.q,vector_map_order.q,ptf_matchpath.q,constprog_dpp.q,vector_groupby_grouping_sets3.q,leftsemijoin_mr.q,results_cache_transactional.q,unionDistinct_2.q,vectorization_14.q,load_data_using_job.q,cbo_stats.q,auto_sortmerge_join_6.q,vector_groupby4.q,ptf.q,update_where_non_partitioned.q,insert_dir_distcp.q,schema_evol_orc_nonvec_part_llap_io.q,vector_windowing_range_multiorder.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10992/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10992/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10992/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923219 - PreCommit-HIVE-Build
> Drop table by using direct sql
> --
>
> Key: HIVE-6980
> URL: https://issues.apache.org/jira/browse/HIVE-6980
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 0.12.0
>Reporter: Selina Zhang
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-6980.2.patch, HIVE-6980.3.patch, HIVE-6980.4.patch,
> HIVE-6980.patch, drop_table_after.png, drop_table_before.png
>
>
> Dropping table which has lots of partitions is slow. Even after applying the
> patch of HIVE-6265, the drop table still takes hours (100K+ partitions).
> The fixes come with two parts:
> 1. use directSQL to query the partitions protect mode;
> the current implementation needs to transfer the Partition object to client
> and check the protect mode for each partition. I'd like to move this part of
> logic to metastore. The check will be done by direct sql (if direct sql is
> disabled, execute the same logic in the ObjectStore);
> 2. use directSQL to drop partitions for table;
> there maybe two solutions here:
> 1. add "DELETE CASCADE" in the schema. In this way we only need to delete
> entries from partitions table use direct sql. May need to change
> datanucleus.deletionPolicy = DataNucleus.
> 2. clean up the dependent tables by issue DELETE statement. This also needs
> to turn on datanucleus.query.sql.allowAll
> Both of above solutions should be able to fix the problem. The DELETE CASCADE
> has to change schemas and prepare upgrade scripts. The second solutions added
> maintenance cost if new tables added in the future releases.
> Please advice.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19496) Check untar folder
[ https://issues.apache.org/jira/browse/HIVE-19496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477829#comment-16477829 ] Vineet Garg commented on HIVE-19496: [~aihuaxu] FYI branch-3 has been closed for 3.0 release to prepare 3.0 RC. Your commit to branch-3 will not go in 3.0 but in 3.1. > Check untar folder > -- > > Key: HIVE-19496 > URL: https://issues.apache.org/jira/browse/HIVE-19496 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Major > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19496.1.patch, HIVE-19496.2.patch > > > We need to check if the file is under untar folder. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19496) Check untar folder
[ https://issues.apache.org/jira/browse/HIVE-19496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19496: --- Fix Version/s: (was: 3.0.0) > Check untar folder > -- > > Key: HIVE-19496 > URL: https://issues.apache.org/jira/browse/HIVE-19496 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19496.1.patch, HIVE-19496.2.patch > > > We need to check if the file is under untar folder. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19572) Add option to mask data size in q files
[
https://issues.apache.org/jira/browse/HIVE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19572:
---
Attachment: HIVE-19572.01.patch
> Add option to mask data size in q files
> ---
>
> Key: HIVE-19572
> URL: https://issues.apache.org/jira/browse/HIVE-19572
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Affects Versions: 3.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19572.01.patch
>
>
> Some tests are flaky because of minimal data size differences, e.g., one
> byte. However, many times we do not actually care about these differences.
> One example is {{default_constraint.q}}.
> Patch adds the possibility to mask printing of data size stats selectively on
> q files by adding the {{-- MASK_DATA_SIZE}} option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19572) Add option to mask data size in q files
[
https://issues.apache.org/jira/browse/HIVE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19572:
---
Attachment: (was: HIVE-19572.patch)
> Add option to mask data size in q files
> ---
>
> Key: HIVE-19572
> URL: https://issues.apache.org/jira/browse/HIVE-19572
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Affects Versions: 3.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19572.01.patch
>
>
> Some tests are flaky because of minimal data size differences, e.g., one
> byte. However, many times we do not actually care about these differences.
> One example is {{default_constraint.q}}.
> Patch adds the possibility to mask printing of data size stats selectively on
> q files by adding the {{-- MASK_DATA_SIZE}} option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18748) Rename table impacts the ACID behaviour as table names are not updated in meta-tables.
[ https://issues.apache.org/jira/browse/HIVE-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477847#comment-16477847 ] Eugene Koifman commented on HIVE-18748: --- fixed checkstyle issues. [~sankarh] could you take a look please > Rename table impacts the ACID behaviour as table names are not updated in > meta-tables. > -- > > Key: HIVE-18748 > URL: https://issues.apache.org/jira/browse/HIVE-18748 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Eugene Koifman >Priority: Critical > Labels: ACID, DDL > Attachments: HIVE-18748.02.patch, HIVE-18748.03.patch, > HIVE-18748.04.patch, HIVE-18748.05.patch > > > ACID implementation uses metatables such as TXN_COMPONENTS, > COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPCTION_QUEUE etc to > manage ACID operations. > Per table write ID implementation (HIVE-18192) introduces couple of > metatables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID to manage write ids > allocated per table. > Now, when we rename any tables, it is necessary to update the corresponding > table names in these metatables as well. Otherwise, ACID table operations > won't work properly. > Since, this change is significant and have other side-effects, we propose to > disable rename tables on ACID tables until a fix is figured out. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18748) Rename table impacts the ACID behaviour as table names are not updated in meta-tables.
[ https://issues.apache.org/jira/browse/HIVE-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18748: -- Attachment: HIVE-18748.05.patch > Rename table impacts the ACID behaviour as table names are not updated in > meta-tables. > -- > > Key: HIVE-18748 > URL: https://issues.apache.org/jira/browse/HIVE-18748 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Eugene Koifman >Priority: Critical > Labels: ACID, DDL > Attachments: HIVE-18748.02.patch, HIVE-18748.03.patch, > HIVE-18748.04.patch, HIVE-18748.05.patch > > > ACID implementation uses metatables such as TXN_COMPONENTS, > COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPCTION_QUEUE etc to > manage ACID operations. > Per table write ID implementation (HIVE-18192) introduces couple of > metatables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID to manage write ids > allocated per table. > Now, when we rename any tables, it is necessary to update the corresponding > table names in these metatables as well. Otherwise, ACID table operations > won't work properly. > Since, this change is significant and have other side-effects, we propose to > disable rename tables on ACID tables until a fix is figured out. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19490) Locking on Insert into for non native and managed tables.
[
https://issues.apache.org/jira/browse/HIVE-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477868#comment-16477868
]
Hive QA commented on HIVE-19490:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} druid-handler in master has 12 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
14s{color} | {color:red} druid-handler in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} druid-handler: The patch generated 0 new + 44
unchanged - 12 fixed = 44 total (was 56) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 1 new + 1023 unchanged - 52
fixed = 1024 total (was 1075) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10993/dev-support/hive-personality.sh
|
| git revision | master / ce0f7b0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10993/yetus/patch-mvninstall-druid-handler.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10993/yetus/diff-checkstyle-ql.txt
|
| modules | C: druid-handler ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10993/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Locking on Insert into for non native and managed tables.
> -
>
> Key: HIVE-19490
> URL: https://issues.apache.org/jira/browse/HIVE-19490
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Labels: druid, locking
> Attachments: HIVE-19490.2.patch, HIVE-19490.3.patch,
> HIVE-19490.4.patch, HIVE-19490.5.patch, HIVE-19490.6.patch, HIVE-19490.patch
>
>
> Current state of the art:
> Managed non native table like Druid Tables, will need to get a Lock on Insert
> into or in
[jira] [Commented] (HIVE-19572) Add option to mask data size in q files
[
https://issues.apache.org/jira/browse/HIVE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477869#comment-16477869
]
Prasanth Jayachandran commented on HIVE-19572:
--
+1
> Add option to mask data size in q files
> ---
>
> Key: HIVE-19572
> URL: https://issues.apache.org/jira/browse/HIVE-19572
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Affects Versions: 3.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19572.01.patch
>
>
> Some tests are flaky because of minimal data size differences, e.g., one
> byte. However, many times we do not actually care about these differences.
> One example is {{default_constraint.q}}.
> Patch adds the possibility to mask printing of data size stats selectively on
> q files by adding the {{-- MASK_DATA_SIZE}} option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19496) Check untar folder
[ https://issues.apache.org/jira/browse/HIVE-19496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477876#comment-16477876 ] Aihua Xu commented on HIVE-19496: - I see. Thanks [~vgarg] for letting me know. > Check untar folder > -- > > Key: HIVE-19496 > URL: https://issues.apache.org/jira/browse/HIVE-19496 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19496.1.patch, HIVE-19496.2.patch > > > We need to check if the file is under untar folder. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19572) Add option to mask stats and data size in q files
[
https://issues.apache.org/jira/browse/HIVE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19572:
---
Summary: Add option to mask stats and data size in q files (was: Add
option to mask data size in q files)
> Add option to mask stats and data size in q files
> -
>
> Key: HIVE-19572
> URL: https://issues.apache.org/jira/browse/HIVE-19572
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Affects Versions: 3.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19572.01.patch
>
>
> Some tests are flaky because of minimal data size differences, e.g., one
> byte. However, many times we do not actually care about these differences.
> One example is {{default_constraint.q}}.
> Patch adds the possibility to mask printing of data size stats selectively on
> q files by adding the {{-- MASK_DATA_SIZE}} option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19572) Add option to mask stats and data size in q files
[
https://issues.apache.org/jira/browse/HIVE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19572:
---
Description:
Some tests are flaky because of minimal data size differences, e.g., one byte.
However, many times we do not actually care about these differences. One
example is {{default_constraint.q}}.
Patch adds the possibility to mask 1) printing of stats selectively on q files
by adding the {{-- MASK_STATS}} option, and 2) printing of data size stats
selectively on q files by adding the {{-- MASK_DATA_SIZE}} option.
was:
Some tests are flaky because of minimal data size differences, e.g., one byte.
However, many times we do not actually care about these differences. One
example is {{default_constraint.q}}.
Patch adds the possibility to mask printing of data size stats selectively on q
files by adding the {{-- MASK_DATA_SIZE}} option.
> Add option to mask stats and data size in q files
> -
>
> Key: HIVE-19572
> URL: https://issues.apache.org/jira/browse/HIVE-19572
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Affects Versions: 3.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19572.01.patch
>
>
> Some tests are flaky because of minimal data size differences, e.g., one
> byte. However, many times we do not actually care about these differences.
> One example is {{default_constraint.q}}.
> Patch adds the possibility to mask 1) printing of stats selectively on q
> files by adding the {{-- MASK_STATS}} option, and 2) printing of data size
> stats selectively on q files by adding the {{-- MASK_DATA_SIZE}} option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[ https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-18079: Assignee: Prasanth Jayachandran (was: Gopal V) > Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator > bit-size > > > Key: HIVE-18079 > URL: https://issues.apache.org/jira/browse/HIVE-18079 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Statistics >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18079.1.patch, HIVE-18079.10.patch, > HIVE-18079.11.patch, HIVE-18079.12.patch, HIVE-18079.13.patch, > HIVE-18079.14.patch, HIVE-18079.15.patch, HIVE-18079.2.patch, > HIVE-18079.4.patch, HIVE-18079.5.patch, HIVE-18079.6.patch, > HIVE-18079.7.patch, HIVE-18079.8.patch, HIVE-18079.9.patch > > > HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its > mathematical hash distribution & construction. > Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second > scan over the data-set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[ https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18079: - Attachment: HIVE-18079.15.patch > Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator > bit-size > > > Key: HIVE-18079 > URL: https://issues.apache.org/jira/browse/HIVE-18079 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Statistics >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18079.1.patch, HIVE-18079.10.patch, > HIVE-18079.11.patch, HIVE-18079.12.patch, HIVE-18079.13.patch, > HIVE-18079.14.patch, HIVE-18079.15.patch, HIVE-18079.2.patch, > HIVE-18079.4.patch, HIVE-18079.5.patch, HIVE-18079.6.patch, > HIVE-18079.7.patch, HIVE-18079.8.patch, HIVE-18079.9.patch > > > HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its > mathematical hash distribution & construction. > Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second > scan over the data-set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[ https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-18079: Assignee: Gopal V (was: Prasanth Jayachandran) > Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator > bit-size > > > Key: HIVE-18079 > URL: https://issues.apache.org/jira/browse/HIVE-18079 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Statistics >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-18079.1.patch, HIVE-18079.10.patch, > HIVE-18079.11.patch, HIVE-18079.12.patch, HIVE-18079.13.patch, > HIVE-18079.14.patch, HIVE-18079.15.patch, HIVE-18079.2.patch, > HIVE-18079.4.patch, HIVE-18079.5.patch, HIVE-18079.6.patch, > HIVE-18079.7.patch, HIVE-18079.8.patch, HIVE-18079.9.patch > > > HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its > mathematical hash distribution & construction. > Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second > scan over the data-set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19575) TestAutoPurgeTables seems flaky
[ https://issues.apache.org/jira/browse/HIVE-19575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-19575: > TestAutoPurgeTables seems flaky > --- > > Key: HIVE-19575 > URL: https://issues.apache.org/jira/browse/HIVE-19575 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19575.1.patch > > > I cannot reproduce the flakiness locally. Maybe we can retry this flaky test > using RetryTestRunner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19575) TestAutoPurgeTables seems flaky
[ https://issues.apache.org/jira/browse/HIVE-19575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19575: - Attachment: HIVE-19575.1.patch > TestAutoPurgeTables seems flaky > --- > > Key: HIVE-19575 > URL: https://issues.apache.org/jira/browse/HIVE-19575 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19575.1.patch > > > I cannot reproduce the flakiness locally. Maybe we can retry this flaky test > using RetryTestRunner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19575) TestAutoPurgeTables seems flaky
[ https://issues.apache.org/jira/browse/HIVE-19575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477907#comment-16477907 ] Jesus Camacho Rodriguez commented on HIVE-19575: +1 > TestAutoPurgeTables seems flaky > --- > > Key: HIVE-19575 > URL: https://issues.apache.org/jira/browse/HIVE-19575 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19575.1.patch > > > I cannot reproduce the flakiness locally. Maybe we can retry this flaky test > using RetryTestRunner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19418) add background stats updater similar to compactor
[ https://issues.apache.org/jira/browse/HIVE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19418: Attachment: HIVE-19418.01.patch > add background stats updater similar to compactor > - > > Key: HIVE-19418 > URL: https://issues.apache.org/jira/browse/HIVE-19418 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19418.01.patch, HIVE-19418.patch > > > There's a JIRA HIVE-19416 to add snapshot version to stats for MM/ACID tables > to make them usable in a transaction without breaking ACID (for metadata-only > optimization). However, stats for ACID tables can still become unusable if > e.g. two parallel inserts run - neither sees the data written by the other, > so after both finish, the snapshots on either set of stats won't match the > current snapshot and the stats will be unusable. > Additionally, for ACID and non-ACID tables alike, a lot of the stats, with > some exceptions like numRows, cannot be aggregated (i.e. you cannot combine > ndvs from two inserts), and for ACID even less can be aggregated (you cannot > derive min/max if some rows are deleted but you don't scan the rest of the > dataset). > Therefore we will add background logic to metastore (similar to, and > partially inside, the ACID compactor) to update stats. > It will have 3 modes of operation. > 1) Off. > 2) Update only the stats that exist but are out of date (generating stats can > be expensive, so if the user is only analyzing a subset of tables it should > be able to only update that subset). We can simply look at existing stats and > only analyze for the relevant partitions and columns. > 3) On: 2 + create stats for all tables and columns missing stats. > There will also be a table parameter to skip stats update. > In phase 1, the process will operate outside of compactor, and run analyze > command on the table. The analyze command will automatically save the stats > with ACID snapshot information if needed, based on HIVE-19416, so we don't > need to do any special state management and this will work for all table > types. However it's also more expensive. > In phase 2, we can explore adding stats collection during MM compaction that > uses a temp table. If we don't have open writers during major compaction (so > we overwrite all of the data), the temp table stats can simply be copied over > to the main table with correct snapshot information, saving us a table scan. > In phase 3, we can add custom stats collection logic to full ACID compactor > that is not query based, the same way as we'd do for (2). Alternatively we > can wait for ACID compactor to become query based and just reuse (2). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19516) TestNegative merge_negative_5 and mm_concatenate are causing timeouts
[
https://issues.apache.org/jira/browse/HIVE-19516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-19516:
Attachment: HIVE-19516.01.patch
> TestNegative merge_negative_5 and mm_concatenate are causing timeouts
> -
>
> Key: HIVE-19516
> URL: https://issues.apache.org/jira/browse/HIVE-19516
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Vineet Garg
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19516.01.patch, HIVE-19516.patch
>
>
> I haven't tried to reproduce this in isolation but it is reproducible if you
> run in batch on local system
> {noformat}
> mvn -B test -Dtest.groups= -Dtest=TestNegativeCliDriver
> -Dqfile=udf_invalid.q,authorization_uri_export.q,default_constraint_complex_default_value.q,druid_datasource2.q,check_constraint_max_length.q,view_update.q,default_partition_name.q,authorization_public_create.q,load_wrong_fileformat_rc_seq.q,default_constraint_invalid_type.q,altern1.q,describe_xpath1.q,drop_view_failure2.q,temp_table_rename.q,invalid_select_column_with_subquery.q,udf_trunc_error1.q,insert_view_failure.q,dbtxnmgr_nodbunlock.q,authorization_show_columns.q,cte_recursion.q,merge_constraint_notnull.q,clusterbyorderby.q,orc_type_promotion2.q,ctas_noperm_loc.q,udf_min.q,udf_instr_wrong_args_len.q,invalid_create_tbl2.q,part_col_complex_type.q,authorization_drop_db_empty.q,smb_mapjoin_14.q,subquery_scalar_multi_rows.q,alter_partition_coltype_2columns.q,subquery_corr_in_agg.q,insert_overwrite_notnull_constraint.q,authorization_show_grant_otheruser_wtab.q,regex_col_groupby.q,ptf_negative_DuplicateWindowAlias.q,exim_22_export_authfail.q,authorization_insert_noinspriv.q,udf_likeany_wrong1.q,groupby_key.q,ambiguous_col.q,groupby3_multi_distinct.q,authorization_alter_drop_ptn.q,invalid_cast_from_binary_5.q,show_create_table_does_not_exist.q,invalid_select_column.q,exim_20_managed_location_over_existing.q,interval_3.q,authorization_compile.q,join35.q,udf_concat_ws_wrong3.q,create_or_replace_view8.q,create_external_with_notnull_constraint.q,split_sample_out_of_range.q,materialized_view_no_transactional_rewrite.q,authorization_show_grant_otherrole.q,create_with_constraints_duplicate_name.q,invalid_stddev_samp_syntax.q,authorization_view_disable_cbo_7.q,autolocal1.q,avro_non_nullable_union.q,load_orc_negative_part.q,drop_view_failure1.q,columnstats_partlvl_invalid_values_autogather.q,exim_13_nonnative_import.q,alter_table_wrong_regex.q,udf_next_day_error_2.q,authorization_select.q,udf_trunc_error2.q,authorization_view_7.q,udf_format_number_wrong5.q,touch2.q,orc_type_promotion1.q,lateral_view_alias.q,show_tables_bad_db1.q,unset_table_property.q,alter_non_native.q,nvl_mismatch_type.q,load_orc_negative3.q,authorization_create_role_no_admin.q,invalid_distinct1.q,authorization_grant_server.q,orc_type_promotion3_acid.q,hms_using_serde_alter_table_update_columns.q,show_tables_bad1.q,macro_unused_parameter.q,drop_invalid_constraint3.q,drop_partition_filter_failure.q,char_pad_convert_fail3.q,exim_23_import_exist_authfail.q,drop_invalid_constraint4.q,authorization_create_macro1.q,archive1.q,subquery_multiple_cols_in_select.q,change_hive_hdfs_session_path.q,udf_trunc_error3.q,invalid_variance_syntax.q,authorization_truncate_2.q,invalid_avg_syntax.q,invalid_select_column_with_tablename.q,mm_truncate_cols.q,groupby_grouping_sets1.q,druid_location.q,groupby2_multi_distinct.q,authorization_sba_drop_table.q,dynamic_partitions_with_whitelist.q,compare_string_bigint_2.q,udf_greatest_error_2.q,authorization_view_6.q,show_tablestatus.q,duplicate_alias_in_transform_schema.q,create_with_fk_uk_same_tab.q,udtf_not_supported3.q,alter_table_constraint_invalid_fk_col2.q,udtf_not_supported1.q,dbtxnmgr_notableunlock.q,ptf_negative_InvalidValueBoundary.q,alter_table_constraint_duplicate_pk.q,udf_printf_wrong4.q,create_view_failure9.q,udf_elt_wrong_type.q,selectDistinctStarNeg_1.q,invalid_mapjoin1.q,load_stored_as_dirs.q,input1.q,udf_sort_array_wrong1.q,invalid_distinct2.q,invalid_select_fn.q,authorization_role_grant_otherrole.q,archive4.q,load_nonpart_authfail.q,recursive_view.q,authorization_view_disable_cbo_1.q,desc_failure4.q,create_not_acid.q,udf_sort_array_wrong3.q,char_pad_convert_fail0.q,udf_map_values_arg_type.q,alter_view_failure6_2.q,alter_partition_change_col_nonexist.q,update_non_acid_table.q,authorization_view_disable_cbo_5.q,ct_noperm_loc.q,interval_1.q,authorization_show_grant_otheruser_all.q,authorization_view_2.q,show_tables_bad2.q,groupby_rollup2.q,truncate_column_seqfile.q,create_view_failure5.q,authorization_create_view.q,ptf_window_boundaries.q,ctasnullcol.q,input_part0_neg_2.q,create_or_replace_view1.q,udf_max.q,exim_01_nonpart_over_loaded.q,msck_repair_1.q,orc_change_fileformat_acid.q,
[jira] [Commented] (HIVE-19258) add originals support to MM tables (and make the conversion a metadata only operation)
[ https://issues.apache.org/jira/browse/HIVE-19258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477937#comment-16477937 ] Sergey Shelukhin commented on HIVE-19258: - Again for HiveQA... yet again. > add originals support to MM tables (and make the conversion a metadata only > operation) > -- > > Key: HIVE-19258 > URL: https://issues.apache.org/jira/browse/HIVE-19258 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19258.01.patch, HIVE-19258.02.patch, > HIVE-19258.03.patch, HIVE-19258.04.patch, HIVE-19258.05.patch, > HIVE-19258.06.patch, HIVE-19258.07.patch, HIVE-19258.08.patch, > HIVE-19258.08.patch, HIVE-19258.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19258) add originals support to MM tables (and make the conversion a metadata only operation)
[ https://issues.apache.org/jira/browse/HIVE-19258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19258: Attachment: HIVE-19258.08.patch > add originals support to MM tables (and make the conversion a metadata only > operation) > -- > > Key: HIVE-19258 > URL: https://issues.apache.org/jira/browse/HIVE-19258 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19258.01.patch, HIVE-19258.02.patch, > HIVE-19258.03.patch, HIVE-19258.04.patch, HIVE-19258.05.patch, > HIVE-19258.06.patch, HIVE-19258.07.patch, HIVE-19258.08.patch, > HIVE-19258.08.patch, HIVE-19258.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19463) TezTask - getting groups may fail (PartialGroupNameException in some tests)
[
https://issues.apache.org/jira/browse/HIVE-19463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477939#comment-16477939
]
Sergey Shelukhin commented on HIVE-19463:
-
Again, HiveQA already lost this from yesterday...
> TezTask - getting groups may fail (PartialGroupNameException in some tests)
> ---
>
> Key: HIVE-19463
> URL: https://issues.apache.org/jira/browse/HIVE-19463
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19463.01.patch, HIVE-19463.02.patch,
> HIVE-19463.03.patch, HIVE-19463.03.patch, HIVE-19463.patch
>
>
> {noformat}
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping$PartialGroupNameException:
> The user name 'hive_test_user' is not found. id: hive_test_user: no such user
> id: hive_test_user: no such user
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.resolvePartialGroupNames(ShellBasedUnixGroupsMapping.java:294)
> ~[hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:207)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:97)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:51)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.security.Groups$GroupCacheLoader.fetchGroupList(Groups.java:384)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:319)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:269)
> [hadoop-common-3.1.0.jar:?]
> at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3542)
> [guava-19.0.jar:?]
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2323)
> [guava-19.0.jar:?]
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2286)
> [guava-19.0.jar:?]
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
> [guava-19.0.jar:?]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3953)
> [guava-19.0.jar:?]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3957)
> [guava-19.0.jar:?]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4875)
> [guava-19.0.jar:?]
> at org.apache.hadoop.security.Groups.getGroups(Groups.java:227)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.security.UserGroupInformation.getGroups(UserGroupInformation.java:1540)
> [hadoop-common-3.1.0.jar:?]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:163)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19463) TezTask - getting groups may fail (PartialGroupNameException in some tests)
[
https://issues.apache.org/jira/browse/HIVE-19463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-19463:
Attachment: HIVE-19463.03.patch
> TezTask - getting groups may fail (PartialGroupNameException in some tests)
> ---
>
> Key: HIVE-19463
> URL: https://issues.apache.org/jira/browse/HIVE-19463
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19463.01.patch, HIVE-19463.02.patch,
> HIVE-19463.03.patch, HIVE-19463.03.patch, HIVE-19463.patch
>
>
> {noformat}
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping$PartialGroupNameException:
> The user name 'hive_test_user' is not found. id: hive_test_user: no such user
> id: hive_test_user: no such user
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.resolvePartialGroupNames(ShellBasedUnixGroupsMapping.java:294)
> ~[hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:207)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:97)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:51)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.security.Groups$GroupCacheLoader.fetchGroupList(Groups.java:384)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:319)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:269)
> [hadoop-common-3.1.0.jar:?]
> at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3542)
> [guava-19.0.jar:?]
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2323)
> [guava-19.0.jar:?]
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2286)
> [guava-19.0.jar:?]
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
> [guava-19.0.jar:?]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3953)
> [guava-19.0.jar:?]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3957)
> [guava-19.0.jar:?]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4875)
> [guava-19.0.jar:?]
> at org.apache.hadoop.security.Groups.getGroups(Groups.java:227)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.security.UserGroupInformation.getGroups(UserGroupInformation.java:1540)
> [hadoop-common-3.1.0.jar:?]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:163)
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19490) Locking on Insert into for non native and managed tables.
[
https://issues.apache.org/jira/browse/HIVE-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477942#comment-16477942
]
Hive QA commented on HIVE-19490:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923674/HIVE-19490.6.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 14399 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,scriptfile1.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=183)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10993/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10993/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10993/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923674 - PreCommit-HIVE-Build
> Locking on Insert into for non native and managed tables.
> -
>
> Key: HIVE-19490
> URL: https://issues.apache.org/jira/browse/HIVE-19490
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Labels: druid, locking
> Attachments: HIVE-19490.2.patch, HIVE-19490.3.patch,
> HIVE-19490.4.patch, HIVE-19490.5.patch, HIVE-19490.6.patch, HIVE-19490.patch
>
>
> Current state of the art:
> Managed non native table like Druid Tables, will need to get a Lock on Insert
> into or insert Over write. The nature of this lock is set to Exclusive by
> default for any non native table.
> This implies that Inserts into Druid table will Lock any read query as well
> during the execution of the insert into. IMO this lock (on insert into) is
> not needed since the insert statement is appending data and the state of
> loading it is managed partially by Hive Storage handler hook and part of it
> by Druid.
> What i am proposing is to relax the lock level to shared for all non native
> tables on insert into operations and keep it as Exclusive Write for insert
> Overwrite for now.
>
> Any feedback is welcome.
> cc [~ekoifman] / [~ashutoshc] / [~jdere] / [~hagleitn]
> Also am not sure what is the best way to unit test this currently am using
> debugger to check if locks are what i except, please let me know if there is
> a better way to do this.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-12342) Set default value of hive.optimize.index.filter to true
[
https://issues.apache.org/jira/browse/HIVE-12342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477946#comment-16477946
]
Hive QA commented on HIVE-12342:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923711/HIVE-12342.14.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10994/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10994/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10994/
Messages:
{noformat}
This message was trimmed, see log for full details
error: a/ql/src/test/results/clientpositive/spark/union22.q.out: does not exist
in index
error: a/ql/src/test/results/clientpositive/spark/union24.q.out: does not exist
in index
error: a/ql/src/test/results/clientpositive/spark/union25.q.out: does not exist
in index
error: a/ql/src/test/results/clientpositive/spark/union26.q.out: does not exist
in index
error: a/ql/src/test/results/clientpositive/spark/union27.q.out: does not exist
in index
error: a/ql/src/test/results/clientpositive/spark/union32.q.out: does not exist
in index
error: a/ql/src/test/results/clientpositive/spark/union33.q.out: does not exist
in index
error: a/ql/src/test/results/clientpositive/spark/union34.q.out: does not exist
in index
error: a/ql/src/test/results/clientpositive/spark/union7.q.out: does not exist
in index
error: a/ql/src/test/results/clientpositive/spark/union_lateralview.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/union_null.q.out: does not
exist in index
error: a/ql/src/test/results/clientpositive/spark/union_ppr.q.out: does not
exist in index
error: a/ql/src/test/results/clientpositive/spark/union_remove_12.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/union_remove_13.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/union_remove_14.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/union_remove_19.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/union_remove_23.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/union_remove_25.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/union_top_level.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/union_view.q.out: does not
exist in index
error: a/ql/src/test/results/clientpositive/spark/vector_between_in.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/vector_decimal_mapjoin.q.out:
does not exist in index
error: a/ql/src/test/results/clientpositive/spark/vector_elt.q.out: does not
exist in index
error: a/ql/src/test/results/clientpositive/spark/vector_inner_join.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out:
does not exist in index
error: a/ql/src/test/results/clientpositive/spark/vectorization_0.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/vectorization_1.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/vectorization_10.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/vectorization_11.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/vectorization_12.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/vectorization_13.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/vectorization_14.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/vectorization_15.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/vectorization_16.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/vectorization_17.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/vectorization_2.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/vectorization_3.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/vectorization_4.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/vectorization_5.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/vectorization_6.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/spark/vectorization_9.q.out: does
not exist in index
error:
a/ql/src/test/results/clientpositive/spark/vectorization_decimal_date.q.out:
does not exist in index
error: a/ql/src/test/results/clientpositive/spark/vectorization_div0.q.out:
does not exist in index
error:
a/ql/src/test/results/clientpositive/spark/vectorization_input_format_exc
[jira] [Updated] (HIVE-19421) Upgrade version of Jetty to 9.3.20.v20170531
[ https://issues.apache.org/jira/browse/HIVE-19421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-19421: --- Attachment: HIVE-19421.4.patch > Upgrade version of Jetty to 9.3.20.v20170531 > > > Key: HIVE-19421 > URL: https://issues.apache.org/jira/browse/HIVE-19421 > Project: Hive > Issue Type: Improvement >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-19421.1.patch, HIVE-19421.2.patch, > HIVE-19421.3.patch, HIVE-19421.3.patch, HIVE-19421.3.patch, > HIVE-19421.4.patch, HIVE-19421.4.patch > > > Move Jetty up to 9.3.20.v20170531 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19317) Handle schema evolution from int like types to decimal
[ https://issues.apache.org/jira/browse/HIVE-19317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477953#comment-16477953 ] Janaki Lahorani commented on HIVE-19317: The test failures from HIVE-19317.5.patch are not related to this patch. > Handle schema evolution from int like types to decimal > -- > > Key: HIVE-19317 > URL: https://issues.apache.org/jira/browse/HIVE-19317 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-19317.1.patch, HIVE-19317.2.patch, > HIVE-19317.3.patch, HIVE-19317.4.patch, HIVE-19317.5.patch > > > If int like type is changed to decimal on parquet data, select results in > errors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19572) Add option to mask stats and data size in q files
[
https://issues.apache.org/jira/browse/HIVE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477961#comment-16477961
]
Hive QA commented on HIVE-19572:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 13s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-10995/patches/PreCommit-HIVE-Build-10995.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10995/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add option to mask stats and data size in q files
> -
>
> Key: HIVE-19572
> URL: https://issues.apache.org/jira/browse/HIVE-19572
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Affects Versions: 3.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19572.01.patch
>
>
> Some tests are flaky because of minimal data size differences, e.g., one
> byte. However, many times we do not actually care about these differences.
> One example is {{default_constraint.q}}.
> Patch adds the possibility to mask 1) printing of stats selectively on q
> files by adding the {{-- MASK_STATS}} option, and 2) printing of data size
> stats selectively on q files by adding the {{-- MASK_DATA_SIZE}} option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-12342) Set default value of hive.optimize.index.filter to true
[ https://issues.apache.org/jira/browse/HIVE-12342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Kryvenko updated HIVE-12342: - Attachment: HIVE-12342.15.patch > Set default value of hive.optimize.index.filter to true > --- > > Key: HIVE-12342 > URL: https://issues.apache.org/jira/browse/HIVE-12342 > Project: Hive > Issue Type: Task > Components: Configuration >Reporter: Ashutosh Chauhan >Assignee: Igor Kryvenko >Priority: Major > Attachments: HIVE-12342.05.patch, HIVE-12342.06.patch, > HIVE-12342.07.patch, HIVE-12342.08.patch, HIVE-12342.09.patch, > HIVE-12342.1.patch, HIVE-12342.10.patch, HIVE-12342.11.patch, > HIVE-12342.12.patch, HIVE-12342.13.patch, HIVE-12342.14.patch, > HIVE-12342.15.patch, HIVE-12342.2.patch, HIVE-12342.3.patch, > HIVE-12342.4.patch, HIVE-12342.patch > > > This configuration governs ppd for storage layer. When applicable, it will > always help. It should be on by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477971#comment-16477971 ] Jason Dere commented on HIVE-19308: --- Can you create an RB link? > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477982#comment-16477982 ] Eric Wohlstadter commented on HIVE-19308: - https://reviews.apache.org/r/67159/ > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477994#comment-16477994 ] Steve Yeom commented on HIVE-19417: --- While I was talking with Eugene, I realized UPD_PARTTXNS table is not needed. I already made relevant changes for the other patch but will generate a new one for this jira. > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-19417.01.patch, HIVE-19417.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19500) Prevent multiple selectivity estimations for the same variable in conjuctions
[
https://issues.apache.org/jira/browse/HIVE-19500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16478001#comment-16478001
]
Nita Dembla commented on HIVE-19500:
I didn't have a good pass rate on TPCDS with 2nd patch here. Though the change
fixes query74 but causes OOM's in other queries.
> Prevent multiple selectivity estimations for the same variable in conjuctions
> -
>
> Key: HIVE-19500
> URL: https://issues.apache.org/jira/browse/HIVE-19500
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19500.01.patch, HIVE-19500.02.patch
>
>
> see HIVE-19097 for problem description
> for filters like: {{(d_year in (2001,2002) and d_year = 2001)}} the current
> estimation is around {{(1/NDV)**2}} (iff column stats are available)
> actually the source of the problem was a small typo in HIVE-17465
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-19575) TestAutoPurgeTables seems flaky
[ https://issues.apache.org/jira/browse/HIVE-19575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez resolved HIVE-19575. Resolution: Fixed Fix Version/s: 3.1.0 Pushed to master, thanks [~prasanth_j] > TestAutoPurgeTables seems flaky > --- > > Key: HIVE-19575 > URL: https://issues.apache.org/jira/browse/HIVE-19575 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19575.1.patch > > > I cannot reproduce the flakiness locally. Maybe we can retry this flaky test > using RetryTestRunner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19572) Add option to mask stats and data size in q files
[
https://issues.apache.org/jira/browse/HIVE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19572:
---
Resolution: Fixed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
Pushed to master, thanks for reviewing [~prasanth_j]
> Add option to mask stats and data size in q files
> -
>
> Key: HIVE-19572
> URL: https://issues.apache.org/jira/browse/HIVE-19572
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Affects Versions: 3.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19572.01.patch
>
>
> Some tests are flaky because of minimal data size differences, e.g., one
> byte. However, many times we do not actually care about these differences.
> One example is {{default_constraint.q}}.
> Patch adds the possibility to mask 1) printing of stats selectively on q
> files by adding the {{-- MASK_STATS}} option, and 2) printing of data size
> stats selectively on q files by adding the {{-- MASK_DATA_SIZE}} option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19572) Add option to mask stats and data size in q files
[
https://issues.apache.org/jira/browse/HIVE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16478070#comment-16478070
]
Hive QA commented on HIVE-19572:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923742/HIVE-19572.01.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 14397 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,scriptfile1.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q]
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.testCliDriver[spark_stage_max_tasks]
(batchId=255)
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.testCliDriver[spark_task_failure]
(batchId=255)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query39]
(batchId=255)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testExternalNoAutoPurge
(batchId=233)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10995/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10995/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10995/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923742 - PreCommit-HIVE-Build
> Add option to mask stats and data size in q files
> -
>
> Key: HIVE-19572
> URL: https://issues.apache.org/jira/browse/HIVE-19572
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Affects Versions: 3.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19572.01.patch
>
>
> Some tests are flaky because of minimal data size differences, e.g., one
> byte. However, many times we do not actually care about these differences.
> One example is {{default_constraint.q}}.
> Patch adds the possibility to mask 1) printing of stats selectively on q
> files by adding the {{-- MASK_STATS}} option, and 2) printing of data size
> stats selectively on q files by adding the {{-- MASK_DATA_SIZE}} option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19424) NPE In MetaDataFormatters
[
https://issues.apache.org/jira/browse/HIVE-19424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-19424:
Resolution: Fixed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks [~afan] for the work.
> NPE In MetaDataFormatters
> -
>
> Key: HIVE-19424
> URL: https://issues.apache.org/jira/browse/HIVE-19424
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Metastore, Standalone Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: BELUGA BEHR
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HIVE-19424.1.patch, HIVE-19424.2.patch
>
>
> h2. Overview
> According to the Hive Schema definition, a table's {{INPUT_FORMAT}} class can
> be set to NULL. However, there are places in the code where we do not
> account for this NULL value, in particular the {{MetaDataFormatters}} classes
> {{TextMetaDataFormatter}} and {{JsonMetaDataFormatter}}. In addition, there
> is no debug level logging in the {{MetaDataFormatters}} classes to tell me
> which table in particular is causing the problem.
> {code:sql|title=hive-schema-2.2.0.mysql.sql}
> CREATE TABLE IF NOT EXISTS `SDS` (
> `SD_ID` bigint(20) NOT NULL,
> `CD_ID` bigint(20) DEFAULT NULL,
> `INPUT_FORMAT` varchar(4000) CHARACTER SET latin1 COLLATE latin1_bin
> DEFAULT NULL,
> `IS_COMPRESSED` bit(1) NOT NULL,
> ...
> {code}
> {code:java|title=TextMetaDataFormatter.java}
> // Not checking for a null return from getInputFormatClass
> inputFormattCls = par.getInputFormatClass().getName();
> outputFormattCls = par.getOutputFormatClass().getName();
> {code}
> h2. Reproduction
> {code:sql}
> -- MySQL Backend
> update SDS SET INPUT_FORMAT=NULL WHERE SD_ID=XXX;
> {code}
> {code}
> // Hive
> SHOW TABLE EXTENDED FROM default LIKE '*';
> // HS2 Logs
> [HiveServer2-Background-Pool: Thread-464]: Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. Exception while processing show table
> status
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:400)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Exception while
> processing show table status
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.showTableStatus(DDLTask.java:3025)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:405)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> ... 11 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.metadata.formatting.TextMetaDataFormatter.showTableStatus(TextMetaDataFormatter.java:202)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.showTableStatus(DDLTask.java:3020)
> ... 20 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19575) TestAutoPurgeTables seems flaky
[ https://issues.apache.org/jira/browse/HIVE-19575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16478099#comment-16478099 ] Jesus Camacho Rodriguez commented on HIVE-19575: Pushed an addendum to try the retry, since it was not being triggered correctly because of the @Before / @After annotations. If that does not work either, I will proceed to disable the flaky tests and add it to the list in HIVE-19509. > TestAutoPurgeTables seems flaky > --- > > Key: HIVE-19575 > URL: https://issues.apache.org/jira/browse/HIVE-19575 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19575.1.patch > > > I cannot reproduce the flakiness locally. Maybe we can retry this flaky test > using RetryTestRunner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18748) Rename table impacts the ACID behaviour as table names are not updated in meta-tables.
[
https://issues.apache.org/jira/browse/HIVE-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16478116#comment-16478116
]
Hive QA commented on HIVE-18748:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
58s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
56s{color} | {color:blue} standalone-metastore in master has 215 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 3 new + 2 unchanged - 0 fixed
= 5 total (was 2) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
21s{color} | {color:red} standalone-metastore: The patch generated 3 new + 620
unchanged - 0 fixed = 623 total (was 620) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
3s{color} | {color:red} standalone-metastore generated 1 new + 215 unchanged -
0 fixed = 216 total (was 215) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore |
| | org.apache.hadoop.hive.metastore.txn.TxnHandler.onRename(String, String,
String, String, String, String, String, String) passes a nonconstant String to
an execute or addBatch method on an SQL statement At TxnHandler.java:String,
String, String) passes a nonconstant String to an execute or addBatch method on
an SQL statement At TxnHandler.java:[line 2918] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10996/dev-support/hive-personality.sh
|
| git revision | master / 3d83467 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10996/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10996/yetus/diff-checkstyle-standalone-metastore.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10996/yetus/new-findbugs-standalone-metastore.html
|
| modules | C: ql standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10996/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Rename table impacts the ACID behaviour as table names are not updated in
> meta-tables.
> ---
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Status: Patch Available (was: Open) > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Status: Open (was: Patch Available) > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Attachment: HIVE-19308.3.patch > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18748) Rename table impacts the ACID behaviour as table names are not updated in meta-tables.
[
https://issues.apache.org/jira/browse/HIVE-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16478159#comment-16478159
]
Hive QA commented on HIVE-18748:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923744/HIVE-18748.05.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14408 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10996/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10996/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10996/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923744 - PreCommit-HIVE-Build
> Rename table impacts the ACID behaviour as table names are not updated in
> meta-tables.
> --
>
> Key: HIVE-18748
> URL: https://issues.apache.org/jira/browse/HIVE-18748
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Eugene Koifman
>Priority: Critical
> Labels: ACID, DDL
> Attachments: HIVE-18748.02.patch, HIVE-18748.03.patch,
> HIVE-18748.04.patch, HIVE-18748.05.patch
>
>
> ACID implementation uses metatables such as TXN_COMPONENTS,
> COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPCTION_QUEUE etc to
> manage ACID operations.
> Per table write ID implementation (HIVE-18192) introduces couple of
> metatables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID to manage write ids
> allocated per table.
> Now, when we rename any tables, it is necessary to update the corresponding
> table names in these metatables as well. Otherwise, ACID table operations
> won't work properly.
> Since, this change is significant and have other side-effects, we propose to
> disable rename tables on ACID tables until a fix is figured out.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[ https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-18079: Assignee: Prasanth Jayachandran (was: Gopal V) > Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator > bit-size > > > Key: HIVE-18079 > URL: https://issues.apache.org/jira/browse/HIVE-18079 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Statistics >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18079-branch-3.patch, HIVE-18079.1.patch, > HIVE-18079.10.patch, HIVE-18079.11.patch, HIVE-18079.12.patch, > HIVE-18079.13.patch, HIVE-18079.14.patch, HIVE-18079.15.patch, > HIVE-18079.2.patch, HIVE-18079.4.patch, HIVE-18079.5.patch, > HIVE-18079.6.patch, HIVE-18079.7.patch, HIVE-18079.8.patch, HIVE-18079.9.patch > > > HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its > mathematical hash distribution & construction. > Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second > scan over the data-set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[ https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18079: - Attachment: HIVE-18079-branch-3.patch > Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator > bit-size > > > Key: HIVE-18079 > URL: https://issues.apache.org/jira/browse/HIVE-18079 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Statistics >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18079-branch-3.patch, HIVE-18079.1.patch, > HIVE-18079.10.patch, HIVE-18079.11.patch, HIVE-18079.12.patch, > HIVE-18079.13.patch, HIVE-18079.14.patch, HIVE-18079.15.patch, > HIVE-18079.2.patch, HIVE-18079.4.patch, HIVE-18079.5.patch, > HIVE-18079.6.patch, HIVE-18079.7.patch, HIVE-18079.8.patch, HIVE-18079.9.patch > > > HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its > mathematical hash distribution & construction. > Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second > scan over the data-set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
