[jira] [Updated] (HIVE-19853) Arrow serializer needs to create a TimeStampMicroTZVector instead of TimeStampMicroVector
[ https://issues.apache.org/jira/browse/HIVE-19853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-19853: -- Attachment: HIVE-19853.1.patch > Arrow serializer needs to create a TimeStampMicroTZVector instead of > TimeStampMicroVector > - > > Key: HIVE-19853 > URL: https://issues.apache.org/jira/browse/HIVE-19853 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-19853.1.patch > > > HIVE-19723 changed nanosecond to microsecond in Arrow serialization. However, > it needs to be microsecond with time zone. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19830) Inconsistent behavior when multiple partitions point to the same location
[ https://issues.apache.org/jira/browse/HIVE-19830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507843#comment-16507843 ] Gabor Kaszab commented on HIVE-19830: - I have heard about users taking advantage of having multiple partitions pointing to the same location. For example with a table that is partitioned by date it is common to create an extra partition called 'latest' and point it to another partition's location and change the location once new partitions are introduced. I feel that considering this pattern is used in production it should be guaranteed that the queries on these partition return consistent results. i.e. the listing of partitions shouldn't show a partition that actually doesn't exist anymore. > Inconsistent behavior when multiple partitions point to the same location > - > > Key: HIVE-19830 > URL: https://issues.apache.org/jira/browse/HIVE-19830 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.4.0 >Reporter: Gabor Kaszab >Assignee: Adam Szita >Priority: Major > > // create a table with 2 partitions where both partitions share the same > location and inserting a single line to one of them. > create table test (i int) partitioned by (j int) stored as parquet; > alter table test add partition (j=1) location > 'hdfs://localhost:20500/test-warehouse/test/j=1'; > alter table test add partition (j=2) location > 'hdfs://localhost:20500/test-warehouse/test/j=1'; > insert into table test partition (j=1) values (1); > // select * show this single line in both partitions as expected. > select * from test; > 1 1 > 1 2 > // however, sum() doesn't add up the line for all the partitions. This is > +Issue #1+. > select sum( i), sum(j) from test; > 1 2 > // On the file system there is a common dir for the 2 partitions that is > expected. > hdfs dfs -ls hdfs://localhost:20500/test-warehouse/test/ > Found 1 items > drwxr-xr-x - gaborkaszab supergroup 0 2018-06-08 10:54 > hdfs://localhost:20500/test-warehouse/test/j=1 > // Let's drop one of the partitions now! > alter table test drop partition (j=2); > // running the same hdfs dfs -ls command shows that the j=1 directory is > dropped. I think this is a good behavior, we just have to document that this > is the expected case. > // select * from test; returns zero rows, this is still as expected. > // Even though the dir is dropped j=1 partition is still visible with show > partitions. This is +Issue #2+. > show partitions test; > j=1 > After dropping the directory with Hive, when Impala reloads it's partitions > it asks Hive to tell what are the existing partitions. Apparently, Hive sends > down a list with j=1 partition included and then Impala takes it as an > existing one and doesn't drop it from Catalog's cache. Here Hive shouldn't > send that partition down. This is +Issue #3+. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19584) Dictionary encoding for string types
[ https://issues.apache.org/jira/browse/HIVE-19584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-19584: -- Attachment: HIVE-19584.5.patch > Dictionary encoding for string types > > > Key: HIVE-19584 > URL: https://issues.apache.org/jira/browse/HIVE-19584 > Project: Hive > Issue Type: Sub-task >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-19584.1.patch, HIVE-19584.2.patch, > HIVE-19584.3.patch, HIVE-19584.4.patch, HIVE-19584.5.patch > > > Apache Arrow supports dictionary encoding for some data types. So implement > dictionary encoding for string types in Arrow SerDe. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15976) Support CURRENT_CATALOG and CURRENT_SCHEMA
[ https://issues.apache.org/jira/browse/HIVE-15976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507842#comment-16507842 ] Laszlo Bodor commented on HIVE-15976: - [~kgyrtkirk]: tests passed with 02.patch, could you please take a look at that, if it's ready to push or not? > Support CURRENT_CATALOG and CURRENT_SCHEMA > -- > > Key: HIVE-15976 > URL: https://issues.apache.org/jira/browse/HIVE-15976 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Carter Shanklin >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-15976.01.patch, HIVE-15976.02.patch > > > Support these keywords for querying the current catalog and schema. SQL > reference: section 6.4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19771) allowNullColumnForMissingStats should not be false when column stats are estimated
[
https://issues.apache.org/jira/browse/HIVE-19771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507834#comment-16507834
]
Hive QA commented on HIVE-19771:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12927234/HIVE-19771.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14517 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_llap]
(batchId=165)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testQueueingWithThreads
(batchId=276)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11703/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11703/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11703/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12927234 - PreCommit-HIVE-Build
> allowNullColumnForMissingStats should not be false when column stats are
> estimated
> --
>
> Key: HIVE-19771
> URL: https://issues.apache.org/jira/browse/HIVE-19771
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19771.01.patch, HIVE-19771.02.patch,
> HIVE-19771.03.patch, HIVE-19771.patch
>
>
> Otherwise we may throw an Exception.
> {noformat}
> 2018-05-26T00:30:22,335 DEBUG [HiveServer2-Background-Pool: Thread-631]:
> stats.StatsUtils (:()) - Estimated average row size: 372
> 2018-05-26T00:30:22,352 DEBUG [HiveServer2-Background-Pool: Thread-631]:
> calcite.RelOptHiveTable (:()) - Stats for column a in table basetable_rebuild
> stored in cache
> 2018-05-26T00:30:22,352 DEBUG [HiveServer2-Background-Pool: Thread-631]:
> calcite.RelOptHiveTable (:()) - colName: a colType: int countDistincts: 4
> numNulls: 1 avgColLen: 4.0 numTrues: 0 numFalses: 0 Range: [ min:
> -9223372036854775808 max: 9223372036854775807 ] isPrimaryKey: false
> isEstimated: true
> 2018-05-26T00:30:22,352 DEBUG [HiveServer2-Background-Pool: Thread-631]:
> calcite.RelOptHiveTable (:()) - Stats for column b in table basetable_rebuild
> stored in cache
> 2018-05-26T00:30:22,352 DEBUG [HiveServer2-Background-Pool: Thread-631]:
> calcite.RelOptHiveTable (:()) - colName: b colType: varchar(256)
> countDistincts: 4 numNulls: 1 avgColLen: 256.0 numTrues: 0 numFalses: 0
> isPrimaryKey: false isEstimated: true
> 2018-05-26T00:30:22,352 ERROR [HiveServer2-Background-Pool: Thread-631]:
> calcite.RelOptHiveTable (:()) - No Stats for default@basetable_rebuild,
> Columns: a, b
> java.lang.RuntimeException: No Stats for default@basetable_rebuild, Columns:
> a, b
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.updateColStats(RelOptHiveTable.java:586)
> ~[hive-exec-3.0.0.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:606)
> ~[hive-exec-3.0.0.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:592)
> ~[hive-exec-3.0.0.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan.getColStat(HiveTableScan.java:155)
> ~[hive-exec-3.0.0.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdDistinctRowCount.getDistinctRowCount(HiveRelMdDistinctRowCount.java:78)
> ~[hive-exec-3.0.0.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdDistinctRowCount.getDistinctRowCount(HiveRelMdDistinctRowCount.java:65)
> ~[hive-exec-3.0.0.jar:3.0.0-SNAPSHOT]
> at
> GeneratedMetadataHandler_DistinctRowCount.getDistinctRowCount_$(Unknown
> Source) ~[?:?]
> at
> GeneratedMetadataHandler_DistinctRowCount.getDistinctRowCount(Unknown Source)
> ~[?:?]
> at
> org.apache.calcite.rel.metadata.RelMetadataQuery.getDistinctRowCount(RelMetadataQuery.java:781)
> ~[calcite-core-1.16.0.jar:1.16.0]
> at
> org.apache.calcite.rel.metadata.RelMdRowCount.getRowCount(RelMdRowCount.java:207)
> ~[calcite-core-1.16.0.jar:1.16.0]
> at GeneratedMetadataHandler_RowCount.getRowCount_$(Unknown Source)
[jira] [Commented] (HIVE-19771) allowNullColumnForMissingStats should not be false when column stats are estimated
[
https://issues.apache.org/jira/browse/HIVE-19771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507812#comment-16507812
]
Hive QA commented on HIVE-19771:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
30s{color} | {color:blue} ql in master has 2279 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
34s{color} | {color:red} ql: The patch generated 2 new + 34 unchanged - 1 fixed
= 36 total (was 35) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11703/dev-support/hive-personality.sh
|
| git revision | master / f37fef0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11703/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11703/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> allowNullColumnForMissingStats should not be false when column stats are
> estimated
> --
>
> Key: HIVE-19771
> URL: https://issues.apache.org/jira/browse/HIVE-19771
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19771.01.patch, HIVE-19771.02.patch,
> HIVE-19771.03.patch, HIVE-19771.patch
>
>
> Otherwise we may throw an Exception.
> {noformat}
> 2018-05-26T00:30:22,335 DEBUG [HiveServer2-Background-Pool: Thread-631]:
> stats.StatsUtils (:()) - Estimated average row size: 372
> 2018-05-26T00:30:22,352 DEBUG [HiveServer2-Background-Pool: Thread-631]:
> calcite.RelOptHiveTable (:()) - Stats for column a in table basetable_rebuild
> stored in cache
> 2018-05-26T00:30:22,352 DEBUG [HiveServer2-Background-Pool: Thread-631]:
> calcite.RelOptHiveTable (:()) - colName: a colType: int countDistincts: 4
> numNulls: 1 avgColLen: 4.0 numTrues: 0 numFalses: 0 Range: [ min:
> -9223372036854775808 max: 9223372036854775807 ] isPrimaryKey: false
> isEstimated: true
> 2018-05-26T00:30:22,352 DEBUG [HiveServer2-Background-Pool: Thread-631]:
> calcite.RelOptHiveTable (:()) - Stats for column b in table basetable_rebuild
> stored in cache
> 2018-05-26T00:30:22,352 DEBUG [HiveServer2-Background-Pool: Thread-631]:
>
[jira] [Commented] (HIVE-19855) TestStatsUpdaterThread.testQueueingWithThreads fails often
[ https://issues.apache.org/jira/browse/HIVE-19855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507809#comment-16507809 ] Peter Vary commented on HIVE-19855: --- What should we do with this [~sershe]? Increase the timeout? Rerun the test? Or this is an actual concurrency problem? Thanks, Peter > TestStatsUpdaterThread.testQueueingWithThreads fails often > -- > > Key: HIVE-19855 > URL: https://issues.apache.org/jira/browse/HIVE-19855 > Project: Hive > Issue Type: Test >Reporter: Peter Vary >Priority: Major > > Taking a look at here, it seems, that > TestStatsUpdaterThread.testQueueingWithThreads fails on almost every second > run: > [https://builds.apache.org/job/PreCommit-HIVE-Build/11698/testReport/junit/org.apache.hadoop.hive.ql.stats/TestStatsUpdaterThread/testQueueingWithThreads/history/] > > We should fix this -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19835) Flaky test: TestWorkloadManager.testAsyncSessionInitFailures
[
https://issues.apache.org/jira/browse/HIVE-19835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507803#comment-16507803
]
Peter Vary commented on HIVE-19835:
---
HIVE-19816 added retry to the tests.
Left this open, so permanent fix could solve the test problem
> Flaky test: TestWorkloadManager.testAsyncSessionInitFailures
>
>
> Key: HIVE-19835
> URL: https://issues.apache.org/jira/browse/HIVE-19835
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Affects Versions: 4.0.0
>Reporter: Aihua Xu
>Priority: Major
>
> Sometimes this test fails with the following issue. Seems it's a flaky test.
> {noformat}
> Error Message
> expected:<0> but was:<1>
> Stacktrace
> java.lang.AssertionError: expected:<0> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testAsyncSessionInitFailures(TestWorkloadManager.java:1138)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19816) TestWorkloadManager.testAsyncSessionInitFailures is flaky
[
https://issues.apache.org/jira/browse/HIVE-19816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-19816:
--
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master.
Thanks for the review [~sershe]!
Left open HIVE-19835 for permanent fix
> TestWorkloadManager.testAsyncSessionInitFailures is flaky
> -
>
> Key: HIVE-19816
> URL: https://issues.apache.org/jira/browse/HIVE-19816
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19816.patch
>
>
> This one seems flaky:
> [https://builds.apache.org/job/PreCommit-HIVE-Build/11551/testReport/org.apache.hadoop.hive.ql.exec.tez/TestWorkloadManager/testAsyncSessionInitFailures/history/]
> Let's add
> {code:java}
> @RunWith(RetryTestRunner.class){code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19823) BytesBytesMultiHashMap estimation should account for loadFactor
[
https://issues.apache.org/jira/browse/HIVE-19823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507801#comment-16507801
]
Zoltan Haindrich commented on HIVE-19823:
-
oh; I've just found it :) but that calculation is somehow ended up being
incorrect:
{code}
threshold = (int)Math.ceil(keyCount / (keyCountAdj * loadFactor));
{code}
but fixing it there would not make this problem entirely go away - I think the
sizing strategy belongs to the datastracture
> BytesBytesMultiHashMap estimation should account for loadFactor
> ---
>
> Key: HIVE-19823
> URL: https://issues.apache.org/jira/browse/HIVE-19823
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19823.01.patch
>
>
> it could happen that the capacity is known beforehand; and the estimated size
> of the hashtable is accurate. but still; because after some time the element
> count violates loadfactor ratio a rehash will occur.
> this by default could happen with a {{1-loadfactor = 25%}} probability
> this rehashing takes around 2 seconds on my system for 6.5M entries
> https://github.com/apache/hive/blob/cfd57348c1ac188e0ba131d5636a62ff7b7c27be/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java#L176-L187
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-12192) Hive should carry out timestamp computations in UTC
[
https://issues.apache.org/jira/browse/HIVE-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507793#comment-16507793
]
Hive QA commented on HIVE-12192:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12927229/HIVE-12192.15.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11702/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11702/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11702/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-06-11 08:09:49.905
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-11702/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-06-11 08:09:49.908
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at f37fef0 HIVE-16505: Support "unknown" boolean truth value
(Laszlo Bodor via Zoltan Haindrich)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at f37fef0 HIVE-16505: Support "unknown" boolean truth value
(Laszlo Bodor via Zoltan Haindrich)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-06-11 08:09:51.203
+ rm -rf ../yetus_PreCommit-HIVE-Build-11702
+ mkdir ../yetus_PreCommit-HIVE-Build-11702
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-11702
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-11702/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
fatal: git diff header lacks filename information when removing 0 leading
pathname components (line 1070)
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query5.q.out:326
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query5.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query77.q.out:279
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query77.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query80.q.out:295
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query80.q.out'
with conflicts.
Going to apply patch with: git apply -p1
/data/hiveptest/working/scratch/build.patch:328: trailing whitespace.
/data/hiveptest/working/scratch/build.patch:40430: trailing whitespace.
min -28830
/data/hiveptest/working/scratch/build.patch:40431: trailing whitespace.
max -28769
/data/hiveptest/working/scratch/build.patch:40434: trailing whitespace.
distinct_count 35
/data/hiveptest/working/scratch/build.patch:41041: trailing whitespace.
totalSize 295599
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query5.q.out:326
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query5.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query77.q.out:279
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query77.q.out'
with conflicts.
error: patch failed:
[jira] [Commented] (HIVE-19847) Create Separate getInputSummary Service
[
https://issues.apache.org/jira/browse/HIVE-19847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507792#comment-16507792
]
Hive QA commented on HIVE-19847:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12927220/HIVE-19847.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 14515 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_llap]
(batchId=165)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitions
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerVertexRawInputSplitsNoKill
(batchId=242)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11701/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11701/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11701/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12927220 - PreCommit-HIVE-Build
> Create Separate getInputSummary Service
> ---
>
> Key: HIVE-19847
> URL: https://issues.apache.org/jira/browse/HIVE-19847
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-19847.1.patch
>
>
> The Hive {{org.apache.hadoop.hive.ql.exec.Utilities.java}} file has taken on
> a life of its own. We should consider separating out the various components
> into their own classes. For this ticket, I propose separating out the
> {{getInputSummary}} functionality into its own class.
> There are several issues with the current implementation:
> # It is
> [synchronized|https://github.com/apache/hive/blob/f27c38ff55902827499192a4f8cf8ed37d6fd967/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2383].
> Only one query can get file input summary at a time. For a query which
> deals with a large data set with a large number of files, this can block
> other queries for a long period of time. This is especially painful when
> most queries use a small data set, but a large data set is submitted on
> occasion.
> # For each query, time is spend setting up and tearing down a ThreadPool
> # It uses deprecated code
> I propose breaking it out into its own class and creating a single thread
> pool that all queries pull from. In this way, the bottle neck will be one
> the number of available threads, not on a single query and if a big query is
> running and a small query is also submitted, the smaller query will be able
> to proceed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (HIVE-19835) Flaky test: TestWorkloadManager.testAsyncSessionInitFailures
[
https://issues.apache.org/jira/browse/HIVE-19835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary reopened HIVE-19835:
---
> Flaky test: TestWorkloadManager.testAsyncSessionInitFailures
>
>
> Key: HIVE-19835
> URL: https://issues.apache.org/jira/browse/HIVE-19835
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Affects Versions: 4.0.0
>Reporter: Aihua Xu
>Priority: Major
>
> Sometimes this test fails with the following issue. Seems it's a flaky test.
> {noformat}
> Error Message
> expected:<0> but was:<1>
> Stacktrace
> java.lang.AssertionError: expected:<0> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testAsyncSessionInitFailures(TestWorkloadManager.java:1138)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-19835) Flaky test: TestWorkloadManager.testAsyncSessionInitFailures
[
https://issues.apache.org/jira/browse/HIVE-19835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary resolved HIVE-19835.
---
Resolution: Duplicate
> Flaky test: TestWorkloadManager.testAsyncSessionInitFailures
>
>
> Key: HIVE-19835
> URL: https://issues.apache.org/jira/browse/HIVE-19835
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Affects Versions: 4.0.0
>Reporter: Aihua Xu
>Priority: Major
>
> Sometimes this test fails with the following issue. Seems it's a flaky test.
> {noformat}
> Error Message
> expected:<0> but was:<1>
> Stacktrace
> java.lang.AssertionError: expected:<0> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testAsyncSessionInitFailures(TestWorkloadManager.java:1138)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19723) Arrow serde: "Unsupported data type: Timestamp(NANOSECOND, null)"
[ https://issues.apache.org/jira/browse/HIVE-19723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507763#comment-16507763 ] Teddy Choi commented on HIVE-19723: --- [~ewohlstadter], I created HIVE-19853. > Arrow serde: "Unsupported data type: Timestamp(NANOSECOND, null)" > - > > Key: HIVE-19723 > URL: https://issues.apache.org/jira/browse/HIVE-19723 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-19723.1.patch, HIVE-19723.3.patch, > HIVE-19723.4.patch, HIVE-19732.2.patch > > > Spark's Arrow support only provides Timestamp at MICROSECOND granularity. > Spark 2.3.0 won't accept NANOSECOND. Switch it back to MICROSECOND. > The unit test org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow will just need > to change the assertion to test microsecond. And we'll need to add this to > documentation on supported datatypes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19847) Create Separate getInputSummary Service
[
https://issues.apache.org/jira/browse/HIVE-19847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507754#comment-16507754
]
Hive QA commented on HIVE-19847:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
24s{color} | {color:blue} ql in master has 2279 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} common: The patch generated 1 new + 428 unchanged - 0
fixed = 429 total (was 428) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
34s{color} | {color:red} ql: The patch generated 19 new + 154 unchanged - 28
fixed = 173 total (was 182) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
37s{color} | {color:green} ql generated 0 new + 2276 unchanged - 3 fixed = 2276
total (was 2279) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11701/dev-support/hive-personality.sh
|
| git revision | master / f37fef0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11701/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11701/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11701/yetus/patch-asflicense-problems.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11701/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Create Separate getInputSummary Service
> ---
>
> Key: HIVE-19847
> URL: https://issues.apache.org/jira/browse/HIVE-19847
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-19847.1.patch
>
>
> The Hive
[jira] [Updated] (HIVE-19569) alter table db1.t1 rename db2.t2 generates MetaStoreEventListener.onDropTable()
[
https://issues.apache.org/jira/browse/HIVE-19569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera updated HIVE-19569:
---
Attachment: HIVE-19569.03.patch
> alter table db1.t1 rename db2.t2 generates
> MetaStoreEventListener.onDropTable()
> ---
>
> Key: HIVE-19569
> URL: https://issues.apache.org/jira/browse/HIVE-19569
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Standalone Metastore, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-19569.01.patch, HIVE-19569.02.patch,
> HIVE-19569.03.patch
>
>
> When renaming a table within the same DB, this operation causes
> {{MetaStoreEventListener.onAlterTable()}} to fire but when changing DB name
> for a table it causes {{MetaStoreEventListener.onDropTable()}} +
> {{MetaStoreEventListener.onCreateTable()}}.
> The files from original table are moved to new table location.
> This creates confusing semantics since any logic in {{onDropTable()}} doesn't
> know about the larger context, i.e. that there will be a matching
> {{onCreateTable()}}.
> In particular, this causes a problem for Acid tables since files moved from
> old table use WriteIDs that are not meaningful with the context of new table.
> Current implementation is due to replication. This should ideally be changed
> to raise a "not supported" error for tables that are marked for replication.
> cc [~sankarh]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19783) Retrieve only locations in HiveMetaStore.dropPartitionsAndGetLocations
[
https://issues.apache.org/jira/browse/HIVE-19783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507741#comment-16507741
]
Peter Vary commented on HIVE-19783:
---
Added review board link: [https://reviews.apache.org/r/67485/]
The patch does the following:
* Adds a new getPartitionLocations method to the RawStore interface.
** Implements getPartitionLocations in ObjectStore using JDQL.
** Implements getPartitionLocations in CachedObjectStore by calling
rawStore.getPartitionLocations (so big tables with many partitions will not
cause problem even if they cannot fit into memory)
* Modifies dropPartitionsAndGetLocations:
** Instead of querying every partition data. Query only the locations using
the new interface method
** Removes partKeys parameter which become unnecessary
[~vihangk1]: Could you please review?
Thanks,
Peter
> Retrieve only locations in HiveMetaStore.dropPartitionsAndGetLocations
> --
>
> Key: HIVE-19783
> URL: https://issues.apache.org/jira/browse/HIVE-19783
> Project: Hive
> Issue Type: Improvement
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-19783.2.patch, HIVE-19783.patch
>
>
> Optimize further the dropTable command.
> Currently {{HiveMetaStore.dropPartitionsAndGetLocations}} retrieves the whole
> partition object, but we need only the locations instead.
> Create a RawStore method to retrieve only the locations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19569) alter table db1.t1 rename db2.t2 generates MetaStoreEventListener.onDropTable()
[
https://issues.apache.org/jira/browse/HIVE-19569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera updated HIVE-19569:
---
Attachment: (was: HIVE-19569.02.patch)
> alter table db1.t1 rename db2.t2 generates
> MetaStoreEventListener.onDropTable()
> ---
>
> Key: HIVE-19569
> URL: https://issues.apache.org/jira/browse/HIVE-19569
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Standalone Metastore, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-19569.01.patch, HIVE-19569.02.patch
>
>
> When renaming a table within the same DB, this operation causes
> {{MetaStoreEventListener.onAlterTable()}} to fire but when changing DB name
> for a table it causes {{MetaStoreEventListener.onDropTable()}} +
> {{MetaStoreEventListener.onCreateTable()}}.
> The files from original table are moved to new table location.
> This creates confusing semantics since any logic in {{onDropTable()}} doesn't
> know about the larger context, i.e. that there will be a matching
> {{onCreateTable()}}.
> In particular, this causes a problem for Acid tables since files moved from
> old table use WriteIDs that are not meaningful with the context of new table.
> Current implementation is due to replication. This should ideally be changed
> to raise a "not supported" error for tables that are marked for replication.
> cc [~sankarh]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19569) alter table db1.t1 rename db2.t2 generates MetaStoreEventListener.onDropTable()
[
https://issues.apache.org/jira/browse/HIVE-19569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera updated HIVE-19569:
---
Attachment: HIVE-19569.02.patch
> alter table db1.t1 rename db2.t2 generates
> MetaStoreEventListener.onDropTable()
> ---
>
> Key: HIVE-19569
> URL: https://issues.apache.org/jira/browse/HIVE-19569
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Standalone Metastore, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-19569.01.patch, HIVE-19569.02.patch,
> HIVE-19569.02.patch
>
>
> When renaming a table within the same DB, this operation causes
> {{MetaStoreEventListener.onAlterTable()}} to fire but when changing DB name
> for a table it causes {{MetaStoreEventListener.onDropTable()}} +
> {{MetaStoreEventListener.onCreateTable()}}.
> The files from original table are moved to new table location.
> This creates confusing semantics since any logic in {{onDropTable()}} doesn't
> know about the larger context, i.e. that there will be a matching
> {{onCreateTable()}}.
> In particular, this causes a problem for Acid tables since files moved from
> old table use WriteIDs that are not meaningful with the context of new table.
> Current implementation is due to replication. This should ideally be changed
> to raise a "not supported" error for tables that are marked for replication.
> cc [~sankarh]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19008) Improve Spark session id logging
[
https://issues.apache.org/jira/browse/HIVE-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507738#comment-16507738
]
Hive QA commented on HIVE-19008:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12927219/HIVE-19008.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14517 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11699/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11699/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11699/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12927219 - PreCommit-HIVE-Build
> Improve Spark session id logging
>
>
> Key: HIVE-19008
> URL: https://issues.apache.org/jira/browse/HIVE-19008
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19008.1.patch, HIVE-19008.2.patch
>
>
> HoS users have two session ids, one id for the Hive session and another id
> for the Spark session, both are UUIDs.
> I think some improvements could be made here:
> The Spark session id could just be a counter that is incremented for each new
> Spark session within a Hive session. Each Spark session is still globally
> identifiable by its associated Hive session id + its own counter. This may
> make more sense since the Hive session - Spark session has a 1-to-many
> relationship, as in a single Hive session can contain multiple Spark
> sessions, and each Spark session must belong to a Hive session.
> Furthermore, we should include both the Hive session id and Spark session id
> in the console logs + the Spark Web UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16505) Support "unknown" boolean truth value
[
https://issues.apache.org/jira/browse/HIVE-16505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507737#comment-16507737
]
Laszlo Bodor commented on HIVE-16505:
-
Thanks for pushing [~kgyrtkirk] !
> Support "unknown" boolean truth value
> -
>
> Key: HIVE-16505
> URL: https://issues.apache.org/jira/browse/HIVE-16505
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-16505.01.patch, HIVE-16505.02.patch,
> HIVE-16505.03.patch, HIVE-16505.04.patch
>
>
> according to the standard, boolean truth value might be:
> {{TRUE|FALSE|UNKNOWN}}.
> similar queries to the following should be supported:
> {code:java}
> select 1 where null is unknown;
> select 1 where (select cast(null as boolean) ) is unknown;
> {code}
> "unknown" behaves similarily to null. {{(null=null) is null}}
>
> "All boolean values and SQL truth values are comparable and all are
> assignable to a site of type boolean. The value True is greater than the
> value False, and any comparison involving the null value or an Unknown truth
> value will return an Unknown result. The values True and False may be
> assigned to any site having a boolean data type; assignment of Unknown, or
> the null value, is subject to the nullability characteristic of the target."
>
> *Truth table for the AND boolean operator*
> AND True False Unknown
> True True False Unknown
> False False False False
> Unknown Unknown False Unknown
> *Truth table for the OR boolean operator*
> OR True False Unknown
> True True True True
> False True False Unknown
> Unknown True Unknown Unknown
> *Truth table for the IS boolean operator*
> IS TRUE FALSE UNKNOWN
> True True False False
> False False True False
> Unknown False False True
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19782) Flash out TestObjectStore.testDirectSQLDropParitionsCleanup
[
https://issues.apache.org/jira/browse/HIVE-19782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-19782:
--
Attachment: HIVE-19782.2.patch
> Flash out TestObjectStore.testDirectSQLDropParitionsCleanup
> ---
>
> Key: HIVE-19782
> URL: https://issues.apache.org/jira/browse/HIVE-19782
> Project: Hive
> Issue Type: Test
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-19782.2.patch, HIVE-19782.patch
>
>
> {{TestObjectStore.testDirectSQLDropParitionsCleanup}} checks that the tables
> are empty after the drop. We should add some rows to every partition related
> table, to see that they are really cleaned up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19829) Incremental replication load should create tasks in execution phase rather than semantic phase
[ https://issues.apache.org/jira/browse/HIVE-19829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-19829: --- Attachment: HIVE-19829.03.patch > Incremental replication load should create tasks in execution phase rather > than semantic phase > -- > > Key: HIVE-19829 > URL: https://issues.apache.org/jira/browse/HIVE-19829 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19829.01.patch, HIVE-19829.02.patch, > HIVE-19829.03.patch > > > Split the incremental load into multiple iterations. In each iteration create > number of tasks equal to the configured value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19853) Arrow serializer needs to create a TimeStampMicroTZVector instead of TimeStampMicroVector
[ https://issues.apache.org/jira/browse/HIVE-19853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi reassigned HIVE-19853: - > Arrow serializer needs to create a TimeStampMicroTZVector instead of > TimeStampMicroVector > - > > Key: HIVE-19853 > URL: https://issues.apache.org/jira/browse/HIVE-19853 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > > HIVE-19723 changed nanosecond to microsecond in Arrow serialization. However, > it needs to be microsecond with time zone. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19237) Only use an operatorId once in a plan
[ https://issues.apache.org/jira/browse/HIVE-19237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507729#comment-16507729 ] Zoltan Haindrich commented on HIVE-19237: - pushed to master. Thank you Ashutosh for reviewing the changes! > Only use an operatorId once in a plan > - > > Key: HIVE-19237 > URL: https://issues.apache.org/jira/browse/HIVE-19237 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-19237.01.patch, HIVE-19237.02.patch, > HIVE-19237.03.patch, HIVE-19237.04.patch, HIVE-19237.05.patch, > HIVE-19237.05.patch, HIVE-19237.06.patch, HIVE-19237.07.patch, > HIVE-19237.08.patch, HIVE-19237.08.patch, HIVE-19237.09.patch, > HIVE-19237.10.patch, HIVE-19237.10.patch, HIVE-19237.11.patch, > HIVE-19237.11.patch, HIVE-19237.11.patch, HIVE-19237.12.patch > > > Column stats autogather plan part is added from a plan compiled by the driver > itself; however that driver starts to use operatorIds from 1 ; so it's > possible that 2 SEL_1 operators end up in the same plan... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16505) Support "unknown" boolean truth value
[
https://issues.apache.org/jira/browse/HIVE-16505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-16505:
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
pushed to master. Thank you [~abstractdog] for making this happen!
> Support "unknown" boolean truth value
> -
>
> Key: HIVE-16505
> URL: https://issues.apache.org/jira/browse/HIVE-16505
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-16505.01.patch, HIVE-16505.02.patch,
> HIVE-16505.03.patch, HIVE-16505.04.patch
>
>
> according to the standard, boolean truth value might be:
> {{TRUE|FALSE|UNKNOWN}}.
> similar queries to the following should be supported:
> {code:java}
> select 1 where null is unknown;
> select 1 where (select cast(null as boolean) ) is unknown;
> {code}
> "unknown" behaves similarily to null. {{(null=null) is null}}
>
> "All boolean values and SQL truth values are comparable and all are
> assignable to a site of type boolean. The value True is greater than the
> value False, and any comparison involving the null value or an Unknown truth
> value will return an Unknown result. The values True and False may be
> assigned to any site having a boolean data type; assignment of Unknown, or
> the null value, is subject to the nullability characteristic of the target."
>
> *Truth table for the AND boolean operator*
> AND True False Unknown
> True True False Unknown
> False False False False
> Unknown Unknown False Unknown
> *Truth table for the OR boolean operator*
> OR True False Unknown
> True True True True
> False True False Unknown
> Unknown True Unknown Unknown
> *Truth table for the IS boolean operator*
> IS TRUE FALSE UNKNOWN
> True True False False
> False False True False
> Unknown False False True
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19008) Improve Spark session id logging
[
https://issues.apache.org/jira/browse/HIVE-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507717#comment-16507717
]
Hive QA commented on HIVE-19008:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
16s{color} | {color:blue} ql in master has 2286 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
33s{color} | {color:red} ql: The patch generated 4 new + 92 unchanged - 1 fixed
= 96 total (was 93) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11699/dev-support/hive-personality.sh
|
| git revision | master / 24a98ea |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11699/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11699/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve Spark session id logging
>
>
> Key: HIVE-19008
> URL: https://issues.apache.org/jira/browse/HIVE-19008
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19008.1.patch, HIVE-19008.2.patch
>
>
> HoS users have two session ids, one id for the Hive session and another id
> for the Spark session, both are UUIDs.
> I think some improvements could be made here:
> The Spark session id could just be a counter that is incremented for each new
> Spark session within a Hive session. Each Spark session is still globally
> identifiable by its associated Hive session id + its own counter. This may
> make more sense since the Hive session - Spark session has a 1-to-many
> relationship, as in a single Hive session can contain multiple Spark
> sessions, and each Spark session must belong to a Hive session.
> Furthermore, we should include both the Hive session id and Spark session id
> in the console logs + the Spark Web UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19744) In Beeline if -u is specified the default connection should not be tried at all
[
https://issues.apache.org/jira/browse/HIVE-19744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507713#comment-16507713
]
Zoltan Haindrich commented on HIVE-19744:
-
[~ashutoshc] could you take a look?
> In Beeline if -u is specified the default connection should not be tried at
> all
> ---
>
> Key: HIVE-19744
> URL: https://issues.apache.org/jira/browse/HIVE-19744
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19744.01.patch
>
>
> I wanted to explicitly connect to a hiveserver by specifying {{-u}} but that
> didn't work because it was not running/etc...
> The strange thing is that somehow the default connection is activated...and
> tried
> The possible "hazard" here is that if someone specifies {{-u $MY_DEV_HS2 -f
> recreate_db.sql}} to run some sql script...beeline may connect somewhere else
> and run the commands there - which might have serious consequences (in
> the above case having default as production might be interesting)
> {code}
> beeline -u jdbc:hive2://localhost:10502/;transportMode=binary -n hrt_qa
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.0.0.0-1406/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.0.0.0-1406/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Connecting to jdbc:hive2://localhost:10502/
> 18/05/31 07:51:20 [main]: WARN jdbc.HiveConnection: Failed to connect to
> localhost:10502
> Unknown HS2 problem when communicating with Thrift server.
> Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://localhost:10502/: Invalid status 72 (state=08S01,code=0)
> Connecting to
> jdbc:hive2://ctr-e138-1518143905142-336795-01-16.hwx.site:2181,ctr-e138-1518143905142-336795-01-08.hwx.site:2181,ctr-e138-1518143905142-336795-01-14.hwx.site:2181,ctr-e138-1518143905142-336795-01-09.hwx.site:2181,ctr-e138-1518143905142-336795-01-15.hwx.site:2181/default;httpPath=cliservice;principal=hive/_h...@example.com;serviceDiscoveryMode=zooKeeper;ssl=true;transportMode=http;zooKeeperNamespace=hiveserver2
> 18/05/31 07:51:21 [main]: INFO jdbc.HiveConnection: Connected to
> ctr-e138-1518143905142-336795-01-03.hwx.site:10001
> 18/05/31 07:51:21 [main]: ERROR jdbc.HiveConnection: Error opening session
> org.apache.thrift.transport.TTransportException:
> javax.net.ssl.SSLHandshakeException:
> sun.security.validator.ValidatorException: PKIX path validation failed:
> java.security.cert.CertPathValidatorException: signature check failed
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
