[jira] [Commented] (HIVE-18385) mergejoin fails with java.lang.IllegalStateException
[
https://issues.apache.org/jira/browse/HIVE-18385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325023#comment-16325023
]
Hive QA commented on HIVE-18385:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905973/HIVE-18385.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 28 failed/errored test(s), 11533 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_create_rewrite_4]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=178)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_alter_table_exchange_partition_fail]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_no_transactional_rewrite]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_no_transactional_rewrite_2]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=121)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=214)
org.apache.hadoop.hive.metastore.TestHiveMetaTool.testListFSRoot (batchId=222)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=254)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMergeUnpartitioned01
(batchId=291)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=226)
org.apache.hive.hcatalog.pig.TestE2EScenarios.testReadOrcAndRCFromPig
(batchId=191)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8608/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8608/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8608/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 28 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12905973 - PreCommit-HIVE-Build
> mergejoin fails with java.lang.IllegalStateException
>
>
> Key: HIVE-18385
> URL: https://issues.apache.org/jira/browse/HIVE-18385
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Jason Dere
> Attachments: HIVE-18385.1.patch, hive.log
>
>
> mergejoin test fails with java.lang.IllegalStateException when run in
> MiniLlapLocal.
> This is the query for which it fails,
> [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Client execution failed
> with error code = 2 running "
> select count(*) from tab a join tab_part b on a.key = b.key join src1 c on
> a.value = c.value" fname=mergejoin.q
> This is the stack trace,
> failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: b initializer failed,
> vertex=vertex_1515180518813_0001_42_05 [Map 8], java.lang.RuntimeException:
> ORC split generation failed with exception: java.lang.IllegalStateException:
> Failed to retrieve dynamic value for
[jira] [Commented] (HIVE-18385) mergejoin fails with java.lang.IllegalStateException
[
https://issues.apache.org/jira/browse/HIVE-18385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325010#comment-16325010
]
Hive QA commented on HIVE-18385:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / b1cdbc6 |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8608/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> mergejoin fails with java.lang.IllegalStateException
>
>
> Key: HIVE-18385
> URL: https://issues.apache.org/jira/browse/HIVE-18385
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Jason Dere
> Attachments: HIVE-18385.1.patch, hive.log
>
>
> mergejoin test fails with java.lang.IllegalStateException when run in
> MiniLlapLocal.
> This is the query for which it fails,
> [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Client execution failed
> with error code = 2 running "
> select count(*) from tab a join tab_part b on a.key = b.key join src1 c on
> a.value = c.value" fname=mergejoin.q
> This is the stack trace,
> failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: b initializer failed,
> vertex=vertex_1515180518813_0001_42_05 [Map 8], java.lang.RuntimeException:
> ORC split generation failed with exception: java.lang.IllegalStateException:
> Failed to retrieve dynamic value for RS_12_a_key_min
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1784)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1872)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:499)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:684)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native
[jira] [Commented] (HIVE-17495) CachedStore: prewarm improvement (avoid multiple sql calls to read partition column stats), refactoring and caching some aggregate stats
[
https://issues.apache.org/jira/browse/HIVE-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325002#comment-16325002
]
Hive QA commented on HIVE-17495:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905925/HIVE-17495.9.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 11138 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_4] (batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_create_rewrite_4]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestContribCliDriver.org.apache.hadoop.hive.cli.TestContribCliDriver
(batchId=241)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_no_transactional_rewrite]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_no_transactional_rewrite_2]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=121)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=214)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=254)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMergeUnpartitioned01
(batchId=291)
org.apache.hadoop.hive.ql.metadata.TestHiveRemote.testDropTableTrash
(batchId=277)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=226)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8607/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8607/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8607/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12905925 - PreCommit-HIVE-Build
> CachedStore: prewarm improvement (avoid multiple sql calls to read partition
> column stats), refactoring and caching some aggregate stats
>
>
> Key: HIVE-17495
> URL: https://issues.apache.org/jira/browse/HIVE-17495
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-17495.1.patch, HIVE-17495.2.patch,
> HIVE-17495.3.patch, HIVE-17495.4.patch, HIVE-17495.5.patch,
> HIVE-17495.6.patch, HIVE-17495.7.patch, HIVE-17495.8.patch, HIVE-17495.9.patch
>
>
> Only when CachedStore is enabled, we would like to make the following
> optimizations:
> 1. During CachedStore prewarm, use one sql call to retrieve column stats
> objects for a db and store it in the cache.
> 2. Cache some aggregate stats (e.g. aggregate stats for all partitions,
> which seems to be commonly used) for query compilation speedup.
> 3. There was a bug in {{MetaStoreUtils.aggrPartitionStats}}, which would use
> an iterator.next w/o checking with iterator.hasNext. This patch refactors
[jira] [Updated] (HIVE-18350) load data should rename files consistent with insert statements
[ https://issues.apache.org/jira/browse/HIVE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18350: -- Attachment: HIVE-18350.3.patch Reuploading the patch after build failure. > load data should rename files consistent with insert statements > --- > > Key: HIVE-18350 > URL: https://issues.apache.org/jira/browse/HIVE-18350 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-18350.1.patch, HIVE-18350.2.patch, > HIVE-18350.3.patch > > > Insert statements create files of format ending with _0, 0001_0 etc. > However, the load data uses the input file name. That results in inconsistent > naming convention which makes SMB joins difficult in some scenarios and may > cause trouble for other types of queries in future. > We need consistent naming convention. > For non-bucketed table, hive renames all the files regardless of how they > were named by the user. > For bucketed table, hive relies on user to name the files matching the bucket > in non-strict mode. Hive assumes that the data belongs to same bucket in a > file. In strict mode, loading bucketed table is disabled. > This will likely affect most of the tests which load data which is pretty > significant due to which it is further divided into two subtasks for smoother > merge. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18385) mergejoin fails with java.lang.IllegalStateException
[ https://issues.apache.org/jira/browse/HIVE-18385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324984#comment-16324984 ] Deepak Jaiswal commented on HIVE-18385: --- Thanks for fixing it. The patch looks good. +1. > mergejoin fails with java.lang.IllegalStateException > > > Key: HIVE-18385 > URL: https://issues.apache.org/jira/browse/HIVE-18385 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Jason Dere > Attachments: HIVE-18385.1.patch, hive.log > > > mergejoin test fails with java.lang.IllegalStateException when run in > MiniLlapLocal. > This is the query for which it fails, > [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Client execution failed > with error code = 2 running " > select count(*) from tab a join tab_part b on a.key = b.key join src1 c on > a.value = c.value" fname=mergejoin.q > This is the stack trace, > failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: b initializer failed, > vertex=vertex_1515180518813_0001_42_05 [Map 8], java.lang.RuntimeException: > ORC split generation failed with exception: java.lang.IllegalStateException: > Failed to retrieve dynamic value for RS_12_a_key_min > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1784) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1872) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:499) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:684) > at > org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253) > at > com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) > at > com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) > at > com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.concurrent.ExecutionException: > java.lang.IllegalStateException: Failed to retrieve dynamic value for > RS_12_a_key_min > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1778) > ... 17 more > Caused by: java.lang.IllegalStateException: Failed to retrieve dynamic value > for RS_12_a_key_min > at > org.apache.hadoop.hive.ql.plan.DynamicValue.getValue(DynamicValue.java:142) > at > org.apache.hadoop.hive.ql.plan.DynamicValue.getJavaValue(DynamicValue.java:97) > at > org.apache.hadoop.hive.ql.plan.DynamicValue.getLiteral(DynamicValue.java:93) > at > org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.getLiteralList(SearchArgumentImpl.java:120) > at > org.apache.orc.impl.RecordReaderImpl.evaluatePredicateMinMax(RecordReaderImpl.java:553) > at > org.apache.orc.impl.RecordReaderImpl.evaluatePredicateRange(RecordReaderImpl.java:463) > at > org.apache.orc.impl.RecordReaderImpl.evaluatePredicate(RecordReaderImpl.java:440) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.isStripeSatisfyPredicate(OrcInputFormat.java:2163) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.pickStripesInternal(OrcInputFormat.java:2140) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.pickStripes(OrcInputFormat.java:2131) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.access$3000(OrcInputFormat.java:157) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1476) > at >
[jira] [Commented] (HIVE-17495) CachedStore: prewarm improvement (avoid multiple sql calls to read partition column stats), refactoring and caching some aggregate stats
[
https://issues.apache.org/jira/browse/HIVE-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324983#comment-16324983
]
Hive QA commented on HIVE-17495:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore: The patch generated 10 new +
1131 unchanged - 31 fixed = 1141 total (was 1162) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / b1cdbc6 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8607/yetus/diff-checkstyle-standalone-metastore.txt
|
| modules | C: standalone-metastore itests/hcatalog-unit itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8607/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CachedStore: prewarm improvement (avoid multiple sql calls to read partition
> column stats), refactoring and caching some aggregate stats
>
>
> Key: HIVE-17495
> URL: https://issues.apache.org/jira/browse/HIVE-17495
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-17495.1.patch, HIVE-17495.2.patch,
> HIVE-17495.3.patch, HIVE-17495.4.patch, HIVE-17495.5.patch,
> HIVE-17495.6.patch, HIVE-17495.7.patch, HIVE-17495.8.patch, HIVE-17495.9.patch
>
>
> Only when CachedStore is enabled, we would like to make the following
> optimizations:
> 1. During CachedStore prewarm, use one sql call to retrieve column stats
> objects for a db and store it in the cache.
> 2. Cache some aggregate stats (e.g. aggregate stats for all partitions,
> which seems to be commonly used) for query compilation speedup.
> 3. There was a bug in {{MetaStoreUtils.aggrPartitionStats}}, which would use
> an iterator.next w/o checking with iterator.hasNext. This patch refactors
> some code to fix that.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18426) Memory leak in RoutingAppender for every hive operation
[
https://issues.apache.org/jira/browse/HIVE-18426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324972#comment-16324972
]
Hive QA commented on HIVE-18426:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905921/HIVE-18426.003.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 11558 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_create_rewrite_4]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_no_transactional_rewrite]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_no_transactional_rewrite_2]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=121)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=214)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=254)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMergeUnpartitioned01
(batchId=291)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=226)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232)
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementParallel
(batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8606/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8606/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8606/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12905921 - PreCommit-HIVE-Build
> Memory leak in RoutingAppender for every hive operation
> ---
>
> Key: HIVE-18426
> URL: https://issues.apache.org/jira/browse/HIVE-18426
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
> Fix For: 3.0.0
>
> Attachments: HIVE-18426.001.patch, HIVE-18426.002.patch,
> HIVE-18426.003.patch, Screen Shot 2018-01-10 at 9.17.19 AM.png
>
>
> Each new operation creates new entry in the ConcurrentMap in RoutingAppender
> but when the operation ends, AppenderControl stored in the map is retrieved
> and stopped but the entry in ConcurrentMap is never cleaned up.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18373) Make it easier to search for column name in a table
[
https://issues.apache.org/jira/browse/HIVE-18373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324965#comment-16324965
]
Prasad Nagaraj Subramanya commented on HIVE-18373:
--
[~siddhantsaraf], mysql syntax is as below -
{code}
show fields in like '%%'
{code}
We could extend `show columns` in similar format
{code}
show columns in like '%%'
{code}
> Make it easier to search for column name in a table
> ---
>
> Key: HIVE-18373
> URL: https://issues.apache.org/jira/browse/HIVE-18373
> Project: Hive
> Issue Type: New Feature
>Reporter: Siddhant Saraf
>Assignee: Madhudeep Petwal
>Priority: Minor
>
> Within a database, to filter for tables with the string 'abc' in its name, I
> can use something like:
> {code:java}
> hive> use my_database;
> hive> show tables '*abc*';
> {code}
> It would be great if I can do something similar to search within the list of
> columns in a table.
> I have a table with around 3200 columns. Searching for the column of interest
> is an onerous task after doing a {{describe}} on it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18426) Memory leak in RoutingAppender for every hive operation
[
https://issues.apache.org/jira/browse/HIVE-18426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324958#comment-16324958
]
Hive QA commented on HIVE-18426:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} common: The patch generated 2 new + 27 unchanged - 0
fixed = 29 total (was 27) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / b1cdbc6 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8606/yetus/diff-checkstyle-common.txt
|
| modules | C: common itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8606/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Memory leak in RoutingAppender for every hive operation
> ---
>
> Key: HIVE-18426
> URL: https://issues.apache.org/jira/browse/HIVE-18426
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
> Fix For: 3.0.0
>
> Attachments: HIVE-18426.001.patch, HIVE-18426.002.patch,
> HIVE-18426.003.patch, Screen Shot 2018-01-10 at 9.17.19 AM.png
>
>
> Each new operation creates new entry in the ConcurrentMap in RoutingAppender
> but when the operation ends, AppenderControl stored in the map is retrieved
> and stopped but the entry in ConcurrentMap is never cleaned up.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18350) load data should rename files consistent with insert statements
[
https://issues.apache.org/jira/browse/HIVE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324948#comment-16324948
]
Hive QA commented on HIVE-18350:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905920/HIVE-18350.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8605/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8605/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8605/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-01-13 04:01:39.732
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-8605/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-01-13 04:01:39.735
+ cd apache-github-source-source
+ git fetch origin
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
+ git reset --hard HEAD
HEAD is now at b1cdbc6 HIVE-18416: Initial support for TABLE function (Jesus
Camacho Rodriguez, reviewed by Ashutosh Chauhan) (addendum)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at b1cdbc6 HIVE-18416: Initial support for TABLE function (Jesus
Camacho Rodriguez, reviewed by Ashutosh Chauhan) (addendum)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-01-13 04:01:44.211
+ rm -rf ../yetus
+ mkdir ../yetus
+ cp -R . ../yetus
cp: cannot stat ?./.git/gc.pid?: No such file or directory
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12905920 - PreCommit-HIVE-Build
> load data should rename files consistent with insert statements
> ---
>
> Key: HIVE-18350
> URL: https://issues.apache.org/jira/browse/HIVE-18350
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-18350.1.patch, HIVE-18350.2.patch
>
>
> Insert statements create files of format ending with _0, 0001_0 etc.
> However, the load data uses the input file name. That results in inconsistent
> naming convention which makes SMB joins difficult in some scenarios and may
> cause trouble for other types of queries in future.
> We need consistent naming convention.
> For non-bucketed table, hive renames all the files regardless of how they
> were named by the user.
> For bucketed table, hive relies on user to name the files matching the bucket
> in non-strict mode. Hive assumes that the data belongs to same bucket in a
> file. In strict mode, loading bucketed table is disabled.
> This will likely affect most of the tests which load data which is pretty
> significant due to which it is further divided into two subtasks for smoother
> merge.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18430) Add new determinism category for runtime constants (current_date, current_timestamp)
[
https://issues.apache.org/jira/browse/HIVE-18430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324947#comment-16324947
]
Hive QA commented on HIVE-18430:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905911/HIVE-18430.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 28 failed/errored test(s), 11559 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_create_rewrite_4]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_no_transactional_rewrite]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_no_transactional_rewrite_2]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=121)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=214)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=254)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMergeUnpartitioned01
(batchId=291)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=226)
org.apache.hive.hcatalog.common.TestHiveClientCache.testCloseAllClients
(batchId=198)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestSchedulerQueue.testQueueMappingCheckDisabled
(batchId=239)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8604/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8604/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8604/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 28 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12905911 - PreCommit-HIVE-Build
> Add new determinism category for runtime constants (current_date,
> current_timestamp)
>
>
> Key: HIVE-18430
> URL: https://issues.apache.org/jira/browse/HIVE-18430
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-18430.1.patch, HIVE-18430.2.patch
>
>
> Add a new piece of metadata to the UDFs to specify if whether a UDF is a
> runtime constant. Runtime constants also exist in SQL Server, and this is
> similar to Postgres' concept of STABLE functions. This metadata may be useful
> for materialized views and query caching.
> Some Hive functions such as the ones listed below are currently labelled as
> deterministic, but really are runtime constants:
> current_timestamp
> current_date
> current_user
> current_database
> The values for these functions are not deterministic between different
> queries - for example current_timestamp will most likely be different every
>
[jira] [Updated] (HIVE-18410) [Performance][Avro] Reading flat Avro tables is very expensive in Hive
[
https://issues.apache.org/jira/browse/HIVE-18410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ratandeep Ratti updated HIVE-18410:
---
Fix Version/s: 3.0.0
> [Performance][Avro] Reading flat Avro tables is very expensive in Hive
> --
>
> Key: HIVE-18410
> URL: https://issues.apache.org/jira/browse/HIVE-18410
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.3.2
>Reporter: Ratandeep Ratti
>Assignee: Ratandeep Ratti
> Fix For: 3.0.0, 2.3.2
>
> Attachments: HIVE-18410.patch, HIVE-18410_1.patch,
> profiling_with_patch.nps, profiling_with_patch.png,
> profiling_without_patch.nps, profiling_without_patch.png
>
>
> There's a performance penalty when reading flat [no nested fields] Avro
> tables. When reading the same flat dataset in Pig, it takes half the time.
> On profiling, a lot of time is spent in
> {{AvroDeserializer.deserializeSingleItemNullableUnion()}}. The bulk of the
> time is spent in GenericData.get().resolveUnion(), which calls
> GenericData.getSchemaName(Object datum), which does a lot of instanceof
> checks. This could be simplified with performance benefits. A approach is
> described in this patch which almost halves the runtime.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18452) work around HADOOP-15171
[ https://issues.apache.org/jira/browse/HIVE-18452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324935#comment-16324935 ] Sergey Shelukhin commented on HIVE-18452: - [~prasanth_j] can you take a look? > work around HADOOP-15171 > > > Key: HIVE-18452 > URL: https://issues.apache.org/jira/browse/HIVE-18452 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18452.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18452) work around HADOOP-15171
[ https://issues.apache.org/jira/browse/HIVE-18452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18452: Attachment: (was: HIVE-18452.patch) > work around HADOOP-15171 > > > Key: HIVE-18452 > URL: https://issues.apache.org/jira/browse/HIVE-18452 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18452.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18452) work around HADOOP-15171
[ https://issues.apache.org/jira/browse/HIVE-18452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18452: Attachment: HIVE-18452.patch > work around HADOOP-15171 > > > Key: HIVE-18452 > URL: https://issues.apache.org/jira/browse/HIVE-18452 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18452.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18452) work around HADOOP-15171
[ https://issues.apache.org/jira/browse/HIVE-18452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18452: Attachment: HIVE-18452.patch > work around HADOOP-15171 > > > Key: HIVE-18452 > URL: https://issues.apache.org/jira/browse/HIVE-18452 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18452.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18452) work around HADOOP-15171
[ https://issues.apache.org/jira/browse/HIVE-18452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18452: Attachment: (was: HIVE-18452.patch) > work around HADOOP-15171 > > > Key: HIVE-18452 > URL: https://issues.apache.org/jira/browse/HIVE-18452 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18452.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18452) work around HADOOP-15171
[ https://issues.apache.org/jira/browse/HIVE-18452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18452: Status: Patch Available (was: Open) > work around HADOOP-15171 > > > Key: HIVE-18452 > URL: https://issues.apache.org/jira/browse/HIVE-18452 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18452.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18452) work around HADOOP-15171
[ https://issues.apache.org/jira/browse/HIVE-18452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18452: Attachment: HIVE-18452.patch This also adds a safety flag to turn off codec pool. I suspected the pool was at fault, and since it could easily have bugs it's good to have a safety option. > work around HADOOP-15171 > > > Key: HIVE-18452 > URL: https://issues.apache.org/jira/browse/HIVE-18452 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18452.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18452) work around HADOOP-15171
[ https://issues.apache.org/jira/browse/HIVE-18452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-18452: --- > work around HADOOP-15171 > > > Key: HIVE-18452 > URL: https://issues.apache.org/jira/browse/HIVE-18452 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18430) Add new determinism category for runtime constants (current_date, current_timestamp)
[
https://issues.apache.org/jira/browse/HIVE-18430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324925#comment-16324925
]
Hive QA commented on HIVE-18430:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 17 new + 1620 unchanged - 6
fixed = 1637 total (was 1626) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / b1cdbc6 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8604/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8604/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add new determinism category for runtime constants (current_date,
> current_timestamp)
>
>
> Key: HIVE-18430
> URL: https://issues.apache.org/jira/browse/HIVE-18430
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-18430.1.patch, HIVE-18430.2.patch
>
>
> Add a new piece of metadata to the UDFs to specify if whether a UDF is a
> runtime constant. Runtime constants also exist in SQL Server, and this is
> similar to Postgres' concept of STABLE functions. This metadata may be useful
> for materialized views and query caching.
> Some Hive functions such as the ones listed below are currently labelled as
> deterministic, but really are runtime constants:
> current_timestamp
> current_date
> current_user
> current_database
> The values for these functions are not deterministic between different
> queries - for example current_timestamp will most likely be different every
> query executed. This makes these functions ineligible for things like
> materialized views or cached query results.
> However the value for the current_timestamp should not change during the life
> of a single query, which allows these values to be used in optimizations such
> as constant folding.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18410) [Performance][Avro] Reading flat Avro tables is very expensive in Hive
[
https://issues.apache.org/jira/browse/HIVE-18410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324915#comment-16324915
]
Hive QA commented on HIVE-18410:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905898/HIVE-18410_1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 11568 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_binary_storage_queries]
(batchId=99)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8603/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8603/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8603/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 19 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12905898 - PreCommit-HIVE-Build
> [Performance][Avro] Reading flat Avro tables is very expensive in Hive
> --
>
> Key: HIVE-18410
> URL: https://issues.apache.org/jira/browse/HIVE-18410
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.3.2
>Reporter: Ratandeep Ratti
>Assignee: Ratandeep Ratti
> Fix For: 2.3.2
>
> Attachments: HIVE-18410.patch, HIVE-18410_1.patch,
> profiling_with_patch.nps, profiling_with_patch.png,
> profiling_without_patch.nps, profiling_without_patch.png
>
>
> There's a performance penalty when reading flat [no nested fields] Avro
> tables. When reading the same flat dataset in Pig, it takes half the time.
> On profiling, a lot of time is spent in
> {{AvroDeserializer.deserializeSingleItemNullableUnion()}}. The bulk of the
> time is spent in GenericData.get().resolveUnion(), which calls
> GenericData.getSchemaName(Object datum), which does a lot of instanceof
> checks. This could be simplified with performance benefits. A approach is
> described in this patch which almost halves the runtime.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18451) Support UDTF Operator in SimpleFetchOptimizer
[
https://issues.apache.org/jira/browse/HIVE-18451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez reassigned HIVE-18451:
--
Assignee: (was: Jesus Camacho Rodriguez)
> Support UDTF Operator in SimpleFetchOptimizer
> -
>
> Key: HIVE-18451
> URL: https://issues.apache.org/jira/browse/HIVE-18451
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>
> After HIVE-18416, simple insert into table values{} is still resulting in
> launching of job as earlier which can be avoided by optimizing that job to
> FetchTask.
> This issue is to extend SimpleFetchOptimizer, which does not fire on insert
> queries as it does not handle UDTF.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18450) Support TABLE function in CBO
[ https://issues.apache.org/jira/browse/HIVE-18450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18450: --- Attachment: HIVE-18450.01.patch > Support TABLE function in CBO > - > > Key: HIVE-18450 > URL: https://issues.apache.org/jira/browse/HIVE-18450 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-18450.01.patch > > > Follow-up of HIVE-18416 to support TABLE function in CBO. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18450) Support TABLE function in CBO
[ https://issues.apache.org/jira/browse/HIVE-18450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18450: --- Status: Patch Available (was: In Progress) > Support TABLE function in CBO > - > > Key: HIVE-18450 > URL: https://issues.apache.org/jira/browse/HIVE-18450 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > Follow-up of HIVE-18416 to support TABLE function in CBO. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18450) Support TABLE function in CBO
[ https://issues.apache.org/jira/browse/HIVE-18450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18450: --- Attachment: (was: HIVE-18450.patch) > Support TABLE function in CBO > - > > Key: HIVE-18450 > URL: https://issues.apache.org/jira/browse/HIVE-18450 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > Follow-up of HIVE-18416 to support TABLE function in CBO. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18416) Initial support for TABLE function
[
https://issues.apache.org/jira/browse/HIVE-18416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-18416:
---
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
[~ashutoshc], I created HIVE-18451 to extend the FetchOptimizer. I fixed
{{TestDbTxnManager2}} (changes expected after this patch) and pushed to master.
> Initial support for TABLE function
> --
>
> Key: HIVE-18416
> URL: https://issues.apache.org/jira/browse/HIVE-18416
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-18416.01.patch, HIVE-18416.patch
>
>
> Include support for TABLE function derived table. In SQL standard, it is
> defined as:
> {noformat}
> ::=
> TABLE
> {noformat}
> Further, include limited support for its usage as LATERAL derived table.
> This will allow Hive to execute queries such as:
> {code}
> SELECT tf.col1, tf.col2, tf.col3
> FROM
> TABLE(VALUES('A', 10, 30.0),('B', 20, 30.0)) AS tf(col1, col2, col3);
> SELECT tf.col1, tf.col2, tf.col3
> FROM
> (SELECT key, value FROM src) t,
> LATERAL TABLE(VALUES('A', 10, t.key),('B', 20, t.key)) AS tf(col1, col2,
> col3);
> {code}
> The idea is to rely on AST rewriting for such cases, as TABLE can be
> implemented using {{inline}} UDTF with an {{array}} of {{structs}}
> representing the rows in the {{VALUES}} clause.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18451) Support UDTF Operator in SimpleFetchOptimizer
[
https://issues.apache.org/jira/browse/HIVE-18451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez reassigned HIVE-18451:
--
> Support UDTF Operator in SimpleFetchOptimizer
> -
>
> Key: HIVE-18451
> URL: https://issues.apache.org/jira/browse/HIVE-18451
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>
> After HIVE-18416, simple insert into table values{} is still resulting in
> launching of job as earlier which can be avoided by optimizing that job to
> FetchTask.
> This issue is to extend SimpleFetchOptimizer, which does not fire on insert
> queries as it does not handle UDTF.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18385) mergejoin fails with java.lang.IllegalStateException
[ https://issues.apache.org/jira/browse/HIVE-18385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-18385: -- Status: Patch Available (was: Open) > mergejoin fails with java.lang.IllegalStateException > > > Key: HIVE-18385 > URL: https://issues.apache.org/jira/browse/HIVE-18385 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Jason Dere > Attachments: HIVE-18385.1.patch, hive.log > > > mergejoin test fails with java.lang.IllegalStateException when run in > MiniLlapLocal. > This is the query for which it fails, > [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Client execution failed > with error code = 2 running " > select count(*) from tab a join tab_part b on a.key = b.key join src1 c on > a.value = c.value" fname=mergejoin.q > This is the stack trace, > failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: b initializer failed, > vertex=vertex_1515180518813_0001_42_05 [Map 8], java.lang.RuntimeException: > ORC split generation failed with exception: java.lang.IllegalStateException: > Failed to retrieve dynamic value for RS_12_a_key_min > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1784) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1872) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:499) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:684) > at > org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253) > at > com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) > at > com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) > at > com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.concurrent.ExecutionException: > java.lang.IllegalStateException: Failed to retrieve dynamic value for > RS_12_a_key_min > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1778) > ... 17 more > Caused by: java.lang.IllegalStateException: Failed to retrieve dynamic value > for RS_12_a_key_min > at > org.apache.hadoop.hive.ql.plan.DynamicValue.getValue(DynamicValue.java:142) > at > org.apache.hadoop.hive.ql.plan.DynamicValue.getJavaValue(DynamicValue.java:97) > at > org.apache.hadoop.hive.ql.plan.DynamicValue.getLiteral(DynamicValue.java:93) > at > org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.getLiteralList(SearchArgumentImpl.java:120) > at > org.apache.orc.impl.RecordReaderImpl.evaluatePredicateMinMax(RecordReaderImpl.java:553) > at > org.apache.orc.impl.RecordReaderImpl.evaluatePredicateRange(RecordReaderImpl.java:463) > at > org.apache.orc.impl.RecordReaderImpl.evaluatePredicate(RecordReaderImpl.java:440) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.isStripeSatisfyPredicate(OrcInputFormat.java:2163) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.pickStripesInternal(OrcInputFormat.java:2140) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.pickStripes(OrcInputFormat.java:2131) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.access$3000(OrcInputFormat.java:157) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1476) > at >
[jira] [Updated] (HIVE-18385) mergejoin fails with java.lang.IllegalStateException
[ https://issues.apache.org/jira/browse/HIVE-18385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-18385: -- Attachment: HIVE-18385.1.patch Looks like HIVE-17631 needed to add another case of using NoDynamicValuesException. Also updating the golden file output due to HIVE-18401. > mergejoin fails with java.lang.IllegalStateException > > > Key: HIVE-18385 > URL: https://issues.apache.org/jira/browse/HIVE-18385 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Jason Dere > Attachments: HIVE-18385.1.patch, hive.log > > > mergejoin test fails with java.lang.IllegalStateException when run in > MiniLlapLocal. > This is the query for which it fails, > [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Client execution failed > with error code = 2 running " > select count(*) from tab a join tab_part b on a.key = b.key join src1 c on > a.value = c.value" fname=mergejoin.q > This is the stack trace, > failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: b initializer failed, > vertex=vertex_1515180518813_0001_42_05 [Map 8], java.lang.RuntimeException: > ORC split generation failed with exception: java.lang.IllegalStateException: > Failed to retrieve dynamic value for RS_12_a_key_min > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1784) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1872) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:499) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:684) > at > org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253) > at > com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) > at > com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) > at > com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.concurrent.ExecutionException: > java.lang.IllegalStateException: Failed to retrieve dynamic value for > RS_12_a_key_min > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1778) > ... 17 more > Caused by: java.lang.IllegalStateException: Failed to retrieve dynamic value > for RS_12_a_key_min > at > org.apache.hadoop.hive.ql.plan.DynamicValue.getValue(DynamicValue.java:142) > at > org.apache.hadoop.hive.ql.plan.DynamicValue.getJavaValue(DynamicValue.java:97) > at > org.apache.hadoop.hive.ql.plan.DynamicValue.getLiteral(DynamicValue.java:93) > at > org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.getLiteralList(SearchArgumentImpl.java:120) > at > org.apache.orc.impl.RecordReaderImpl.evaluatePredicateMinMax(RecordReaderImpl.java:553) > at > org.apache.orc.impl.RecordReaderImpl.evaluatePredicateRange(RecordReaderImpl.java:463) > at > org.apache.orc.impl.RecordReaderImpl.evaluatePredicate(RecordReaderImpl.java:440) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.isStripeSatisfyPredicate(OrcInputFormat.java:2163) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.pickStripesInternal(OrcInputFormat.java:2140) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.pickStripes(OrcInputFormat.java:2131) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.access$3000(OrcInputFormat.java:157) > at >
[jira] [Assigned] (HIVE-18385) mergejoin fails with java.lang.IllegalStateException
[ https://issues.apache.org/jira/browse/HIVE-18385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere reassigned HIVE-18385: - Assignee: Jason Dere (was: Deepak Jaiswal) > mergejoin fails with java.lang.IllegalStateException > > > Key: HIVE-18385 > URL: https://issues.apache.org/jira/browse/HIVE-18385 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Jason Dere > Attachments: hive.log > > > mergejoin test fails with java.lang.IllegalStateException when run in > MiniLlapLocal. > This is the query for which it fails, > [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Client execution failed > with error code = 2 running " > select count(*) from tab a join tab_part b on a.key = b.key join src1 c on > a.value = c.value" fname=mergejoin.q > This is the stack trace, > failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: b initializer failed, > vertex=vertex_1515180518813_0001_42_05 [Map 8], java.lang.RuntimeException: > ORC split generation failed with exception: java.lang.IllegalStateException: > Failed to retrieve dynamic value for RS_12_a_key_min > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1784) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1872) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:499) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:684) > at > org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253) > at > com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) > at > com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) > at > com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.concurrent.ExecutionException: > java.lang.IllegalStateException: Failed to retrieve dynamic value for > RS_12_a_key_min > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1778) > ... 17 more > Caused by: java.lang.IllegalStateException: Failed to retrieve dynamic value > for RS_12_a_key_min > at > org.apache.hadoop.hive.ql.plan.DynamicValue.getValue(DynamicValue.java:142) > at > org.apache.hadoop.hive.ql.plan.DynamicValue.getJavaValue(DynamicValue.java:97) > at > org.apache.hadoop.hive.ql.plan.DynamicValue.getLiteral(DynamicValue.java:93) > at > org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.getLiteralList(SearchArgumentImpl.java:120) > at > org.apache.orc.impl.RecordReaderImpl.evaluatePredicateMinMax(RecordReaderImpl.java:553) > at > org.apache.orc.impl.RecordReaderImpl.evaluatePredicateRange(RecordReaderImpl.java:463) > at > org.apache.orc.impl.RecordReaderImpl.evaluatePredicate(RecordReaderImpl.java:440) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.isStripeSatisfyPredicate(OrcInputFormat.java:2163) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.pickStripesInternal(OrcInputFormat.java:2140) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.pickStripes(OrcInputFormat.java:2131) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.access$3000(OrcInputFormat.java:157) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1476) > at >
[jira] [Commented] (HIVE-18410) [Performance][Avro] Reading flat Avro tables is very expensive in Hive
[
https://issues.apache.org/jira/browse/HIVE-18410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324886#comment-16324886
]
Hive QA commented on HIVE-18410:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} serde: The patch generated 2 new + 58 unchanged - 2
fixed = 60 total (was 60) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 8m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0a62507 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8603/yetus/diff-checkstyle-serde.txt
|
| modules | C: serde U: serde |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8603/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> [Performance][Avro] Reading flat Avro tables is very expensive in Hive
> --
>
> Key: HIVE-18410
> URL: https://issues.apache.org/jira/browse/HIVE-18410
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.3.2
>Reporter: Ratandeep Ratti
>Assignee: Ratandeep Ratti
> Fix For: 2.3.2
>
> Attachments: HIVE-18410.patch, HIVE-18410_1.patch,
> profiling_with_patch.nps, profiling_with_patch.png,
> profiling_without_patch.nps, profiling_without_patch.png
>
>
> There's a performance penalty when reading flat [no nested fields] Avro
> tables. When reading the same flat dataset in Pig, it takes half the time.
> On profiling, a lot of time is spent in
> {{AvroDeserializer.deserializeSingleItemNullableUnion()}}. The bulk of the
> time is spent in GenericData.get().resolveUnion(), which calls
> GenericData.getSchemaName(Object datum), which does a lot of instanceof
> checks. This could be simplified with performance benefits. A approach is
> described in this patch which almost halves the runtime.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18051) qfiles: dataset support
[
https://issues.apache.org/jira/browse/HIVE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324866#comment-16324866
]
Hive QA commented on HIVE-18051:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905894/HIVE-18051.05.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 106 failed/errored test(s), 11571 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_role_grant2]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_5]
(batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_5a]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_schema_evolution_native]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_sets1]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_sets2]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_sets3]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_schema_evol_1a]
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_schema_evol_1b]
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_schema_evol_2a]
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_schema_evol_2b]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[print_header] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_between_columns]
(batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_complex_join]
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_interval_arithmetic]
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_order_null]
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_reference_windowed]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols]
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_tablesample_rows]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_date_funcs]
(batchId=76)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby_grouping_id2]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_ppd_schema_evol_1a]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_ppd_schema_evol_1b]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_ppd_schema_evol_2a]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_ppd_schema_evol_2b]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acid_part]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acid_part_update]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acid_table]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acid_table_update]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acidvec_part]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acidvec_part_update]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acidvec_table]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acidvec_table_update]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_nonvec_part]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_nonvec_part_all_complex]
(batchId=151)
[jira] [Commented] (HIVE-18051) qfiles: dataset support
[
https://issues.apache.org/jira/browse/HIVE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324840#comment-16324840
]
Hive QA commented on HIVE-18051:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
30s{color} | {color:red} ql: The patch generated 1 new + 2 unchanged - 4 fixed
= 3 total (was 6) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
39s{color} | {color:red} root: The patch generated 7 new + 238 unchanged - 14
fixed = 245 total (was 252) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} itests/util: The patch generated 6 new + 236 unchanged
- 10 fixed = 242 total (was 246) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0a62507 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8602/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8602/yetus/diff-checkstyle-root.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8602/yetus/diff-checkstyle-itests_util.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8602/yetus/whitespace-eol.txt
|
| modules | C: ql . itests/qtest itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8602/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> qfiles: dataset support
> ---
>
> Key: HIVE-18051
> URL: https://issues.apache.org/jira/browse/HIVE-18051
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
> Attachments: HIVE-18051.01.patch, HIVE-18051.02.patch,
> HIVE-18051.03.patch, HIVE-18051.04.patch, HIVE-18051.05.patch
>
>
> it would be great to have some kind of test dataset support; currently there
> is the {{q_test_init.sql}} which is quite large; and I'm often override it
> with an invalid string;
[jira] [Commented] (HIVE-18323) Vectorization: add the support of timestamp in VectorizedPrimitiveColumnReader for parquet
[
https://issues.apache.org/jira/browse/HIVE-18323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324834#comment-16324834
]
Vihang Karajgaonkar commented on HIVE-18323:
I talked with [~aihuaxu] offline and he is okay if I continue the work on this
patch. Assigning this JIRA to myself. Thanks for the initial patch [~aihuaxu] I
will add some tests and make some changes to your patch and submit the second
version in some time.
> Vectorization: add the support of timestamp in
> VectorizedPrimitiveColumnReader for parquet
> --
>
> Key: HIVE-18323
> URL: https://issues.apache.org/jira/browse/HIVE-18323
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-18323.1.patch
>
>
> {noformat}
> CREATE TABLE `t1`(
> `ts` timestamp,
> `s1` string)
> STORED AS PARQUET;
> set hive.vectorized.execution.enabled=true;
> SELECT * from t1 SORT BY s1;
> {noformat}
> This query will throw exception since timestamp is not supported here yet.
> {noformat}
> Caused by: java.io.IOException: java.io.IOException: Unsupported type:
> optional int96 ts
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:116)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18450) Support TABLE function in CBO
[ https://issues.apache.org/jira/browse/HIVE-18450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324801#comment-16324801 ] Jesus Camacho Rodriguez commented on HIVE-18450: Cc [~ashutoshc] > Support TABLE function in CBO > - > > Key: HIVE-18450 > URL: https://issues.apache.org/jira/browse/HIVE-18450 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-18450.patch > > > Follow-up of HIVE-18416 to support TABLE function in CBO. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18450) Support TABLE function in CBO
[ https://issues.apache.org/jira/browse/HIVE-18450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18450: --- Attachment: HIVE-18450.patch > Support TABLE function in CBO > - > > Key: HIVE-18450 > URL: https://issues.apache.org/jira/browse/HIVE-18450 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-18450.patch > > > Follow-up of HIVE-18416 to support TABLE function in CBO. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Work started] (HIVE-18450) Support TABLE function in CBO
[ https://issues.apache.org/jira/browse/HIVE-18450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-18450 started by Jesus Camacho Rodriguez. -- > Support TABLE function in CBO > - > > Key: HIVE-18450 > URL: https://issues.apache.org/jira/browse/HIVE-18450 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > Follow-up of HIVE-18416 to support TABLE function in CBO. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18450) Support TABLE function in CBO
[ https://issues.apache.org/jira/browse/HIVE-18450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-18450: -- > Support TABLE function in CBO > - > > Key: HIVE-18450 > URL: https://issues.apache.org/jira/browse/HIVE-18450 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > Follow-up of HIVE-18416 to support TABLE function in CBO. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18309) qtests: smb_mapjoin_19.q breaks bucketsortoptimize_insert_2.q
[
https://issues.apache.org/jira/browse/HIVE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324770#comment-16324770
]
Hive QA commented on HIVE-18309:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905878/HIVE-18309.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 11567 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=228)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8601/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8601/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8601/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12905878 - PreCommit-HIVE-Build
> qtests: smb_mapjoin_19.q breaks bucketsortoptimize_insert_2.q
> -
>
> Key: HIVE-18309
> URL: https://issues.apache.org/jira/browse/HIVE-18309
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
> Attachments: HIVE-18309.01.patch
>
>
> {code}
> mvn install -q -am -pl itests/qtest -DskipSparkTests -pl itests/qtest
> -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=smb_mapjoin_19.q,bucketsortoptimize_insert_2.q
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18449) Add configurable policy for choosing the HMS URI from hive.metastore.uris
[
https://issues.apache.org/jira/browse/HIVE-18449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Janaki Lahorani reassigned HIVE-18449:
--
Assignee: Janaki Lahorani
> Add configurable policy for choosing the HMS URI from hive.metastore.uris
> -
>
> Key: HIVE-18449
> URL: https://issues.apache.org/jira/browse/HIVE-18449
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Sahil Takiar
>Assignee: Janaki Lahorani
>
> HIVE-10815 added logic to randomly choose a HMS URI from
> {{hive.metastore.uris}}. It would be nice if there was a configurable policy
> that determined how a URI is chosen from this list - e.g. one option can be
> to randomly pick a URI, another option can be to choose the first URI in the
> list (which was the behavior prior to HIVE-10815).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18309) qtests: smb_mapjoin_19.q breaks bucketsortoptimize_insert_2.q
[
https://issues.apache.org/jira/browse/HIVE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324705#comment-16324705
]
Hive QA commented on HIVE-18309:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0a62507 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8601/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> qtests: smb_mapjoin_19.q breaks bucketsortoptimize_insert_2.q
> -
>
> Key: HIVE-18309
> URL: https://issues.apache.org/jira/browse/HIVE-18309
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
> Attachments: HIVE-18309.01.patch
>
>
> {code}
> mvn install -q -am -pl itests/qtest -DskipSparkTests -pl itests/qtest
> -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=smb_mapjoin_19.q,bucketsortoptimize_insert_2.q
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18314) qtests: semijoin_hint.q breaks hybridgrace_hashjoin_2.q
[
https://issues.apache.org/jira/browse/HIVE-18314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324701#comment-16324701
]
Hive QA commented on HIVE-18314:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905873/HIVE-18314.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 11567 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_binary_storage_queries]
(batchId=99)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.metastore.TestMetastoreExpr.testPartitionExpr
(batchId=256)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8599/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8599/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8599/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 19 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12905873 - PreCommit-HIVE-Build
> qtests: semijoin_hint.q breaks hybridgrace_hashjoin_2.q
>
>
> Key: HIVE-18314
> URL: https://issues.apache.org/jira/browse/HIVE-18314
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
> Attachments: HIVE-18314.01.patch
>
>
> {code}
> mvn install -q -am -pl itests/qtest -DskipSparkTests
> -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=semijoin_hint.q,hybridgrace_hashjoin_2.q
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18314) qtests: semijoin_hint.q breaks hybridgrace_hashjoin_2.q
[
https://issues.apache.org/jira/browse/HIVE-18314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324593#comment-16324593
]
Hive QA commented on HIVE-18314:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0a62507 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8599/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> qtests: semijoin_hint.q breaks hybridgrace_hashjoin_2.q
>
>
> Key: HIVE-18314
> URL: https://issues.apache.org/jira/browse/HIVE-18314
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
> Attachments: HIVE-18314.01.patch
>
>
> {code}
> mvn install -q -am -pl itests/qtest -DskipSparkTests
> -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=semijoin_hint.q,hybridgrace_hashjoin_2.q
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18061) q.outs: be more selective with masking hdfs paths
[
https://issues.apache.org/jira/browse/HIVE-18061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324584#comment-16324584
]
Hive QA commented on HIVE-18061:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905871/HIVE-18061.07.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 60 failed/errored test(s), 11568 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_unencrypted_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge10]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge1]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge2]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge3]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge4]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[reduce_deduplicate]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[table_nonprintable]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_16]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[leftsemijoin_mr]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[smb_mapjoin_15]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucket5]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketmapjoin7]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[disable_merge_for_bucketing]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[list_bucket_dml_10]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge1]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge2]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge3]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge4]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge5]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge6]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge7]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge8]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge_incompat1]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge_incompat2]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[reduce_deduplicate]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez]
(batchId=103)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=102)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[local_mapred_error_cache]
(batchId=95)
[jira] [Assigned] (HIVE-18447) JDBC: Provide a way for JDBC users to pass cookie info via connection string
[ https://issues.apache.org/jira/browse/HIVE-18447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta reassigned HIVE-18447: --- Assignee: Vaibhav Gumashta > JDBC: Provide a way for JDBC users to pass cookie info via connection string > > > Key: HIVE-18447 > URL: https://issues.apache.org/jira/browse/HIVE-18447 > Project: Hive > Issue Type: Bug > Components: JDBC >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > > Some authentication mechanisms like Single Sign On, need the ability to pass > a cookie to some intermediate authentication service like Knox via the JDBC > driver. We need to add the mechanism in Hive's JDBC driver (when used in HTTP > transport mode). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17495) CachedStore: prewarm improvement (avoid multiple sql calls to read partition column stats), refactoring and caching some aggregate stats
[
https://issues.apache.org/jira/browse/HIVE-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-17495:
Attachment: HIVE-17495.9.patch
> CachedStore: prewarm improvement (avoid multiple sql calls to read partition
> column stats), refactoring and caching some aggregate stats
>
>
> Key: HIVE-17495
> URL: https://issues.apache.org/jira/browse/HIVE-17495
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-17495.1.patch, HIVE-17495.2.patch,
> HIVE-17495.3.patch, HIVE-17495.4.patch, HIVE-17495.5.patch,
> HIVE-17495.6.patch, HIVE-17495.7.patch, HIVE-17495.8.patch, HIVE-17495.9.patch
>
>
> Only when CachedStore is enabled, we would like to make the following
> optimizations:
> 1. During CachedStore prewarm, use one sql call to retrieve column stats
> objects for a db and store it in the cache.
> 2. Cache some aggregate stats (e.g. aggregate stats for all partitions,
> which seems to be commonly used) for query compilation speedup.
> 3. There was a bug in {{MetaStoreUtils.aggrPartitionStats}}, which would use
> an iterator.next w/o checking with iterator.hasNext. This patch refactors
> some code to fix that.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18061) q.outs: be more selective with masking hdfs paths
[
https://issues.apache.org/jira/browse/HIVE-18061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324538#comment-16324538
]
Hive QA commented on HIVE-18061:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} itests/util: The patch generated 1 new + 188 unchanged
- 6 fixed = 189 total (was 194) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0a62507 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8598/yetus/diff-checkstyle-itests_util.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8598/yetus/whitespace-eol.txt
|
| modules | C: ql itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8598/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> q.outs: be more selective with masking hdfs paths
> -
>
> Key: HIVE-18061
> URL: https://issues.apache.org/jira/browse/HIVE-18061
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
> Attachments: HIVE-18061.01.patch, HIVE-18061.02.patch,
> HIVE-18061.03.patch, HIVE-18061.04.patch, HIVE-18061.05.patch,
> HIVE-18061.06.patch, HIVE-18061.07.patch
>
>
> currently any line which contains a path which looks like an hdfs location is
> replaced with a "masked pattern was here"...
> it might be releavant to record these messages; since even an exception
> message might contain an hdfs location
> noticed in
> HIVE-18012
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18355) Add builder for metastore Thrift classes missed in the first pass - FunctionBuilder
[ https://issues.apache.org/jira/browse/HIVE-18355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324541#comment-16324541 ] Alan Gates commented on HIVE-18355: --- +1 > Add builder for metastore Thrift classes missed in the first pass - > FunctionBuilder > --- > > Key: HIVE-18355 > URL: https://issues.apache.org/jira/browse/HIVE-18355 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-18355.2.patch, HIVE-18355.patch > > > Add a FunctionBuilder class -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18372) Create testing infra to test different HMS instances
[
https://issues.apache.org/jira/browse/HIVE-18372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324524#comment-16324524
]
Hive QA commented on HIVE-18372:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905856/HIVE-18372.5.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 11598 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveAndKill
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8597/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8597/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8597/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12905856 - PreCommit-HIVE-Build
> Create testing infra to test different HMS instances
>
>
> Key: HIVE-18372
> URL: https://issues.apache.org/jira/browse/HIVE-18372
> Project: Hive
> Issue Type: Sub-task
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-18372.2.patch, HIVE-18372.3.patch,
> HIVE-18372.5.patch, HIVE-18372.patch
>
>
> Since there will be multiple tests, it would be good to have a good
> infrastructure to help creating those faster, easier.
> This patch will also include the test cases for the Database related methods
> to showcase the infra
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18401) load data should rename files consistent with insert statements (bucketed tables only) Part3
[ https://issues.apache.org/jira/browse/HIVE-18401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18401: -- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the review [~jdere] Pushed to master. > load data should rename files consistent with insert statements (bucketed > tables only) Part3 > > > Key: HIVE-18401 > URL: https://issues.apache.org/jira/browse/HIVE-18401 > Project: Hive > Issue Type: Sub-task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-18401.1.patch, HIVE-18401.2.patch, > HIVE-18401.3.patch, HIVE-18401.4.patch, HIVE-18401.5.patch, HIVE-18401.6.patch > > > As part of this effort, this JIRA tracks updating tests which use load data > files srcbucket20 etc > Following is the mapping of new data files vs old data files > bmj/00_0 —> srcbucket20.txt > bmj/01_0 —> srcbucket21.txt > bmj/02_0 —> srcbucket22.txt > bmj/03_0 —> srcbucket23.txt > bmj1/01_0 —> srcbucket22.txt > bmj2/00_0 —> srcbucket22.txt > bmj2/01_0 —> srcbucket23.txt -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18426) Memory leak in RoutingAppender for every hive operation
[ https://issues.apache.org/jira/browse/HIVE-18426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated HIVE-18426: --- Attachment: HIVE-18426.003.patch > Memory leak in RoutingAppender for every hive operation > --- > > Key: HIVE-18426 > URL: https://issues.apache.org/jira/browse/HIVE-18426 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 3.0.0 > > Attachments: HIVE-18426.001.patch, HIVE-18426.002.patch, > HIVE-18426.003.patch, Screen Shot 2018-01-10 at 9.17.19 AM.png > > > Each new operation creates new entry in the ConcurrentMap in RoutingAppender > but when the operation ends, AppenderControl stored in the map is retrieved > and stopped but the entry in ConcurrentMap is never cleaned up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18350) load data should rename files consistent with insert statements
[ https://issues.apache.org/jira/browse/HIVE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324501#comment-16324501 ] Deepak Jaiswal commented on HIVE-18350: --- [~jdere] [~ekoifman] can you please review? > load data should rename files consistent with insert statements > --- > > Key: HIVE-18350 > URL: https://issues.apache.org/jira/browse/HIVE-18350 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-18350.1.patch, HIVE-18350.2.patch > > > Insert statements create files of format ending with _0, 0001_0 etc. > However, the load data uses the input file name. That results in inconsistent > naming convention which makes SMB joins difficult in some scenarios and may > cause trouble for other types of queries in future. > We need consistent naming convention. > For non-bucketed table, hive renames all the files regardless of how they > were named by the user. > For bucketed table, hive relies on user to name the files matching the bucket > in non-strict mode. Hive assumes that the data belongs to same bucket in a > file. In strict mode, loading bucketed table is disabled. > This will likely affect most of the tests which load data which is pretty > significant due to which it is further divided into two subtasks for smoother > merge. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18350) load data should rename files consistent with insert statements
[ https://issues.apache.org/jira/browse/HIVE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18350: -- Attachment: HIVE-18350.2.patch Patch with all the code changes and new tests. > load data should rename files consistent with insert statements > --- > > Key: HIVE-18350 > URL: https://issues.apache.org/jira/browse/HIVE-18350 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-18350.1.patch, HIVE-18350.2.patch > > > Insert statements create files of format ending with _0, 0001_0 etc. > However, the load data uses the input file name. That results in inconsistent > naming convention which makes SMB joins difficult in some scenarios and may > cause trouble for other types of queries in future. > We need consistent naming convention. > For non-bucketed table, hive renames all the files regardless of how they > were named by the user. > For bucketed table, hive relies on user to name the files matching the bucket > in non-strict mode. Hive assumes that the data belongs to same bucket in a > file. In strict mode, loading bucketed table is disabled. > This will likely affect most of the tests which load data which is pretty > significant due to which it is further divided into two subtasks for smoother > merge. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17982) Move metastore specific itests
[ https://issues.apache.org/jira/browse/HIVE-17982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324483#comment-16324483 ] Alan Gates commented on HIVE-17982: --- Failures all have age greater than 1 except first three, all of which are some variation of out of memory/GC errors. I don't think moving tests around caused that. > Move metastore specific itests > -- > > Key: HIVE-17982 > URL: https://issues.apache.org/jira/browse/HIVE-17982 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Alan Gates >Assignee: Alan Gates > Labels: pull-request-available > Attachments: HIVE-17982.2.patch, HIVE-17982.patch > > > There are a number of tests in itests/hive-unit/.../metastore that are > metastore specific. I suspect they were initially placed in itests only > because the metastore pulling in a few plugins from ql. > Given that we need to be able to release the metastore separately, we need to > be able to test it completely as a standalone entity. So I propose to move a > number of the itests over into standalone-metastore. I will only move tests > that are isolated to the metastore. Anything that tests wider functionality > I plan to leave in itests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18401) load data should rename files consistent with insert statements (bucketed tables only) Part3
[ https://issues.apache.org/jira/browse/HIVE-18401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324475#comment-16324475 ] Jason Dere commented on HIVE-18401: --- +1, thanks for creating the mapping for 1 and 2-bucket versions of the tables. > load data should rename files consistent with insert statements (bucketed > tables only) Part3 > > > Key: HIVE-18401 > URL: https://issues.apache.org/jira/browse/HIVE-18401 > Project: Hive > Issue Type: Sub-task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-18401.1.patch, HIVE-18401.2.patch, > HIVE-18401.3.patch, HIVE-18401.4.patch, HIVE-18401.5.patch, HIVE-18401.6.patch > > > As part of this effort, this JIRA tracks updating tests which use load data > files srcbucket20 etc > Following is the mapping of new data files vs old data files > bmj/00_0 —> srcbucket20.txt > bmj/01_0 —> srcbucket21.txt > bmj/02_0 —> srcbucket22.txt > bmj/03_0 —> srcbucket23.txt > bmj1/01_0 —> srcbucket22.txt > bmj2/00_0 —> srcbucket22.txt > bmj2/01_0 —> srcbucket23.txt -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18416) Initial support for TABLE function
[
https://issues.apache.org/jira/browse/HIVE-18416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324445#comment-16324445
]
Ashutosh Chauhan commented on HIVE-18416:
-
Simple insert into table values{} is still resulting in launching of job as
earlier which can be avoided by optimizing that job to FetchTask.
I guess SimpleFetchOptimizer doesn't fire on insert queries and it doesn't
handle UDTF which it now needs to handle. Can you please file a follow-up jira
for it.
Are failures {{TestDbTxnManager2}} related? +1, if not.
> Initial support for TABLE function
> --
>
> Key: HIVE-18416
> URL: https://issues.apache.org/jira/browse/HIVE-18416
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Labels: TODOC3.0
> Attachments: HIVE-18416.01.patch, HIVE-18416.patch
>
>
> Include support for TABLE function derived table. In SQL standard, it is
> defined as:
> {noformat}
> ::=
> TABLE
> {noformat}
> Further, include limited support for its usage as LATERAL derived table.
> This will allow Hive to execute queries such as:
> {code}
> SELECT tf.col1, tf.col2, tf.col3
> FROM
> TABLE(VALUES('A', 10, 30.0),('B', 20, 30.0)) AS tf(col1, col2, col3);
> SELECT tf.col1, tf.col2, tf.col3
> FROM
> (SELECT key, value FROM src) t,
> LATERAL TABLE(VALUES('A', 10, t.key),('B', 20, t.key)) AS tf(col1, col2,
> col3);
> {code}
> The idea is to rely on AST rewriting for such cases, as TABLE can be
> implemented using {{inline}} UDTF with an {{array}} of {{structs}}
> representing the rows in the {{VALUES}} clause.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18430) Add new determinism category for runtime constants (current_date, current_timestamp)
[ https://issues.apache.org/jira/browse/HIVE-18430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-18430: -- Attachment: HIVE-18430.2.patch Updating patch - missed a place in ConstantPropagateProcFactory where the deterministic UDFType annotation was being used. This should fix the relevant failures. Some of the other failures do not appear related. I opened HIVE-18446 for bucket_map_join_tez1. > Add new determinism category for runtime constants (current_date, > current_timestamp) > > > Key: HIVE-18430 > URL: https://issues.apache.org/jira/browse/HIVE-18430 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-18430.1.patch, HIVE-18430.2.patch > > > Add a new piece of metadata to the UDFs to specify if whether a UDF is a > runtime constant. Runtime constants also exist in SQL Server, and this is > similar to Postgres' concept of STABLE functions. This metadata may be useful > for materialized views and query caching. > Some Hive functions such as the ones listed below are currently labelled as > deterministic, but really are runtime constants: > current_timestamp > current_date > current_user > current_database > The values for these functions are not deterministic between different > queries - for example current_timestamp will most likely be different every > query executed. This makes these functions ineligible for things like > materialized views or cached query results. > However the value for the current_timestamp should not change during the life > of a single query, which allows these values to be used in optimizations such > as constant folding. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18372) Create testing infra to test different HMS instances
[
https://issues.apache.org/jira/browse/HIVE-18372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324430#comment-16324430
]
Hive QA commented on HIVE-18372:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / fd4e222 |
| Default Java | 1.8.0_111 |
| modules | C: standalone-metastore U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8597/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Create testing infra to test different HMS instances
>
>
> Key: HIVE-18372
> URL: https://issues.apache.org/jira/browse/HIVE-18372
> Project: Hive
> Issue Type: Sub-task
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-18372.2.patch, HIVE-18372.3.patch,
> HIVE-18372.5.patch, HIVE-18372.patch
>
>
> Since there will be multiple tests, it would be good to have a good
> infrastructure to help creating those faster, easier.
> This patch will also include the test cases for the Database related methods
> to showcase the infra
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-14162) Allow disabling of long running job on Hive On Spark On YARN
[ https://issues.apache.org/jira/browse/HIVE-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324429#comment-16324429 ] BELUGA BEHR commented on HIVE-14162: The size of the driver is only configurable before the ApplicationMaster is running. Once it's running, my understanding is that there is no way to change it. Setting Executors down to 0 might be a good option. I have contributed my thoughts to this ticket because I have spoken with several institutions that have complained about giving up resources to people that are not even using them, especially in a 24x7 cluster where users forgetting to log out in one time zone uses resources available to another time zone. I cannot however say that setting the minimum to 0 would work for everyone: there is still the ApplicaitonMaster. > Allow disabling of long running job on Hive On Spark On YARN > > > Key: HIVE-14162 > URL: https://issues.apache.org/jira/browse/HIVE-14162 > Project: Hive > Issue Type: New Feature > Components: Spark >Reporter: Thomas Scott >Assignee: Aihua Xu >Priority: Minor > Attachments: HIVE-14162.1.patch > > > Hive On Spark launches a long running process on the first query to handle > all queries for that user session. In some use cases this is not desired, for > instance when using Hue with large intervals between query executions. > Could we have a property that would cause long running spark jobs to be > terminated after each query execution and started again for the next one? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17833) Publish split generation counters
[ https://issues.apache.org/jira/browse/HIVE-17833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324426#comment-16324426 ] Sergey Shelukhin commented on HIVE-17833: - The files might need to be updated... > Publish split generation counters > - > > Key: HIVE-17833 > URL: https://issues.apache.org/jira/browse/HIVE-17833 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17833.1.patch, HIVE-17833.2.patch, > HIVE-17833.3.patch, HIVE-17833.4.patch, HIVE-17833.5.patch > > > With TEZ-3856, tez counters are exposed via input initializers which can be > used to publish split generation counters. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18422) Vectorized input format should not be used when input format is excluded and row.serde is enabled
[
https://issues.apache.org/jira/browse/HIVE-18422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324409#comment-16324409
]
Matt McCline commented on HIVE-18422:
-
[~vihangk1] Yes, I'd appreciate the patch being posted to RB. Thanks
> Vectorized input format should not be used when input format is excluded and
> row.serde is enabled
> -
>
> Key: HIVE-18422
> URL: https://issues.apache.org/jira/browse/HIVE-18422
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-18422.01.patch, HIVE-18422.02.patch
>
>
> HIVE-17534 introduced a config which gives a capability to exclude certain
> inputformat from vectorized execution without affecting other input formats.
> If an input format is excluded and row.serde is enabled at the same time,
> vectorizer still sets the {{useVectorizedInputFormat}} to true which causes
> Vectorized readers to be used in row.serde mode.
> In order to reproduce:
> {noformat}
> set hive.fetch.task.conversion=none;
> set hive.vectorized.use.row.serde.deserialize=true;
> set hive.vectorized.use.vector.serde.deserialize=true;
> set hive.vectorized.execution.enabled=true;
> set hive.vectorized.execution.reduce.enabled=true;
> set hive.vectorized.row.serde.inputformat.excludes=;
> -- SORT_QUERY_RESULTS
> -- exclude MapredParquetInputFormat from vectorization, this should cause
> mapwork vectorization to be disabled
> set
> hive.vectorized.input.format.excludes=org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat,org.apache.hadoop.hive.ql.io.orc.OrcInputFormat;
> set hive.vectorized.use.vectorized.input.format=true;
> create table orcTbl (t1 tinyint, t2 tinyint)
> stored as orc;
> insert into orcTbl values (54, 9), (-104, 25), (-112, 24);
> explain vectorization select t1, t2, (t1+t2) from orcTbl where (t1+t2) > 10;
> select t1, t2, (t1+t2) from orcTbl where (t1+t2) > 10;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-14162) Allow disabling of long running job on Hive On Spark On YARN
[ https://issues.apache.org/jira/browse/HIVE-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324408#comment-16324408 ] Xuefu Zhang commented on HIVE-14162: The size of the driver is configurable. Minimum number of executors can be 0. Would this be a problem for you? > Allow disabling of long running job on Hive On Spark On YARN > > > Key: HIVE-14162 > URL: https://issues.apache.org/jira/browse/HIVE-14162 > Project: Hive > Issue Type: New Feature > Components: Spark >Reporter: Thomas Scott >Assignee: Aihua Xu >Priority: Minor > Attachments: HIVE-14162.1.patch > > > Hive On Spark launches a long running process on the first query to handle > all queries for that user session. In some use cases this is not desired, for > instance when using Hue with large intervals between query executions. > Could we have a property that would cause long running spark jobs to be > terminated after each query execution and started again for the next one? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-14162) Allow disabling of long running job on Hive On Spark On YARN
[ https://issues.apache.org/jira/browse/HIVE-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324405#comment-16324405 ] BELUGA BEHR commented on HIVE-14162: [~xuefuz] I am familiar with dynamic allocation. So at a minimum the client will be reserving a container for the ApplicationMaster/Driver (which can be up to 12GB in some environments) and the default value of 1 Executor (which can be up to 32GB in some environments). Even if the user has stepped away from the Hive client for some amount of time. These are resources which could be used by other users and is compounded by the number of users one environment supports. Thanks for the consideration. > Allow disabling of long running job on Hive On Spark On YARN > > > Key: HIVE-14162 > URL: https://issues.apache.org/jira/browse/HIVE-14162 > Project: Hive > Issue Type: New Feature > Components: Spark >Reporter: Thomas Scott >Assignee: Aihua Xu >Priority: Minor > Attachments: HIVE-14162.1.patch > > > Hive On Spark launches a long running process on the first query to handle > all queries for that user session. In some use cases this is not desired, for > instance when using Hue with large intervals between query executions. > Could we have a property that would cause long running spark jobs to be > terminated after each query execution and started again for the next one? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18231) validate resource plan - part 2 - validate action and trigger expressions
[
https://issues.apache.org/jira/browse/HIVE-18231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324403#comment-16324403
]
Hive QA commented on HIVE-18231:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905843/HIVE-18231.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 11457 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_bmj_schema_evolution]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=177)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.org.apache.hadoop.hive.cli.TestTezPerfCliDriver
(batchId=245)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8596/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8596/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8596/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12905843 - PreCommit-HIVE-Build
> validate resource plan - part 2 - validate action and trigger expressions
> -
>
> Key: HIVE-18231
> URL: https://issues.apache.org/jira/browse/HIVE-18231
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Harish Jaiprakash
> Attachments: HIVE-18231.01.patch
>
>
> The expression factories can probably be moved to metastore or common. Action
> should be easy to verify; counter names in triggers cannot be verified, but
> at least the expression form can be
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-14162) Allow disabling of long running job on Hive On Spark On YARN
[ https://issues.apache.org/jira/browse/HIVE-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324349#comment-16324349 ] Xuefu Zhang commented on HIVE-14162: [~belugabehr], Spark on Yarn is powered by a feature call dynamic allocation, which is strongly recommended in a multi-tenancy or cost sensitive env. This is something that you might have missed. With it, unused executors are returned to the cluster so others can use. SparkContext is stateful, and can be considered as an extension of HIve session. If you allow a SparkSession to time out, then part of the state is gone. In that case, you might just as well let the whole session expire. What you requested isn't completely off, but I'd like to see if existing features are enough to reach what you want achieve. > Allow disabling of long running job on Hive On Spark On YARN > > > Key: HIVE-14162 > URL: https://issues.apache.org/jira/browse/HIVE-14162 > Project: Hive > Issue Type: New Feature > Components: Spark >Reporter: Thomas Scott >Assignee: Aihua Xu >Priority: Minor > Attachments: HIVE-14162.1.patch > > > Hive On Spark launches a long running process on the first query to handle > all queries for that user session. In some use cases this is not desired, for > instance when using Hue with large intervals between query executions. > Could we have a property that would cause long running spark jobs to be > terminated after each query execution and started again for the next one? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18410) [Performance][Avro] Reading flat Avro tables is very expensive in Hive
[
https://issues.apache.org/jira/browse/HIVE-18410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ratandeep Ratti updated HIVE-18410:
---
Affects Version/s: 3.0.0
Status: In Progress (was: Patch Available)
> [Performance][Avro] Reading flat Avro tables is very expensive in Hive
> --
>
> Key: HIVE-18410
> URL: https://issues.apache.org/jira/browse/HIVE-18410
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 2.3.2, 3.0.0
>Reporter: Ratandeep Ratti
>Assignee: Ratandeep Ratti
> Fix For: 2.3.2
>
> Attachments: HIVE-18410.patch, HIVE-18410_1.patch,
> profiling_with_patch.nps, profiling_with_patch.png,
> profiling_without_patch.nps, profiling_without_patch.png
>
>
> There's a performance penalty when reading flat [no nested fields] Avro
> tables. When reading the same flat dataset in Pig, it takes half the time.
> On profiling, a lot of time is spent in
> {{AvroDeserializer.deserializeSingleItemNullableUnion()}}. The bulk of the
> time is spent in GenericData.get().resolveUnion(), which calls
> GenericData.getSchemaName(Object datum), which does a lot of instanceof
> checks. This could be simplified with performance benefits. A approach is
> described in this patch which almost halves the runtime.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18410) [Performance][Avro] Reading flat Avro tables is very expensive in Hive
[
https://issues.apache.org/jira/browse/HIVE-18410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324342#comment-16324342
]
Ratandeep Ratti commented on HIVE-18410:
https://reviews.apache.org/r/65036
> [Performance][Avro] Reading flat Avro tables is very expensive in Hive
> --
>
> Key: HIVE-18410
> URL: https://issues.apache.org/jira/browse/HIVE-18410
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.3.2
>Reporter: Ratandeep Ratti
>Assignee: Ratandeep Ratti
> Fix For: 2.3.2
>
> Attachments: HIVE-18410.patch, HIVE-18410_1.patch,
> profiling_with_patch.nps, profiling_with_patch.png,
> profiling_without_patch.nps, profiling_without_patch.png
>
>
> There's a performance penalty when reading flat [no nested fields] Avro
> tables. When reading the same flat dataset in Pig, it takes half the time.
> On profiling, a lot of time is spent in
> {{AvroDeserializer.deserializeSingleItemNullableUnion()}}. The bulk of the
> time is spent in GenericData.get().resolveUnion(), which calls
> GenericData.getSchemaName(Object datum), which does a lot of instanceof
> checks. This could be simplified with performance benefits. A approach is
> described in this patch which almost halves the runtime.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18410) [Performance][Avro] Reading flat Avro tables is very expensive in Hive
[
https://issues.apache.org/jira/browse/HIVE-18410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ratandeep Ratti updated HIVE-18410:
---
Status: Patch Available (was: In Progress)
> [Performance][Avro] Reading flat Avro tables is very expensive in Hive
> --
>
> Key: HIVE-18410
> URL: https://issues.apache.org/jira/browse/HIVE-18410
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 2.3.2, 3.0.0
>Reporter: Ratandeep Ratti
>Assignee: Ratandeep Ratti
> Fix For: 2.3.2
>
> Attachments: HIVE-18410.patch, HIVE-18410_1.patch,
> profiling_with_patch.nps, profiling_with_patch.png,
> profiling_without_patch.nps, profiling_without_patch.png
>
>
> There's a performance penalty when reading flat [no nested fields] Avro
> tables. When reading the same flat dataset in Pig, it takes half the time.
> On profiling, a lot of time is spent in
> {{AvroDeserializer.deserializeSingleItemNullableUnion()}}. The bulk of the
> time is spent in GenericData.get().resolveUnion(), which calls
> GenericData.getSchemaName(Object datum), which does a lot of instanceof
> checks. This could be simplified with performance benefits. A approach is
> described in this patch which almost halves the runtime.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18410) [Performance][Avro] Reading flat Avro tables is very expensive in Hive
[
https://issues.apache.org/jira/browse/HIVE-18410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ratandeep Ratti updated HIVE-18410:
---
Attachment: HIVE-18410_1.patch
> [Performance][Avro] Reading flat Avro tables is very expensive in Hive
> --
>
> Key: HIVE-18410
> URL: https://issues.apache.org/jira/browse/HIVE-18410
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 2.3.2
>Reporter: Ratandeep Ratti
>Assignee: Ratandeep Ratti
> Fix For: 2.3.2
>
> Attachments: HIVE-18410.patch, HIVE-18410_1.patch,
> profiling_with_patch.nps, profiling_with_patch.png,
> profiling_without_patch.nps, profiling_without_patch.png
>
>
> There's a performance penalty when reading flat [no nested fields] Avro
> tables. When reading the same flat dataset in Pig, it takes half the time.
> On profiling, a lot of time is spent in
> {{AvroDeserializer.deserializeSingleItemNullableUnion()}}. The bulk of the
> time is spent in GenericData.get().resolveUnion(), which calls
> GenericData.getSchemaName(Object datum), which does a lot of instanceof
> checks. This could be simplified with performance benefits. A approach is
> described in this patch which almost halves the runtime.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18231) validate resource plan - part 2 - validate action and trigger expressions
[
https://issues.apache.org/jira/browse/HIVE-18231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324295#comment-16324295
]
Hive QA commented on HIVE-18231:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / fd4e222 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8596/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> validate resource plan - part 2 - validate action and trigger expressions
> -
>
> Key: HIVE-18231
> URL: https://issues.apache.org/jira/browse/HIVE-18231
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Harish Jaiprakash
> Attachments: HIVE-18231.01.patch
>
>
> The expression factories can probably be moved to metastore or common. Action
> should be easy to verify; counter names in triggers cannot be verified, but
> at least the expression form can be
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18445) qtests: auto_join25.q fails permanently
[ https://issues.apache.org/jira/browse/HIVE-18445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor reassigned HIVE-18445: --- > qtests: auto_join25.q fails permanently > --- > > Key: HIVE-18445 > URL: https://issues.apache.org/jira/browse/HIVE-18445 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor > > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] > (batchId=72) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18341) Add repl load support for adding "raw" namespace for TDE with same encryption keys
[
https://issues.apache.org/jira/browse/HIVE-18341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324285#comment-16324285
]
Hive QA commented on HIVE-18341:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905802/HIVE-18341.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 11568 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample_islocalmode_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=254)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSchedulerQueue.testQueueMappingCheckDisabled
(batchId=239)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8595/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8595/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8595/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12905802 - PreCommit-HIVE-Build
> Add repl load support for adding "raw" namespace for TDE with same encryption
> keys
> --
>
> Key: HIVE-18341
> URL: https://issues.apache.org/jira/browse/HIVE-18341
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
> Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18341.0.patch, HIVE-18341.1.patch,
> HIVE-18341.2.patch, HIVE-18341.3.patch
>
>
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html#Running_as_the_superuser
> "a new virtual path prefix, /.reserved/raw/, that gives superusers direct
> access to the underlying block data in the filesystem. This allows superusers
> to distcp data without needing having access to encryption keys, and also
> avoids the overhead of decrypting and re-encrypting data."
> We need to introduce a new option in "Repl Load" command that will change the
> files being copied in distcp to have this "/.reserved/raw/" namespace before
> the file paths.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18051) qfiles: dataset support
[
https://issues.apache.org/jira/browse/HIVE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-18051:
Attachment: HIVE-18051.05.patch
> qfiles: dataset support
> ---
>
> Key: HIVE-18051
> URL: https://issues.apache.org/jira/browse/HIVE-18051
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
> Attachments: HIVE-18051.01.patch, HIVE-18051.02.patch,
> HIVE-18051.03.patch, HIVE-18051.04.patch, HIVE-18051.05.patch
>
>
> it would be great to have some kind of test dataset support; currently there
> is the {{q_test_init.sql}} which is quite large; and I'm often override it
> with an invalid string; because I write independent qtests most of the time -
> and the load of {{src}} and other tables are just a waste of time for me ;
> not to mention that the loading of those tables may also trigger breakpoints
> - which is a bit annoying.
> Most of the tests are "only" using the {{src}} table and possibly 2 others;
> however the main init script contains a bunch of tables - meanwhile there are
> quite few other tests which could possibly also benefit from a more general
> feature; for example the creation of {{bucket_small}} is present in 20 q
> files.
> the proposal would be to enable the qfiles to be annotated with metadata like
> datasets:
> {code}
> --! qt:dataset:src,bucket_small
> {code}
> proposal for storing a dataset:
> * the loader script would be at: {{data/datasets/__NAME__/load.hive.sql}}
> * the table data could be stored under that location
> a draft about this; and other qfiles related ideas:
> https://docs.google.com/document/d/1KtcIx8ggL9LxDintFuJo8NQuvNWkmtvv_ekbWrTLNGc/edit?usp=sharing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18341) Add repl load support for adding "raw" namespace for TDE with same encryption keys
[
https://issues.apache.org/jira/browse/HIVE-18341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324243#comment-16324243
]
Hive QA commented on HIVE-18341:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} common: The patch generated 3 new + 938 unchanged - 0
fixed = 941 total (was 938) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
32s{color} | {color:red} ql: The patch generated 12 new + 5 unchanged - 1 fixed
= 17 total (was 6) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / fd4e222 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8595/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8595/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8595/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add repl load support for adding "raw" namespace for TDE with same encryption
> keys
> --
>
> Key: HIVE-18341
> URL: https://issues.apache.org/jira/browse/HIVE-18341
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
> Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18341.0.patch, HIVE-18341.1.patch,
> HIVE-18341.2.patch, HIVE-18341.3.patch
>
>
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html#Running_as_the_superuser
> "a new virtual path prefix, /.reserved/raw/, that gives superusers direct
> access to the underlying block data in the filesystem. This allows superusers
> to distcp data without needing having access to encryption keys, and also
> avoids the overhead of decrypting and re-encrypting data."
> We need to introduce a new option in "Repl Load" command that will change the
> files being copied in distcp to have
[jira] [Commented] (HIVE-18341) Add repl load support for adding "raw" namespace for TDE with same encryption keys
[
https://issues.apache.org/jira/browse/HIVE-18341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324205#comment-16324205
]
Hive QA commented on HIVE-18341:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905802/HIVE-18341.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 11568 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=254)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8594/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8594/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8594/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12905802 - PreCommit-HIVE-Build
> Add repl load support for adding "raw" namespace for TDE with same encryption
> keys
> --
>
> Key: HIVE-18341
> URL: https://issues.apache.org/jira/browse/HIVE-18341
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
> Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18341.0.patch, HIVE-18341.1.patch,
> HIVE-18341.2.patch, HIVE-18341.3.patch
>
>
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html#Running_as_the_superuser
> "a new virtual path prefix, /.reserved/raw/, that gives superusers direct
> access to the underlying block data in the filesystem. This allows superusers
> to distcp data without needing having access to encryption keys, and also
> avoids the overhead of decrypting and re-encrypting data."
> We need to introduce a new option in "Repl Load" command that will change the
> files being copied in distcp to have this "/.reserved/raw/" namespace before
> the file paths.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18341) Add repl load support for adding "raw" namespace for TDE with same encryption keys
[
https://issues.apache.org/jira/browse/HIVE-18341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324150#comment-16324150
]
Hive QA commented on HIVE-18341:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} common: The patch generated 3 new + 938 unchanged - 0
fixed = 941 total (was 938) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
31s{color} | {color:red} ql: The patch generated 12 new + 5 unchanged - 1 fixed
= 17 total (was 6) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / fd4e222 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8594/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8594/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8594/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add repl load support for adding "raw" namespace for TDE with same encryption
> keys
> --
>
> Key: HIVE-18341
> URL: https://issues.apache.org/jira/browse/HIVE-18341
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
> Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18341.0.patch, HIVE-18341.1.patch,
> HIVE-18341.2.patch, HIVE-18341.3.patch
>
>
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html#Running_as_the_superuser
> "a new virtual path prefix, /.reserved/raw/, that gives superusers direct
> access to the underlying block data in the filesystem. This allows superusers
> to distcp data without needing having access to encryption keys, and also
> avoids the overhead of decrypting and re-encrypting data."
> We need to introduce a new option in "Repl Load" command that will change the
> files being copied in distcp to have
[jira] [Commented] (HIVE-14162) Allow disabling of long running job on Hive On Spark On YARN
[ https://issues.apache.org/jira/browse/HIVE-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324121#comment-16324121 ] BELUGA BEHR commented on HIVE-14162: [~xuefuz] Thanks for the review. The issue here is that holding open a Hive session is relatively lightweight. However, holding open a Spark context is very heavy weight on the cluster and reserves resources from YARN that are not being utilized and cannot be used by other users. For convenience, a user may want to preserve their Hive session, with their session configurations in place, the entire work day but, for example, while they go out to lunch for 30 minutes, would like to allow the Spark resources to be returned to the cluster. This would require a Hive session timeout of 8-12 hours but a Spark context timeout of 15 minutes. If there are 100 employees using Hue for example, that's 200 containers reserved and not being used (1 for AM and 1 for Executor). > Allow disabling of long running job on Hive On Spark On YARN > > > Key: HIVE-14162 > URL: https://issues.apache.org/jira/browse/HIVE-14162 > Project: Hive > Issue Type: New Feature > Components: Spark >Reporter: Thomas Scott >Assignee: Aihua Xu >Priority: Minor > Attachments: HIVE-14162.1.patch > > > Hive On Spark launches a long running process on the first query to handle > all queries for that user session. In some use cases this is not desired, for > instance when using Hue with large intervals between query executions. > Could we have a property that would cause long running spark jobs to be > terminated after each query execution and started again for the next one? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17751) Separate HMS Client and HMS server into separate sub-modules
[
https://issues.apache.org/jira/browse/HIVE-17751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324109#comment-16324109
]
Hive QA commented on HIVE-17751:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905779/HIVE-17751.01-standalone-metastore.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8593/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8593/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8593/
Messages:
{noformat}
This message was trimmed, see log for full details
location: class org.apache.hadoop.hive.druid.DruidStorageHandler
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:[452,31]
cannot find symbol
symbol: class Table
location: class org.apache.hadoop.hive.druid.DruidStorageHandler
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:[452,71]
cannot find symbol
symbol: class MetaException
location: class org.apache.hadoop.hive.druid.DruidStorageHandler
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:[484,33]
cannot find symbol
symbol: class Table
location: class org.apache.hadoop.hive.druid.DruidStorageHandler
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:[484,72]
cannot find symbol
symbol: class MetaException
location: class org.apache.hadoop.hive.druid.DruidStorageHandler
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:[494,30]
cannot find symbol
symbol: class Table
location: class org.apache.hadoop.hive.druid.DruidStorageHandler
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:[494,69]
cannot find symbol
symbol: class MetaException
location: class org.apache.hadoop.hive.druid.DruidStorageHandler
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:[499,35]
cannot find symbol
symbol: class Table
location: class org.apache.hadoop.hive.druid.DruidStorageHandler
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:[499,74]
cannot find symbol
symbol: class MetaException
location: class org.apache.hadoop.hive.druid.DruidStorageHandler
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile (default-compile)
on project hive-jdbc-handler: Compilation failure: Compilation failure:
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcStorageHandler.java:[18,40]
cannot find symbol
[ERROR] symbol: class HiveMetaHook
[ERROR] location: package org.apache.hadoop.hive.metastore
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcStorageHandler.java:[75,10]
cannot find symbol
[ERROR] symbol: class HiveMetaHook
[ERROR] location: class org.apache.hive.storage.jdbc.JdbcStorageHandler
[ERROR] -> [Help 1]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile (default-compile)
on project hive-hbase-handler: Compilation failure: Compilation failure:
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java:[42,40]
cannot find symbol
[ERROR] symbol: class HiveMetaHook
[ERROR] location: package org.apache.hadoop.hive.metastore
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java:[43,44]
package org.apache.hadoop.hive.metastore.api does not exist
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java:[44,44]
package org.apache.hadoop.hive.metastore.api does not exist
[ERROR]
/data/hiveptest/working/apache-github-standalone-metastore-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java:[152,10]
cannot find symbol
[ERROR] symbol: class HiveMetaHook
[ERROR] location: class
[jira] [Commented] (HIVE-17833) Publish split generation counters
[
https://issues.apache.org/jira/browse/HIVE-17833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324105#comment-16324105
]
Hive QA commented on HIVE-17833:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905775/HIVE-17833.5.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 33 failed/errored test(s), 11576 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters1]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dp_counter_mm]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dp_counter_non_mm]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=121)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=214)
org.apache.hadoop.hive.ql.TestAcidOnTez.testGetSplitsLocks (batchId=223)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=254)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=226)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testDataTypes (batchId=233)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testEscapedStrings (batchId=233)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd
(batchId=233)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testNonAsciiStrings (batchId=233)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveBackKill
(batchId=236)
org.apache.hive.jdbc.TestTriggersNoTezSessionPool.testTriggerTotalTasks
(batchId=233)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=236)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerVertexInputSplits
(batchId=236)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=233)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8592/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8592/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8592/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 33 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12905775 - PreCommit-HIVE-Build
> Publish split generation counters
> -
>
> Key: HIVE-17833
> URL: https://issues.apache.org/jira/browse/HIVE-17833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17833.1.patch, HIVE-17833.2.patch,
> HIVE-17833.3.patch, HIVE-17833.4.patch, HIVE-17833.5.patch
>
>
> With TEZ-3856, tez counters are exposed via input initializers which can be
> used to publish split generation counters.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18309) qtests: smb_mapjoin_19.q breaks bucketsortoptimize_insert_2.q
[
https://issues.apache.org/jira/browse/HIVE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-18309:
Attachment: HIVE-18309.01.patch
> qtests: smb_mapjoin_19.q breaks bucketsortoptimize_insert_2.q
> -
>
> Key: HIVE-18309
> URL: https://issues.apache.org/jira/browse/HIVE-18309
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
> Attachments: HIVE-18309.01.patch
>
>
> {code}
> mvn install -q -am -pl itests/qtest -DskipSparkTests -pl itests/qtest
> -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=smb_mapjoin_19.q,bucketsortoptimize_insert_2.q
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18309) qtests: smb_mapjoin_19.q breaks bucketsortoptimize_insert_2.q
[
https://issues.apache.org/jira/browse/HIVE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-18309:
Status: Patch Available (was: Open)
> qtests: smb_mapjoin_19.q breaks bucketsortoptimize_insert_2.q
> -
>
> Key: HIVE-18309
> URL: https://issues.apache.org/jira/browse/HIVE-18309
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
> Attachments: HIVE-18309.01.patch
>
>
> {code}
> mvn install -q -am -pl itests/qtest -DskipSparkTests -pl itests/qtest
> -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=smb_mapjoin_19.q,bucketsortoptimize_insert_2.q
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-18130) Update table path to storage description parameter when alter a table
[ https://issues.apache.org/jira/browse/HIVE-18130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324040#comment-16324040 ] Zhe Sun edited comment on HIVE-18130 at 1/12/18 2:45 PM: - Dear committer, I fixed the related test failure `org.apache.hive.hcatalog.listener.TestDbNotificationListener.alterTable`. The rest test failures are unrelated. Do I need a test case for the piece of code I changed? Existing `testAlterTable` function doesn't test SerdrInfo field. Can you please help me a bit? Thanks! was (Author: ymwdalex): I fixed the related test failure `org.apache.hive.hcatalog.listener.TestDbNotificationListener.alterTable`. The rest test failures are unrelated. > Update table path to storage description parameter when alter a table > - > > Key: HIVE-18130 > URL: https://issues.apache.org/jira/browse/HIVE-18130 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Zhe Sun > Attachments: HIVE-18130.02.patch, HIVE-18130.patch > > > When an managed table is created by Spark, table path information is not only > store in `location` field (first figure), but also in `parameters.path` > fields (second figure). > The `parameters.path` should be also checked and updated when alter a table > Example explaining this issue can be found here > https://github.com/apache/hive/pull/273 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18130) Update table path to storage description parameter when alter a table
[ https://issues.apache.org/jira/browse/HIVE-18130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Sun updated HIVE-18130: --- Status: In Progress (was: Patch Available) I fixed the related test failure `org.apache.hive.hcatalog.listener.TestDbNotificationListener.alterTable`. The rest test failures are unrelated. > Update table path to storage description parameter when alter a table > - > > Key: HIVE-18130 > URL: https://issues.apache.org/jira/browse/HIVE-18130 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Zhe Sun > Attachments: HIVE-18130.02.patch, HIVE-18130.patch > > > When an managed table is created by Spark, table path information is not only > store in `location` field (first figure), but also in `parameters.path` > fields (second figure). > The `parameters.path` should be also checked and updated when alter a table > Example explaining this issue can be found here > https://github.com/apache/hive/pull/273 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18130) Update table path to storage description parameter when alter a table
[ https://issues.apache.org/jira/browse/HIVE-18130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Sun updated HIVE-18130: --- Attachment: HIVE-18130.02.patch > Update table path to storage description parameter when alter a table > - > > Key: HIVE-18130 > URL: https://issues.apache.org/jira/browse/HIVE-18130 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Zhe Sun > Attachments: HIVE-18130.02.patch, HIVE-18130.patch > > > When an managed table is created by Spark, table path information is not only > store in `location` field (first figure), but also in `parameters.path` > fields (second figure). > The `parameters.path` should be also checked and updated when alter a table > Example explaining this issue can be found here > https://github.com/apache/hive/pull/273 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17833) Publish split generation counters
[
https://issues.apache.org/jira/browse/HIVE-17833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324020#comment-16324020
]
Hive QA commented on HIVE-17833:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
34s{color} | {color:red} ql: The patch generated 15 new + 323 unchanged - 2
fixed = 338 total (was 325) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / fd4e222 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8592/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql itests itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8592/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Publish split generation counters
> -
>
> Key: HIVE-17833
> URL: https://issues.apache.org/jira/browse/HIVE-17833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17833.1.patch, HIVE-17833.2.patch,
> HIVE-17833.3.patch, HIVE-17833.4.patch, HIVE-17833.5.patch
>
>
> With TEZ-3856, tez counters are exposed via input initializers which can be
> used to publish split generation counters.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18314) qtests: semijoin_hint.q breaks hybridgrace_hashjoin_2.q
[
https://issues.apache.org/jira/browse/HIVE-18314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-18314:
Attachment: HIVE-18314.01.patch
> qtests: semijoin_hint.q breaks hybridgrace_hashjoin_2.q
>
>
> Key: HIVE-18314
> URL: https://issues.apache.org/jira/browse/HIVE-18314
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
> Attachments: HIVE-18314.01.patch
>
>
> {code}
> mvn install -q -am -pl itests/qtest -DskipSparkTests
> -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=semijoin_hint.q,hybridgrace_hashjoin_2.q
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18314) qtests: semijoin_hint.q breaks hybridgrace_hashjoin_2.q
[
https://issues.apache.org/jira/browse/HIVE-18314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-18314:
Status: Patch Available (was: Open)
> qtests: semijoin_hint.q breaks hybridgrace_hashjoin_2.q
>
>
> Key: HIVE-18314
> URL: https://issues.apache.org/jira/browse/HIVE-18314
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
> Attachments: HIVE-18314.01.patch
>
>
> {code}
> mvn install -q -am -pl itests/qtest -DskipSparkTests
> -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=semijoin_hint.q,hybridgrace_hashjoin_2.q
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18061) q.outs: be more selective with masking hdfs paths
[ https://issues.apache.org/jira/browse/HIVE-18061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-18061: Attachment: HIVE-18061.07.patch > q.outs: be more selective with masking hdfs paths > - > > Key: HIVE-18061 > URL: https://issues.apache.org/jira/browse/HIVE-18061 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Laszlo Bodor > Attachments: HIVE-18061.01.patch, HIVE-18061.02.patch, > HIVE-18061.03.patch, HIVE-18061.04.patch, HIVE-18061.05.patch, > HIVE-18061.06.patch, HIVE-18061.07.patch > > > currently any line which contains a path which looks like an hdfs location is > replaced with a "masked pattern was here"... > it might be releavant to record these messages; since even an exception > message might contain an hdfs location > noticed in > HIVE-18012 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17982) Move metastore specific itests
[
https://issues.apache.org/jira/browse/HIVE-17982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323982#comment-16323982
]
Hive QA commented on HIVE-17982:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905773/HIVE-17982.2.patch
{color:green}SUCCESS:{color} +1 due to 28 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 11568 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[repl_load_requires_admin]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8591/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8591/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8591/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 19 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12905773 - PreCommit-HIVE-Build
> Move metastore specific itests
> --
>
> Key: HIVE-17982
> URL: https://issues.apache.org/jira/browse/HIVE-17982
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Labels: pull-request-available
> Attachments: HIVE-17982.2.patch, HIVE-17982.patch
>
>
> There are a number of tests in itests/hive-unit/.../metastore that are
> metastore specific. I suspect they were initially placed in itests only
> because the metastore pulling in a few plugins from ql.
> Given that we need to be able to release the metastore separately, we need to
> be able to test it completely as a standalone entity. So I propose to move a
> number of the itests over into standalone-metastore. I will only move tests
> that are isolated to the metastore. Anything that tests wider functionality
> I plan to leave in itests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18298) Fix TestReplicationScenarios.testConstraints
[ https://issues.apache.org/jira/browse/HIVE-18298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323973#comment-16323973 ] Sankar Hariappan commented on HIVE-18298: - +1 Patch looks good to me. > Fix TestReplicationScenarios.testConstraints > > > Key: HIVE-18298 > URL: https://issues.apache.org/jira/browse/HIVE-18298 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-18298.1.patch > > > The test if broken by HIVE-16603. Currently on constraints are created > without order on replication destination cluster during bootstrap, after > HIVE-16603, it is no longer possible. We need to create foreign keys at last > after all primary keys are created. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18442) HoS: No FileSystem for scheme: nullscan
[
https://issues.apache.org/jira/browse/HIVE-18442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323967#comment-16323967
]
Rui Li commented on HIVE-18442:
---
The issue seems to related to class loading. FileSystem uses ServiceLoader to
load FS implementations. But it will only load once. So if hive-exec.jar is not
loaded by the context class loader when {{FileSystem::loadFileSystems}} is
called, the NullScanFileSystem will not be loaded and thus the failure.
The issue is specific to cluster mode. I don't see it in client mode.
I think we can set {{fs.nullscan.impl}} in JobConf to fix it.
> HoS: No FileSystem for scheme: nullscan
> ---
>
> Key: HIVE-18442
> URL: https://issues.apache.org/jira/browse/HIVE-18442
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
>
> Hit the issue when I run following query in yarn-cluster mode:
> {code}
> select * from (select key from src where false) a left outer join (select key
> from srcpart limit 0) b on a.key=b.key;
> {code}
> Stack trace:
> {noformat}
> Job failed with java.io.IOException: No FileSystem for scheme: nullscan
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2799)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.isEmptyPath(Utilities.java:2605)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.isEmptyPath(Utilities.java:2601)
> at
> org.apache.hadoop.hive.ql.exec.Utilities$GetInputPathsCallable.call(Utilities.java:3409)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getInputPaths(Utilities.java:3347)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.cloneJobConf(SparkPlanGenerator.java:299)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generate(SparkPlanGenerator.java:222)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generate(SparkPlanGenerator.java:109)
> at
> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:354)
> at
> org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:358)
> at
> org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:323)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18442) HoS: No FileSystem for scheme: nullscan
[
https://issues.apache.org/jira/browse/HIVE-18442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-18442:
--
Description:
Hit the issue when I run following query in yarn-cluster mode:
{code}
select * from (select key from src where false) a left outer join (select key
from srcpart limit 0) b on a.key=b.key;
{code}
Stack trace:
{noformat}
Job failed with java.io.IOException: No FileSystem for scheme: nullscan
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2799)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
at
org.apache.hadoop.hive.ql.exec.Utilities.isEmptyPath(Utilities.java:2605)
at
org.apache.hadoop.hive.ql.exec.Utilities.isEmptyPath(Utilities.java:2601)
at
org.apache.hadoop.hive.ql.exec.Utilities$GetInputPathsCallable.call(Utilities.java:3409)
at
org.apache.hadoop.hive.ql.exec.Utilities.getInputPaths(Utilities.java:3347)
at
org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.cloneJobConf(SparkPlanGenerator.java:299)
at
org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generate(SparkPlanGenerator.java:222)
at
org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generate(SparkPlanGenerator.java:109)
at
org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:354)
at
org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:358)
at
org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:323)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
> HoS: No FileSystem for scheme: nullscan
> ---
>
> Key: HIVE-18442
> URL: https://issues.apache.org/jira/browse/HIVE-18442
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
>
> Hit the issue when I run following query in yarn-cluster mode:
> {code}
> select * from (select key from src where false) a left outer join (select key
> from srcpart limit 0) b on a.key=b.key;
> {code}
> Stack trace:
> {noformat}
> Job failed with java.io.IOException: No FileSystem for scheme: nullscan
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2799)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.isEmptyPath(Utilities.java:2605)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.isEmptyPath(Utilities.java:2601)
> at
> org.apache.hadoop.hive.ql.exec.Utilities$GetInputPathsCallable.call(Utilities.java:3409)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getInputPaths(Utilities.java:3347)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.cloneJobConf(SparkPlanGenerator.java:299)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generate(SparkPlanGenerator.java:222)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generate(SparkPlanGenerator.java:109)
> at
> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:354)
> at
> org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:358)
> at
> org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:323)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17982) Move metastore specific itests
[
https://issues.apache.org/jira/browse/HIVE-17982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323934#comment-16323934
]
Hive QA commented on HIVE-17982:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} standalone-metastore: The patch generated 0 new +
195 unchanged - 1 fixed = 195 total (was 196) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
6s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / fd4e222 |
| Default Java | 1.8.0_111 |
| modules | C: standalone-metastore itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8591/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Move metastore specific itests
> --
>
> Key: HIVE-17982
> URL: https://issues.apache.org/jira/browse/HIVE-17982
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Labels: pull-request-available
> Attachments: HIVE-17982.2.patch, HIVE-17982.patch
>
>
> There are a number of tests in itests/hive-unit/.../metastore that are
> metastore specific. I suspect they were initially placed in itests only
> because the metastore pulling in a few plugins from ql.
> Given that we need to be able to release the metastore separately, we need to
> be able to test it completely as a standalone entity. So I propose to move a
> number of the itests over into standalone-metastore. I will only move tests
> that are isolated to the metastore. Anything that tests wider functionality
> I plan to leave in itests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17257) Hive should merge empty files
[
https://issues.apache.org/jira/browse/HIVE-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323901#comment-16323901
]
Hive QA commented on HIVE-17257:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905762/HIVE-17257.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 11567 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query39]
(batchId=247)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8590/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8590/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8590/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12905762 - PreCommit-HIVE-Build
> Hive should merge empty files
> -
>
> Key: HIVE-17257
> URL: https://issues.apache.org/jira/browse/HIVE-17257
> Project: Hive
> Issue Type: Bug
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-17257.0.patch, HIVE-17257.1.patch,
> HIVE-17257.2.patch
>
>
> Currently if merging file option is turned on and the dest dir contains large
> number of empty files, Hive will not trigger merge task:
> {code}
> private long getMergeSize(FileSystem inpFs, Path dirPath, long avgSize) {
> AverageSize averageSize = getAverageSize(inpFs, dirPath);
> if (averageSize.getTotalSize() <= 0) {
> return -1;
> }
> if (averageSize.getNumFiles() <= 1) {
> return -1;
> }
> if (averageSize.getTotalSize()/averageSize.getNumFiles() < avgSize) {
> return averageSize.getTotalSize();
> }
> return -1;
> }
> {code}
> This logic doesn't seem right as the it seems better to combine these empty
> files into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18372) Create testing infra to test different HMS instances
[ https://issues.apache.org/jira/browse/HIVE-18372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-18372: -- Attachment: HIVE-18372.5.patch Addressed RB comments > Create testing infra to test different HMS instances > > > Key: HIVE-18372 > URL: https://issues.apache.org/jira/browse/HIVE-18372 > Project: Hive > Issue Type: Sub-task >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-18372.2.patch, HIVE-18372.3.patch, > HIVE-18372.5.patch, HIVE-18372.patch > > > Since there will be multiple tests, it would be good to have a good > infrastructure to help creating those faster, easier. > This patch will also include the test cases for the Database related methods > to showcase the infra -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17257) Hive should merge empty files
[
https://issues.apache.org/jira/browse/HIVE-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323859#comment-16323859
]
Hive QA commented on HIVE-17257:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / fd4e222 |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8590/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive should merge empty files
> -
>
> Key: HIVE-17257
> URL: https://issues.apache.org/jira/browse/HIVE-17257
> Project: Hive
> Issue Type: Bug
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-17257.0.patch, HIVE-17257.1.patch,
> HIVE-17257.2.patch
>
>
> Currently if merging file option is turned on and the dest dir contains large
> number of empty files, Hive will not trigger merge task:
> {code}
> private long getMergeSize(FileSystem inpFs, Path dirPath, long avgSize) {
> AverageSize averageSize = getAverageSize(inpFs, dirPath);
> if (averageSize.getTotalSize() <= 0) {
> return -1;
> }
> if (averageSize.getNumFiles() <= 1) {
> return -1;
> }
> if (averageSize.getTotalSize()/averageSize.getNumFiles() < avgSize) {
> return averageSize.getTotalSize();
> }
> return -1;
> }
> {code}
> This logic doesn't seem right as the it seems better to combine these empty
> files into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18429) Compaction should handle a case when it produces no output
[
https://issues.apache.org/jira/browse/HIVE-18429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323846#comment-16323846
]
Hive QA commented on HIVE-18429:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905761/HIVE-18429.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 11571 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query39]
(batchId=247)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.hcatalog.common.TestHiveClientCache.testCloseAllClients
(batchId=197)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveConflictKill
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8589/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8589/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8589/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12905761 - PreCommit-HIVE-Build
> Compaction should handle a case when it produces no output
> --
>
> Key: HIVE-18429
> URL: https://issues.apache.org/jira/browse/HIVE-18429
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18429.01.patch, HIVE-18429.02.patch,
> HIVE-18429.03.patch
>
>
> Suppose we start with empty delta_8_8 and delta_9_9 and compaction runs.
> It will currently produce an MR job with 0 splits and so
> {{CompactorMR.TMP_LOCATION}} never gets created. This causes
> {{CompactorOutputCommitted.commitJob()}} to fail when it tries to do
> {{FileStatus[] contents = fs.listStatus(tmpLocation);}} since tmpLocation
> doesn't exist.
> If compactor fails to produce delta_8_9 here it will fail to do further
> compaction unless new delta with data is created.
> If the number of empty deltas is > than
> HiveConf.ConfVars.COMPACTOR_MAX_NUM_DELTA, compaction will not be able to
> proceed at all.
> It should produce a delta_8_9 in this case even if it's empty.
> The error (in the log of standalone metastore process) would look like this
> {noformat}
> 2017-12-27 17:19:28,850 ERROR CommitterEvent Processor #1
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Could not
> commit job
> java.io.FileNotFoundException: File
> hdfs://OTCHaaS/apps/hive/warehouse/momi.db/sensor_data/babyid=5911806ebf6964014257/_tmp_b4c5a3f3-44e5-4d45-86af-5b773bf0fc96
> does not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:923)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:114)
> at
>
[jira] [Commented] (HIVE-18429) Compaction should handle a case when it produces no output
[
https://issues.apache.org/jira/browse/HIVE-18429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323809#comment-16323809
]
Hive QA commented on HIVE-18429:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / fd4e222 |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8589/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Compaction should handle a case when it produces no output
> --
>
> Key: HIVE-18429
> URL: https://issues.apache.org/jira/browse/HIVE-18429
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18429.01.patch, HIVE-18429.02.patch,
> HIVE-18429.03.patch
>
>
> Suppose we start with empty delta_8_8 and delta_9_9 and compaction runs.
> It will currently produce an MR job with 0 splits and so
> {{CompactorMR.TMP_LOCATION}} never gets created. This causes
> {{CompactorOutputCommitted.commitJob()}} to fail when it tries to do
> {{FileStatus[] contents = fs.listStatus(tmpLocation);}} since tmpLocation
> doesn't exist.
> If compactor fails to produce delta_8_9 here it will fail to do further
> compaction unless new delta with data is created.
> If the number of empty deltas is > than
> HiveConf.ConfVars.COMPACTOR_MAX_NUM_DELTA, compaction will not be able to
> proceed at all.
> It should produce a delta_8_9 in this case even if it's empty.
> The error (in the log of standalone metastore process) would look like this
> {noformat}
> 2017-12-27 17:19:28,850 ERROR CommitterEvent Processor #1
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Could not
> commit job
> java.io.FileNotFoundException: File
> hdfs://OTCHaaS/apps/hive/warehouse/momi.db/sensor_data/babyid=5911806ebf6964014257/_tmp_b4c5a3f3-44e5-4d45-86af-5b773bf0fc96
> does not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:923)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:114)
> at
>
[jira] [Commented] (HIVE-17833) Publish split generation counters
[
https://issues.apache.org/jira/browse/HIVE-17833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323792#comment-16323792
]
Hive QA commented on HIVE-17833:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905775/HIVE-17833.5.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 35 failed/errored test(s), 11576 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters1]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dp_counter_mm]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dp_counter_non_mm]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=121)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=214)
org.apache.hadoop.hive.ql.TestAcidOnTez.testGetSplitsLocks (batchId=223)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=254)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=226)
org.apache.hive.hcatalog.templeton.TestConcurrentJobRequestsThreadsAndTimeout.ConcurrentListJobsVerifyExceptions
(batchId=189)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testDataTypes (batchId=233)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testEscapedStrings (batchId=233)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd
(batchId=233)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testNonAsciiStrings (batchId=233)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveBackKill
(batchId=236)
org.apache.hive.jdbc.TestTriggersNoTezSessionPool.testTriggerTotalTasks
(batchId=233)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=236)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=233)
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementParallel
(batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8588/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8588/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8588/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 35 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12905775 - PreCommit-HIVE-Build
> Publish split generation counters
> -
>
> Key: HIVE-17833
> URL: https://issues.apache.org/jira/browse/HIVE-17833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17833.1.patch, HIVE-17833.2.patch,
> HIVE-17833.3.patch, HIVE-17833.4.patch, HIVE-17833.5.patch
>
>
> With TEZ-3856, tez counters are exposed via input initializers
