[jira] [Issue Comment Deleted] (HIVE-18309) qtests: smb_mapjoin_19.q breaks bucketsortoptimize_insert_2.q
[
https://issues.apache.org/jira/browse/HIVE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-18309:
Comment: was deleted
(was: may be the same issue, please find comments)
> qtests: smb_mapjoin_19.q breaks bucketsortoptimize_insert_2.q
> -
>
> Key: HIVE-18309
> URL: https://issues.apache.org/jira/browse/HIVE-18309
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>
> {code}
> mvn install -q -am -pl itests/qtest -DskipSparkTests -pl itests/qtest
> -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=smb_mapjoin_19.q,bucketsortoptimize_insert_2.q
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (HIVE-18182) execution of semijoin_hint.q breaks hybridgrace_hashjoin_2.q
[
https://issues.apache.org/jira/browse/HIVE-18182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor resolved HIVE-18182.
-
Resolution: Duplicate
Assignee: Laszlo Bodor
Duplicate of https://issues.apache.org/jira/browse/HIVE-18314
> execution of semijoin_hint.q breaks hybridgrace_hashjoin_2.q
>
>
> Key: HIVE-18182
> URL: https://issues.apache.org/jira/browse/HIVE-18182
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>
> {code}
> M_OPTS+=" -q "
> M_OPTS+=" -Pitests -DskipSparkTests"
> M_OPTS+=" -pl itests/qtest"
> M_OPTS+=" -Dtest=TestMiniLlapLocalCliDriver"
> M_OPTS+=" -Dqfile=semijoin_hint.q,hybridgrace_hashjoin_2.q"
> M_OPTS+=" install"
> mvn $M_OPTS -DfailIfNoTests
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18362) Introduce a parameter to control the max row number for map join convertion
[ https://issues.apache.org/jira/browse/HIVE-18362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wan kun reassigned HIVE-18362: -- > Introduce a parameter to control the max row number for map join convertion > --- > > Key: HIVE-18362 > URL: https://issues.apache.org/jira/browse/HIVE-18362 > Project: Hive > Issue Type: Bug > Components: Query Processor >Reporter: wan kun >Assignee: wan kun >Priority: Minor > > The compression ratio of the Orc compressed file will be very high in some > cases. > The test table has three Int columns, with twelve million records, but the > compressed file size is only 4M. Hive will automatically converts the Join to > Map join, but this will cause memory overflow. So I think it is better to > have a parameter to limit to the total number of table records in the Map > Join convertion, and if the total number of records is larger than that, it > can not be converted to Map join. > *hive.auto.convert.join.max.number = 250L* > The default value for this parameter is 250, because so many records > occupy about 700M memory in clint JVM, and 250 records for Map Join are > also large tables. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18361) Extend shared work optimizer to reuse computation beyond work boundaries
[
https://issues.apache.org/jira/browse/HIVE-18361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309315#comment-16309315
]
Hive QA commented on HIVE-18361:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904318/HIVE-18361.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 42 failed/errored test(s), 11543 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver[generatehfiles_require_family_path]
(batchId=243)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_multi]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_notin]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_select]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_views]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_id2]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets4]
(batchId=156)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query2]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query33]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query44]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query47]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query54]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query56]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query57]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query58]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query59]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query60]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query61]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query64]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query83]
(batchId=245)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8413/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8413/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8413/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException:
[jira] [Updated] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[ https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-18079: --- Attachment: HIVE-18079.6.patch > Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator > bit-size > > > Key: HIVE-18079 > URL: https://issues.apache.org/jira/browse/HIVE-18079 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Statistics >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-18079.1.patch, HIVE-18079.2.patch, > HIVE-18079.4.patch, HIVE-18079.5.patch, HIVE-18079.6.patch > > > HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its > mathematical hash distribution & construction. > Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second > scan over the data-set. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18190) Consider looking at ORC file schema rather than using _metadata_acid file
[
https://issues.apache.org/jira/browse/HIVE-18190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309334#comment-16309334
]
Hive QA commented on HIVE-18190:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 5b0d993 |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8414/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Consider looking at ORC file schema rather than using _metadata_acid file
> -
>
> Key: HIVE-18190
> URL: https://issues.apache.org/jira/browse/HIVE-18190
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18190.01.patch, HIVE-18190.02.patch,
> HIVE-18190.04.patch, HIVE-18190.05.patch, HIVE-18190.06.patch,
> HIVE-18190.07.patch
>
>
> See if it's possible to just look at the schema of the file in base_ or
> delta_ to see if it has Acid metadata columns. If not, it's an 'original'
> file and needs ROW_IDs generated.
> see more discussion at https://reviews.apache.org/r/64131/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18301) Investigate to enable MapInput cache in Hive on Spark
[
https://issues.apache.org/jira/browse/HIVE-18301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309361#comment-16309361
]
liyunzhang commented on HIVE-18301:
---
Here some update about NPE:
the normal case when enable rdd cache
the stacktrace of initIOContext which will intialize
ExecMapperContext#setCurrentInputPath is
{code}
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.initIOContext(HiveContextAwareRecordReader.java:175)
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.initIOContext(HiveContextAwareRecordReader.java:211)
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.(CombineHiveRecordReader.java:101)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:257)
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.(HadoopShimsSecure.java:217)
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:346)
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:712)
org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:246)
org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:209)
org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:102)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:332)
org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:330)
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:919)
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:910)
org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:866)
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:910)
org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:668)
org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:330)
org.apache.spark.rdd.RDD.iterator(RDD.scala:281)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:332)
org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:330)
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:919)
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:910)
org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:866)
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:910)
org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:668)
org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:330)
org.apache.spark.rdd.RDD.iterator(RDD.scala:281)
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
org.apache.spark.scheduler.Task.run(Task.scala:85)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
{code}
the stacktrace of ExecMapperContext#getCurrentInputPath
{code}
org.apache.hadoop.hive.ql.exec.mr.ExecMapperContext.getCurrentInputPath(ExecMapperContext.java:113)
org.apache.hadoop.hive.ql.exec.MapOperator.cleanUpInputFileChangedOp(MapOperator.java:512)
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1187)
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:543)
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:136)
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:85)
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:213)
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:919)
org.apache.spark.storage.
[jira] [Work started] (HIVE-18362) Introduce a parameter to control the max row number for map join convertion
[ https://issues.apache.org/jira/browse/HIVE-18362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-18362 started by wan kun. -- > Introduce a parameter to control the max row number for map join convertion > --- > > Key: HIVE-18362 > URL: https://issues.apache.org/jira/browse/HIVE-18362 > Project: Hive > Issue Type: Bug > Components: Query Processor >Reporter: wan kun >Assignee: wan kun >Priority: Minor > > The compression ratio of the Orc compressed file will be very high in some > cases. > The test table has three Int columns, with twelve million records, but the > compressed file size is only 4M. Hive will automatically converts the Join to > Map join, but this will cause memory overflow. So I think it is better to > have a parameter to limit to the total number of table records in the Map > Join convertion, and if the total number of records is larger than that, it > can not be converted to Map join. > *hive.auto.convert.join.max.number = 250L* > The default value for this parameter is 250, because so many records > occupy about 700M memory in clint JVM, and 250 records for Map Join are > also large tables. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18362) Introduce a parameter to control the max row number for map join convertion
[ https://issues.apache.org/jira/browse/HIVE-18362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wan kun updated HIVE-18362: --- Attachment: HIVE-18362-branch-1.2.patch > Introduce a parameter to control the max row number for map join convertion > --- > > Key: HIVE-18362 > URL: https://issues.apache.org/jira/browse/HIVE-18362 > Project: Hive > Issue Type: Bug > Components: Query Processor >Reporter: wan kun >Assignee: wan kun >Priority: Minor > Attachments: HIVE-18362-branch-1.2.patch > > > The compression ratio of the Orc compressed file will be very high in some > cases. > The test table has three Int columns, with twelve million records, but the > compressed file size is only 4M. Hive will automatically converts the Join to > Map join, but this will cause memory overflow. So I think it is better to > have a parameter to limit to the total number of table records in the Map > Join convertion, and if the total number of records is larger than that, it > can not be converted to Map join. > *hive.auto.convert.join.max.number = 250L* > The default value for this parameter is 250, because so many records > occupy about 700M memory in clint JVM, and 250 records for Map Join are > also large tables. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18362) Introduce a parameter to control the max row number for map join convertion
[ https://issues.apache.org/jira/browse/HIVE-18362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wan kun updated HIVE-18362: --- Status: Patch Available (was: In Progress) > Introduce a parameter to control the max row number for map join convertion > --- > > Key: HIVE-18362 > URL: https://issues.apache.org/jira/browse/HIVE-18362 > Project: Hive > Issue Type: Bug > Components: Query Processor >Reporter: wan kun >Assignee: wan kun >Priority: Minor > > The compression ratio of the Orc compressed file will be very high in some > cases. > The test table has three Int columns, with twelve million records, but the > compressed file size is only 4M. Hive will automatically converts the Join to > Map join, but this will cause memory overflow. So I think it is better to > have a parameter to limit to the total number of table records in the Map > Join convertion, and if the total number of records is larger than that, it > can not be converted to Map join. > *hive.auto.convert.join.max.number = 250L* > The default value for this parameter is 250, because so many records > occupy about 700M memory in clint JVM, and 250 records for Map Join are > also large tables. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18362) Introduce a parameter to control the max row number for map join convertion
[
https://issues.apache.org/jira/browse/HIVE-18362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309392#comment-16309392
]
Gopal V commented on HIVE-18362:
Is this a dup for {{hive.auto.convert.join.hashtable.max.entries}} setting?
> Introduce a parameter to control the max row number for map join convertion
> ---
>
> Key: HIVE-18362
> URL: https://issues.apache.org/jira/browse/HIVE-18362
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: wan kun
>Assignee: wan kun
>Priority: Minor
> Attachments: HIVE-18362-branch-1.2.patch
>
>
> The compression ratio of the Orc compressed file will be very high in some
> cases.
> The test table has three Int columns, with twelve million records, but the
> compressed file size is only 4M. Hive will automatically converts the Join to
> Map join, but this will cause memory overflow. So I think it is better to
> have a parameter to limit to the total number of table records in the Map
> Join convertion, and if the total number of records is larger than that, it
> can not be converted to Map join.
> *hive.auto.convert.join.max.number = 250L*
> The default value for this parameter is 250, because so many records
> occupy about 700M memory in clint JVM, and 250 records for Map Join are
> also large tables.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (HIVE-18174) Vectorization: De-dup Group-by key expressions (identical keys are irrelevant)
[
https://issues.apache.org/jira/browse/HIVE-18174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V resolved HIVE-18174.
Resolution: Duplicate
> Vectorization: De-dup Group-by key expressions (identical keys are irrelevant)
> --
>
> Key: HIVE-18174
> URL: https://issues.apache.org/jira/browse/HIVE-18174
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 3.0.0
>Reporter: Gopal V
>
> {code}
> hive.vectorized.execution.reduce.enabled=true;
> hive.vectorized.execution.reduce.groupby.enabled=true;
> create temporary table foo (x int) stored as orc;
> insert into foo values(1),(2),(3);
> insert into foo values(1),(2),(3);
> set hive.cbo.enable=false;
> select distinct concat('x', x) x, concat('x', x), 'Foo', 'Foo' from foo;
> {code}
> {code}
> Caused by: java.lang.RuntimeException: null STRING entry: batchIndex 0
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.nullBytesReadError(VectorExtractRow.java:476)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.extractRowColumn(VectorExtractRow.java:288)
> {code}
> The key has duplicate references - {{keys: KEY._col0 (type: string),
> KEY._col0 (type: string), 'Foo' (type: string), 'Foo' (type: string)}}
> {code}
> STAGE PLANS:
> Stage: Stage-1
> Tez
> DagId: gopal_20171128220857_9c9def2e-d0a4-461a-8fd6-f9fdaea2d5ce:26
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> DagName:
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: foo
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
> Column stats: NONE
> Select Operator
> expressions: x (type: int)
> outputColumnNames: x
> Statistics: Num rows: 1 Data size: 4 Basic stats:
> COMPLETE Column stats: NONE
> Group By Operator
> keys: concat('x', x) (type: string), concat('x', x)
> (type: string), 'Foo' (type: string), 'Foo' (type: string)
> mode: hash
> outputColumnNames: _col0, _col1, _col2, _col3
> Statistics: Num rows: 1 Data size: 4 Basic stats:
> COMPLETE Column stats: NONE
> Reduce Output Operator
> key expressions: _col1 (type: string), 'Foo' (type:
> string)
> sort order: ++
> Map-reduce partition columns: _col1 (type: string),
> 'Foo' (type: string)
> Statistics: Num rows: 1 Data size: 4 Basic stats:
> COMPLETE Column stats: NONE
> Execution mode: vectorized, llap
> LLAP IO: all inputs
> Reducer 2
> Execution mode: vectorized, llap
> Reduce Operator Tree:
> Group By Operator
> keys: KEY._col0 (type: string), KEY._col0 (type: string),
> 'Foo' (type: string), 'Foo' (type: string)
> mode: mergepartial
> outputColumnNames: _col0, _col1, _col2, _col3
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
> Column stats: NONE
> Select Operator
> expressions: _col1 (type: string), _col1 (type: string),
> 'Foo' (type: string), 'Foo' (type: string)
> outputColumnNames: _col0, _col1, _col2, _col3
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
> Column stats: NONE
> File Output Operator
> compressed: false
> Statistics: Num rows: 1 Data size: 4 Basic stats:
> COMPLETE Column stats: NONE
> table:
> input format:
> org.apache.hadoop.mapred.SequenceFileInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
> serde:
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18362) Introduce a parameter to control the max row number for map join convertion
[ https://issues.apache.org/jira/browse/HIVE-18362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V reassigned HIVE-18362: -- Assignee: Gopal V (was: wan kun) > Introduce a parameter to control the max row number for map join convertion > --- > > Key: HIVE-18362 > URL: https://issues.apache.org/jira/browse/HIVE-18362 > Project: Hive > Issue Type: Bug > Components: Query Processor >Reporter: wan kun >Assignee: Gopal V >Priority: Minor > Attachments: HIVE-18362-branch-1.2.patch > > > The compression ratio of the Orc compressed file will be very high in some > cases. > The test table has three Int columns, with twelve million records, but the > compressed file size is only 4M. Hive will automatically converts the Join to > Map join, but this will cause memory overflow. So I think it is better to > have a parameter to limit to the total number of table records in the Map > Join convertion, and if the total number of records is larger than that, it > can not be converted to Map join. > *hive.auto.convert.join.max.number = 250L* > The default value for this parameter is 250, because so many records > occupy about 700M memory in clint JVM, and 250 records for Map Join are > also large tables. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18359) Extend grouping set limits from int to long
[ https://issues.apache.org/jira/browse/HIVE-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18359: - Attachment: HIVE-18359.1.patch > Extend grouping set limits from int to long > --- > > Key: HIVE-18359 > URL: https://issues.apache.org/jira/browse/HIVE-18359 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-18359.1.patch > > > Grouping sets is broken for >32 columns because of usage of Int for bitmap > (also GROUPING__ID virtual column). This assumption breaks grouping > sets/rollups/cube when number of participating aggregation columns is >32. > The easier fix would be extend it to Long for now. The correct fix would be > to use BitSets everywhere but that would require GROUPING__ID column type to > binary which will make predicates on GROUPING__ID difficult to deal with. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18359) Extend grouping set limits from int to long
[ https://issues.apache.org/jira/browse/HIVE-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18359: - Status: Patch Available (was: Open) > Extend grouping set limits from int to long > --- > > Key: HIVE-18359 > URL: https://issues.apache.org/jira/browse/HIVE-18359 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-18359.1.patch > > > Grouping sets is broken for >32 columns because of usage of Int for bitmap > (also GROUPING__ID virtual column). This assumption breaks grouping > sets/rollups/cube when number of participating aggregation columns is >32. > The easier fix would be extend it to Long for now. The correct fix would be > to use BitSets everywhere but that would require GROUPING__ID column type to > binary which will make predicates on GROUPING__ID difficult to deal with. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18190) Consider looking at ORC file schema rather than using _metadata_acid file
[
https://issues.apache.org/jira/browse/HIVE-18190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309405#comment-16309405
]
Hive QA commented on HIVE-18190:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904322/HIVE-18190.07.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 11542 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_var_samp]
(batchId=85)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=177)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8414/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8414/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8414/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 19 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12904322 - PreCommit-HIVE-Build
> Consider looking at ORC file schema rather than using _metadata_acid file
> -
>
> Key: HIVE-18190
> URL: https://issues.apache.org/jira/browse/HIVE-18190
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18190.01.patch, HIVE-18190.02.patch,
> HIVE-18190.04.patch, HIVE-18190.05.patch, HIVE-18190.06.patch,
> HIVE-18190.07.patch
>
>
> See if it's possible to just look at the schema of the file in base_ or
> delta_ to see if it has Acid metadata columns. If not, it's an 'original'
> file and needs ROW_IDs generated.
> see more discussion at https://reviews.apache.org/r/64131/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[
https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309439#comment-16309439
]
Hive QA commented on HIVE-18079:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} standalone-metastore: The patch generated 1 new + 41
unchanged - 0 fixed = 42 total (was 41) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 5b0d993 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8415/yetus/diff-checkstyle-standalone-metastore.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8415/yetus/whitespace-eol.txt
|
| modules | C: standalone-metastore ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8415/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator
> bit-size
>
>
> Key: HIVE-18079
> URL: https://issues.apache.org/jira/browse/HIVE-18079
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore, Statistics
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-18079.1.patch, HIVE-18079.2.patch,
> HIVE-18079.4.patch, HIVE-18079.5.patch, HIVE-18079.6.patch
>
>
> HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its
> mathematical hash distribution & construction.
> Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second
> scan over the data-set.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18356) Fixing license headers in checkstyle
[
https://issues.apache.org/jira/browse/HIVE-18356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-18356:
--
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master.
Thanks [~prasanth_j] and [~asherman] for the review!
> Fixing license headers in checkstyle
>
>
> Key: HIVE-18356
> URL: https://issues.apache.org/jira/browse/HIVE-18356
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-18356.patch
>
>
> The checkstyle header contains the following ASF header:
> {code}
> /**
> * Licensed to the Apache Software Foundation (ASF) under one
> * or more contributor license agreements. See the NOTICE file
> [..]
> {code}
> Even if we undecided what to do with the already existing headers
> (HIVE-17952), the new ones should use the proper one with 1 '*' in the first
> line:
> {code}
> /*
> * Licensed to the Apache Software Foundation (ASF) under one
> * or more contributor license agreements. See the NOTICE file
> [..]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18346) Beeline could not launch because of size of history
[
https://issues.apache.org/jira/browse/HIVE-18346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309474#comment-16309474
]
Madhudeep Petwal commented on HIVE-18346:
-
The max size for history is
DEFAULT_MAX_SIZE = 500;
and this function maybeResize() is keeping a check on history size
It looks like you manually inserted some history records,
in that case also beeline truncate history size to 3k
-rw---. 1 user group 3.0K Jan 3 16:17 .beeline/history
am I wrong ??
I am not able to reproduce this bug. I think it is handled [~asherman]
> Beeline could not launch because of size of history
> ---
>
> Key: HIVE-18346
> URL: https://issues.apache.org/jira/browse/HIVE-18346
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.2.1
>Reporter: Jumping
>Assignee: Madhudeep Petwal
>Priority: Minor
>
> Beeline version 1.2.1 could not launch when the size of
> ${user.home}/.beeline/history larger than 39MB. Which reports
> "java.lang.outofmemoryerror" .
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18342) Remove LinkedList from HiveAlterHandler.java
[
https://issues.apache.org/jira/browse/HIVE-18342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309485#comment-16309485
]
Adam Szita commented on HIVE-18342:
---
+1 non-binding on [^HIVE-18342.1.patch]
> Remove LinkedList from HiveAlterHandler.java
>
>
> Key: HIVE-18342
> URL: https://issues.apache.org/jira/browse/HIVE-18342
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-18342.1.patch
>
>
> Remove {{LinkedList}} in favor of {{ArrayList}} for class
> {{org.apache.hadoop.hive.metastore.HiveAlterHandler}}.
> {quote}
> The size, isEmpty, get, set, iterator, and listIterator operations run in
> constant time. The add operation runs in amortized constant time, that is,
> adding n elements requires O(n) time. All of the other operations run in
> linear time (roughly speaking). *The constant factor is low compared to that
> for the LinkedList implementation.*
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18344) Remove LinkedList from SharedWorkOptimizer.java
[
https://issues.apache.org/jira/browse/HIVE-18344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309488#comment-16309488
]
Peter Vary commented on HIVE-18344:
---
+1
> Remove LinkedList from SharedWorkOptimizer.java
> ---
>
> Key: HIVE-18344
> URL: https://issues.apache.org/jira/browse/HIVE-18344
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-18344.1.patch
>
>
> Prefer {{ArrayList}} over {{LinkedList}} especially in this class because the
> initial size of the collection is known.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[
https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309502#comment-16309502
]
Hive QA commented on HIVE-18079:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904357/HIVE-18079.6.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 52 failed/errored test(s), 11570 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_join1]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_6]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_groupby]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez2]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainanalyze_2]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby2]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join1]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join32_lessSize]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mapjoin_decimal]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[quotedid_smb]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[semijoin]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[semijoin_hint]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[skewjoin]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[smb_cache]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[smb_mapjoin_15]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[smb_mapjoin_6]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_corr]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_in]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_notin]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_bmj_schema_evolution]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_join_hash]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_union_multiinsert]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_cast_constant]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_count_distinct]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_context]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=102)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=102)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[tez-tag]
(batchId=102)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket_map_join_tez2]
(batchId=109)
org.apache.hadoop.hive.cli
[jira] [Commented] (HIVE-17896) TopNKey: Create a standalone vectorizable TopNKey operator
[
https://issues.apache.org/jira/browse/HIVE-17896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309533#comment-16309533
]
Hive QA commented on HIVE-17896:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} common: The patch generated 1 new + 932 unchanged - 0
fixed = 933 total (was 932) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 42 new + 584 unchanged - 1
fixed = 626 total (was 585) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 98c95c6 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8416/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8416/yetus/diff-checkstyle-ql.txt
|
| modules | C: common serde ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8416/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TopNKey: Create a standalone vectorizable TopNKey operator
> --
>
> Key: HIVE-17896
> URL: https://issues.apache.org/jira/browse/HIVE-17896
> Project: Hive
> Issue Type: New Feature
> Components: Operators
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Teddy Choi
> Attachments: HIVE-17896.1.patch, HIVE-17896.3.patch,
> HIVE-17896.4.patch, HIVE-17896.5.patch
>
>
> For TPC-DS Query27, the TopN operation is delayed by the group-by - the
> group-by operator buffers up all the rows before discarding the 99% of the
> rows in the TopN Hash within the ReduceSink Operator.
> The RS TopN operator is very restrictive as it only supports doing the
> filtering on the shuffle keys, but it is better to do this before breaking
> the vectors into rows and losing the isRepeating properties.
> Adding a TopN Key operator in the physical operator tree allows the following
> to happen.
> GBY->RS(Top=1)
> can become
> TNK(1)->GBY->RS(Top=1)
> So that, the TopNKey can remove
[jira] [Commented] (HIVE-18358) from_unixtime returns wrong year for Dec 31 timestamps with format 'YYYY'
[
https://issues.apache.org/jira/browse/HIVE-18358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309543#comment-16309543
]
Nick Orka commented on HIVE-18358:
--
To be precise, according to the documentation, it's "Week year". "Week in year"
is "w". The problem is that whole year "" was providing correct 2017 year
and it changed to 2018 for 12/31/2017 only. There is no real explanation what
"Week year" means. I assume, this is a year of last day of a week and as
12/31/2017 is Sunday, which is the first day of the week in USA. Thus ""
returns a year for last day of the week which is 1/6/2018.
> from_unixtime returns wrong year for Dec 31 timestamps with format ''
> -
>
> Key: HIVE-18358
> URL: https://issues.apache.org/jira/browse/HIVE-18358
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
> Environment: AWS EMR with Hive 2.1.0-amzn-0
>Reporter: Nick Orka
>Assignee: Andrew Sherman
> Labels: timezone
>
> If you use capital Ys as a year format in from_unixtime() it returns next
> year for Dec 31 only. All other days work as intended.
> Here is reproduction code:
> {code:sql}
> hive> select from_unixtime(1514754599, '-MM-dd HH-mm-ss'),
> from_unixtime(1514754599, '-MM-dd HH-mm-ss');
> OK
> 2018-12-31 21-09-59 2017-12-31 21-09-59
> Time taken: 0.025 seconds, Fetched: 1 row(s)
> hive>
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17896) TopNKey: Create a standalone vectorizable TopNKey operator
[
https://issues.apache.org/jira/browse/HIVE-17896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309581#comment-16309581
]
Hive QA commented on HIVE-17896:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904342/HIVE-17896.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 11548 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=177)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query84]
(batchId=247)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testApplyPlanQpChanges
(batchId=284)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8416/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8416/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8416/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12904342 - PreCommit-HIVE-Build
> TopNKey: Create a standalone vectorizable TopNKey operator
> --
>
> Key: HIVE-17896
> URL: https://issues.apache.org/jira/browse/HIVE-17896
> Project: Hive
> Issue Type: New Feature
> Components: Operators
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Teddy Choi
> Attachments: HIVE-17896.1.patch, HIVE-17896.3.patch,
> HIVE-17896.4.patch, HIVE-17896.5.patch
>
>
> For TPC-DS Query27, the TopN operation is delayed by the group-by - the
> group-by operator buffers up all the rows before discarding the 99% of the
> rows in the TopN Hash within the ReduceSink Operator.
> The RS TopN operator is very restrictive as it only supports doing the
> filtering on the shuffle keys, but it is better to do this before breaking
> the vectors into rows and losing the isRepeating properties.
> Adding a TopN Key operator in the physical operator tree allows the following
> to happen.
> GBY->RS(Top=1)
> can become
> TNK(1)->GBY->RS(Top=1)
> So that, the TopNKey can remove rows before they are buffered into the GBY
> and consume memory.
> Here's the equivalent implementation in Presto
> https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/operator/TopNOperator.java#L35
> Adding this as a sub-feature of GroupBy prevents further optimizations if the
> GBY is on keys "a,b,c" and the TopNKey is on just "a".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18305) travis-ci builds are timing out
[
https://issues.apache.org/jira/browse/HIVE-18305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18305:
Attachment: HIVE-18305.01.patch
unfortunately this issue needs either some manual tweaking the maven
installation or the resolver lib(MRESOLVER-39)
patch#1) disabling paralell build fixes the issue...
> travis-ci builds are timing out
> ---
>
> Key: HIVE-18305
> URL: https://issues.apache.org/jira/browse/HIVE-18305
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18305.01.patch
>
>
> {code}
> No output has been received in the last 10m0s, this potentially indicates a
> stalled build or something wrong with the build itself.
> Check the details on how to adjust your build configuration on:
> https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17982) Move metastore specific itests
[
https://issues.apache.org/jira/browse/HIVE-17982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309601#comment-16309601
]
Peter Vary commented on HIVE-17982:
---
Thanks for all your work done in this patch.
I tried to review it, so we can continue on the standalone metastore work.
I surely missed some stuff, but found some questions:
- In several cases (for example {{TestRetryingHMSHandler}}) I see that
{{MetaStoreTestUtils.startMetaStoreWithRetry()}} is replaced by this code:
{code}
int port = MetaStoreTestUtils.findFreePort();
[..]
MetaStoreTestUtils.startMetaStoreWithRetry(port,
HadoopThriftAuthBridge.getBridge(), conf);
{code}
I am not sure that the original version was atomic (but at least we had the
possibility to create an atomic version of the method), but adding more code
between getting a free port and starting the metastore could cause racing
condition problems. Probably that is why the original version was "WithRetry".
- {{MetaStoreTestUtils.startMetaStoreWithRetry}} - I have concerns with this
change - see my previous comment
- {{TestHiveMetaStore}}
-- Line 153: I understand why you removed the {{assert(false)}} - I would
removed the try-catch too - the test should throw the exception if it is a
failure
-- Line 956, 1812, 3140: The original table definition contained bucketed cols
as well. The new table is not bucketed - maybe not important at all. Pointing
out just in case :)
-- Line 990 in the new file: nit: We might rely on the default value of the
{{ownerName}} too, or do not rely on default for the {{ownerType}} - just a
thought
-- Line 992 in the new file: nit: use only {{assertEquals}} instead
{{Assert.assertEquals}}, since it is already imported.
-- {{testGetSchemaWithNoClassDefFoundError}} - I do not understand the original
intent of this test - it seems like a very-very edge case (throwing a
NoClassDefFoundError when initializing the SerDe), but the new version does not
test the original case (it tests how the SerDe is not found).
-- Line 2131, 2143: I understand why you removed the {{assert(false)}} - I
would removed the try-catch too - the test should throw the exception if it is
a failure
-- Line 2156: nit: Why not drop database with cascade option?
-- Line 2916: nit: I am curious why this was run with retries
- {{TestMetaStoreEndFunctionListener}} in line 118, 132: Really every exception
means all good? I know it was there previously, but seem like a bad decision at
that time :)
- {{TestMetaStoreEventListener}} in line 245, 262, 282, 300, 313, 329, 368,
407, 429, 441,... in the new file: nit: use only {{assertEquals}} instead
{{Assert.assertEquals}}, since it is already imported.
Thanks,
Peter
> Move metastore specific itests
> --
>
> Key: HIVE-17982
> URL: https://issues.apache.org/jira/browse/HIVE-17982
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Labels: pull-request-available
> Attachments: HIVE-17982.patch
>
>
> There are a number of tests in itests/hive-unit/.../metastore that are
> metastore specific. I suspect they were initially placed in itests only
> because the metastore pulling in a few plugins from ql.
> Given that we need to be able to release the metastore separately, we need to
> be able to test it completely as a standalone entity. So I propose to move a
> number of the itests over into standalone-metastore. I will only move tests
> that are isolated to the metastore. Anything that tests wider functionality
> I plan to leave in itests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[
https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309605#comment-16309605
]
Hive QA commented on HIVE-18079:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} standalone-metastore: The patch generated 1 new + 37
unchanged - 0 fixed = 38 total (was 37) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 98c95c6 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8417/yetus/diff-checkstyle-standalone-metastore.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8417/yetus/whitespace-eol.txt
|
| modules | C: standalone-metastore ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8417/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator
> bit-size
>
>
> Key: HIVE-18079
> URL: https://issues.apache.org/jira/browse/HIVE-18079
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore, Statistics
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-18079.1.patch, HIVE-18079.2.patch,
> HIVE-18079.4.patch, HIVE-18079.5.patch, HIVE-18079.6.patch
>
>
> HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its
> mathematical hash distribution & construction.
> Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second
> scan over the data-set.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18305) travis-ci builds are timing out
[
https://issues.apache.org/jira/browse/HIVE-18305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309619#comment-16309619
]
Zoltan Haindrich commented on HIVE-18305:
-
I see no reason to run this on the ptest server; so I don't set it to patch
available; a travis run: https://travis-ci.org/kgyrtkirk/hive/builds/324566225
[~prasanth_j] Could you please take a look?
> travis-ci builds are timing out
> ---
>
> Key: HIVE-18305
> URL: https://issues.apache.org/jira/browse/HIVE-18305
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18305.01.patch
>
>
> {code}
> No output has been received in the last 10m0s, this potentially indicates a
> stalled build or something wrong with the build itself.
> Check the details on how to adjust your build configuration on:
> https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[
https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309657#comment-16309657
]
Hive QA commented on HIVE-18079:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904357/HIVE-18079.6.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 50 failed/errored test(s), 11570 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_join1]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_6]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_groupby]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez2]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainanalyze_2]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby2]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join1]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join32_lessSize]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mapjoin_decimal]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[quotedid_smb]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[semijoin]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[semijoin_hint]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[skewjoin]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[smb_cache]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[smb_mapjoin_15]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[smb_mapjoin_6]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_corr]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_in]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_notin]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_bmj_schema_evolution]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_join_hash]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_union_multiinsert]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_cast_constant]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_count_distinct]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_context]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=102)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=102)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[tez-tag]
(batchId=102)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket_map_join_tez2]
(batchId=109)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.io.Te
[jira] [Updated] (HIVE-18353) CompactorMR should call jobclient.close() to trigger cleanup
[
https://issues.apache.org/jira/browse/HIVE-18353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated HIVE-18353:
-
Attachment: HIVE-18353.patch
> CompactorMR should call jobclient.close() to trigger cleanup
>
>
> Key: HIVE-18353
> URL: https://issues.apache.org/jira/browse/HIVE-18353
> Project: Hive
> Issue Type: Bug
> Components: Hive, Transactions
>Affects Versions: 1.2.1
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
> Attachments: HIVE-18353.patch
>
>
> HiveMetastore process is leaking TrustStore reloader threads when running
> compaction as JobClient close is not called from CompactorMR - MAPREDUCE-6618
> and MAPREDUCE-6621
> {code}
> "Truststore reloader thread" #2814 daemon prio=1 os_prio=0
> tid=0x00cdc800 nid=0x2f05a waiting on condition [0x7fdaef403000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.security.ssl.ReloadingX509TrustManager.run(ReloadingX509TrustManager.java:194)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18353) CompactorMR should call jobclient.close() to trigger cleanup
[
https://issues.apache.org/jira/browse/HIVE-18353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated HIVE-18353:
-
Status: Patch Available (was: Open)
> CompactorMR should call jobclient.close() to trigger cleanup
>
>
> Key: HIVE-18353
> URL: https://issues.apache.org/jira/browse/HIVE-18353
> Project: Hive
> Issue Type: Bug
> Components: Hive, Transactions
>Affects Versions: 1.2.1
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
> Attachments: HIVE-18353.patch
>
>
> HiveMetastore process is leaking TrustStore reloader threads when running
> compaction as JobClient close is not called from CompactorMR - MAPREDUCE-6618
> and MAPREDUCE-6621
> {code}
> "Truststore reloader thread" #2814 daemon prio=1 os_prio=0
> tid=0x00cdc800 nid=0x2f05a waiting on condition [0x7fdaef403000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.security.ssl.ReloadingX509TrustManager.run(ReloadingX509TrustManager.java:194)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18362) Introduce a parameter to control the max row number for map join convertion
[
https://issues.apache.org/jira/browse/HIVE-18362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309740#comment-16309740
]
Hive QA commented on HIVE-18362:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904365/HIVE-18362-branch-1.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 155 failed/errored test(s), 7897 tests
executed
*Failed tests:*
{noformat}
TestAdminUser - did not produce a TEST-*.xml file (likely timed out)
(batchId=339)
TestAuthorizationPreEventListener - did not produce a TEST-*.xml file (likely
timed out) (batchId=370)
TestAuthzApiEmbedAuthorizerInEmbed - did not produce a TEST-*.xml file (likely
timed out) (batchId=349)
TestAuthzApiEmbedAuthorizerInRemote - did not produce a TEST-*.xml file (likely
timed out) (batchId=355)
TestBeeLineWithArgs - did not produce a TEST-*.xml file (likely timed out)
(batchId=377)
TestCLIAuthzSessionContext - did not produce a TEST-*.xml file (likely timed
out) (batchId=393)
TestClientSideAuthorizationProvider - did not produce a TEST-*.xml file (likely
timed out) (batchId=369)
TestCompactor - did not produce a TEST-*.xml file (likely timed out)
(batchId=359)
TestCreateUdfEntities - did not produce a TEST-*.xml file (likely timed out)
(batchId=358)
TestCustomAuthentication - did not produce a TEST-*.xml file (likely timed out)
(batchId=378)
TestDBTokenStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=324)
TestDDLWithRemoteMetastoreSecondNamenode - did not produce a TEST-*.xml file
(likely timed out) (batchId=357)
TestDynamicSerDe - did not produce a TEST-*.xml file (likely timed out)
(batchId=327)
TestEmbeddedHiveMetaStore - did not produce a TEST-*.xml file (likely timed
out) (batchId=336)
TestEmbeddedThriftBinaryCLIService - did not produce a TEST-*.xml file (likely
timed out) (batchId=381)
TestFilterHooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=331)
TestFolderPermissions - did not produce a TEST-*.xml file (likely timed out)
(batchId=364)
TestHS2AuthzContext - did not produce a TEST-*.xml file (likely timed out)
(batchId=396)
TestHS2AuthzSessionContext - did not produce a TEST-*.xml file (likely timed
out) (batchId=397)
TestHS2ImpersonationWithRemoteMS - did not produce a TEST-*.xml file (likely
timed out) (batchId=385)
TestHiveAuthorizerCheckInvocation - did not produce a TEST-*.xml file (likely
timed out) (batchId=373)
TestHiveAuthorizerShowFilters - did not produce a TEST-*.xml file (likely timed
out) (batchId=372)
TestHiveHistory - did not produce a TEST-*.xml file (likely timed out)
(batchId=375)
TestHiveMetaStoreTxns - did not produce a TEST-*.xml file (likely timed out)
(batchId=351)
TestHiveMetaStoreWithEnvironmentContext - did not produce a TEST-*.xml file
(likely timed out) (batchId=341)
TestHiveMetaTool - did not produce a TEST-*.xml file (likely timed out)
(batchId=354)
TestHiveServer2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=399)
TestHiveServer2SessionTimeout - did not produce a TEST-*.xml file (likely timed
out) (batchId=400)
TestHiveSessionImpl - did not produce a TEST-*.xml file (likely timed out)
(batchId=382)
TestHs2Hooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=356)
TestHs2HooksWithMiniKdc - did not produce a TEST-*.xml file (likely timed out)
(batchId=428)
TestJdbcDriver2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=387)
TestJdbcMetadataApiAuth - did not produce a TEST-*.xml file (likely timed out)
(batchId=398)
TestJdbcWithLocalClusterSpark - did not produce a TEST-*.xml file (likely timed
out) (batchId=392)
TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=389)
TestJdbcWithMiniKdc - did not produce a TEST-*.xml file (likely timed out)
(batchId=425)
TestJdbcWithMiniKdcCookie - did not produce a TEST-*.xml file (likely timed
out) (batchId=424)
TestJdbcWithMiniKdcSQLAuthBinary - did not produce a TEST-*.xml file (likely
timed out) (batchId=422)
TestJdbcWithMiniKdcSQLAuthHttp - did not produce a TEST-*.xml file (likely
timed out) (batchId=427)
TestJdbcWithMiniMr - did not produce a TEST-*.xml file (likely timed out)
(batchId=388)
TestJdbcWithSQLAuthUDFBlacklist - did not produce a TEST-*.xml file (likely
timed out) (batchId=394)
TestJdbcWithSQLAuthorization - did not produce a TEST-*.xml file (likely timed
out) (batchId=395)
TestLocationQueries - did not produce a TEST-*.xml file (likely timed out)
(batchId=362)
TestMTQueries - did not produce a TEST-*.xml file (likely timed out)
(batchId=360)
TestMarkPartition - did not produce a TEST-*.xml file (likely timed out)
(batchId=348)
TestMarkPartitionRemote - did not produce a TEST-*.xml file (likely timed out)
(batchId=352)
TestMetaStoreAuthorization - did not produce a TEST-*.xml file (likely time
[jira] [Commented] (HIVE-18359) Extend grouping set limits from int to long
[
https://issues.apache.org/jira/browse/HIVE-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309774#comment-16309774
]
Hive QA commented on HIVE-18359:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 2 new + 1416 unchanged - 1
fixed = 1418 total (was 1417) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 47 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 98c95c6 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8419/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8419/yetus/whitespace-eol.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8419/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Extend grouping set limits from int to long
> ---
>
> Key: HIVE-18359
> URL: https://issues.apache.org/jira/browse/HIVE-18359
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-18359.1.patch
>
>
> Grouping sets is broken for >32 columns because of usage of Int for bitmap
> (also GROUPING__ID virtual column). This assumption breaks grouping
> sets/rollups/cube when number of participating aggregation columns is >32.
> The easier fix would be extend it to Long for now. The correct fix would be
> to use BitSets everywhere but that would require GROUPING__ID column type to
> binary which will make predicates on GROUPING__ID difficult to deal with.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18362) Introduce a parameter to control the max row number for map join convertion
[ https://issues.apache.org/jira/browse/HIVE-18362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309791#comment-16309791 ] wan kun commented on HIVE-18362: Hi，[~gopalv] What we do is similar, but there is some difference in the implementation. My implementation is to take the table or partition's ROW_COUNT information directly from hive metastore, which does not need additional calculations. I also have a few questions to ask: 1. Why do you use NDV instead of using ROW_COUNT directly ? I think NDV will be less than the actual number of ROW, but the actual memory is linearly related to the number of ROW. 2., I'm sorry, I haven't had the test environment of hive 2.* for a while. Hive branch-2.* depends on ColStatistics's statistics. Can you tell me where does ColStatistics come from ? Is this nesessary to add extra calculation for additional column statistics before our job? 3. The checkNumberOfEntriesForHashTable function only checks the number of Entry of one RS at a time. Does it happen that multiple map table is loaded into memory together, resulting in OOM? There are also two following questions: 1. ConvertJoinMapJoin optimization is only used in TezCompiler ? Spark use SparkMapJoinOptimizer. There is no optimizer for MapReduce ? 2. in hive branch-1.2 does not have this part of the code (but this parameter is added in hive-default.xml.template, which should not be effective) > Introduce a parameter to control the max row number for map join convertion > --- > > Key: HIVE-18362 > URL: https://issues.apache.org/jira/browse/HIVE-18362 > Project: Hive > Issue Type: Bug > Components: Query Processor >Reporter: wan kun >Assignee: Gopal V >Priority: Minor > Attachments: HIVE-18362-branch-1.2.patch > > > The compression ratio of the Orc compressed file will be very high in some > cases. > The test table has three Int columns, with twelve million records, but the > compressed file size is only 4M. Hive will automatically converts the Join to > Map join, but this will cause memory overflow. So I think it is better to > have a parameter to limit to the total number of table records in the Map > Join convertion, and if the total number of records is larger than that, it > can not be converted to Map join. > *hive.auto.convert.join.max.number = 250L* > The default value for this parameter is 250, because so many records > occupy about 700M memory in clint JVM, and 250 records for Map Join are > also large tables. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18359) Extend grouping set limits from int to long
[
https://issues.apache.org/jira/browse/HIVE-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309835#comment-16309835
]
Hive QA commented on HIVE-18359:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904366/HIVE-18359.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 74 failed/errored test(s), 11543 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_groupby2]
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_groupby]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_cube1]
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_cube_multi_gby]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_id3]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_sets1]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_sets2]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_sets3]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_sets4]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_sets5]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_sets6]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_sets_grouping]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_sets_limit]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_window]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_rollup1]
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_rollup_empty]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_grouping_operators]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[limit_pushdown2]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_width_bucket]
(batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_grouping_sets]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[view_cbo] (batchId=67)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[multi_count_distinct_null]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cte_1]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby_rollup_empty]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_cube1]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_id1]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_id2]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_id3]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets1]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets2]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets3]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets4]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets5]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets6]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets_grouping]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets_limit]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.test
[jira] [Updated] (HIVE-18096) add a user-friendly show plan command
[ https://issues.apache.org/jira/browse/HIVE-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-18096: - Status: Patch Available (was: Open) > add a user-friendly show plan command > - > > Key: HIVE-18096 > URL: https://issues.apache.org/jira/browse/HIVE-18096 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Harish Jaiprakash > Attachments: HIVE-18096.01.patch > > > For admin to be able to get an overview of a resource plan. > We need to try to do this using sysdb. > If that is not possible to do in a nice way, we'd do a text-based one like > query explain, or desc extended table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18096) add a user-friendly show plan command
[ https://issues.apache.org/jira/browse/HIVE-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-18096: - Attachment: HIVE-18096.01.patch [~sershe], Sorry for the delay. Attaching the patch. > add a user-friendly show plan command > - > > Key: HIVE-18096 > URL: https://issues.apache.org/jira/browse/HIVE-18096 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Harish Jaiprakash > Attachments: HIVE-18096.01.patch > > > For admin to be able to get an overview of a resource plan. > We need to try to do this using sysdb. > If that is not possible to do in a nice way, we'd do a text-based one like > query explain, or desc extended table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[
https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18238:
Attachment: HIVE-18238.03.patch
#3)
* {{hive.doing.acid}} is also problematic; I suspect that after 1 acid
operation this flag is left turned on...
* transactional tests are working by changing the config in flight...so for
those test cases the isolation is disabled
* removed the {{Driver.resetQueryState()}} method...I don't think it was ok to
call that method even without isolation...
* removed a driver caching mechanism from {{CommandFactory}}; it seemed to me
that it probably didn't worked lately ; but the conf isolation have bringed it
back to life, and caused some issues - however since hs2 didn't use the cleaner
it provided, I've removed it.
> Driver execution may not have configuration changing sideeffects
> -
>
> Key: HIVE-18238
> URL: https://issues.apache.org/jira/browse/HIVE-18238
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18238.01wip01.patch, HIVE-18238.02.patch,
> HIVE-18238.03.patch
>
>
> {{Driver}} executes sql statements which use "hiveconf" settings;
> but the {{Driver}} itself may *not* change the configuration...
> I've found an example; which shows how hazardous this is...
> {code}
> set hive.mapred.mode=strict;
> select "${hiveconf:hive.mapred.mode}";
> create table t (a int);
> analyze table t compute statistics;
> select "${hiveconf:hive.mapred.mode}";
> {code}
> currently; the last select returns {{nonstrict}} because of
> [this|https://github.com/apache/hive/blob/7ddd915bf82a68c8ab73b0c4ca409f1a6d43d227/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1696]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18353) CompactorMR should call jobclient.close() to trigger cleanup
[
https://issues.apache.org/jira/browse/HIVE-18353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309867#comment-16309867
]
Hive QA commented on HIVE-18353:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
33s{color} | {color:red} ql: The patch generated 1 new + 67 unchanged - 2 fixed
= 68 total (was 69) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 98c95c6 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8420/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8420/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CompactorMR should call jobclient.close() to trigger cleanup
>
>
> Key: HIVE-18353
> URL: https://issues.apache.org/jira/browse/HIVE-18353
> Project: Hive
> Issue Type: Bug
> Components: Hive, Transactions
>Affects Versions: 1.2.1
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
> Attachments: HIVE-18353.patch
>
>
> HiveMetastore process is leaking TrustStore reloader threads when running
> compaction as JobClient close is not called from CompactorMR - MAPREDUCE-6618
> and MAPREDUCE-6621
> {code}
> "Truststore reloader thread" #2814 daemon prio=1 os_prio=0
> tid=0x00cdc800 nid=0x2f05a waiting on condition [0x7fdaef403000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.security.ssl.ReloadingX509TrustManager.run(ReloadingX509TrustManager.java:194)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18361) Extend shared work optimizer to reuse computation beyond work boundaries
[ https://issues.apache.org/jira/browse/HIVE-18361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18361: --- Attachment: HIVE-18361.01.patch > Extend shared work optimizer to reuse computation beyond work boundaries > > > Key: HIVE-18361 > URL: https://issues.apache.org/jira/browse/HIVE-18361 > Project: Hive > Issue Type: New Feature > Components: Physical Optimizer >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Labels: TODOC3.0 > Attachments: HIVE-18361.01.patch, HIVE-18361.patch > > > Follow-up of the work in HIVE-16867. > HIVE-16867 introduced an optimization that identifies scans on input tables > that can be merged and reuses the computation that is done in the work > containing those scans. In particular, we traverse both parts of the plan > upstream and reuse the operators if possible. > Currently, the optimizer will not go beyond the output edge(s) of that work. > This extension removes that limitation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18190) Consider looking at ORC file schema rather than using _metadata_acid file
[ https://issues.apache.org/jira/browse/HIVE-18190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309907#comment-16309907 ] Eugene Koifman commented on HIVE-18190: --- no related failures [~alangates] could you review please > Consider looking at ORC file schema rather than using _metadata_acid file > - > > Key: HIVE-18190 > URL: https://issues.apache.org/jira/browse/HIVE-18190 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18190.01.patch, HIVE-18190.02.patch, > HIVE-18190.04.patch, HIVE-18190.05.patch, HIVE-18190.06.patch, > HIVE-18190.07.patch > > > See if it's possible to just look at the schema of the file in base_ or > delta_ to see if it has Acid metadata columns. If not, it's an 'original' > file and needs ROW_IDs generated. > see more discussion at https://reviews.apache.org/r/64131/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17683) Annotate Query Plan with locking information
[ https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17683: -- Priority: Critical (was: Major) > Annotate Query Plan with locking information > > > Key: HIVE-17683 > URL: https://issues.apache.org/jira/browse/HIVE-17683 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > > Explore if it's possible to add info about what locks will be asked for to > the query plan. > Lock acquisition (for Acid Lock Manager) is done in > DbTxnManager.acquireLocks() which is called once the query starts running. > Would need to refactor that. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18358) from_unixtime returns wrong year for Dec 31 timestamps with format 'YYYY'
[
https://issues.apache.org/jira/browse/HIVE-18358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309927#comment-16309927
]
Andrew Sherman commented on HIVE-18358:
---
I tried
{noformat}
long unixtime = 1514754599;
SimpleDateFormat formatter = new SimpleDateFormat("-MM-dd HH-mm-ss");
formatter.setTimeZone(TimeZone.getTimeZone("Europe/London"));
Date date = new Date(unixtime * 1000L);
System.out.println("date = " + formatter.format(date));
{noformat}
which gives output
{noformat}
date = 2018-12-31 21-09-59
{noformat}
So it seems like the problem may be with SimpleDateFormat itself?
> from_unixtime returns wrong year for Dec 31 timestamps with format ''
> -
>
> Key: HIVE-18358
> URL: https://issues.apache.org/jira/browse/HIVE-18358
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
> Environment: AWS EMR with Hive 2.1.0-amzn-0
>Reporter: Nick Orka
>Assignee: Andrew Sherman
> Labels: timezone
>
> If you use capital Ys as a year format in from_unixtime() it returns next
> year for Dec 31 only. All other days work as intended.
> Here is reproduction code:
> {code:sql}
> hive> select from_unixtime(1514754599, '-MM-dd HH-mm-ss'),
> from_unixtime(1514754599, '-MM-dd HH-mm-ss');
> OK
> 2018-12-31 21-09-59 2017-12-31 21-09-59
> Time taken: 0.025 seconds, Fetched: 1 row(s)
> hive>
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18353) CompactorMR should call jobclient.close() to trigger cleanup
[
https://issues.apache.org/jira/browse/HIVE-18353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309935#comment-16309935
]
Hive QA commented on HIVE-18353:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904406/HIVE-18353.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 11542 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8420/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8420/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8420/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12904406 - PreCommit-HIVE-Build
> CompactorMR should call jobclient.close() to trigger cleanup
>
>
> Key: HIVE-18353
> URL: https://issues.apache.org/jira/browse/HIVE-18353
> Project: Hive
> Issue Type: Bug
> Components: Hive, Transactions
>Affects Versions: 1.2.1
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
> Attachments: HIVE-18353.patch
>
>
> HiveMetastore process is leaking TrustStore reloader threads when running
> compaction as JobClient close is not called from CompactorMR - MAPREDUCE-6618
> and MAPREDUCE-6621
> {code}
> "Truststore reloader thread" #2814 daemon prio=1 os_prio=0
> tid=0x00cdc800 nid=0x2f05a waiting on condition [0x7fdaef403000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.security.ssl.ReloadingX509TrustManager.run(ReloadingX509TrustManager.java:194)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit
[
https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-16484:
Status: Patch Available (was: Open)
> Investigate SparkLauncher for HoS as alternative to bin/spark-submit
>
>
> Key: HIVE-16484
> URL: https://issues.apache.org/jira/browse/HIVE-16484
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16484.1.patch, HIVE-16484.2.patch,
> HIVE-16484.3.patch, HIVE-16484.4.patch, HIVE-16484.5.patch,
> HIVE-16484.6.patch, HIVE-16484.7.patch, HIVE-16484.8.patch
>
>
> The {{SparkClientImpl#startDriver}} currently looks for the {{SPARK_HOME}}
> directory and invokes the {{bin/spark-submit}} script, which spawns a
> separate process to run the Spark application.
> {{SparkLauncher}} was added in SPARK-4924 and is a programatic way to launch
> Spark applications.
> I see a few advantages:
> * No need to spawn a separate process to launch a HoS --> lower startup time
> * Simplifies the code in {{SparkClientImpl}} --> easier to debug
> * {{SparkLauncher#startApplication}} returns a {{SparkAppHandle}} which
> contains some useful utilities for querying the state of the Spark job
> ** It also allows the launcher to specify a list of job listeners
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18096) add a user-friendly show plan command
[
https://issues.apache.org/jira/browse/HIVE-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309972#comment-16309972
]
Hive QA commented on HIVE-18096:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} standalone-metastore: The patch generated 1 new + 2378
unchanged - 1 fixed = 2379 total (was 2379) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 16 new + 977 unchanged - 2
fixed = 993 total (was 979) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 98c95c6 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8421/yetus/diff-checkstyle-standalone-metastore.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8421/yetus/diff-checkstyle-ql.txt
|
| modules | C: standalone-metastore ql itests/hcatalog-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8421/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> add a user-friendly show plan command
> -
>
> Key: HIVE-18096
> URL: https://issues.apache.org/jira/browse/HIVE-18096
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Harish Jaiprakash
> Attachments: HIVE-18096.01.patch
>
>
> For admin to be able to get an overview of a resource plan.
> We need to try to do this using sysdb.
> If that is not possible to do in a nice way, we'd do a text-based one like
> query explain, or desc extended table.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18255) spark-client jar should be prefixed with hive-
[ https://issues.apache.org/jira/browse/HIVE-18255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-18255: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master > spark-client jar should be prefixed with hive- > -- > > Key: HIVE-18255 > URL: https://issues.apache.org/jira/browse/HIVE-18255 > Project: Hive > Issue Type: Bug > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Fix For: 3.0.0 > > Attachments: HIVE-18255.1.patch, HIVE-18255.2.patch, > HIVE-18255.3.patch > > > Other Hive jars are prefixed with "hive-" except for the spark-client jar. > Fixing this to make sure the jar name is consistent across all Hive jars. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18346) Beeline could not launch because of size of history
[
https://issues.apache.org/jira/browse/HIVE-18346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310007#comment-16310007
]
Andrew Sherman commented on HIVE-18346:
---
Thanks [~minions] sounds like you are right, though I am not an expert. Thanks
for doing this work and sorry the bug I pointed you to was not so interesting.
> Beeline could not launch because of size of history
> ---
>
> Key: HIVE-18346
> URL: https://issues.apache.org/jira/browse/HIVE-18346
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.2.1
>Reporter: Jumping
>Assignee: Madhudeep Petwal
>Priority: Minor
>
> Beeline version 1.2.1 could not launch when the size of
> ${user.home}/.beeline/history larger than 39MB. Which reports
> "java.lang.outofmemoryerror" .
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18360) NPE in TezSessionState
[ https://issues.apache.org/jira/browse/HIVE-18360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310023#comment-16310023 ] Jason Dere commented on HIVE-18360: --- +1 > NPE in TezSessionState > -- > > Key: HIVE-18360 > URL: https://issues.apache.org/jira/browse/HIVE-18360 > Project: Hive > Issue Type: Bug >Reporter: Deepesh Khandelwal >Assignee: Sergey Shelukhin > Attachments: HIVE-18360.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18365) netty-all jar is not present in the llap tarball
[ https://issues.apache.org/jira/browse/HIVE-18365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-18365: - > netty-all jar is not present in the llap tarball > > > Key: HIVE-18365 > URL: https://issues.apache.org/jira/browse/HIVE-18365 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely > Fix For: 3.0.0 > > > netty-all jar should be part of the llap tarball, as it requires it. Earlier > it was part of the tez libs, but it was removed from there, so it must be > added to it explicitly. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18361) Extend shared work optimizer to reuse computation beyond work boundaries
[ https://issues.apache.org/jira/browse/HIVE-18361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310029#comment-16310029 ] Ashutosh Chauhan commented on HIVE-18361: - +1 > Extend shared work optimizer to reuse computation beyond work boundaries > > > Key: HIVE-18361 > URL: https://issues.apache.org/jira/browse/HIVE-18361 > Project: Hive > Issue Type: New Feature > Components: Physical Optimizer >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Labels: TODOC3.0 > Attachments: HIVE-18361.01.patch, HIVE-18361.patch > > > Follow-up of the work in HIVE-16867. > HIVE-16867 introduced an optimization that identifies scans on input tables > that can be merged and reuses the computation that is done in the work > containing those scans. In particular, we traverse both parts of the plan > upstream and reuse the operators if possible. > Currently, the optimizer will not go beyond the output edge(s) of that work. > This extension removes that limitation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18365) netty-all jar is not present in the llap tarball
[ https://issues.apache.org/jira/browse/HIVE-18365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-18365: -- Attachment: HIVE-18365.patch > netty-all jar is not present in the llap tarball > > > Key: HIVE-18365 > URL: https://issues.apache.org/jira/browse/HIVE-18365 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely > Fix For: 3.0.0 > > Attachments: HIVE-18365.patch > > > netty-all jar should be part of the llap tarball, as it requires it. Earlier > it was part of the tez libs, but it was removed from there, so it must be > added to it explicitly. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18365) netty-all jar is not present in the llap tarball
[ https://issues.apache.org/jira/browse/HIVE-18365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-18365: Status: Patch Available (was: Open) > netty-all jar is not present in the llap tarball > > > Key: HIVE-18365 > URL: https://issues.apache.org/jira/browse/HIVE-18365 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely > Fix For: 3.0.0 > > Attachments: HIVE-18365.patch > > > netty-all jar should be part of the llap tarball, as it requires it. Earlier > it was part of the tez libs, but it was removed from there, so it must be > added to it explicitly. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18096) add a user-friendly show plan command
[
https://issues.apache.org/jira/browse/HIVE-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310035#comment-16310035
]
Hive QA commented on HIVE-18096:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904420/HIVE-18096.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 11527 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=146)
[mapreduce2.q,orc_llap_counters1.q,bucket6.q,insert_into1.q,empty_dir_in_table.q,orc_merge1.q,parquet_types_vectorization.q,orc_merge_diff_fs.q,llap_stats.q,llapdecider.q,load_hdfs_file_with_space_in_the_name.q,llap_nullscan.q,orc_ppd_basic.q,rcfile_merge4.q,orc_merge3.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8421/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8421/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8421/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 19 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12904420 - PreCommit-HIVE-Build
> add a user-friendly show plan command
> -
>
> Key: HIVE-18096
> URL: https://issues.apache.org/jira/browse/HIVE-18096
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Harish Jaiprakash
> Attachments: HIVE-18096.01.patch
>
>
> For admin to be able to get an overview of a resource plan.
> We need to try to do this using sysdb.
> If that is not possible to do in a nice way, we'd do a text-based one like
> query explain, or desc extended table.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18353) CompactorMR should call jobclient.close() to trigger cleanup
[
https://issues.apache.org/jira/browse/HIVE-18353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated HIVE-18353:
-
Attachment: HIVE-18353.1.patch
> CompactorMR should call jobclient.close() to trigger cleanup
>
>
> Key: HIVE-18353
> URL: https://issues.apache.org/jira/browse/HIVE-18353
> Project: Hive
> Issue Type: Bug
> Components: Hive, Transactions
>Affects Versions: 1.2.1
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
> Attachments: HIVE-18353.1.patch, HIVE-18353.patch
>
>
> HiveMetastore process is leaking TrustStore reloader threads when running
> compaction as JobClient close is not called from CompactorMR - MAPREDUCE-6618
> and MAPREDUCE-6621
> {code}
> "Truststore reloader thread" #2814 daemon prio=1 os_prio=0
> tid=0x00cdc800 nid=0x2f05a waiting on condition [0x7fdaef403000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.security.ssl.ReloadingX509TrustManager.run(ReloadingX509TrustManager.java:194)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[
https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310065#comment-16310065
]
Eugene Koifman commented on HIVE-18238:
---
Why not make the Driver make clone of the passed in Configuration object so
that the side effects don't propagate outside the context of the query?
In fact Driver used to do exactly that in the past - I'm not sure when that was
changed or why.
> Driver execution may not have configuration changing sideeffects
> -
>
> Key: HIVE-18238
> URL: https://issues.apache.org/jira/browse/HIVE-18238
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18238.01wip01.patch, HIVE-18238.02.patch,
> HIVE-18238.03.patch
>
>
> {{Driver}} executes sql statements which use "hiveconf" settings;
> but the {{Driver}} itself may *not* change the configuration...
> I've found an example; which shows how hazardous this is...
> {code}
> set hive.mapred.mode=strict;
> select "${hiveconf:hive.mapred.mode}";
> create table t (a int);
> analyze table t compute statistics;
> select "${hiveconf:hive.mapred.mode}";
> {code}
> currently; the last select returns {{nonstrict}} because of
> [this|https://github.com/apache/hive/blob/7ddd915bf82a68c8ab73b0c4ca409f1a6d43d227/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1696]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[
https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310065#comment-16310065
]
Eugene Koifman edited comment on HIVE-18238 at 1/3/18 6:53 PM:
---
Why not make the Driver make clone of the passed in Configuration object so
that the side effects don't propagate outside the context of the query?
In fact Driver used to do exactly that in the past - I'm not sure when that was
changed or why.
Configuration is passed to all the tasks executing on behalf of the query so
setting properties in that object is necessary to control the tasks.
was (Author: ekoifman):
Why not make the Driver make clone of the passed in Configuration object so
that the side effects don't propagate outside the context of the query?
In fact Driver used to do exactly that in the past - I'm not sure when that was
changed or why.
> Driver execution may not have configuration changing sideeffects
> -
>
> Key: HIVE-18238
> URL: https://issues.apache.org/jira/browse/HIVE-18238
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18238.01wip01.patch, HIVE-18238.02.patch,
> HIVE-18238.03.patch
>
>
> {{Driver}} executes sql statements which use "hiveconf" settings;
> but the {{Driver}} itself may *not* change the configuration...
> I've found an example; which shows how hazardous this is...
> {code}
> set hive.mapred.mode=strict;
> select "${hiveconf:hive.mapred.mode}";
> create table t (a int);
> analyze table t compute statistics;
> select "${hiveconf:hive.mapred.mode}";
> {code}
> currently; the last select returns {{nonstrict}} because of
> [this|https://github.com/apache/hive/blob/7ddd915bf82a68c8ab73b0c4ca409f1a6d43d227/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1696]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18365) netty-all jar is not present in the llap tarball
[ https://issues.apache.org/jira/browse/HIVE-18365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-18365: --- Component/s: (was: HiveServer2) llap > netty-all jar is not present in the llap tarball > > > Key: HIVE-18365 > URL: https://issues.apache.org/jira/browse/HIVE-18365 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely > Fix For: 3.0.0 > > Attachments: HIVE-18365.patch > > > netty-all jar should be part of the llap tarball, as it requires it. Earlier > it was part of the tez libs, but it was removed from there, so it must be > added to it explicitly. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18365) netty-all jar is not present in the llap tarball
[ https://issues.apache.org/jira/browse/HIVE-18365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310100#comment-16310100 ] Gopal V commented on HIVE-18365: +1 tests pending. Can you confirm it is picking up netty-all and not the split apart netty jars, with a listing of the llap tarball files? > netty-all jar is not present in the llap tarball > > > Key: HIVE-18365 > URL: https://issues.apache.org/jira/browse/HIVE-18365 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely > Fix For: 3.0.0 > > Attachments: HIVE-18365.patch > > > netty-all jar should be part of the llap tarball, as it requires it. Earlier > it was part of the tez libs, but it was removed from there, so it must be > added to it explicitly. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[
https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310112#comment-16310112
]
Hive QA commented on HIVE-18238:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 13 new + 1159 unchanged - 3
fixed = 1172 total (was 1162) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} cli: The patch generated 0 new + 38 unchanged - 1
fixed = 38 total (was 39) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} hcatalog/core: The patch generated 0 new + 33
unchanged - 1 fixed = 33 total (was 34) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch hcatalog-pig-adapter passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch server-extensions passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / ed99182 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8422/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql cli hcatalog/core hcatalog/hcatalog-pig-adapter
hcatalog/server-extensions itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8422/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Driver execution may not have configuration changing sideeffects
> -
>
> Key: HIVE-18238
> URL: https://issues.apache.org/jira/browse/HIVE-18238
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18238.01wip01.patch
[jira] [Commented] (HIVE-18291) An exception should be raised if the result is outside the range of decimal
[
https://issues.apache.org/jira/browse/HIVE-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310116#comment-16310116
]
Daniel Voros commented on HIVE-18291:
-
[~sershe] I think that ticket is HIVE-13098. Based on the size of that WIP
patch, I think I'll put this on hold for now.
> An exception should be raised if the result is outside the range of decimal
> ---
>
> Key: HIVE-18291
> URL: https://issues.apache.org/jira/browse/HIVE-18291
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 2.3.0
>Reporter: Marco Gaido
>Assignee: Daniel Voros
>
> Citing SQL:2011 on page 27 available at
> http://standards.iso.org/ittf/PubliclyAvailableStandards/c053681_ISO_IEC_9075-1_2011.zip:
> {noformat}
> If the result cannot be represented exactly in the result type, then whether
> it is rounded
> or truncated is implementation-defined. An exception condition is raised if
> the result is
> outside the range of numeric values of the result type, or if the arithmetic
> operation
> is not defined for the operands.
> {noformat}
> Currently Hive is returning NULL instead of throwing an exception if the
> result is out of range, eg.:
> {code}
> > select 100.01*100.01;
> +---+
> | _c0 |
> +---+
> | NULL |
> +---+
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18366) Update HBaseSerDe to use hbase.mapreduce.hfileoutputformat.table.name instead of hbase.table.name as the table name property
[
https://issues.apache.org/jira/browse/HIVE-18366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu reassigned HIVE-18366:
---
> Update HBaseSerDe to use hbase.mapreduce.hfileoutputformat.table.name instead
> of hbase.table.name as the table name property
>
>
> Key: HIVE-18366
> URL: https://issues.apache.org/jira/browse/HIVE-18366
> Project: Hive
> Issue Type: Sub-task
> Components: HBase Handler
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>
> HBase 2.0 changes the table name property to
> hbase.mapreduce.hfileoutputformat.table.name. HiveHFileOutputFormat is using
> the new property name while HiveHBaseTableOutputFormat is not. If we create
> the table as follows, HiveHBaseTableOutputFormat is used which still uses the
> old property hbase.table.name.
> {noformat}
> create table hbase_table2(key int, val string) stored by
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties
> ('hbase.columns.mapping' = ':key,cf:val') tblproperties
> ('hbase.mapreduce.hfileoutputformat.table.name' =
> 'positive_hbase_handler_bulk')
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-14615) Temp table leaves behind insert command
[
https://issues.apache.org/jira/browse/HIVE-14615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated HIVE-14615:
--
Attachment: HIVE-14615.2.patch
> Temp table leaves behind insert command
> ---
>
> Key: HIVE-14615
> URL: https://issues.apache.org/jira/browse/HIVE-14615
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Chaoyu Tang
>Assignee: Andrew Sherman
> Attachments: HIVE-14615.1.patch, HIVE-14615.2.patch
>
>
> {code}
> create table test (key int, value string);
> insert into test values (1, 'val1');
> show tables;
> test
> values__tmp__table__1
> {code}
> the temp table values__tmp__table__1 was resulted from insert into ...values
> and exists until logout the session.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18214) Flaky test: TestSparkClient
[
https://issues.apache.org/jira/browse/HIVE-18214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310134#comment-16310134
]
Sahil Takiar commented on HIVE-18214:
-
Test failures look un-related.
[~pvary], [~aihuaxu] could you review?
> Flaky test: TestSparkClient
> ---
>
> Key: HIVE-18214
> URL: https://issues.apache.org/jira/browse/HIVE-18214
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-18214.1.patch
>
>
> Looks like there is a race condition in {{TestSparkClient#runTest}}. The test
> creates a {{RemoteDriver}} in memory, which creates a {{JavaSparkContext}}. A
> new {{JavaSparkContext}} is created for each test that is run. There is a
> race condition where the {{RemoteDriver}} isn't given enough time to
> shutdown, so when the next test starts running it creates another
> {{JavaSparkContext}} which causes an exception like
> {{org.apache.spark.SparkException: Only one SparkContext may be running in
> this JVM (see SPARK-2243)}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[
https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310180#comment-16310180
]
Sahil Takiar commented on HIVE-18238:
-
[~kgyrtkirk] I think there are some users who are expecting
{{hive.query.string}} to be set to the last executed query. Although, that gets
a bit murky when there are queries being run in the same session in parallel.
https://www.quora.com/How-do-I-get-the-Hive-query-statement-that-I-have-executed-in-Yarn
https://mail-archives.apache.org/mod_mbox/hive-user/201312.mbox/%3c068f8f823eee4b379e6346a919d05...@blupr08mb296.namprd08.prod.outlook.com%3E
> Driver execution may not have configuration changing sideeffects
> -
>
> Key: HIVE-18238
> URL: https://issues.apache.org/jira/browse/HIVE-18238
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18238.01wip01.patch, HIVE-18238.02.patch,
> HIVE-18238.03.patch
>
>
> {{Driver}} executes sql statements which use "hiveconf" settings;
> but the {{Driver}} itself may *not* change the configuration...
> I've found an example; which shows how hazardous this is...
> {code}
> set hive.mapred.mode=strict;
> select "${hiveconf:hive.mapred.mode}";
> create table t (a int);
> analyze table t compute statistics;
> select "${hiveconf:hive.mapred.mode}";
> {code}
> currently; the last select returns {{nonstrict}} because of
> [this|https://github.com/apache/hive/blob/7ddd915bf82a68c8ab73b0c4ca409f1a6d43d227/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1696]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17929) Use sessionId for HoS Remote Driver Client id
[
https://issues.apache.org/jira/browse/HIVE-17929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17929:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master.
> Use sessionId for HoS Remote Driver Client id
> -
>
> Key: HIVE-17929
> URL: https://issues.apache.org/jira/browse/HIVE-17929
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Fix For: 3.0.0
>
> Attachments: HIVE-17929.1.patch, HIVE-17929.2.patch,
> HIVE-17929.3.patch
>
>
> Each {{SparkClientImpl}} creates a client connection using a client id. The
> client id is created via {{UUID.randomUUID()}}.
> Since each HoS session has a single client connection we should just use the
> sessionId instead (which is also a UUID). This should help simplify the code
> and some of the client logging.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[
https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310192#comment-16310192
]
Hive QA commented on HIVE-18238:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904425/HIVE-18238.03.patch
{color:green}SUCCESS:{color} +1 due to 9 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 95 failed/errored test(s), 11148 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=92)
[nopart_insert.q,insert_into_with_schema.q,input41.q,having1.q,create_table_failure3.q,database_drop_not_empty_restrict.q,windowing_after_orderby.q,orderbysortby.q,subquery_select_distinct2.q,authorization_uri_alterpart_loc.q,udf_last_day_error_1.q,create_table_failure4.q,semijoin5.q,udf_format_number_wrong4.q,deletejar.q,exim_11_nonpart_noncompat_sorting.q,show_tables_bad_db2.q,drop_func_nonexistent.q,nopart_load.q,alter_table_non_partitioned_table_cascade.q,load_wrong_fileformat.q,lockneg_try_db_lock_conflict.q,udf_field_wrong_args_len.q,create_table_failure2.q,groupby2_map_skew_multi_distinct.q,udf_min.q,authorization_update_noupdatepriv.q,show_columns2.q,authorization_insert_noselectpriv.q,orc_replace_columns3_acid.q,udf_instr_wrong_args_len.q,compare_double_bigint.q,authorization_set_nonexistent_conf.q,alter_rename_partition_failure3.q,split_sample_wrong_format2.q,create_with_fk_pk_same_tab.q,authorization_show_roles_no_admin.q,materialized_view_authorization_rebuild_no_grant.q,unionLimit.q,authorization_revoke_table_fail2.q,authorization_insert_noinspriv.q,duplicate_insert3.q,authorization_desc_table_nosel.q,invalid_select_column.q,stats_noscan_non_native.q,orc_change_serde_acid.q,create_or_replace_view7.q,exim_07_nonpart_noncompat_ifof.q,udf_concat_ws_wrong2.q,fileformat_bad_class.q,merge_negative_2.q,exim_15_part_nonpart.q,authorization_not_owner_drop_view.q,external1.q,authorization_uri_insert.q,create_with_fk_wrong_ref.q,columnstats_tbllvl_incorrect_column.q,authorization_show_parts_nosel.q,merge_negative_1.q,authorization_not_owner_drop_tab.q,external2.q,authorization_deletejar.q,temp_table_create_like_partitions.q,udf_greatest_error_1.q,ptf_negative_AggrFuncsWithNoGBYNoPartDef.q,alter_view_as_select_not_exist.q,touch1.q,groupby3_map_skew_multi_distinct.q,exchange_partition_neg_partition_missing.q,groupby_cube_multi_gby.q,columnstats_tbllvl.q,drop_invalid_constraint2.q,alter_table_add_partition.q,update_not_acid.q,archive5.q,alter_table_constraint_invalid_pk_col.q,ivyDownload.q,udf_instr_wrong_type.q,bad_sample_clause.q,authorization_not_owner_drop_tab2.q,authorization_alter_db_owner.q,show_columns1.q,orc_type_promotion3.q,create_view_failure8.q,strict_join.q,udf_add_months_error_1.q,groupby_cube2.q,drop_partition_filter_failure.q,groupby_cube1.q,groupby_rollup1.q,genericFileFormat.q,authorization_create_macro1.q,invalid_cast_from_binary_4.q,drop_invalid_constraint1.q,serde_regex.q,show_partitions1.q,invalid_cast_from_binary_6.q,create_with_multi_pk_constraint.q,udf_field_wrong_type.q,groupby_grouping_sets4.q,groupby_grouping_sets3.q,load_data_into_acid.q,insertsel_fail.q,udf_locate_wrong_type.q,orc_type_promotion1_acid.q,set_table_property.q,create_or_replace_view2.q,groupby_grouping_sets2.q,alter_view_failure.q,distinct_windowing_failure1.q,invalid_t_alter2.q,alter_table_constraint_invalid_fk_col1.q,invalid_varchar_length_2.q,authorization_show_grant_otheruser_alltabs.q,subquery_windowing_corr.q,compact_non_acid_table.q,authorization_view_4.q,authorization_disallow_transform.q,materialized_view_authorization_rebuild_other.q,authorization_fail_4.q,dbtxnmgr_nodblock.q,set_hiveconf_internal_variable1.q,input_part0_neg.q,udf_printf_wrong3.q,load_orc_negative2.q,druid_buckets.q,archive2.q,authorization_addjar.q,invalid_sum_syntax.q,insert_into_with_schema1.q,udf_add_months_error_2.q,dyn_part_max_per_node.q,authorization_revoke_table_fail1.q,udf_printf_wrong2.q,archive_multi3.q,udf_printf_wrong1.q,subquery_subquery_chain.q,authorization_view_disable_cbo_4.q,no_matching_udf.q,char_pad_convert_fail0.q,create_view_failure7.q,drop_native_udf.q,truncate_column_list_bucketing.q,authorization_uri_add_partition.q,authorization_view_disable_cbo_3.q,bad_exec_hooks.q,authorization_view_disable_cbo_2.q,fetchtask_ioexception.q,char_pad_convert_fail2.q,ptf_window_boundaries.q,authorization_set_role_neg1.q,serde_regex3.q,authorization_delete_nodeletepriv.q,materialized_view_delete.q,create_or_replace_view6.q,bucket_mapjoin_wrong_table_metadata_2.q,msck_repair_3.q,udf_sort_array_by_wrong2.q,local_mapred_error_cache.q,alter_external_acid.q,mm_concatenate.q,authorization_fail_3.q,set_hiveconf_internal_variable0.q,udf_last_day_error_2.q,alter_table_constraint_invalid_ref.q,create_table_wrong_regex.q,describe_xpath4.q,join32.q,insert_sorted.q,describe_xpath2.q,authori
[jira] [Commented] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[
https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310209#comment-16310209
]
Eugene Koifman commented on HIVE-18238:
---
[~stakiar] does Hive really support parallel query execution from a single
session? Is there any infrastructure somewhere to ensure that they don't step
on each other somehow?
> Driver execution may not have configuration changing sideeffects
> -
>
> Key: HIVE-18238
> URL: https://issues.apache.org/jira/browse/HIVE-18238
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18238.01wip01.patch, HIVE-18238.02.patch,
> HIVE-18238.03.patch
>
>
> {{Driver}} executes sql statements which use "hiveconf" settings;
> but the {{Driver}} itself may *not* change the configuration...
> I've found an example; which shows how hazardous this is...
> {code}
> set hive.mapred.mode=strict;
> select "${hiveconf:hive.mapred.mode}";
> create table t (a int);
> analyze table t compute statistics;
> select "${hiveconf:hive.mapred.mode}";
> {code}
> currently; the last select returns {{nonstrict}} because of
> [this|https://github.com/apache/hive/blob/7ddd915bf82a68c8ab73b0c4ca409f1a6d43d227/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1696]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-13000) Hive returns useless parsing error
[ https://issues.apache.org/jira/browse/HIVE-13000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13000: Status: Open (was: Patch Available) > Hive returns useless parsing error > --- > > Key: HIVE-13000 > URL: https://issues.apache.org/jira/browse/HIVE-13000 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0, 1.2.1, 1.0.0, 0.13.0 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Attachments: HIVE-13000.1.patch, HIVE-13000.2.patch, > HIVE-13000.3.patch, HIVE-13000.4.patch, HIVE-13000.5.patch, HIVE-13000.6.patch > > > When I run query like these I receive unclear exception > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Error in parsing > It will be clearer if it would be like: > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Expression not in GROUP BY key record -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-13000) Hive returns useless parsing error
[ https://issues.apache.org/jira/browse/HIVE-13000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13000: Attachment: HIVE-13000.6.patch > Hive returns useless parsing error > --- > > Key: HIVE-13000 > URL: https://issues.apache.org/jira/browse/HIVE-13000 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.0, 1.0.0, 1.2.1, 2.2.0 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Attachments: HIVE-13000.1.patch, HIVE-13000.2.patch, > HIVE-13000.3.patch, HIVE-13000.4.patch, HIVE-13000.5.patch, HIVE-13000.6.patch > > > When I run query like these I receive unclear exception > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Error in parsing > It will be clearer if it would be like: > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Expression not in GROUP BY key record -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-13000) Hive returns useless parsing error
[ https://issues.apache.org/jira/browse/HIVE-13000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13000: Status: Patch Available (was: Open) > Hive returns useless parsing error > --- > > Key: HIVE-13000 > URL: https://issues.apache.org/jira/browse/HIVE-13000 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0, 1.2.1, 1.0.0, 0.13.0 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Attachments: HIVE-13000.1.patch, HIVE-13000.2.patch, > HIVE-13000.3.patch, HIVE-13000.4.patch, HIVE-13000.5.patch, HIVE-13000.6.patch > > > When I run query like these I receive unclear exception > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Error in parsing > It will be clearer if it would be like: > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Expression not in GROUP BY key record -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18360) NPE in TezSessionState
[ https://issues.apache.org/jira/browse/HIVE-18360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18360: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the review! cc [~deepesh] > NPE in TezSessionState > -- > > Key: HIVE-18360 > URL: https://issues.apache.org/jira/browse/HIVE-18360 > Project: Hive > Issue Type: Bug >Reporter: Deepesh Khandelwal >Assignee: Sergey Shelukhin > Fix For: 3.0.0 > > Attachments: HIVE-18360.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18349) Misc metastore changes for debuggability
[ https://issues.apache.org/jira/browse/HIVE-18349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18349: - Attachment: HIVE-18349.1.patch > Misc metastore changes for debuggability > > > Key: HIVE-18349 > URL: https://issues.apache.org/jira/browse/HIVE-18349 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran > Attachments: HIVE-18349.1.patch > > > 1) Hive metastore audit event log/metastore log does not log the final status > (success or failed) of the event. Some operations like for example, > drop_table returns a boolean success flag but it never gets logged anywhere. > However the same is sent to end event listeners or other metastore event > listeners. It will be good to log the final status of the events. > 2) Make connection timeout when using connection pool configurable. Currently > its hard coded to 30 seconds. > 3) Provide a config to enable connection leak detection for HikariCP or > enable when debug logging is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18349) Misc metastore changes for debuggability
[ https://issues.apache.org/jira/browse/HIVE-18349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18349: - Attachment: HIVE-18349.1.patch > Misc metastore changes for debuggability > > > Key: HIVE-18349 > URL: https://issues.apache.org/jira/browse/HIVE-18349 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran > Attachments: HIVE-18349.1.patch > > > 1) Hive metastore audit event log/metastore log does not log the final status > (success or failed) of the event. Some operations like for example, > drop_table returns a boolean success flag but it never gets logged anywhere. > However the same is sent to end event listeners or other metastore event > listeners. It will be good to log the final status of the events. > 2) Make connection timeout when using connection pool configurable. Currently > its hard coded to 30 seconds. > 3) Provide a config to enable connection leak detection for HikariCP or > enable when debug logging is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18349) Misc metastore changes for debuggability
[ https://issues.apache.org/jira/browse/HIVE-18349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18349: - Attachment: (was: HIVE-18349.1.patch) > Misc metastore changes for debuggability > > > Key: HIVE-18349 > URL: https://issues.apache.org/jira/browse/HIVE-18349 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran > Attachments: HIVE-18349.1.patch > > > 1) Hive metastore audit event log/metastore log does not log the final status > (success or failed) of the event. Some operations like for example, > drop_table returns a boolean success flag but it never gets logged anywhere. > However the same is sent to end event listeners or other metastore event > listeners. It will be good to log the final status of the events. > 2) Make connection timeout when using connection pool configurable. Currently > its hard coded to 30 seconds. > 3) Provide a config to enable connection leak detection for HikariCP or > enable when debug logging is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18349) Misc metastore changes for debuggability
[ https://issues.apache.org/jira/browse/HIVE-18349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18349: - Status: Patch Available (was: Open) > Misc metastore changes for debuggability > > > Key: HIVE-18349 > URL: https://issues.apache.org/jira/browse/HIVE-18349 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-18349.1.patch > > > 1) Hive metastore audit event log/metastore log does not log the final status > (success or failed) of the event. Some operations like for example, > drop_table returns a boolean success flag but it never gets logged anywhere. > However the same is sent to end event listeners or other metastore event > listeners. It will be good to log the final status of the events. > 2) Make connection timeout when using connection pool configurable. Currently > its hard coded to 30 seconds. > 3) Provide a config to enable connection leak detection for HikariCP or > enable when debug logging is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18349) Misc metastore changes for debuggability
[ https://issues.apache.org/jira/browse/HIVE-18349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-18349: Assignee: Prasanth Jayachandran > Misc metastore changes for debuggability > > > Key: HIVE-18349 > URL: https://issues.apache.org/jira/browse/HIVE-18349 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-18349.1.patch > > > 1) Hive metastore audit event log/metastore log does not log the final status > (success or failed) of the event. Some operations like for example, > drop_table returns a boolean success flag but it never gets logged anywhere. > However the same is sent to end event listeners or other metastore event > listeners. It will be good to log the final status of the events. > 2) Make connection timeout when using connection pool configurable. Currently > its hard coded to 30 seconds. > 3) Provide a config to enable connection leak detection for HikariCP or > enable when debug logging is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18365) netty-all jar is not present in the llap tarball
[ https://issues.apache.org/jira/browse/HIVE-18365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310223#comment-16310223 ] Miklos Gergely commented on HIVE-18365: --- Yes, as a result of this change netty-all-4.0.52.Final.jar was added to the llap tarball file in the lib folder. > netty-all jar is not present in the llap tarball > > > Key: HIVE-18365 > URL: https://issues.apache.org/jira/browse/HIVE-18365 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely > Fix For: 3.0.0 > > Attachments: HIVE-18365.patch > > > netty-all jar should be part of the llap tarball, as it requires it. Earlier > it was part of the tez libs, but it was removed from there, so it must be > added to it explicitly. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18361) Extend shared work optimizer to reuse computation beyond work boundaries
[
https://issues.apache.org/jira/browse/HIVE-18361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310229#comment-16310229
]
Hive QA commented on HIVE-18361:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
19s{color} | {color:red} common: The patch generated 2 new + 932 unchanged - 0
fixed = 934 total (was 932) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
31s{color} | {color:red} ql: The patch generated 23 new + 46 unchanged - 0
fixed = 69 total (was 46) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 006d69d |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8423/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8423/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8423/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Extend shared work optimizer to reuse computation beyond work boundaries
>
>
> Key: HIVE-18361
> URL: https://issues.apache.org/jira/browse/HIVE-18361
> Project: Hive
> Issue Type: New Feature
> Components: Physical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Labels: TODOC3.0
> Attachments: HIVE-18361.01.patch, HIVE-18361.patch
>
>
> Follow-up of the work in HIVE-16867.
> HIVE-16867 introduced an optimization that identifies scans on input tables
> that can be merged and reuses the computation that is done in the work
> containing those scans. In particular, we traverse both parts of the plan
> upstream and reuse the operators if possible.
> Currently, the optimizer will not go beyond the output edge(s) of that work.
> This extension removes that limitation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[
https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310230#comment-16310230
]
Sahil Takiar commented on HIVE-18238:
-
I'm not sure what support Hive provides for parallel query execution in a
single session. I don't think we have explicit tests for it either. However, I
know there have been issues with it - e.g. HIVE-18054
> Driver execution may not have configuration changing sideeffects
> -
>
> Key: HIVE-18238
> URL: https://issues.apache.org/jira/browse/HIVE-18238
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18238.01wip01.patch, HIVE-18238.02.patch,
> HIVE-18238.03.patch
>
>
> {{Driver}} executes sql statements which use "hiveconf" settings;
> but the {{Driver}} itself may *not* change the configuration...
> I've found an example; which shows how hazardous this is...
> {code}
> set hive.mapred.mode=strict;
> select "${hiveconf:hive.mapred.mode}";
> create table t (a int);
> analyze table t compute statistics;
> select "${hiveconf:hive.mapred.mode}";
> {code}
> currently; the last select returns {{nonstrict}} because of
> [this|https://github.com/apache/hive/blob/7ddd915bf82a68c8ab73b0c4ca409f1a6d43d227/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1696]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[
https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310230#comment-16310230
]
Sahil Takiar edited comment on HIVE-18238 at 1/3/18 8:20 PM:
-
I'm not sure what support Hive provides for parallel query execution in a
single session. I don't think we have explicit tests for it either. However, I
know there have been issues with it - e.g. HIVE-18054
HIVE-11402 is relevant too
was (Author: stakiar):
I'm not sure what support Hive provides for parallel query execution in a
single session. I don't think we have explicit tests for it either. However, I
know there have been issues with it - e.g. HIVE-18054
> Driver execution may not have configuration changing sideeffects
> -
>
> Key: HIVE-18238
> URL: https://issues.apache.org/jira/browse/HIVE-18238
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18238.01wip01.patch, HIVE-18238.02.patch,
> HIVE-18238.03.patch
>
>
> {{Driver}} executes sql statements which use "hiveconf" settings;
> but the {{Driver}} itself may *not* change the configuration...
> I've found an example; which shows how hazardous this is...
> {code}
> set hive.mapred.mode=strict;
> select "${hiveconf:hive.mapred.mode}";
> create table t (a int);
> analyze table t compute statistics;
> select "${hiveconf:hive.mapred.mode}";
> {code}
> currently; the last select returns {{nonstrict}} because of
> [this|https://github.com/apache/hive/blob/7ddd915bf82a68c8ab73b0c4ca409f1a6d43d227/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1696]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[
https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310235#comment-16310235
]
Eugene Koifman commented on HIVE-18238:
---
in that case it may be better to explicitly state that it does not - it
simplifies implementation and it it's not clear how useful it is to have this
"feature" even if it did work.
> Driver execution may not have configuration changing sideeffects
> -
>
> Key: HIVE-18238
> URL: https://issues.apache.org/jira/browse/HIVE-18238
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18238.01wip01.patch, HIVE-18238.02.patch,
> HIVE-18238.03.patch
>
>
> {{Driver}} executes sql statements which use "hiveconf" settings;
> but the {{Driver}} itself may *not* change the configuration...
> I've found an example; which shows how hazardous this is...
> {code}
> set hive.mapred.mode=strict;
> select "${hiveconf:hive.mapred.mode}";
> create table t (a int);
> analyze table t compute statistics;
> select "${hiveconf:hive.mapred.mode}";
> {code}
> currently; the last select returns {{nonstrict}} because of
> [this|https://github.com/apache/hive/blob/7ddd915bf82a68c8ab73b0c4ca409f1a6d43d227/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1696]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18335) Vectorization : Check bounds of array before the allocation in VectorMapJoinFastBytesHashTable
[ https://issues.apache.org/jira/browse/HIVE-18335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18335: -- Attachment: HIVE-18335.3.patch Updated the tests with try catch to catch the error. > Vectorization : Check bounds of array before the allocation in > VectorMapJoinFastBytesHashTable > -- > > Key: HIVE-18335 > URL: https://issues.apache.org/jira/browse/HIVE-18335 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-18335.1.patch, HIVE-18335.2.patch, > HIVE-18335.3.patch > > > VectorMapJoinFastBytesHashTable.allocateBucketArray() needs to check bounds > before allocating array. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18221) test acid default
[ https://issues.apache.org/jira/browse/HIVE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18221: -- Attachment: HIVE-18221.21.patch > test acid default > - > > Key: HIVE-18221 > URL: https://issues.apache.org/jira/browse/HIVE-18221 > Project: Hive > Issue Type: Test > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18221.01.patch, HIVE-18221.02.patch, > HIVE-18221.03.patch, HIVE-18221.04.patch, HIVE-18221.07.patch, > HIVE-18221.08.patch, HIVE-18221.09.patch, HIVE-18221.10.patch, > HIVE-18221.11.patch, HIVE-18221.12.patch, HIVE-18221.13.patch, > HIVE-18221.14.patch, HIVE-18221.16.patch, HIVE-18221.18.patch, > HIVE-18221.19.patch, HIVE-18221.20.patch, HIVE-18221.21.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18221) test acid default
[ https://issues.apache.org/jira/browse/HIVE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310257#comment-16310257 ] Eugene Koifman commented on HIVE-18221: --- patch 21 includes HIVE-18125 (temp WIP to reduce noise), ensures HCat tests pick up concurrency prop from hive-site.xml. > test acid default > - > > Key: HIVE-18221 > URL: https://issues.apache.org/jira/browse/HIVE-18221 > Project: Hive > Issue Type: Test > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18221.01.patch, HIVE-18221.02.patch, > HIVE-18221.03.patch, HIVE-18221.04.patch, HIVE-18221.07.patch, > HIVE-18221.08.patch, HIVE-18221.09.patch, HIVE-18221.10.patch, > HIVE-18221.11.patch, HIVE-18221.12.patch, HIVE-18221.13.patch, > HIVE-18221.14.patch, HIVE-18221.16.patch, HIVE-18221.18.patch, > HIVE-18221.19.patch, HIVE-18221.20.patch, HIVE-18221.21.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[
https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310266#comment-16310266
]
Sahil Takiar commented on HIVE-18238:
-
I believe there was some discussion on removing support for parallel query
execution in a single session in HIVE-11402
> Driver execution may not have configuration changing sideeffects
> -
>
> Key: HIVE-18238
> URL: https://issues.apache.org/jira/browse/HIVE-18238
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18238.01wip01.patch, HIVE-18238.02.patch,
> HIVE-18238.03.patch
>
>
> {{Driver}} executes sql statements which use "hiveconf" settings;
> but the {{Driver}} itself may *not* change the configuration...
> I've found an example; which shows how hazardous this is...
> {code}
> set hive.mapred.mode=strict;
> select "${hiveconf:hive.mapred.mode}";
> create table t (a int);
> analyze table t compute statistics;
> select "${hiveconf:hive.mapred.mode}";
> {code}
> currently; the last select returns {{nonstrict}} because of
> [this|https://github.com/apache/hive/blob/7ddd915bf82a68c8ab73b0c4ca409f1a6d43d227/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1696]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18096) add a user-friendly show plan command
[ https://issues.apache.org/jira/browse/HIVE-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310267#comment-16310267 ] Sergey Shelukhin commented on HIVE-18096: - Looks good so far... can you add displaying triggers, and trigger list in the pools? Also, this doesn't appear to be using sysdb. Have you tried using sysdb? Just wondering if you may have some unfinished patch. If not, we can continue with this for now. > add a user-friendly show plan command > - > > Key: HIVE-18096 > URL: https://issues.apache.org/jira/browse/HIVE-18096 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Harish Jaiprakash > Attachments: HIVE-18096.01.patch > > > For admin to be able to get an overview of a resource plan. > We need to try to do this using sysdb. > If that is not possible to do in a nice way, we'd do a text-based one like > query explain, or desc extended table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18335) Vectorization : Check bounds of array before the allocation in VectorMapJoinFastBytesHashTable
[ https://issues.apache.org/jira/browse/HIVE-18335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310281#comment-16310281 ] Matt McCline commented on HIVE-18335: - I added one comment to the review board. Otherwise, looks good to me. +1 > Vectorization : Check bounds of array before the allocation in > VectorMapJoinFastBytesHashTable > -- > > Key: HIVE-18335 > URL: https://issues.apache.org/jira/browse/HIVE-18335 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-18335.1.patch, HIVE-18335.2.patch, > HIVE-18335.3.patch > > > VectorMapJoinFastBytesHashTable.allocateBucketArray() needs to check bounds > before allocating array. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[
https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310285#comment-16310285
]
Zoltan Haindrich commented on HIVE-18238:
-
[~ekoifman] ...in the latest patch I'm doing a very similar thing - by creating
a new conf in {{QueryState}} ; but the broken test number have raised again so
I might probably step back to only prevent hiveconf changes for intra project
parts (service/cli/etc) - just like in the earlier patches.
In the near future I would like to add some re-execution logic which in this
way haven't has to tweak everything in the driver - instead; I will try to
clean some parts of it.
I will keep in mind that earlier Driver had a separate config; but
unfortunately currently there are some pieces which can't live without modify
session level configs...
I've collected the configs which have been "restored" to do session level
changes in the latest patch:
| {{hive.query.string}} | the user can take a look at the last query...I
haven't seen any other uses so far.. thank you [~stakiar] for the info; it's
very intresting, IIRC there is some rest api which give much more detailed
information about the executed queries... I feel that using that service would
be better for the user |
| {{hive.query.id}} | the druid storage driver uses {{Session.get().getConf()}}
to acquire the current query id; to calculate a scratch-dir name - in case of
parallel execution this might be cause severe problems - I've tried to send the
correct conf; but I ended up changing too many things... I think that the
Storage driver is more closer to be a session level thing; than to be a query
level construct; so I feel that probably the Druid storage handler should be
modified to use a random id to get a scratch dir... |
| {{hive.doing.acid}} | I think this is some leftover flag...I feel that once
set; it's never been unset; and it also needs session level - I think this can
be removed |
> Driver execution may not have configuration changing sideeffects
> -
>
> Key: HIVE-18238
> URL: https://issues.apache.org/jira/browse/HIVE-18238
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18238.01wip01.patch, HIVE-18238.02.patch,
> HIVE-18238.03.patch
>
>
> {{Driver}} executes sql statements which use "hiveconf" settings;
> but the {{Driver}} itself may *not* change the configuration...
> I've found an example; which shows how hazardous this is...
> {code}
> set hive.mapred.mode=strict;
> select "${hiveconf:hive.mapred.mode}";
> create table t (a int);
> analyze table t compute statistics;
> select "${hiveconf:hive.mapred.mode}";
> {code}
> currently; the last select returns {{nonstrict}} because of
> [this|https://github.com/apache/hive/blob/7ddd915bf82a68c8ab73b0c4ca409f1a6d43d227/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1696]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18361) Extend shared work optimizer to reuse computation beyond work boundaries
[
https://issues.apache.org/jira/browse/HIVE-18361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310287#comment-16310287
]
Hive QA commented on HIVE-18361:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904427/HIVE-18361.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 40 failed/errored test(s), 11543 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_multi]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_notin]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_select]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query33]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query44]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query47]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query54]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query56]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query57]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query58]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query60]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query61]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query64]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query70]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query83]
(batchId=245)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query90]
(batchId=245)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout
(batchId=235)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8423/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8423/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8423/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 40 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12904427 - PreCommit-HIVE-Build
> Extend shared work optimizer to reuse computation beyond work boundaries
> --
[jira] [Updated] (HIVE-18335) Vectorization : Check bounds of array before the allocation in VectorMapJoinFastBytesHashTable
[ https://issues.apache.org/jira/browse/HIVE-18335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18335: -- Attachment: HIVE-18335.4.patch > Vectorization : Check bounds of array before the allocation in > VectorMapJoinFastBytesHashTable > -- > > Key: HIVE-18335 > URL: https://issues.apache.org/jira/browse/HIVE-18335 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-18335.1.patch, HIVE-18335.2.patch, > HIVE-18335.3.patch, HIVE-18335.4.patch > > > VectorMapJoinFastBytesHashTable.allocateBucketArray() needs to check bounds > before allocating array. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18335) Vectorization : Check bounds of array before the allocation in VectorMapJoinFastBytesHashTable
[ https://issues.apache.org/jira/browse/HIVE-18335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310289#comment-16310289 ] Deepak Jaiswal commented on HIVE-18335: --- Implemented the suggested comment. > Vectorization : Check bounds of array before the allocation in > VectorMapJoinFastBytesHashTable > -- > > Key: HIVE-18335 > URL: https://issues.apache.org/jira/browse/HIVE-18335 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-18335.1.patch, HIVE-18335.2.patch, > HIVE-18335.3.patch, HIVE-18335.4.patch > > > VectorMapJoinFastBytesHashTable.allocateBucketArray() needs to check bounds > before allocating array. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit
[
https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310290#comment-16310290
]
Hive QA commented on HIVE-16484:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904293/HIVE-16484.8.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8424/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8424/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8424/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-01-03 21:09:26.686
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-8424/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-01-03 21:09:26.690
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
d015656..006d69d master -> origin/master
+ git reset --hard HEAD
HEAD is now at d015656 HIVE-17929: Use sessionId for HoS Remote Driver Client
id (Sahil Takiar, reviewed by Rui Li)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 006d69d HIVE-18360 : NPE in TezSessionState (Sergey Shelukhin,
reviewed by Jason Dere)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-01-03 21:09:30.024
+ rm -rf ../yetus
+ mkdir ../yetus
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-8424/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java:
does not exist in index
error:
a/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientFactory.java:
does not exist in index
error:
a/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java:
does not exist in index
error:
a/spark-client/src/test/java/org/apache/hive/spark/client/TestSparkClient.java:
does not exist in index
error: patch failed:
spark-client/src/main/java/org/apache/hive/spark/client/SparkClientFactory.java:84
Falling back to three-way merge...
Applied patch to
'spark-client/src/main/java/org/apache/hive/spark/client/SparkClientFactory.java'
with conflicts.
error: patch failed:
spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java:82
Falling back to three-way merge...
Applied patch to
'spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java'
with conflicts.
error: patch failed:
spark-client/src/test/java/org/apache/hive/spark/client/TestSparkClient.java:307
Falling back to three-way merge...
Applied patch to
'spark-client/src/test/java/org/apache/hive/spark/client/TestSparkClient.java'
with conflicts.
Going to apply patch with: git apply -p1
error: patch failed:
spark-client/src/main/java/org/apache/hive/spark/client/SparkClientFactory.java:84
Falling back to three-way merge...
Applied patch to
'spark-client/src/main/java/org/apache/hive/spark/client/SparkClientFactory.java'
with conflicts.
error: patch failed:
spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java:82
Falling back to three-way merge...
Applied patch to
'spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java'
with conflicts.
error: patch failed:
spark-client/src/test/java/org/apache/hive/spark/client/TestSparkClient.java:307
Falling back to three-way merge...
Applied patch to
'spark-client/src/test/java/org
[jira] [Commented] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[
https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310297#comment-16310297
]
Eugene Koifman commented on HIVE-18238:
---
I think AcidUtils.CONF_ACID_KEY (hive.doing.acid) is used to disable
CombineInputFormat for queries reading Acid tables.
It should be per query rather than per session.
It was most likely introduced back when Driver copied Configuration so the
"unset" happened automatically.
If it gets set permanently I don't think that would show up in UTs unless they
explicitly check for use of CombineInputFormat
> Driver execution may not have configuration changing sideeffects
> -
>
> Key: HIVE-18238
> URL: https://issues.apache.org/jira/browse/HIVE-18238
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18238.01wip01.patch, HIVE-18238.02.patch,
> HIVE-18238.03.patch
>
>
> {{Driver}} executes sql statements which use "hiveconf" settings;
> but the {{Driver}} itself may *not* change the configuration...
> I've found an example; which shows how hazardous this is...
> {code}
> set hive.mapred.mode=strict;
> select "${hiveconf:hive.mapred.mode}";
> create table t (a int);
> analyze table t compute statistics;
> select "${hiveconf:hive.mapred.mode}";
> {code}
> currently; the last select returns {{nonstrict}} because of
> [this|https://github.com/apache/hive/blob/7ddd915bf82a68c8ab73b0c4ca409f1a6d43d227/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1696]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18365) netty-all jar is not present in the llap tarball
[
https://issues.apache.org/jira/browse/HIVE-18365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310304#comment-16310304
]
Hive QA commented on HIVE-18365:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 006d69d |
| Default Java | 1.8.0_111 |
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8425/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> netty-all jar is not present in the llap tarball
>
>
> Key: HIVE-18365
> URL: https://issues.apache.org/jira/browse/HIVE-18365
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 3.0.0
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
> Fix For: 3.0.0
>
> Attachments: HIVE-18365.patch
>
>
> netty-all jar should be part of the llap tarball, as it requires it. Earlier
> it was part of the tez libs, but it was removed from there, so it must be
> added to it explicitly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-15102) Hiveptest is killing nodes where IP is reused after previous node termination
[ https://issues.apache.org/jira/browse/HIVE-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310311#comment-16310311 ] Sergio Peña commented on HIVE-15102: [~szita] I see you fixed this issue on HIVE-18263, can I close this issue as duplicate or this issue still exists? > Hiveptest is killing nodes where IP is reused after previous node termination > - > > Key: HIVE-15102 > URL: https://issues.apache.org/jira/browse/HIVE-15102 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.2.0 >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-15102.1.patch > > > NO PRECOMMIT TESTS > The Hiveptest framework has a background thread that runs every hour, and > attempts to kill zombie nodes that are not being used by the test execution > anymore. > These killed nodes are kept in a list of terminated nodes, and next time the > background thread is executed, it will attempt to kill all those nodes again > because Hiveptest consider them as zombie nodes. > The problem is that cloud providers can give you the same IP numbers for new > nodes, and when the background thread runs, it will kill those nodes that may > still be in used by Hiveptest. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit
[
https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-16484:
Attachment: HIVE-16484.9.patch
Re-based patch.
> Investigate SparkLauncher for HoS as alternative to bin/spark-submit
>
>
> Key: HIVE-16484
> URL: https://issues.apache.org/jira/browse/HIVE-16484
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16484.1.patch, HIVE-16484.2.patch,
> HIVE-16484.3.patch, HIVE-16484.4.patch, HIVE-16484.5.patch,
> HIVE-16484.6.patch, HIVE-16484.7.patch, HIVE-16484.8.patch, HIVE-16484.9.patch
>
>
> The {{SparkClientImpl#startDriver}} currently looks for the {{SPARK_HOME}}
> directory and invokes the {{bin/spark-submit}} script, which spawns a
> separate process to run the Spark application.
> {{SparkLauncher}} was added in SPARK-4924 and is a programatic way to launch
> Spark applications.
> I see a few advantages:
> * No need to spawn a separate process to launch a HoS --> lower startup time
> * Simplifies the code in {{SparkClientImpl}} --> easier to debug
> * {{SparkLauncher#startApplication}} returns a {{SparkAppHandle}} which
> contains some useful utilities for querying the state of the Spark job
> ** It also allows the launcher to specify a list of job listeners
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-15077) Acid LockManager is unfair
[
https://issues.apache.org/jira/browse/HIVE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15077:
--
Priority: Blocker (was: Major)
> Acid LockManager is unfair
> --
>
> Key: HIVE-15077
> URL: https://issues.apache.org/jira/browse/HIVE-15077
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
>
> HIVE-10242 made the acid LM unfair.
> In TxnHandler.checkLock(), suppose we are trying to acquire SR5 (the number
> is extLockId).
> Then
> LockInfo[] locks = lockSet.toArray(new LockInfo[lockSet.size()]);
> may look like this (all explicitly listed locks are in Waiting state)
> {, SR5 SW3 X4}
> So the algorithm will find SR5 in the list and start looking backwards.
> According to IDs, SR5 should wait for X4 to be granted but X4 won't even be
> examined and so SR5 may be granted.
> Theoretically, this could cause starvation.
> The query that generates the list already has
> query.append(" and hl_lock_ext_id <= ").append(extLockId);
> but it should use "<" rather than "<=" to exclude the locks being checked
> from "locks" list which will make the algorithm look at all locks "in front"
> of a given lock.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[
https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310330#comment-16310330
]
Zoltan Haindrich commented on HIVE-18238:
-
thank you [~ekoifman], I will take a closer look on it tomorrow - and try to
make it work at a query only level
> Driver execution may not have configuration changing sideeffects
> -
>
> Key: HIVE-18238
> URL: https://issues.apache.org/jira/browse/HIVE-18238
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18238.01wip01.patch, HIVE-18238.02.patch,
> HIVE-18238.03.patch
>
>
> {{Driver}} executes sql statements which use "hiveconf" settings;
> but the {{Driver}} itself may *not* change the configuration...
> I've found an example; which shows how hazardous this is...
> {code}
> set hive.mapred.mode=strict;
> select "${hiveconf:hive.mapred.mode}";
> create table t (a int);
> analyze table t compute statistics;
> select "${hiveconf:hive.mapred.mode}";
> {code}
> currently; the last select returns {{nonstrict}} because of
> [this|https://github.com/apache/hive/blob/7ddd915bf82a68c8ab73b0c4ca409f1a6d43d227/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1696]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
