[jira] [Commented] (HIVE-9323) Merge from trunk to spark 1/8/2015
[
https://issues.apache.org/jira/browse/HIVE-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270719#comment-14270719
]
Chengxiang Li commented on HIVE-9323:
-
[~Szehon], I take a look at the hive log, the failed reason is quite strange
and a little different from HIVE-9094. HIVE-9094 failed due to get executor
count timeout because of spark cluster launch time is longer than spark client
future timeout interval(5s, and 30s after HIVE-9094), while this timeout
failure is due to RemoteDriver is not response in time(spark client wait 10s
for RemoteDriver to register).
>From the hive.log, RemoteDriver processor is launched at 2015-01-08
>18:43:03,938
{noformat}
2015-01-08 18:43:03,938 DEBUG [main]: client.SparkClientImpl
(SparkClientImpl.java:startDriver(298)) - Running client driver with argv:
/home/hiveptest/54.177.142.77-hiveptest-1/apache-svn-spark-source/itests/qtest-spark/../../itests/qtest-spark/target/spark/bin/spark-submit
--properties-file
/home/hiveptest/54.177.142.77-hiveptest-1/apache-svn-spark-source/itests/qtest-spark/target/tmp/spark-submit.1097041260552550316.properties
--class org.apache.hive.spark.client.RemoteDriver
/home/hiveptest/54.177.142.77-hiveptest-1/maven/org/apache/hive/hive-exec/0.15.0-SNAPSHOT/hive-exec-0.15.0-SNAPSHOT.jar
--remote-host ip-10-228-130-250.us-west-1.compute.internal --remote-port 40406
{noformat}
In spark.log, RemoteDriver register back to SparkClient at 2015-01-08
18:43:13,891 which should just more than timeout interval which is 10s.
{noformat}
2015-01-08 18:43:13,891 DEBUG [Driver-RPC-Handler-0]: rpc.RpcDispatcher
(RpcDispatcher.java:registerRpc(185)) - [DriverProtocol] Registered outstanding
rpc 0 (org.apache.hive.spark.client.rpc.Rpc$Hello).
{noformat}
The strange thing is that RemoteDriver processor is unusual slow, as it's
launched at 2015-01-08 18:43:03,938 but we get it's first debug info at
2015-01-08 18:43:13,161, RemoteDriver hardly do anything before this debug info.
{noformat}
2015-01-08 18:43:13,161 INFO [main]: client.RemoteDriver
(RemoteDriver.java:(118)) - Connecting to:
ip-10-228-130-250.us-west-1.compute.internal:40406
{noformat}
I not sure why this happens, but this should be a quite rarely case, we can
check whether it happens again, besides expand timeout interval, i don't have a
good solution for this issue now.
> Merge from trunk to spark 1/8/2015
> --
>
> Key: HIVE-9323
> URL: https://issues.apache.org/jira/browse/HIVE-9323
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Affects Versions: spark-branch
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Fix For: spark-branch
>
> Attachments: HIVE-9323-spark.patch, HIVE-9323.2-spark.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9323) Merge from trunk to spark 1/8/2015
[ https://issues.apache.org/jira/browse/HIVE-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-9323: Resolution: Fixed Fix Version/s: spark-branch Status: Resolved (was: Patch Available) Committed to spark branch. > Merge from trunk to spark 1/8/2015 > -- > > Key: HIVE-9323 > URL: https://issues.apache.org/jira/browse/HIVE-9323 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Szehon Ho >Assignee: Szehon Ho > Fix For: spark-branch > > Attachments: HIVE-9323-spark.patch, HIVE-9323.2-spark.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9323) Merge from trunk to spark 1/8/2015
[ https://issues.apache.org/jira/browse/HIVE-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270701#comment-14270701 ] Szehon Ho commented on HIVE-9323: - Remaining new failures are caused by merge of HIVE-9290 and should be fixed there. > Merge from trunk to spark 1/8/2015 > -- > > Key: HIVE-9323 > URL: https://issues.apache.org/jira/browse/HIVE-9323 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-9323-spark.patch, HIVE-9323.2-spark.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9323) Merge from trunk to spark 1/8/2015
[
https://issues.apache.org/jira/browse/HIVE-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270695#comment-14270695
]
Hive QA commented on HIVE-9323:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12691051/HIVE-9323.2-spark.patch
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 7301 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_memcheck
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_limit_pushdown
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_gby_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_string_concat
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_windowing
org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/623/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/623/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-623/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12691051 - PreCommit-HIVE-SPARK-Build
> Merge from trunk to spark 1/8/2015
> --
>
> Key: HIVE-9323
> URL: https://issues.apache.org/jira/browse/HIVE-9323
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Affects Versions: spark-branch
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-9323-spark.patch, HIVE-9323.2-spark.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8931) Test TestAccumuloCliDriver is not completing
[
https://issues.apache.org/jira/browse/HIVE-8931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270685#comment-14270685
]
Josh Elser commented on HIVE-8931:
--
Getting back to this, I'm a little stuck here. Backing up, {{hive-metastore}}
is bringing in libthrift-0.9.2 which is breaking things. The qtests ultimately
pull from $CLASSPATH to star the Accumulo minicluster (which includes stuff
from HIVE_HADOOP_TEST_CLASSPATH), that ultimately comes back to the maven test
classpath. Without getting libthrift-0.9.1 somehow on the maven classpath, I
don't know where the libthirft-0.9.1.jar even exists on the local m2 repository
(and thus can't do any trickery to substitute it in place of the
libthrift-0.9.2 dependency). My assumption is that excluding libthrift from the
hive-metastore dependency will break the other qtests (but that is only a
guess).
Assuming I can't exclude libthrift from hive-metastore, I'm not sure what I
could even do at this point aside from introducing a new maven module
specifically for the Accumulo qtests (and gives me carte blanche over the
classpath). [~brocknoland], any ideas?
> Test TestAccumuloCliDriver is not completing
>
>
> Key: HIVE-8931
> URL: https://issues.apache.org/jira/browse/HIVE-8931
> Project: Hive
> Issue Type: Bug
>Reporter: Brock Noland
>Assignee: Josh Elser
>
> Tests are taking 3 hours due to {{TestAccumuloCliDriver}} not finishing.
> Logs:
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1848/failed/TestAccumuloCliDriver/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-9325) Handle the case of insert overwrite statement with a qualified path that the destination path does not have a schema.
[ https://issues.apache.org/jira/browse/HIVE-9325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland resolved HIVE-9325. Resolution: Fixed Fix Version/s: encryption-branch Thank you Ferd! I have committed this to branch! > Handle the case of insert overwrite statement with a qualified path that the > destination path does not have a schema. > - > > Key: HIVE-9325 > URL: https://issues.apache.org/jira/browse/HIVE-9325 > Project: Hive > Issue Type: Sub-task >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Fix For: encryption-branch > > Attachments: HIVE-9325-encryption.patch > > > The insert overwrite statement can pass in a path that does not have a schema > specified. This will make the isSub method out of work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9325) Handle the case of insert overwrite statement with a qualified path that the destination path does not have a schema.
[ https://issues.apache.org/jira/browse/HIVE-9325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270674#comment-14270674 ] Brock Noland commented on HIVE-9325: +1 > Handle the case of insert overwrite statement with a qualified path that the > destination path does not have a schema. > - > > Key: HIVE-9325 > URL: https://issues.apache.org/jira/browse/HIVE-9325 > Project: Hive > Issue Type: Sub-task >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-9325-encryption.patch > > > The insert overwrite statement can pass in a path that does not have a schema > specified. This will make the isSub method out of work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9325) Handle the case of insert overwrite statement with a qualified path that the destination path does not have a schema.
[ https://issues.apache.org/jira/browse/HIVE-9325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-9325: --- Attachment: HIVE-9325-encryption.patch > Handle the case of insert overwrite statement with a qualified path that the > destination path does not have a schema. > - > > Key: HIVE-9325 > URL: https://issues.apache.org/jira/browse/HIVE-9325 > Project: Hive > Issue Type: Sub-task >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-9325-encryption.patch > > > The insert overwrite statement can pass in a path that does not have a schema > specified. This will make the isSub method out of work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9325) Handle the case of insert overwrite statement with a qualified path that the destination path does not have a schema.
Ferdinand Xu created HIVE-9325: -- Summary: Handle the case of insert overwrite statement with a qualified path that the destination path does not have a schema. Key: HIVE-9325 URL: https://issues.apache.org/jira/browse/HIVE-9325 Project: Hive Issue Type: Sub-task Reporter: Ferdinand Xu Assignee: Ferdinand Xu The insert overwrite statement can pass in a path that does not have a schema specified. This will make the isSub method out of work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9249) java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to org.apache.hadoop.hive.common.type.HiveVarchar when joining tables
[
https://issues.apache.org/jira/browse/HIVE-9249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-9249:
--
Attachment: HIVE-9249.hive-14.patch
[~mmccline]: I have regenerated the golden files for the branch. Can you verify
the mapjoin.out files with a look-see?
> java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
> org.apache.hadoop.hive.common.type.HiveVarchar when joining tables
> ---
>
> Key: HIVE-9249
> URL: https://issues.apache.org/jira/browse/HIVE-9249
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 0.14.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 0.15.0
>
> Attachments: HIVE-9249.01.patch, HIVE-9249.hive-14.patch
>
>
> VectorColumnAssignFactory doesn't handle HiveCharWritable /
> HiveVarcharWritable objects.
> Either:
> HiveVarcharWritable cannot be cast to ... HiveVarchar
> or
> HiveCharWritable cannot be cast to ... HiveChar
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
> org.apache.hadoop.hive.common.type.HiveVarchar
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorColumnAssignFactory$17.assignObjectValue(VectorColumnAssignFactory.java:417)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.internalForward(VectorMapJoinOperator.java:196)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:670)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:748)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:299)
> ... 24 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-3405) UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase
[ https://issues.apache.org/jira/browse/HIVE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270644#comment-14270644 ] Jason Dere commented on HIVE-3405: -- +1 if precommit tests still look good > UDF initcap to obtain a string with the first letter of each word in > uppercase other letters in lowercase > - > > Key: HIVE-3405 > URL: https://issues.apache.org/jira/browse/HIVE-3405 > Project: Hive > Issue Type: New Feature > Components: UDF >Affects Versions: 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.11.0, 0.13.0, 0.14.0, > 0.15.0, 0.14.1 >Reporter: Archana Nair >Assignee: Alexander Pivovarov > Labels: patch > Attachments: HIVE-3405.1.patch.txt, HIVE-3405.2.patch, > HIVE-3405.3.patch, HIVE-3405.4.patch, HIVE-3405.5.patch > > > Hive current releases lacks a INITCAP function which returns String with > first letter of the word in uppercase.INITCAP returns String, with the first > letter of each word in uppercase, all other letters in same case. Words are > delimited by white space.This will be useful report generation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9094) TimeoutException when trying get executor count from RSC [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270604#comment-14270604
]
Szehon Ho commented on HIVE-9094:
-
Just as FYI this happened again in latest ptest run :
[https://issues.apache.org/jira/browse/HIVE-9323?focusedCommentId=14270602&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14270602|https://issues.apache.org/jira/browse/HIVE-9323?focusedCommentId=14270602&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14270602]
> TimeoutException when trying get executor count from RSC [Spark Branch]
> ---
>
> Key: HIVE-9094
> URL: https://issues.apache.org/jira/browse/HIVE-9094
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Xuefu Zhang
>Assignee: Chengxiang Li
> Labels: TODOC-SPARK
> Fix For: spark-branch
>
> Attachments: HIVE-9094.1-spark.patch, HIVE-9094.2-spark.patch
>
>
> In
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/532/testReport,
> join25.q failed because:
> {code}
> 2014-12-12 19:14:50,084 ERROR [main]: ql.Driver
> (SessionState.java:printError(838)) - FAILED: SemanticException Failed to get
> spark memory/core info: java.util.concurrent.TimeoutException
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to get spark
> memory/core info: java.util.concurrent.TimeoutException
> at
> org.apache.hadoop.hive.ql.optimizer.spark.SetSparkReducerParallelism.process(SetSparkReducerParallelism.java:120)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
> at
> org.apache.hadoop.hive.ql.lib.ForwardWalker.walk(ForwardWalker.java:79)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
> at
> org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.optimizeOperatorPlan(SparkCompiler.java:134)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:99)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10202)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:420)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:306)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1108)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1045)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1035)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:199)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:151)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:362)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:297)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:837)
> at
> org.apache.hadoop.hive.cli.TestSparkCliDriver.runTest(TestSparkCliDriver.java:234)
> at
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join25(TestSparkCliDriver.java:162)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at junit.framework.TestCase.runBare(TestCase.java:141)
> at junit.framework.TestResult$1.protect(TestResult.java:122)
> at junit.framework.TestResult.runProtected(TestResult.java:142)
> at junit.framework.TestResult.run(TestResult.java:125)
> at junit.framework.TestCase.run(TestCase.java:129)
> at junit.framework.TestSuite.runTest(TestSuite.java:255)
> at junit.framework.TestSuite.run(TestSuite.java:250)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
>
[jira] [Updated] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9251: - Attachment: HIVE-9251.4-spark.patch Update more golden files. > SetSparkReducerParallelism is likely to set too small number of reducers > [Spark Branch] > --- > > Key: HIVE-9251 > URL: https://issues.apache.org/jira/browse/HIVE-9251 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-9251.1-spark.patch, HIVE-9251.2-spark.patch, > HIVE-9251.3-spark.patch, HIVE-9251.4-spark.patch > > > This may hurt performance or even lead to task failures. For example, spark's > netty-based shuffle limits the max frame size to be 2G. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9323) Merge from trunk to spark 1/8/2015
[
https://issues.apache.org/jira/browse/HIVE-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270602#comment-14270602
]
Szehon Ho commented on HIVE-9323:
-
The ctas failure is due to HIVE-9094:
{noformat}
2015-01-08 18:43:14,025 WARN [main]: client.SparkClientImpl
(SparkClientImpl.java:(88)) - Error while waiting for client to connect.
java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException:
Timed out waiting for client connection.
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
at
org.apache.hive.spark.client.SparkClientImpl.(SparkClientImpl.java:86)
at
org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:75)
at
org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.(RemoteHiveSparkClient.java:82)
at
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:53)
at
org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:56)
at
org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:128)
at
org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:84)
at
org.apache.hadoop.hive.ql.optimizer.spark.SetSparkReducerParallelism.process(SetSparkReducerParallelism.java:116)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
at
org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.optimizeOperatorPlan(SparkCompiler.java:133)
at
org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:99)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10053)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:189)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:224)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:224)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:420)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:306)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1108)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1156)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1045)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1035)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:206)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:158)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:369)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:304)
at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:880)
at
org.apache.hadoop.hive.cli.TestSparkCliDriver.runTest(TestSparkCliDriver.java:234)
at
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ctas(TestSparkCliDriver.java:166)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.ma
[jira] [Updated] (HIVE-9323) Merge from trunk to spark 1/8/2015
[ https://issues.apache.org/jira/browse/HIVE-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-9323: Attachment: HIVE-9323.2-spark.patch Regenerate the golden files for auto_join7, join7, and groupby_cube1 due to various changes in the trunk. > Merge from trunk to spark 1/8/2015 > -- > > Key: HIVE-9323 > URL: https://issues.apache.org/jira/browse/HIVE-9323 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-9323-spark.patch, HIVE-9323.2-spark.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-9324) Reduce side joins failing with IOException from RowContainer.nextBlock
[
https://issues.apache.org/jira/browse/HIVE-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270588#comment-14270588
]
Amareshwari Sriramadasu edited comment on HIVE-9324 at 1/9/15 5:54 AM:
---
More task log :
{noformat}
2014-12-14 16:58:03,905 INFO org.apache.hadoop.hive.ql.exec.mr.ObjectCache:
Ignoring retrieval request: __REDUCE_PLAN__
2014-12-14 16:58:03,945 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
2014-12-14 16:58:03,945 INFO org.apache.hadoop.hive.ql.exec.Utilities:
Deserializing ReduceWork via kryo
2014-12-14 16:58:04,987 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
2014-12-14 16:58:04,988 INFO org.apache.hadoop.hive.ql.exec.mr.ObjectCache:
Ignoring cache key: __REDUCE_PLAN__
2014-12-14 16:58:05,327 INFO ExecReducer:
Id =0
Id =1
<\Children>
Id = 0 null<\Parent>
<\FS>
<\Children>
<\Parent>
<\JOIN>
2014-12-14 16:58:05,327 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Initializing Self 0 JOIN
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
JOIN
struct<_col23:string,_col65:double,_col99:double,_col237:double,_col240:double,_col250:string,_col367:int>
totalsz = 7
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Operator 0 JOIN initialized
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Initializing children of 0 JOIN
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initializing child 1 FS
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initializing Self 1 FS
2014-12-14 16:58:05,394 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Operator 1 FS initialized
2014-12-14 16:58:05,394 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initialization Done 1 FS
2014-12-14 16:58:05,395 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Initialization Done 0 JOIN
2014-12-14 16:58:05,401 INFO ExecReducer: ExecReducer: processing 1 rows: used
memory = 242598168
2014-12-14 16:58:05,406 INFO ExecReducer: ExecReducer: processing 10 rows: used
memory = 242759392
2014-12-14 16:58:05,437 INFO ExecReducer: ExecReducer: processing 100 rows:
used memory = 242759392
2014-12-14 16:58:05,657 INFO ExecReducer: ExecReducer: processing 1000 rows:
used memory = 243653240
2014-12-14 16:58:06,976 INFO ExecReducer: ExecReducer: processing 1 rows:
used memory = 247197944
2014-12-14 16:58:07,646 INFO ExecReducer: ExecReducer: processing 10 rows:
used memory = 277801256
2014-12-14 16:58:11,511 INFO ExecReducer: ExecReducer: processing 100 rows:
used memory = 283150744
2014-12-14 16:58:14,993 INFO ExecReducer: ExecReducer: processing 200 rows:
used memory = 293036992
2014-12-14 16:58:18,497 INFO ExecReducer: ExecReducer: processing 300 rows:
used memory = 311449488
2014-12-14 16:58:20,815 INFO ExecReducer: ExecReducer: processing 400 rows:
used memory = 285251752
2014-12-14 16:58:26,460 INFO ExecReducer: ExecReducer: processing 500 rows:
used memory = 328223864
2014-12-14 16:58:29,412 INFO ExecReducer: ExecReducer: processing 600 rows:
used memory = 263175576
2014-12-14 16:58:31,331 INFO ExecReducer: ExecReducer: processing 700 rows:
used memory = 282021320
2014-12-14 16:58:35,099 INFO ExecReducer: ExecReducer: processing 800 rows:
used memory = 299301184
2014-12-14 16:58:37,981 INFO ExecReducer: ExecReducer: processing 900 rows:
used memory = 306925648
2014-12-14 16:58:40,506 INFO ExecReducer: ExecReducer: processing 1000
rows: used memory = 307407920
2014-12-14 16:58:42,242 INFO ExecReducer: ExecReducer: processing 1100
rows: used memory = 304664048
2014-12-14 16:58:46,142 INFO ExecReducer: ExecReducer: processing 1200
rows: used memory = 298347024
2014-12-14 16:58:48,549 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
table 0 has 1000 rows for join key [003b9de7876541c2bcce9029ff0d3873]
2014-12-14 16:58:48,622 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
table 0 has 2000 rows for join key [003b9de7876541c2bcce9029ff0d3873]
2014-12-14 16:58:48,677 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
table 0 has 4000 rows for join key [003b9de7876541c2bcce9029ff0d3873]
2014-12-14 16:58:48,679 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Final Path: FS
hdfs://test-machine:8020/tmp/hive-dataqa/hive_2014-12-14_16-49-14_996_1630664550753106415-32/_tmp.-mr-10002/00_0
2014-12-14 16:58:48,680 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Writing to temp file: FS
hdfs://test-machine:8020/tmp/hive-dataqa/hive_2014-12-14_16-49-14_996_1630664550753106415-32/_task_tmp.-mr-10002/_tmp.00_0
2014-12-14 16:58:48,680 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
New Final Path: FS
hdfs://test-machine:8020/tmp/hive-dataqa/hive_2014-12-14_16-49-14_996_1630664550753106415-32/_tmp.-mr-10002/0
[jira] [Commented] (HIVE-9324) Reduce side joins failing with IOException from RowContainer.nextBlock
[
https://issues.apache.org/jira/browse/HIVE-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270588#comment-14270588
]
Amareshwari Sriramadasu commented on HIVE-9324:
---
More task log :
{noformat}
2014-12-14 16:58:03,905 INFO org.apache.hadoop.hive.ql.exec.mr.ObjectCache:
Ignoring retrieval request: __REDUCE_PLAN__
2014-12-14 16:58:03,945 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
2014-12-14 16:58:03,945 INFO org.apache.hadoop.hive.ql.exec.Utilities:
Deserializing ReduceWork via kryo
2014-12-14 16:58:04,987 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
2014-12-14 16:58:04,988 INFO org.apache.hadoop.hive.ql.exec.mr.ObjectCache:
Ignoring cache key: __REDUCE_PLAN__
2014-12-14 16:58:05,327 INFO ExecReducer:
Id =0
Id =1
<\Children>
Id = 0 null<\Parent>
<\FS>
<\Children>
<\Parent>
<\JOIN>
2014-12-14 16:58:05,327 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Initializing Self 0 JOIN
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
JOIN
struct<_col23:string,_col65:double,_col99:double,_col237:double,_col240:double,_col250:string,_col367:int>
totalsz = 7
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Operator 0 JOIN initialized
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Initializing children of 0 JOIN
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initializing child 1 FS
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initializing Self 1 FS
2014-12-14 16:58:05,394 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Operator 1 FS initialized
2014-12-14 16:58:05,394 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initialization Done 1 FS
2014-12-14 16:58:05,395 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Initialization Done 0 JOIN
2014-12-14 16:58:05,401 INFO ExecReducer: ExecReducer: processing 1 rows: used
memory = 242598168
2014-12-14 16:58:05,406 INFO ExecReducer: ExecReducer: processing 10 rows: used
memory = 242759392
2014-12-14 16:58:05,437 INFO ExecReducer: ExecReducer: processing 100 rows:
used memory = 242759392
2014-12-14 16:58:05,657 INFO ExecReducer: ExecReducer: processing 1000 rows:
used memory = 243653240
2014-12-14 16:58:06,976 INFO ExecReducer: ExecReducer: processing 1 rows:
used memory = 247197944
2014-12-14 16:58:07,646 INFO ExecReducer: ExecReducer: processing 10 rows:
used memory = 277801256
2014-12-14 16:58:11,511 INFO ExecReducer: ExecReducer: processing 100 rows:
used memory = 283150744
2014-12-14 16:58:14,993 INFO ExecReducer: ExecReducer: processing 200 rows:
used memory = 293036992
2014-12-14 16:58:18,497 INFO ExecReducer: ExecReducer: processing 300 rows:
used memory = 311449488
2014-12-14 16:58:20,815 INFO ExecReducer: ExecReducer: processing 400 rows:
used memory = 285251752
2014-12-14 16:58:26,460 INFO ExecReducer: ExecReducer: processing 500 rows:
used memory = 328223864
2014-12-14 16:58:29,412 INFO ExecReducer: ExecReducer: processing 600 rows:
used memory = 263175576
2014-12-14 16:58:31,331 INFO ExecReducer: ExecReducer: processing 700 rows:
used memory = 282021320
2014-12-14 16:58:35,099 INFO ExecReducer: ExecReducer: processing 800 rows:
used memory = 299301184
2014-12-14 16:58:37,981 INFO ExecReducer: ExecReducer: processing 900 rows:
used memory = 306925648
2014-12-14 16:58:40,506 INFO ExecReducer: ExecReducer: processing 1000
rows: used memory = 307407920
2014-12-14 16:58:42,242 INFO ExecReducer: ExecReducer: processing 1100
rows: used memory = 304664048
2014-12-14 16:58:46,142 INFO ExecReducer: ExecReducer: processing 1200
rows: used memory = 298347024
2014-12-14 16:58:48,549 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
table 0 has 1000 rows for join key [003b9de7876541c2bcce9029ff0d3873]
2014-12-14 16:58:48,622 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
table 0 has 2000 rows for join key [003b9de7876541c2bcce9029ff0d3873]
2014-12-14 16:58:48,677 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
table 0 has 4000 rows for join key [003b9de7876541c2bcce9029ff0d3873]
2014-12-14 16:58:48,679 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Final Path: FS
hdfs://data-grill300-null.arshad.ev1.inmobi.com:8020/tmp/hive-dataqa/hive_2014-12-14_16-49-14_996_1630664550753106415-32/_tmp.-mr-10002/00_0
2014-12-14 16:58:48,680 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Writing to temp file: FS
hdfs://data-grill300-null.arshad.ev1.inmobi.com:8020/tmp/hive-dataqa/hive_2014-12-14_16-49-14_996_1630664550753106415-32/_task_tmp.-mr-10002/_tmp.00_0
2014-12-14 16:58:48,680 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
New Final Path: FS
hdfs://data-grill300-null.arshad.ev1.inmobi.com:8020/tmp/hive-dataqa/hive_2014-12-14_16-49-14_996_16
[jira] [Created] (HIVE-9324) Reduce side joins failing with IOException from RowContainer.nextBlock
Amareshwari Sriramadasu created HIVE-9324:
-
Summary: Reduce side joins failing with IOException from
RowContainer.nextBlock
Key: HIVE-9324
URL: https://issues.apache.org/jira/browse/HIVE-9324
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.13.1
Reporter: Amareshwari Sriramadasu
We are seeing some reduce side join mapreduce jobs failing with following
exception :
{noformat}
2014-12-14 16:58:51,296 ERROR
org.apache.hadoop.hive.ql.exec.persistence.RowContainer:
org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer@42610e8 read 1 bytes, should read
27264
java.io.IOException: org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer@42610e8 read
1 bytes, should read 27264
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2435)
at
org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:76)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.nextBlock(RowContainer.java:360)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:230)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:74)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:644)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:758)
at
org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256)
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:216)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:506)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
2014-12-14 16:58:51,334 FATAL ExecReducer:
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer@42610e8 read 1 bytes, should read
27264
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:237)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:74)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:644)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:758)
at
org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256)
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:216)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:506)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.io.IOException: org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer@42610e8 read
1 bytes, should read 27264
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.nextBlock(RowContainer.java:385)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:230)
... 12 more
Caused by: java.io.IOException:
org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer@42610e8 read 1 bytes, should read
27264
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2435)
at
org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:76)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.nextBlock(RowContainer.java:360)
... 13 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9309) schematool fails on Postgres 8.1
[
https://issues.apache.org/jira/browse/HIVE-9309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270570#comment-14270570
]
Hive QA commented on HIVE-9309:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690741/HIVE-9309.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6740 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2299/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2299/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2299/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690741 - PreCommit-HIVE-TRUNK-Build
> schematool fails on Postgres 8.1
>
>
> Key: HIVE-9309
> URL: https://issues.apache.org/jira/browse/HIVE-9309
> Project: Hive
> Issue Type: Bug
> Components: Database/Schema
>Reporter: Mohit Sabharwal
>Assignee: Mohit Sabharwal
> Attachments: HIVE-9309.patch
>
>
> Postgres upgrade scripts set {{standard_conforming_strings}} which is not
> allowed in 8.1:
> {code}
> ERROR: parameter "standard_conforming_strings" cannot be changed
> (state=55P02,code=0)
> {code}
> Postgres [8.1 Release
> notes|http://www.postgresql.org/docs/8.2/static/release-8-1.html] say that
> "standard_conforming_strings value is read-only"
> Postgres [8.2
> notes|http://www.postgresql.org/docs/8.2/static/release-8-2.html] say that it
> can be set at runtime.
> It'd be nice to address this for those still using Postgres 8.1
> This patch provides a schemaTool db option "postgres.filter.81" which, if
> set, filters out the "standard_conforming_strings" statement from upgrade
> scripts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9307) Use metastore warehouse dir variable from conf instead of hard coded dir in encryption test
[
https://issues.apache.org/jira/browse/HIVE-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-9307:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thank you Dong! I have committed this to branch!
> Use metastore warehouse dir variable from conf instead of hard coded dir in
> encryption test
> ---
>
> Key: HIVE-9307
> URL: https://issues.apache.org/jira/browse/HIVE-9307
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Fix For: encryption-branch
>
> Attachments: HIVE-9307.1.patch, HIVE-9307.patch
>
>
> NO PRECOMMIT TESTS
> Use the following variable to get the metastore directory
> $\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Review Request 29753: Support split sampling for parallel order by
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29753/ --- Review request for hive. Bugs: HIVE-9210 https://issues.apache.org/jira/browse/HIVE-9210 Repository: hive-git Description --- Random sampling cannot sure it covers whole range of input data. See HIVE-7661 Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8264b16 ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 0ccab02 ql/src/java/org/apache/hadoop/hive/ql/exec/PartitionKeySampler.java 96f4530 ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 2227e6f ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 9f8c091 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java ac2cc86 ql/src/test/queries/clientpositive/parallel_orderby.q 73c3940 ql/src/test/results/clientpositive/parallel_orderby.q.out 2f4ac8f Diff: https://reviews.apache.org/r/29753/diff/ Testing --- Thanks, Navis Ryu
[jira] [Updated] (HIVE-9242) Many places in CBO code eat exceptions
[ https://issues.apache.org/jira/browse/HIVE-9242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-9242: Resolution: Fixed Fix Version/s: 0.15.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks Brock. > Many places in CBO code eat exceptions > -- > > Key: HIVE-9242 > URL: https://issues.apache.org/jira/browse/HIVE-9242 > Project: Hive > Issue Type: Bug >Reporter: Brock Noland >Priority: Blocker > Fix For: 0.15.0 > > Attachments: HIVE-9242.1.patch.txt > > > I've noticed that there are a number of places in the CBO code which eat > exceptions. This is not acceptable. Example: > https://github.com/apache/hive/blob/357b473a354aace3bd59b522ad7108be561e9d0f/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java#L274 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9242) Many places in CBO code eat exceptions
[
https://issues.apache.org/jira/browse/HIVE-9242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270519#comment-14270519
]
Hive QA commented on HIVE-9242:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690708/HIVE-9242.1.patch.txt
{color:green}SUCCESS:{color} +1 6739 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2298/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2298/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2298/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690708 - PreCommit-HIVE-TRUNK-Build
> Many places in CBO code eat exceptions
> --
>
> Key: HIVE-9242
> URL: https://issues.apache.org/jira/browse/HIVE-9242
> Project: Hive
> Issue Type: Bug
>Reporter: Brock Noland
>Priority: Blocker
> Attachments: HIVE-9242.1.patch.txt
>
>
> I've noticed that there are a number of places in the CBO code which eat
> exceptions. This is not acceptable. Example:
> https://github.com/apache/hive/blob/357b473a354aace3bd59b522ad7108be561e9d0f/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java#L274
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9310) CLI JLine does not flush history back to ~/.hivehistory
[
https://issues.apache.org/jira/browse/HIVE-9310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270515#comment-14270515
]
Navis commented on HIVE-9310:
-
Should it be called in shutdown hook or at least in signal handler either?
> CLI JLine does not flush history back to ~/.hivehistory
> ---
>
> Key: HIVE-9310
> URL: https://issues.apache.org/jira/browse/HIVE-9310
> Project: Hive
> Issue Type: Bug
> Components: CLI
>Affects Versions: 0.15.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Minor
> Attachments: HIVE-9310.1.patch
>
>
> Hive CLI does not seem to be saving history anymore.
> In JLine with the PersistentHistory class, to keep history across sessions,
> you need to do {{reader.getHistory().flush()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7550) Extend cached evaluation to multiple expressions
[ https://issues.apache.org/jira/browse/HIVE-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-7550: Attachment: HIVE-7550.2.patch.txt > Extend cached evaluation to multiple expressions > > > Key: HIVE-7550 > URL: https://issues.apache.org/jira/browse/HIVE-7550 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Navis >Assignee: Navis >Priority: Trivial > Attachments: HIVE-7550.1.patch.txt, HIVE-7550.2.patch.txt > > > Currently, hive.cache.expr.evaluation caches per expression. But cache > context might be shared for multiple expressions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-4790) MapredLocalTask task does not make virtual columns
[
https://issues.apache.org/jira/browse/HIVE-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-4790:
Attachment: HIVE-4790.14.patch.txt
> MapredLocalTask task does not make virtual columns
> --
>
> Key: HIVE-4790
> URL: https://issues.apache.org/jira/browse/HIVE-4790
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D11511.3.patch, D11511.4.patch, HIVE-4790.10.patch.txt,
> HIVE-4790.11.patch.txt, HIVE-4790.12.patch.txt, HIVE-4790.13.patch.txt,
> HIVE-4790.14.patch.txt, HIVE-4790.5.patch.txt, HIVE-4790.6.patch.txt,
> HIVE-4790.7.patch.txt, HIVE-4790.8.patch.txt, HIVE-4790.9.patch.txt,
> HIVE-4790.D11511.1.patch, HIVE-4790.D11511.2.patch
>
>
> From mailing list,
> http://www.mail-archive.com/user@hive.apache.org/msg08264.html
> {noformat}
> SELECT *,b.BLOCK__OFFSET__INSIDE__FILE FROM a JOIN b ON
> b.rownumber = a.number;
> fails with this error:
>
> > SELECT *,b.BLOCK__OFFSET__INSIDE__FILE FROM a JOIN b ON b.rownumber =
> a.number;
> Automatically selecting local only mode for query
> Total MapReduce jobs = 1
> setting HADOOP_USER_NAMEpmarron
> 13/06/25 10:52:56 WARN conf.HiveConf: DEPRECATED: Configuration property
> hive.metastore.local no longer has any effect. Make sure to provide a valid
> value for hive.metastore.uris if you are connecting to a remote metastore.
> Execution log at: /tmp/pmarron/.log
> 2013-06-25 10:52:56 Starting to launch local task to process map join;
> maximum memory = 932118528
> java.lang.RuntimeException: cannot find field block__offset__inside__file
> from [0:rownumber, 1:offset]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:366)
> at
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.getStructFieldRef(LazySimpleStructObjectInspector.java:168)
> at
> org.apache.hadoop.hive.serde2.objectinspector.DelegatedStructObjectInspector.getStructFieldRef(DelegatedStructObjectInspector.java:74)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:68)
> at
> org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:222)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:451)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:407)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:186)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at
> org.apache.hadoop.hive.ql.exec.MapredLocalTask.initializeOperators(MapredLocalTask.java:394)
> at
> org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:277)
> at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:676)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Execution failed with exit status: 2
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7550) Extend cached evaluation to multiple expressions
[ https://issues.apache.org/jira/browse/HIVE-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-7550: Status: Patch Available (was: Open) > Extend cached evaluation to multiple expressions > > > Key: HIVE-7550 > URL: https://issues.apache.org/jira/browse/HIVE-7550 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Navis >Assignee: Navis >Priority: Trivial > Attachments: HIVE-7550.1.patch.txt, HIVE-7550.2.patch.txt > > > Currently, hive.cache.expr.evaluation caches per expression. But cache > context might be shared for multiple expressions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8872) Hive view of HBase range scan intermittently returns incorrect data.
[ https://issues.apache.org/jira/browse/HIVE-8872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270500#comment-14270500 ] 杨昆 commented on HIVE-8872: -- how can I reappear this bug? [~ychena] Can you provide your code and the hive runtime configuration? > Hive view of HBase range scan intermittently returns incorrect data. > > > Key: HIVE-8872 > URL: https://issues.apache.org/jira/browse/HIVE-8872 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.1 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 0.15.0 > > Attachments: HIVE-8872.1.patch, HIVE-8872.2.patch > > > This need running in cluster: > 1. Create a hive external table pointing to a hbase table. > 2. Create views to the hive table(for example 30 views), each view looks like > following with different range check: > CREATE VIEW hview_nn AS SELECT * FROM hivehbasetable WHERE (pk >='pk_nn_0' > AND pk <= pk_nn_A') > 3. Create same number of hive new tables as views. > 4. then runs several queries in parallel (30 threads): > INSERT OVERWRITE TABLE hivenewtable_nn SELECT * FROM hview_nn //nn is from > 01 to 30 > 5 After insert, check the hivenewtables, some values are not right. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9323) Merge from trunk to spark 1/8/2015
[
https://issues.apache.org/jira/browse/HIVE-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270497#comment-14270497
]
Hive QA commented on HIVE-9323:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12691015/HIVE-9323-spark.patch
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 7301 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ctas
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_cube1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_windowing
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/622/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/622/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-622/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12691015 - PreCommit-HIVE-SPARK-Build
> Merge from trunk to spark 1/8/2015
> --
>
> Key: HIVE-9323
> URL: https://issues.apache.org/jira/browse/HIVE-9323
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Affects Versions: spark-branch
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-9323-spark.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9307) Use metastore warehouse dir variable from conf instead of hard coded dir in encryption test
[
https://issues.apache.org/jira/browse/HIVE-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270488#comment-14270488
]
Brock Noland commented on HIVE-9307:
+1
> Use metastore warehouse dir variable from conf instead of hard coded dir in
> encryption test
> ---
>
> Key: HIVE-9307
> URL: https://issues.apache.org/jira/browse/HIVE-9307
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Fix For: encryption-branch
>
> Attachments: HIVE-9307.1.patch, HIVE-9307.patch
>
>
> NO PRECOMMIT TESTS
> Use the following variable to get the metastore directory
> $\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9290) Make some test results deterministic
[
https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-9290:
--
Resolution: Fixed
Fix Version/s: 0.15.0
spark-branch
Status: Resolved (was: Patch Available)
Committed to trunk and merged to Spark branch. Thanks, Rui.
> Make some test results deterministic
>
>
> Key: HIVE-9290
> URL: https://issues.apache.org/jira/browse/HIVE-9290
> Project: Hive
> Issue Type: Test
>Reporter: Rui Li
>Assignee: Rui Li
> Fix For: spark-branch, 0.15.0
>
> Attachments: HIVE-9290.1.patch, HIVE-9290.1.patch
>
>
> {noformat}
> limit_pushdown.q
> optimize_nullscan.q
> ppd_gby_join.q
> vector_string_concat.q
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9307) Use metastore warehouse dir variable from conf instead of hard coded dir in encryption test
[
https://issues.apache.org/jira/browse/HIVE-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270486#comment-14270486
]
Dong Chen commented on HIVE-9307:
-
Not necessary. Other .q files result keep same.
The trick is that:
In {{QTestUtil.planMask}}, one is {{LOCATION}}. This cause the table location
in create query in PREHOOK / POSTHOOK masked.
In the 2 .q file of this patch, it is {{location}} so that it is not masked, as
expected.
> Use metastore warehouse dir variable from conf instead of hard coded dir in
> encryption test
> ---
>
> Key: HIVE-9307
> URL: https://issues.apache.org/jira/browse/HIVE-9307
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Fix For: encryption-branch
>
> Attachments: HIVE-9307.1.patch, HIVE-9307.patch
>
>
> NO PRECOMMIT TESTS
> Use the following variable to get the metastore directory
> $\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-3405) UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase
[ https://issues.apache.org/jira/browse/HIVE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Pivovarov updated HIVE-3405: -- Attachment: HIVE-3405.5.patch - use WorkUtils.capitalizeFully instead of custom implementation - check arguments.length == 1 in GenericUDFInitCap.initialize > UDF initcap to obtain a string with the first letter of each word in > uppercase other letters in lowercase > - > > Key: HIVE-3405 > URL: https://issues.apache.org/jira/browse/HIVE-3405 > Project: Hive > Issue Type: New Feature > Components: UDF >Affects Versions: 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.11.0, 0.13.0, 0.14.0, > 0.15.0, 0.14.1 >Reporter: Archana Nair >Assignee: Alexander Pivovarov > Labels: patch > Attachments: HIVE-3405.1.patch.txt, HIVE-3405.2.patch, > HIVE-3405.3.patch, HIVE-3405.4.patch, HIVE-3405.5.patch > > > Hive current releases lacks a INITCAP function which returns String with > first letter of the word in uppercase.INITCAP returns String, with the first > letter of each word in uppercase, all other letters in same case. Words are > delimited by white space.This will be useful report generation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-3405) UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase
[ https://issues.apache.org/jira/browse/HIVE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Pivovarov updated HIVE-3405: -- Status: Patch Available (was: In Progress) HIVE-3405.5.patch > UDF initcap to obtain a string with the first letter of each word in > uppercase other letters in lowercase > - > > Key: HIVE-3405 > URL: https://issues.apache.org/jira/browse/HIVE-3405 > Project: Hive > Issue Type: New Feature > Components: UDF >Affects Versions: 0.14.0, 0.13.0, 0.11.0, 0.10.0, 0.9.0, 0.8.1, 0.15.0, > 0.14.1, 0.9.1 >Reporter: Archana Nair >Assignee: Alexander Pivovarov > Labels: patch > Attachments: HIVE-3405.1.patch.txt, HIVE-3405.2.patch, > HIVE-3405.3.patch, HIVE-3405.4.patch, HIVE-3405.5.patch > > > Hive current releases lacks a INITCAP function which returns String with > first letter of the word in uppercase.INITCAP returns String, with the first > letter of each word in uppercase, all other letters in same case. Words are > delimited by white space.This will be useful report generation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-3405) UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase
[ https://issues.apache.org/jira/browse/HIVE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Pivovarov updated HIVE-3405: -- Status: In Progress (was: Patch Available) > UDF initcap to obtain a string with the first letter of each word in > uppercase other letters in lowercase > - > > Key: HIVE-3405 > URL: https://issues.apache.org/jira/browse/HIVE-3405 > Project: Hive > Issue Type: New Feature > Components: UDF >Affects Versions: 0.14.0, 0.13.0, 0.11.0, 0.10.0, 0.9.0, 0.8.1, 0.15.0, > 0.14.1, 0.9.1 >Reporter: Archana Nair >Assignee: Alexander Pivovarov > Labels: patch > Attachments: HIVE-3405.1.patch.txt, HIVE-3405.2.patch, > HIVE-3405.3.patch, HIVE-3405.4.patch > > > Hive current releases lacks a INITCAP function which returns String with > first letter of the word in uppercase.INITCAP returns String, with the first > letter of each word in uppercase, all other letters in same case. Words are > delimited by white space.This will be useful report generation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-9156) Investigate CBO [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland reassigned HIVE-9156: -- Assignee: Chao > Investigate CBO [Spark Branch] > -- > > Key: HIVE-9156 > URL: https://issues.apache.org/jira/browse/HIVE-9156 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Brock Noland >Assignee: Chao > > We need to get a list of the following: > * What happens when you enable the CBO with HOS > * What portions of the CBO integration are tez or MR specific > * Some basic idea of what our cost model will look like -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9307) Use metastore warehouse dir variable from conf instead of hard coded dir in encryption test
[
https://issues.apache.org/jira/browse/HIVE-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270469#comment-14270469
]
Brock Noland commented on HIVE-9307:
Do the other .q files which were modified need to be updated?
> Use metastore warehouse dir variable from conf instead of hard coded dir in
> encryption test
> ---
>
> Key: HIVE-9307
> URL: https://issues.apache.org/jira/browse/HIVE-9307
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Fix For: encryption-branch
>
> Attachments: HIVE-9307.1.patch, HIVE-9307.patch
>
>
> NO PRECOMMIT TESTS
> Use the following variable to get the metastore directory
> $\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9290) Make some test results deterministic
[
https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270456#comment-14270456
]
Rui Li commented on HIVE-9290:
--
The failed test is not related to the patch here.
> Make some test results deterministic
>
>
> Key: HIVE-9290
> URL: https://issues.apache.org/jira/browse/HIVE-9290
> Project: Hive
> Issue Type: Test
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-9290.1.patch, HIVE-9290.1.patch
>
>
> {noformat}
> limit_pushdown.q
> optimize_nullscan.q
> ppd_gby_join.q
> vector_string_concat.q
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9290) Make some test results deterministic
[
https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270450#comment-14270450
]
Hive QA commented on HIVE-9290:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690996/HIVE-9290.1.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6739 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2297/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2297/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2297/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690996 - PreCommit-HIVE-TRUNK-Build
> Make some test results deterministic
>
>
> Key: HIVE-9290
> URL: https://issues.apache.org/jira/browse/HIVE-9290
> Project: Hive
> Issue Type: Test
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-9290.1.patch, HIVE-9290.1.patch
>
>
> {noformat}
> limit_pushdown.q
> optimize_nullscan.q
> ppd_gby_join.q
> vector_string_concat.q
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-4639) Add has null flag to ORC internal index
[ https://issues.apache.org/jira/browse/HIVE-4639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-4639: Attachment: HIVE-4639.3.patch I missed out few test failure diffs in previous patch. Added them in this patch. > Add has null flag to ORC internal index > --- > > Key: HIVE-4639 > URL: https://issues.apache.org/jira/browse/HIVE-4639 > Project: Hive > Issue Type: Improvement > Components: File Formats >Reporter: Owen O'Malley >Assignee: Prasanth Jayachandran > Attachments: HIVE-4639.1.patch, HIVE-4639.2.patch, HIVE-4639.3.patch > > > It would enable more predicate pushdown if we added a flag to the index entry > recording if there were any null values in the column for the 10k rows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9304) [Refactor] remove unused method in SemAly
[
https://issues.apache.org/jira/browse/HIVE-9304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-9304:
---
Resolution: Fixed
Fix Version/s: 0.15.0
Status: Resolved (was: Patch Available)
Committed to trunk.
> [Refactor] remove unused method in SemAly
> -
>
> Key: HIVE-9304
> URL: https://issues.apache.org/jira/browse/HIVE-9304
> Project: Hive
> Issue Type: Task
> Components: Query Processor
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
>Priority: Trivial
> Fix For: 0.15.0
>
> Attachments: HIVE-9304.patch
>
>
> Seems like method {{genConversionOps}} don't serve any purpose any longer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8327) mvn site -Pfindbugs
[ https://issues.apache.org/jira/browse/HIVE-8327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270437#comment-14270437 ] Ashutosh Chauhan commented on HIVE-8327: +1 > mvn site -Pfindbugs > --- > > Key: HIVE-8327 > URL: https://issues.apache.org/jira/browse/HIVE-8327 > Project: Hive > Issue Type: Test > Components: Diagnosability >Reporter: Gopal V >Assignee: Gopal V > Fix For: 0.15.0 > > Attachments: HIVE-8327.1.patch, HIVE-8327.2.patch, ql-findbugs.html > > > HIVE-3099 originally added findbugs into the old ant build. > Get basic findbugs working for the maven build. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8853) Make vectorization work with Spark [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270418#comment-14270418
]
Brock Noland commented on HIVE-8853:
[~jxiang] I took some thread dumps of an executor JVM during execution with
vectorization turned on and I saw a ton of thread dumps here like the ones
below.
{noformat}
"Executor task launch worker-4" daemon prio=10 tid=0x7f8394048800
nid=0x707a runnable [0x7f8457dfb000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x000281c57b70> (a sun.nio.ch.Util$2)
- locked <0x000281c57b80> (a java.util.Collections$UnmodifiableSet)
- locked <0x000281c57b28> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.readChannelFully(PacketReceiver.java:258)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:209)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:171)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:102)
at
org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:186)
at
org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:146)
- locked <0x000718424118> (a
org.apache.hadoop.hdfs.RemoteBlockReader2)
at
org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:693)
at
org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:749)
- eliminated <0x0007184169b8> (a
org.apache.hadoop.hdfs.DFSInputStream)
at
org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:806)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:847)
- locked <0x0007184169b8> (a org.apache.hadoop.hdfs.DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hive.com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
at
org.apache.hive.com.esotericsoftware.kryo.io.Input.require(Input.java:178)
at
org.apache.hive.com.esotericsoftware.kryo.io.Input.readUtf8_slow(Input.java:542)
at
org.apache.hive.com.esotericsoftware.kryo.io.Input.readUtf8(Input.java:535)
at
org.apache.hive.com.esotericsoftware.kryo.io.Input.readString(Input.java:465)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.read(DefaultSerializers.java:171)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.read(DefaultSerializers.java:160)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at
org.apache.hive.com.esotericsoftwa
[jira] [Assigned] (HIVE-8853) Make vectorization work with Spark [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland reassigned HIVE-8853: -- Assignee: Jimmy Xiang Jimmy mentioned he was interested in this. > Make vectorization work with Spark [Spark Branch] > - > > Key: HIVE-8853 > URL: https://issues.apache.org/jira/browse/HIVE-8853 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Xuefu Zhang >Assignee: Jimmy Xiang > > In Hive to make vectorization work, the reader needs to be also vectorized, > which means that the reader can read a chunk of rows (or a list of column > chunks) instead of one row at a time. However, we use Spark RDD for reading, > which again utilized the underlying inputformat to read. Subsequent > processing also needs to hapen in batches. We need to make sure that > vectorizatoin is working as expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9323) Merge from trunk to spark 1/8/2015
[ https://issues.apache.org/jira/browse/HIVE-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-9323: Status: Patch Available (was: Open) Giving this a try for precommit test. > Merge from trunk to spark 1/8/2015 > -- > > Key: HIVE-9323 > URL: https://issues.apache.org/jira/browse/HIVE-9323 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-9323-spark.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9307) Use metastore warehouse dir variable from conf instead of hard coded dir in encryption test
[
https://issues.apache.org/jira/browse/HIVE-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9307:
Fix Version/s: encryption-branch
Status: Patch Available (was: Open)
> Use metastore warehouse dir variable from conf instead of hard coded dir in
> encryption test
> ---
>
> Key: HIVE-9307
> URL: https://issues.apache.org/jira/browse/HIVE-9307
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Fix For: encryption-branch
>
> Attachments: HIVE-9307.1.patch, HIVE-9307.patch
>
>
> NO PRECOMMIT TESTS
> Use the following variable to get the metastore directory
> $\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9323) Merge from trunk to spark 1/8/2015
[ https://issues.apache.org/jira/browse/HIVE-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-9323: Attachment: HIVE-9323-spark.patch > Merge from trunk to spark 1/8/2015 > -- > > Key: HIVE-9323 > URL: https://issues.apache.org/jira/browse/HIVE-9323 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-9323-spark.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9307) Use metastore warehouse dir variable from conf instead of hard coded dir in encryption test
[
https://issues.apache.org/jira/browse/HIVE-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9307:
Attachment: HIVE-9307.1.patch
> Use metastore warehouse dir variable from conf instead of hard coded dir in
> encryption test
> ---
>
> Key: HIVE-9307
> URL: https://issues.apache.org/jira/browse/HIVE-9307
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Attachments: HIVE-9307.1.patch, HIVE-9307.patch
>
>
> NO PRECOMMIT TESTS
> Use the following variable to get the metastore directory
> $\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9307) Use metastore warehouse dir variable from conf instead of hard coded dir in encryption test
[
https://issues.apache.org/jira/browse/HIVE-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270398#comment-14270398
]
Dong Chen commented on HIVE-9307:
-
Update patch V1.
The variable is used in 3 statement: {{CREATE table sql}}, {{dfs command}}, and
{{CRYPTO command}}. The first 2 already can handle this variable. For CRYPTO
command, since we only use it for test, I added the handling in
{{QTestUtil.executeTestCommand()}} for simplicity.
The output of 2 cases changed by 2 lines, which are the table location in
create query in PREHOOK / POSTHOOK is masked. After double check, I think it is
OK and I misunderstood before.
> Use metastore warehouse dir variable from conf instead of hard coded dir in
> encryption test
> ---
>
> Key: HIVE-9307
> URL: https://issues.apache.org/jira/browse/HIVE-9307
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Attachments: HIVE-9307.patch
>
>
> NO PRECOMMIT TESTS
> Use the following variable to get the metastore directory
> $\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9323) Merge from trunk to spark 1/8/2015
Szehon Ho created HIVE-9323: --- Summary: Merge from trunk to spark 1/8/2015 Key: HIVE-9323 URL: https://issues.apache.org/jira/browse/HIVE-9323 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Szehon Ho Assignee: Szehon Ho -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-836) Add syntax to force a new mapreduce job / transform subquery in mapper
[
https://issues.apache.org/jira/browse/HIVE-836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Kramer resolved HIVE-836.
--
Resolution: Won't Fix
Release Note: See comments for workarounds.
> Add syntax to force a new mapreduce job / transform subquery in mapper
> --
>
> Key: HIVE-836
> URL: https://issues.apache.org/jira/browse/HIVE-836
> Project: Hive
> Issue Type: Wish
>Reporter: Adam Kramer
>
> Hive currently does a lot of awesome work to figure out when my transformers
> should be used in the mapper and when they should be used in the reducer.
> However, sometimes I have a different plan.
> For example, consider this:
> {code:title=foo.sql}
> SELECT TRANSFORM(a.val1, a.val2)
> USING './niftyscript'
> AS part1, part2, part3
> FROM (
> SELECT b.val AS val1, c.val AS val2
> FROM tblb b JOIN tblc c on (b.key=c.key)
> ) a
> {code}
> ...now, assume that the join step is very easy and 'niftyscript' is really
> processor intensive. The ideal format for this is a MR task with few mappers
> and few reducers, and then a second MR task with lots of mappers.
> Currently, there is no way to even require the outer TRANSFORM statement
> occur in a separate map phase. Implementing a "hint" such as /* +MAP */, akin
> to /* +MAPJOIN(x) */, would be awesome.
> Current workaround is to dump everything to a temporary table and then start
> over, but that is not an easy to scale--the subquery structure effectively
> (and easily) "locks" the mid-points so no other job can touch the table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-836) Add syntax to force a new mapreduce job / transform subquery in mapper
[
https://issues.apache.org/jira/browse/HIVE-836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270390#comment-14270390
]
Adam Kramer commented on HIVE-836:
--
Oh hey there five year old task.
Workaround: Use CLUSTER BY to force a reduce phase, and a staging table to
force a map phase. Hive writes all the data to disk in every phase anyway so
the staging table isn't actually a performance hit.
Also protip: DON'T get distracted by the Hive keywords "MAP" and "REDUCE", they
are just synonyms for TRANSFORM and do not do what anybody expects.
> Add syntax to force a new mapreduce job / transform subquery in mapper
> --
>
> Key: HIVE-836
> URL: https://issues.apache.org/jira/browse/HIVE-836
> Project: Hive
> Issue Type: Wish
>Reporter: Adam Kramer
>
> Hive currently does a lot of awesome work to figure out when my transformers
> should be used in the mapper and when they should be used in the reducer.
> However, sometimes I have a different plan.
> For example, consider this:
> {code:title=foo.sql}
> SELECT TRANSFORM(a.val1, a.val2)
> USING './niftyscript'
> AS part1, part2, part3
> FROM (
> SELECT b.val AS val1, c.val AS val2
> FROM tblb b JOIN tblc c on (b.key=c.key)
> ) a
> {code}
> ...now, assume that the join step is very easy and 'niftyscript' is really
> processor intensive. The ideal format for this is a MR task with few mappers
> and few reducers, and then a second MR task with lots of mappers.
> Currently, there is no way to even require the outer TRANSFORM statement
> occur in a separate map phase. Implementing a "hint" such as /* +MAP */, akin
> to /* +MAPJOIN(x) */, would be awesome.
> Current workaround is to dump everything to a temporary table and then start
> over, but that is not an easy to scale--the subquery structure effectively
> (and easily) "locks" the mid-points so no other job can touch the table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9319) Cleanup Modified Files [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-9319: Resolution: Fixed Fix Version/s: spark-branch Status: Resolved (was: Patch Available) Committed to spark-branch. Thanks Xuefu for review. > Cleanup Modified Files [Spark Branch] > - > > Key: HIVE-9319 > URL: https://issues.apache.org/jira/browse/HIVE-9319 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Minor > Fix For: spark-branch > > Attachments: HIVE-9319-spark.patch > > > Cleanup the code that is modified based on checkstyle/TODO/warnings. > It is a follow-up of HIVE-9281 which is for new files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9319) Cleanup Modified Files [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270369#comment-14270369
]
Hive QA commented on HIVE-9319:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690973/HIVE-9319-spark.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 7285 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby3_map_skew
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_windowing
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/621/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/621/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-621/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690973 - PreCommit-HIVE-SPARK-Build
> Cleanup Modified Files [Spark Branch]
> -
>
> Key: HIVE-9319
> URL: https://issues.apache.org/jira/browse/HIVE-9319
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Affects Versions: spark-branch
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Minor
> Attachments: HIVE-9319-spark.patch
>
>
> Cleanup the code that is modified based on checkstyle/TODO/warnings.
> It is a follow-up of HIVE-9281 which is for new files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9272) Tests for utf-8 support
[ https://issues.apache.org/jira/browse/HIVE-9272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-9272: - Assignee: Aswathy Chellammal Sreekumar > Tests for utf-8 support > --- > > Key: HIVE-9272 > URL: https://issues.apache.org/jira/browse/HIVE-9272 > Project: Hive > Issue Type: Test > Components: Tests, WebHCat >Reporter: Aswathy Chellammal Sreekumar >Assignee: Aswathy Chellammal Sreekumar >Priority: Minor > Attachments: HIVE-9272.1.patch, HIVE-9272.patch > > > Including some test cases for utf8 support in webhcat. The first four tests > invoke hive, pig, mapred and streaming apis for testing the utf8 support for > data processed, file names and job name. The last test case tests the > filtering of job name with utf8 character -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9272) Tests for utf-8 support
[ https://issues.apache.org/jira/browse/HIVE-9272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270365#comment-14270365 ] Eugene Koifman commented on HIVE-9272: -- 1. could you add a comment to the change in deploy_e2e_artifacts.sh to indicate what set of tests the artifact is for? 2. I'm not sure how you are generating the patch, but it has some headers that may not apply. I usually use "git diff --no-prefix SHA SHA1 > foo.patch" 3. my knowledge of Perl is very limited. Is there someone else who can review .pm changes? > Tests for utf-8 support > --- > > Key: HIVE-9272 > URL: https://issues.apache.org/jira/browse/HIVE-9272 > Project: Hive > Issue Type: Test > Components: Tests, WebHCat >Reporter: Aswathy Chellammal Sreekumar >Priority: Minor > Attachments: HIVE-9272.1.patch, HIVE-9272.patch > > > Including some test cases for utf8 support in webhcat. The first four tests > invoke hive, pig, mapred and streaming apis for testing the utf8 support for > data processed, file names and job name. The last test case tests the > filtering of job name with utf8 character -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4639) Add has null flag to ORC internal index
[
https://issues.apache.org/jira/browse/HIVE-4639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270348#comment-14270348
]
Hive QA commented on HIVE-4639:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690690/HIVE-4639.2.patch
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 6747 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.io.orc.TestOrcNullOptimization.testColumnsWithNullAndCompression
org.apache.hadoop.hive.ql.io.orc.TestOrcNullOptimization.testMultiStripeWithNull
org.apache.hadoop.hive.ql.io.orc.TestOrcNullOptimization.testMultiStripeWithoutNull
org.apache.hadoop.hive.ql.io.orc.TestOrcSerDeStats.testOrcSerDeStatsComplex
org.apache.hadoop.hive.ql.io.orc.TestOrcSerDeStats.testOrcSerDeStatsComplexOldFormat
org.apache.hadoop.hive.ql.io.orc.TestOrcSerDeStats.testSerdeStatsOldFormat
org.apache.hadoop.hive.ql.io.orc.TestOrcSerDeStats.testStringAndBinaryStatistics
org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2296/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2296/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2296/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690690 - PreCommit-HIVE-TRUNK-Build
> Add has null flag to ORC internal index
> ---
>
> Key: HIVE-4639
> URL: https://issues.apache.org/jira/browse/HIVE-4639
> Project: Hive
> Issue Type: Improvement
> Components: File Formats
>Reporter: Owen O'Malley
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-4639.1.patch, HIVE-4639.2.patch
>
>
> It would enable more predicate pushdown if we added a flag to the index entry
> recording if there were any null values in the column for the 10k rows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9290) Make some test results deterministic
[
https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-9290:
-
Attachment: HIVE-9290.1.patch
Reload patch to trigger test
> Make some test results deterministic
>
>
> Key: HIVE-9290
> URL: https://issues.apache.org/jira/browse/HIVE-9290
> Project: Hive
> Issue Type: Test
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-9290.1.patch, HIVE-9290.1.patch
>
>
> {noformat}
> limit_pushdown.q
> optimize_nullscan.q
> ppd_gby_join.q
> vector_string_concat.q
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8696) HCatClientHMSImpl doesn't use a Retrying-HiveMetastoreClient.
[ https://issues.apache.org/jira/browse/HIVE-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mithun Radhakrishnan updated HIVE-8696: --- Status: Patch Available (was: Open) > HCatClientHMSImpl doesn't use a Retrying-HiveMetastoreClient. > - > > Key: HIVE-8696 > URL: https://issues.apache.org/jira/browse/HIVE-8696 > Project: Hive > Issue Type: Bug > Components: HCatalog, Metastore >Affects Versions: 0.13.1, 0.12.0 >Reporter: Mithun Radhakrishnan >Assignee: Mithun Radhakrishnan > Attachments: HIVE-8696.1.patch > > > The HCatClientHMSImpl doesn't use a RetryingHiveMetastoreClient. Users of the > HCatClient API that log in through keytabs will fail without retry, when > their TGTs expire. > The fix is inbound. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9322) Make null-checks consistent for MapObjectInspector subclasses.
[
https://issues.apache.org/jira/browse/HIVE-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-9322:
---
Status: Patch Available (was: Open)
> Make null-checks consistent for MapObjectInspector subclasses.
> --
>
> Key: HIVE-9322
> URL: https://issues.apache.org/jira/browse/HIVE-9322
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 0.14.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
>Priority: Minor
> Attachments: HIVE-9322.1.patch
>
>
> {{LazyBinaryMapObjectInspector}}, {{DeepParquetHiveMapInspector}}, etc. check
> both the map-column value and the map-key for null, before dereferencing
> them. {{OrcMapObjectInspector}} and {{LazyMapObjectInspector}} do not.
> This patch brings them all in sync. Might not be a real problem, unless (for
> example) the lookup key is itself a (possibly null) value from another column.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9322) Make null-checks consistent for MapObjectInspector subclasses.
[
https://issues.apache.org/jira/browse/HIVE-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-9322:
---
Attachment: HIVE-9322.1.patch
> Make null-checks consistent for MapObjectInspector subclasses.
> --
>
> Key: HIVE-9322
> URL: https://issues.apache.org/jira/browse/HIVE-9322
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 0.14.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
>Priority: Minor
> Attachments: HIVE-9322.1.patch
>
>
> {{LazyBinaryMapObjectInspector}}, {{DeepParquetHiveMapInspector}}, etc. check
> both the map-column value and the map-key for null, before dereferencing
> them. {{OrcMapObjectInspector}} and {{LazyMapObjectInspector}} do not.
> This patch brings them all in sync. Might not be a real problem, unless (for
> example) the lookup key is itself a (possibly null) value from another column.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9322) Make null-checks consistent for MapObjectInspector subclasses.
Mithun Radhakrishnan created HIVE-9322:
--
Summary: Make null-checks consistent for MapObjectInspector
subclasses.
Key: HIVE-9322
URL: https://issues.apache.org/jira/browse/HIVE-9322
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.14.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Priority: Minor
{{LazyBinaryMapObjectInspector}}, {{DeepParquetHiveMapInspector}}, etc. check
both the map-column value and the map-key for null, before dereferencing them.
{{OrcMapObjectInspector}} and {{LazyMapObjectInspector}} do not.
This patch brings them all in sync. Might not be a real problem, unless (for
example) the lookup key is itself a (possibly null) value from another column.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9321) Notification message size can be arbitrarily long, DbNotificationListener limits to 1024
Sushanth Sowmyan created HIVE-9321: -- Summary: Notification message size can be arbitrarily long, DbNotificationListener limits to 1024 Key: HIVE-9321 URL: https://issues.apache.org/jira/browse/HIVE-9321 Project: Hive Issue Type: Bug Reporter: Sushanth Sowmyan Assignee: Alan Gates As part of HIVE-9174, a new notification listener, DbNotificationListener was introduced that stores notification messages in the metastore. Now, the "message" field of a Notification event can be arbitrarily long, since it is a JSON-serialized message of what event actually occurred, and an AddPartitionMessage which captured the result of an add_partitions can be arbitrarily long (as long as the number of partitions that were added). However, the "message" field for the storage in db is limited to VARCHAR(1024), which it will easily break. We need to fix this somehow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9319) Cleanup Modified Files [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270303#comment-14270303 ] Xuefu Zhang commented on HIVE-9319: --- +1 pending on test > Cleanup Modified Files [Spark Branch] > - > > Key: HIVE-9319 > URL: https://issues.apache.org/jira/browse/HIVE-9319 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Minor > Attachments: HIVE-9319-spark.patch > > > Cleanup the code that is modified based on checkstyle/TODO/warnings. > It is a follow-up of HIVE-9281 which is for new files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29733: HIVE-9319 : Cleanup Modified Files [Spark Branch]
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29733/#review67348 --- Ship it! Ship It! - Xuefu Zhang On Jan. 9, 2015, 12:01 a.m., Szehon Ho wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/29733/ > --- > > (Updated Jan. 9, 2015, 12:01 a.m.) > > > Review request for hive and Xuefu Zhang. > > > Repository: hive-git > > > Description > --- > > Note that this limits cleanup to lines of code changed in spark-branch in the > merge to trunk, not cleanup of all of the modified files, in order to reduce > merge conflicts. > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/Driver.java fa40082 > ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java b25a639 > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java ee42f4c > > ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java > abdb6af > ql/src/java/org/apache/hadoop/hive/ql/io/HiveKey.java 33aeda4 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 6f216c9 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java a6d5c62 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/unionproc/UnionProcessor.java > fec6822 > ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java 1b6de64 > ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 1efbb12 > > ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java > 4582678 > ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 076d2fa > shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java > f1743ae > > Diff: https://reviews.apache.org/r/29733/diff/ > > > Testing > --- > > > Thanks, > > Szehon Ho > >
[jira] [Updated] (HIVE-8966) Delta files created by hive hcatalog streaming cannot be compacted
[ https://issues.apache.org/jira/browse/HIVE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-8966: - Status: Open (was: Patch Available) > Delta files created by hive hcatalog streaming cannot be compacted > -- > > Key: HIVE-8966 > URL: https://issues.apache.org/jira/browse/HIVE-8966 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 0.14.0 > Environment: hive >Reporter: Jihong Liu >Assignee: Alan Gates >Priority: Critical > Fix For: 0.14.1 > > Attachments: HIVE-8966.2.patch, HIVE-8966.3.patch, HIVE-8966.patch > > > hive hcatalog streaming will also create a file like bucket_n_flush_length in > each delta directory. Where "n" is the bucket number. But the > compactor.CompactorMR think this file also needs to compact. However this > file of course cannot be compacted, so compactor.CompactorMR will not > continue to do the compaction. > Did a test, after removed the bucket_n_flush_length file, then the "alter > table partition compact" finished successfully. If don't delete that file, > nothing will be compacted. > This is probably a very severity bug. Both 0.13 and 0.14 have this issue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8966) Delta files created by hive hcatalog streaming cannot be compacted
[ https://issues.apache.org/jira/browse/HIVE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270282#comment-14270282 ] Alan Gates commented on HIVE-8966: -- The issue is that since the writer died with an unclosed batch it left the orc file in a state where it cannot be read without the length file. So removing the length file means any reader will fail when reading it. The proper solution is for the compactor to stop at that partition until it has determined all transactions in that file have committed or aborted. Then it should compact it using the length file, but properly ignore the length file. I'll work on the fix. > Delta files created by hive hcatalog streaming cannot be compacted > -- > > Key: HIVE-8966 > URL: https://issues.apache.org/jira/browse/HIVE-8966 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 0.14.0 > Environment: hive >Reporter: Jihong Liu >Assignee: Alan Gates >Priority: Critical > Fix For: 0.14.1 > > Attachments: HIVE-8966.2.patch, HIVE-8966.3.patch, HIVE-8966.patch > > > hive hcatalog streaming will also create a file like bucket_n_flush_length in > each delta directory. Where "n" is the bucket number. But the > compactor.CompactorMR think this file also needs to compact. However this > file of course cannot be compacted, so compactor.CompactorMR will not > continue to do the compaction. > Did a test, after removed the bucket_n_flush_length file, then the "alter > table partition compact" finished successfully. If don't delete that file, > nothing will be compacted. > This is probably a very severity bug. Both 0.13 and 0.14 have this issue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9320) Add UnionEliminatorRule on cbo path
[ https://issues.apache.org/jira/browse/HIVE-9320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-9320: --- Status: Patch Available (was: Open) > Add UnionEliminatorRule on cbo path > --- > > Key: HIVE-9320 > URL: https://issues.apache.org/jira/browse/HIVE-9320 > Project: Hive > Issue Type: Improvement > Components: CBO, Logical Optimizer >Affects Versions: 0.14.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-9320.patch > > > Shorten the pipeline, where possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9320) Add UnionEliminatorRule on cbo path
[ https://issues.apache.org/jira/browse/HIVE-9320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-9320: --- Attachment: HIVE-9320.patch > Add UnionEliminatorRule on cbo path > --- > > Key: HIVE-9320 > URL: https://issues.apache.org/jira/browse/HIVE-9320 > Project: Hive > Issue Type: Improvement > Components: CBO, Logical Optimizer >Affects Versions: 0.14.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-9320.patch > > > Shorten the pipeline, where possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9320) Add UnionEliminatorRule on cbo path
Ashutosh Chauhan created HIVE-9320: -- Summary: Add UnionEliminatorRule on cbo path Key: HIVE-9320 URL: https://issues.apache.org/jira/browse/HIVE-9320 Project: Hive Issue Type: Improvement Components: CBO, Logical Optimizer Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Shorten the pipeline, where possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8485) HMS on Oracle incompatibility
[
https://issues.apache.org/jira/browse/HIVE-8485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270263#comment-14270263
]
Sushanth Sowmyan commented on HIVE-8485:
[~ctang.ma]/[~sershe], could you please check/review this patch, and this
approach?
> HMS on Oracle incompatibility
> -
>
> Key: HIVE-8485
> URL: https://issues.apache.org/jira/browse/HIVE-8485
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Environment: Oracle as metastore DB
>Reporter: Ryan Pridgeon
>Assignee: Chaoyu Tang
> Attachments: HIVE-8485.2.patch, HIVE-8485.patch
>
>
> Oracle does not distinguish between empty strings and NULL,which proves
> problematic for DataNucleus.
> In the event a user creates a table with some property stored as an empty
> string the table will no longer be accessible.
> i.e. TBLPROPERTIES ('serialization.null.format'='')
> If they try to select, describe, drop, etc the client prints the following
> exception.
> ERROR ql.Driver: FAILED: SemanticException [Error 10001]: Table not found
>
> The work around for this was to go into the hive metastore on the Oracle
> database and replace NULL with some other string. Users could then drop the
> tables or alter their data to use the new null format they just set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Review Request 29733: HIVE-9319 : Cleanup Modified Files [Spark Branch]
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29733/ --- Review request for hive and Xuefu Zhang. Repository: hive-git Description --- Note that this limits cleanup to lines of code changed in spark-branch in the merge to trunk, not cleanup of all of the modified files, in order to reduce merge conflicts. Diffs - ql/src/java/org/apache/hadoop/hive/ql/Driver.java fa40082 ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java b25a639 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java ee42f4c ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java abdb6af ql/src/java/org/apache/hadoop/hive/ql/io/HiveKey.java 33aeda4 ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 6f216c9 ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java a6d5c62 ql/src/java/org/apache/hadoop/hive/ql/optimizer/unionproc/UnionProcessor.java fec6822 ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java 1b6de64 ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 1efbb12 ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 4582678 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 076d2fa shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java f1743ae Diff: https://reviews.apache.org/r/29733/diff/ Testing --- Thanks, Szehon Ho
[jira] [Updated] (HIVE-9319) Cleanup Modified Files [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-9319: Status: Patch Available (was: Open) > Cleanup Modified Files [Spark Branch] > - > > Key: HIVE-9319 > URL: https://issues.apache.org/jira/browse/HIVE-9319 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Minor > Attachments: HIVE-9319-spark.patch > > > Cleanup the code that is modified based on checkstyle/TODO/warnings. > It is a follow-up of HIVE-9281 which is for new files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9319) Cleanup Modified Files [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-9319: Attachment: HIVE-9319-spark.patch > Cleanup Modified Files [Spark Branch] > - > > Key: HIVE-9319 > URL: https://issues.apache.org/jira/browse/HIVE-9319 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Minor > Attachments: HIVE-9319-spark.patch > > > Cleanup the code that is modified based on checkstyle/TODO/warnings. > It is a follow-up of HIVE-9281 which is for new files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9316) TestSqoop tests in WebHCat testsuite hardcode libdir path to hdfs
[ https://issues.apache.org/jira/browse/HIVE-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270244#comment-14270244 ] Eugene Koifman commented on HIVE-9316: -- +1 > TestSqoop tests in WebHCat testsuite hardcode libdir path to hdfs > - > > Key: HIVE-9316 > URL: https://issues.apache.org/jira/browse/HIVE-9316 > Project: Hive > Issue Type: Bug > Components: Tests, WebHCat >Reporter: Deepesh Khandelwal >Assignee: Deepesh Khandelwal >Priority: Minor > Fix For: 0.15.0 > > Attachments: HIVE-9316.1.patch > > > Currently the TestSqoop tests in WebHCat Perl based testsuite has hdfs:// > prefix in the jdbc jar path in libdir, we should remove this to enable it to > run against other file systems. > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9319) Cleanup Modified Files [Spark Branch]
Szehon Ho created HIVE-9319: --- Summary: Cleanup Modified Files [Spark Branch] Key: HIVE-9319 URL: https://issues.apache.org/jira/browse/HIVE-9319 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Szehon Ho Assignee: Szehon Ho Priority: Minor Cleanup the code that is modified based on checkstyle/TODO/warnings. It is a follow-up of HIVE-9281 which is for new files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9175) Add alters to list of events handled by NotificationListener
[ https://issues.apache.org/jira/browse/HIVE-9175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270216#comment-14270216 ] Alan Gates commented on HIVE-9175: -- I don't believe the TestStreaming failure is related, as it passes locally and I didn't change any code in this area. I thought I fixed the flakiness in that test but I guess not. > Add alters to list of events handled by NotificationListener > > > Key: HIVE-9175 > URL: https://issues.apache.org/jira/browse/HIVE-9175 > Project: Hive > Issue Type: New Feature > Components: HCatalog >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.15.0 > > Attachments: HIVE-9175.patch > > > HCatalog currently doesn't implement onAlterTable and onAlterPartition. It > should. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9318) Add UnionMerge rule on cbo path
[ https://issues.apache.org/jira/browse/HIVE-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-9318: --- Affects Version/s: 0.14.0 Status: Patch Available (was: Open) > Add UnionMerge rule on cbo path > --- > > Key: HIVE-9318 > URL: https://issues.apache.org/jira/browse/HIVE-9318 > Project: Hive > Issue Type: Improvement > Components: CBO, Logical Optimizer >Affects Versions: 0.14.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-9318.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9318) Add UnionMerge rule on cbo path
[ https://issues.apache.org/jira/browse/HIVE-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-9318: --- Component/s: Logical Optimizer CBO > Add UnionMerge rule on cbo path > --- > > Key: HIVE-9318 > URL: https://issues.apache.org/jira/browse/HIVE-9318 > Project: Hive > Issue Type: Improvement > Components: CBO, Logical Optimizer >Affects Versions: 0.14.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-9318.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9318) Add UnionMerge rule on cbo path
[ https://issues.apache.org/jira/browse/HIVE-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-9318: --- Attachment: HIVE-9318.patch If there are subsequent unions, we can merge them. > Add UnionMerge rule on cbo path > --- > > Key: HIVE-9318 > URL: https://issues.apache.org/jira/browse/HIVE-9318 > Project: Hive > Issue Type: Improvement >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-9318.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9306) Let Context.isLocalOnlyExecutionMode() return false if execution engine is Spark [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-9306: -- Resolution: Fixed Fix Version/s: spark-branch Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks to Szehon for the review. > Let Context.isLocalOnlyExecutionMode() return false if execution engine is > Spark [Spark Branch] > --- > > Key: HIVE-9306 > URL: https://issues.apache.org/jira/browse/HIVE-9306 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Xuefu Zhang >Assignee: Xuefu Zhang > Fix For: spark-branch > > Attachments: HIVE-9306.1-spark.patch, HIVE-9306.2-spark.patch, > HIVE-9306.3-spark.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9306) Let Context.isLocalOnlyExecutionMode() return false if execution engine is Spark [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270202#comment-14270202 ] Xuefu Zhang commented on HIVE-9306: --- Test failure above, org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23.q, doesn't seem related to the patch. It didn't happen in previous run, and neither in my local run. > Let Context.isLocalOnlyExecutionMode() return false if execution engine is > Spark [Spark Branch] > --- > > Key: HIVE-9306 > URL: https://issues.apache.org/jira/browse/HIVE-9306 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Xuefu Zhang >Assignee: Xuefu Zhang > Attachments: HIVE-9306.1-spark.patch, HIVE-9306.2-spark.patch, > HIVE-9306.3-spark.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9296) Need to add schema upgrade changes for queueing events in the database
[ https://issues.apache.org/jira/browse/HIVE-9296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270198#comment-14270198 ] Sushanth Sowmyan commented on HIVE-9296: +1. Thanks, Alan! > Need to add schema upgrade changes for queueing events in the database > -- > > Key: HIVE-9296 > URL: https://issues.apache.org/jira/browse/HIVE-9296 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.15.0 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.15.0 > > Attachments: HIVE-9296.patch > > > HIVE-9174 added the ability to queue notification events in the database, but > did not include the schema upgrade scripts. > Also, in the thrift changes the convention was not followed properly in > naming the thrift methods. HIVE-9174 used camel case, where the thrift > methods use all lower case separated by underscores. > Both of these issues should be fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-9318) Add UnionMerge rule on cbo path
[ https://issues.apache.org/jira/browse/HIVE-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan reassigned HIVE-9318: -- Assignee: Ashutosh Chauhan > Add UnionMerge rule on cbo path > --- > > Key: HIVE-9318 > URL: https://issues.apache.org/jira/browse/HIVE-9318 > Project: Hive > Issue Type: Improvement >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9318) Add UnionMerge rule on cbo path
Ashutosh Chauhan created HIVE-9318: -- Summary: Add UnionMerge rule on cbo path Key: HIVE-9318 URL: https://issues.apache.org/jira/browse/HIVE-9318 Project: Hive Issue Type: Improvement Reporter: Ashutosh Chauhan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9317) move Microsoft copyright to NOTICE file
Owen O'Malley created HIVE-9317:
---
Summary: move Microsoft copyright to NOTICE file
Key: HIVE-9317
URL: https://issues.apache.org/jira/browse/HIVE-9317
Project: Hive
Issue Type: Bug
Reporter: Owen O'Malley
Fix For: 0.15.0
There are a set of files that still have the Microsoft copyright notices. Those
notices need to be moved into NOTICES and replaced with the standard Apache
headers.
{code}
./common/src/java/org/apache/hadoop/hive/common/type/Decimal128.java
./common/src/java/org/apache/hadoop/hive/common/type/SignedInt128.java
./common/src/java/org/apache/hadoop/hive/common/type/SqlMathUtil.java
./common/src/java/org/apache/hadoop/hive/common/type/UnsignedInt128.java
./common/src/test/org/apache/hadoop/hive/common/type/TestDecimal128.java
./common/src/test/org/apache/hadoop/hive/common/type/TestSignedInt128.java
./common/src/test/org/apache/hadoop/hive/common/type/TestSqlMathUtil.java
./common/src/test/org/apache/hadoop/hive/common/type/TestUnsignedInt128.java
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9304) [Refactor] remove unused method in SemAly
[
https://issues.apache.org/jira/browse/HIVE-9304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270175#comment-14270175
]
Vikram Dixit K commented on HIVE-9304:
--
+1 LGTM.
> [Refactor] remove unused method in SemAly
> -
>
> Key: HIVE-9304
> URL: https://issues.apache.org/jira/browse/HIVE-9304
> Project: Hive
> Issue Type: Task
> Components: Query Processor
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
>Priority: Trivial
> Attachments: HIVE-9304.patch
>
>
> Seems like method {{genConversionOps}} don't serve any purpose any longer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9304) [Refactor] remove unused method in SemAly
[
https://issues.apache.org/jira/browse/HIVE-9304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-9304:
---
Priority: Trivial (was: Major)
> [Refactor] remove unused method in SemAly
> -
>
> Key: HIVE-9304
> URL: https://issues.apache.org/jira/browse/HIVE-9304
> Project: Hive
> Issue Type: Task
> Components: Query Processor
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
>Priority: Trivial
> Attachments: HIVE-9304.patch
>
>
> Seems like method {{genConversionOps}} don't serve any purpose any longer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9306) Let Context.isLocalOnlyExecutionMode() return false if execution engine is Spark [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270170#comment-14270170
]
Hive QA commented on HIVE-9306:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690946/HIVE-9306.3-spark.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 7285 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_windowing
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchEmptyCommit
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/620/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/620/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-620/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690946 - PreCommit-HIVE-SPARK-Build
> Let Context.isLocalOnlyExecutionMode() return false if execution engine is
> Spark [Spark Branch]
> ---
>
> Key: HIVE-9306
> URL: https://issues.apache.org/jira/browse/HIVE-9306
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-9306.1-spark.patch, HIVE-9306.2-spark.patch,
> HIVE-9306.3-spark.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9316) TestSqoop tests in WebHCat testsuite hardcode libdir path to hdfs
[ https://issues.apache.org/jira/browse/HIVE-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepesh Khandelwal updated HIVE-9316: - Status: Patch Available (was: Open) > TestSqoop tests in WebHCat testsuite hardcode libdir path to hdfs > - > > Key: HIVE-9316 > URL: https://issues.apache.org/jira/browse/HIVE-9316 > Project: Hive > Issue Type: Bug > Components: Tests, WebHCat >Reporter: Deepesh Khandelwal >Assignee: Deepesh Khandelwal >Priority: Minor > Fix For: 0.15.0 > > Attachments: HIVE-9316.1.patch > > > Currently the TestSqoop tests in WebHCat Perl based testsuite has hdfs:// > prefix in the jdbc jar path in libdir, we should remove this to enable it to > run against other file systems. > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9217) UnionProcessor misses results for multi-insert when hive.optimize.union.remove=true
[ https://issues.apache.org/jira/browse/HIVE-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270161#comment-14270161 ] Laljo John Pullokkaran commented on HIVE-9217: -- Committed to trunk; thanks [~pxiong] > UnionProcessor misses results for multi-insert when > hive.optimize.union.remove=true > --- > > Key: HIVE-9217 > URL: https://issues.apache.org/jira/browse/HIVE-9217 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-9217.01.patch, HIVE-9217.02.patch, > HIVE-9217.03.patch > > > To reproduce: > In union_remove_6.q, just change > FROM ( > SELECT key, count(1) as values from inputTbl1 group by key > UNION ALL > SELECT key, count(1) as values from inputTbl1 group by key > ) a > insert overwrite table outputTbl1 select * > insert overwrite table outputTbl2 select *; > to > FROM ( > select * from( > SELECT key, count(1) as values from inputTbl1 group by key > UNION ALL > SELECT key, count(1) as values from inputTbl1 group by key > )subq > ) a > insert overwrite table outputTbl1 select * > insert overwrite table outputTbl2 select *; > select * from outputtbl2 will output nothing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9217) UnionProcessor misses results for multi-insert when hive.optimize.union.remove=true
[ https://issues.apache.org/jira/browse/HIVE-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-9217: - Resolution: Fixed Status: Resolved (was: Patch Available) > UnionProcessor misses results for multi-insert when > hive.optimize.union.remove=true > --- > > Key: HIVE-9217 > URL: https://issues.apache.org/jira/browse/HIVE-9217 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-9217.01.patch, HIVE-9217.02.patch, > HIVE-9217.03.patch > > > To reproduce: > In union_remove_6.q, just change > FROM ( > SELECT key, count(1) as values from inputTbl1 group by key > UNION ALL > SELECT key, count(1) as values from inputTbl1 group by key > ) a > insert overwrite table outputTbl1 select * > insert overwrite table outputTbl2 select *; > to > FROM ( > select * from( > SELECT key, count(1) as values from inputTbl1 group by key > UNION ALL > SELECT key, count(1) as values from inputTbl1 group by key > )subq > ) a > insert overwrite table outputTbl1 select * > insert overwrite table outputTbl2 select *; > select * from outputtbl2 will output nothing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9304) [Refactor] remove unused method in SemAly
[
https://issues.apache.org/jira/browse/HIVE-9304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270152#comment-14270152
]
Hive QA commented on HIVE-9304:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690693/HIVE-9304.patch
{color:green}SUCCESS:{color} +1 6733 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2295/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2295/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2295/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690693 - PreCommit-HIVE-TRUNK-Build
> [Refactor] remove unused method in SemAly
> -
>
> Key: HIVE-9304
> URL: https://issues.apache.org/jira/browse/HIVE-9304
> Project: Hive
> Issue Type: Task
> Components: Query Processor
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-9304.patch
>
>
> Seems like method {{genConversionOps}} don't serve any purpose any longer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9316) TestSqoop tests in WebHCat testsuite hardcode libdir path to hdfs
[ https://issues.apache.org/jira/browse/HIVE-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepesh Khandelwal updated HIVE-9316: - Attachment: HIVE-9316.1.patch Attaching the patch that removes the prefix. Please review. > TestSqoop tests in WebHCat testsuite hardcode libdir path to hdfs > - > > Key: HIVE-9316 > URL: https://issues.apache.org/jira/browse/HIVE-9316 > Project: Hive > Issue Type: Bug > Components: Tests, WebHCat >Reporter: Deepesh Khandelwal >Assignee: Deepesh Khandelwal >Priority: Minor > Fix For: 0.15.0 > > Attachments: HIVE-9316.1.patch > > > Currently the TestSqoop tests in WebHCat Perl based testsuite has hdfs:// > prefix in the jdbc jar path in libdir, we should remove this to enable it to > run against other file systems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9316) TestSqoop tests in WebHCat testsuite hardcode libdir path to hdfs
[ https://issues.apache.org/jira/browse/HIVE-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepesh Khandelwal updated HIVE-9316: - Description: Currently the TestSqoop tests in WebHCat Perl based testsuite has hdfs:// prefix in the jdbc jar path in libdir, we should remove this to enable it to run against other file systems. NO PRECOMMIT TESTS was:Currently the TestSqoop tests in WebHCat Perl based testsuite has hdfs:// prefix in the jdbc jar path in libdir, we should remove this to enable it to run against other file systems. > TestSqoop tests in WebHCat testsuite hardcode libdir path to hdfs > - > > Key: HIVE-9316 > URL: https://issues.apache.org/jira/browse/HIVE-9316 > Project: Hive > Issue Type: Bug > Components: Tests, WebHCat >Reporter: Deepesh Khandelwal >Assignee: Deepesh Khandelwal >Priority: Minor > Fix For: 0.15.0 > > Attachments: HIVE-9316.1.patch > > > Currently the TestSqoop tests in WebHCat Perl based testsuite has hdfs:// > prefix in the jdbc jar path in libdir, we should remove this to enable it to > run against other file systems. > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9316) TestSqoop tests in WebHCat testsuite hardcode libdir path to hdfs
Deepesh Khandelwal created HIVE-9316: Summary: TestSqoop tests in WebHCat testsuite hardcode libdir path to hdfs Key: HIVE-9316 URL: https://issues.apache.org/jira/browse/HIVE-9316 Project: Hive Issue Type: Bug Components: Tests, WebHCat Reporter: Deepesh Khandelwal Assignee: Deepesh Khandelwal Priority: Minor Fix For: 0.15.0 Currently the TestSqoop tests in WebHCat Perl based testsuite has hdfs:// prefix in the jdbc jar path in libdir, we should remove this to enable it to run against other file systems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8988) Support advanced aggregation in Hive to Calcite path
[ https://issues.apache.org/jira/browse/HIVE-8988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-8988: - Resolution: Fixed Status: Resolved (was: Patch Available) > Support advanced aggregation in Hive to Calcite path > - > > Key: HIVE-8988 > URL: https://issues.apache.org/jira/browse/HIVE-8988 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.15.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Labels: grouping, logical, optiq > Fix For: 0.15.0 > > Attachments: HIVE-8988.01.patch, HIVE-8988.02.patch, > HIVE-8988.03.patch, HIVE-8988.04.patch, HIVE-8988.05.patch, > HIVE-8988.05.patch, HIVE-8988.06.patch, HIVE-8988.07.patch, HIVE-8988.patch > > > CLEAR LIBRARY CACHE > To close the gap between Hive and Calcite, we need to support the translation > of GroupingSets into Calcite; currently this is not implemented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8988) Support advanced aggregation in Hive to Calcite path
[ https://issues.apache.org/jira/browse/HIVE-8988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270133#comment-14270133 ] Laljo John Pullokkaran commented on HIVE-8988: -- Committed to trunk, thanks [~jcamachorodriguez] > Support advanced aggregation in Hive to Calcite path > - > > Key: HIVE-8988 > URL: https://issues.apache.org/jira/browse/HIVE-8988 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.15.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Labels: grouping, logical, optiq > Fix For: 0.15.0 > > Attachments: HIVE-8988.01.patch, HIVE-8988.02.patch, > HIVE-8988.03.patch, HIVE-8988.04.patch, HIVE-8988.05.patch, > HIVE-8988.05.patch, HIVE-8988.06.patch, HIVE-8988.07.patch, HIVE-8988.patch > > > CLEAR LIBRARY CACHE > To close the gap between Hive and Calcite, we need to support the translation > of GroupingSets into Calcite; currently this is not implemented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8327) mvn site -Pfindbugs
[ https://issues.apache.org/jira/browse/HIVE-8327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270123#comment-14270123 ] Prasanth Jayachandran commented on HIVE-8327: - We have been bitten by this in other cases as well. I remember LazyMap implementing equals() but not hashCode() causing issues 5 years later. Its definitely good to catch such cases before committing patches. 4 min spent in QA better than hours later :) > mvn site -Pfindbugs > --- > > Key: HIVE-8327 > URL: https://issues.apache.org/jira/browse/HIVE-8327 > Project: Hive > Issue Type: Test > Components: Diagnosability >Reporter: Gopal V >Assignee: Gopal V > Fix For: 0.15.0 > > Attachments: HIVE-8327.1.patch, HIVE-8327.2.patch, ql-findbugs.html > > > HIVE-3099 originally added findbugs into the old ant build. > Get basic findbugs working for the maven build. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9249) java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to org.apache.hadoop.hive.common.type.HiveVarchar when joining tables
[
https://issues.apache.org/jira/browse/HIVE-9249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270119#comment-14270119
]
Gopal V commented on HIVE-9249:
---
Committed to 0.15.0.
Not committed to 0.14.1 because of plan formatting changes in the golden files.
{code}
clientpositive/vector_varchar_mapjoin1.q.out
148,150d147
< condition expressions:
< 0 {c1}
< 1 {c1} {c2}
167,169d163
< condition expressions:
< 0 {c1} {c2}
< 1 {c1} {c2}
244,246d237
< condition expressions:
< 0 {c1}
< 1 {c1} {c2}
263,265d253
< condition expressions:
< 0 {c1} {c2}
< 1 {c1} {c2}
342,344d329
< condition expressions:
< 0 {c1} {c2}
< 1 {c1}
361,363d345
< condition expressions:
< 0 {c1} {c2}
< 1 {c1} {c2}
{code}
Will generate a new patch, in a few mins.
> java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
> org.apache.hadoop.hive.common.type.HiveVarchar when joining tables
> ---
>
> Key: HIVE-9249
> URL: https://issues.apache.org/jira/browse/HIVE-9249
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 0.14.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 0.15.0
>
> Attachments: HIVE-9249.01.patch
>
>
> VectorColumnAssignFactory doesn't handle HiveCharWritable /
> HiveVarcharWritable objects.
> Either:
> HiveVarcharWritable cannot be cast to ... HiveVarchar
> or
> HiveCharWritable cannot be cast to ... HiveChar
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
> org.apache.hadoop.hive.common.type.HiveVarchar
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorColumnAssignFactory$17.assignObjectValue(VectorColumnAssignFactory.java:417)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.internalForward(VectorMapJoinOperator.java:196)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:670)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:748)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:299)
> ... 24 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
