[jira] [Commented] (HIVE-4790) MapredLocalTask task does not make virtual columns
[
https://issues.apache.org/jira/browse/HIVE-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272343#comment-14272343
]
Hive QA commented on HIVE-4790:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12691034/HIVE-4790.14.patch.txt
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6747 tests executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2314/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2314/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2314/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12691034 - PreCommit-HIVE-TRUNK-Build
MapredLocalTask task does not make virtual columns
--
Key: HIVE-4790
URL: https://issues.apache.org/jira/browse/HIVE-4790
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
Attachments: D11511.3.patch, D11511.4.patch, HIVE-4790.10.patch.txt,
HIVE-4790.11.patch.txt, HIVE-4790.12.patch.txt, HIVE-4790.13.patch.txt,
HIVE-4790.14.patch.txt, HIVE-4790.5.patch.txt, HIVE-4790.6.patch.txt,
HIVE-4790.7.patch.txt, HIVE-4790.8.patch.txt, HIVE-4790.9.patch.txt,
HIVE-4790.D11511.1.patch, HIVE-4790.D11511.2.patch
From mailing list,
http://www.mail-archive.com/user@hive.apache.org/msg08264.html
{noformat}
SELECT *,b.BLOCK__OFFSET__INSIDE__FILE FROM a JOIN b ON
b.rownumber = a.number;
fails with this error:
SELECT *,b.BLOCK__OFFSET__INSIDE__FILE FROM a JOIN b ON b.rownumber =
a.number;
Automatically selecting local only mode for query
Total MapReduce jobs = 1
setting HADOOP_USER_NAMEpmarron
13/06/25 10:52:56 WARN conf.HiveConf: DEPRECATED: Configuration property
hive.metastore.local no longer has any effect. Make sure to provide a valid
value for hive.metastore.uris if you are connecting to a remote metastore.
Execution log at: /tmp/pmarron/.log
2013-06-25 10:52:56 Starting to launch local task to process map join;
maximum memory = 932118528
java.lang.RuntimeException: cannot find field block__offset__inside__file
from [0:rownumber, 1:offset]
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:366)
at
org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.getStructFieldRef(LazySimpleStructObjectInspector.java:168)
at
org.apache.hadoop.hive.serde2.objectinspector.DelegatedStructObjectInspector.getStructFieldRef(DelegatedStructObjectInspector.java:74)
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
at
org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:68)
at
org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:222)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:451)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:407)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:186)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at
org.apache.hadoop.hive.ql.exec.MapredLocalTask.initializeOperators(MapredLocalTask.java:394)
at
org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:277)
at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:676)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Execution failed with exit status: 2
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7550) Extend cached evaluation to multiple expressions
[
https://issues.apache.org/jira/browse/HIVE-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272318#comment-14272318
]
Hive QA commented on HIVE-7550:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12691035/HIVE-7550.2.patch.txt
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6747 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2313/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2313/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2313/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12691035 - PreCommit-HIVE-TRUNK-Build
Extend cached evaluation to multiple expressions
Key: HIVE-7550
URL: https://issues.apache.org/jira/browse/HIVE-7550
Project: Hive
Issue Type: Improvement
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-7550.1.patch.txt, HIVE-7550.2.patch.txt
Currently, hive.cache.expr.evaluation caches per expression. But cache
context might be shared for multiple expressions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9104) windowing.q failed when mapred.reduce.tasks is set to larger than one
[
https://issues.apache.org/jira/browse/HIVE-9104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272334#comment-14272334
]
Chao commented on HIVE-9104:
[~xuefuz] OK, will do.
windowing.q failed when mapred.reduce.tasks is set to larger than one
-
Key: HIVE-9104
URL: https://issues.apache.org/jira/browse/HIVE-9104
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Chao
Assignee: Chao
Attachments: HIVE-9104.patch
Test {{windowing.q}} is actually not enabled in Spark branch - in test
configurations it is {{windowing.q.q}}.
I just run this test, and query
{code}
-- 12. testFirstLastWithWhere
select p_mfgr,p_name, p_size,
rank() over(distribute by p_mfgr sort by p_name) as r,
sum(p_size) over (distribute by p_mfgr sort by p_name rows between current
row and current row) as s2,
first_value(p_size) over w1 as f,
last_value(p_size, false) over w1 as l
from part
where p_mfgr = 'Manufacturer#3'
window w1 as (distribute by p_mfgr sort by p_name rows between 2 preceding
and 2 following);
{code}
failed with the following exception:
{noformat}
java.lang.RuntimeException: Hive Runtime Error while closing operators: null
at
org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.close(SparkReduceRecordHandler.java:446)
at
org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunctionResultList.closeRecordProcessor(HiveReduceFunctionResultList.java:58)
at
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:108)
at
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$2.apply(AsyncRDDActions.scala:115)
at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$2.apply(AsyncRDDActions.scala:115)
at org.apache.spark.SparkContext$$anonfun$30.apply(SparkContext.scala:1390)
at org.apache.spark.SparkContext$$anonfun$30.apply(SparkContext.scala:1390)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.NoSuchElementException
at java.util.ArrayDeque.getFirst(ArrayDeque.java:318)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFFirstValue$FirstValStreamingFixedWindow.terminate(GenericUDAFFirstValue.java:290)
at
org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.finishPartition(WindowingTableFunction.java:413)
at
org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:337)
at org.apache.hadoop.hive.ql.exec.PTFOperator.closeOp(PTFOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:598)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
at
org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.close(SparkReduceRecordHandler.java:431)
... 15 more
{noformat}
We need to find out:
- Since which commit this test started failing, and
- Why it fails
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 28964: HIVE-8121 Create micro-benchmarks for ParquetSerde and evaluate performance
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28964/#review67581 --- Thank you for your patch! I have a general question about the current solution. Can we add more dimensions to show the performance like throughput, average time? Besides that, I leave some minor issues open. See http://hg.openjdk.java.net/code-tools/jmh/file/1ba1bb91b83c/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_02_BenchmarkModes.java itests/hive-jmh/src/main/java/org/apache/hive/benchmark/storage/ColumnarStorageBench.java https://reviews.apache.org/r/28964/#comment111619 Please only import the packages you really needed. itests/hive-jmh/src/main/java/org/apache/hive/benchmark/storage/ColumnarStorageBench.java https://reviews.apache.org/r/28964/#comment111621 The same as above. itests/hive-jmh/src/main/java/org/apache/hive/benchmark/storage/ColumnarStorageBench.java https://reviews.apache.org/r/28964/#comment111622 The same as above. itests/hive-jmh/src/main/java/org/apache/hive/benchmark/storage/ColumnarStorageBench.java https://reviews.apache.org/r/28964/#comment111623 Can you sort the import statements in alphabetical order? itests/hive-jmh/src/main/java/org/apache/hive/benchmark/storage/ColumnarStorageBench.java https://reviews.apache.org/r/28964/#comment111624 Add static here for final variable. itests/hive-jmh/src/main/java/org/apache/hive/benchmark/storage/ColumnarStorageBench.java https://reviews.apache.org/r/28964/#comment111625 Please add space before and after less-than sign. itests/hive-jmh/src/main/java/org/apache/hive/benchmark/storage/ColumnarStorageBench.java https://reviews.apache.org/r/28964/#comment111626 Can we add some space here before and after the definition for public method and some annotations for them? itests/hive-jmh/src/main/java/org/apache/hive/benchmark/storage/ColumnarStorageBench.java https://reviews.apache.org/r/28964/#comment111627 No need to break line here. - cheng xu On Jan. 9, 2015, 6:38 p.m., Sergio Pena wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28964/ --- (Updated Jan. 9, 2015, 6:38 p.m.) Review request for hive, Brock Noland and cheng xu. Bugs: HIVE-8121 https://issues.apache.org/jira/browse/HIVE-8121 Repository: hive-git Description --- This is a new tool used to test ORC PARQUET file format performance. Diffs - itests/hive-jmh/pom.xml PRE-CREATION itests/hive-jmh/src/main/java/org/apache/hive/benchmark/storage/ColumnarStorageBench.java PRE-CREATION itests/pom.xml 0a154d6eb8c119e4e6419777c28b59b9d2108ba0 Diff: https://reviews.apache.org/r/28964/diff/ Testing --- Thanks, Sergio Pena
[jira] [Commented] (HIVE-9104) windowing.q failed when mapred.reduce.tasks is set to larger than one
[
https://issues.apache.org/jira/browse/HIVE-9104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272187#comment-14272187
]
Xuefu Zhang commented on HIVE-9104:
---
+1. Code looks reasonable to me. However, it's great if [~rhbutani] or someone
else familiar to this part of code to take a look.
windowing.q failed when mapred.reduce.tasks is set to larger than one
-
Key: HIVE-9104
URL: https://issues.apache.org/jira/browse/HIVE-9104
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Chao
Assignee: Chao
Attachments: HIVE-9104.patch
Test {{windowing.q}} is actually not enabled in Spark branch - in test
configurations it is {{windowing.q.q}}.
I just run this test, and query
{code}
-- 12. testFirstLastWithWhere
select p_mfgr,p_name, p_size,
rank() over(distribute by p_mfgr sort by p_name) as r,
sum(p_size) over (distribute by p_mfgr sort by p_name rows between current
row and current row) as s2,
first_value(p_size) over w1 as f,
last_value(p_size, false) over w1 as l
from part
where p_mfgr = 'Manufacturer#3'
window w1 as (distribute by p_mfgr sort by p_name rows between 2 preceding
and 2 following);
{code}
failed with the following exception:
{noformat}
java.lang.RuntimeException: Hive Runtime Error while closing operators: null
at
org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.close(SparkReduceRecordHandler.java:446)
at
org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunctionResultList.closeRecordProcessor(HiveReduceFunctionResultList.java:58)
at
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:108)
at
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$2.apply(AsyncRDDActions.scala:115)
at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$2.apply(AsyncRDDActions.scala:115)
at org.apache.spark.SparkContext$$anonfun$30.apply(SparkContext.scala:1390)
at org.apache.spark.SparkContext$$anonfun$30.apply(SparkContext.scala:1390)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.NoSuchElementException
at java.util.ArrayDeque.getFirst(ArrayDeque.java:318)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFFirstValue$FirstValStreamingFixedWindow.terminate(GenericUDAFFirstValue.java:290)
at
org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.finishPartition(WindowingTableFunction.java:413)
at
org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:337)
at org.apache.hadoop.hive.ql.exec.PTFOperator.closeOp(PTFOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:598)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
at
org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.close(SparkReduceRecordHandler.java:431)
... 15 more
{noformat}
We need to find out:
- Since which commit this test started failing, and
- Why it fails
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9194) Support select distinct *
[ https://issues.apache.org/jira/browse/HIVE-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-9194: -- Status: Open (was: Patch Available) Support select distinct * - Key: HIVE-9194 URL: https://issues.apache.org/jira/browse/HIVE-9194 Project: Hive Issue Type: Sub-task Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-9194.00.patch As per [~jpullokkaran]'s review comments, implement select distinct * -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29671: Support select distinct *
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29671/#review67566 --- ql/src/test/queries/clientpositive/selectDistinctStar.q https://reviews.apache.org/r/29671/#comment111603 Add more test cases to cover more operators: ex: 1. select distinct * from src union all select distinct * from src1 2. select distinct * from src join src1 on src.key=src1.key 3. select * from (select distinct * from src)src1 join (select distinct * from src)src2 on src1.key=src2.key Also could you add tests to negative test driver - John Pullokkaran On Jan. 7, 2015, 9:16 p.m., pengcheng xiong wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29671/ --- (Updated Jan. 7, 2015, 9:16 p.m.) Review request for hive and John Pullokkaran. Repository: hive-git Description --- Support select distinct * in operator genaration phase. Diffs - itests/src/test/resources/testconfiguration.properties 5d07fba ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 306edf6 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c2d5c8c ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java 57868b7 ql/src/java/org/apache/hadoop/hive/ql/parse/UnparseTranslator.java 9ad6714 ql/src/test/queries/clientpositive/selectDistinctStar.q PRE-CREATION ql/src/test/results/clientpositive/selectDistinctStar.q.out PRE-CREATION ql/src/test/results/clientpositive/tez/selectDistinctStar.q.out PRE-CREATION Diff: https://reviews.apache.org/r/29671/diff/ Testing --- Thanks, pengcheng xiong
[jira] [Updated] (HIVE-9321) Notification message size can be arbitrarily long, DbNotificationListener limits to 1024
[ https://issues.apache.org/jira/browse/HIVE-9321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-9321: - Attachment: HIVE-9321.patch This patch moves MESSAGE field from VARCHAR to TEXT/CLOB. Notification message size can be arbitrarily long, DbNotificationListener limits to 1024 Key: HIVE-9321 URL: https://issues.apache.org/jira/browse/HIVE-9321 Project: Hive Issue Type: Bug Affects Versions: 0.15.0 Reporter: Sushanth Sowmyan Assignee: Alan Gates Fix For: 0.15.0 Attachments: HIVE-9321.patch As part of HIVE-9174, a new notification listener, DbNotificationListener was introduced that stores notification messages in the metastore. Now, the message field of a Notification event can be arbitrarily long, since it is a JSON-serialized message of what event actually occurred, and an AddPartitionMessage which captured the result of an add_partitions can be arbitrarily long (as long as the number of partitions that were added). However, the message field for the storage in db is limited to VARCHAR(1024), which it will easily break. We need to fix this somehow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3405) UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase
[
https://issues.apache.org/jira/browse/HIVE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272180#comment-14272180
]
Hive QA commented on HIVE-3405:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12691027/HIVE-3405.5.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6741 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2312/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2312/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2312/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12691027 - PreCommit-HIVE-TRUNK-Build
UDF initcap to obtain a string with the first letter of each word in
uppercase other letters in lowercase
-
Key: HIVE-3405
URL: https://issues.apache.org/jira/browse/HIVE-3405
Project: Hive
Issue Type: New Feature
Components: UDF
Affects Versions: 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.11.0, 0.13.0, 0.14.0,
0.15.0, 0.14.1
Reporter: Archana Nair
Assignee: Alexander Pivovarov
Labels: patch
Attachments: HIVE-3405.1.patch.txt, HIVE-3405.2.patch,
HIVE-3405.3.patch, HIVE-3405.4.patch, HIVE-3405.5.patch, HIVE-3405.5.patch
Hive current releases lacks a INITCAP function which returns String with
first letter of the word in uppercase.INITCAP returns String, with the first
letter of each word in uppercase, all other letters in same case. Words are
delimited by white space.This will be useful report generation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9194) Support select distinct *
[ https://issues.apache.org/jira/browse/HIVE-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-9194: -- Status: Patch Available (was: Open) Support select distinct * - Key: HIVE-9194 URL: https://issues.apache.org/jira/browse/HIVE-9194 Project: Hive Issue Type: Sub-task Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-9194.00.patch, HIVE-9194.01.patch As per [~jpullokkaran]'s review comments, implement select distinct * -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29671: Support select distinct *
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29671/ --- (Updated Jan. 10, 2015, 4:02 a.m.) Review request for hive and John Pullokkaran. Changes --- Address John's comments. (1) add more test cases including both pos and neg tests. (2) remove spaces in code. Repository: hive-git Description --- Support select distinct * in operator genaration phase. Diffs (updated) - itests/src/test/resources/testconfiguration.properties 68e9b7d ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 3534551 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 971910a ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java 57868b7 ql/src/java/org/apache/hadoop/hive/ql/parse/UnparseTranslator.java 9ad6714 ql/src/test/queries/clientnegative/selectDistinctStarNeg_1.q PRE-CREATION ql/src/test/queries/clientnegative/selectDistinctStarNeg_2.q PRE-CREATION ql/src/test/queries/clientpositive/selectDistinctStar.q PRE-CREATION ql/src/test/results/clientnegative/selectDistinctStarNeg_1.q.out PRE-CREATION ql/src/test/results/clientnegative/selectDistinctStarNeg_2.q.out PRE-CREATION ql/src/test/results/clientpositive/selectDistinctStar.q.out PRE-CREATION ql/src/test/results/clientpositive/tez/selectDistinctStar.q.out PRE-CREATION Diff: https://reviews.apache.org/r/29671/diff/ Testing --- Thanks, pengcheng xiong
[jira] [Updated] (HIVE-9194) Support select distinct *
[ https://issues.apache.org/jira/browse/HIVE-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-9194: -- Attachment: HIVE-9194.01.patch Address [~jpullokkaran]'s comments. (1) add more test cases including both pos and neg tests. (2) remove spaces in code. Support select distinct * - Key: HIVE-9194 URL: https://issues.apache.org/jira/browse/HIVE-9194 Project: Hive Issue Type: Sub-task Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-9194.00.patch, HIVE-9194.01.patch As per [~jpullokkaran]'s review comments, implement select distinct * -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8327) mvn site -Pfindbugs
[ https://issues.apache.org/jira/browse/HIVE-8327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272349#comment-14272349 ] Lefty Leverenz commented on HIVE-8327: -- Should this be documented somewhere in the wiki? * [Hive Testing Docs | https://cwiki.apache.org/confluence/display/Hive/TestingDocs] mvn site -Pfindbugs --- Key: HIVE-8327 URL: https://issues.apache.org/jira/browse/HIVE-8327 Project: Hive Issue Type: Test Components: Diagnosability Reporter: Gopal V Assignee: Gopal V Fix For: 0.15.0 Attachments: HIVE-8327.1.patch, HIVE-8327.2.patch, ql-findbugs.html HIVE-3099 originally added findbugs into the old ant build. Get basic findbugs working for the maven build. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9104) windowing.q failed when mapred.reduce.tasks is set to larger than one
[
https://issues.apache.org/jira/browse/HIVE-9104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272290#comment-14272290
]
Harish Butani commented on HIVE-9104:
-
+1, thanks for tracking this down.
windowing.q failed when mapred.reduce.tasks is set to larger than one
-
Key: HIVE-9104
URL: https://issues.apache.org/jira/browse/HIVE-9104
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Chao
Assignee: Chao
Attachments: HIVE-9104.patch
Test {{windowing.q}} is actually not enabled in Spark branch - in test
configurations it is {{windowing.q.q}}.
I just run this test, and query
{code}
-- 12. testFirstLastWithWhere
select p_mfgr,p_name, p_size,
rank() over(distribute by p_mfgr sort by p_name) as r,
sum(p_size) over (distribute by p_mfgr sort by p_name rows between current
row and current row) as s2,
first_value(p_size) over w1 as f,
last_value(p_size, false) over w1 as l
from part
where p_mfgr = 'Manufacturer#3'
window w1 as (distribute by p_mfgr sort by p_name rows between 2 preceding
and 2 following);
{code}
failed with the following exception:
{noformat}
java.lang.RuntimeException: Hive Runtime Error while closing operators: null
at
org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.close(SparkReduceRecordHandler.java:446)
at
org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunctionResultList.closeRecordProcessor(HiveReduceFunctionResultList.java:58)
at
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:108)
at
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$2.apply(AsyncRDDActions.scala:115)
at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$2.apply(AsyncRDDActions.scala:115)
at org.apache.spark.SparkContext$$anonfun$30.apply(SparkContext.scala:1390)
at org.apache.spark.SparkContext$$anonfun$30.apply(SparkContext.scala:1390)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.NoSuchElementException
at java.util.ArrayDeque.getFirst(ArrayDeque.java:318)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFFirstValue$FirstValStreamingFixedWindow.terminate(GenericUDAFFirstValue.java:290)
at
org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.finishPartition(WindowingTableFunction.java:413)
at
org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:337)
at org.apache.hadoop.hive.ql.exec.PTFOperator.closeOp(PTFOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:598)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
at
org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.close(SparkReduceRecordHandler.java:431)
... 15 more
{noformat}
We need to find out:
- Since which commit this test started failing, and
- Why it fails
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Move ancient Hive issues from Hadoop project to Hive
Hi all, Hive started out as Hadoop subproject. That time Hadoop's jira is used to track Hive's bugs and features. As I try to find lineage of some very old code in Hive, I sometimes end up on those jiras. It will be nice to move those issues from Hadoop to Hive so that its easy to search as all jiras relevant to Hive is contained in one project. A representative list is : *http://s.apache.org/Hive-issues-in-Hadoop http://s.apache.org/Hive-issues-in-Hadoop* Unless some one objects, I will start to move those issues to Hive some time over next week. Thanks, Ashutosh
[jira] [Commented] (HIVE-9315) CBO (Calcite Return Path): Inline FileSinkOperator, Properties
[
https://issues.apache.org/jira/browse/HIVE-9315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272366#comment-14272366
]
Hive QA commented on HIVE-9315:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12691116/HIVE-9315.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6747 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2315/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2315/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2315/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12691116 - PreCommit-HIVE-TRUNK-Build
CBO (Calcite Return Path): Inline FileSinkOperator, Properties
--
Key: HIVE-9315
URL: https://issues.apache.org/jira/browse/HIVE-9315
Project: Hive
Issue Type: Sub-task
Components: CBO
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez
Fix For: 0.15.0
Attachments: HIVE-9315.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9104) windowing.q failed when mapred.reduce.tasks is set to larger than one
[
https://issues.apache.org/jira/browse/HIVE-9104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao updated HIVE-9104:
---
Attachment: HIVE-9104.2.patch
windowing.q failed when mapred.reduce.tasks is set to larger than one
-
Key: HIVE-9104
URL: https://issues.apache.org/jira/browse/HIVE-9104
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Chao
Assignee: Chao
Attachments: HIVE-9104.2.patch, HIVE-9104.patch
Test {{windowing.q}} is actually not enabled in Spark branch - in test
configurations it is {{windowing.q.q}}.
I just run this test, and query
{code}
-- 12. testFirstLastWithWhere
select p_mfgr,p_name, p_size,
rank() over(distribute by p_mfgr sort by p_name) as r,
sum(p_size) over (distribute by p_mfgr sort by p_name rows between current
row and current row) as s2,
first_value(p_size) over w1 as f,
last_value(p_size, false) over w1 as l
from part
where p_mfgr = 'Manufacturer#3'
window w1 as (distribute by p_mfgr sort by p_name rows between 2 preceding
and 2 following);
{code}
failed with the following exception:
{noformat}
java.lang.RuntimeException: Hive Runtime Error while closing operators: null
at
org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.close(SparkReduceRecordHandler.java:446)
at
org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunctionResultList.closeRecordProcessor(HiveReduceFunctionResultList.java:58)
at
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:108)
at
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$2.apply(AsyncRDDActions.scala:115)
at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$2.apply(AsyncRDDActions.scala:115)
at org.apache.spark.SparkContext$$anonfun$30.apply(SparkContext.scala:1390)
at org.apache.spark.SparkContext$$anonfun$30.apply(SparkContext.scala:1390)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.NoSuchElementException
at java.util.ArrayDeque.getFirst(ArrayDeque.java:318)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFFirstValue$FirstValStreamingFixedWindow.terminate(GenericUDAFFirstValue.java:290)
at
org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.finishPartition(WindowingTableFunction.java:413)
at
org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:337)
at org.apache.hadoop.hive.ql.exec.PTFOperator.closeOp(PTFOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:598)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
at
org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.close(SparkReduceRecordHandler.java:431)
... 15 more
{noformat}
We need to find out:
- Since which commit this test started failing, and
- Why it fails
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9104) windowing.q failed when mapred.reduce.tasks is set to larger than one
[
https://issues.apache.org/jira/browse/HIVE-9104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272299#comment-14272299
]
Xuefu Zhang commented on HIVE-9104:
---
[~csun] Could you add a test case in which perhaps the same query runs with
multiple reducers. It can be in the same .q file.
windowing.q failed when mapred.reduce.tasks is set to larger than one
-
Key: HIVE-9104
URL: https://issues.apache.org/jira/browse/HIVE-9104
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Chao
Assignee: Chao
Attachments: HIVE-9104.patch
Test {{windowing.q}} is actually not enabled in Spark branch - in test
configurations it is {{windowing.q.q}}.
I just run this test, and query
{code}
-- 12. testFirstLastWithWhere
select p_mfgr,p_name, p_size,
rank() over(distribute by p_mfgr sort by p_name) as r,
sum(p_size) over (distribute by p_mfgr sort by p_name rows between current
row and current row) as s2,
first_value(p_size) over w1 as f,
last_value(p_size, false) over w1 as l
from part
where p_mfgr = 'Manufacturer#3'
window w1 as (distribute by p_mfgr sort by p_name rows between 2 preceding
and 2 following);
{code}
failed with the following exception:
{noformat}
java.lang.RuntimeException: Hive Runtime Error while closing operators: null
at
org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.close(SparkReduceRecordHandler.java:446)
at
org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunctionResultList.closeRecordProcessor(HiveReduceFunctionResultList.java:58)
at
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:108)
at
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$2.apply(AsyncRDDActions.scala:115)
at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$2.apply(AsyncRDDActions.scala:115)
at org.apache.spark.SparkContext$$anonfun$30.apply(SparkContext.scala:1390)
at org.apache.spark.SparkContext$$anonfun$30.apply(SparkContext.scala:1390)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.NoSuchElementException
at java.util.ArrayDeque.getFirst(ArrayDeque.java:318)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFFirstValue$FirstValStreamingFixedWindow.terminate(GenericUDAFFirstValue.java:290)
at
org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.finishPartition(WindowingTableFunction.java:413)
at
org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:337)
at org.apache.hadoop.hive.ql.exec.PTFOperator.closeOp(PTFOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:598)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
at
org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.close(SparkReduceRecordHandler.java:431)
... 15 more
{noformat}
We need to find out:
- Since which commit this test started failing, and
- Why it fails
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272350#comment-14272350 ] Szehon Ho commented on HIVE-9257: - For reference, patch is uploaded here: [http://ec2-50-18-79-139.us-west-1.compute.amazonaws.com/data/spark-to-trunk-merge/HIVE-9257.patch|http://ec2-50-18-79-139.us-west-1.compute.amazonaws.com/data/spark-to-trunk-merge/HIVE-9257.patch] to get around the JIRA 10MB restriction. Ran a separate build that uses this patch, and the existing tests are passing: [http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build-Upload/lastCompletedBuild/testReport/|http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build-Upload/lastCompletedBuild/testReport/]. But spark tests are not running as ptest-master is missing spark test properties. Going to reconfigure the build machine with sparkCliDriver properties and give it another spin. Merge from spark to trunk January 2015 -- Key: HIVE-9257 URL: https://issues.apache.org/jira/browse/HIVE-9257 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: 0.15.0 Reporter: Szehon Ho Assignee: Szehon Ho The hive on spark work has reached a point where we can merge it into the trunk branch. Note that spark execution engine is optional and no current users should be impacted. This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9321) Notification message size can be arbitrarily long, DbNotificationListener limits to 1024
[ https://issues.apache.org/jira/browse/HIVE-9321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-9321: - Fix Version/s: 0.15.0 Affects Version/s: 0.15.0 Status: Patch Available (was: Open) Notification message size can be arbitrarily long, DbNotificationListener limits to 1024 Key: HIVE-9321 URL: https://issues.apache.org/jira/browse/HIVE-9321 Project: Hive Issue Type: Bug Affects Versions: 0.15.0 Reporter: Sushanth Sowmyan Assignee: Alan Gates Fix For: 0.15.0 Attachments: HIVE-9321.patch As part of HIVE-9174, a new notification listener, DbNotificationListener was introduced that stores notification messages in the metastore. Now, the message field of a Notification event can be arbitrarily long, since it is a JSON-serialized message of what event actually occurred, and an AddPartitionMessage which captured the result of an add_partitions can be arbitrarily long (as long as the number of partitions that were added). However, the message field for the storage in db is limited to VARCHAR(1024), which it will easily break. We need to fix this somehow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3405) UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase
[ https://issues.apache.org/jira/browse/HIVE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272279#comment-14272279 ] Alexander Pivovarov commented on HIVE-3405: --- it's only one test failed on build 2312 - TestMiniTezCliDriver.testCliDriver_optimize_nullscan This test failed in previous build 2311 as well So, the patch HIVE-3405.5 should be good. UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase - Key: HIVE-3405 URL: https://issues.apache.org/jira/browse/HIVE-3405 Project: Hive Issue Type: New Feature Components: UDF Affects Versions: 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.11.0, 0.13.0, 0.14.0, 0.15.0, 0.14.1 Reporter: Archana Nair Assignee: Alexander Pivovarov Labels: patch Attachments: HIVE-3405.1.patch.txt, HIVE-3405.2.patch, HIVE-3405.3.patch, HIVE-3405.4.patch, HIVE-3405.5.patch, HIVE-3405.5.patch Hive current releases lacks a INITCAP function which returns String with first letter of the word in uppercase.INITCAP returns String, with the first letter of each word in uppercase, all other letters in same case. Words are delimited by white space.This will be useful report generation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4639) Add has null flag to ORC internal index
[
https://issues.apache.org/jira/browse/HIVE-4639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272090#comment-14272090
]
Hive QA commented on HIVE-4639:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12691023/HIVE-4639.3.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6747 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_joins
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2311/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2311/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2311/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12691023 - PreCommit-HIVE-TRUNK-Build
Add has null flag to ORC internal index
---
Key: HIVE-4639
URL: https://issues.apache.org/jira/browse/HIVE-4639
Project: Hive
Issue Type: Improvement
Components: File Formats
Reporter: Owen O'Malley
Assignee: Prasanth Jayachandran
Attachments: HIVE-4639.1.patch, HIVE-4639.2.patch, HIVE-4639.3.patch
It would enable more predicate pushdown if we added a flag to the index entry
recording if there were any null values in the column for the 10k rows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 29671: Support select distinct *
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29671/#review67558 --- ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java https://reviews.apache.org/r/29671/#comment111596 Why do we need Select here since all you are looking for is column names to build ast, can't you get that from srcRel. - John Pullokkaran On Jan. 7, 2015, 9:16 p.m., pengcheng xiong wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29671/ --- (Updated Jan. 7, 2015, 9:16 p.m.) Review request for hive and John Pullokkaran. Repository: hive-git Description --- Support select distinct * in operator genaration phase. Diffs - itests/src/test/resources/testconfiguration.properties 5d07fba ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 306edf6 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c2d5c8c ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java 57868b7 ql/src/java/org/apache/hadoop/hive/ql/parse/UnparseTranslator.java 9ad6714 ql/src/test/queries/clientpositive/selectDistinctStar.q PRE-CREATION ql/src/test/results/clientpositive/selectDistinctStar.q.out PRE-CREATION ql/src/test/results/clientpositive/tez/selectDistinctStar.q.out PRE-CREATION Diff: https://reviews.apache.org/r/29671/diff/ Testing --- Thanks, pengcheng xiong
Re: Review Request 29671: Support select distinct *
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29671/#review67559 --- ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java https://reviews.apache.org/r/29671/#comment111597 Fix spacing - John Pullokkaran On Jan. 7, 2015, 9:16 p.m., pengcheng xiong wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29671/ --- (Updated Jan. 7, 2015, 9:16 p.m.) Review request for hive and John Pullokkaran. Repository: hive-git Description --- Support select distinct * in operator genaration phase. Diffs - itests/src/test/resources/testconfiguration.properties 5d07fba ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 306edf6 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c2d5c8c ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java 57868b7 ql/src/java/org/apache/hadoop/hive/ql/parse/UnparseTranslator.java 9ad6714 ql/src/test/queries/clientpositive/selectDistinctStar.q PRE-CREATION ql/src/test/results/clientpositive/selectDistinctStar.q.out PRE-CREATION ql/src/test/results/clientpositive/tez/selectDistinctStar.q.out PRE-CREATION Diff: https://reviews.apache.org/r/29671/diff/ Testing --- Thanks, pengcheng xiong
Re: Review Request 29671: Support select distinct *
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29671/#review67560 --- ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java https://reviews.apache.org/r/29671/#comment111598 Fix Spacing - John Pullokkaran On Jan. 7, 2015, 9:16 p.m., pengcheng xiong wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29671/ --- (Updated Jan. 7, 2015, 9:16 p.m.) Review request for hive and John Pullokkaran. Repository: hive-git Description --- Support select distinct * in operator genaration phase. Diffs - itests/src/test/resources/testconfiguration.properties 5d07fba ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 306edf6 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c2d5c8c ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java 57868b7 ql/src/java/org/apache/hadoop/hive/ql/parse/UnparseTranslator.java 9ad6714 ql/src/test/queries/clientpositive/selectDistinctStar.q PRE-CREATION ql/src/test/results/clientpositive/selectDistinctStar.q.out PRE-CREATION ql/src/test/results/clientpositive/tez/selectDistinctStar.q.out PRE-CREATION Diff: https://reviews.apache.org/r/29671/diff/ Testing --- Thanks, pengcheng xiong
Re: Review Request 29671: Support select distinct *
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29671/#review67562 --- ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java https://reviews.apache.org/r/29671/#comment111599 You can get RR from relToHiveRR map; i don't think you need this aditional arg. - John Pullokkaran On Jan. 7, 2015, 9:16 p.m., pengcheng xiong wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29671/ --- (Updated Jan. 7, 2015, 9:16 p.m.) Review request for hive and John Pullokkaran. Repository: hive-git Description --- Support select distinct * in operator genaration phase. Diffs - itests/src/test/resources/testconfiguration.properties 5d07fba ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 306edf6 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c2d5c8c ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java 57868b7 ql/src/java/org/apache/hadoop/hive/ql/parse/UnparseTranslator.java 9ad6714 ql/src/test/queries/clientpositive/selectDistinctStar.q PRE-CREATION ql/src/test/results/clientpositive/selectDistinctStar.q.out PRE-CREATION ql/src/test/results/clientpositive/tez/selectDistinctStar.q.out PRE-CREATION Diff: https://reviews.apache.org/r/29671/diff/ Testing --- Thanks, pengcheng xiong
Review Request 29787: HIVE-9257 : Merge spark to trunk January 2015 (Modified files)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29787/ --- Review request for hive. Bugs: HIVE-9257 https://issues.apache.org/jira/browse/HIVE-9257 Repository: hive-git Description --- As the entire patch is too big, this shows the modified files. These have been cleanuped as part of HIVE-9319, HIVE-9306, HIVE-9305. The new files can be found here: http://svn.apache.org/repos/asf/hive/branches/spark/ or https://github.com/apache/hive/tree/spark under: # data/conf/spark/ # itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithLocalClusterSpark.java # itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestMultiSessionsHS2WithLocalClusterSpark.java # itests/qtest-spark/ # ql/src/java/org/apache/hadoop/hive/ql/exec/SparkHashTableSinkOperator.java # ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ # ql/src/java/org/apache/hadoop/hive/ql/lib/TypeRule.java # ql/src/java/org/apache/hadoop/hive/ql/optimizer/SparkMapJoinProcessor.java # ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenSparkSkewJoinProcessor.java # ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkCrossProductCheck.java # ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java # ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/ # ql/src/java/org/apache/hadoop/hive/ql/parse/spark/ # ql/src/java/org/apache/hadoop/hive/ql/plan/SparkBucketMapJoinContext.java # ql/src/java/org/apache/hadoop/hive/ql/plan/SparkEdgeProperty.java # ql/src/java/org/apache/hadoop/hive/ql/plan/SparkHashTableSinkDesc.java # ql/src/java/org/apache/hadoop/hive/ql/plan/SparkWork.java # ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorSpark.java # ql/src/test/org/apache/hadoop/hive/ql/exec/spark/ # ql/src/test/queries/clientpositive/auto_join_stats.q # ql/src/test/queries/clientpositive/auto_join_stats2.q # ql/src/test/queries/clientpositive/bucket_map_join_spark1.q # ql/src/test/queries/clientpositive/bucket_map_join_spark2.q # ql/src/test/queries/clientpositive/bucket_map_join_spark3.q # ql/src/test/queries/clientpositive/bucket_map_join_spark4.q # ql/src/test/queries/clientpositive/multi_insert_mixed.q # ql/src/test/queries/clientpositive/multi_insert_union_src.q # ql/src/test/queries/clientpositive/parallel_join0.q # ql/src/test/queries/clientpositive/parallel_join1.q # ql/src/test/queries/clientpositive/spark_test.q # ql/src/test/queries/clientpositive/udf_example_add.q # ql/src/test/results/clientpositive/auto_join_stats.q.out # ql/src/test/results/clientpositive/auto_join_stats2.q.out # ql/src/test/results/clientpositive/bucket_map_join_spark1.q.out # ql/src/test/results/clientpositive/bucket_map_join_spark2.q.out # ql/src/test/results/clientpositive/bucket_map_join_spark3.q.out # ql/src/test/results/clientpositive/bucket_map_join_spark4.q.out # ql/src/test/results/clientpositive/multi_insert_mixed.q.out # ql/src/test/results/clientpositive/multi_insert_union_src.q.out # ql/src/test/results/clientpositive/parallel_join0.q.out # ql/src/test/results/clientpositive/parallel_join1.q.out # ql/src/test/results/clientpositive/spark/ # ql/src/test/results/clientpositive/spark_test.q.out # ql/src/test/results/clientpositive/udf_example_add.q.out # spark-client/ Cleanup and review of those have been done as part of HIVE-9281 and HIVE-9288. Diffs - common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java cd4beeb common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8264b16 data/conf/hive-log4j.properties a5b9c9a itests/hive-unit/pom.xml f9f59c9 itests/pom.xml 0a154d6 itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 878202a pom.xml efe5e3a ql/pom.xml 84e912e ql/src/java/org/apache/hadoop/hive/ql/Context.java 0373273 ql/src/java/org/apache/hadoop/hive/ql/Driver.java 8bb6d0f ql/src/java/org/apache/hadoop/hive/ql/HashTableLoaderFactory.java 10ad933 ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 3f5f475 ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java b96faa4 ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java f0e04e7 ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 561bf3b ql/src/java/org/apache/hadoop/hive/ql/exec/ScriptOperator.java 87eaa8f ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java e3469a0 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java ed03bb9 ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapperContext.java 8b92f32
[jira] [Commented] (HIVE-9257) Merge from spark to trunk January 2015
[ https://issues.apache.org/jira/browse/HIVE-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272107#comment-14272107 ] Szehon Ho commented on HIVE-9257: - Adding review request: [https://reviews.apache.org/r/29787/|https://reviews.apache.org/r/29787/]. As the entire patch is too big, this shows the modified files. These have been cleanuped as part of HIVE-9319, HIVE-9306, HIVE-9305. The new files can be found here: [http://svn.apache.org/repos/asf/hive/branches/spark/|http://svn.apache.org/repos/asf/hive/branches/spark/] or [https://github.com/apache/hive/tree/spark|https://github.com/apache/hive/tree/spark] under: data/conf/spark/ itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithLocalClusterSpark.java itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestMultiSessionsHS2WithLocalClusterSpark.java itests/qtest-spark/ ql/src/java/org/apache/hadoop/hive/ql/exec/SparkHashTableSinkOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ ql/src/java/org/apache/hadoop/hive/ql/lib/TypeRule.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/SparkMapJoinProcessor.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenSparkSkewJoinProcessor.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkCrossProductCheck.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/ ql/src/java/org/apache/hadoop/hive/ql/parse/spark/ ql/src/java/org/apache/hadoop/hive/ql/plan/SparkBucketMapJoinContext.java ql/src/java/org/apache/hadoop/hive/ql/plan/SparkEdgeProperty.java ql/src/java/org/apache/hadoop/hive/ql/plan/SparkHashTableSinkDesc.java ql/src/java/org/apache/hadoop/hive/ql/plan/SparkWork.java ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorSpark.java ql/src/test/org/apache/hadoop/hive/ql/exec/spark/ ql/src/test/queries/clientpositive/auto_join_stats.q ql/src/test/queries/clientpositive/auto_join_stats2.q ql/src/test/queries/clientpositive/bucket_map_join_spark1.q ql/src/test/queries/clientpositive/bucket_map_join_spark2.q ql/src/test/queries/clientpositive/bucket_map_join_spark3.q ql/src/test/queries/clientpositive/bucket_map_join_spark4.q ql/src/test/queries/clientpositive/multi_insert_mixed.q ql/src/test/queries/clientpositive/multi_insert_union_src.q ql/src/test/queries/clientpositive/parallel_join0.q ql/src/test/queries/clientpositive/parallel_join1.q ql/src/test/queries/clientpositive/spark_test.q ql/src/test/queries/clientpositive/udf_example_add.q ql/src/test/results/clientpositive/auto_join_stats.q.out ql/src/test/results/clientpositive/auto_join_stats2.q.out ql/src/test/results/clientpositive/bucket_map_join_spark1.q.out ql/src/test/results/clientpositive/bucket_map_join_spark2.q.out ql/src/test/results/clientpositive/bucket_map_join_spark3.q.out ql/src/test/results/clientpositive/bucket_map_join_spark4.q.out ql/src/test/results/clientpositive/multi_insert_mixed.q.out ql/src/test/results/clientpositive/multi_insert_union_src.q.out ql/src/test/results/clientpositive/parallel_join0.q.out ql/src/test/results/clientpositive/parallel_join1.q.out ql/src/test/results/clientpositive/spark/ ql/src/test/results/clientpositive/spark_test.q.out ql/src/test/results/clientpositive/udf_example_add.q.out spark-client/ Cleanup and review of those have been done as part of HIVE-9281 and HIVE-9288. Merge from spark to trunk January 2015 -- Key: HIVE-9257 URL: https://issues.apache.org/jira/browse/HIVE-9257 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: 0.15.0 Reporter: Szehon Ho Assignee: Szehon Ho The hive on spark work has reached a point where we can merge it into the trunk branch. Note that spark execution engine is optional and no current users should be impacted. This JIRA will be used to track the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9334) PredicateTransitivePropagate optimizer should run after PredicatePushDown
Ashutosh Chauhan created HIVE-9334: -- Summary: PredicateTransitivePropagate optimizer should run after PredicatePushDown Key: HIVE-9334 URL: https://issues.apache.org/jira/browse/HIVE-9334 Project: Hive Issue Type: Improvement Components: Logical Optimizer Affects Versions: 0.14.0, 0.13.0, 0.12.0, 0.11.0, 0.10.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan This way PredicateTransitivePropagate will be more effective as it has more filters to push for other branches of joins. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8966) Delta files created by hive hcatalog streaming cannot be compacted
[ https://issues.apache.org/jira/browse/HIVE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-8966: - Attachment: HIVE-8966.4.patch This patch takes a new approach. Rather than changing AcidUtils.getAcidState (as previous 2 attempts) this patch gives a new implementation of ValidTxnList that only returns isTxnRangeValid ALL or NONE, and gives NONE if there are any open transactions = the max transaction in the range (even if it's below the range). This new implementation is used only by the compactor so that it's understanding of what files it should compact are different than what files a reader views as available for reading. I've also added tests to TestCompactor to test compaction during streaming and compaction after a streamer has aborted and died without cleaning up. Delta files created by hive hcatalog streaming cannot be compacted -- Key: HIVE-8966 URL: https://issues.apache.org/jira/browse/HIVE-8966 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.14.0 Environment: hive Reporter: Jihong Liu Assignee: Alan Gates Priority: Critical Fix For: 0.14.1 Attachments: HIVE-8966.2.patch, HIVE-8966.3.patch, HIVE-8966.4.patch, HIVE-8966.patch hive hcatalog streaming will also create a file like bucket_n_flush_length in each delta directory. Where n is the bucket number. But the compactor.CompactorMR think this file also needs to compact. However this file of course cannot be compacted, so compactor.CompactorMR will not continue to do the compaction. Did a test, after removed the bucket_n_flush_length file, then the alter table partition compact finished successfully. If don't delete that file, nothing will be compacted. This is probably a very severity bug. Both 0.13 and 0.14 have this issue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8966) Delta files created by hive hcatalog streaming cannot be compacted
[ https://issues.apache.org/jira/browse/HIVE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-8966: - Status: Patch Available (was: Open) Delta files created by hive hcatalog streaming cannot be compacted -- Key: HIVE-8966 URL: https://issues.apache.org/jira/browse/HIVE-8966 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.14.0 Environment: hive Reporter: Jihong Liu Assignee: Alan Gates Priority: Critical Fix For: 0.14.1 Attachments: HIVE-8966.2.patch, HIVE-8966.3.patch, HIVE-8966.4.patch, HIVE-8966.patch hive hcatalog streaming will also create a file like bucket_n_flush_length in each delta directory. Where n is the bucket number. But the compactor.CompactorMR think this file also needs to compact. However this file of course cannot be compacted, so compactor.CompactorMR will not continue to do the compaction. Did a test, after removed the bucket_n_flush_length file, then the alter table partition compact finished successfully. If don't delete that file, nothing will be compacted. This is probably a very severity bug. Both 0.13 and 0.14 have this issue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9334) PredicateTransitivePropagate optimizer should run after PredicatePushDown
[
https://issues.apache.org/jira/browse/HIVE-9334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-9334:
---
Attachment: HIVE-9334.patch
[~navis] Since you implemented {{PredicateTransitivePropagate}} optimization.
Does this make sense?
PredicateTransitivePropagate optimizer should run after PredicatePushDown
-
Key: HIVE-9334
URL: https://issues.apache.org/jira/browse/HIVE-9334
Project: Hive
Issue Type: Improvement
Components: Logical Optimizer
Affects Versions: 0.10.0, 0.11.0, 0.12.0, 0.13.0, 0.14.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Attachments: HIVE-9334.patch
This way PredicateTransitivePropagate will be more effective as it has more
filters to push for other branches of joins.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9334) PredicateTransitivePropagate optimizer should run after PredicatePushDown
[ https://issues.apache.org/jira/browse/HIVE-9334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-9334: --- Status: Patch Available (was: Open) PredicateTransitivePropagate optimizer should run after PredicatePushDown - Key: HIVE-9334 URL: https://issues.apache.org/jira/browse/HIVE-9334 Project: Hive Issue Type: Improvement Components: Logical Optimizer Affects Versions: 0.14.0, 0.13.0, 0.12.0, 0.11.0, 0.10.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-9334.patch This way PredicateTransitivePropagate will be more effective as it has more filters to push for other branches of joins. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-4639) Add has null flag to ORC internal index
[ https://issues.apache.org/jira/browse/HIVE-4639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-4639: Resolution: Fixed Fix Version/s: 0.15.0 Release Note: Support for hasNull flag in ORC row group index. Status: Resolved (was: Patch Available) Committed to trunk. Thanks [~gopalv] for the review and test run! Add has null flag to ORC internal index --- Key: HIVE-4639 URL: https://issues.apache.org/jira/browse/HIVE-4639 Project: Hive Issue Type: Improvement Components: File Formats Reporter: Owen O'Malley Assignee: Prasanth Jayachandran Fix For: 0.15.0 Attachments: HIVE-4639.1.patch, HIVE-4639.2.patch, HIVE-4639.3.patch It would enable more predicate pushdown if we added a flag to the index entry recording if there were any null values in the column for the 10k rows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4639) Add has null flag to ORC internal index
[ https://issues.apache.org/jira/browse/HIVE-4639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272122#comment-14272122 ] Gopal V commented on HIVE-4639: --- for the sake of documentation this does not change the ORC format version (i.e ORC files with hasNull flags can be read by hive-14). [~leftylev]: FYI. Add has null flag to ORC internal index --- Key: HIVE-4639 URL: https://issues.apache.org/jira/browse/HIVE-4639 Project: Hive Issue Type: Improvement Components: File Formats Reporter: Owen O'Malley Assignee: Prasanth Jayachandran Fix For: 0.15.0 Attachments: HIVE-4639.1.patch, HIVE-4639.2.patch, HIVE-4639.3.patch It would enable more predicate pushdown if we added a flag to the index entry recording if there were any null values in the column for the 10k rows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-3405) UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase
[ https://issues.apache.org/jira/browse/HIVE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Pivovarov updated HIVE-3405: -- Status: Patch Available (was: In Progress) UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase - Key: HIVE-3405 URL: https://issues.apache.org/jira/browse/HIVE-3405 Project: Hive Issue Type: New Feature Components: UDF Affects Versions: 0.14.0, 0.13.0, 0.11.0, 0.10.0, 0.9.0, 0.8.1, 0.15.0, 0.14.1, 0.9.1 Reporter: Archana Nair Assignee: Alexander Pivovarov Labels: patch Attachments: HIVE-3405.1.patch.txt, HIVE-3405.2.patch, HIVE-3405.3.patch, HIVE-3405.4.patch, HIVE-3405.5.patch, HIVE-3405.5.patch Hive current releases lacks a INITCAP function which returns String with first letter of the word in uppercase.INITCAP returns String, with the first letter of each word in uppercase, all other letters in same case. Words are delimited by white space.This will be useful report generation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-3405) UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase
[ https://issues.apache.org/jira/browse/HIVE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Pivovarov updated HIVE-3405: -- Status: In Progress (was: Patch Available) UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase - Key: HIVE-3405 URL: https://issues.apache.org/jira/browse/HIVE-3405 Project: Hive Issue Type: New Feature Components: UDF Affects Versions: 0.14.0, 0.13.0, 0.11.0, 0.10.0, 0.9.0, 0.8.1, 0.15.0, 0.14.1, 0.9.1 Reporter: Archana Nair Assignee: Alexander Pivovarov Labels: patch Attachments: HIVE-3405.1.patch.txt, HIVE-3405.2.patch, HIVE-3405.3.patch, HIVE-3405.4.patch, HIVE-3405.5.patch Hive current releases lacks a INITCAP function which returns String with first letter of the word in uppercase.INITCAP returns String, with the first letter of each word in uppercase, all other letters in same case. Words are delimited by white space.This will be useful report generation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-3405) UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase
[ https://issues.apache.org/jira/browse/HIVE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Pivovarov updated HIVE-3405: -- Attachment: HIVE-3405.5.patch UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase - Key: HIVE-3405 URL: https://issues.apache.org/jira/browse/HIVE-3405 Project: Hive Issue Type: New Feature Components: UDF Affects Versions: 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.11.0, 0.13.0, 0.14.0, 0.15.0, 0.14.1 Reporter: Archana Nair Assignee: Alexander Pivovarov Labels: patch Attachments: HIVE-3405.1.patch.txt, HIVE-3405.2.patch, HIVE-3405.3.patch, HIVE-3405.4.patch, HIVE-3405.5.patch, HIVE-3405.5.patch Hive current releases lacks a INITCAP function which returns String with first letter of the word in uppercase.INITCAP returns String, with the first letter of each word in uppercase, all other letters in same case. Words are delimited by white space.This will be useful report generation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3405) UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase
[ https://issues.apache.org/jira/browse/HIVE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272129#comment-14272129 ] Alexander Pivovarov commented on HIVE-3405: --- how to trigger HIVE build? for some reason 3405 in not in the queue http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/ UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase - Key: HIVE-3405 URL: https://issues.apache.org/jira/browse/HIVE-3405 Project: Hive Issue Type: New Feature Components: UDF Affects Versions: 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.11.0, 0.13.0, 0.14.0, 0.15.0, 0.14.1 Reporter: Archana Nair Assignee: Alexander Pivovarov Labels: patch Attachments: HIVE-3405.1.patch.txt, HIVE-3405.2.patch, HIVE-3405.3.patch, HIVE-3405.4.patch, HIVE-3405.5.patch, HIVE-3405.5.patch Hive current releases lacks a INITCAP function which returns String with first letter of the word in uppercase.INITCAP returns String, with the first letter of each word in uppercase, all other letters in same case. Words are delimited by white space.This will be useful report generation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3405) UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase
[ https://issues.apache.org/jira/browse/HIVE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272134#comment-14272134 ] Szehon Ho commented on HIVE-3405: - Current build should be testing 3405: [http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2312/|http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2312/], it says in the parameters. UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase - Key: HIVE-3405 URL: https://issues.apache.org/jira/browse/HIVE-3405 Project: Hive Issue Type: New Feature Components: UDF Affects Versions: 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.11.0, 0.13.0, 0.14.0, 0.15.0, 0.14.1 Reporter: Archana Nair Assignee: Alexander Pivovarov Labels: patch Attachments: HIVE-3405.1.patch.txt, HIVE-3405.2.patch, HIVE-3405.3.patch, HIVE-3405.4.patch, HIVE-3405.5.patch, HIVE-3405.5.patch Hive current releases lacks a INITCAP function which returns String with first letter of the word in uppercase.INITCAP returns String, with the first letter of each word in uppercase, all other letters in same case. Words are delimited by white space.This will be useful report generation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4639) Add has null flag to ORC internal index
[ https://issues.apache.org/jira/browse/HIVE-4639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272141#comment-14272141 ] Lefty Leverenz commented on HIVE-4639: -- Thanks [~gopalv]. I assume that means no documentation is needed, since this is internal and backward-compatible. Add has null flag to ORC internal index --- Key: HIVE-4639 URL: https://issues.apache.org/jira/browse/HIVE-4639 Project: Hive Issue Type: Improvement Components: File Formats Reporter: Owen O'Malley Assignee: Prasanth Jayachandran Fix For: 0.15.0 Attachments: HIVE-4639.1.patch, HIVE-4639.2.patch, HIVE-4639.3.patch It would enable more predicate pushdown if we added a flag to the index entry recording if there were any null values in the column for the 10k rows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9329) The qtest can't handle the statements which contains semicolons in qfile
Ferdinand Xu created HIVE-9329:
--
Summary: The qtest can't handle the statements which contains
semicolons in qfile
Key: HIVE-9329
URL: https://issues.apache.org/jira/browse/HIVE-9329
Project: Hive
Issue Type: Sub-task
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
The previous logic is trying to split the line by semicolons. The following
statement can't be handled.
compile `import org.apache.hadoop.hive.ql.exec.UDF \;
public class Pyth extends UDF {
public double evaluate(double a, double b){
return Math.sqrt((a*a) + (b*b)) \;
}
} `AS GROOVY NAMED Pyth.groovy;
CREATE TEMPORARY FUNCTION Pyth as 'Pyth';
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9329) The qtest can't handle the statements which contains semicolons in qfile
[
https://issues.apache.org/jira/browse/HIVE-9329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferdinand Xu updated HIVE-9329:
---
Fix Version/s: encryption-branch
The qtest can't handle the statements which contains semicolons in qfile
Key: HIVE-9329
URL: https://issues.apache.org/jira/browse/HIVE-9329
Project: Hive
Issue Type: Sub-task
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
Fix For: encryption-branch
The previous logic is trying to split the line by semicolons. The following
statement can't be handled.
compile `import org.apache.hadoop.hive.ql.exec.UDF \;
public class Pyth extends UDF {
public double evaluate(double a, double b){
return Math.sqrt((a*a) + (b*b)) \;
}
} `AS GROOVY NAMED Pyth.groovy;
CREATE TEMPORARY FUNCTION Pyth as 'Pyth';
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9328) Tests cannot move files due to change on HIVE-9325
[ https://issues.apache.org/jira/browse/HIVE-9328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-9328: -- Attachment: HIVE-9325.1.patch Tests cannot move files due to change on HIVE-9325 -- Key: HIVE-9328 URL: https://issues.apache.org/jira/browse/HIVE-9328 Project: Hive Issue Type: Sub-task Affects Versions: encryption-branch Reporter: Sergio Peña Assignee: Sergio Peña Attachments: HIVE-9325.1.patch The change on HIVE-9325 that modified the isSubDir() method is checking that the source path is part of the source path? This always returns true and many tests are failing. The method should check if source path is part of destination path. This is the line that is failing: +String fullF1 = getQualifiedPathWithoutSchemeAndAuthority(srcf, fs); +String fullF2 = getQualifiedPathWithoutSchemeAndAuthority(srcf, fs); It should be: +String fullF1 = getQualifiedPathWithoutSchemeAndAuthority(srcf, fs); +String fullF2 = getQualifiedPathWithoutSchemeAndAuthority(destf, fs); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-9251: -- Resolution: Fixed Fix Version/s: spark-branch Status: Resolved (was: Patch Available) Committed to spark branch. Thanks, Rui. SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch] --- Key: HIVE-9251 URL: https://issues.apache.org/jira/browse/HIVE-9251 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Rui Li Assignee: Rui Li Fix For: spark-branch Attachments: HIVE-9251.1-spark.patch, HIVE-9251.2-spark.patch, HIVE-9251.3-spark.patch, HIVE-9251.4-spark.patch, HIVE-9251.5-spark.patch, HIVE-9251.6-spark.patch This may hurt performance or even lead to task failures. For example, spark's netty-based shuffle limits the max frame size to be 2G. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9290) Make some test results deterministic
[
https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271116#comment-14271116
]
Xuefu Zhang commented on HIVE-9290:
---
I was aware of this but knew Rui was immediately working on HIVE-9251 which
depends on this issue. Yes, there will be a little time period where the tests
would fail, but I think it's okay as long as we are aware.
Make some test results deterministic
Key: HIVE-9290
URL: https://issues.apache.org/jira/browse/HIVE-9290
Project: Hive
Issue Type: Test
Reporter: Rui Li
Assignee: Rui Li
Fix For: spark-branch, 0.15.0
Attachments: HIVE-9290-spark.patch, HIVE-9290.1.patch,
HIVE-9290.1.patch
{noformat}
limit_pushdown.q
optimize_nullscan.q
ppd_gby_join.q
vector_string_concat.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9328) Tests cannot move files due to change on HIVE-9325
Sergio Peña created HIVE-9328: - Summary: Tests cannot move files due to change on HIVE-9325 Key: HIVE-9328 URL: https://issues.apache.org/jira/browse/HIVE-9328 Project: Hive Issue Type: Sub-task Affects Versions: encryption-branch Reporter: Sergio Peña Assignee: Sergio Peña The change on HIVE-9325 that modified the isSubDir() method is checking that the source path is part of the source path? This always returns true and many tests are failing. The method should check if source path is part of destination path. This is the line that is failing: +String fullF1 = getQualifiedPathWithoutSchemeAndAuthority(srcf, fs); +String fullF2 = getQualifiedPathWithoutSchemeAndAuthority(srcf, fs); It should be: +String fullF1 = getQualifiedPathWithoutSchemeAndAuthority(srcf, fs); +String fullF2 = getQualifiedPathWithoutSchemeAndAuthority(destf, fs); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-9290) Make some test results deterministic
[
https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271116#comment-14271116
]
Xuefu Zhang edited comment on HIVE-9290 at 1/9/15 3:42 PM:
---
I was aware of this but knew Rui was immediately working on HIVE-9251 which
depends on this issue. Yes, there would be a little time period where the tests
would fail, but I thought it's okay as long as we are aware. Sorry for the
inconvenience.
was (Author: xuefuz):
I was aware of this but knew Rui was immediately working on HIVE-9251 which
depends on this issue. Yes, there will be a little time period where the tests
would fail, but I think it's okay as long as we are aware.
Make some test results deterministic
Key: HIVE-9290
URL: https://issues.apache.org/jira/browse/HIVE-9290
Project: Hive
Issue Type: Test
Reporter: Rui Li
Assignee: Rui Li
Fix For: spark-branch, 0.15.0
Attachments: HIVE-9290-spark.patch, HIVE-9290.1.patch,
HIVE-9290.1.patch
{noformat}
limit_pushdown.q
optimize_nullscan.q
ppd_gby_join.q
vector_string_concat.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9314) Writable object inspector should use Writable classes from Hadoop for get
[
https://issues.apache.org/jira/browse/HIVE-9314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271274#comment-14271274
]
Hive QA commented on HIVE-9314:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690926/HIVE-9314.1.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6737 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2306/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2306/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2306/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690926 - PreCommit-HIVE-TRUNK-Build
Writable object inspector should use Writable classes from Hadoop for get
-
Key: HIVE-9314
URL: https://issues.apache.org/jira/browse/HIVE-9314
Project: Hive
Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
Attachments: HIVE-9314.1.patch
In WritableByteObjectInspector, we cast objects to
org.apache.hadoop.hive.serde2.io.ByteWritable to get the value. We should
use the hadoop one since org.apache.hadoop.hive.serde2.io.ByteWritable
extends the hadoop one.
If customer OI uses the hadoop writable, in some queries, they could get
class cast exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9310) CLI JLine does not flush history back to ~/.hivehistory
[
https://issues.apache.org/jira/browse/HIVE-9310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270726#comment-14270726
]
Hive QA commented on HIVE-9310:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690748/HIVE-9310.1.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6737 tests executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2300/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2300/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2300/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690748 - PreCommit-HIVE-TRUNK-Build
CLI JLine does not flush history back to ~/.hivehistory
---
Key: HIVE-9310
URL: https://issues.apache.org/jira/browse/HIVE-9310
Project: Hive
Issue Type: Bug
Components: CLI
Affects Versions: 0.15.0
Reporter: Gopal V
Assignee: Gopal V
Priority: Minor
Attachments: HIVE-9310.1.patch
Hive CLI does not seem to be saving history anymore.
In JLine with the PersistentHistory class, to keep history across sessions,
you need to do {{reader.getHistory().flush()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HIVE-9290) Make some test results deterministic
[
https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho reopened HIVE-9290:
-
Make some test results deterministic
Key: HIVE-9290
URL: https://issues.apache.org/jira/browse/HIVE-9290
Project: Hive
Issue Type: Test
Reporter: Rui Li
Assignee: Rui Li
Fix For: spark-branch, 0.15.0
Attachments: HIVE-9290-spark.patch, HIVE-9290.1.patch,
HIVE-9290.1.patch
{noformat}
limit_pushdown.q
optimize_nullscan.q
ppd_gby_join.q
vector_string_concat.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9290) Make some test results deterministic
[
https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-9290:
Attachment: HIVE-9290-spark.patch
I'm attaching a patch to regenerate the output files in spark branch version.
Xuefu, can the committer in these scenarios remember to also regenerate the
tests whenever merging to spark-branch? Or alternatively the contributor can
generate two versions of the patch if plan to merge to spark. Thanks!
Make some test results deterministic
Key: HIVE-9290
URL: https://issues.apache.org/jira/browse/HIVE-9290
Project: Hive
Issue Type: Test
Reporter: Rui Li
Assignee: Rui Li
Fix For: spark-branch, 0.15.0
Attachments: HIVE-9290-spark.patch, HIVE-9290.1.patch,
HIVE-9290.1.patch
{noformat}
limit_pushdown.q
optimize_nullscan.q
ppd_gby_join.q
vector_string_concat.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9323) Merge from trunk to spark 1/8/2015
[ https://issues.apache.org/jira/browse/HIVE-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270728#comment-14270728 ] Szehon Ho commented on HIVE-9323: - Got it, thanks for the analysis. Maybe we need to increase that timeout (hive.spark.client.server.connect.timeout.ms) if it happens again. Merge from trunk to spark 1/8/2015 -- Key: HIVE-9323 URL: https://issues.apache.org/jira/browse/HIVE-9323 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Szehon Ho Assignee: Szehon Ho Fix For: spark-branch Attachments: HIVE-9323-spark.patch, HIVE-9323.2-spark.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9290) Make some test results deterministic
[
https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-9290:
Status: Patch Available (was: Reopened)
Make some test results deterministic
Key: HIVE-9290
URL: https://issues.apache.org/jira/browse/HIVE-9290
Project: Hive
Issue Type: Test
Reporter: Rui Li
Assignee: Rui Li
Fix For: spark-branch, 0.15.0
Attachments: HIVE-9290-spark.patch, HIVE-9290.1.patch,
HIVE-9290.1.patch
{noformat}
limit_pushdown.q
optimize_nullscan.q
ppd_gby_join.q
vector_string_concat.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270776#comment-14270776
]
Hive QA commented on HIVE-9251:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12691052/HIVE-9251.4-spark.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 7285 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_transform_acid
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_windowing
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/624/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/624/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-624/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12691052 - PreCommit-HIVE-SPARK-Build
SetSparkReducerParallelism is likely to set too small number of reducers
[Spark Branch]
---
Key: HIVE-9251
URL: https://issues.apache.org/jira/browse/HIVE-9251
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Rui Li
Assignee: Rui Li
Attachments: HIVE-9251.1-spark.patch, HIVE-9251.2-spark.patch,
HIVE-9251.3-spark.patch, HIVE-9251.4-spark.patch
This may hurt performance or even lead to task failures. For example, spark's
netty-based shuffle limits the max frame size to be 2G.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9249) java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to org.apache.hadoop.hive.common.type.HiveVarchar when joining tables
[
https://issues.apache.org/jira/browse/HIVE-9249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270795#comment-14270795
]
Matt McCline commented on HIVE-9249:
Yes, [~gopalv] the hive-14 patch looks good. Thanks.
java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
org.apache.hadoop.hive.common.type.HiveVarchar when joining tables
---
Key: HIVE-9249
URL: https://issues.apache.org/jira/browse/HIVE-9249
Project: Hive
Issue Type: Bug
Components: Vectorization
Affects Versions: 0.14.0
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical
Fix For: 0.15.0
Attachments: HIVE-9249.01.patch, HIVE-9249.hive-14.patch
VectorColumnAssignFactory doesn't handle HiveCharWritable /
HiveVarcharWritable objects.
Either:
HiveVarcharWritable cannot be cast to ... HiveVarchar
or
HiveCharWritable cannot be cast to ... HiveChar
{code}
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
org.apache.hadoop.hive.common.type.HiveVarchar
at
org.apache.hadoop.hive.ql.exec.vector.VectorColumnAssignFactory$17.assignObjectValue(VectorColumnAssignFactory.java:417)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.internalForward(VectorMapJoinOperator.java:196)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:670)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:748)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:299)
... 24 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-9249) java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to org.apache.hadoop.hive.common.type.HiveVarchar when joining tables
[
https://issues.apache.org/jira/browse/HIVE-9249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V resolved HIVE-9249.
---
Resolution: Fixed
Fix Version/s: 0.14.1
Release Note: HIVE-9249: Vectorization: Join involving CHAR/VARCHAR fails
during execution. (Matt McCline, reviewed by Gopal V)
Committed to branch and trunk, thanks [~mmccline].
java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
org.apache.hadoop.hive.common.type.HiveVarchar when joining tables
---
Key: HIVE-9249
URL: https://issues.apache.org/jira/browse/HIVE-9249
Project: Hive
Issue Type: Bug
Components: Vectorization
Affects Versions: 0.14.0
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical
Fix For: 0.15.0, 0.14.1
Attachments: HIVE-9249.01.patch, HIVE-9249.hive-14.patch
VectorColumnAssignFactory doesn't handle HiveCharWritable /
HiveVarcharWritable objects.
Either:
HiveVarcharWritable cannot be cast to ... HiveVarchar
or
HiveCharWritable cannot be cast to ... HiveChar
{code}
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
org.apache.hadoop.hive.common.type.HiveVarchar
at
org.apache.hadoop.hive.ql.exec.vector.VectorColumnAssignFactory$17.assignObjectValue(VectorColumnAssignFactory.java:417)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.internalForward(VectorMapJoinOperator.java:196)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:670)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:748)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:299)
... 24 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-9311) I tryed to run max query using jdbc it didint work
[
https://issues.apache.org/jira/browse/HIVE-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis resolved HIVE-9311.
-
Resolution: Not a Problem
Use mailing list for simple questions.
I tryed to run max query using jdbc it didint work
--
Key: HIVE-9311
URL: https://issues.apache.org/jira/browse/HIVE-9311
Project: Hive
Issue Type: Bug
Components: Configuration
Affects Versions: 0.12.0
Reporter: payal
I tryed to run max query using jdbc it didint work please look at it
Class.forName(org.apache.hive.jdbc.HiveDriver);
try {
con1 =3D DriverManager.getConnection(
jdbc:hive2://dkhc3012:1/default, hive, );
} catch (SQLException sqle) {
System.out.println(SQL Exception thrown: + sqle);
}
System.out.println(Connected);
System.out.println(columns + tables);
Statement stmt =3D con1.createStatement();
result =3D stmt.executeQuery(SELECT max(date_id) FROM date_dim);
while (result.next()) {
System.out.println(In while);
val =3D result.getInt(1);
System.out.println(val);
System.out.println(result.getString(1));
}
System.out.println(JDBC Done);
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9324) Reduce side joins failing with IOException from RowContainer.nextBlock
[
https://issues.apache.org/jira/browse/HIVE-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270779#comment-14270779
]
Amareshwari Sriramadasu commented on HIVE-9324:
---
After doing some code walkthrough, here is what i found,
On JoinOperator, whenever any key as more values than BLOCKSIZE(hardcoded to
25000), it spills the values to a file on disk, and spill uses SequenceFile
format.
Here is the table description for spill (from
org.apache.hadoop.hive.ql.exec.JoinUtil.java)
{noformat}
TableDesc tblDesc = new TableDesc(
SequenceFileInputFormat.class, HiveSequenceFileOutputFormat.class,
Utilities.makeProperties(
org.apache.hadoop.hive.serde.serdeConstants.SERIALIZATION_FORMAT,
+ Utilities.ctrlaCode,
org.apache.hadoop.hive.serde.serdeConstants.LIST_COLUMNS, colNames
.toString(),
org.apache.hadoop.hive.serde.serdeConstants.LIST_COLUMN_TYPES,
colTypes.toString(),
serdeConstants.SERIALIZATION_LIB,LazyBinarySerDe.class.getName()));
spillTableDesc[tag] = tblDesc;
{noformat}
From the exception:
{noformat}
Caused by: java.io.IOException:
org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer@42610e8 read 1 bytes, should read
27264
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2435)
at
org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:76)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.nextBlock(RowContainer.java:360)
... 13 more
{noformat}
I see that the value in SequenceFile is RCFile$KeyBuffer, dont know why. Also
couldnt figure out the reason why the reading went wrong.
Following is the code snippet from SequenceFile.java for the exception we are
hitting :
{noformat}
2417 public synchronized Object next(Object key) throws IOException {
2418 if (key != null key.getClass() != getKeyClass()) {
2419 throw new IOException(wrong key class: +key.getClass().getName()
2420 + is not +keyClass);
2421 }
2422
2423 if (!blockCompressed) {
2424 outBuf.reset();
2425
2426 keyLength = next(outBuf);
2427 if (keyLength 0)
2428 return null;
2429
2430 valBuffer.reset(outBuf.getData(), outBuf.getLength());
2431
2432 key = deserializeKey(key);
2433 valBuffer.mark(0);
2434 if (valBuffer.getPosition() != keyLength)
2435 throw new IOException(key + read + valBuffer.getPosition()
2436 + bytes, should read + keyLength);
{noformat}
Reduce side joins failing with IOException from RowContainer.nextBlock
--
Key: HIVE-9324
URL: https://issues.apache.org/jira/browse/HIVE-9324
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.13.1
Reporter: Amareshwari Sriramadasu
We are seeing some reduce side join mapreduce jobs failing with following
exception :
{noformat}
2014-12-14 16:58:51,296 ERROR
org.apache.hadoop.hive.ql.exec.persistence.RowContainer:
org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer@42610e8 read 1 bytes, should
read 27264
java.io.IOException: org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer@42610e8
read 1 bytes, should read 27264
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2435)
at
org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:76)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.nextBlock(RowContainer.java:360)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:230)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:74)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:644)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:758)
at
org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256)
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:216)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:506)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
2014-12-14 16:58:51,334 FATAL ExecReducer:
org.apache.hadoop.hive.ql.metadata.HiveException:
[jira] [Commented] (HIVE-9310) CLI JLine does not flush history back to ~/.hivehistory
[
https://issues.apache.org/jira/browse/HIVE-9310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270780#comment-14270780
]
Gopal V commented on HIVE-9310:
---
[~navis]: did not want to leak the implementation out of the same class.
I saw issues with ShutdownHooks and different classloaders in some other work,
didn't want to go that way because there's no easy way to test these things for
slow leaks.
My patch is a 5-min safe fix.
CLI JLine does not flush history back to ~/.hivehistory
---
Key: HIVE-9310
URL: https://issues.apache.org/jira/browse/HIVE-9310
Project: Hive
Issue Type: Bug
Components: CLI
Affects Versions: 0.15.0
Reporter: Gopal V
Assignee: Gopal V
Priority: Minor
Attachments: HIVE-9310.1.patch
Hive CLI does not seem to be saving history anymore.
In JLine with the PersistentHistory class, to keep history across sessions,
you need to do {{reader.getHistory().flush()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9251: - Attachment: HIVE-9251.5-spark.patch I missed some update to optimize_nullscan.q Update patch. SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch] --- Key: HIVE-9251 URL: https://issues.apache.org/jira/browse/HIVE-9251 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Rui Li Assignee: Rui Li Attachments: HIVE-9251.1-spark.patch, HIVE-9251.2-spark.patch, HIVE-9251.3-spark.patch, HIVE-9251.4-spark.patch, HIVE-9251.5-spark.patch This may hurt performance or even lead to task failures. For example, spark's netty-based shuffle limits the max frame size to be 2G. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9326) BaseProtocol.Error failed to deserialization due to NPE.[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-9326: Status: Patch Available (was: Open) BaseProtocol.Error failed to deserialization due to NPE.[Spark Branch] -- Key: HIVE-9326 URL: https://issues.apache.org/jira/browse/HIVE-9326 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Chengxiang Li Labels: Spark-M5 Attachments: HIVE-9326.1-spark.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9326) BaseProtocol.Error failed to deserialization due to NPE.[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-9326: Description: Throwables.getStackTraceAsString(cause) throw NPE if cause is null. BaseProtocol.Error failed to deserialization due to NPE.[Spark Branch] -- Key: HIVE-9326 URL: https://issues.apache.org/jira/browse/HIVE-9326 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Chengxiang Li Labels: Spark-M5 Attachments: HIVE-9326.1-spark.patch Throwables.getStackTraceAsString(cause) throw NPE if cause is null. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9249) java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to org.apache.hadoop.hive.common.type.HiveVarchar when joining tables
[
https://issues.apache.org/jira/browse/HIVE-9249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270834#comment-14270834
]
Gopal V commented on HIVE-9249:
---
All tests on 0.14 branch branch passes -
https://builds.apache.org/job/Hive-0.14/818/.
java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
org.apache.hadoop.hive.common.type.HiveVarchar when joining tables
---
Key: HIVE-9249
URL: https://issues.apache.org/jira/browse/HIVE-9249
Project: Hive
Issue Type: Bug
Components: Vectorization
Affects Versions: 0.14.0
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical
Fix For: 0.15.0, 0.14.1
Attachments: HIVE-9249.01.patch, HIVE-9249.hive-14.patch
VectorColumnAssignFactory doesn't handle HiveCharWritable /
HiveVarcharWritable objects.
Either:
HiveVarcharWritable cannot be cast to ... HiveVarchar
or
HiveCharWritable cannot be cast to ... HiveChar
{code}
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
org.apache.hadoop.hive.common.type.HiveVarchar
at
org.apache.hadoop.hive.ql.exec.vector.VectorColumnAssignFactory$17.assignObjectValue(VectorColumnAssignFactory.java:417)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.internalForward(VectorMapJoinOperator.java:196)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:670)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:748)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:299)
... 24 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9290) Make some test results deterministic
[
https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270835#comment-14270835
]
Hive QA commented on HIVE-9290:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12691076/HIVE-9290-spark.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 7301 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_windowing
org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/625/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/625/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-625/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12691076 - PreCommit-HIVE-SPARK-Build
Make some test results deterministic
Key: HIVE-9290
URL: https://issues.apache.org/jira/browse/HIVE-9290
Project: Hive
Issue Type: Test
Reporter: Rui Li
Assignee: Rui Li
Fix For: spark-branch, 0.15.0
Attachments: HIVE-9290-spark.patch, HIVE-9290.1.patch,
HIVE-9290.1.patch
{noformat}
limit_pushdown.q
optimize_nullscan.q
ppd_gby_join.q
vector_string_concat.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9330) DummyTxnManager will throw NPE if WriteEntity writeType has not been set
Chaoyu Tang created HIVE-9330:
-
Summary: DummyTxnManager will throw NPE if WriteEntity writeType
has not been set
Key: HIVE-9330
URL: https://issues.apache.org/jira/browse/HIVE-9330
Project: Hive
Issue Type: Bug
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang
Priority: Minor
DummyTxnManager line 259:
{code}
switch (we.getWriteType()) {
case DDL_EXCLUSIVE:
return HiveLockMode.EXCLUSIVE;
case DDL_SHARED:
return HiveLockMode.SHARED;
...
{code}
will throw out NPE in case the WriteEntity(we) writeType has not been set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 28964: HIVE-8121 Create micro-benchmarks for ParquetSerde and evaluate performance
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28964/ --- (Updated Ene. 9, 2015, 6:26 p.m.) Review request for hive. Changes --- These new changes converts the last patch to use the JMH framework Bugs: HIVE-8121 https://issues.apache.org/jira/browse/HIVE-8121 Repository: hive-git Description --- This is a new tool used to test ORC PARQUET file format performance. Diffs (updated) - itests/hive-jmh/pom.xml PRE-CREATION itests/hive-jmh/src/main/java/org/apache/hive/benchmark/storage/ColumnarStorageBench.java PRE-CREATION itests/pom.xml 0a154d6eb8c119e4e6419777c28b59b9d2108ba0 Diff: https://reviews.apache.org/r/28964/diff/ Testing --- Thanks, Sergio Pena
[jira] [Commented] (HIVE-8121) Create micro-benchmarks for ParquetSerde and evaluate performance
[ https://issues.apache.org/jira/browse/HIVE-8121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271653#comment-14271653 ] Sergio Peña commented on HIVE-8121: --- Hi All, I added a new patch to review that uses the JMH framework. https://reviews.apache.org/r/28964/diff/# Create micro-benchmarks for ParquetSerde and evaluate performance - Key: HIVE-8121 URL: https://issues.apache.org/jira/browse/HIVE-8121 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Sergio Peña These benchmarks should not execute queries but test only the ParquetSerde code to ensure we are as efficient as possible. The output of this JIRA is: 1) Benchmark tool exists 2) We create new tasks under HIVE-8120 to track the improvements required -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 28964: HIVE-8121 Create micro-benchmarks for ParquetSerde and evaluate performance
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28964/ --- (Updated Ene. 9, 2015, 6:38 p.m.) Review request for hive, Brock Noland and cheng xu. Bugs: HIVE-8121 https://issues.apache.org/jira/browse/HIVE-8121 Repository: hive-git Description --- This is a new tool used to test ORC PARQUET file format performance. Diffs - itests/hive-jmh/pom.xml PRE-CREATION itests/hive-jmh/src/main/java/org/apache/hive/benchmark/storage/ColumnarStorageBench.java PRE-CREATION itests/pom.xml 0a154d6eb8c119e4e6419777c28b59b9d2108ba0 Diff: https://reviews.apache.org/r/28964/diff/ Testing --- Thanks, Sergio Pena
[jira] [Created] (HIVE-9331) get rid of pre-optimized-hashtable memory optimizations
Sergey Shelukhin created HIVE-9331: -- Summary: get rid of pre-optimized-hashtable memory optimizations Key: HIVE-9331 URL: https://issues.apache.org/jira/browse/HIVE-9331 Project: Hive Issue Type: Bug Affects Versions: 0.15.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin These were added in 13 because optimized hashtable couldn't make it in; they reduced memory usage by some amount (10-25%), and informed the design of the optimized hashtable, but now extra settings and code branches are just confusing and may have their own bugs. Might as well remove them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9290) Make some test results deterministic
[
https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-9290:
Resolution: Fixed
Status: Resolved (was: Patch Available)
Make some test results deterministic
Key: HIVE-9290
URL: https://issues.apache.org/jira/browse/HIVE-9290
Project: Hive
Issue Type: Test
Reporter: Rui Li
Assignee: Rui Li
Fix For: spark-branch, 0.15.0
Attachments: HIVE-9290-spark.patch, HIVE-9290.1.patch,
HIVE-9290.1.patch
{noformat}
limit_pushdown.q
optimize_nullscan.q
ppd_gby_join.q
vector_string_concat.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9314) Writable object inspector should use Writable classes from Hadoop for get
[ https://issues.apache.org/jira/browse/HIVE-9314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271561#comment-14271561 ] Brock Noland commented on HIVE-9314: +1 LGTM [~jxiang] - could you check to ensure that TestPigHBaseStorageHandler passes locally with the patch? Writable object inspector should use Writable classes from Hadoop for get - Key: HIVE-9314 URL: https://issues.apache.org/jira/browse/HIVE-9314 Project: Hive Issue Type: Improvement Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: HIVE-9314.1.patch In WritableByteObjectInspector, we cast objects to org.apache.hadoop.hive.serde2.io.ByteWritable to get the value. We should use the hadoop one since org.apache.hadoop.hive.serde2.io.ByteWritable extends the hadoop one. If customer OI uses the hadoop writable, in some queries, they could get class cast exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9329) The qtest can't handle the statements which contains semicolons in qfile
[
https://issues.apache.org/jira/browse/HIVE-9329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-9329:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thank you much Ferd! I have committed this to branch!
The qtest can't handle the statements which contains semicolons in qfile
Key: HIVE-9329
URL: https://issues.apache.org/jira/browse/HIVE-9329
Project: Hive
Issue Type: Sub-task
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
Fix For: encryption-branch
Attachments: HIVE-9329-encryption.patch
The previous logic is trying to split the line by semicolons. The following
statement can't be handled.
compile `import org.apache.hadoop.hive.ql.exec.UDF \;
public class Pyth extends UDF {
public double evaluate(double a, double b){
return Math.sqrt((a*a) + (b*b)) \;
}
} `AS GROOVY NAMED Pyth.groovy;
CREATE TEMPORARY FUNCTION Pyth as 'Pyth';
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9328) Tests cannot move files due to change on HIVE-9325
[ https://issues.apache.org/jira/browse/HIVE-9328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-9328: --- Resolution: Fixed Fix Version/s: encryption-branch Status: Resolved (was: Patch Available) Thank you much Sergio! I have committed to branch. Tests cannot move files due to change on HIVE-9325 -- Key: HIVE-9328 URL: https://issues.apache.org/jira/browse/HIVE-9328 Project: Hive Issue Type: Sub-task Affects Versions: encryption-branch Reporter: Sergio Peña Assignee: Sergio Peña Fix For: encryption-branch Attachments: HIVE-9325.1.patch The change on HIVE-9325 that modified the isSubDir() method is checking that the source path is part of the source path? This always returns true and many tests are failing. The method should check if source path is part of destination path. This is the line that is failing: +String fullF1 = getQualifiedPathWithoutSchemeAndAuthority(srcf, fs); +String fullF2 = getQualifiedPathWithoutSchemeAndAuthority(srcf, fs); It should be: +String fullF1 = getQualifiedPathWithoutSchemeAndAuthority(srcf, fs); +String fullF2 = getQualifiedPathWithoutSchemeAndAuthority(destf, fs); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9330) DummyTxnManager will throw NPE if WriteEntity writeType has not been set
[
https://issues.apache.org/jira/browse/HIVE-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-9330:
--
Status: Patch Available (was: Open)
DummyTxnManager will throw NPE if WriteEntity writeType has not been set
Key: HIVE-9330
URL: https://issues.apache.org/jira/browse/HIVE-9330
Project: Hive
Issue Type: Bug
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang
Priority: Minor
Attachments: HIVE-9330.patch
DummyTxnManager line 259:
{code}
switch (we.getWriteType()) {
case DDL_EXCLUSIVE:
return HiveLockMode.EXCLUSIVE;
case DDL_SHARED:
return HiveLockMode.SHARED;
...
{code}
will throw out NPE in case the WriteEntity(we) writeType has not been set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9330) DummyTxnManager will throw NPE if WriteEntity writeType has not been set
[
https://issues.apache.org/jira/browse/HIVE-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-9330:
--
Attachment: HIVE-9330.patch
The patch checks if the writeType is null or not before it is used in switch.
DummyTxnManager will throw NPE if WriteEntity writeType has not been set
Key: HIVE-9330
URL: https://issues.apache.org/jira/browse/HIVE-9330
Project: Hive
Issue Type: Bug
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang
Priority: Minor
Attachments: HIVE-9330.patch
DummyTxnManager line 259:
{code}
switch (we.getWriteType()) {
case DDL_EXCLUSIVE:
return HiveLockMode.EXCLUSIVE;
case DDL_SHARED:
return HiveLockMode.SHARED;
...
{code}
will throw out NPE in case the WriteEntity(we) writeType has not been set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9320) Add UnionEliminatorRule on cbo path
[
https://issues.apache.org/jira/browse/HIVE-9320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271654#comment-14271654
]
Hive QA commented on HIVE-9320:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690977/HIVE-9320.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6739 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2308/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2308/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2308/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690977 - PreCommit-HIVE-TRUNK-Build
Add UnionEliminatorRule on cbo path
---
Key: HIVE-9320
URL: https://issues.apache.org/jira/browse/HIVE-9320
Project: Hive
Issue Type: Improvement
Components: CBO, Logical Optimizer
Affects Versions: 0.14.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Attachments: HIVE-9320.patch
Shorten the pipeline, where possible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9320) Add UnionEliminatorRule on cbo path
[ https://issues.apache.org/jira/browse/HIVE-9320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271743#comment-14271743 ] Ashutosh Chauhan commented on HIVE-9320: Failure is unrelated. [~jpullokkaran] this is ready for review as well. Add UnionEliminatorRule on cbo path --- Key: HIVE-9320 URL: https://issues.apache.org/jira/browse/HIVE-9320 Project: Hive Issue Type: Improvement Components: CBO, Logical Optimizer Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-9320.patch Shorten the pipeline, where possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9318) Add UnionMerge rule on cbo path
[ https://issues.apache.org/jira/browse/HIVE-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271626#comment-14271626 ] Ashutosh Chauhan commented on HIVE-9318: Failures are unrelated. [~jpullokkaran] this is ready for review. Add UnionMerge rule on cbo path --- Key: HIVE-9318 URL: https://issues.apache.org/jira/browse/HIVE-9318 Project: Hive Issue Type: Improvement Components: CBO, Logical Optimizer Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-9318.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8327) mvn site -Pfindbugs
[ https://issues.apache.org/jira/browse/HIVE-8327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-8327: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Gopal. Gopal, since you have report handy would you mind filing a jira for findbugs reported issues which you think are worth fixing. mvn site -Pfindbugs --- Key: HIVE-8327 URL: https://issues.apache.org/jira/browse/HIVE-8327 Project: Hive Issue Type: Test Components: Diagnosability Reporter: Gopal V Assignee: Gopal V Fix For: 0.15.0 Attachments: HIVE-8327.1.patch, HIVE-8327.2.patch, ql-findbugs.html HIVE-3099 originally added findbugs into the old ant build. Get basic findbugs working for the maven build. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9332) Warn user when parquet mm kicks in
Brock Noland created HIVE-9332: -- Summary: Warn user when parquet mm kicks in Key: HIVE-9332 URL: https://issues.apache.org/jira/browse/HIVE-9332 Project: Hive Issue Type: Sub-task Components: File Formats Reporter: Brock Noland In PARQUET-164 will add a warning mechanism when the mm kicks in. It'd be great if we could feed that warning back to the user in form of a console warning. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9290) Make some test results deterministic
[
https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-9290:
Assignee: Rui Li (was: Szehon Ho)
Make some test results deterministic
Key: HIVE-9290
URL: https://issues.apache.org/jira/browse/HIVE-9290
Project: Hive
Issue Type: Test
Reporter: Rui Li
Assignee: Rui Li
Fix For: spark-branch, 0.15.0
Attachments: HIVE-9290-spark.patch, HIVE-9290.1.patch,
HIVE-9290.1.patch
{noformat}
limit_pushdown.q
optimize_nullscan.q
ppd_gby_join.q
vector_string_concat.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-9290) Make some test results deterministic
[
https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho reassigned HIVE-9290:
---
Assignee: Szehon Ho (was: Rui Li)
Make some test results deterministic
Key: HIVE-9290
URL: https://issues.apache.org/jira/browse/HIVE-9290
Project: Hive
Issue Type: Test
Reporter: Rui Li
Assignee: Szehon Ho
Fix For: spark-branch, 0.15.0
Attachments: HIVE-9290-spark.patch, HIVE-9290.1.patch,
HIVE-9290.1.patch
{noformat}
limit_pushdown.q
optimize_nullscan.q
ppd_gby_join.q
vector_string_concat.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8931) Test TestAccumuloCliDriver is not completing
[
https://issues.apache.org/jira/browse/HIVE-8931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271525#comment-14271525
]
Brock Noland commented on HIVE-8931:
Yes the HMS has code which depends specifically on the 0.9.2 version of
thrift...
Test TestAccumuloCliDriver is not completing
Key: HIVE-8931
URL: https://issues.apache.org/jira/browse/HIVE-8931
Project: Hive
Issue Type: Bug
Reporter: Brock Noland
Assignee: Josh Elser
Tests are taking 3 hours due to {{TestAccumuloCliDriver}} not finishing.
Logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1848/failed/TestAccumuloCliDriver/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8931) Test TestAccumuloCliDriver is not completing
[
https://issues.apache.org/jira/browse/HIVE-8931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271537#comment-14271537
]
Brock Noland commented on HIVE-8931:
They actually don't start the server so you might get away with 0.9.1.
Test TestAccumuloCliDriver is not completing
Key: HIVE-8931
URL: https://issues.apache.org/jira/browse/HIVE-8931
Project: Hive
Issue Type: Bug
Reporter: Brock Noland
Assignee: Josh Elser
Tests are taking 3 hours due to {{TestAccumuloCliDriver}} not finishing.
Logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1848/failed/TestAccumuloCliDriver/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 29702: HIVE-9309 : schematool fails on Postgres 8.1
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29702/#review67460 --- Ship it! Ship It! - Brock Noland On Jan. 8, 2015, 8:16 a.m., Mohit Sabharwal wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29702/ --- (Updated Jan. 8, 2015, 8:16 a.m.) Review request for hive. Bugs: HIVE-9309 https://issues.apache.org/jira/browse/HIVE-9309 Repository: hive-git Description --- HIVE-9309 : schematool fails on Postgres 8.1 Postgres upgrade scripts set standard_conforming_strings which is not allowed in 8.1: ERROR: parameter standard_conforming_strings cannot be changed (state=55P02,code=0) Postgres 8.1 Release notes say that standard_conforming_strings value is read-only Postgres 8.2 notes say that it can be set at runtime. It'd be nice to address this for those still using Postgres 8.1 This patch provides a schemaTool db option postgres.filter.81 which, if set, filters out the standard_conforming_strings statement from upgrade scripts. Diffs - beeline/src/java/org/apache/hive/beeline/HiveSchemaHelper.java d13d8b67d09704ac8575b8f8bc80192569b06aba itests/hive-unit/src/test/java/org/apache/hive/beeline/TestSchemaTool.java 9ae9bc0a267e41a852aa6f87bab532763f9c54b3 Diff: https://reviews.apache.org/r/29702/diff/ Testing --- Added unit test Thanks, Mohit Sabharwal
[jira] [Updated] (HIVE-9329) The qtest can't handle the statements which contains semicolons in qfile
[
https://issues.apache.org/jira/browse/HIVE-9329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferdinand Xu updated HIVE-9329:
---
Attachment: HIVE-9329-encryption.patch
The qtest can't handle the statements which contains semicolons in qfile
Key: HIVE-9329
URL: https://issues.apache.org/jira/browse/HIVE-9329
Project: Hive
Issue Type: Sub-task
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
Fix For: encryption-branch
Attachments: HIVE-9329-encryption.patch
The previous logic is trying to split the line by semicolons. The following
statement can't be handled.
compile `import org.apache.hadoop.hive.ql.exec.UDF \;
public class Pyth extends UDF {
public double evaluate(double a, double b){
return Math.sqrt((a*a) + (b*b)) \;
}
} `AS GROOVY NAMED Pyth.groovy;
CREATE TEMPORARY FUNCTION Pyth as 'Pyth';
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9329) The qtest can't handle the statements which contains semicolons in qfile
[
https://issues.apache.org/jira/browse/HIVE-9329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferdinand Xu updated HIVE-9329:
---
Status: Patch Available (was: Open)
The qtest can't handle the statements which contains semicolons in qfile
Key: HIVE-9329
URL: https://issues.apache.org/jira/browse/HIVE-9329
Project: Hive
Issue Type: Sub-task
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
Fix For: encryption-branch
Attachments: HIVE-9329-encryption.patch
The previous logic is trying to split the line by semicolons. The following
statement can't be handled.
compile `import org.apache.hadoop.hive.ql.exec.UDF \;
public class Pyth extends UDF {
public double evaluate(double a, double b){
return Math.sqrt((a*a) + (b*b)) \;
}
} `AS GROOVY NAMED Pyth.groovy;
CREATE TEMPORARY FUNCTION Pyth as 'Pyth';
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8931) Test TestAccumuloCliDriver is not completing
[
https://issues.apache.org/jira/browse/HIVE-8931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271529#comment-14271529
]
Josh Elser commented on HIVE-8931:
--
bq. Yes the HMS has code which depends specifically on the 0.9.2 version of
thrift...
I meant I'm assuming that the QTests themselves are exercising the metastore in
such a way that the thrift dependency is directly needed (and not doing some
mock thing).
Test TestAccumuloCliDriver is not completing
Key: HIVE-8931
URL: https://issues.apache.org/jira/browse/HIVE-8931
Project: Hive
Issue Type: Bug
Reporter: Brock Noland
Assignee: Josh Elser
Tests are taking 3 hours due to {{TestAccumuloCliDriver}} not finishing.
Logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1848/failed/TestAccumuloCliDriver/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9318) Add UnionMerge rule on cbo path
[
https://issues.apache.org/jira/browse/HIVE-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271545#comment-14271545
]
Hive QA commented on HIVE-9318:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690969/HIVE-9318.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6739 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_handler_bulk
org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2307/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2307/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2307/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690969 - PreCommit-HIVE-TRUNK-Build
Add UnionMerge rule on cbo path
---
Key: HIVE-9318
URL: https://issues.apache.org/jira/browse/HIVE-9318
Project: Hive
Issue Type: Improvement
Components: CBO, Logical Optimizer
Affects Versions: 0.14.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Attachments: HIVE-9318.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9314) Writable object inspector should use Writable classes from Hadoop for get
[ https://issues.apache.org/jira/browse/HIVE-9314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271577#comment-14271577 ] Jimmy Xiang commented on HIVE-9314: --- Yes, TestPigHBaseStorageHandler passes locally with the patch for me. Thanks. Writable object inspector should use Writable classes from Hadoop for get - Key: HIVE-9314 URL: https://issues.apache.org/jira/browse/HIVE-9314 Project: Hive Issue Type: Improvement Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: HIVE-9314.1.patch In WritableByteObjectInspector, we cast objects to org.apache.hadoop.hive.serde2.io.ByteWritable to get the value. We should use the hadoop one since org.apache.hadoop.hive.serde2.io.ByteWritable extends the hadoop one. If customer OI uses the hadoop writable, in some queries, they could get class cast exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29635: HiveServer2 leaks FileSystem objects in FileSystem.CACHE
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29635/#review67455 --- Hi, I happened to see this randomly. Since you are cleaning up some stuff, may I request the two cleanups below? Please feel free to push them off to another jira if you've already updated the patch! service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java https://reviews.apache.org/r/29635/#comment111478 Would you mind updating: fail to load reloadable jar file path + e; to Failed to load reloadable jar file path: + e; service/src/java/org/apache/hive/service/cli/session/SessionManager.java https://reviews.apache.org/r/29635/#comment111487 would you mind updating Failed to open new session, e to Failed to open new session: + e, e - Brock Noland On Jan. 6, 2015, 7:28 p.m., Vaibhav Gumashta wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29635/ --- (Updated Jan. 6, 2015, 7:28 p.m.) Review request for hive and Thejas Nair. Bugs: HIVE-9234 https://issues.apache.org/jira/browse/HIVE-9234 Repository: hive-git Description --- https://issues.apache.org/jira/browse/HIVE-9234 Diffs - service/src/java/org/apache/hive/service/cli/session/HiveSession.java 6359a5b service/src/java/org/apache/hive/service/cli/session/HiveSessionBase.java 50094d2 service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java cabf32a service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java 7acd113 service/src/java/org/apache/hive/service/cli/session/SessionManager.java 137359b Diff: https://reviews.apache.org/r/29635/diff/ Testing --- Thanks, Vaibhav Gumashta
Re: Review Request 29625: HIVE-9200
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29625/#review67459 --- ql/src/java/org/apache/hadoop/hive/ql/optimizer/NonBlockingOpDeDupProc.java https://reviews.apache.org/r/29625/#comment111491 Please trim the trailing whitespace. It shows up in red. - Brock Noland On Jan. 9, 2015, 1:42 p.m., Jesús Camacho Rodríguez wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29625/ --- (Updated Jan. 9, 2015, 1:42 p.m.) Review request for hive and John Pullokkaran. Bugs: HIVE-9200 https://issues.apache.org/jira/browse/HIVE-9200 Repository: hive-git Description --- CBO (Calcite Return Path): Inline Join, Properties Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java ed03bb92bc2cc3c08cd7357ad4053dfd6aa739d3 ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractBucketJoinProc.java 70c23a693bbb4d11808452bc24e67ffbb441b05e ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractSMBJoinProc.java c9e808605da4218605a402fd0099469cdc20a289 ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketMapjoinProc.java 1260c831d86ae770cf7f9f0356932f9e0fdf7ba8 ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 7ab35eec5987c78dee0349431e06ee65a20ee2cd ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 9a74e1e63e8d210f1d41909acf170f5279bd1d14 ql/src/java/org/apache/hadoop/hive/ql/optimizer/JoinReorder.java 9238e0e541b748f5e45fe572e6b4575cc3299b7f ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java ccb3ce54797abbb6d1f58d9d260194b67d7ec77f ql/src/java/org/apache/hadoop/hive/ql/optimizer/NonBlockingOpDeDupProc.java 5291851b105730490033ff91e583ee44022ed24f ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java ea06503b0377ffb98f2583869e2c51ac1ea4e398 ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedMergeBucketMapjoinProc.java 11ce47eb4ff4b8ae1162eb5f3842b8e32d3a21e1 ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedMergeJoinProc.java 8a0c47477718141cab85a4d6f71070117372df91 ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationOptimizer.java c52f7530b10c81a662118d2cb43599c82f7dbb4f ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/AbstractJoinTaskDispatcher.java 33ef581a97768d6391c67558e768d10e46a366f2 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java 9c26907544ad8ced31d5cf47ed27c8a240f93925 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SortMergeJoinTaskDispatcher.java 6f92b13ff7c1cdd4c651f5e1bff42626dee52750 ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 8215c26f48f1611d3c64f6df5bcfac02069e3a67 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c2d5c8c448f784397e5f08ffb0f386726df3ff41 ql/src/java/org/apache/hadoop/hive/ql/parse/TableAccessAnalyzer.java da14ab4e96bcc9089e10eb3a9d4e5d575b51d5ab ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 23fbbe11198ac5893a84bdf94f9c843c4ee2ccb4 ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java c144d8c05c73025ba33b300229125e74930e ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 9f8c0918179d9226e36cecc3bd955946d6b5fe98 Diff: https://reviews.apache.org/r/29625/diff/ Testing --- Existing tests. Thanks, Jesús Camacho Rodríguez
[jira] [Commented] (HIVE-8966) Delta files created by hive hcatalog streaming cannot be compacted
[ https://issues.apache.org/jira/browse/HIVE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271601#comment-14271601 ] Jihong Liu commented on HIVE-8966: -- Make sense. It is so great if that solution can be implemented.Thanks Delta files created by hive hcatalog streaming cannot be compacted -- Key: HIVE-8966 URL: https://issues.apache.org/jira/browse/HIVE-8966 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.14.0 Environment: hive Reporter: Jihong Liu Assignee: Alan Gates Priority: Critical Fix For: 0.14.1 Attachments: HIVE-8966.2.patch, HIVE-8966.3.patch, HIVE-8966.patch hive hcatalog streaming will also create a file like bucket_n_flush_length in each delta directory. Where n is the bucket number. But the compactor.CompactorMR think this file also needs to compact. However this file of course cannot be compacted, so compactor.CompactorMR will not continue to do the compaction. Did a test, after removed the bucket_n_flush_length file, then the alter table partition compact finished successfully. If don't delete that file, nothing will be compacted. This is probably a very severity bug. Both 0.13 and 0.14 have this issue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9329) The qtest can't handle the statements which contains semicolons in qfile
[
https://issues.apache.org/jira/browse/HIVE-9329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271580#comment-14271580
]
Brock Noland commented on HIVE-9329:
=1
The qtest can't handle the statements which contains semicolons in qfile
Key: HIVE-9329
URL: https://issues.apache.org/jira/browse/HIVE-9329
Project: Hive
Issue Type: Sub-task
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
Fix For: encryption-branch
Attachments: HIVE-9329-encryption.patch
The previous logic is trying to split the line by semicolons. The following
statement can't be handled.
compile `import org.apache.hadoop.hive.ql.exec.UDF \;
public class Pyth extends UDF {
public double evaluate(double a, double b){
return Math.sqrt((a*a) + (b*b)) \;
}
} `AS GROOVY NAMED Pyth.groovy;
CREATE TEMPORARY FUNCTION Pyth as 'Pyth';
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9329) The qtest can't handle the statements which contains semicolons in qfile
[
https://issues.apache.org/jira/browse/HIVE-9329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271579#comment-14271579
]
Sergio Peña commented on HIVE-9329:
---
+1
Thanks Ferd
The qtest can't handle the statements which contains semicolons in qfile
Key: HIVE-9329
URL: https://issues.apache.org/jira/browse/HIVE-9329
Project: Hive
Issue Type: Sub-task
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
Fix For: encryption-branch
Attachments: HIVE-9329-encryption.patch
The previous logic is trying to split the line by semicolons. The following
statement can't be handled.
compile `import org.apache.hadoop.hive.ql.exec.UDF \;
public class Pyth extends UDF {
public double evaluate(double a, double b){
return Math.sqrt((a*a) + (b*b)) \;
}
} `AS GROOVY NAMED Pyth.groovy;
CREATE TEMPORARY FUNCTION Pyth as 'Pyth';
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9328) Tests cannot move files due to change on HIVE-9325
[ https://issues.apache.org/jira/browse/HIVE-9328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-9328: -- Status: Patch Available (was: Open) Tests cannot move files due to change on HIVE-9325 -- Key: HIVE-9328 URL: https://issues.apache.org/jira/browse/HIVE-9328 Project: Hive Issue Type: Sub-task Affects Versions: encryption-branch Reporter: Sergio Peña Assignee: Sergio Peña Attachments: HIVE-9325.1.patch The change on HIVE-9325 that modified the isSubDir() method is checking that the source path is part of the source path? This always returns true and many tests are failing. The method should check if source path is part of destination path. This is the line that is failing: +String fullF1 = getQualifiedPathWithoutSchemeAndAuthority(srcf, fs); +String fullF2 = getQualifiedPathWithoutSchemeAndAuthority(srcf, fs); It should be: +String fullF1 = getQualifiedPathWithoutSchemeAndAuthority(srcf, fs); +String fullF2 = getQualifiedPathWithoutSchemeAndAuthority(destf, fs); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
hive 0.14.1 + tez 0.4 sometimes throws NPE on small data
I noticed that if amount of data which should be sent to reducers is small
then tez might throw NPE.
workaround is to set number of reducers to some small value.
e.g. if initially I have
set hive.exec.reducers.max=25;
then to solve the issue I might try
set hive.exec.reducers.max=13;
or even smaller value
I'm using hdp-2.1.5 (which is hive-0.14.1 + tez-0.4.0)
Map 1: 17/21Map 11: 1/1 Map 14: 4/4 Map 16: 4/4 Map 18:
4/4 Map 19: 1/1 Map 21: 1/1 Map 22: 5/5 Map 23: 18/21 Map
24: 1/1 Map 4: 21/21Map 6: 1/1 Map 9: 1/11
Status: Failed
Vertex re-running, vertexName=Map 6,
vertexId=vertex_1420058390346_15040_1_20
Vertex re-running, vertexName=Map 18,
vertexId=vertex_1420058390346_15040_1_10
Vertex re-running, vertexName=Map 22,
vertexId=vertex_1420058390346_15040_1_14
Vertex re-running, vertexName=Map 4,
vertexId=vertex_1420058390346_15040_1_22
Vertex re-running, vertexName=Map 9,
vertexId=vertex_1420058390346_15040_1_17
Vertex re-running, vertexName=Map 19,
vertexId=vertex_1420058390346_15040_1_02
Vertex re-running, vertexName=Map 24,
vertexId=vertex_1420058390346_15040_1_09
Vertex re-running, vertexName=Map 1,
vertexId=vertex_1420058390346_15040_1_24
Vertex re-running, vertexName=Map 14,
vertexId=vertex_1420058390346_15040_1_15
Vertex re-running, vertexName=Map 16,
vertexId=vertex_1420058390346_15040_1_12
Vertex re-running, vertexName=Map 14,
vertexId=vertex_1420058390346_15040_1_15
Vertex re-running, vertexName=Map 6,
vertexId=vertex_1420058390346_15040_1_20
Vertex re-running, vertexName=Map 18,
vertexId=vertex_1420058390346_15040_1_10
Vertex re-running, vertexName=Map 22,
vertexId=vertex_1420058390346_15040_1_14
Vertex re-running, vertexName=Map 21,
vertexId=vertex_1420058390346_15040_1_19
Vertex re-running, vertexName=Map 16,
vertexId=vertex_1420058390346_15040_1_12
Vertex re-running, vertexName=Map 24,
vertexId=vertex_1420058390346_15040_1_09
Vertex re-running, vertexName=Map 11,
vertexId=vertex_1420058390346_15040_1_06
Vertex failed, vertexName=Reducer 17,
vertexId=vertex_1420058390346_15040_1_11, diagnostics=[Task failed,
taskId=task_1420058390346_15040_1_11_07,
diagnostics=[AttemptID:attempt_1420058390346_15040_1_11}
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:188)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:564)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:553)
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row (tag=0)
{key:{reducesinkkey0:61419,reducesinkkey1:2011-10-02 13:04:53}
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:349)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:228)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:165)
... 6 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row (tag=0)
{key:{reducesinkkey0:61419,reducesinkkey1:2011-10-02
13:04:53,reducesinkkey2:2011-11-0}
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:334)
... 8 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:341)
at
org.apache.hadoop.hive.ql.exec.PTFOperator.processOp(PTFOperator.java:115)
at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
at
org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:325)
... 8 more
Container released by application,
AttemptID:attempt_1420058390346_15040_1_11_07_1 Info:Node blacklisted,
AttemptID:attempt_1420058390346_15040_1_11_07_2 Info:Node blacklisted
Container released by application,
AttemptID:attempt_1420058390346_15040_1_11_07_3 Info:Error:
java.lang.RuntimeException: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: H}
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:188)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:564)
at java.security.AccessController.doPrivileged(Native Method)
at
[jira] [Updated] (HIVE-9326) BaseProtocol.Error failed to deserialization due to NPE.[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-9326: -- Resolution: Fixed Fix Version/s: spark-branch Status: Resolved (was: Patch Available) The test failures are known and unrelated. Committed to Spark branch. Thanks, Chengxiang. BaseProtocol.Error failed to deserialization due to NPE.[Spark Branch] -- Key: HIVE-9326 URL: https://issues.apache.org/jira/browse/HIVE-9326 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Chengxiang Li Labels: Spark-M5 Fix For: spark-branch Attachments: HIVE-9326.1-spark.patch Throwables.getStackTraceAsString(cause) throw NPE if cause is null. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9331) get rid of pre-optimized-hashtable memory optimizations
[ https://issues.apache.org/jira/browse/HIVE-9331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-9331: --- Attachment: HIVE-9331.patch mostly removing code, there was also a small bug in one of the classes. Path for now has optimized hashtable itself disabled, so that tests could run on modified code; PLEASE DO NOT COMMIT this patch even if tests pass and there's +1 get rid of pre-optimized-hashtable memory optimizations --- Key: HIVE-9331 URL: https://issues.apache.org/jira/browse/HIVE-9331 Project: Hive Issue Type: Bug Affects Versions: 0.15.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.15.0 Attachments: HIVE-9331.patch These were added in 13 because optimized hashtable couldn't make it in; they reduced memory usage by some amount (10-25%), and informed the design of the optimized hashtable, but now extra settings and code branches are just confusing and may have their own bugs. Might as well remove them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
