Re: Review Request 21138: Support more generic way of using composite key for HBaseHandler
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21138/#review42436
---
hbase-handler/src/java/org/apache/hadoop/hive/hbase/CompositeHBaseKeyFactory.java
https://reviews.apache.org/r/21138/#comment76234
Is the comment meant for setupFilter()?
hbase-handler/src/java/org/apache/hadoop/hive/hbase/CompositeHBaseKeyFactory.java
https://reviews.apache.org/r/21138/#comment76233
Is FamilyFilter appropriate here?
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java
https://reviews.apache.org/r/21138/#comment76235
Can we have some comments here? I had difficulty understanding this.
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java
https://reviews.apache.org/r/21138/#comment76236
Same as above.
serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java
https://reviews.apache.org/r/21138/#comment76237
I don't see any use of this method.
- Xuefu Zhang
On May 6, 2014, 11:26 p.m., Swarnim Kulkarni wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21138/
---
(Updated May 6, 2014, 11:26 p.m.)
Review request for hive.
Repository: hive-git
Description
---
HIVE-2599 introduced using custom object for the row key. But it forces key
objects to extend HBaseCompositeKey, which is again extension of LazyStruct.
If user provides proper Object and OI, we can replace internal key and keyOI
with those.
Initial implementation is based on factory interface.
{code}
public interface HBaseKeyFactory {
void init(SerDeParameters parameters, Properties properties) throws
SerDeException;
ObjectInspector createObjectInspector(TypeInfo type) throws SerDeException;
LazyObjectBase createObject(ObjectInspector inspector) throws
SerDeException;
}
{code}
Diffs
-
hbase-handler/pom.xml 132af43
hbase-handler/src/java/org/apache/hadoop/hive/hbase/AbstractHBaseKeyFactory.java
PRE-CREATION
hbase-handler/src/java/org/apache/hadoop/hive/hbase/ColumnMappings.java
PRE-CREATION
hbase-handler/src/java/org/apache/hadoop/hive/hbase/CompositeHBaseKeyFactory.java
PRE-CREATION
hbase-handler/src/java/org/apache/hadoop/hive/hbase/DefaultHBaseKeyFactory.java
PRE-CREATION
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseCompositeKey.java
5008f15
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseKeyFactory.java
PRE-CREATION
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseLazyObjectFactory.java
PRE-CREATION
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java
PRE-CREATION
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseScanRange.java
PRE-CREATION
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java
b64590d
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
4fe1b1b
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
142bfd8
hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java
fc40195
hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestCompositeKey.java
13c344b
hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseKeyFactory.java
PRE-CREATION
hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseKeyFactory2.java
PRE-CREATION
hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java
7c4fc9f
hbase-handler/src/test/queries/positive/hbase_custom_key.q PRE-CREATION
hbase-handler/src/test/queries/positive/hbase_custom_key2.q PRE-CREATION
hbase-handler/src/test/results/positive/hbase_custom_key.q.out PRE-CREATION
hbase-handler/src/test/results/positive/hbase_custom_key2.q.out
PRE-CREATION
itests/util/pom.xml e9720df
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 113227d
ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java
d39ee2e
ql/src/java/org/apache/hadoop/hive/ql/index/IndexSearchCondition.java
5f1329c
ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 4921966
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcStruct.java 293b74e
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ArrayWritableObjectInspector.java
2a7fdf9
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStoragePredicateHandler.java
9f35575
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java e50026b
ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java ecb82d7
[jira] [Updated] (HIVE-5823) Support for DECIMAL primitive type in AvroSerDe
[ https://issues.apache.org/jira/browse/HIVE-5823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-5823: -- Release Note: The feature may need to be properly documented on Hive avro support. Support for DECIMAL primitive type in AvroSerDe --- Key: HIVE-5823 URL: https://issues.apache.org/jira/browse/HIVE-5823 Project: Hive Issue Type: New Feature Components: Serializers/Deserializers Affects Versions: 0.12.0 Reporter: Mariano Dominguez Assignee: Xuefu Zhang Labels: avro, serde Fix For: 0.14.0 Attachments: HIVE-5823.1.patch, HIVE-5823.2.patch, HIVE-5823.3.patch, HIVE-5823.4.patch, HIVE-5823.5.patch, HIVE-5823.6.patch, HIVE-5823.7.patch, HIVE-5823.patch, dec.avro This new feature request would be tied to AVRO-1402. Adding DECIMAL support would be particularly interesting when converting types from Avro to Hive, since DECIMAL is already a supported data type in Hive. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5823) Support for DECIMAL primitive type in AvroSerDe
[ https://issues.apache.org/jira/browse/HIVE-5823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-5823: --- Resolution: Fixed Fix Version/s: 0.14.0 Status: Resolved (was: Patch Available) Thank you Xuefu for your contribution! I have committed this to trunk. Support for DECIMAL primitive type in AvroSerDe --- Key: HIVE-5823 URL: https://issues.apache.org/jira/browse/HIVE-5823 Project: Hive Issue Type: New Feature Components: Serializers/Deserializers Affects Versions: 0.12.0 Reporter: Mariano Dominguez Assignee: Xuefu Zhang Labels: avro, serde Fix For: 0.14.0 Attachments: HIVE-5823.1.patch, HIVE-5823.2.patch, HIVE-5823.3.patch, HIVE-5823.4.patch, HIVE-5823.5.patch, HIVE-5823.6.patch, HIVE-5823.7.patch, HIVE-5823.patch, dec.avro This new feature request would be tied to AVRO-1402. Adding DECIMAL support would be particularly interesting when converting types from Avro to Hive, since DECIMAL is already a supported data type in Hive. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7037) Add additional tests for transform clauses with Tez
[ https://issues.apache.org/jira/browse/HIVE-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-7037: - Description: Enabling some q tests for Tez wrt to ScriptOperator/Stream/Transform. Add additional tests for transform clauses with Tez --- Key: HIVE-7037 URL: https://issues.apache.org/jira/browse/HIVE-7037 Project: Hive Issue Type: Bug Components: Tez Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-7037.1.patch Enabling some q tests for Tez wrt to ScriptOperator/Stream/Transform. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 21095: HIVE-7015 Failing to inherit group/permission should not fail the operation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21095/ --- (Updated May 9, 2014, 7:13 p.m.) Review request for hive and Brock Noland. Changes --- The diff did not have some files (with minor call changes), adding them. Bugs: HIVE-7015 https://issues.apache.org/jira/browse/HIVE-7015 Repository: hive-git Description --- There was some reported permission errors hit in Fs.setOwner during table creation, when the inherit permission flag was on. I realized that HDFS follows the BSD rule and anyway sets new directory to be the same group as the parent, so actually no need to call 'setOwner' for changing the group during mkdirs. Minor cleanups elsewhere. Changed the other call in mkdirs (Fs.setPermission) to use the shell, so it doesnt throw an error (although I don't see why it should, as folder should be owned by the current user whether its impersonated user or hive). Also changing other places as well to not throw error on failure, although again they shouldn't have failed as these were already using the shell. Diffs (updated) - common/src/java/org/apache/hadoop/hive/common/FileUtils.java 23a4b8e metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java 6d9b559 ql/src/java/org/apache/hadoop/hive/ql/exec/CopyTask.java 43064b3 ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java d8ad873 Diff: https://reviews.apache.org/r/21095/diff/ Testing --- Ran TestFolderPermissions. Thanks, Szehon Ho
Re: Review Request 20899: HIVE-6994 - parquet-hive createArray strips null elements
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20899/#review42416 --- Mostly looks good, some comments. ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java https://reviews.apache.org/r/20899/#comment76184 Is it necessary to remove final? ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java https://reviews.apache.org/r/20899/#comment76183 Is it necessary to remove final on these? - Szehon Ho On April 30, 2014, 5:22 p.m., justin coffey wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20899/ --- (Updated April 30, 2014, 5:22 p.m.) Review request for hive. Repository: hive-git Description --- - Fix for bug in createArray() that strips null elements. - In the process refactored serde for simplification purposes. - Refactored tests for better regression testing. Diffs - data/files/parquet_create.txt ccd48ee ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java b689336 ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetSerDe.java be518b9 ql/src/test/org/apache/hadoop/hive/ql/io/parquet/serde/TestParquetHiveSerDe.java PRE-CREATION ql/src/test/queries/clientpositive/parquet_create.q 0b976bd ql/src/test/results/clientpositive/parquet_create.q.out 3220be5 Diff: https://reviews.apache.org/r/20899/diff/ Testing --- Thanks, justin coffey
[jira] [Updated] (HIVE-7012) Wrong RS de-duplication in the ReduceSinkDeDuplication Optimizer
[
https://issues.apache.org/jira/browse/HIVE-7012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-7012:
Attachment: HIVE-7012.2.patch.txt
Wrong RS de-duplication in the ReduceSinkDeDuplication Optimizer
Key: HIVE-7012
URL: https://issues.apache.org/jira/browse/HIVE-7012
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.13.0
Reporter: Sun Rui
Attachments: HIVE-7012.1.patch.txt, HIVE-7012.2.patch.txt
With HIVE 0.13.0, run the following test case:
{code:sql}
create table src(key bigint, value string);
select
count(distinct key) as col0
from src
order by col0;
{code}
The following exception will be thrown:
{noformat}
java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:485)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 9 more
Caused by: java.lang.RuntimeException: Reduce operator initialization failed
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:173)
... 14 more
Caused by: java.lang.RuntimeException: cannot find field _col0 from
[0:reducesinkkey0]
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:415)
at
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:150)
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:79)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.initializeOp(GroupByOperator.java:288)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:166)
... 14 more
{noformat}
This issue is related to HIVE-6455. When hive.optimize.reducededuplication is
set to false, then this issue will be gone.
Logical plan when hive.optimize.reducededuplication=false;
{noformat}
src
TableScan (TS_0)
alias: src
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
Select Operator (SEL_1)
expressions: key (type: bigint)
outputColumnNames: key
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats:
NONE
Group By Operator (GBY_2)
aggregations: count(DISTINCT key)
keys: key (type: bigint)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats:
NONE
Reduce Output Operator (RS_3)
istinctColumnIndices:
key expressions: _col0 (type: bigint)
DistributionKeys: 0
sort order: +
OutputKeyColumnNames: _col0
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column
stats: NONE
Group By Operator (GBY_4)
aggregations: count(DISTINCT KEY._col0:0._col0)
mode: mergepartial
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE
Column stats: NONE
Select Operator (SEL_5)
expressions: _col0 (type: bigint)
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE
Column stats: NONE
Reduce Output Operator (RS_6)
key expressions: _col0 (type: bigint)
DistributionKeys: 1
sort order: +
OutputKeyColumnNames: reducesinkkey0
OutputVAlueColumnNames: _col0
[jira] [Created] (HIVE-7038) Join of external tables of elasticsearch giving an error.
Ashwin SL created HIVE-7038: --- Summary: Join of external tables of elasticsearch giving an error. Key: HIVE-7038 URL: https://issues.apache.org/jira/browse/HIVE-7038 Project: Hive Issue Type: Bug Components: Query Processor, Serializers/Deserializers Affects Versions: 0.13.0, 0.12.0 Reporter: Ashwin SL Select * is working while the Join of the tables is giving the following error: hive select * from failedauth f, failedauth2 f1 where f.username=f1.username; Total jobs = 1 14/05/09 10:57:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/05/09 10:57:11 WARN conf.Configuration: file:/tmp/hduser/hive_2014-05-09_10-57-09_954_5441752347301140125-1/-local-10006/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 14/05/09 10:57:11 WARN conf.Configuration: file:/tmp/hduser/hive_2014-05-09_10-57-09_954_5441752347301140125-1/-local-10006/jobconf.xml:an attempt to override final parameter: mapreduce.jobtracker.system.dir; Ignoring. 14/05/09 10:57:11 WARN conf.Configuration: file:/tmp/hduser/hive_2014-05-09_10-57-09_954_5441752347301140125-1/-local-10006/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 14/05/09 10:57:12 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces 14/05/09 10:57:12 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize 14/05/09 10:57:12 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative 14/05/09 10:57:12 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node 14/05/09 10:57:12 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive 14/05/09 10:57:12 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack 14/05/09 10:57:12 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize 14/05/09 10:57:12 INFO Configuration.deprecation: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed Execution log at: /tmp/hduser/hduser_20140509105757_945cc986-7fb1-491e-9bc1-a17cc150c6c6.log 2014-05-09 10:57:12 Starting to launch local task to process map join; maximum memory = 503840768 Execution failed with exit status: 2 Obtaining error information Task failed! Task ID: Stage-4 Logs: /tmp/hduser/hive.log FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask The Following exception was seen in /tmp/hduser/hive.log 2014-05-07 15:31:58,942 INFO mr.ExecDriver (SessionState.java:printInfo(410)) - Execution log at: /tmp/hduser/.log 2014-05-07 15:31:59,016 WARN conf.Configuration (Configuration.java:loadProperty(2172)) - file:/tmp/hduser/hive_2014-05-07_15-31-57_274_2380982880290259806-1/-local-10005/jobconf.xml:an attempt to override final parameter: dfs.datanode.data.dir; Ignoring. 2014-05-07 15:31:59,017 WARN conf.Configuration (Configuration.java:loadProperty(2172)) - file:/tmp/hduser/hive_2014-05-07_15-31-57_274_2380982880290259806-1/-local-10005/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 2014-05-07 15:31:59,019 WARN conf.Configuration (Configuration.java:loadProperty(2172)) - file:/tmp/hduser/hive_2014-05-07_15-31-57_274_2380982880290259806-1/-local-10005/jobconf.xml:an attempt to override final parameter: dfs.namenode.name.dir; Ignoring. 2014-05-07 15:31:59,020 WARN conf.Configuration (Configuration.java:loadProperty(2172)) - file:/tmp/hduser/hive_2014-05-07_15-31-57_274_2380982880290259806-1/-local-10005/jobconf.xml:an attempt to override final parameter: dfs.namenode.name.dir; Ignoring. 2014-05-07 15:31:59,020 WARN conf.Configuration (Configuration.java:loadProperty(2172)) - file:/tmp/hduser/hive_2014-05-07_15-31-57_274_2380982880290259806-1/-local-10005/jobconf.xml:an attempt to override final parameter: mapreduce.jobtracker.system.dir; Ignoring. 2014-05-07 15:31:59,021 WARN conf.Configuration (Configuration.java:loadProperty(2172)) - file:/tmp/hduser/hive_2014-05-07_15-31-57_274_2380982880290259806-1/-local-10005/jobconf.xml:an attempt to override final parameter: dfs.datanode.data.dir; Ignoring. 2014-05-07 15:31:59,021 WARN conf.Configuration (Configuration.java:loadProperty(2172)) -
[jira] [Commented] (HIVE-7016) Hive returns wrong results when execute UDF on top of DISTINCT column
[ https://issues.apache.org/jira/browse/HIVE-7016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993438#comment-13993438 ] Navis commented on HIVE-7016: - [~selinazh] You are right. Fail of testUniqueErrorCode is caused by that. Thanks. Hive returns wrong results when execute UDF on top of DISTINCT column - Key: HIVE-7016 URL: https://issues.apache.org/jira/browse/HIVE-7016 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.12.0, 0.13.1 Reporter: Selina Zhang Assignee: Navis Fix For: 0.14.0 Attachments: HIVE-7016.1.patch.txt The following query returns wrong result: select hash(distinct value) from table; This kind of query should be identified as syntax error. However, Hive ignores DISTINCT and returns the result. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7039) Hive run on MR2 must set mapreduce.jobtracker.address
[
https://issues.apache.org/jira/browse/HIVE-7039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fengdong Yu updated HIVE-7039:
--
Description:
I am using hive-0.13.0 and hadoop-2.4.0,
why I must set 'mapreduce.jobtracker.address' in yarn-site.xml? otherwise,
there are exceptions and job failed.
And, 'mapreduce.jobtracker.address' can be set to any value.
The following messages are gened without set 'mapreduce.jobtracker.address'.
Job output on the console:
{code}
Execution log at:
/tmp/test/test_20140507180505_bcd4d89f-017c-4cf4-81a3-5fa619de0ad0.log
Job running in-process (local Hadoop)
Hadoop job information for null: number of mappers: 1; number of reducers: 1
2014-05-07 18:06:25,782 null map = 0%, reduce = 0%
2014-05-07 18:06:33,699 null map = 100%, reduce = 0%
2014-05-07 18:06:34,774 null map = 0%, reduce = 0%
2014-05-07 18:06:49,222 null map = 100%, reduce = 100%
Ended Job = job_1399453944131_0006 with errors
Error during job, obtaining debugging information...
{code}
Container error:
{code}
2014-05-07 18:06:33,634 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities:
No plan file found:
file:/tmp/test/hive_2014-05-07_18-06-08_349_1526907284076641211-1/-mr-10001/0a1c9ebe-cdb0-4adc-9e93-8f176019f19a/map.xml
2014-05-07 18:06:33,635 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:437)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:430)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.init(MapTask.java:168)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:409)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
{code}
was:
I am using hive-0.13.0 and hadoop-2.4.0,
why I must set 'mapreduce.jobtracker.address' in yarn-site.xml? otherwise,
there are exceptions and job failed.
And, 'mapreduce.jobtracker.address' can be set to any value.
The following messages are gened without set 'mapreduce.jobtracker.address'.
Job output on the console:
Execution log at:
/tmp/test/test_20140507180505_bcd4d89f-017c-4cf4-81a3-5fa619de0ad0.log
Job running in-process (local Hadoop)
Hadoop job information for null: number of mappers: 1; number of reducers: 1
2014-05-07 18:06:25,782 null map = 0%, reduce = 0%
2014-05-07 18:06:33,699 null map = 100%, reduce = 0%
2014-05-07 18:06:34,774 null map = 0%, reduce = 0%
2014-05-07 18:06:49,222 null map = 100%, reduce = 100%
Ended Job = job_1399453944131_0006 with errors
Error during job, obtaining debugging information...
Container error:
2014-05-07 18:06:33,634 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities:
No plan file found:
file:/tmp/test/hive_2014-05-07_18-06-08_349_1526907284076641211-1/-mr-10001/0a1c9ebe-cdb0-4adc-9e93-8f176019f19a/map.xml
2014-05-07 18:06:33,635 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:437)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:430)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.init(MapTask.java:168)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:409)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Hive run on MR2 must set mapreduce.jobtracker.address
-
Key: HIVE-7039
URL: https://issues.apache.org/jira/browse/HIVE-7039
Project: Hive
Issue Type: Bug
Components:
[jira] [Updated] (HIVE-7037) Add additional tests for transform clauses with Tez
[ https://issues.apache.org/jira/browse/HIVE-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-7037: - Attachment: HIVE-7037.1.patch Add additional tests for transform clauses with Tez --- Key: HIVE-7037 URL: https://issues.apache.org/jira/browse/HIVE-7037 Project: Hive Issue Type: Bug Components: Tez Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-7037.1.patch Enabling some q tests for Tez wrt to ScriptOperator/Stream/Transform. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7023) Bucket mapjoin is broken when the number of small aliases is two or more
[
https://issues.apache.org/jira/browse/HIVE-7023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993647#comment-13993647
]
Hive QA commented on HIVE-7023:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12643890/HIVE-7023.2.patch.txt
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 5500 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket_map_join_tez1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_dml
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/152/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/152/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12643890
Bucket mapjoin is broken when the number of small aliases is two or more
Key: HIVE-7023
URL: https://issues.apache.org/jira/browse/HIVE-7023
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.13.0
Reporter: Navis
Assignee: Navis
Fix For: 0.14.0
Attachments: HIVE-7023.1.patch.txt, HIVE-7023.2.patch.txt
From auto_sortmerge_join_11.q,
{noformat}
-- small 1 part, 2 bucket big 2 part, 4 bucket
CREATE TABLE bucket_small (key string, value string) partitioned by (ds
string) CLUSTERED BY (key) INTO 2 BUCKETS STORED AS TEXTFILE;
load data local inpath '../../data/files/smallsrcsortbucket1outof4.txt' INTO
TABLE bucket_small partition(ds='2008-04-08');
load data local inpath '../../data/files/smallsrcsortbucket2outof4.txt' INTO
TABLE bucket_small partition(ds='2008-04-08');
CREATE TABLE bucket_big (key string, value string) partitioned by (ds string)
CLUSTERED BY (key) INTO 4 BUCKETS STORED AS TEXTFILE;
load data local inpath '../../data/files/srcsortbucket1outof4.txt' INTO TABLE
bucket_big partition(ds='2008-04-08');
load data local inpath '../../data/files/srcsortbucket2outof4.txt' INTO TABLE
bucket_big partition(ds='2008-04-08');
load data local inpath '../../data/files/srcsortbucket3outof4.txt' INTO TABLE
bucket_big partition(ds='2008-04-08');
load data local inpath '../../data/files/srcsortbucket4outof4.txt' INTO TABLE
bucket_big partition(ds='2008-04-08');
load data local inpath '../../data/files/srcsortbucket1outof4.txt' INTO TABLE
bucket_big partition(ds='2008-04-09');
load data local inpath '../../data/files/srcsortbucket2outof4.txt' INTO TABLE
bucket_big partition(ds='2008-04-09');
load data local inpath '../../data/files/srcsortbucket3outof4.txt' INTO TABLE
bucket_big partition(ds='2008-04-09');
load data local inpath '../../data/files/srcsortbucket4outof4.txt' INTO TABLE
bucket_big partition(ds='2008-04-09');
set hive.auto.convert.join=true;
set hive.ignore.mapjoin.hint=false;
set hive.auto.convert.sortmerge.join=true;
set hive.optimize.bucketmapjoin=true;
set hive.optimize.bucketmapjoin.sortedmerge=true;
select /* + MAPJOIN(a,b) */ count(*) FROM bucket_small a JOIN bucket_big b ON
a.key = b.key JOIN bucket_big c ON a.key = c.key;
{noformat}
The last query produces 0 row, instead of 180 rows, which is correct.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7012) Wrong RS de-duplication in the ReduceSinkDeDuplication Optimizer
[
https://issues.apache.org/jira/browse/HIVE-7012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13994368#comment-13994368
]
Sun Rui commented on HIVE-7012:
---
[~navis] I verified that your patch solved my problem.
[~navis] and [~yhuai] However, I suspect that the optimizer may still have bug
when there are distinct expressions. It seems that the optimizer has not taken
support for distinct keys into consideration when it was being implemented.
Note that keyCols in ReduceSinkDesc is composed of groupby keys and possibly
distinct keys. For example, assume cRS and pRS both have KeyCols as (a, b, c,
d) and numDistributionKeys=2. cRS may have distinct expressions like
distinct(c, d) while pRS may have distinct expressions like distinct(c),
distinct(d). In this case, they have different sort keys while their KeyCols
are same. [~yhuai] what do you think?
Wrong RS de-duplication in the ReduceSinkDeDuplication Optimizer
Key: HIVE-7012
URL: https://issues.apache.org/jira/browse/HIVE-7012
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.13.0
Reporter: Sun Rui
Assignee: Navis
Attachments: HIVE-7012.1.patch.txt, HIVE-7012.2.patch.txt
With HIVE 0.13.0, run the following test case:
{code:sql}
create table src(key bigint, value string);
select
count(distinct key) as col0
from src
order by col0;
{code}
The following exception will be thrown:
{noformat}
java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:485)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 9 more
Caused by: java.lang.RuntimeException: Reduce operator initialization failed
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:173)
... 14 more
Caused by: java.lang.RuntimeException: cannot find field _col0 from
[0:reducesinkkey0]
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:415)
at
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:150)
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:79)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.initializeOp(GroupByOperator.java:288)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:166)
... 14 more
{noformat}
This issue is related to HIVE-6455. When hive.optimize.reducededuplication is
set to false, then this issue will be gone.
Logical plan when hive.optimize.reducededuplication=false;
{noformat}
src
TableScan (TS_0)
alias: src
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
Select Operator (SEL_1)
expressions: key (type: bigint)
outputColumnNames: key
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats:
NONE
Group By Operator (GBY_2)
aggregations: count(DISTINCT key)
keys: key (type: bigint)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats:
NONE
Reduce Output Operator (RS_3)
istinctColumnIndices:
key expressions: _col0 (type: bigint)
DistributionKeys: 0
sort order: +
OutputKeyColumnNames: _col0
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column
stats: NONE
[jira] [Updated] (HIVE-5664) Drop cascade database fails when the db has any tables with indexes
[
https://issues.apache.org/jira/browse/HIVE-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-5664:
Fix Version/s: (was: 0.13.0)
0.14.0
Drop cascade database fails when the db has any tables with indexes
---
Key: HIVE-5664
URL: https://issues.apache.org/jira/browse/HIVE-5664
Project: Hive
Issue Type: Bug
Components: Indexing, Metastore
Affects Versions: 0.10.0, 0.11.0, 0.12.0
Reporter: Venki Korukanti
Assignee: Venki Korukanti
Fix For: 0.14.0
Attachments: HIVE-5664.1.patch.txt
{code}
CREATE DATABASE db2;
USE db2;
CREATE TABLE tab1 (id int, name string);
CREATE INDEX idx1 ON TABLE tab1(id) as 'COMPACT' with DEFERRED REBUILD IN
TABLE tab1_indx;
DROP DATABASE db2 CASCADE;
{code}
Last DDL fails with the following error:
{code}
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. Database does not exist: db2
Hive.log has following exception
2013-10-27 20:46:16,629 ERROR exec.DDLTask (DDLTask.java:execute(434)) -
org.apache.hadoop.hive.ql.metadata.HiveException: Database does not exist: db2
at
org.apache.hadoop.hive.ql.exec.DDLTask.dropDatabase(DDLTask.java:3473)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1441)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1219)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1047)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:915)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:422)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:790)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:623)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
Caused by: NoSuchObjectException(message:db2.tab1_indx table not found)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1376)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:103)
at com.sun.proxy.$Proxy7.get_table(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:890)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:660)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:652)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropDatabase(HiveMetaStoreClient.java:546)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
at com.sun.proxy.$Proxy8.dropDatabase(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.dropDatabase(Hive.java:284)
at
org.apache.hadoop.hive.ql.exec.DDLTask.dropDatabase(DDLTask.java:3470)
... 18 more
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7037) Add additional tests for transform clauses with Tez
[
https://issues.apache.org/jira/browse/HIVE-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13994381#comment-13994381
]
Hive QA commented on HIVE-7037:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12644001/HIVE-7037.1.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 5436 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/168/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/168/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12644001
Add additional tests for transform clauses with Tez
---
Key: HIVE-7037
URL: https://issues.apache.org/jira/browse/HIVE-7037
Project: Hive
Issue Type: Bug
Components: Tez
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
Attachments: HIVE-7037.1.patch
Enabling some q tests for Tez wrt to ScriptOperator/Stream/Transform.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6815) Version of the HIVE-6374 for Hive 0.13
[ https://issues.apache.org/jira/browse/HIVE-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-6815: Fix Version/s: (was: 0.13.0) 0.14.0 Version of the HIVE-6374 for Hive 0.13 -- Key: HIVE-6815 URL: https://issues.apache.org/jira/browse/HIVE-6815 Project: Hive Issue Type: Sub-task Components: CLI Affects Versions: 0.13.0 Reporter: Benjamin Zhitomirsky Assignee: Benjamin Zhitomirsky Fix For: 0.14.0 Attachments: hive-6374.branch-0.13.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6410) Allow output serializations separators to be set for HDFS path as well.
[ https://issues.apache.org/jira/browse/HIVE-6410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-6410: Fix Version/s: (was: 0.13.0) 0.14.0 Allow output serializations separators to be set for HDFS path as well. --- Key: HIVE-6410 URL: https://issues.apache.org/jira/browse/HIVE-6410 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Fix For: 0.14.0 Attachments: HIVE-6410.patch HIVE-3682 adds functionality for users to set serialization constants for 'insert overwrite local directory'. The same functionality should be available for hdfs path as well. The workaround suggested is to create a table with required format and insert into the table, which enforces the users to know the schema of the result and create the table ahead. Though that works, it is good to have the functionality for loading into directory as well. I'm planning to add the same functionality in 'insert overwrite directory' in this jira. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5342) Remove pre hadoop-0.20.0 related codes
[ https://issues.apache.org/jira/browse/HIVE-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-5342: - Attachment: HIVE-5342.1.patch - Tried removing 0.17 workarounds from LazySimpleSerde/LazyBinarySerde, which has different code for Text/BytesWritable because they didn't share common interface in 0.17. [~ashutoshc] Does this look correct? - exec/Utilities has some 0.17 workarounds relating to file paths. Not everything could be removed here since it looks like both file:/// and just the path string are supposed to work. The changes now allow file:/ to work in addition to file:///. Remove pre hadoop-0.20.0 related codes -- Key: HIVE-5342 URL: https://issues.apache.org/jira/browse/HIVE-5342 Project: Hive Issue Type: Task Reporter: Navis Assignee: Jason Dere Priority: Trivial Attachments: D13047.1.patch, HIVE-5342.1.patch Recently, we discussed not supporting hadoop-0.20.0. If it would be done like that or not, 0.17 related codes would be removed before that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-7037) Add additional tests for transform clauses with Tez
Gunther Hagleitner created HIVE-7037: Summary: Add additional tests for transform clauses with Tez Key: HIVE-7037 URL: https://issues.apache.org/jira/browse/HIVE-7037 Project: Hive Issue Type: Bug Components: Tez Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-7037.1.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory overhead
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18936/#review42555 --- ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java https://reviews.apache.org/r/18936/#comment76364 see other comment. shouldn't need at all - or at least not in the operator... serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java https://reviews.apache.org/r/18936/#comment76365 this file also has a bunch of no curly conditions... serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java https://reviews.apache.org/r/18936/#comment76363 randomaccess doesn't extend output? - Gunther Hagleitner On May 1, 2014, 2:29 a.m., Sergey Shelukhin wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18936/ --- (Updated May 1, 2014, 2:29 a.m.) Review request for hive, Gopal V and Gunther Hagleitner. Repository: hive-git Description --- See JIRA Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 604bea7 conf/hive-default.xml.template 2552560 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 142bfd8 ql/src/java/org/apache/hadoop/hive/ql/Driver.java bf9d4c1 ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java f5d4670 ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b93ea7a ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 175d3ab ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19 ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 64f0be2 ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java e392592 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java d4be78d ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 3077d75 ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java f7b499b ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 157d072 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339 ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779 ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 093da55 ql/src/test/queries/clientpositive/mapjoin_decimal.q b65a7be ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6 ql/src/test/queries/clientpositive/tez_union.q f80d94c ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 8350670 ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out 3c55b5c ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03 serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29 serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 9079b9d serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/OutputByteBuffer.java 1b09d41 serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884 serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
[jira] [Commented] (HIVE-7034) Explain result of TezWork is not deterministic
[ https://issues.apache.org/jira/browse/HIVE-7034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993904#comment-13993904 ] Ashutosh Chauhan commented on HIVE-7034: +1 Explain result of TezWork is not deterministic -- Key: HIVE-7034 URL: https://issues.apache.org/jira/browse/HIVE-7034 Project: Hive Issue Type: Task Components: Tests Reporter: Navis Assignee: Navis Priority: Trivial Fix For: 0.14.0 Attachments: HIVE-7034.1.patch.txt Recent failure on tez tests are caused by different iteration order of HashMap implementations. Let's fix that. -- This message was sent by Atlassian JIRA (v6.2#6252)
Tutorial link not working
Hi, This link http://www.grouplens.org/sites/www.grouplens.org/external_files/data/ml-data.tar.gz from this page https://cwiki.apache.org/confluence/display/Hive/GettingStarted is not working. Could someone provide link to a file that has the same contents? Thank you.
