[jira] [Created] (HIVE-11217) CTAS statements throws error, when the table is stored as ORC File format and select clause has NULL/VOID type column
Gaurav Kohli created HIVE-11217: --- Summary: CTAS statements throws error, when the table is stored as ORC File format and select clause has NULL/VOID type column Key: HIVE-11217 URL: https://issues.apache.org/jira/browse/HIVE-11217 Project: Hive Issue Type: Bug Components: File Formats Affects Versions: 0.13.1 Reporter: Gaurav Kohli Priority: Minor If you try to use create-table-as-select (CTAS) statement and create a ORC File format based table, then you can't use NULL as a column value in select clause CREATE TABLE empty (x int); CREATE TABLE orc_table_with_null STORED AS ORC AS SELECT x, null FROM empty; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 36353: HIVE-10882
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36353/ --- Review request for hive and Ashutosh Chauhan. Bugs: HIVE-10882 https://issues.apache.org/jira/browse/HIVE-10882 Repository: hive-git Description --- CBO: Calcite Operator To Hive Operator (Calcite Return Path) empty filtersMap of join operator causes wrong results Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOptUtil.java 9ebb24ffc33c49b0d1479b17af99bba78ca4c5fc ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveJoin.java 668960ec18a4ed8a40b4e08bc814f6ca14b13fd3 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveMultiJoin.java 911cedaccc1051a0b0dff56cee0c5758120fb139 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSemiJoin.java dd1691c7482128bed15608b03df84eb20d4b06e6 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinToMultiJoinRule.java c5e0e1153341218fa36bd3de2155d17478a17436 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java f72f67fec787ff8db729f998a96b1264bc6cf8cb ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java 84c6cc8b0e89dafc4cbbc9e6c337c3215d10ddbb ql/src/test/queries/clientpositive/fouter_join_ppr.q PRE-CREATION ql/src/test/results/clientpositive/cbo_rp_auto_join0.q.out 6fdc935af99971b89c58270014fab856c3e18d6e ql/src/test/results/clientpositive/cbo_rp_auto_join1.q.out 79b4650cc9605fd462b0f4c7080fdc860b32217e ql/src/test/results/clientpositive/cbo_rp_join0.q.out 1894110bca316d8b3fc44e3c3e087cd83ab01bf5 ql/src/test/results/clientpositive/cbo_rp_join1.q.out 69ce6d27571310b05b97ae159de50a5764b4c7f4 ql/src/test/results/clientpositive/fouter_join_ppr.q.out PRE-CREATION Diff: https://reviews.apache.org/r/36353/diff/ Testing --- Existing tests + new test added with return path on. Thanks, Jesús Camacho Rodríguez
Review Request 36354: HIVE-10882
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36354/ --- Review request for hive and Ashutosh Chauhan. Bugs: HIVE-10882 https://issues.apache.org/jira/browse/HIVE-10882 Repository: hive-git Description --- CBO: Calcite Operator To Hive Operator (Calcite Return Path) empty filtersMap of join operator causes wrong results Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOptUtil.java 9ebb24ffc33c49b0d1479b17af99bba78ca4c5fc ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveJoin.java 668960ec18a4ed8a40b4e08bc814f6ca14b13fd3 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveMultiJoin.java 911cedaccc1051a0b0dff56cee0c5758120fb139 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSemiJoin.java dd1691c7482128bed15608b03df84eb20d4b06e6 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinToMultiJoinRule.java c5e0e1153341218fa36bd3de2155d17478a17436 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java f72f67fec787ff8db729f998a96b1264bc6cf8cb ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java 84c6cc8b0e89dafc4cbbc9e6c337c3215d10ddbb ql/src/test/queries/clientpositive/fouter_join_ppr.q PRE-CREATION ql/src/test/results/clientpositive/fouter_join_ppr.q.out PRE-CREATION Diff: https://reviews.apache.org/r/36354/diff/ Testing --- Existing tests + new test added with return path on. Thanks, Jesús Camacho Rodríguez
Hive-0.14 - Build # 1007 - Still Failing
Changes for Build #987 Changes for Build #988 Changes for Build #989 Changes for Build #990 Changes for Build #991 Changes for Build #992 Changes for Build #993 Changes for Build #994 Changes for Build #995 Changes for Build #996 Changes for Build #997 Changes for Build #998 Changes for Build #999 Changes for Build #1000 Changes for Build #1001 Changes for Build #1002 Changes for Build #1003 Changes for Build #1004 Changes for Build #1005 Changes for Build #1006 Changes for Build #1007 No tests ran. The Apache Jenkins build system has built Hive-0.14 (build #1007) Status: Still Failing Check console output at https://builds.apache.org/job/Hive-0.14/1007/ to view the results.
Review Request 36355: Remove OFFLINE and NO_DROP from tables and partitions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36355/ --- Review request for hive, Ashutosh Chauhan, Sushanth Sowmyan, and Thejas Nair. Repository: hive-git Description --- Currently a table or partition can be marked no_drop or offline. This prevents users from dropping or reading (and dropping) the table or partition. This was built in 0.7 before SQL standard authorization was an option. Diffs - hbase-handler/src/test/results/positive/external_table_ppd.q.out 57424ce hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out 578ddb2 hbase-handler/src/test/results/positive/hbase_stats.q.out f12b136 hbase-handler/src/test/results/positive/hbase_stats2.q.out a60dee2 hbase-handler/src/test/results/positive/hbase_stats3.q.out 114847c hbase-handler/src/test/results/positive/hbase_stats_empty_partition.q.out 65e0a6f hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/HCatSemanticAnalyzer.java 11d0743 hcatalog/webhcat/java-client/src/main/java/org/apache/hive/hcatalog/api/HCatClientHMSImpl.java 3a69581 metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 0bcd053 metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java a5f5053 metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 341b0ca metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 38dc406 metastore/src/java/org/apache/hadoop/hive/metastore/PartitionDropOptions.java 5b2811f metastore/src/java/org/apache/hadoop/hive/metastore/ProtectMode.java b8f1390 ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 8bcf860 ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java 968c1e1 ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java a6193ba ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 08ff2e9 ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java e53933e ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java bc09fc3 ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 24ca663 ql/src/java/org/apache/hadoop/hive/ql/parse/EximUtil.java a4c5d0e ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g bdd7cb7 ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 15f1f11 ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 187dc20 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f41668b ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 97d02ea ql/src/java/org/apache/hadoop/hive/ql/plan/AlterTableDesc.java 9e1ac80 ql/src/java/org/apache/hadoop/hive/ql/plan/DropTableDesc.java bb0e7f7 ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java 75cdf16 ql/src/test/queries/clientnegative/alter_partition_invalidspec.q 8cbb25c ql/src/test/queries/clientnegative/alter_partition_nodrop.q 3c0ff02 ql/src/test/queries/clientnegative/alter_partition_nodrop_table.q f2135b1 ql/src/test/queries/clientnegative/alter_partition_offline.q 7376d8b ql/src/test/queries/clientnegative/drop_table_failure3.q 534ce0b ql/src/test/queries/clientnegative/protectmode_part.q 5415999 ql/src/test/queries/clientnegative/protectmode_part1.q 99256da ql/src/test/queries/clientnegative/protectmode_part2.q 3fdc036 ql/src/test/queries/clientnegative/protectmode_part_no_drop.q b4e508f ql/src/test/queries/clientnegative/protectmode_part_no_drop2.q e7e8c42 ql/src/test/queries/clientnegative/protectmode_tbl1.q 2361299 ql/src/test/queries/clientnegative/protectmode_tbl2.q 05964c3 ql/src/test/queries/clientnegative/protectmode_tbl3.q bbaa267 ql/src/test/queries/clientnegative/protectmode_tbl4.q c7880de ql/src/test/queries/clientnegative/protectmode_tbl5.q cd848fd ql/src/test/queries/clientnegative/protectmode_tbl6.q 26248cc ql/src/test/queries/clientnegative/protectmode_tbl7.q afff840 ql/src/test/queries/clientnegative/protectmode_tbl8.q 809c287 ql/src/test/queries/clientnegative/protectmode_tbl_no_drop.q a4ef2ac ql/src/test/queries/clientnegative/sa_fail_hook3.q e54201c ql/src/test/queries/clientpositive/alter_partition_protect_mode.q 7a1f3dd ql/src/test/queries/clientpositive/drop_partitions_ignore_protection.q e825df9 ql/src/test/queries/clientpositive/protectmode.q 27055fb ql/src/test/queries/clientpositive/protectmode2.q 978b032 ql/src/test/results/beelinepositive/alter_numbuckets_partitioned_table.q.out b5b089a ql/src/test/results/beelinepositive/create_like.q.out 40b6cb7 ql/src/test/results/beelinepositive/create_like2.q.out 8bdb44f ql/src/test/results/beelinepositive/create_like_view.q.out 80483c3 ql/src/test/results/beelinepositive/create_skewed_table1.q.out 60b54a7
[jira] [Created] (HIVE-11218) alter1.q test fails in hbase-metastore branch
Alan Gates created HIVE-11218: - Summary: alter1.q test fails in hbase-metastore branch Key: HIVE-11218 URL: https://issues.apache.org/jira/browse/HIVE-11218 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: hbase-metastore-branch Reporter: Alan Gates Assignee: Alan Gates alter1.q fails -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 36284: HIVE-11197
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36284/ --- (Updated July 9, 2015, 6:19 p.m.) Review request for hive and Jesús Camacho Rodríguez. Changes --- Updated with latest changes. Bugs: HIVE-11197 https://issues.apache.org/jira/browse/HIVE-11197 Repository: hive-git Description --- While extracting join conditions follow Hive rules for type conversion instead of Calcite Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 024097e ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOptUtil.java 9ebb24f ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveOnTezCostModel.java fb67309 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveJoin.java 668960e ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveMultiJoin.java 911ceda ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveInsertExchange4JoinRule.java c5ab055 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java a4484ec ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinToMultiJoinRule.java c5e0e11 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java 960ec40 Diff: https://reviews.apache.org/r/36284/diff/ Testing --- Existing tests. Thanks, Ashutosh Chauhan
Re: Only a small subset of Hive UTs are executed when running mvn test
I¹ve noticed that not all test are run.
For example ³mvn test -Phadoop-2² from ql/ only runs 30-40 Test*.java
files while there are 245 such files under ql/.
Something similar happens under itests/
On 7/9/15, 5:07 PM, Thomas Friedrich tfried...@yahoo.com.INVALID wrote:
Hi, I am trying to run all the UTs on the master branch using the
hadoop-2 profilemvn test -Phadoop-2
but it only runs 272 tests:
org.apache.hive:hive-accumulo-handler 23 org.apache.hive:hive-common 73
org.apache.hive:hive-contrib 6 org.apache.hive:hive-exec 168
org.apache.hive:hive-serde 2
All other tests are skipped. This seems to be related to HIVE-10941 where
an additional option to skip the Spark UTs was added. When I take out
exclude%regex[${skip.spark.files}]/excludefrom the exclude list of
tests, all tests are executed again.
Did anyone else run into this as well or am I missing something in my
test command? This seems like a bug to me because HIVE-10477 mentions
that by default the Spark tests should be included and no additional
spark profile needs to be set.
In addition, HIVE-10941 should also update the documentation to explain
the use of the spark-test profile.
https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ#HiveDeve
loperFAQ-Testing
Review Request 36386: Auto-gather column stats
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36386/ --- Review request for hive and John Pullokkaran. Repository: hive-git Description --- Hive will collect table stats when set hive.stats.autogather=true during the INSERT OVERWRITE command. And then the users need to collect the column stats themselves using Analyze command. In this patch, the column stats will also be collected automatically when a flag is set. Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 39477d6 metastore/src/java/org/apache/hadoop/hive/metastore/AggregateStatsCache.java 44106f5 ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java d688ef1 ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 3b5dbe2 ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsAutoGatherContext.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java a5f0a7f ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 5872e8e ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 14a7e9c ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java b02374e ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java ba11e41 ql/src/test/queries/clientpositive/autoColumnStats.q PRE-CREATION ql/src/test/results/clientpositive/autoColumnStats.q.out PRE-CREATION Diff: https://reviews.apache.org/r/36386/diff/ Testing --- Thanks, pengcheng xiong
[jira] [Created] (HIVE-11224) AggregateStatsCache triggers java.util.ConcurrentModificationException under some conditions
Pengcheng Xiong created HIVE-11224:
--
Summary: AggregateStatsCache triggers
java.util.ConcurrentModificationException under some conditions
Key: HIVE-11224
URL: https://issues.apache.org/jira/browse/HIVE-11224
Project: Hive
Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong
Stack
{code}
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922)
at java.util.HashMap$EntryIterator.next(HashMap.java:962)
at java.util.HashMap$EntryIterator.next(HashMap.java:960)
at
org.apache.hadoop.hive.metastore.AggregateStatsCache.findBestMatch(AggregateStatsCache.java:244)
at
org.apache.hadoop.hive.metastore.AggregateStatsCache.get(AggregateStatsCache.java:186)
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.aggrColStatsForPartitions(MetaStoreDirectSql.java:1131)
at
org.apache.hadoop.hive.metastore.ObjectStore$9.getSqlResult(ObjectStore.java:6174)
at
org.apache.hadoop.hive.metastore.ObjectStore$9.getSqlResult(ObjectStore.java:6170)
at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2405)
at
org.apache.hadoop.hive.metastore.ObjectStore.get_aggr_stats_for(ObjectStore.java:6170)
at sun.reflect.GeneratedMethodAccessor103.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
at com.sun.proxy.$Proxy6.get_aggr_stats_for(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_aggr_stats_for(HiveMetaStore.java:5707)
at sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy8.get_aggr_stats_for(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAggrColStatsFor(HiveMetaStoreClient.java:2067)
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-11225) Running all Hive UTs or itests executes only small subset of tests
Thomas Friedrich created HIVE-11225:
---
Summary: Running all Hive UTs or itests executes only small subset
of tests
Key: HIVE-11225
URL: https://issues.apache.org/jira/browse/HIVE-11225
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Thomas Friedrich
Trying to run all unit tests runs only a small sub-set of the tests:
For example:
mvn test -Phadoop-2
only ran 272 tests.
Others ran into similar issues when running the UTs from a particular package
like ql.
This is potentially related to HIVE-10941 where an additional option to skip
the Spark UTs was added. When I take out
exclude%regex[${skip.spark.files}]/exclude
from the exclude list of tests, all tests are executed again. This is not a fix
though, instead the spark-test profile should be fixed.
I see the problem in both the 1.2 branch as well as master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Only a small subset of Hive UTs are executed when running mvn test
Thanks for confirming that this is a real issue. I opened HIVE-11225 to track
it.
On Thursday, July 9, 2015 6:16 PM, Sergey Shelukhin
ser...@hortonworks.com wrote:
Can you file a bug? I noticed this too under itests when trying to run q
file tests locally when HiveQA was down.
On 15/7/9, 17:59, Eugene Koifman ekoif...@hortonworks.com wrote:
I¹ve noticed that not all test are run.
For example ³mvn test -Phadoop-2² from ql/ only runs 30-40 Test*.java
files while there are 245 such files under ql/.
Something similar happens under itests/
On 7/9/15, 5:07 PM, Thomas Friedrich tfried...@yahoo.com.INVALID
wrote:
Hi, I am trying to run all the UTs on the master branch using the
hadoop-2 profilemvn test -Phadoop-2
but it only runs 272 tests:
org.apache.hive:hive-accumulo-handler 23 org.apache.hive:hive-common 73
org.apache.hive:hive-contrib 6 org.apache.hive:hive-exec 168
org.apache.hive:hive-serde 2
All other tests are skipped. This seems to be related to HIVE-10941 where
an additional option to skip the Spark UTs was added. When I take out
exclude%regex[${skip.spark.files}]/excludefrom the exclude list of
tests, all tests are executed again.
Did anyone else run into this as well or am I missing something in my
test command? This seems like a bug to me because HIVE-10477 mentions
that by default the Spark tests should be included and no additional
spark profile needs to be set.
In addition, HIVE-10941 should also update the documentation to explain
the use of the spark-test profile.
https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ#HiveDev
e
loperFAQ-Testing
[jira] [Created] (HIVE-11223) CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered
Jesus Camacho Rodriguez created HIVE-11223: -- Summary: CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered Key: HIVE-11223 URL: https://issues.apache.org/jira/browse/HIVE-11223 Project: Hive Issue Type: Sub-task Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Information in aux data structures is not complete, thus MapJoin and SMBJoin conversion are not triggered. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Only a small subset of Hive UTs are executed when running mvn test
Can you file a bug? I noticed this too under itests when trying to run q
file tests locally when HiveQA was down.
On 15/7/9, 17:59, Eugene Koifman ekoif...@hortonworks.com wrote:
I¹ve noticed that not all test are run.
For example ³mvn test -Phadoop-2² from ql/ only runs 30-40 Test*.java
files while there are 245 such files under ql/.
Something similar happens under itests/
On 7/9/15, 5:07 PM, Thomas Friedrich tfried...@yahoo.com.INVALID
wrote:
Hi, I am trying to run all the UTs on the master branch using the
hadoop-2 profilemvn test -Phadoop-2
but it only runs 272 tests:
org.apache.hive:hive-accumulo-handler 23 org.apache.hive:hive-common 73
org.apache.hive:hive-contrib 6 org.apache.hive:hive-exec 168
org.apache.hive:hive-serde 2
All other tests are skipped. This seems to be related to HIVE-10941 where
an additional option to skip the Spark UTs was added. When I take out
exclude%regex[${skip.spark.files}]/excludefrom the exclude list of
tests, all tests are executed again.
Did anyone else run into this as well or am I missing something in my
test command? This seems like a bug to me because HIVE-10477 mentions
that by default the Spark tests should be included and no additional
spark profile needs to be set.
In addition, HIVE-10941 should also update the documentation to explain
the use of the spark-test profile.
https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ#HiveDev
e
loperFAQ-Testing
[jira] [Created] (HIVE-11215) Vectorized grace hash-join throws FileUtil warnings
Gopal V created HIVE-11215:
--
Summary: Vectorized grace hash-join throws FileUtil warnings
Key: HIVE-11215
URL: https://issues.apache.org/jira/browse/HIVE-11215
Project: Hive
Issue Type: Bug
Components: Vectorization
Affects Versions: 1.2.0, 1.3.0, 2.0.0
Reporter: Gopal V
Priority: Minor
TPC-DS query13 warnings about a null-file deletion.
{code}
2015-07-09 03:14:18,880 INFO [TezChild] exec.MapJoinOperator: Hybrid Grace Hash
Join: Number of rows restored from KeyValueContainer: 31184
2015-07-09 03:14:18,881 INFO [TezChild] exec.MapJoinOperator: Hybrid Grace Hash
Join: Deserializing spilled hash partition...
2015-07-09 03:14:18,881 INFO [TezChild] exec.MapJoinOperator: Hybrid Grace Hash
Join: Number of rows in hashmap: 31184
2015-07-09 03:14:18,897 INFO [TezChild] exec.MapJoinOperator: spilled: true
abort: false. Clearing spilled partitions.
2015-07-09 03:14:18,898 WARN [TezChild] fs.FileUtil: null file argument.
2015-07-09 03:14:18,898 WARN [TezChild] fs.FileUtil: null file argument.
2015-07-09 03:14:18,898 WARN [TezChild] fs.FileUtil: null file argument.
2015-07-09 03:14:18,898 WARN [TezChild] fs.FileUtil: null file argument.
2015-07-09 03:14:18,898 WARN [TezChild] fs.FileUtil: null file argument.
2015-07-09 03:14:18,898 WARN [TezChild] fs.FileUtil: null file argument.
2015-07-09 03:14:18,898 WARN [TezChild] fs.FileUtil: null file argument.
2015-07-09 03:14:18,898 WARN [TezChild] fs.FileUtil: null file argument.
2015-07-09 03:14:18,898 WARN [TezChild] fs.FileUtil: null file argument.
2015-07-09 03:14:18,898 WARN [TezChild] fs.FileUtil: null file argument.
2015-07-09 03:14:18,898 WARN [TezChild] fs.FileUtil: null file argument.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 36300: HIVE-11191 Beeline-cli: support hive.cli.errors.ignore in new CLI
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36300/ --- (Updated July 9, 2015, 3:07 p.m.) Review request for hive and Xuefu Zhang. Changes --- Summary for V2: 1. Add the fields of conf in beelineOpts which is used to parsing client side configuration 2. Add new API to update the configuration in client side Bugs: HIVE-11191 https://issues.apache.org/jira/browse/HIVE-11191 Repository: hive-git Description --- Summary: 1. Add option update stage after connection established in CLI mode 2. Update the getHiveConf API Diffs (updated) - beeline/src/java/org/apache/hive/beeline/BeeLine.java c4dbcd4 beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java c1ec82a beeline/src/java/org/apache/hive/beeline/Commands.java d490273 beeline/src/test/org/apache/hive/beeline/cli/TestHiveCli.java ff8ab17 Diff: https://reviews.apache.org/r/36300/diff/ Testing --- Partial UT passed and local cluster test passed. Thanks, cheng xu
[jira] [Created] (HIVE-11216) UDF GenericUDFMapKeys throws NPE when a null map value is passed in
Yibing Shi created HIVE-11216:
-
Summary: UDF GenericUDFMapKeys throws NPE when a null map value is
passed in
Key: HIVE-11216
URL: https://issues.apache.org/jira/browse/HIVE-11216
Project: Hive
Issue Type: Bug
Components: UDF
Affects Versions: 1.2.0
Reporter: Yibing Shi
We can reproduce the problem as below:
{noformat}
hive show create table map_txt;
OK
CREATE TABLE `map_txt`(
`id` int,
`content` mapint,string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
...
Time taken: 0.233 seconds, Fetched: 18 row(s)
hive select * from map_txt;
OK
1 NULL
Time taken: 0.679 seconds, Fetched: 1 row(s)
hive select id, map_keys(content) from map_txt;
Error during job, obtaining debugging information...
Examining task ID: task_1435534231122_0025_m_00 (and more) from job
job_1435534231122_0025
Task with the most failures(4):
-
Task ID:
task_1435534231122_0025_m_00
URL:
http://host-10-17-80-40.coe.cloudera.com:8088/taskdetails.jsp?jobid=job_1435534231122_0025tipid=task_1435534231122_0025_m_00
-
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {id:1,content:null}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:198)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row {id:1,content:null}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:559)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:180)
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating
map_keys(content)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
... 9 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDFMapKeys.evaluate(GenericUDFMapKeys.java:64)
at
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
at
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
at
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:79)
... 13 more
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL
hive
{noformat}
The error is as below (in mappers):
{noformat}
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDFMapKeys.evaluate(GenericUDFMapKeys.java:64)
at
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
at
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
at
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
at
org.apache.hadoop.hive.ql.exec.KeyWrapperFactory$ListKeyWrapper.getNewKey(KeyWrapperFactory.java:113)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:778)
... 17 more
{noformat}
Looking at the source code:
{code}
public Object evaluate(DeferredObject[] arguments) throws HiveException {
retArray.clear();
Object mapObj = arguments[0].get();
retArray.addAll(mapOI.getMap(mapObj).keySet());
return retArray;
}
{code}
It is obvious that we will have a NPE when a NULL map value is passed in
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-11221) In Tez mode, alter table concatenate orc files can intermittently fail with NPE
Prasanth Jayachandran created HIVE-11221:
Summary: In Tez mode, alter table concatenate orc files can
intermittently fail with NPE
Key: HIVE-11221
URL: https://issues.apache.org/jira/browse/HIVE-11221
Project: Hive
Issue Type: Bug
Affects Versions: 1.3.0, 2.0.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran
We are not waiting for input ready events which can trigger occasional NPE if
input is not actually ready.
Stacktrace:
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186)
at
org.apache.hadoop.hive.ql.exec.tez.MergeFileTezProcessor.run(MergeFileTezProcessor.java:42)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:265)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:478)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:471)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:648)
at
org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:146)
at
org.apache.tez.mapreduce.lib.MRReaderMapred.init(MRReaderMapred.java:73)
at
org.apache.tez.mapreduce.input.MRInput.initializeInternal(MRInput.java:483)
at
org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:108)
at
org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.getMRInput(MergeFileRecordProcessor.java:220)
at
org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.init(MergeFileRecordProcessor.java:72)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162)
... 13 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 36335: Reset the fields in class JoinStatsRule in StatsRulesProcFactory
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36335/#review91173 --- Ship it! Ship It! - John Pullokkaran On July 8, 2015, 11:37 p.m., pengcheng xiong wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36335/ --- (Updated July 8, 2015, 11:37 p.m.) Review request for hive, John Pullokkaran and Mostafa Mokhtar. Repository: hive-git Description --- The static fields in static class JoinStatsRule in StatsRulesProcFactory were carried on in between different calls. It would be better to move it inside the function, just like all the other StatsRuls do. Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java 0982059 Diff: https://reviews.apache.org/r/36335/diff/ Testing --- Thanks, pengcheng xiong
[jira] [Created] (HIVE-11220) HS2 Lineage leakage with 16x concurrency tests
Gopal V created HIVE-11220: -- Summary: HS2 Lineage leakage with 16x concurrency tests Key: HIVE-11220 URL: https://issues.apache.org/jira/browse/HIVE-11220 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: llap, 1.3.0 Reporter: Gopal V Test scenario is the HS2 + LLAP, 16x concurrency of TPCDS queries which take less than 4 seconds. session.LineageState accumulates optimizer lineage info and HS2 OOMs due to the amount of data being held in the SessionState, since the sessions are continously being used without pause. The issue seems to be triggered due to the volume of fast queries or the life time of a single JDBC connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11219) Transactional documentation is unclear
Johndee Burks created HIVE-11219: Summary: Transactional documentation is unclear Key: HIVE-11219 URL: https://issues.apache.org/jira/browse/HIVE-11219 Project: Hive Issue Type: Improvement Components: Transactions Reporter: Johndee Burks Priority: Minor At the [this|https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions] link the following is said. If a table is to be used in ACID writes (insert, update, delete) then the table property transactional must be set on that table, starting with Hive 0.14.0. Without this value, inserts will be done in the old style; updates and deletes will be prohibited. However, this does not apply to Hive 0.13.0. It does not tell you what the value of transactional should be. I think we should say it needs to be true and we should show an example DDL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 36300: HIVE-11191 Beeline-cli: support hive.cli.errors.ignore in new CLI
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36300/#review91240 --- Ship it! Ship It! - Xuefu Zhang On July 9, 2015, 7:07 a.m., cheng xu wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36300/ --- (Updated July 9, 2015, 7:07 a.m.) Review request for hive and Xuefu Zhang. Bugs: HIVE-11191 https://issues.apache.org/jira/browse/HIVE-11191 Repository: hive-git Description --- Summary: 1. Add option update stage after connection established in CLI mode 2. Update the getHiveConf API Diffs - beeline/src/java/org/apache/hive/beeline/BeeLine.java c4dbcd4 beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java c1ec82a beeline/src/java/org/apache/hive/beeline/Commands.java d490273 beeline/src/test/org/apache/hive/beeline/cli/TestHiveCli.java ff8ab17 Diff: https://reviews.apache.org/r/36300/diff/ Testing --- Partial UT passed and local cluster test passed. Thanks, cheng xu
[jira] [Created] (HIVE-11222) LLAP: occasional NPE in parallel queries in ORC reader
Sergey Shelukhin created HIVE-11222:
---
Summary: LLAP: occasional NPE in parallel queries in ORC reader
Key: HIVE-11222
URL: https://issues.apache.org/jira/browse/HIVE-11222
Project: Hive
Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
{noformat}
Caused by: java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:275)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:227)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:155)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:101)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
... 22 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$SargApplier.pickRowGroups(RecordReaderImpl.java:709)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.determineRgsToRead(OrcEncodedDataReader.java:618)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:195)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:59)
at
org.apache.hadoop.hive.common.CallableWithNdc.call(CallableWithNdc.java:37)
... 4 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 36280: HIVE-11196
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36280/ --- (Updated July 9, 2015, 10:04 p.m.) Review request for hive and John Pullokkaran. Repository: hive-git Description --- Utilities.getPartitionDesc() should try to reuse TableDesc object Diffs (updated) - hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatMultiOutputFormat.java 049de54 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java afecb1e ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 29854d8 ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java 317454d ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java b123511 Diff: https://reviews.apache.org/r/36280/diff/ Testing --- Thanks, Hari Sankar Sivarama Subramaniyan
Only a small subset of Hive UTs are executed when running mvn test
Hi, I am trying to run all the UTs on the master branch using the hadoop-2
profilemvn test -Phadoop-2
but it only runs 272 tests:
org.apache.hive:hive-accumulo-handler 23 org.apache.hive:hive-common 73
org.apache.hive:hive-contrib 6 org.apache.hive:hive-exec 168
org.apache.hive:hive-serde 2
All other tests are skipped. This seems to be related to HIVE-10941 where an
additional option to skip the Spark UTs was added. When I take out
exclude%regex[${skip.spark.files}]/excludefrom the exclude list of tests,
all tests are executed again.
Did anyone else run into this as well or am I missing something in my test
command? This seems like a bug to me because HIVE-10477 mentions that by
default the Spark tests should be included and no additional spark profile
needs to be set.
In addition, HIVE-10941 should also update the documentation to explain the use
of the spark-test profile.
https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ#HiveDeveloperFAQ-Testing
