[jira] [Created] (HIVE-11083) Make test cbo_windowing robust
Ashutosh Chauhan created HIVE-11083: --- Summary: Make test cbo_windowing robust Key: HIVE-11083 URL: https://issues.apache.org/jira/browse/HIVE-11083 Project: Hive Issue Type: Test Components: Tests Affects Versions: 1.2.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Make result set deterministic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 34757: HIVE-10844: Combine equivalent Works for HoS[Spark Branch]
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34757/#review88966 --- ql/src/test/results/clientpositive/spark/groupby10.q.out (line 60) https://reviews.apache.org/r/34757/#comment141575 Interesting. How come we got more stages now? - Xuefu Zhang On June 23, 2015, 7:24 a.m., chengxiang li wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34757/ --- (Updated June 23, 2015, 7:24 a.m.) Review request for hive and Xuefu Zhang. Bugs: HIVE-10844 https://issues.apache.org/jira/browse/HIVE-10844 Repository: hive-git Description --- Some Hive queries(like TPCDS Q39) may share the same subquery, which translated into sperate, but equivalent Works in SparkWork, combining these equivalent Works into a single one would help to benifit from following dynamic RDD caching optimization. Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/OperatorComparatorFactory.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/CombineEquivalentWorkResolver.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 19aae70 ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java b307b16 ql/src/test/results/clientpositive/spark/auto_join30.q.out 7b5c5e7 ql/src/test/results/clientpositive/spark/auto_smb_mapjoin_14.q.out 8a43d78 ql/src/test/results/clientpositive/spark/groupby10.q.out dd9d9fe ql/src/test/results/clientpositive/spark/groupby7_map.q.out abd6459 ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out 5e69b31 ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 3418b99 ql/src/test/results/clientpositive/spark/groupby7_noskew_multi_single_reducer.q.out 2cb126d ql/src/test/results/clientpositive/spark/groupby8.q.out c249b61 ql/src/test/results/clientpositive/spark/groupby8_map_skew.q.out 2fb1d73 ql/src/test/results/clientpositive/spark/insert_into3.q.out 7df5ba8 ql/src/test/results/clientpositive/spark/join22.q.out b1e5b67 ql/src/test/results/clientpositive/spark/skewjoinopt11.q.out 8a278ef ql/src/test/results/clientpositive/spark/union10.q.out 5e8fe38 ql/src/test/results/clientpositive/spark/union11.q.out 20c27c7 ql/src/test/results/clientpositive/spark/union20.q.out 6f0dca6 ql/src/test/results/clientpositive/spark/union28.q.out 98582df ql/src/test/results/clientpositive/spark/union3.q.out 834b6d4 ql/src/test/results/clientpositive/spark/union30.q.out 3409623 ql/src/test/results/clientpositive/spark/union4.q.out c121ef0 ql/src/test/results/clientpositive/spark/union5.q.out afee988 ql/src/test/results/clientpositive/spark/union_remove_1.q.out ba0e293 ql/src/test/results/clientpositive/spark/union_remove_15.q.out 26cfbab ql/src/test/results/clientpositive/spark/union_remove_16.q.out 7a7aaf2 ql/src/test/results/clientpositive/spark/union_remove_18.q.out a5e15c5 ql/src/test/results/clientpositive/spark/union_remove_19.q.out ad44400 ql/src/test/results/clientpositive/spark/union_remove_20.q.out 1d67177 ql/src/test/results/clientpositive/spark/union_remove_21.q.out 9f5b070 ql/src/test/results/clientpositive/spark/union_remove_22.q.out 2e01432 ql/src/test/results/clientpositive/spark/union_remove_24.q.out 2659798 ql/src/test/results/clientpositive/spark/union_remove_25.q.out 0a94684 ql/src/test/results/clientpositive/spark/union_remove_4.q.out 6c3d596 ql/src/test/results/clientpositive/spark/union_remove_6.q.out cd36189 ql/src/test/results/clientpositive/spark/union_remove_6_subq.q.out c981ae4 ql/src/test/results/clientpositive/spark/union_remove_7.q.out 084fbd6 ql/src/test/results/clientpositive/spark/union_top_level.q.out dede1ef Diff: https://reviews.apache.org/r/34757/diff/ Testing --- Thanks, chengxiang li
[jira] [Created] (HIVE-11084) Issue in Parquet Hove Table
Chanchal Kumar Ghosh created HIVE-11084:
---
Summary: Issue in Parquet Hove Table
Key: HIVE-11084
URL: https://issues.apache.org/jira/browse/HIVE-11084
Project: Hive
Issue Type: Bug
Components: File Formats
Affects Versions: 0.9.0
Environment: GNU/Linux
Reporter: Chanchal Kumar Ghosh
{quote}
hive CREATE TABLE intable_p (
sr_no int,
name string,
emp_id int
) PARTITIONED BY (
a string,
b string,
c string
) ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
STORED AS PARQUET;
hive insert overwrite table intable_p partition (a='a', b='b', c='c') select *
from intable;
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1 Cumulative CPU: 2.59 sec HDFS Read: 247 HDFS Write:
410 SUCCESS
Total MapReduce CPU Time Spent: 2 seconds 590 msec
OK
Time taken: 30.382 seconds
hive show create table intable_p;
OK
CREATE TABLE `intable_p`(
`sr_no` int,
`name` string,
`emp_id` int)
PARTITIONED BY (
`a` string,
`b` string,
`c` string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
'hdfs://nameservice1/hive/db/intable_p'
TBLPROPERTIES (
'transient_lastDdlTime'='1435080569')
Time taken: 0.212 seconds, Fetched: 19 row(s)
hive CREATE TABLE `intable_p2`(
`sr_no` int,
`name` string,
`emp_id` int)
PARTITIONED BY (
`a` string,
`b` string,
`c` string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat';
OK
Time taken: 0.179 seconds
hive insert overwrite table intable_p2 partition (a='a', b='b', c='c') select
* from intable;
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
...
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2015-06-23 17:34:40,471 Stage-1 map = 0%, reduce = 0%
2015-06-23 17:35:10,753 Stage-1 map = 100%, reduce = 0%
Ended Job = job_1433246369760_7947 with errors
Error during job, obtaining debugging information...
Examining task ID: task_ (and more) from job job_
Task with the most failures(4):
-
Task ID:
task_
URL:
-
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {sr_no:1,name:ABC,emp_id:1001}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:198)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row {sr_no:1,name:ABC,emp_id:1001}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:180)
... 8 more
Caused by: {color:red}java.lang.ClassCastException: org.apache.hadoop.io.Text
cannot be cast to org.apache.hadoop.io.ArrayWritable{color}
at
org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:105)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:628)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:539)
... 9 more
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Re: [VOTE] Apache Hive 1.2.1 Release Candidate 0
Thanks for testing and verifying, folks! With 4 PMC votes and 105 hours( 72 hours ) now having passed, the vote for releasing 1.2.1 RC0 as Hive 1.2.1 passes. I will go ahead and publish artifacts for the 1.2.1 release and send out mail about general availability. With this release, please note that commits to branch-1.2 are now restricted to a higher bar of necessity, and will require it to be fixing a product outage(such as an NPE when you run a query). I will update the wiki to that effect to indicate the process for further commits to the branch. For most part, please restrict commits to branch-1 and master from now on. I am amenable to doing a 1.2.2 release eventually if we have enough such issues, maybe about 3+ months out. Thanks all! -Sushanth On Sun, Jun 21, 2015 at 6:13 PM, Vikram Dixit K vikram.di...@gmail.com wrote: +1 built on both profiles and ran a simple query on the rc. Thanks Vikram. On Sat, Jun 20, 2015 at 7:47 AM, Thejas Nair thejas.n...@gmail.com wrote: +1 Checked signatures, checksums Checked release notes Reviewed changes in pom files. Built with hadoop2 and hadoop1. Ran some simple queries in local mode. On Fri, Jun 19, 2015 at 5:00 PM, Gunther Hagleitner ghagleit...@hortonworks.com wrote: +1 Checked signatures, compiled, ran some tests. Thanks, Gunther. -- *From:* Alan Gates alanfga...@gmail.com *Sent:* Friday, June 19, 2015 11:44 AM *To:* dev@hive.apache.org *Subject:* Re: [VOTE] Apache Hive 1.2.1 Release Candidate 0 +1. Checked signatures, looked for binary files, compiled the code, and ran a rat check. Alan. Sushanth Sowmyan khorg...@gmail.com June 19, 2015 at 2:44 Hi Folks, It's been a month since 1.2.0, and I promised to do a stabilization 1.2.1 release, and this is it. A large number of patches have been applied since 1.2.0, and major known issues have been cleared/fixed. A few jiras were deferred out to 1.3/2.0 as not being ready to commit into 1.2.1 at this time. More details are available here : https://cwiki.apache.org/confluence/display/Hive/Hive+1.2+Release+Status Apache Hive 1.2.1 Release Candidate 0 is available here: https://people.apache.org/~khorgath/releases/1.2.1_RC0/artifacts/ My public key used for signing is as available from the hive committers key list : http://www.apache.org/dist/hive/KEYS Maven artifacts are available here: https://repository.apache.org/content/repositories/orgapachehive-1040/ Source tag for RC0 is up on the apache git repo as tag release-1.2.1-rc0 (Browseable view over at https://git-wip-us.apache.org/repos/asf?p=hive.git;a=tag;h=0f6ee99efc911cbc1566f9bbbc63a51600302703 ) Voting will conclude in 72 hours. Hive PMC Members: Please test and vote. Thanks, -Sushanth -- Nothing better than when appreciated for hard work. -Mark
[jira] [Created] (HIVE-11085) Alter table fail with NPE if schema change
Daniel Dai created HIVE-11085:
-
Summary: Alter table fail with NPE if schema change
Key: HIVE-11085
URL: https://issues.apache.org/jira/browse/HIVE-11085
Project: Hive
Issue Type: Bug
Components: Metastore
Reporter: Daniel Dai
Fix For: hbase-metastore-branch
alter1.q fail. Specifically, the following statement fail:
create table alter1(a int, b int);
add jar itests/test-serde/target/hive-it-test-serde-1.3.0-SNAPSHOT.jar;
alter table alter1 set serde 'org.apache.hadoop.hive.serde2.TestSerDe' with
serdeproperties('s1'='9');
Error stack:
{code}
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter table.
java.lang.NullPointerException
at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:498)
at org.apache.hadoop.hive.ql.exec.DDLTask.alterTable(DDLTask.java:3418)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:338)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1660)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1419)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1200)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1067)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1057)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:311)
at
org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1116)
at
org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1090)
at
org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:146)
at
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter1(TestCliDriver.java:130)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: MetaException(message:java.lang.NullPointerException)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:5301)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3443)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_cascade(HiveMetaStore.java:3395)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table(HiveMetaStoreClient.java:352)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.alter_table(SessionHiveMetaStoreClient.java:251)
at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:496)
... 36 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.metastore.HiveAlterHandler.updateTableColumnStatsForAlterTable(HiveAlterHandler.java:673)
at
org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:241)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3423)
{code}
If changing the alter statement to:
alter table alter1 set serde 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' with
serdeproperties('s1'='9');
The
Review Request 35792: HIVE-10438 - Architecture for ResultSet Compression via external plugin
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35792/ --- Review request for hive, Vaibhav Gumashta and Xuefu Zhang. Repository: hive-git Description --- This patch enables ResultSet compression for Hive using external plugins. The patch proposes a plugin architecture that enables using external plugins to compress ResultSets on-the-fly. Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 27f68df conf/hive-site.xml dab494e service/if/TCLIService.thrift baf583f service/src/gen/thrift/gen-cpp/TCLIService_types.h 7bceabd service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 86eeea3 service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TEnColumn.java PRE-CREATION service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementReq.java 4f157ad service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTablesReq.java c973fcc service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionReq.java c048161 service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionResp.java 351f78b service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java a4279d2 service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRowSet.java d16c8a4 service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TStatus.java 24a746e service/src/gen/thrift/gen-py/TCLIService/ttypes.py 068727c service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb b482533 service/src/java/org/apache/hive/service/cli/Column.java 2e21f18 service/src/java/org/apache/hive/service/cli/ColumnBasedSet.java 47a582e service/src/java/org/apache/hive/service/cli/ColumnCompressor.java PRE-CREATION service/src/java/org/apache/hive/service/cli/ColumnCompressorService.java PRE-CREATION service/src/java/org/apache/hive/service/cli/EncodedColumnBasedSet.java PRE-CREATION service/src/java/org/apache/hive/service/cli/RowSetFactory.java e8f68ea service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java dfb7faa Diff: https://reviews.apache.org/r/35792/diff/ Testing --- Testing has been done using a docker container-based query submitter that has an integer decompressor as part of it. Using the integer compressor (also provided) and the decompressor, the end-to-end functionality can be observed. File Attachments Patch file https://reviews.apache.org/media/uploaded/files/2015/06/23/16aa08f8-2393-460a-83ef-72464fc537db__HIVE-10438.patch Thanks, Rohit Dholakia
Review Request 35803: HIVE-10895 ObjectStore does not close Query objects in some calls, causing a potential leak in some metastore db resources
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35803/ --- Review request for hive. Repository: hive-git Description --- HIVE-10895 ObjectStore does not close Query objects in some calls, causing a potential leak in some metastore db resources Diffs - metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 417ecc825e4bca8c92f3e42c19e693b624c96dc1 metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java PRE-CREATION Diff: https://reviews.apache.org/r/35803/diff/ Testing --- Thanks, Aihua Xu
[jira] [Created] (HIVE-11086) Remove use of ErrorMsg in Orc's RunLengthIntegerReaderV2
Owen O'Malley created HIVE-11086: Summary: Remove use of ErrorMsg in Orc's RunLengthIntegerReaderV2 Key: HIVE-11086 URL: https://issues.apache.org/jira/browse/HIVE-11086 Project: Hive Issue Type: Sub-task Reporter: Owen O'Malley Assignee: Owen O'Malley ORC's rle v2 reader uses a string literal from ErrorMsg, which forces a large dependency on the rle v2 reader. Pulling the string literal in directly doesn't change the behavior and fixes the linkage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11088) Investigate intermitten failure of join28.q for Spark
Xuefu Zhang created HIVE-11088: -- Summary: Investigate intermitten failure of join28.q for Spark Key: HIVE-11088 URL: https://issues.apache.org/jira/browse/HIVE-11088 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: 1.3.0 Reporter: Xuefu Zhang Assignee: Mohit Sabharwal Please refer to https://issues.apache.org/jira/browse/HIVE-10996?focusedCommentId=14598349page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14598349. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11087) DbTxnManager exceptions should include txnid
Eugene Koifman created HIVE-11087: - Summary: DbTxnManager exceptions should include txnid Key: HIVE-11087 URL: https://issues.apache.org/jira/browse/HIVE-11087 Project: Hive Issue Type: Sub-task Components: Transactions Affects Versions: 1.0.0 Reporter: Eugene Koifman Assignee: Eugene Koifman must include txnid in the exception so that user visible error can be correlated with log file info -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 34757: HIVE-10844: Combine equivalent Works for HoS[Spark Branch]
On 六月 19, 2015, 3:42 a.m., Xuefu Zhang wrote: ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/CombineEquivalentWorkResolver.java, line 207 https://reviews.apache.org/r/34757/diff/2/?file=986303#file986303line207 I think in SparkWork, there couldn't be two parents connectting to the same child. UnionWork wold be such a child, but SparkWork doesn't have UnionWork, if I'm not mistaken. I don't think SparkPlan has a limitation of only link between to trans. If there are two links between a parent to a child, the input will be self unioned and the result is the input to the child. chengxiang li wrote: Take self-join for example, there would be 2 MapWork connect to same ReduceWork. if we combine these 2 MapWorks into 1, SparkPlan::connect would throw exception during SparkPlan generation. Xuefu Zhang wrote: I see. Thanks for the explanation. However, I'm wondering if we should remove the restriction. Otherwise, certain cases such as self join will not take the advantage of this feature, right? Yes, this is a further optimization we can continue to work on, i would create a following up JIRA to research on this. - chengxiang --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34757/#review88484 --- On 六月 19, 2015, 7:22 a.m., chengxiang li wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34757/ --- (Updated 六月 19, 2015, 7:22 a.m.) Review request for hive and Xuefu Zhang. Bugs: HIVE-10844 https://issues.apache.org/jira/browse/HIVE-10844 Repository: hive-git Description --- Some Hive queries(like TPCDS Q39) may share the same subquery, which translated into sperate, but equivalent Works in SparkWork, combining these equivalent Works into a single one would help to benifit from following dynamic RDD caching optimization. Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/OperatorComparatorFactory.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/CombineEquivalentWorkResolver.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 19aae70 ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java b307b16 ql/src/test/results/clientpositive/spark/auto_join30.q.out 7b5c5e7 ql/src/test/results/clientpositive/spark/auto_smb_mapjoin_14.q.out 8a43d78 ql/src/test/results/clientpositive/spark/groupby10.q.out 9d3cf36 ql/src/test/results/clientpositive/spark/groupby7_map.q.out abd6459 ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out 5e69b31 ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 3418b99 ql/src/test/results/clientpositive/spark/groupby7_noskew_multi_single_reducer.q.out 2cb126d ql/src/test/results/clientpositive/spark/groupby8.q.out 307395f ql/src/test/results/clientpositive/spark/groupby8_map_skew.q.out ba04a57 ql/src/test/results/clientpositive/spark/insert_into3.q.out 7df5ba8 ql/src/test/results/clientpositive/spark/join22.q.out b1e5b67 ql/src/test/results/clientpositive/spark/skewjoinopt11.q.out 8a278ef ql/src/test/results/clientpositive/spark/union10.q.out 5e8fe38 ql/src/test/results/clientpositive/spark/union11.q.out 20c27c7 ql/src/test/results/clientpositive/spark/union20.q.out 6f0dca6 ql/src/test/results/clientpositive/spark/union28.q.out 98582df ql/src/test/results/clientpositive/spark/union3.q.out 834b6d4 ql/src/test/results/clientpositive/spark/union30.q.out 3409623 ql/src/test/results/clientpositive/spark/union4.q.out c121ef0 ql/src/test/results/clientpositive/spark/union5.q.out afee988 ql/src/test/results/clientpositive/spark/union_remove_1.q.out ba0e293 ql/src/test/results/clientpositive/spark/union_remove_15.q.out 26cfbab ql/src/test/results/clientpositive/spark/union_remove_16.q.out 7a7aaf2 ql/src/test/results/clientpositive/spark/union_remove_18.q.out a5e15c5 ql/src/test/results/clientpositive/spark/union_remove_19.q.out ad44400 ql/src/test/results/clientpositive/spark/union_remove_20.q.out 1d67177 ql/src/test/results/clientpositive/spark/union_remove_21.q.out 9f5b070 ql/src/test/results/clientpositive/spark/union_remove_22.q.out 2e01432 ql/src/test/results/clientpositive/spark/union_remove_24.q.out 2659798 ql/src/test/results/clientpositive/spark/union_remove_25.q.out 0a94684 ql/src/test/results/clientpositive/spark/union_remove_4.q.out 6c3d596 ql/src/test/results/clientpositive/spark/union_remove_6.q.out cd36189 ql/src/test/results/clientpositive/spark/union_remove_6_subq.q.out c981ae4
[jira] [Created] (HIVE-11082) Support multi edge between nodes in SparkPlan[Spark Branch]
Chengxiang Li created HIVE-11082: Summary: Support multi edge between nodes in SparkPlan[Spark Branch] Key: HIVE-11082 URL: https://issues.apache.org/jira/browse/HIVE-11082 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li For Dynamic RDD caching optimization, we found SparkPlan::connect throw exception while we try to combine 2 works with same child, support multi edge between nodes in SparkPlan would help to enable dynamic RDD caching in more use cases, like self join and self union. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 34757: HIVE-10844: Combine equivalent Works for HoS[Spark Branch]
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34757/ --- (Updated June 23, 2015, 7:24 a.m.) Review request for hive and Xuefu Zhang. Changes --- fix Xuefu's second round comments. Bugs: HIVE-10844 https://issues.apache.org/jira/browse/HIVE-10844 Repository: hive-git Description --- Some Hive queries(like TPCDS Q39) may share the same subquery, which translated into sperate, but equivalent Works in SparkWork, combining these equivalent Works into a single one would help to benifit from following dynamic RDD caching optimization. Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/optimizer/OperatorComparatorFactory.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/CombineEquivalentWorkResolver.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 19aae70 ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java b307b16 ql/src/test/results/clientpositive/spark/auto_join30.q.out 7b5c5e7 ql/src/test/results/clientpositive/spark/auto_smb_mapjoin_14.q.out 8a43d78 ql/src/test/results/clientpositive/spark/groupby10.q.out dd9d9fe ql/src/test/results/clientpositive/spark/groupby7_map.q.out abd6459 ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out 5e69b31 ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 3418b99 ql/src/test/results/clientpositive/spark/groupby7_noskew_multi_single_reducer.q.out 2cb126d ql/src/test/results/clientpositive/spark/groupby8.q.out c249b61 ql/src/test/results/clientpositive/spark/groupby8_map_skew.q.out 2fb1d73 ql/src/test/results/clientpositive/spark/insert_into3.q.out 7df5ba8 ql/src/test/results/clientpositive/spark/join22.q.out b1e5b67 ql/src/test/results/clientpositive/spark/skewjoinopt11.q.out 8a278ef ql/src/test/results/clientpositive/spark/union10.q.out 5e8fe38 ql/src/test/results/clientpositive/spark/union11.q.out 20c27c7 ql/src/test/results/clientpositive/spark/union20.q.out 6f0dca6 ql/src/test/results/clientpositive/spark/union28.q.out 98582df ql/src/test/results/clientpositive/spark/union3.q.out 834b6d4 ql/src/test/results/clientpositive/spark/union30.q.out 3409623 ql/src/test/results/clientpositive/spark/union4.q.out c121ef0 ql/src/test/results/clientpositive/spark/union5.q.out afee988 ql/src/test/results/clientpositive/spark/union_remove_1.q.out ba0e293 ql/src/test/results/clientpositive/spark/union_remove_15.q.out 26cfbab ql/src/test/results/clientpositive/spark/union_remove_16.q.out 7a7aaf2 ql/src/test/results/clientpositive/spark/union_remove_18.q.out a5e15c5 ql/src/test/results/clientpositive/spark/union_remove_19.q.out ad44400 ql/src/test/results/clientpositive/spark/union_remove_20.q.out 1d67177 ql/src/test/results/clientpositive/spark/union_remove_21.q.out 9f5b070 ql/src/test/results/clientpositive/spark/union_remove_22.q.out 2e01432 ql/src/test/results/clientpositive/spark/union_remove_24.q.out 2659798 ql/src/test/results/clientpositive/spark/union_remove_25.q.out 0a94684 ql/src/test/results/clientpositive/spark/union_remove_4.q.out 6c3d596 ql/src/test/results/clientpositive/spark/union_remove_6.q.out cd36189 ql/src/test/results/clientpositive/spark/union_remove_6_subq.q.out c981ae4 ql/src/test/results/clientpositive/spark/union_remove_7.q.out 084fbd6 ql/src/test/results/clientpositive/spark/union_top_level.q.out dede1ef Diff: https://reviews.apache.org/r/34757/diff/ Testing --- Thanks, chengxiang li
[jira] [Created] (HIVE-11089) Hive Streaming: connection fails when using a proxy user UGI
Adam Kunicki created HIVE-11089: --- Summary: Hive Streaming: connection fails when using a proxy user UGI Key: HIVE-11089 URL: https://issues.apache.org/jira/browse/HIVE-11089 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 1.2.0, 1.0.0, 0.14.0 Reporter: Adam Kunicki HIVE-8427 adds a call to ugi.hasKerberosCredentials() to check whether the connection is supposed to be a secure connection. This however breaks support for Proxy Users as a proxy user UGI will always return false to hasKerberosCredentials(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
