Build failed in Jenkins: HIVE-TRUNK-JAVA8 #64
See http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/64/changes Changes: [Ashutosh Chauhan] HIVE-10451 : PTF deserializer fails if values are not used in reducer (Ashutosh Chauhan via Gopal V) [j.prasanth.j] HIVE-10370: Hive does not compile with -Phadoop-1 option (Prasanth Jayachandran reviewed by Vaibhav Gumashta) [j.prasanth.j] HIVE-10431: HIVE-9555 broke hadoop-1 build (Sergey Shelukhin reviewed by Prasanth Jayachandran) [j.prasanth.j] HIVE-10443: HIVE-9870 broke hadoop-1 build (Vaibhav Gumashta reviewed by Prasanth Jayachandran) [Xuefu Zhang] HIVE-10372: Update parquet version to 1.6.0 (Ferdinand via Xuefu) [thejas] HIVE-10447 : Beeline JDBC Driver to support 2 way SSL (Hari Sankar Sivarama Subramaniyan via Thejas Nair) [Ashutosh Chauhan] HIVE-10462 : CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered (Jesus Camacho Rodriguez via Ashutosh Chauhan) [Ashutosh Chauhan] HIVE-10490 : HBase Snapshot IF fails at run time with missing dependency of MetricsRegistry (Ashutosh Chauhan via Gunther Hagleitner) [jdere] HIVE-10421: DROP TABLE with qualified table name ignores database name when checking partitions (Jason Dere via Thejas Nair) [jdere] HIVE-10441: Fix confusing log statement in SessionState about hive.execution.engine setting (Jason Dere reviewed by Gunther Hagnleitner) [thejas] HIVE-4625 : HS2 should not attempt to get delegation token from metastore if using embedded metastore (Hari Sankar Sivarama Subramaniyan via Thejas Nair) [thejas] HIVE-10465 : whitelist restrictions don't get initialized in new copy of HiveConf (Thejas Nair, reviewed by Daniel Dai) [j.prasanth.j] HIVE-10442: HIVE-10098 broke hadoop-1 build (Yongzhi Chen reviewed by Prasanth Jayachandran) [khorgath] HIVE-10477 : Provide option to disable Spark tests (Hari Subramaniyan via Sushanth Sowmyan) -- [...truncated 321 lines...] A ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testBasic.approved.txt A ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testPrepNone.approved.txt A ptest2/src/test/java/org/apache/hive/ptest/execution/TestExecutionPhase.testFailingQFile.approved.txt A ptest2/src/test/java/org/apache/hive/ptest/execution/MockLocalCommandFactory.java A ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testAlternativeTestJVM.approved.txt Aptest2/src/test/java/org/apache/hive/ptest/execution/context A ptest2/src/test/java/org/apache/hive/ptest/execution/context/TestCloudComputeService.java A ptest2/src/test/java/org/apache/hive/ptest/execution/context/TestCloudExecutionContextProvider.java Aptest2/src/test/java/org/apache/hive/ptest/execution/TestJIRAService.java A ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testIsolatedFailsOnRsyncOne.approved.txt A ptest2/src/test/java/org/apache/hive/ptest/execution/TestExecutionPhase.testFailingUnitTest.approved.txt A ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testShutdownBeforeExec.approved.txt A ptest2/src/test/java/org/apache/hive/ptest/execution/TestPhase.testExecInstancesWithFailure.approved.txt Aptest2/src/test/java/org/apache/hive/ptest/execution/ssh A ptest2/src/test/java/org/apache/hive/ptest/execution/ssh/TestSSHCommandExecutor.java A ptest2/src/test/java/org/apache/hive/ptest/execution/ssh/TestRSyncCommandExecutor.java A ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testPrepGit.approved.txt A ptest2/src/test/java/org/apache/hive/ptest/execution/TestExecutionPhase.java A ptest2/src/test/java/org/apache/hive/ptest/execution/TestPhase.testRsyncFromLocalToRemoteInstancesWithFailureUnknown.approved.txt A ptest2/src/test/java/org/apache/hive/ptest/execution/MockSSHCommandExecutor.java A ptest2/src/test/java/org/apache/hive/ptest/execution/MockRSyncCommandExecutor.java Aptest2/src/test/java/org/apache/hive/ptest/execution/TestPrepPhase.java Aptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.java Aptest2/src/test/java/org/apache/hive/ptest/execution/TestReportParser.java A ptest2/src/test/java/org/apache/hive/ptest/execution/TestReportingPhase.java Aptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.java A ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testIsolatedFailsOnExec.approved.txt A ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testParallelFailsOnRsync.approved.txt Aptest2/src/test/java/org/apache/hive/ptest/execution/TestPhase.java A ptest2/src/test/java/org/apache/hive/ptest/execution/TestLogDirectoryCleaner.java A ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testPrepSvn.approved.txt A ptest2/src/test/java/org/apache/hive/ptest/execution/TestExecutionPhase.testPassingUnitTest.approved.txt A
Re: Do we still support hadoop-1.2.x API (-Phadoop-1)?
Hadoop 2 has been GA for a little over a year, there is still a fairly significant user base that uses hadoop-1 and would not be happy with this change. It should be removed at some point, but I'm not in favor of removing in next release which would be too soon. Perhaps we can declare it deprecated in some future release (perhaps 1.3), then another release to formally remove it, as was done in HBase. HBase did the formal removal in a major release (1.0) which is a lot cleaner, not sure if we have that luxury now that Hive 1.0 is forked. Thanks, Szehon On Tue, Apr 28, 2015 at 5:03 PM, Prasanth Jayachandran pjayachand...@hortonworks.com wrote: I recently filed 5 issues (HIVE-10430, 10431, 10442, 10443, 10444) related to build breakage of hadoop-1. There could potentially be more breakage. Some patches were added to reduce the number of file systems calls to improve performance but with supporting hadoop-1 we cannot directly use such APIs. Also we are not enforcing hadoop-1 build checks in hive QA to make sure every commit comes out clean on hadoop-1 and hadoop-2. I think it will be good if we can focus only hadoop-2. Not only it will simplify development but also will reduce the shims layer. Thanks Prasanth On Apr 28, 2015, at 4:40 PM, Owen O'Malley omal...@apache.org wrote: It has been three years since Hadoop 2.0.0 was first released and I believe that the vast majority of users that want to run Hive 1.x have moved over to Hadoop 2.x already. It will dramatically simplify Hive development if we drop backwards compatibility with the old Hadoop 1.x line. Thanks, Owen
Re: Do we still support hadoop-1.2.x API (-Phadoop-1)?
I recently filed 5 issues (HIVE-10430, 10431, 10442, 10443, 10444) related to build breakage of hadoop-1. There could potentially be more breakage. Some patches were added to reduce the number of file systems calls to improve performance but with supporting hadoop-1 we cannot directly use such APIs. Also we are not enforcing hadoop-1 build checks in hive QA to make sure every commit comes out clean on hadoop-1 and hadoop-2. I think it will be good if we can focus only hadoop-2. Not only it will simplify development but also will reduce the shims layer. Thanks Prasanth On Apr 28, 2015, at 4:40 PM, Owen O'Malley omal...@apache.org wrote: It has been three years since Hadoop 2.0.0 was first released and I believe that the vast majority of users that want to run Hive 1.x have moved over to Hadoop 2.x already. It will dramatically simplify Hive development if we drop backwards compatibility with the old Hadoop 1.x line. Thanks, Owen
Review Request 33650: HIVE-10516:Measure Hive CLI's performance difference before and after implementation is switched
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33650/ --- Review request for hive and Xuefu Zhang. Repository: hive-git Description --- Add microbenchmark for the current cli tool Diffs - itests/hive-jmh/pom.xml f3764c7 itests/hive-jmh/src/main/java/org/apache/hive/benchmark/cli/CliBench.java PRE-CREATION Diff: https://reviews.apache.org/r/33650/diff/ Testing --- Test locally Thanks, cheng xu
Re: Do we still support hadoop-1.2.x API (-Phadoop-1)?
It has been three years since Hadoop 2.0.0 was first released and I believe that the vast majority of users that want to run Hive 1.x have moved over to Hadoop 2.x already. It will dramatically simplify Hive development if we drop backwards compatibility with the old Hadoop 1.x line. Thanks, Owen
[jira] [Created] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
Laljo John Pullokkaran created HIVE-10526: - Summary: CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account Key: HIVE-10526 URL: https://issues.apache.org/jira/browse/HIVE-10526 Project: Hive Issue Type: Sub-task Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Do we still support hadoop-1.2.x API (-Phadoop-1)?
This thread needs [DISCUSS] in the subject. -- Lefty On Tue, Apr 28, 2015 at 10:58 PM, Owen O'Malley omal...@apache.org wrote: On Tue, Apr 28, 2015 at 5:25 PM, Szehon Ho sze...@cloudera.com wrote: Hadoop 2 has been GA for a little over a year, there is still a fairly significant user base that uses hadoop-1 and would not be happy with this change. The question isn't whether there are people running Hadoop 1.x, it is whether those people are likely to install a new version of Hive running on their old Hadoop cluster. As a point of reference, CDH 4 shipped Hadoop 2.0 and Hive 0.10 and HDP 2.0 shipped Hadoop 2.0 and Hive 0.12. Perhaps we can declare it deprecated in some future release (perhaps 1.3), then another release to formally remove it, as was done in HBase. Are you interested in managing a Hadoop 1.x compatible version of Hive? Maybe we should call the new release Hive 2.0 and enable you to maintain the Hive 1.x branch with backwards compatibility with Hadoop 1.x. That would enable the rest of the Hive community to move forward and take advantage of the powerful new features in Hadoop 2.x. .. Owen
[jira] [Created] (HIVE-10528) Hiveserver2 in HTTP mode is not applying auth_to_local rules
Abdelrahman Shettia created HIVE-10528: -- Summary: Hiveserver2 in HTTP mode is not applying auth_to_local rules Key: HIVE-10528 URL: https://issues.apache.org/jira/browse/HIVE-10528 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.14.0 Environment: Centos 6 Reporter: Abdelrahman Shettia PROBLEM: Authenticating to HS2 in HTTP mode with Kerberos, auth_to_local mappings do not get applied. Because of this various permissions checks which rely on the local cluster name for a user are going to fail. STEPS TO REPRODUCE: 1. Create kerberos cluster and HS2 in HTTP mode 2. Create a new user, test, along with a kerberos principal for this user 3. Create a separate principal, mapped-test 4. Create an auth_to_local rule to make sure that mapped-test is mapped to test 5. As the test user, connect to HS2 with beeline and create a simple table: {code} CREATE TABLE permtest (field1 int); {code} There is no need to load anything into this table. 6. Establish that it works as the test user: {code} show create table permtest; {code} 7. Drop the test identity and become mapped-test 8. Re-connect to HS2 with beeline, re-run the above command: {code} show create table permtest; {code} You will find that when this is done in HTTP mode, you will get an HDFS error (because of StorageBasedAuthorization doing a HDFS permissions check) and the user will be mapped-test and NOT test as it should be. ANALYSIS: This appears to be HTTP specific and the problem seems to come in {{ThriftHttpServlet$HttpKerberosServerAction.getPrincipalWithoutRealmAndHost()}}: {code} try { fullKerberosName = ShimLoader.getHadoopShims().getKerberosNameShim(fullPrincipal); } catch (IOException e) { throw new HttpAuthenticationException(e); } return fullKerberosName.getServiceName(); {code} getServiceName applies no auth_to_local rules. Seems like maybe this should be getShortName()? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Do we still support hadoop-1.2.x API (-Phadoop-1)?
On Tue, Apr 28, 2015 at 5:25 PM, Szehon Ho sze...@cloudera.com wrote: Hadoop 2 has been GA for a little over a year, there is still a fairly significant user base that uses hadoop-1 and would not be happy with this change. The question isn't whether there are people running Hadoop 1.x, it is whether those people are likely to install a new version of Hive running on their old Hadoop cluster. As a point of reference, CDH 4 shipped Hadoop 2.0 and Hive 0.10 and HDP 2.0 shipped Hadoop 2.0 and Hive 0.12. Perhaps we can declare it deprecated in some future release (perhaps 1.3), then another release to formally remove it, as was done in HBase. Are you interested in managing a Hadoop 1.x compatible version of Hive? Maybe we should call the new release Hive 2.0 and enable you to maintain the Hive 1.x branch with backwards compatibility with Hadoop 1.x. That would enable the rest of the Hive community to move forward and take advantage of the powerful new features in Hadoop 2.x. .. Owen
[jira] [Created] (HIVE-10527) NPE in SparkUtilities::isDedicatedCluster
Rui Li created HIVE-10527: - Summary: NPE in SparkUtilities::isDedicatedCluster Key: HIVE-10527 URL: https://issues.apache.org/jira/browse/HIVE-10527 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Rui Li Assignee: Rui Li We should add {{spark.master}} to HiveConf when it doesn't exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10529) Cleanup tezcontext reference in org.apache.hadoop.hive.ql.exec.tez.HashTableLoader
Rajesh Balamohan created HIVE-10529: --- Summary: Cleanup tezcontext reference in org.apache.hadoop.hive.ql.exec.tez.HashTableLoader Key: HIVE-10529 URL: https://issues.apache.org/jira/browse/HIVE-10529 Project: Hive Issue Type: Bug Reporter: Rajesh Balamohan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 33633: HIVE-10499 Ensure Session/ZooKeeperClient instances are closed
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33633/#review81878 --- Ship it! Ship It! - Szehon Ho On April 28, 2015, 5:56 p.m., Jimmy Xiang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33633/ --- (Updated April 28, 2015, 5:56 p.m.) Review request for hive and Szehon Ho. Bugs: HIVE-10499 https://issues.apache.org/jira/browse/HIVE-10499 Repository: hive-git Description --- Make sure Session/SessionState/ZooKeeperHiveClient instances are closed if some exception is thrown in a wrong place, so as to avoid some possible resource leakage. Diffs - jdbc/src/java/org/apache/hive/jdbc/ZooKeeperHiveClientHelper.java 496c820 service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java f14b974 service/src/java/org/apache/hive/service/cli/session/SessionManager.java e02997a service/src/java/org/apache/hive/service/server/HiveServer2.java 222cb45 Diff: https://reviews.apache.org/r/33633/diff/ Testing --- Thanks, Jimmy Xiang
Re: Review Request 33633: HIVE-10499 Ensure Session/ZooKeeperClient instances are closed
On April 28, 2015, 6:03 p.m., Szehon Ho wrote: service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java, line 568 https://reviews.apache.org/r/33633/diff/1/?file=944227#file944227line568 Is it necessary to call close twice? Can we consolidate the two finally blocks? The first one is in a block and may not get a chance to be executed. After the first one closes it, it is set to null. The second one is to make sure it is really closed. The first one throws an Exception. The second one just logs a warning. I thought about consolidating them but preferred not to change the existing behavior. - Jimmy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33633/#review81845 --- On April 28, 2015, 5:56 p.m., Jimmy Xiang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33633/ --- (Updated April 28, 2015, 5:56 p.m.) Review request for hive and Szehon Ho. Bugs: HIVE-10499 https://issues.apache.org/jira/browse/HIVE-10499 Repository: hive-git Description --- Make sure Session/SessionState/ZooKeeperHiveClient instances are closed if some exception is thrown in a wrong place, so as to avoid some possible resource leakage. Diffs - jdbc/src/java/org/apache/hive/jdbc/ZooKeeperHiveClientHelper.java 496c820 service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java f14b974 service/src/java/org/apache/hive/service/cli/session/SessionManager.java e02997a service/src/java/org/apache/hive/service/server/HiveServer2.java 222cb45 Diff: https://reviews.apache.org/r/33633/diff/ Testing --- Thanks, Jimmy Xiang
[jira] [Created] (HIVE-10525) loading data into list bucketing table when null in skew column
Gabriel C Balan created HIVE-10525: -- Summary: loading data into list bucketing table when null in skew column Key: HIVE-10525 URL: https://issues.apache.org/jira/browse/HIVE-10525 Project: Hive Issue Type: Bug Components: Hive Affects Versions: 1.1.0 Environment: linux Reporter: Gabriel C Balan Priority: Minor I'm trying to load data into a list bucketing table. The insert statement fails when there are nulls going into the skew column. If this is the expected behavior, there is no mention of this restriction in the doc. more *null.csv :: has-null.csv :: 1 2 \N 3 :: no-null.csv :: 1 2 3 set hive.mapred.supports.subdirectories=true; set hive.optimize.listbucketing=true; set mapred.input.dir.recursive=true; set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat; create table src_with_null (x int); load data local inpath 'has-null.csv' overwrite into table src_with_null; create table src_no_null (x int); load data local inpath 'no-null.csv' overwrite into table src_no_null; create table lb (x int) partitioned by (p string) skewed by ( x ) on (1) STORED AS DIRECTORIES stored as rcfile; insert overwrite table lb partition (p = 'foo') select * from src_with_null; --fails insert overwrite table lb partition (p = 'foo') select * from src_no_null; --succeeds I see this in ${hive.log.dir}/hive.log 2015-04-28 13:43:47,646 WARN [Thread-82]: mapred.LocalJobRunner (LocalJobRunner.java:run(560)) - job_local402607316_0001 java.lang.Exception: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {x:null} at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522) Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {x:null} at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:179) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {x:null} at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170) ... 10 more Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.FileSinkOperator.generateListBucketingDirName(FileSinkOperator.java:833) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:615) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10524) Add utility method ExprNodeDescUtils.forwardTrack()
Jason Dere created HIVE-10524: - Summary: Add utility method ExprNodeDescUtils.forwardTrack() Key: HIVE-10524 URL: https://issues.apache.org/jira/browse/HIVE-10524 Project: Hive Issue Type: Bug Reporter: Jason Dere Assignee: Jason Dere ExprNodeDescUtils has a method backtrack(), which is able to take an ExprNodeDesc from an operator and convert it to an equivalent expression based on the columns of a parent operator. Adding a forwardTrack() method to do something similar, but for a child operator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 28372: Address review comments
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/#review81884 --- Looking good to me. I found a few things to fix, and I'm also wondering where the qtest outputs are. ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java https://reviews.apache.org/r/28372/#comment132397 Sergio discovered that MAP_KEY_VALUE was incorrectly used in place of MAP. So instead of throwing UnsupportedOperationException, MAP_KEY_VALUE should be used as a synonym for MAP. That way we can read old data. ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java https://reviews.apache.org/r/28372/#comment132398 Shouldn't you also pass the exception so that the stack trace is printed? ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetToHiveSchemaConverter.java https://reviews.apache.org/r/28372/#comment132399 Minor: Would be good to use the Types API instead of constructors. ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetToHiveSchemaConverter.java https://reviews.apache.org/r/28372/#comment132401 I think the maps and lists should be constructed using the current best practice, ConversionPatterns, instead of by hand. That way we're testing what is actually going to be in data files. - Ryan Blue On April 16, 2015, 8:01 p.m., Ashish Singh wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/ --- (Updated April 16, 2015, 8:01 p.m.) Review request for hive. Bugs: HIVE-8950 https://issues.apache.org/jira/browse/HIVE-8950 Repository: hive-git Description --- HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e138800e6dadd6fe76345f21eb76c906165c438d hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 53d913435cd43c96f044cf8668461fc686817ef4 metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 3f267ff0eb20560c36a19b74353f9d6749c8b333 metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 907ad2b8d7bae9b13eb4d9605ff2a3fe60c03ee8 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 7fd5e9612d4e3c9bf3b816bc48dbdbe59fb8a5a8 ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java a029f10c116fd46070b2d41790043f0a7001390f ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetToHiveSchemaConverter.java PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
Re: Failing minimr tests
I have a patch for this, will be nice if someone can review and verify it. Thanks Hari From: Szehon Ho sze...@cloudera.com Sent: Monday, April 27, 2015 6:25 PM To: dev@hive.apache.org Subject: Re: Failing minimr tests I got around to create a JIRA for this: HIVE-10514 https://issues.apache.org/jira/browse/HIVE-10514, be great if somebody with some cycles could take a look. Thanks Szehon On Tue, Apr 14, 2015 at 8:43 AM, Thejas Nair thejas.n...@gmail.com wrote: code too large seems to refer to a method that is larger than 64kb - http://stackoverflow.com/questions/2407912/code-too-large-compilation-error-in-java I wonder if some auto generated method is getting too large. On Mon, Apr 13, 2015 at 11:20 PM, Szehon Ho sze...@cloudera.com wrote: FYI, was just taking a brief look at the recent failing minimr tests and found this exception when the build framework generates the tests: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project hive-it-qfile: Compilation failure [ERROR] /home/hiveptest/54.147.211.107-hiveptest-0/apache-svn-trunk-source/itests/qtest/target/generated-test-sources/java/org/apache/hadoop/hive/cli/TestCliDriver.java:[100,22] code too large I believe this is the command that it's running : timeout 2h mvn -B test -Dmaven.repo.local=/home/hiveptest//54.166.80.165-hiveptest-0/maven -Phadoop-2 -Phadoop-2 -Dtest=TestMinimrCliDriver -Dminimr.query.files=infer_bucket_sort_map_operators.q,join1.q,bucketmapjoin7.q,udf_using.q Thanks, Szehon
[jira] [Created] (HIVE-10520) LLAP: Must reset small table result columns for Native Vectorization of Map Join
Matt McCline created HIVE-10520: --- Summary: LLAP: Must reset small table result columns for Native Vectorization of Map Join Key: HIVE-10520 URL: https://issues.apache.org/jira/browse/HIVE-10520 Project: Hive Issue Type: Sub-task Components: Vectorization Reporter: Matt McCline Assignee: Matt McCline Priority: Blocker -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Do we still support hadoop-1.2.x API (-Phadoop-1)?
Hi Everyone I tried to compile the latest hive with hadoop-1 profile. It failed because TestLazySimpleFast (164) uses Text.copyBytes() which is hadoop-2.x API So, which hadoop API should we use in hive? old hadoop-1.x or new hadoop-2.x? Alex
Re: Review Request 33633: HIVE-10499 Ensure Session/ZooKeeperClient instances are closed
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33633/#review81845 --- service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java https://reviews.apache.org/r/33633/#comment132336 Is it necessary to call close twice? Can we consolidate the two finally blocks? - Szehon Ho On April 28, 2015, 5:56 p.m., Jimmy Xiang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33633/ --- (Updated April 28, 2015, 5:56 p.m.) Review request for hive and Szehon Ho. Bugs: HIVE-10499 https://issues.apache.org/jira/browse/HIVE-10499 Repository: hive-git Description --- Make sure Session/SessionState/ZooKeeperHiveClient instances are closed if some exception is thrown in a wrong place, so as to avoid some possible resource leakage. Diffs - jdbc/src/java/org/apache/hive/jdbc/ZooKeeperHiveClientHelper.java 496c820 service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java f14b974 service/src/java/org/apache/hive/service/cli/session/SessionManager.java e02997a service/src/java/org/apache/hive/service/server/HiveServer2.java 222cb45 Diff: https://reviews.apache.org/r/33633/diff/ Testing --- Thanks, Jimmy Xiang
Review Request 33633: HIVE-10499 Ensure Session/ZooKeeperClient instances are closed
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33633/ --- Review request for hive and Szehon Ho. Bugs: HIVE-10499 https://issues.apache.org/jira/browse/HIVE-10499 Repository: hive-git Description --- Make sure Session/SessionState/ZooKeeperHiveClient instances are closed if some exception is thrown in a wrong place, so as to avoid some possible resource leakage. Diffs - jdbc/src/java/org/apache/hive/jdbc/ZooKeeperHiveClientHelper.java 496c820 service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java f14b974 service/src/java/org/apache/hive/service/cli/session/SessionManager.java e02997a service/src/java/org/apache/hive/service/server/HiveServer2.java 222cb45 Diff: https://reviews.apache.org/r/33633/diff/ Testing --- Thanks, Jimmy Xiang
Re: Do we still support hadoop-1.2.x API (-Phadoop-1)?
I think its time to discuss about dropping support for Hadoop-1 line. What do folks think about Hive-1.2 being last release supporting Hadoop-1 line? Thanks, Ashutosh On Tue, Apr 28, 2015 at 11:02 AM, Alexander Pivovarov apivova...@gmail.com wrote: Hi Everyone I tried to compile the latest hive with hadoop-1 profile. It failed because TestLazySimpleFast (164) uses Text.copyBytes() which is hadoop-2.x API So, which hadoop API should we use in hive? old hadoop-1.x or new hadoop-2.x? Alex
[jira] [Created] (HIVE-10521) TxnHandler.timeOutTxns only times out some of the expired transactions
Alan Gates created HIVE-10521: - Summary: TxnHandler.timeOutTxns only times out some of the expired transactions Key: HIVE-10521 URL: https://issues.apache.org/jira/browse/HIVE-10521 Project: Hive Issue Type: Bug Components: Transactions Affects Versions: 1.0.0, 0.14.0, 1.1.0 Reporter: Alan Gates Assignee: Alan Gates {code} for (int i = 0; i 20 rs.next(); i++) deadTxns.add(rs.getLong(1)); // We don't care whether all of the transactions get deleted or not, // if some didn't it most likely means someone else deleted them in the interum if (deadTxns.size() 0) abortTxns(dbConn, deadTxns); {code} While it makes sense to limit the number of transactions aborted in one pass (since this get's translated to an IN clause) we should still make sure all are timed out. Also, 20 seems pretty small as a batch size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 28109: HiveServer2 dynamic service discovery: use persistent ephemeral nodes curator recipe
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28109/ --- (Updated April 28, 2015, 6:40 p.m.) Review request for hive and Thejas Nair. Bugs: HIVE-8890 https://issues.apache.org/jira/browse/HIVE-8890 Repository: hive-git Description --- https://issues.apache.org/jira/browse/HIVE-8890 Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c9ee423 pom.xml 9a1dae9 service/pom.xml 421bb9b service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java ca1eae6 service/src/java/org/apache/hive/service/server/HiveServer2.java 222cb45 Diff: https://reviews.apache.org/r/28109/diff/ Testing --- Thanks, Vaibhav Gumashta
Review Request 33637: HIVE-10519 Move TestGenericUDF classes to udf.generic package
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33637/ --- Review request for hive and Jason Dere. Bugs: HIVE-10519 https://issues.apache.org/jira/browse/HIVE-10519 Repository: hive-git Description --- HIVE-10519 Move TestGenericUDF classes to udf.generic package Diffs - ql/src/test/org/apache/hadoop/hive/ql/udf/TestGenericUDFDate.java 0d40ff7cb3c47ed4c2a8d2e88d9d8c32eb87d09d ql/src/test/org/apache/hadoop/hive/ql/udf/TestGenericUDFDateAdd.java 0f3ecd18ae04ca9e44f0d55400fbacd3a879f9fc ql/src/test/org/apache/hadoop/hive/ql/udf/TestGenericUDFDateDiff.java 849e70c17a1e2c839bf857ca22b5b61cda6de9e1 ql/src/test/org/apache/hadoop/hive/ql/udf/TestGenericUDFDateSub.java ae1b3db5568f86c26e281c3cedb42821ebc5b450 ql/src/test/org/apache/hadoop/hive/ql/udf/TestGenericUDFUtils.java 7399a7cd964913a1ae7f18b85cb050bba56acfe6 Diff: https://reviews.apache.org/r/33637/diff/ Testing --- Thanks, Alexander Pivovarov
Re: Review Request 33637: HIVE-10519 Move TestGenericUDF classes to udf.generic package
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33637/ --- (Updated April 28, 2015, 6:59 p.m.) Review request for hive and Jason Dere. Changes --- Organize Imports Bugs: HIVE-10519 https://issues.apache.org/jira/browse/HIVE-10519 Repository: hive-git Description --- HIVE-10519 Move TestGenericUDF classes to udf.generic package Diffs (updated) - ql/src/test/org/apache/hadoop/hive/ql/udf/TestGenericUDFDate.java 0d40ff7cb3c47ed4c2a8d2e88d9d8c32eb87d09d ql/src/test/org/apache/hadoop/hive/ql/udf/TestGenericUDFDateAdd.java 0f3ecd18ae04ca9e44f0d55400fbacd3a879f9fc ql/src/test/org/apache/hadoop/hive/ql/udf/TestGenericUDFDateDiff.java 849e70c17a1e2c839bf857ca22b5b61cda6de9e1 ql/src/test/org/apache/hadoop/hive/ql/udf/TestGenericUDFDateSub.java ae1b3db5568f86c26e281c3cedb42821ebc5b450 ql/src/test/org/apache/hadoop/hive/ql/udf/TestGenericUDFUtils.java 7399a7cd964913a1ae7f18b85cb050bba56acfe6 Diff: https://reviews.apache.org/r/33637/diff/ Testing --- Thanks, Alexander Pivovarov
[jira] [Created] (HIVE-10522) CBO (Calcite Return Path): fix the wrong needed column names when TS is created
Pengcheng Xiong created HIVE-10522: -- Summary: CBO (Calcite Return Path): fix the wrong needed column names when TS is created Key: HIVE-10522 URL: https://issues.apache.org/jira/browse/HIVE-10522 Project: Hive Issue Type: Sub-task Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Priority: Critical -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 28109: HiveServer2 dynamic service discovery: use persistent ephemeral nodes curator recipe
On Jan. 30, 2015, 8:26 p.m., Thejas Nair wrote: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 1337 https://reviews.apache.org/r/28109/diff/3/?file=836237#file836237line1337 how about using seconds instead of milliseconds, most people would not care about the millisecond part. HIVE_ZOOKEEPER_CONNECTION_BASESLEEPTIME uses ms and it is reasonable to expect it to be configured 1s. I'm thinking it's better to keep consistent units. What do you think? - Vaibhav --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28109/#review70303 --- On April 28, 2015, 6:40 p.m., Vaibhav Gumashta wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28109/ --- (Updated April 28, 2015, 6:40 p.m.) Review request for hive and Thejas Nair. Bugs: HIVE-8890 https://issues.apache.org/jira/browse/HIVE-8890 Repository: hive-git Description --- https://issues.apache.org/jira/browse/HIVE-8890 Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c9ee423 pom.xml 9a1dae9 service/pom.xml 421bb9b service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java ca1eae6 service/src/java/org/apache/hive/service/server/HiveServer2.java 222cb45 Diff: https://reviews.apache.org/r/28109/diff/ Testing --- Thanks, Vaibhav Gumashta
Review Request 33618: HIVE-10518 Replace US_ASCII charset with US-ASCII in encode and decode description
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33618/ --- Review request for hive and Jason Dere. Bugs: HIVE-10518 https://issues.apache.org/jira/browse/HIVE-10518 Repository: hive-git Description --- HIVE-10518 Replace US_ASCII charset with US-ASCII in encode and decode description Diffs - ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDecode.java f4366c7975c09d915f98d8e04c1039802cabdefa ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEncode.java 64f8a14f894635f49e32b095dc0974f9d5500e38 ql/src/test/results/clientpositive/udf_decode.q.out 97f0acf96dfe6e41b6841a65bdd33c7efe955e42 Diff: https://reviews.apache.org/r/33618/diff/ Testing --- Thanks, Alexander Pivovarov
[jira] [Created] (HIVE-10519) Move TestGenericUDF classes to udf.generic package
Alexander Pivovarov created HIVE-10519: -- Summary: Move TestGenericUDF classes to udf.generic package Key: HIVE-10519 URL: https://issues.apache.org/jira/browse/HIVE-10519 Project: Hive Issue Type: Bug Components: UDF Reporter: Alexander Pivovarov Assignee: Alexander Pivovarov Priority: Trivial 5 TestGenericUDF classes are located in udf package instead of udf.generic {code} TestGenericUDFDate.java TestGenericUDFDateAdd.java TestGenericUDFDateDiff.java TestGenericUDFDateSub.java TestGenericUDFUtils.java {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10523) Hive HCatalog Core 1.2.0 can not be built with hadoop-1 profile
Alexander Pivovarov created HIVE-10523: -- Summary: Hive HCatalog Core 1.2.0 can not be built with hadoop-1 profile Key: HIVE-10523 URL: https://issues.apache.org/jira/browse/HIVE-10523 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 1.2.0 Reporter: Alexander Pivovarov I tried to built HCatalog with hadoop-1 profile and got the following error {code} $ git status On branch branch-1.2 Your branch is up-to-date with 'origin/branch-1.2' $ mvn clean install -DskipTests -Phadoop-1 [ERROR] COMPILATION ERROR : [INFO] - [ERROR] /workhive/hive/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java:[515,19] cannot find symbol symbol: method isFile() location: variable fileStatus of type org.apache.hadoop.fs.FileStatus [ERROR] /workhive/hive/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java:[545,26] cannot find symbol symbol: method isDirectory() location: variable fileStatus of type org.apache.hadoop.fs.FileStatus [INFO] 2 errors [INFO] - [INFO] [INFO] Reactor Summary: [INFO] [INFO] Hive ... SUCCESS [ 3.181 s] [INFO] Hive Shims Common .. SUCCESS [ 4.292 s] [INFO] Hive Shims 0.20S ... SUCCESS [ 1.035 s] [INFO] Hive Shims 0.23 SUCCESS [ 5.692 s] [INFO] Hive Shims Scheduler ... SUCCESS [ 1.681 s] [INFO] Hive Shims . SUCCESS [ 1.302 s] [INFO] Hive Common SUCCESS [ 4.787 s] [INFO] Hive Serde . SUCCESS [ 5.501 s] [INFO] Hive Metastore . SUCCESS [ 15.634 s] [INFO] Hive Ant Utilities . SUCCESS [ 0.695 s] [INFO] Spark Remote Client SUCCESS [ 9.376 s] [INFO] Hive Query Language SUCCESS [01:19 min] [INFO] Hive Service ... SUCCESS [ 5.310 s] [INFO] Hive Accumulo Handler .. SUCCESS [ 2.462 s] [INFO] Hive JDBC .. SUCCESS [ 8.817 s] [INFO] Hive Beeline ... SUCCESS [ 1.636 s] [INFO] Hive CLI ... SUCCESS [ 4.843 s] [INFO] Hive Contrib ... SUCCESS [ 1.501 s] [INFO] Hive HBase Handler . SUCCESS [ 11.925 s] [INFO] Hive HCatalog .. SUCCESS [ 0.265 s] [INFO] Hive HCatalog Core . FAILURE [ 1.003 s] [INFO] Hive HCatalog Pig Adapter .. SKIPPED [INFO] Hive HCatalog Server Extensions SKIPPED [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED [INFO] Hive HCatalog Webhcat .. SKIPPED [INFO] Hive HCatalog Streaming SKIPPED [INFO] Hive HWI ... SKIPPED [INFO] Hive ODBC .. SKIPPED [INFO] Hive Shims Aggregator .. SKIPPED [INFO] Hive TestUtils . SKIPPED [INFO] Hive Packaging . SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 02:51 min [INFO] Finished at: 2015-04-28T12:20:09-07:00 [INFO] Final Memory: 179M/649M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-hcatalog-core: Compilation failure: Compilation failure: [ERROR] /workhive/hive/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java:[515,19] cannot find symbol [ERROR] symbol: method isFile() [ERROR] location: variable fileStatus of type org.apache.hadoop.fs.FileStatus [ERROR] /workhive/hive/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java:[545,26] cannot find symbol [ERROR] symbol: method isDirectory() [ERROR] location: variable fileStatus of type org.apache.hadoop.fs.FileStatus [ERROR] - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
Re: Preparation for Hive-1.2 release
Hi Folks, As per the timeline set on https://cwiki.apache.org/confluence/display/Hive/Hive+1.2+Release+Status , the time for requests for inclusion of any new features to branch-1.2 is now closed. Any JIRAS already marked in the lists previously will be honoured and considered release blockers, but no new feature JIRAs will be included. The list is still open for any new bugfix jira to be added to the list. I will start getting RC0 ready now. The purpose of RC0 is not for it to be a true release-candidate, but simply to go through the process of creating a release, verifying instructions set to create a release, so that we can discover any hiccups along the way. Once that is done, the next deadline of note for the community is 15:01, Thursday, 30th Apr 2015 PDT, which is the deadline for requesting any bugfix jiras for inclusion. After this deadline, the list will go into further lockdown, and allow inclusion of only jiras already in the currently accepted lists, and either critical bugfixes which are product breakages without workarounds, or bugs discovered during testing of the RCs. Thanks, -Sushanth On Mon, Apr 27, 2015 at 4:51 PM, Sushanth Sowmyan khorg...@gmail.com wrote: HIVE-10509 and HIVE-10510 have been created with the individual patches that upgraded master and branch-1.2 respectively, and marked as resolved. I'll also add it in to the HowToRelease doc to ensure that from the next release onward, we create a jira for this. Also, friendly reminder for folks : If you have any other jira you want included for 1.2, please go and add it to https://cwiki.apache.org/confluence/display/Hive/Hive+1.2+Release+Status Tonight midnight will be the deadline for feature include requests, and this list will go on lockdown for any further requests for feature inclusions. Bugfixes will still be okay to request for inclusion. On Mon, Apr 27, 2015 at 4:41 PM, Sushanth Sowmyan khorg...@gmail.com wrote: Also, Agreed with Sergio/Szehon, we should have a jira that's marked as committed with a patch attached for the version bump patch. Creating one. On Mon, Apr 27, 2015 at 4:39 PM, Sushanth Sowmyan khorg...@gmail.com wrote: Vinod, If 2.7.0 is not-yet-ready, then we should not be bumping 1.2 to 2.7.0. But, we should be okay to bump master(1.3) up to 2.7.0 so that we start general-purpose testing against that. Please go ahead and create a jira for that, and we should be able to bump up trunk to go against that. Thanks, -Sushanth