Hive-trunk-h0.21 - Build # 1113 - Fixed
Changes for Build #1112 [jvs] HIVE-2561 [jira] Add an annotation to UDFs to allow them to specify additional FILE/JAR resources necessary for execution Summary: Add requirements. Often times UDFs will have dependencies to external JARs/FILEs. It makes sense for these to be encoded by the UDF (rather than having the caller remember the set of files that need to be ADDed). Let's add an annotation to UDFs which will cause these resources to be auto-added. Test Plan: EMPTY Reviewers: JIRA, jsichi Reviewed By: jsichi CC: jsichi, jonchang, atfiore Differential Revision: 507 Changes for Build #1113 [jvs] HIVE-2613. HiveCli eclipse launch configuration hangs (Carl Steinbach via jvs) All tests passed The Apache Jenkins build system has built Hive-trunk-h0.21 (build #1113) Status: Fixed Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1113/ to view the results.
[jira] [Commented] (HIVE-2613) HiveCli eclipse launch configuration hangs
[ https://issues.apache.org/jira/browse/HIVE-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13159896#comment-13159896 ] Hudson commented on HIVE-2613: -- Integrated in Hive-trunk-h0.21 #1113 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1113/]) HIVE-2613. HiveCli eclipse launch configuration hangs (Carl Steinbach via jvs) jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1208228 Files : * /hive/trunk/common/src/java/conf/hive-log4j.properties * /hive/trunk/eclipse-templates/HiveCLI.launchtemplate HiveCli eclipse launch configuration hangs -- Key: HIVE-2613 URL: https://issues.apache.org/jira/browse/HIVE-2613 Project: Hive Issue Type: Bug Components: Build Infrastructure Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.9.0 Attachments: HIVE-2613.1.patch.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2468) Make Hive compile against Hadoop 0.23
[ https://issues.apache.org/jira/browse/HIVE-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13159922#comment-13159922 ] Amareshwari Sriramadasu commented on HIVE-2468: --- I could compile hive against Hadoop 0.23 without any issues with the latest patch. Thanks Tom. Can you also include the change in bin/hive and make it Patch available? I also started running tests. Will get back if i find any issues. Make Hive compile against Hadoop 0.23 - Key: HIVE-2468 URL: https://issues.apache.org/jira/browse/HIVE-2468 Project: Hive Issue Type: Task Reporter: Konstantin Shvachko Assignee: Tom White Attachments: HIVE-2468.patch, HIVE-2468.patch, HIVE-2468.patch, HIVE-2468.patch Due to restructure of Hadoop 0.22 branch compared to Hadoop 0.20 Hive does not compile against 0.22 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
HIVE-2526 patch need review and commit
Hi, I have added patch for HIVE-2526. Please review it. Thanks, Priyadarshini
Re: Review Request: HIVE-2468 - Make Hive compile against Hadoop 0.22
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2762/ --- (Updated 2011-11-30 16:57:27.344021) Review request for hive. Summary --- See https://issues.apache.org/jira/browse/HIVE-2468 This addresses bug HIVE-2468. https://issues.apache.org/jira/browse/HIVE-2468 Diffs (updated) - trunk/bin/hive 1203794 trunk/build-common.xml 1203794 trunk/build.properties 1203794 trunk/build.xml 1203794 trunk/conf/hive-default.xml 1203794 trunk/contrib/build.xml 1203794 trunk/hbase-handler/build.xml 1203794 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 1203794 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java 1203794 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java 1203794 trunk/hwi/build.xml 1203794 trunk/jdbc/build.xml 1203794 trunk/ql/build.xml 1203794 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1203794 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JobTrackerURLResolver.java 1203794 trunk/service/build.xml 1203794 trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1203794 trunk/shims/build.xml 1203794 trunk/shims/src/0.20/java/org/apache/hadoop/fs/ProxyFileSystem.java 1203794 trunk/shims/src/0.20/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java 1203794 trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 1203794 trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 1203794 trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Jetty23Shims.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenIdentifier23.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSelector23.java PRE-CREATION trunk/shims/src/common/java/org/apache/hadoop/fs/ProxyFileSystem.java PRE-CREATION trunk/shims/src/common/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java PRE-CREATION trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java 1203794 trunk/shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java 1203794 Diff: https://reviews.apache.org/r/2762/diff Testing --- Thanks, Tom
[jira] [Commented] (HIVE-2468) Make Hive compile against Hadoop 0.23
[ https://issues.apache.org/jira/browse/HIVE-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160145#comment-13160145 ] jirapos...@reviews.apache.org commented on HIVE-2468: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2762/ --- (Updated 2011-11-30 16:57:27.344021) Review request for hive. Summary --- See https://issues.apache.org/jira/browse/HIVE-2468 This addresses bug HIVE-2468. https://issues.apache.org/jira/browse/HIVE-2468 Diffs (updated) - trunk/bin/hive 1203794 trunk/build-common.xml 1203794 trunk/build.properties 1203794 trunk/build.xml 1203794 trunk/conf/hive-default.xml 1203794 trunk/contrib/build.xml 1203794 trunk/hbase-handler/build.xml 1203794 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 1203794 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java 1203794 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java 1203794 trunk/hwi/build.xml 1203794 trunk/jdbc/build.xml 1203794 trunk/ql/build.xml 1203794 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1203794 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JobTrackerURLResolver.java 1203794 trunk/service/build.xml 1203794 trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1203794 trunk/shims/build.xml 1203794 trunk/shims/src/0.20/java/org/apache/hadoop/fs/ProxyFileSystem.java 1203794 trunk/shims/src/0.20/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java 1203794 trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 1203794 trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 1203794 trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Jetty23Shims.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenIdentifier23.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSelector23.java PRE-CREATION trunk/shims/src/common/java/org/apache/hadoop/fs/ProxyFileSystem.java PRE-CREATION trunk/shims/src/common/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java PRE-CREATION trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java 1203794 trunk/shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java 1203794 Diff: https://reviews.apache.org/r/2762/diff Testing --- Thanks, Tom Make Hive compile against Hadoop 0.23 - Key: HIVE-2468 URL: https://issues.apache.org/jira/browse/HIVE-2468 Project: Hive Issue Type: Task Reporter: Konstantin Shvachko Assignee: Tom White Attachments: HIVE-2468.patch, HIVE-2468.patch, HIVE-2468.patch, HIVE-2468.patch, HIVE-2468.patch Due to restructure of Hadoop 0.22 branch compared to Hadoop 0.20 Hive does not compile against 0.22 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2467) HA Support for Metastore Server
[ https://issues.apache.org/jira/browse/HIVE-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160178#comment-13160178 ] Thomas Weise commented on HIVE-2467: All unit tests pass with the patch. HA Support for Metastore Server Key: HIVE-2467 URL: https://issues.apache.org/jira/browse/HIVE-2467 Project: Hive Issue Type: Improvement Components: Metastore, Security, Server Infrastructure Affects Versions: 0.8.0, 0.9.0 Reporter: Thomas Weise Assignee: Thomas Weise Fix For: 0.9.0 Attachments: HIVE-2467.2.patch, HIVE-2467.patch We require HA deployment for metastore server for HCatalog: * Multiple server instances run behind VIP * Database provides HA Metastore server instances will need to be able to share any state required for VIP outside RDBMS. As of Hive 0.8 affected conversational state that needs to support VIP/HA setup is limited to current delegation tokens. Is this correct? We are planning to use ZooKeeper to share current delegation tokens and master keys between nodes of the VIP. ZK is already (optionally) used by Hive for concurrency control. Access to ZK would be limited on the network level or in the future, when ZooKeeper supports security, through Kerberos, similar to NN access. Currently Hive taps into Hadoop core security delegation token support through extension of org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManagerTokenIdent A solution could amend the Hive specific extension to support: * Pluggable delegation token and master key store (ZooKeeper as alternative for in-memory AbstractDelegationTokenSecretManager) * Delegation token retrieval from token store when not found in memory (wrap/extend retrievePassword(...)) * Cancellation of token in token store * Purging of expired tokens from token store http://www.mail-archive.com/hcatalog-user@incubator.apache.org/msg00053.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2617) Insert overwrite table db.tname fails if partition already exists
Insert overwrite table db.tname fails if partition already exists -- Key: HIVE-2617 URL: https://issues.apache.org/jira/browse/HIVE-2617 Project: Hive Issue Type: Bug Components: Metastore Reporter: Aniket Mokashi Insert Overwrite table db.tname fails if partition already exists. For example- insert overwrite table db.tname PARTITION(part='p') select .. from t2 where part='p'; fails if partition 'p' already exists. Workaround is - use db; and the fire the command. From the source code- alterPartition(tbl.getTableName(), new Partition(tbl, tpart)); takes String tablename as argument and loses db information. Table table = newTable(tablename) is called to retrieve table from name. But, it relies on currentDatabase value (hence the workaround). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2468) Make Hive compile against Hadoop 0.23
[ https://issues.apache.org/jira/browse/HIVE-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160209#comment-13160209 ] Thomas Weise commented on HIVE-2468: I assume (based on findings in linked Pig issue) that when Hive is built against .23 then it won't be byte code compatible with .20 and so even though the shim is present cannot run on .20? Make Hive compile against Hadoop 0.23 - Key: HIVE-2468 URL: https://issues.apache.org/jira/browse/HIVE-2468 Project: Hive Issue Type: Task Reporter: Konstantin Shvachko Assignee: Tom White Attachments: HIVE-2468.patch, HIVE-2468.patch, HIVE-2468.patch, HIVE-2468.patch, HIVE-2468.patch Due to restructure of Hadoop 0.22 branch compared to Hadoop 0.20 Hive does not compile against 0.22 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2618) Describe partition returns table columns but should return partition columns
Describe partition returns table columns but should return partition columns Key: HIVE-2618 URL: https://issues.apache.org/jira/browse/HIVE-2618 Project: Hive Issue Type: Bug Reporter: Kevin Wilfong If a partitioned table and some partitions are created, and then the table is altered adding a columns, if describe is called on the partitions created before the columns were added it will show the new columns, even though it should not. In particular, in the metastore, the partition will not have these columns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2618) Describe partition returns table columns but should return partition columns
[ https://issues.apache.org/jira/browse/HIVE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong reassigned HIVE-2618: --- Assignee: Namit Jain Describe partition returns table columns but should return partition columns Key: HIVE-2618 URL: https://issues.apache.org/jira/browse/HIVE-2618 Project: Hive Issue Type: Bug Reporter: Kevin Wilfong Assignee: Namit Jain If a partitioned table and some partitions are created, and then the table is altered adding a columns, if describe is called on the partitions created before the columns were added it will show the new columns, even though it should not. In particular, in the metastore, the partition will not have these columns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2468) Make Hive compile against Hadoop 0.23
[ https://issues.apache.org/jira/browse/HIVE-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160250#comment-13160250 ] Carl Steinbach commented on HIVE-2468: -- +1. Will commit if tests pass. Make Hive compile against Hadoop 0.23 - Key: HIVE-2468 URL: https://issues.apache.org/jira/browse/HIVE-2468 Project: Hive Issue Type: Task Reporter: Konstantin Shvachko Assignee: Tom White Attachments: HIVE-2468.patch, HIVE-2468.patch, HIVE-2468.patch, HIVE-2468.patch, HIVE-2468.patch Due to restructure of Hadoop 0.22 branch compared to Hadoop 0.20 Hive does not compile against 0.22 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 1114 - Failure
Changes for Build #1114 2 tests failed. REGRESSION: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket4 Error Message: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. Stack Trace: junit.framework.AssertionFailedError: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. at junit.framework.Assert.fail(Assert.java:50) at org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket4(TestMinimrCliDriver.java:146) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:422) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:931) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:785) REGRESSION: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6 Error Message: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. Stack Trace: junit.framework.AssertionFailedError: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. at junit.framework.Assert.fail(Assert.java:50) at org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6(TestMinimrCliDriver.java:242) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:422) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:931) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:785) The Apache Jenkins build system has built Hive-trunk-h0.21 (build #1114) Status: Failure Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1114/ to view the results.
[jira] [Updated] (HIVE-2616) Passing user identity from metastore client to server in non-secure mode
[ https://issues.apache.org/jira/browse/HIVE-2616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-2616: --- Attachment: hive-2616_1.patch Previous patch had some bugs. Updated patch fixes them, all the test in metasore directory passes with this. Passing user identity from metastore client to server in non-secure mode Key: HIVE-2616 URL: https://issues.apache.org/jira/browse/HIVE-2616 Project: Hive Issue Type: Bug Components: Metastore Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: hive-2616.patch, hive-2616_1.patch Currently in unsecure mode client don't pass on user identity. As a result hdfs and other operations done by server gets executed by user running metastore process instead of being done in context of client. This results in problem as reported here: http://mail-archives.apache.org/mod_mbox/hive-user/20.mbox/%3CCAK0mCrRC3aPqtRHDe2J25Rm0JX6TS1KXxd7KPjqJjoqBjg=a...@mail.gmail.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2619) Add hook to run in meatastore's endFunction which can collect more fb303 counters
Add hook to run in meatastore's endFunction which can collect more fb303 counters - Key: HIVE-2619 URL: https://issues.apache.org/jira/browse/HIVE-2619 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Create the potential for hooks to run in the endFunction method of HMSHandler which take the name of a function and whether or not it succeeded. Also, override getCounters from fb303 to allow these hooks to add counters which they collect, should this be desired. These hooks can be similar to EventListeners, but they should be more generic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2468) Make Hive compile against Hadoop 0.23
[ https://issues.apache.org/jira/browse/HIVE-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160623#comment-13160623 ] Carl Steinbach commented on HIVE-2468: -- Committed to trunk. Thanks Tom! Make Hive compile against Hadoop 0.23 - Key: HIVE-2468 URL: https://issues.apache.org/jira/browse/HIVE-2468 Project: Hive Issue Type: Task Reporter: Konstantin Shvachko Assignee: Tom White Attachments: HIVE-2468.patch, HIVE-2468.patch, HIVE-2468.patch, HIVE-2468.patch, HIVE-2468.patch Due to restructure of Hadoop 0.22 branch compared to Hadoop 0.20 Hive does not compile against 0.22 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HIVE-2339) Preserve RS key columns in columnExprMap after CP optimization, which might be useful to other optimizers
[ https://issues.apache.org/jira/browse/HIVE-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis reopened HIVE-2339: - It seemed to be fixed by HIVE-1989. It was true for join cases but for group-by cases, mapping information of keys are still missing. Preserve RS key columns in columnExprMap after CP optimization, which might be useful to other optimizers - Key: HIVE-2339 URL: https://issues.apache.org/jira/browse/HIVE-2339 Project: Hive Issue Type: Sub-task Components: Query Processor Reporter: Navis Priority: Trivial Fix For: 0.9.0 Attachments: HIVE-2339.1.patch In ColumnPrunerProcFactory#pruneReduceSinkOperator, only VALUE parts are retained from columnExprMap. Doesn't anyone want KEY parts to retained, either? In my case, it was very useful for backtracking column names and removing RS in *-RS-*-RS-GBY case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2619) Add hook to run in meatastore's endFunction which can collect more fb303 counters
[ https://issues.apache.org/jira/browse/HIVE-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160651#comment-13160651 ] Namit Jain commented on HIVE-2619: -- https://reviews.facebook.net/D561 Add hook to run in meatastore's endFunction which can collect more fb303 counters - Key: HIVE-2619 URL: https://issues.apache.org/jira/browse/HIVE-2619 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2619.1.patch.txt Create the potential for hooks to run in the endFunction method of HMSHandler which take the name of a function and whether or not it succeeded. Also, override getCounters from fb303 to allow these hooks to add counters which they collect, should this be desired. These hooks can be similar to EventListeners, but they should be more generic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2619) Add hook to run in meatastore's endFunction which can collect more fb303 counters
[ https://issues.apache.org/jira/browse/HIVE-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160658#comment-13160658 ] Phabricator commented on HIVE-2619: --- njain has commented on the revision HIVE-2619 [jira] Add hook to run in meatastore's endFunction which can collect more fb303 counters. REVISION DETAIL https://reviews.facebook.net/D561 Add hook to run in meatastore's endFunction which can collect more fb303 counters - Key: HIVE-2619 URL: https://issues.apache.org/jira/browse/HIVE-2619 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2619.1.patch.txt Create the potential for hooks to run in the endFunction method of HMSHandler which take the name of a function and whether or not it succeeded. Also, override getCounters from fb303 to allow these hooks to add counters which they collect, should this be desired. These hooks can be similar to EventListeners, but they should be more generic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2619) Add hook to run in meatastore's endFunction which can collect more fb303 counters
[ https://issues.apache.org/jira/browse/HIVE-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160659#comment-13160659 ] Kevin Wilfong commented on HIVE-2619: - I also created a diff here https://reviews.facebook.net/D555 I thought it was supposed to update here automatically. Add hook to run in meatastore's endFunction which can collect more fb303 counters - Key: HIVE-2619 URL: https://issues.apache.org/jira/browse/HIVE-2619 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2619.1.patch.txt Create the potential for hooks to run in the endFunction method of HMSHandler which take the name of a function and whether or not it succeeded. Also, override getCounters from fb303 to allow these hooks to add counters which they collect, should this be desired. These hooks can be similar to EventListeners, but they should be more generic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2496) Allow ALTER TABLE RENAME between schemas
[ https://issues.apache.org/jira/browse/HIVE-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Natkins updated HIVE-2496: --- Assignee: (was: Jonathan Natkins) Allow ALTER TABLE RENAME between schemas Key: HIVE-2496 URL: https://issues.apache.org/jira/browse/HIVE-2496 Project: Hive Issue Type: New Feature Components: Metastore Reporter: Patrick Angeles Attachments: HIVE-2496.1.patch, HIVE-2496.2.patch Currently, this is not allowed which is unfortunate: ALTER TABLE db1.foo RENAME TO db2.foo ; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2619) Add hook to run in meatastore's endFunction which can collect more fb303 counters
[ https://issues.apache.org/jira/browse/HIVE-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160660#comment-13160660 ] Namit Jain commented on HIVE-2619: -- I thought so too - but it did not for some reason. John, do you know ? I did: ant arc-setup arc diff --jira HIVE-2619 Add hook to run in meatastore's endFunction which can collect more fb303 counters - Key: HIVE-2619 URL: https://issues.apache.org/jira/browse/HIVE-2619 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2619.1.patch.txt Create the potential for hooks to run in the endFunction method of HMSHandler which take the name of a function and whether or not it succeeded. Also, override getCounters from fb303 to allow these hooks to add counters which they collect, should this be desired. These hooks can be similar to EventListeners, but they should be more generic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2618) Describe partition returns table columns but should return partition columns
[ https://issues.apache.org/jira/browse/HIVE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160661#comment-13160661 ] Namit Jain commented on HIVE-2618: -- https://reviews.facebook.net/D105 Describe partition returns table columns but should return partition columns Key: HIVE-2618 URL: https://issues.apache.org/jira/browse/HIVE-2618 Project: Hive Issue Type: Bug Reporter: Kevin Wilfong Assignee: Namit Jain If a partitioned table and some partitions are created, and then the table is altered adding a columns, if describe is called on the partitions created before the columns were added it will show the new columns, even though it should not. In particular, in the metastore, the partition will not have these columns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-1003) optimize metadata only queries
[ https://issues.apache.org/jira/browse/HIVE-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-1003: -- Attachment: HIVE-1003.D105.3.patch njain updated the revision HIVE-1003 [jira] optimize metadata only queries. Reviewers: JIRA HIVE-2618 REVISION DETAIL https://reviews.facebook.net/D105 AFFECTED FILES ql/src/test/results/clientpositive/alter_table_serde.q.out ql/src/test/results/clientpositive/exim_04_evolved_parts.q.out ql/src/test/results/clientpositive/partition_schema1.q.out ql/src/test/queries/clientpositive/partition_schema1.q ql/src/java/org/apache/hadoop/hive/ql/metadata/MetaDataFormatUtils.java ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java optimize metadata only queries -- Key: HIVE-1003 URL: https://issues.apache.org/jira/browse/HIVE-1003 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain Assignee: Marcin Kurczych Fix For: 0.9.0 Attachments: D105.1.patch, D105.2.patch, HIVE-1003.1.patch, HIVE-1003.D105.3.patch, hive.1003.2.patch, hive.1003.3.patch, hive.1003.4.patch Queries like: select max(ds) from T where ds is a partitioning column should be optimized. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2619) Add hook to run in meatastore's endFunction which can collect more fb303 counters
[ https://issues.apache.org/jira/browse/HIVE-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160663#comment-13160663 ] Namit Jain commented on HIVE-2619: -- It did show up for me - a little delayed. I should have been more patient. Add hook to run in meatastore's endFunction which can collect more fb303 counters - Key: HIVE-2619 URL: https://issues.apache.org/jira/browse/HIVE-2619 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2619.1.patch.txt Create the potential for hooks to run in the endFunction method of HMSHandler which take the name of a function and whether or not it succeeded. Also, override getCounters from fb303 to allow these hooks to add counters which they collect, should this be desired. These hooks can be similar to EventListeners, but they should be more generic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2619) Add hook to run in meatastore's endFunction which can collect more fb303 counters
[ https://issues.apache.org/jira/browse/HIVE-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160664#comment-13160664 ] John Sichi commented on HIVE-2619: -- It should have added JIRA as one of the reviewers on the diff (that's how the comments get forwarded). Did you remove JIRA from the reviewers list? You can add it back on manually by editing the diff, and then arc diff again to update the diff and it should add info here. +Marek. Mm...meatastore. Can I get bacon with that? Add hook to run in meatastore's endFunction which can collect more fb303 counters - Key: HIVE-2619 URL: https://issues.apache.org/jira/browse/HIVE-2619 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2619.1.patch.txt Create the potential for hooks to run in the endFunction method of HMSHandler which take the name of a function and whether or not it succeeded. Also, override getCounters from fb303 to allow these hooks to add counters which they collect, should this be desired. These hooks can be similar to EventListeners, but they should be more generic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2619) Add hook to run in meatastore's endFunction which can collect more fb303 counters
[ https://issues.apache.org/jira/browse/HIVE-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2619: -- Attachment: HIVE-2619.D561.1.patch njain requested code review of HIVE-2619 [jira] Add hook to run in meatastore's endFunction which can collect more fb303 counters. Reviewers: JIRA HIVE-2619 Create the potential for hooks to run in the endFunction method of HMSHandler which take the name of a function and whether or not it succeeded. Also, override getCounters from fb303 to allow these hooks to add counters which they collect, should this be desired. These hooks can be similar to EventListeners, but they should be more generic. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D561 AFFECTED FILES conf/hive-default.xml metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEndFunctionListener.java metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java common/src/java/org/apache/hadoop/hive/conf/HiveConf.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/1251/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. Add hook to run in meatastore's endFunction which can collect more fb303 counters - Key: HIVE-2619 URL: https://issues.apache.org/jira/browse/HIVE-2619 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2619.1.patch.txt, HIVE-2619.D561.1.patch Create the potential for hooks to run in the endFunction method of HMSHandler which take the name of a function and whether or not it succeeded. Also, override getCounters from fb303 to allow these hooks to add counters which they collect, should this be desired. These hooks can be similar to EventListeners, but they should be more generic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2097) Explore mechanisms for better compression with RC Files
[ https://issues.apache.org/jira/browse/HIVE-2097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160687#comment-13160687 ] alex gemini commented on HIVE-2097: --- selectivity play an important role in columnar database is because they use run-length encoding compression to compress most dimension-attribute column,for example,we have a log table:create table (gender,age,region,message),we know that the selectivity order is :gender=1/2 age= 1/20 1/300, we can order table column like #1(gender,age,region,message) or #2(region,age,gender,message). for #1,we only need (2 + 2*20 + 2*20*300 +num_of_message) to store all the record in one dfs block, but if we organized table like #2,we will need (300 + 300*20 + 300*20*2 + num_of_message),discard num_of_message,the #1 is only take 66% of space #2 required,only difference is because run-length encoding will take more efficiently space when we organize table base on selectivity. Explore mechanisms for better compression with RC Files --- Key: HIVE-2097 URL: https://issues.apache.org/jira/browse/HIVE-2097 Project: Hive Issue Type: Improvement Components: Query Processor, Serializers/Deserializers Reporter: Krishna Kumar Assignee: Krishna Kumar Priority: Minor Optimization of the compression mechanisms used by RC File to be explored. Some initial ideas 1. More efficient serialization/deserialization based on type-specific and storage-specific knowledge. For instance, storing sorted numeric values efficiently using some delta coding techniques 2. More efficient compression based on type-specific and storage-specific knowledge Enable compression codecs to be specified based on types or individual columns 3. Reordering the on-disk storage for better compression efficiency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2065) RCFile issues
[
https://issues.apache.org/jira/browse/HIVE-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-2065:
-
Component/s: Serializers/Deserializers
Tags: rcfile
RCFile issues
-
Key: HIVE-2065
URL: https://issues.apache.org/jira/browse/HIVE-2065
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Krishna Kumar
Assignee: Krishna Kumar
Priority: Minor
Attachments: HIVE.2065.patch.0.txt, HIVE.2065.patch.1.txt,
Slide1.png, proposal.png
Some potential issues with RCFile
1. Remove unwanted synchronized modifiers on the methods of RCFile. As per
yongqiang he, the class is not meant to be thread-safe (and it is not). Might
as well get rid of the confusing and performance-impacting lock acquisitions.
2. Record Length overstated for compressed files. IIUC, the key compression
happens after we have written the record length.
{code}
int keyLength = key.getSize();
if (keyLength 0) {
throw new IOException(negative length keys not allowed: + key);
}
out.writeInt(keyLength + valueLength); // total record length
out.writeInt(keyLength); // key portion length
if (!isCompressed()) {
out.writeInt(keyLength);
key.write(out); // key
} else {
keyCompressionBuffer.reset();
keyDeflateFilter.resetState();
key.write(keyDeflateOut);
keyDeflateOut.flush();
keyDeflateFilter.finish();
int compressedKeyLen = keyCompressionBuffer.getLength();
out.writeInt(compressedKeyLen);
out.write(keyCompressionBuffer.getData(), 0, compressedKeyLen);
}
{code}
3. For sequence file compatibility, the compressed key length should be the
next field to record length, not the uncompressed key length.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 1115 - Still Failing
Changes for Build #1114 Changes for Build #1115 [cws] HIVE-2468. Make Hive compile against Hadoop 0.23 (Tom White via cws) 8 tests failed. REGRESSION: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_isnull_isnotnull Error Message: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. Stack Trace: junit.framework.AssertionFailedError: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. at junit.framework.Assert.fail(Assert.java:50) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_isnull_isnotnull(TestCliDriver.java:35198) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) REGRESSION: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_length Error Message: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. Stack Trace: junit.framework.AssertionFailedError: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. at junit.framework.Assert.fail(Assert.java:50) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_length(TestCliDriver.java:35342) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) REGRESSION: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_lessthan Error Message: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. Stack Trace: junit.framework.AssertionFailedError: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. at junit.framework.Assert.fail(Assert.java:50) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_lessthan(TestCliDriver.java:35390) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at
