[jira] [Updated] (HIVE-2908) Hive: Extend ALTER TABLE DROP PARTITION syntax to use all comparators
[ https://issues.apache.org/jira/browse/HIVE-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2908: -- Attachment: HIVE-2908.D2523.1.patch sambavim requested code review of HIVE-2908 [jira] Hive: Extend ALTER TABLE DROP PARTITION syntax to use all comparators. Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2908 This diff introduces drop partition with non-equality comparisons on the partition key. You can now issue commands of the form: alter table tbl drop partition (keyvalue) to drop all partitions which satisfy the predicate. Changes were introduced to hive.g to allow the new operators: , , , !=, =, = in alter table drop partition statements. The change is implemented by using the method getPartitionsByFilter on the Hive object. In order to cleanup the passing around of the partition spec information, I introduced a new class called PartitionSpec which encapsulates one complete partition spec (multiple key value comparisons). The drop statement thus has a list of the PartitionSpec objects. Unit tests have been added to test the new functionality. Some existing test results needed an update due to the change in the AST for the partition spec. All unit tests passed with this change. The test results of the escape1.q test need an update as well but arc diff complains since it concludes the file is binary. I will work with the committer to make sure that change is captured before commit. To drop a partition from a Hive table, this works: ALTER TABLE foo DROP PARTITION(ds = 'date') ...but it should also work to drop all partitions prior to date. ALTER TABLE foo DROP PARTITION(ds 'date') This task is to implement ALTER TABLE DROP PARTITION for all of the comparators, = = = != instead of just for =. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2523 AFFECTED FILES ql/src/test/results/clientpositive/drop_partitions_filter.q.out ql/src/test/results/clientpositive/drop_multi_partitions.q.out ql/src/test/results/clientnegative/drop_partition_filter_failure.q.out ql/src/test/queries/clientpositive/drop_partitions_filter.q ql/src/test/queries/clientnegative/drop_partition_filter_failure.q ql/src/java/org/apache/hadoop/hive/ql/plan/DropTableDesc.java ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionSpec.java ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/5685/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. Hive: Extend ALTER TABLE DROP PARTITION syntax to use all comparators - Key: HIVE-2908 URL: https://issues.apache.org/jira/browse/HIVE-2908 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Sambavi Muthukrishnan Assignee: Sambavi Muthukrishnan Priority: Minor Attachments: HIVE-2908.D2523.1.patch Original Estimate: 96h Remaining Estimate: 96h To drop a partition from a Hive table, this works: ALTER TABLE foo DROP PARTITION(ds = 'date') ...but it should also work to drop all partitions prior to date. ALTER TABLE foo DROP PARTITION(ds 'date') This task is to implement ALTER TABLE DROP PARTITION for all of the comparators, = = = != instead of just for =. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2910) Improve the HWI interface
Improve the HWI interface - Key: HIVE-2910 URL: https://issues.apache.org/jira/browse/HIVE-2910 Project: Hive Issue Type: Improvement Components: Web UI Reporter: Hugo Trippaers Priority: Minor Attachments: hive-hwi.patch I've made some improvements to the HWI interface with the Twitter bootstrap system. I'm looking for feedback on the new design. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2910) Improve the HWI interface
[ https://issues.apache.org/jira/browse/HIVE-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hugo Trippaers updated HIVE-2910: - Attachment: hive-hwi.patch Patch (generated by Eclipse) Improve the HWI interface - Key: HIVE-2910 URL: https://issues.apache.org/jira/browse/HIVE-2910 Project: Hive Issue Type: Improvement Components: Web UI Reporter: Hugo Trippaers Priority: Minor Labels: newbie, patch Attachments: hive-hwi.patch I've made some improvements to the HWI interface with the Twitter bootstrap system. I'm looking for feedback on the new design. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2910) Improve the HWI interface
[ https://issues.apache.org/jira/browse/HIVE-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hugo Trippaers updated HIVE-2910: - Attachment: screenie002.PNG screenie001.PNG Sample screenshots Improve the HWI interface - Key: HIVE-2910 URL: https://issues.apache.org/jira/browse/HIVE-2910 Project: Hive Issue Type: Improvement Components: Web UI Reporter: Hugo Trippaers Priority: Minor Labels: newbie, patch Attachments: hive-hwi.patch, screenie001.PNG, screenie002.PNG I've made some improvements to the HWI interface with the Twitter bootstrap system. I'm looking for feedback on the new design. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: HWI Interface
Hey Edward, Thanks for the feedback :-) I've created the ticket (HIVE-2910), but i could not find how to add you as a watcher. Cheers, Hugo On Tue, Mar 27, 2012 at 6:46 PM, Edward Capriolo edlinuxg...@gmail.com wrote: Awesome. Read the hive wiki about adding contributing. Open a jira issue and mark me as a watcher and I will take a look at your patch. On Tue, Mar 27, 2012 at 10:43 AM, Hugo h...@trippaers.nl wrote: Heya all, Completely new to this list, but i've been working with hive for some time now. In the company i work at we use hive as part of our workflow for generating statistics, graphs etc. (Other tools we use are OpenTSDB, HBase and Tableau.). One of my data guys likes using the HWI interface for testing queries, but the interface is purely functional now. Playing around with the bootstrap toolkit from Twitter i revamped the user interface a bit, without cluttering it with javascript and other fancy stuff. Its simple, but a bit more pleasing to the eye. I'm happy to contribute this back to hive if there is any interest in it. Attached two sample screens of my current work in progress version. Cheers, Hugo -- Tsune ni ite, kyu ni awasu. Hugo h...@strocamp.net -- Tsune ni ite, kyu ni awasu. Hugo h...@strocamp.net
[jira] [Commented] (HIVE-2891) TextConverter for UDF's is inefficient if the input object is already Text or Lazy
[ https://issues.apache.org/jira/browse/HIVE-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13240300#comment-13240300 ] Hudson commented on HIVE-2891: -- Integrated in Hive-trunk-h0.21 #1336 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1336/]) HIVE-2891: TextConverter for UDF's is inefficient if the input object is already Text or Lazy (Cliff Engle via Ashutosh Chauhan) (Revision 1306096) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1306096 Files : * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java * /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java TextConverter for UDF's is inefficient if the input object is already Text or Lazy -- Key: HIVE-2891 URL: https://issues.apache.org/jira/browse/HIVE-2891 Project: Hive Issue Type: Improvement Components: Serializers/Deserializers Affects Versions: 0.7.0, 0.7.1, 0.8.1 Reporter: Cliff Engle Assignee: Cliff Engle Priority: Minor Fix For: 0.9.0 Attachments: HIVE-2891.1.patch.txt, HIVE-2891.2.patch.txt The TextConverter in PrimitiveObjectInspectorConverter.java is very inefficient if the input object is already Text or Lazy. Since it calls getPrimitiveJavaObject, each Text is decoded into a String and then re-encoded into Text. The solution is to check if preferWritable() is true, then call getPrimitiveWritable(input). To test performance, I ran the Grep query from https://issues.apache.org/jira/browse/HIVE-396 on a cluster of 3 ec2 large nodes (2 slaves 1 master) on 6GB of data. It took 21 map tasks. With the current 0.8.1 version, it took 81 seconds. After patching, it took 66 seconds. I will attach a patch and testcases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 1336 - Still Failing
Changes for Build #1332 [hashutosh] HIVE-2797: Make the IP address of a Thrift client available to HMSHandler. (Kevin Wilfong via Ashutosh Chauhan) Changes for Build #1333 Changes for Build #1334 Changes for Build #1335 Changes for Build #1336 [hashutosh] HIVE-2891: TextConverter for UDF's is inefficient if the input object is already Text or Lazy (Cliff Engle via Ashutosh Chauhan) 1 tests failed. FAILED: org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1 Error Message: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. Stack Trace: junit.framework.AssertionFailedError: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. at junit.framework.Assert.fail(Assert.java:50) at org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1(TestNegativeCliDriver.java:10340) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) The Apache Jenkins build system has built Hive-trunk-h0.21 (build #1336) Status: Still Failing Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1336/ to view the results.
[jira] [Commented] (HIVE-2890) Example UDAFs do not run
[ https://issues.apache.org/jira/browse/HIVE-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13240365#comment-13240365 ] Bharath Ganesh commented on HIVE-2890: -- I dont think this issue exists on the trunk. I verified the UDAF constructors are public and I was able to execute them. Can you confirm? Example UDAFs do not run Key: HIVE-2890 URL: https://issues.apache.org/jira/browse/HIVE-2890 Project: Hive Issue Type: Bug Components: Contrib, Documentation Affects Versions: 0.8.1 Reporter: robin morris Priority: Minor Original Estimate: 1h Remaining Estimate: 1h The example UDAFs in src/contrib/src/java/org/apache/hadoop/hive/contrib/udaf/example have their constructors declared as private. This causes the udaf to fail at run-time. Removing the declaration of the constructor fixes the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2910) Improve the HWI interface
[ https://issues.apache.org/jira/browse/HIVE-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hugo Trippaers updated HIVE-2910: - Attachment: hive-hwi-2.patch Some more updates, this replaces the first patch Improve the HWI interface - Key: HIVE-2910 URL: https://issues.apache.org/jira/browse/HIVE-2910 Project: Hive Issue Type: Improvement Components: Web UI Reporter: Hugo Trippaers Priority: Minor Labels: newbie, patch Attachments: hive-hwi-2.patch, hive-hwi.patch, screenie001.PNG, screenie002.PNG I've made some improvements to the HWI interface with the Twitter bootstrap system. I'm looking for feedback on the new design. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2910) Improve the HWI interface
[ https://issues.apache.org/jira/browse/HIVE-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan reassigned HIVE-2910: -- Assignee: Hugo Trippaers Improve the HWI interface - Key: HIVE-2910 URL: https://issues.apache.org/jira/browse/HIVE-2910 Project: Hive Issue Type: Improvement Components: Web UI Reporter: Hugo Trippaers Assignee: Hugo Trippaers Priority: Minor Labels: newbie, patch Attachments: hive-hwi-2.patch, hive-hwi.patch, screenie001.PNG, screenie002.PNG I've made some improvements to the HWI interface with the Twitter bootstrap system. I'm looking for feedback on the new design. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2898) Add nicer helper functions for adding and reading metadata from RCFiles
[ https://issues.apache.org/jira/browse/HIVE-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-2898: --- Resolution: Fixed Fix Version/s: 0.9.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Owen! Add nicer helper functions for adding and reading metadata from RCFiles --- Key: HIVE-2898 URL: https://issues.apache.org/jira/browse/HIVE-2898 Project: Hive Issue Type: New Feature Components: Serializers/Deserializers Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.9.0 Attachments: HIVE-2898.D2433.1.patch, HIVE-2898.D2433.2.patch, HIVE-2898.D2433.3.patch, HIVE-2898.D2433.4.patch, HIVE-2898.D2433.5.patch Currently, to use the metadata in RCFile, you need to manipulate it using SequenceFile.Metadata. I'd like to add two helper functions that make it more convenient. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2518) pull junit jar from maven repos via ivy
[ https://issues.apache.org/jira/browse/HIVE-2518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-2518: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. Thanks Kevin pull junit jar from maven repos via ivy --- Key: HIVE-2518 URL: https://issues.apache.org/jira/browse/HIVE-2518 Project: Hive Issue Type: Improvement Reporter: He Yongqiang Assignee: Kevin Wilfong Attachments: HIVE-2518.D2493.1.patch see https://issues.apache.org/jira/browse/HIVE-2505 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2518) pull junit jar from maven repos via ivy
[ https://issues.apache.org/jira/browse/HIVE-2518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-2518: --- Fix Version/s: 0.9.0 pull junit jar from maven repos via ivy --- Key: HIVE-2518 URL: https://issues.apache.org/jira/browse/HIVE-2518 Project: Hive Issue Type: Improvement Reporter: He Yongqiang Assignee: Kevin Wilfong Fix For: 0.9.0 Attachments: HIVE-2518.D2493.1.patch see https://issues.apache.org/jira/browse/HIVE-2505 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2910) Improve the HWI interface
[ https://issues.apache.org/jira/browse/HIVE-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13240556#comment-13240556 ] Edward Capriolo commented on HIVE-2910: --- Looks nice. I will review the code. This has needed a face lift for a long time :) Improve the HWI interface - Key: HIVE-2910 URL: https://issues.apache.org/jira/browse/HIVE-2910 Project: Hive Issue Type: Improvement Components: Web UI Reporter: Hugo Trippaers Assignee: Hugo Trippaers Priority: Minor Labels: newbie, patch Attachments: hive-hwi-2.patch, hive-hwi.patch, screenie001.PNG, screenie002.PNG I've made some improvements to the HWI interface with the Twitter bootstrap system. I'm looking for feedback on the new design. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1362) column level statistics
[ https://issues.apache.org/jira/browse/HIVE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13240644#comment-13240644 ] Ashutosh Chauhan commented on HIVE-1362: @Namit, Bharath plans to work on this and has a gsoc proposal : http://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/bharathv/18002 column level statistics --- Key: HIVE-1362 URL: https://issues.apache.org/jira/browse/HIVE-1362 Project: Hive Issue Type: Sub-task Components: Statistics Reporter: Ning Zhang Labels: gsoc, gsoc2012 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-0.8.1-SNAPSHOT-h0.21 - Build # 236 - Fixed
Changes for Build #235 Changes for Build #236 All tests passed The Apache Jenkins build system has built Hive-0.8.1-SNAPSHOT-h0.21 (build #236) Status: Fixed Check console output at https://builds.apache.org/job/Hive-0.8.1-SNAPSHOT-h0.21/236/ to view the results.
[jira] [Updated] (HIVE-2870) Throw an error when a nonexistent partition is accessed in strict mode
[ https://issues.apache.org/jira/browse/HIVE-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2870: -- Attachment: HIVE-2870.D2319.4.patch lucian updated the revision HIVE-2870 [jira] Throw an error when a nonexistent partition is accessed in strict mode. Reviewers: JIRA, njain, kevinwilfong, akramer found a suitable ConfVar REVISION DETAIL https://reviews.facebook.net/D2319 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java ql/src/test/queries/clientnegative/nopart_after_pruning.q ql/src/test/queries/clientpositive/nopart_after_pruning.q ql/src/test/results/clientnegative/nopart_after_pruning.q.out ql/src/test/results/clientpositive/nopart_after_pruning.q.out Throw an error when a nonexistent partition is accessed in strict mode -- Key: HIVE-2870 URL: https://issues.apache.org/jira/browse/HIVE-2870 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Lucian Adrian Grijincu Priority: Minor Attachments: HIVE-2870.D2319.1.patch, HIVE-2870.D2319.2.patch, HIVE-2870.D2319.3.patch, HIVE-2870.D2319.4.patch Original Estimate: 24h Remaining Estimate: 24h When a table does not exist and someone tries to read from it in a query, Hive throws an error. But if a partition is directly accessed that does not exist, an error is not thrown. This is inconsistent and also leads to a lot of confused users who get no output. This task is to cause Hive to throw an error when the partition pruner for a query eliminates ALL existing partitions for some table when running in strict mode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 1337 - Still Failing
Changes for Build #1332 [hashutosh] HIVE-2797: Make the IP address of a Thrift client available to HMSHandler. (Kevin Wilfong via Ashutosh Chauhan) Changes for Build #1333 Changes for Build #1334 Changes for Build #1335 Changes for Build #1336 [hashutosh] HIVE-2891: TextConverter for UDF's is inefficient if the input object is already Text or Lazy (Cliff Engle via Ashutosh Chauhan) Changes for Build #1337 1 tests failed. FAILED: org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1 Error Message: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. Stack Trace: junit.framework.AssertionFailedError: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. at junit.framework.Assert.fail(Assert.java:50) at org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1(TestNegativeCliDriver.java:10340) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:422) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:931) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:785) The Apache Jenkins build system has built Hive-trunk-h0.21 (build #1337) Status: Still Failing Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1337/ to view the results.
[jira] [Created] (HIVE-2911) Move global .hiverc file
Move global .hiverc file
Key: HIVE-2911
URL: https://issues.apache.org/jira/browse/HIVE-2911
Project: Hive
Issue Type: Improvement
Components: CLI
Reporter: Owen O'Malley
Assignee: Owen O'Malley
Currently, the .hiverc files are loaded from:
{code}
$HIVE_HOME/bin/.hiverc
~/.hiverc
{code}
It seems more ops-friendly to have it in the config directory.
{code}
$HIVE_HOME/bin/.hiverc - for backwards compatibility
$HIVE_CONF_DIR/.hiverc
~/.hiverc
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2903) Numeric binary type keys are not compared properly
[
https://issues.apache.org/jira/browse/HIVE-2903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HIVE-2903:
--
Attachment: HIVE-2903.D2481.2.patch
navis updated the revision HIVE-2903 [jira] Numeric binary type keys are not
compared properly.
Reviewers: JIRA
1. Separated storage option 'lbinary' for handling minus values.
2. Added COLUMNS, COLUMN_TYPES to HBase table job property
REVISION DETAIL
https://reviews.facebook.net/D2481
AFFECTED FILES
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java
hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java
hbase-handler/src/test/queries/external_table_ppd.q
hbase-handler/src/test/results/external_table_ppd.q.out
Numeric binary type keys are not compared properly
--
Key: HIVE-2903
URL: https://issues.apache.org/jira/browse/HIVE-2903
Project: Hive
Issue Type: Bug
Components: HBase Handler
Reporter: Navis
Assignee: Navis
Attachments: HIVE-2903.D2481.1.patch, HIVE-2903.D2481.2.patch
In current binary format for numbers, minus values are always greater than
plus values, for example.
{code}
System.our.println(Bytes.compareTo(Bytes.toBytes(-100), Bytes.toBytes(100)));
// 255
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2903) Numeric binary type keys are not compared properly
[
https://issues.apache.org/jira/browse/HIVE-2903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13240800#comment-13240800
]
Navis commented on HIVE-2903:
-
It would be better to be handled in hbase rather then hive, IMHO. Before that,
I've separated the code by option 'lbinary' ('signedbinary' conflicts with
'string' which starts with 's').
Numeric binary type keys are not compared properly
--
Key: HIVE-2903
URL: https://issues.apache.org/jira/browse/HIVE-2903
Project: Hive
Issue Type: Bug
Components: HBase Handler
Reporter: Navis
Assignee: Navis
Attachments: HIVE-2903.D2481.1.patch, HIVE-2903.D2481.2.patch
In current binary format for numbers, minus values are always greater than
plus values, for example.
{code}
System.our.println(Bytes.compareTo(Bytes.toBytes(-100), Bytes.toBytes(100)));
// 255
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2908) Hive: Extend ALTER TABLE DROP PARTITION syntax to use all comparators
[ https://issues.apache.org/jira/browse/HIVE-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13240823#comment-13240823 ] Phabricator commented on HIVE-2908: --- kevinwilfong has commented on the revision HIVE-2908 [jira] Hive: Extend ALTER TABLE DROP PARTITION syntax to use all comparators. Looks good, just some cosmetic things. INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionSpec.java:61 Why do you change the != to , why not use whichever the user chose to use? ql/src/test/results/clientnegative/drop_partition_filter_failure.q.out:18 This error message seems confusing, the filter was 1, so it should not be surprising that the partition '1' was not found. (I know why this exception was thrown, I'm just saying the message could be improved) ql/src/test/queries/clientpositive/drop_partitions_filter.q:18 Out of curiosity, does the behavior change if you use d=2 vs. d='2'? REVISION DETAIL https://reviews.facebook.net/D2523 Hive: Extend ALTER TABLE DROP PARTITION syntax to use all comparators - Key: HIVE-2908 URL: https://issues.apache.org/jira/browse/HIVE-2908 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Sambavi Muthukrishnan Assignee: Sambavi Muthukrishnan Priority: Minor Attachments: HIVE-2908.D2523.1.patch Original Estimate: 96h Remaining Estimate: 96h To drop a partition from a Hive table, this works: ALTER TABLE foo DROP PARTITION(ds = 'date') ...but it should also work to drop all partitions prior to date. ALTER TABLE foo DROP PARTITION(ds 'date') This task is to implement ALTER TABLE DROP PARTITION for all of the comparators, = = = != instead of just for =. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2848) Add validation to HiveConf ConfVars
[ https://issues.apache.org/jira/browse/HIVE-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2848: -- Attachment: HIVE-2848.D2313.2.patch navis updated the revision HIVE-2848 [jira] Add validation to HiveConf ConfVars. Reviewers: JIRA 1. use map for finding conf var. 2. catch Exception instead of NumberFormatException. REVISION DETAIL https://reviews.facebook.net/D2313 AFFECTED FILES cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java ql/src/test/queries/clientnegative/set_hiveconf_validation0.q ql/src/test/queries/clientnegative/set_hiveconf_validation1.q ql/src/test/queries/clientnegative/set_hiveconf_validation2.q ql/src/test/results/clientnegative/set_hiveconf_validation0.q.out ql/src/test/results/clientnegative/set_hiveconf_validation1.q.out ql/src/test/results/clientnegative/set_hiveconf_validation2.q.out Add validation to HiveConf ConfVars --- Key: HIVE-2848 URL: https://issues.apache.org/jira/browse/HIVE-2848 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Navis Attachments: HIVE-2848.D2313.1.patch, HIVE-2848.D2313.2.patch It would be good if we could add some validation to ConfVars in the HiveConf when they are set. At least if a variable is supposed to be boolean, numeric, or strict/nonstrict, it shouldn't be too difficult to check for these. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2848) Add validation to HiveConf ConfVars
[ https://issues.apache.org/jira/browse/HIVE-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-2848: Affects Version/s: 0.9.0 Status: Patch Available (was: Open) Add validation to HiveConf ConfVars --- Key: HIVE-2848 URL: https://issues.apache.org/jira/browse/HIVE-2848 Project: Hive Issue Type: Improvement Affects Versions: 0.9.0 Reporter: Kevin Wilfong Assignee: Navis Attachments: HIVE-2848.D2313.1.patch, HIVE-2848.D2313.2.patch It would be good if we could add some validation to ConfVars in the HiveConf when they are set. At least if a variable is supposed to be boolean, numeric, or strict/nonstrict, it shouldn't be too difficult to check for these. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2870) Throw an error when a nonexistent partition is accessed in strict mode
[ https://issues.apache.org/jira/browse/HIVE-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13240840#comment-13240840 ] Phabricator commented on HIVE-2870: --- kevinwilfong has commented on the revision HIVE-2870 [jira] Throw an error when a nonexistent partition is accessed in strict mode. That conf var is currently being used to throw an error if a query is written to create dynamic partitions and it doesn't produce any partitions. That is a weird dependency to introduce, I think it would be better to create a new variable. Could you print an warning or at least log one if strict mode is set to false and a partition is empty. This would be useful because users could still see that they are querying a partition unnecessarily, but without the risk of breaking existing queries. REVISION DETAIL https://reviews.facebook.net/D2319 Throw an error when a nonexistent partition is accessed in strict mode -- Key: HIVE-2870 URL: https://issues.apache.org/jira/browse/HIVE-2870 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Lucian Adrian Grijincu Priority: Minor Attachments: HIVE-2870.D2319.1.patch, HIVE-2870.D2319.2.patch, HIVE-2870.D2319.3.patch, HIVE-2870.D2319.4.patch Original Estimate: 24h Remaining Estimate: 24h When a table does not exist and someone tries to read from it in a query, Hive throws an error. But if a partition is directly accessed that does not exist, an error is not thrown. This is inconsistent and also leads to a lot of confused users who get no output. This task is to cause Hive to throw an error when the partition pruner for a query eliminates ALL existing partitions for some table when running in strict mode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table with large number of partitions
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13240846#comment-13240846 ] Phabricator commented on HIVE-2907: --- kevinwilfong has requested changes to the revision HIVE-2907 [jira] Hive error when dropping a table with large number of partitions. I'm seeing a lot of test failures in TestNegativeCliDriver. They seem to work ok when run individually, but when I run ant test or ant test -Dtestcase=TestNegativeCliDriver they fail. Could you take a look? REVISION DETAIL https://reviews.facebook.net/D2505 BRANCH svn Hive error when dropping a table with large number of partitions Key: HIVE-2907 URL: https://issues.apache.org/jira/browse/HIVE-2907 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.9.0 Environment: General. Hive Metastore bug. Reporter: Mousom Dhar Gupta Assignee: Mousom Dhar Gupta Priority: Minor Fix For: 0.9.0 Attachments: HIVE-2907.1.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch Original Estimate: 10h Remaining Estimate: 10h Running into an Out Of Memory error when trying to drop a table with 128K partitions. The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors when dropping tables with lots of partitions because they try to load the metadata for every partition into memory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2908) Hive: Extend ALTER TABLE DROP PARTITION syntax to use all comparators
[ https://issues.apache.org/jira/browse/HIVE-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13240860#comment-13240860 ] Phabricator commented on HIVE-2908: --- sambavim has commented on the revision HIVE-2908 [jira] Hive: Extend ALTER TABLE DROP PARTITION syntax to use all comparators. Will update review once I make the change to the error message. INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionSpec.java:61 I found the != produced the unexpected side effect that it actually behaved like an =. That was obviously a problem, so I decided to just convert it to a for safety. I didn't debug why I got that behavior with a !=, but I suspect it is because != is not a valid operator on a filter against the database. What was surprising is the outcome of deleting the partition that was specified in the !=. ql/src/test/queries/clientpositive/drop_partitions_filter.q:18 I'll test it and report back. ql/src/test/results/clientnegative/drop_partition_filter_failure.q.out:18 Will fix. Good catch. REVISION DETAIL https://reviews.facebook.net/D2523 Hive: Extend ALTER TABLE DROP PARTITION syntax to use all comparators - Key: HIVE-2908 URL: https://issues.apache.org/jira/browse/HIVE-2908 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Sambavi Muthukrishnan Assignee: Sambavi Muthukrishnan Priority: Minor Attachments: HIVE-2908.D2523.1.patch Original Estimate: 96h Remaining Estimate: 96h To drop a partition from a Hive table, this works: ALTER TABLE foo DROP PARTITION(ds = 'date') ...but it should also work to drop all partitions prior to date. ALTER TABLE foo DROP PARTITION(ds 'date') This task is to implement ALTER TABLE DROP PARTITION for all of the comparators, = = = != instead of just for =. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2906) Support providing some table properties by user via SQL
[
https://issues.apache.org/jira/browse/HIVE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13240863#comment-13240863
]
Phabricator commented on HIVE-2906:
---
navis has commented on the revision HIVE-2906 [jira] Support providing some
table properties by user via SQL.
It was intended for specifying 'max version' for hbase handler and it seemed
require some modifications. So I was thinking it could be done in a separated
issue after this. But if it's OK, I'll work on that in this issue, too.
REVISION DETAIL
https://reviews.facebook.net/D2499
Support providing some table properties by user via SQL
---
Key: HIVE-2906
URL: https://issues.apache.org/jira/browse/HIVE-2906
Project: Hive
Issue Type: Improvement
Components: Query Processor
Reporter: Navis
Assignee: Navis
Attachments: HIVE-2906.D2499.1.patch
Some properties are needed to be provided to StorageHandler by user in
runtime. It might be an address for remote resource or retry count for access
or maximum version count(for hbase), etc.
For example,
{code}
select emp.empno, emp.ename from hbase_emp ('max.version'='3') emp;
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2898) Add nicer helper functions for adding and reading metadata from RCFiles
[ https://issues.apache.org/jira/browse/HIVE-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13240907#comment-13240907 ] Hudson commented on HIVE-2898: -- Integrated in Hive-trunk-h0.21 #1338 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1338/]) HIVE-2898: Add nicer helper functions for adding and reading metadata from RCFiles (Owen Omalley via Ashutosh Chauhan) (Revision 1306464) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1306464 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/RCFile.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/TestRCFile.java Add nicer helper functions for adding and reading metadata from RCFiles --- Key: HIVE-2898 URL: https://issues.apache.org/jira/browse/HIVE-2898 Project: Hive Issue Type: New Feature Components: Serializers/Deserializers Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.9.0 Attachments: HIVE-2898.D2433.1.patch, HIVE-2898.D2433.2.patch, HIVE-2898.D2433.3.patch, HIVE-2898.D2433.4.patch, HIVE-2898.D2433.5.patch Currently, to use the metadata in RCFile, you need to manipulate it using SequenceFile.Metadata. I'd like to add two helper functions that make it more convenient. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 1338 - Still Failing
Changes for Build #1332 [hashutosh] HIVE-2797: Make the IP address of a Thrift client available to HMSHandler. (Kevin Wilfong via Ashutosh Chauhan) Changes for Build #1333 Changes for Build #1334 Changes for Build #1335 Changes for Build #1336 [hashutosh] HIVE-2891: TextConverter for UDF's is inefficient if the input object is already Text or Lazy (Cliff Engle via Ashutosh Chauhan) Changes for Build #1337 Changes for Build #1338 [namit] HIVE-2518 pull junit jar from maven repos via ivy (Kevin Wilfong via namit) [hashutosh] HIVE-2898: Add nicer helper functions for adding and reading metadata from RCFiles (Owen Omalley via Ashutosh Chauhan) 1 tests failed. FAILED: org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1 Error Message: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. Stack Trace: junit.framework.AssertionFailedError: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. at junit.framework.Assert.fail(Assert.java:47) at org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1(TestNegativeCliDriver.java:10340) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:154) at junit.framework.TestCase.runBare(TestCase.java:127) at junit.framework.TestResult$1.protect(TestResult.java:106) at junit.framework.TestResult.runProtected(TestResult.java:124) at junit.framework.TestResult.run(TestResult.java:109) at junit.framework.TestCase.run(TestCase.java:118) at junit.framework.TestSuite.runTest(TestSuite.java:208) at junit.framework.TestSuite.run(TestSuite.java:203) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) The Apache Jenkins build system has built Hive-trunk-h0.21 (build #1338) Status: Still Failing Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1338/ to view the results.
Make hive support various charsets
Hi all I've been working with hive for some time. In my company, we use hive for querying on large datasets and found it's very easy to use. However we also found hive is lack of various charsets support so that we have to manually transform data files to utf-8 encoding before loading them into hive. So I have made a patch to make hive supports setting charset when creating a table. And the charset property will be used by SerDe when it serialize or deserialize data. The modified hql is like: CREATE TABLE tbl1 (col1 STRING) ROW FORMAT CHARET GBK DELIMITED FIELDS TERMINATED BY '\t'; I'm very happy to contribute this to the community and looking forward to your feedbacks. Thanks, Kai Zhang
Re: Make hive support various charsets
Kai, That would be great. Please file a jura, and submit a patch. We would definitely like to get it for the whole community Thanks, -namit On 3/28/12 8:46 PM, Zhang Kai zhangk1...@gmail.com wrote: Hi all I've been working with hive for some time. In my company, we use hive for querying on large datasets and found it's very easy to use. However we also found hive is lack of various charsets support so that we have to manually transform data files to utf-8 encoding before loading them into hive. So I have made a patch to make hive supports setting charset when creating a table. And the charset property will be used by SerDe when it serialize or deserialize data. The modified hql is like: CREATE TABLE tbl1 (col1 STRING) ROW FORMAT CHARET GBK DELIMITED FIELDS TERMINATED BY '\t'; I'm very happy to contribute this to the community and looking forward to your feedbacks. Thanks, Kai Zhang
