[jira] [Updated] (HIVE-2848) Add validation to HiveConf ConfVars
[ https://issues.apache.org/jira/browse/HIVE-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-2848: - Status: Open (was: Patch Available) Comments on arc diff Add validation to HiveConf ConfVars --- Key: HIVE-2848 URL: https://issues.apache.org/jira/browse/HIVE-2848 Project: Hive Issue Type: Improvement Affects Versions: 0.9.0 Reporter: Kevin Wilfong Assignee: Navis Attachments: HIVE-2848.D2313.1.patch, HIVE-2848.D2313.2.patch It would be good if we could add some validation to ConfVars in the HiveConf when they are set. At least if a variable is supposed to be boolean, numeric, or strict/nonstrict, it shouldn't be too difficult to check for these. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2848) Add validation to HiveConf ConfVars
[ https://issues.apache.org/jira/browse/HIVE-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13243994#comment-13243994 ] Phabricator commented on HIVE-2848: --- njain has commented on the revision HIVE-2848 [jira] Add validation to HiveConf ConfVars. INLINE COMMENTS common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:600 Looks good - a minor change, can you change the type to a ENUM ? It will just make the code easier. Otherwise, the changes look good. ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java:138 Is this check valid ? I mean, aren't there hive parameters which dont start with hive. ? common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:534 What will it take to change the default to true ? It should work - right, or will anything break ? REVISION DETAIL https://reviews.facebook.net/D2313 Add validation to HiveConf ConfVars --- Key: HIVE-2848 URL: https://issues.apache.org/jira/browse/HIVE-2848 Project: Hive Issue Type: Improvement Affects Versions: 0.9.0 Reporter: Kevin Wilfong Assignee: Navis Attachments: HIVE-2848.D2313.1.patch, HIVE-2848.D2313.2.patch It would be good if we could add some validation to ConfVars in the HiveConf when they are set. At least if a variable is supposed to be boolean, numeric, or strict/nonstrict, it shouldn't be too difficult to check for these. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2858) Cache remote map reduce job stack traces for additional logging
[ https://issues.apache.org/jira/browse/HIVE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-2858: - Status: Open (was: Patch Available) Cache remote map reduce job stack traces for additional logging --- Key: HIVE-2858 URL: https://issues.apache.org/jira/browse/HIVE-2858 Project: Hive Issue Type: Improvement Components: Logging Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2858.D2223.1.patch Currently we are parsing the task logs for failed jobs for information to display to the user in the CLI. In addition, we could parse those logs for stack traces and store e them in the SessionState. This way, when we log failed queries, these will give us a decent idea of why those queries failed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2858) Cache remote map reduce job stack traces for additional logging
[ https://issues.apache.org/jira/browse/HIVE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13243998#comment-13243998 ] Phabricator commented on HIVE-2858: --- njain has commented on the revision HIVE-2858 [jira] Cache remote map reduce job stack traces for additional logging. can you refresh and re-apply the patch ? It is not applying cleanly for me. REVISION DETAIL https://reviews.facebook.net/D2223 Cache remote map reduce job stack traces for additional logging --- Key: HIVE-2858 URL: https://issues.apache.org/jira/browse/HIVE-2858 Project: Hive Issue Type: Improvement Components: Logging Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2858.D2223.1.patch Currently we are parsing the task logs for failed jobs for information to display to the user in the CLI. In addition, we could parse those logs for stack traces and store e them in the SessionState. This way, when we log failed queries, these will give us a decent idea of why those queries failed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2860) TestNegativeCliDriver autolocal1.q fails on 0.23
[ https://issues.apache.org/jira/browse/HIVE-2860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2860: -- Attachment: HIVE-2860.D2565.1.patch cwsteinbach requested code review of HIVE-2860 [jira] TestNegativeCliDriver autolocal1.q fails on 0.23. Reviewers: JIRA HIVE-2860. TestNegativeCliDriver autolocal1.q fails on 0.23 TEST PLAN NONE REVISION DETAIL https://reviews.facebook.net/D2565 AFFECTED FILES ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/5823/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. TestNegativeCliDriver autolocal1.q fails on 0.23 Key: HIVE-2860 URL: https://issues.apache.org/jira/browse/HIVE-2860 Project: Hive Issue Type: Bug Components: Testing Infrastructure Affects Versions: 0.9.0 Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.9.0 Attachments: HIVE-2860.D2253.1.patch, HIVE-2860.D2253.1.patch, HIVE-2860.D2565.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2848) Add validation to HiveConf ConfVars
[ https://issues.apache.org/jira/browse/HIVE-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244001#comment-13244001 ] Phabricator commented on HIVE-2848: --- navis has commented on the revision HIVE-2848 [jira] Add validation to HiveConf ConfVars. INLINE COMMENTS common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:534 I'll try to test with this option set 'true'. common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:600 Ok, I'll try. ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java:138 I'm always misspelling 'hive.~' configuration, so it was added for me. Agree on this is not a common case and will be removed. REVISION DETAIL https://reviews.facebook.net/D2313 Add validation to HiveConf ConfVars --- Key: HIVE-2848 URL: https://issues.apache.org/jira/browse/HIVE-2848 Project: Hive Issue Type: Improvement Affects Versions: 0.9.0 Reporter: Kevin Wilfong Assignee: Navis Attachments: HIVE-2848.D2313.1.patch, HIVE-2848.D2313.2.patch It would be good if we could add some validation to ConfVars in the HiveConf when they are set. At least if a variable is supposed to be boolean, numeric, or strict/nonstrict, it shouldn't be too difficult to check for these. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
How to check for year condition in hive
Hello all, I am trying to check age in hive. select * from tbl_name where datediff(date,dob)/365 = 18; Is it right to check date condition in hive? Or do I need to do something else. Pls suggest me as soon as possible -- Thanks and Regards, Bhavesh Shah
[jira] [Updated] (HIVE-2913) BlockMergeTask Doesn't Honor Job Configuration Properties when used directly
[ https://issues.apache.org/jira/browse/HIVE-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HIVE-2913: --- Status: Patch Available (was: Open) BlockMergeTask Doesn't Honor Job Configuration Properties when used directly Key: HIVE-2913 URL: https://issues.apache.org/jira/browse/HIVE-2913 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.8.1 Reporter: Thiruvel Thirumoolan Assignee: Thiruvel Thirumoolan Priority: Minor Fix For: 0.9.0 Attachments: HIVE-2913.D2547.1.patch, HIVE-2913_1.patch, blockmerge.sh BlockMergeTask has a main() and when used directly (instead of say partition concatenate feature), the -jobconf arguments are not honored. This is not something most people directly use. Usage: BlockMergeTask -input colon seperated input paths -outputDir outputDir [-jobconf k1=v1 [-jobconf k2=v2] ...] To reproduce: Run BlockMergeTask with say -jobconf mapred.job.name=test and launched job will have a different name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2918) Hive Dynamic Partition Insert - move task not considering 'hive.exec.max.dynamic.partitions' from CLI
Hive Dynamic Partition Insert - move task not considering 'hive.exec.max.dynamic.partitions' from CLI - Key: HIVE-2918 URL: https://issues.apache.org/jira/browse/HIVE-2918 Project: Hive Issue Type: Bug Affects Versions: 0.7.1 Environment: Cent OS 64 bit Reporter: Bejoy KS Dynamic Partition insert showing an error with the number of partitions created even after the default value of 'hive.exec.max.dynamic.partitions' is bumped high to 2000. Error Message: Failed with exception Number of dynamic partitions created is 1413, which is more than 1000. To solve this try to set hive.exec.max.dynamic.partitions to at least 1413. These are the following properties set on hive CLI hive set hive.exec.dynamic.partition=true; hive set hive.exec.dynamic.partition.mode=nonstrict; hive set hive.exec.max.dynamic.partitions=2000; hive set hive.exec.max.dynamic.partitions.pernode=2000; This is the query with console error log hive INSERT OVERWRITE TABLE partn_dyn Partition (pobox) SELECT country,state,pobox FROM non_partn_dyn; Total MapReduce jobs = 2 Launching Job 1 out of 2 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201204021529_0002, Tracking URL = http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201204021529_0002 Kill Command = /usr/lib/hadoop/bin/hadoop job -Dmapred.job.tracker=0.0.0.0:8021 -kill job_201204021529_0002 2012-04-02 16:05:28,619 Stage-1 map = 0%, reduce = 0% 2012-04-02 16:05:39,701 Stage-1 map = 100%, reduce = 0% 2012-04-02 16:05:50,800 Stage-1 map = 100%, reduce = 100% Ended Job = job_201204021529_0002 Ended Job = 248865587, job is filtered out (removed at runtime). Moving data to: hdfs://0.0.0.0/tmp/hive-cloudera/hive_2012-04-02_16-05-24_919_5976014408587784412/-ext-1 Loading data to table default.partn_dyn partition (pobox=null) Failed with exception Number of dynamic partitions created is 1413, which is more than 1000. To solve this try to set hive.exec.max.dynamic.partitions to at least 1413. FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask I checked the job.xml of the first map only job, there the value hive.exec.max.dynamic.partitions=2000 is reflected but the move task is taking the default value from hive-site.xml . If I change the value in hive-site.xml then the job completes successfully. Bottom line,the property 'hive.exec.max.dynamic.partitions'set on CLI is not being considered by move task -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2881) Remove redundant key comparing in SMBMapJoinOperator
[ https://issues.apache.org/jira/browse/HIVE-2881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244141#comment-13244141 ] Hudson commented on HIVE-2881: -- Integrated in Hive-trunk-h0.21 #1346 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1346/]) HIVE-2881 Remove redundant key comparing in SMBMapJoinOperator (Navis via namit) (Revision 1308241) Result = ABORTED namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1308241 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java Remove redundant key comparing in SMBMapJoinOperator Key: HIVE-2881 URL: https://issues.apache.org/jira/browse/HIVE-2881 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-2881.D2379.1.patch Currently, SMBJoin compares keys twice in #findSmallestKey and #joinObject. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2805) Move metastore upgrade scripts labeled 0.10.0 into scripts labeled 0.9.0
[ https://issues.apache.org/jira/browse/HIVE-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-2805: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. Thanks Kevin Move metastore upgrade scripts labeled 0.10.0 into scripts labeled 0.9.0 Key: HIVE-2805 URL: https://issues.apache.org/jira/browse/HIVE-2805 Project: Hive Issue Type: Task Components: Metastore Affects Versions: 0.9.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Fix For: 0.9.0 Attachments: HIVE-2805.D1743.1.patch Move contents of upgrade-0.9.0-to-0.10.0.mysql.sql, upgrade-0.9.0-to-0.10.0.derby.sql into upgrade-0.8.0-to-0.9.0.mysql.sql, upgrade-0.8.0-to-0.9.0.derby.sql Rename hive-schema-0.10.0.derby.sql, hive-schema-0.10.0.mysql.sql to hive-schema-0.9.0.derby.sql, hive-schema-0.9.0.mysql.sql -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2853) Add pre event listeners to metastore
[ https://issues.apache.org/jira/browse/HIVE-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-2853: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. Thanks Kevin Add pre event listeners to metastore Key: HIVE-2853 URL: https://issues.apache.org/jira/browse/HIVE-2853 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2853.D2175.1.patch, HIVE-2853.D2175.2.patch Currently there are event listeners in the metastore which run after the completion of a method. It would be useful to have similar hooks which run before the metastore method is executed. These can be used to make validating names, locations, etc. customizable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2801) When join key is null, random distribute this tuple
[ https://issues.apache.org/jira/browse/HIVE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-2801: - Assignee: binlijin Status: Open (was: Patch Available) Please make it 'Patch Available' after addresssing Ashutosh's concerns When join key is null, random distribute this tuple --- Key: HIVE-2801 URL: https://issues.apache.org/jira/browse/HIVE-2801 Project: Hive Issue Type: Improvement Reporter: binlijin Assignee: binlijin Attachments: HIVE-2801.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2863) Ambiguous table name or column reference message displays when table and column names are the same
[ https://issues.apache.org/jira/browse/HIVE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244284#comment-13244284 ] Phabricator commented on HIVE-2863: --- njain has accepted the revision HIVE-2863 [jira] Ambiguous table name or column reference message displays when table and column names are the same. REVISION DETAIL https://reviews.facebook.net/D2361 BRANCH DPAL-975 Ambiguous table name or column reference message displays when table and column names are the same -- Key: HIVE-2863 URL: https://issues.apache.org/jira/browse/HIVE-2863 Project: Hive Issue Type: Bug Reporter: Mauro Cazzari Assignee: Navis Attachments: HIVE-2863.D2361.1.patch Given the following table: CREATE TABLE `Y` (`y` DOUBLE) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' STORED AS TEXTFILE; The following query fails: SELECT `Y`.`y` FROM `Y` WHERE ( `y` = 1 ) ERROR: java.sql.SQLException: Query returned non-zero code: 10, cause: FAILED: Error in semantic analysis: Line 1:36 Ambiguous table alias or column reference '`y`' ERROR: Unable to execute Hadoop query. ERROR: Prepare error. SQL statement: SELECT `Y`.`y` FROM `Y` WHERE ( `y` = 1 ). The problem goes away if the table and column names do not match. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2919) Duplicated Hadoop Package Structure in Hive exec
Duplicated Hadoop Package Structure in Hive exec Key: HIVE-2919 URL: https://issues.apache.org/jira/browse/HIVE-2919 Project: Hive Issue Type: Bug Components: Shims Affects Versions: 0.8.1 Environment: Ubuntu 10.04 Reporter: Buddhika Chamith De Alwis There are some Hadoop packages present inside hive-exec jar. (for example org.apache.hadoop.fs etc.). This causes issues when trying to run Hive under an OSGi environment since duplicate packages coming from Hadoop and Hive are causing conflicts. Moving these to a non Hadoop package structure a possible solution? Related discussion in hive-user [1]. [1] http://mail-archives.apache.org/mod_mbox/hive-user/201204.mbox/%3ccaenxbwwtz+lqnr0qht+ykgdgszq19ajrchw5oyghto+t_pn...@mail.gmail.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2919) Duplicated Hadoop Package Structure in Hive exec
[ https://issues.apache.org/jira/browse/HIVE-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Buddhika Chamith De Alwis updated HIVE-2919: Issue Type: Improvement (was: Bug) Duplicated Hadoop Package Structure in Hive exec Key: HIVE-2919 URL: https://issues.apache.org/jira/browse/HIVE-2919 Project: Hive Issue Type: Improvement Components: Shims Affects Versions: 0.8.1 Environment: Ubuntu 10.04 Reporter: Buddhika Chamith De Alwis There are some Hadoop packages present inside hive-exec jar. (for example org.apache.hadoop.fs etc.). This causes issues when trying to run Hive under an OSGi environment since duplicate packages coming from Hadoop and Hive are causing conflicts. Moving these to a non Hadoop package structure a possible solution? Related discussion in hive-user [1]. [1] http://mail-archives.apache.org/mod_mbox/hive-user/201204.mbox/%3ccaenxbwwtz+lqnr0qht+ykgdgszq19ajrchw5oyghto+t_pn...@mail.gmail.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-2908) Hive: Extend ALTER TABLE DROP PARTITION syntax to use all comparators
[ https://issues.apache.org/jira/browse/HIVE-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong resolved HIVE-2908. - Resolution: Fixed Committed. Thanks, Sambavi. Hive: Extend ALTER TABLE DROP PARTITION syntax to use all comparators - Key: HIVE-2908 URL: https://issues.apache.org/jira/browse/HIVE-2908 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.9.0 Reporter: Sambavi Muthukrishnan Assignee: Sambavi Muthukrishnan Priority: Minor Attachments: HIVE-2908.D2523.1.patch, HIVE-2908.D2523.2.patch, HIVE-2908.final.patch.txt Original Estimate: 96h Remaining Estimate: 96h To drop a partition from a Hive table, this works: ALTER TABLE foo DROP PARTITION(ds = 'date') ...but it should also work to drop all partitions prior to date. ALTER TABLE foo DROP PARTITION(ds 'date') This task is to implement ALTER TABLE DROP PARTITION for all of the comparators, = = = != instead of just for =. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2908) Hive: Extend ALTER TABLE DROP PARTITION syntax to use all comparators
[ https://issues.apache.org/jira/browse/HIVE-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-2908: Affects Version/s: 0.9.0 Hive: Extend ALTER TABLE DROP PARTITION syntax to use all comparators - Key: HIVE-2908 URL: https://issues.apache.org/jira/browse/HIVE-2908 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.9.0 Reporter: Sambavi Muthukrishnan Assignee: Sambavi Muthukrishnan Priority: Minor Attachments: HIVE-2908.D2523.1.patch, HIVE-2908.D2523.2.patch, HIVE-2908.final.patch.txt Original Estimate: 96h Remaining Estimate: 96h To drop a partition from a Hive table, this works: ALTER TABLE foo DROP PARTITION(ds = 'date') ...but it should also work to drop all partitions prior to date. ALTER TABLE foo DROP PARTITION(ds 'date') This task is to implement ALTER TABLE DROP PARTITION for all of the comparators, = = = != instead of just for =. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Hive / Hadoop Log Retrieval Problem
I was just trying to build a patch for this, to submit against https://issues.apache.org/jira/browse/HIVE-1579 and it seems that the default hadoop build is 0.20.1, which has a different set of parameters required in the Hadoop TaskLogServlet to a more recent version, like 0.20.205. Are there any plans to update the default hadoop build? Cheers, Phil. On 6 March 2012 14:43, Philip Tromans philip.j.trom...@gmail.com wrote: Hi, It appears that no recent version of hadoop supports this - the hadoop code appears to have changed in 2010: http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/TaskLogServlet.java?r1=918036r2=918037diff_format=f#l209 If someone could confirm that I'm working on the right lines then I'll open up a JIRA (or add to HIVE-1579) and submit a patch. Cheers, Phil. On 6 March 2012 14:13, Philip Tromans philip.j.trom...@gmail.com wrote: Hi all, I'm running into a problem - I'm using Hive trunk (pretty recent, but I see the bug's in trunk at time of writing as well), with Hadoop 0.20.205.0. I have a job which fails (for a reason which is entirely my own fault), and when it does fail Hive dies with the following exception: Ended Job = job_201202291327_1399 with errors Error during job, obtaining debugging information... Examining task ID: task_201202291327_1399_m_03 (and more) from job job_201202291327_1399 Examining task ID: task_201202291327_1399_r_02 (and more) from job job_201202291327_1399 Exception in thread Thread-342 java.lang.RuntimeException: Error while reading from task log url at org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:130) at org.apache.hadoop.hive.ql.exec.JobDebugger.showJobFailDebugInfo(JobDebugger.java:211) at org.apache.hadoop.hive.ql.exec.JobDebugger.run(JobDebugger.java:81) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Server returned HTTP response code: 400 for URL: http://...:50060/tasklog?taskid=attempt_201202291327_1399_r_02_2start=-8193 at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436) at java.net.URL.openStream(URL.java:1010) at org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:120) ... 3 more When I point a web browser at the given URL, I get the following message: HTTP ERROR 400 Problem accessing /tasklog. Reason: Argument attemptid is required If I change taskid to attemptid, everything works perfectly. The code which generates this URL appears to be in org.apache.hadoop.hive.ql.exec.JobDebugger.java. I presume that this code is correct for a given version of Hadoop. Which version is currently in use in the Jenkins/Hudson build environment? I'd be happy to change it and submit a patch to JIRA, but I guess that'd probably break the other version of Hadoop, so perhaps some more profound versioning type thing might be needed. This is the issue that HIVE-1579 is referring to. Cheers, Phil.
[jira] [Updated] (HIVE-2822) Add JSON output to the hive ddl commands
[
https://issues.apache.org/jira/browse/HIVE-2822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HIVE-2822:
--
Attachment: HIVE-2822.D2475.2.patch
ctdean updated the revision HIVE-2822 [jira] Add JSON output to the hive ddl
commands.
Reviewers: JIRA, ashutoshc
- Change control variable to hive.ddl.output.format
- Fixed negative tests that were failing because of changed error
messages
- Create org.apache.hadoop.hive.ql.metadata.formatting
- Remove unused import in MetaDataFormatter
- Fix mulitple stack trace generation in DDLTask.java
- Be more consistent about errors going through the formatter in
DDLTask
- Added template types to JsonMetaDataFormatter and MapBuilder.
- Merged a few methods in the formatters. Don't actually agree with
this as it makes long methods longer, but got carried awaya and did
it anyway.
Unchanged:
- MetaDataFormatter error codes - I think ints are clearer here, since
the names are just human readable cues for the actual int error
messages.
REVISION DETAIL
https://reviews.facebook.net/D2475
AFFECTED FILES
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MapBuilder.java
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
ql/src/java/org/apache/hadoop/hive/ql/metadata/MetaDataFormatUtils.java
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatter.java
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/TextMetaDataFormatter.java
ql/src/test/queries/clientpositive/describe_database_json.q
ql/src/test/queries/clientpositive/describe_table_json.q
ql/src/test/queries/clientpositive/misc_json.q
ql/src/test/queries/clientpositive/partitions_json.q
ql/src/test/results/clientnegative/column_rename1.q.out
ql/src/test/results/clientnegative/column_rename4.q.out
ql/src/test/results/clientnegative/database_create_already_exists.q.out
ql/src/test/results/clientnegative/database_drop_does_not_exist.q.out
ql/src/test/results/clientpositive/describe_database_json.q.out
ql/src/test/results/clientpositive/describe_table_json.q.out
ql/src/test/results/clientpositive/misc_json.q.out
ql/src/test/results/clientpositive/partitions_json.q.out
Add JSON output to the hive ddl commands
Key: HIVE-2822
URL: https://issues.apache.org/jira/browse/HIVE-2822
Project: Hive
Issue Type: Improvement
Reporter: Chris Dean
Assignee: Chris Dean
Attachments: HIVE-2822.03-branch0-8.patch, HIVE-2822.03.patch,
HIVE-2822.03b.patch, HIVE-2822.04-branch-08.patch,
HIVE-2822.05-branch0-8-1.patch, HIVE-2822.05-branch0-8.patch,
HIVE-2822.05.patch, HIVE-2822.D2475.1.patch, HIVE-2822.D2475.2.patch,
hive-json-01-branch0-8.patch, hive-json-01.patch,
hive-json-02-branch0-8.patch, hive-json-02.patch
The goal is to have an option to produce JSON output of the DDL commands that
is easily machine parseable.
For example, desc my_table currently gives
{noformat}
idbigint
user string
{noformat}
and we want to allow a json output:
{noformat}
{
columns: [
{name: id, type: bigint},
{name: user, type: string}
]
}
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2711) Make the header of RCFile unique
[ https://issues.apache.org/jira/browse/HIVE-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2711: -- Attachment: HIVE-2711.D2571.1.patch omalley requested code review of HIVE-2711 [jira] Make the header of RCFile unique. Reviewers: JIRA HIVE-2711 Make the header of RCFile unique wrt SequenceFile The RCFile implementation was copied from Hadoop's SequenceFile and copied the 'magic' string in the header. This means that you can't use the header to distinguish between RCFiles and SequenceFiles. I'd propose that we create a new header for RCFiles (RCF?) to replace the current SEQ. To maintain compatibility, we'll need to continue to accept the current 'SEQ\06' and just make new files contain the new header. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2571 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/io/RCFile.java ql/src/test/data/rc-file-v0.rc ql/src/test/org/apache/hadoop/hive/ql/io/TestRCFile.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/5835/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. Make the header of RCFile unique Key: HIVE-2711 URL: https://issues.apache.org/jira/browse/HIVE-2711 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-2711.D2115.1.patch, HIVE-2711.D2571.1.patch The RCFile implementation was copied from Hadoop's SequenceFile and copied the 'magic' string in the header. This means that you can't use the header to distinguish between RCFiles and SequenceFiles. I'd propose that we create a new header for RCFiles (RCF?) to replace the current SEQ. To maintain compatibility, we'll need to continue to accept the current 'SEQ\06' and just make new files contain the new header. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2711) Make the header of RCFile unique
[ https://issues.apache.org/jira/browse/HIVE-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244404#comment-13244404 ] Phabricator commented on HIVE-2711: --- omalley has abandoned the revision HIVE-2711 [jira] Make the header of RCFile unique. I accidentally got a new revision. This is the same as 2511. REVISION DETAIL https://reviews.facebook.net/D2571 Make the header of RCFile unique Key: HIVE-2711 URL: https://issues.apache.org/jira/browse/HIVE-2711 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-2711.D2115.1.patch, HIVE-2711.D2571.1.patch The RCFile implementation was copied from Hadoop's SequenceFile and copied the 'magic' string in the header. This means that you can't use the header to distinguish between RCFiles and SequenceFiles. I'd propose that we create a new header for RCFiles (RCF?) to replace the current SEQ. To maintain compatibility, we'll need to continue to accept the current 'SEQ\06' and just make new files contain the new header. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2711) Make the header of RCFile unique
[ https://issues.apache.org/jira/browse/HIVE-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2711: -- Attachment: HIVE-2711.D2115.2.patch omalley updated the revision HIVE-2711 [jira] Make the header of RCFile unique. Reviewers: JIRA updated to current trunk REVISION DETAIL https://reviews.facebook.net/D2115 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/io/RCFile.java ql/src/test/data/rc-file-v0.rc ql/src/test/org/apache/hadoop/hive/ql/io/TestRCFile.java Make the header of RCFile unique Key: HIVE-2711 URL: https://issues.apache.org/jira/browse/HIVE-2711 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-2711.D2115.1.patch, HIVE-2711.D2115.2.patch, HIVE-2711.D2571.1.patch The RCFile implementation was copied from Hadoop's SequenceFile and copied the 'magic' string in the header. This means that you can't use the header to distinguish between RCFiles and SequenceFiles. I'd propose that we create a new header for RCFiles (RCF?) to replace the current SEQ. To maintain compatibility, we'll need to continue to accept the current 'SEQ\06' and just make new files contain the new header. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2920) TestStatsPublisherEnhanced throws NPE on JDBC connection failure
TestStatsPublisherEnhanced throws NPE on JDBC connection failure Key: HIVE-2920 URL: https://issues.apache.org/jira/browse/HIVE-2920 Project: Hive Issue Type: Bug Components: Statistics, Testing Infrastructure Reporter: Carl Steinbach Assignee: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2920) TestStatsPublisherEnhanced throws NPE on JDBC connection failure
[
https://issues.apache.org/jira/browse/HIVE-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244501#comment-13244501
]
Carl Steinbach commented on HIVE-2920:
--
I have seen intermittent failures in TestStatsPublisherEnhanced with the
following stack:
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.Utilities.prepareWithRetry(Utilities.java:2176)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsAggregator.cleanUp(JDBCStatsAggregator.java:215)
at
org.apache.hadoop.hive.ql.exec.TestStatsPublisherEnhanced.setUp(TestStatsPublisherEnhanced.java:60)
at junit.framework.TestCase.runBare(TestCase.java:132)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
{noformat}
TestStatsPublisherEnhanced throws NPE on JDBC connection failure
Key: HIVE-2920
URL: https://issues.apache.org/jira/browse/HIVE-2920
Project: Hive
Issue Type: Bug
Components: Statistics, Testing Infrastructure
Reporter: Carl Steinbach
Assignee: Carl Steinbach
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2822) Add JSON output to the hive ddl commands
[
https://issues.apache.org/jira/browse/HIVE-2822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244556#comment-13244556
]
Phabricator commented on HIVE-2822:
---
ashutoshc has requested changes to the revision HIVE-2822 [jira] Add JSON
output to the hive ddl commands.
Seems like output in *.q.out has changed in unintended ways.
INLINE COMMENTS
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:560 This also needs
to be added in file conf/hive-default.xml.template This template file is not
used anywhere but exists for documentation purposes.
ql/src/test/results/clientnegative/column_rename1.q.out:26 This line is
certainly useful and should appear in the output. After the change it seems it
will get lost.
ql/src/test/results/clientnegative/column_rename4.q.out:26 Here also, column
does not exist is useful bit of error message, which is not there anymore in
patched version.
ql/src/test/results/clientnegative/database_create_already_exists.q.out:14
Here, it seems wrong error message got generated. This will confuse users.
ql/src/test/results/clientnegative/database_drop_does_not_exist.q.out:9 Same
as above.
REVISION DETAIL
https://reviews.facebook.net/D2475
BRANCH
HIVE-2822-dev-branch
Add JSON output to the hive ddl commands
Key: HIVE-2822
URL: https://issues.apache.org/jira/browse/HIVE-2822
Project: Hive
Issue Type: Improvement
Reporter: Chris Dean
Assignee: Chris Dean
Attachments: HIVE-2822.03-branch0-8.patch, HIVE-2822.03.patch,
HIVE-2822.03b.patch, HIVE-2822.04-branch-08.patch,
HIVE-2822.05-branch0-8-1.patch, HIVE-2822.05-branch0-8.patch,
HIVE-2822.05.patch, HIVE-2822.D2475.1.patch, HIVE-2822.D2475.2.patch,
hive-json-01-branch0-8.patch, hive-json-01.patch,
hive-json-02-branch0-8.patch, hive-json-02.patch
The goal is to have an option to produce JSON output of the DDL commands that
is easily machine parseable.
For example, desc my_table currently gives
{noformat}
idbigint
user string
{noformat}
and we want to allow a json output:
{noformat}
{
columns: [
{name: id, type: bigint},
{name: user, type: string}
]
}
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2805) Move metastore upgrade scripts labeled 0.10.0 into scripts labeled 0.9.0
[ https://issues.apache.org/jira/browse/HIVE-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244633#comment-13244633 ] Hudson commented on HIVE-2805: -- Integrated in Hive-trunk-h0.21 #1347 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1347/]) HIVE-2805 Move metastore upgrade scripts labeled 0.10.0 into scripts labeled 0.9.0 (Kevin Wilfong via namit) (Revision 1308392) Result = SUCCESS namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1308392 Files : * /hive/trunk/metastore/scripts/upgrade/derby/009-HIVE-2612.derby.sql * /hive/trunk/metastore/scripts/upgrade/derby/010-HIVE-2612.derby.sql * /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.10.0.derby.sql * /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.9.0.derby.sql * /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.8.0-to-0.9.0.derby.sql * /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.9.0-to-0.10.0.derby.sql * /hive/trunk/metastore/scripts/upgrade/mysql/009-HIVE-2612.mysql.sql * /hive/trunk/metastore/scripts/upgrade/mysql/010-HIVE-2612.mysql.sql * /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.10.0.mysql.sql * /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.9.0.mysql.sql * /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.8.0-to-0.9.0.mysql.sql * /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.9.0-to-0.10.0.mysql.sql * /hive/trunk/metastore/scripts/upgrade/postgres/009-HIVE-2612.postgres.sql * /hive/trunk/metastore/scripts/upgrade/postgres/010-HIVE-2612.postgres.sql Move metastore upgrade scripts labeled 0.10.0 into scripts labeled 0.9.0 Key: HIVE-2805 URL: https://issues.apache.org/jira/browse/HIVE-2805 Project: Hive Issue Type: Task Components: Metastore Affects Versions: 0.9.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Fix For: 0.9.0 Attachments: HIVE-2805.D1743.1.patch Move contents of upgrade-0.9.0-to-0.10.0.mysql.sql, upgrade-0.9.0-to-0.10.0.derby.sql into upgrade-0.8.0-to-0.9.0.mysql.sql, upgrade-0.8.0-to-0.9.0.derby.sql Rename hive-schema-0.10.0.derby.sql, hive-schema-0.10.0.mysql.sql to hive-schema-0.9.0.derby.sql, hive-schema-0.9.0.mysql.sql -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2853) Add pre event listeners to metastore
[ https://issues.apache.org/jira/browse/HIVE-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244632#comment-13244632 ] Hudson commented on HIVE-2853: -- Integrated in Hive-trunk-h0.21 #1347 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1347/]) HIVE-2853 Add pre event listeners to metastore (Kevin Wilfong via namit) (Revision 1308399) Result = SUCCESS namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1308399 Files : * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStorePreEventListener.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/PreAddPartitionEvent.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/PreAlterPartitionEvent.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/PreAlterTableEvent.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/PreCreateDatabaseEvent.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/PreCreateTableEvent.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/PreDropDatabaseEvent.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/PreDropPartitionEvent.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/PreDropTableEvent.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/PreEventContext.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/PreLoadPartitionDoneEvent.java * /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyPreListener.java * /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java Add pre event listeners to metastore Key: HIVE-2853 URL: https://issues.apache.org/jira/browse/HIVE-2853 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2853.D2175.1.patch, HIVE-2853.D2175.2.patch Currently there are event listeners in the metastore which run after the completion of a method. It would be useful to have similar hooks which run before the metastore method is executed. These can be used to make validating names, locations, etc. customizable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2920) TestStatsPublisherEnhanced throws NPE on JDBC connection failure
[ https://issues.apache.org/jira/browse/HIVE-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2920: -- Attachment: HIVE-2920.D2577.1.patch cwsteinbach requested code review of HIVE-2920 [jira] TestStatsPublisherEnhanced throws NPE on JDBC connection failure. Reviewers: JIRA HIVE-2920. TestStatsPublisherEnhanced throws NPE on JDBC connection failure TEST PLAN NONE REVISION DETAIL https://reviews.facebook.net/D2577 AFFECTED FILES ql/src/test/org/apache/hadoop/hive/ql/exec/TestStatsPublisherEnhanced.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/5847/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. TestStatsPublisherEnhanced throws NPE on JDBC connection failure Key: HIVE-2920 URL: https://issues.apache.org/jira/browse/HIVE-2920 Project: Hive Issue Type: Bug Components: Statistics, Testing Infrastructure Reporter: Carl Steinbach Assignee: Carl Steinbach Attachments: HIVE-2920.D2577.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2921) StatsPublisher and StatsAggregator interfaces should throw exceptions
StatsPublisher and StatsAggregator interfaces should throw exceptions - Key: HIVE-2921 URL: https://issues.apache.org/jira/browse/HIVE-2921 Project: Hive Issue Type: Bug Components: Statistics Reporter: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2921) StatsPublisher and StatsAggregator interfaces should throw exceptions
[ https://issues.apache.org/jira/browse/HIVE-2921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244650#comment-13244650 ] Carl Steinbach commented on HIVE-2921: -- Currently all of the methods in the StatsAggregator and StatsPublisher interfaces use a boolean return value to indicate success or failure, which ends up hiding a lot of information when things go wrong since underlying methods are required to squelch exceptions. These methods should instead wrap and then re-throw the underlying exception. StatsPublisher and StatsAggregator interfaces should throw exceptions - Key: HIVE-2921 URL: https://issues.apache.org/jira/browse/HIVE-2921 Project: Hive Issue Type: Bug Components: Statistics Reporter: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2920) TestStatsPublisherEnhanced throws NPE on JDBC connection failure
[ https://issues.apache.org/jira/browse/HIVE-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-2920: - Attachment: HIVE-2920.D2577.1.patch TestStatsPublisherEnhanced throws NPE on JDBC connection failure Key: HIVE-2920 URL: https://issues.apache.org/jira/browse/HIVE-2920 Project: Hive Issue Type: Bug Components: Statistics, Testing Infrastructure Reporter: Carl Steinbach Assignee: Carl Steinbach Attachments: HIVE-2920.D2577.1.patch, HIVE-2920.D2577.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2920) TestStatsPublisherEnhanced throws NPE on JDBC connection failure
[ https://issues.apache.org/jira/browse/HIVE-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-2920: - Release Note: Status: Patch Available (was: Open) This patch moves the StatsAggregator cleanup code from the setUp() method to the tearDown() method in order to insure that it gets called *after* StatsPublisher.init() (which creates some tables that StatsAggregator.connect() depends on). I also added some assert statements to the tearDown() code. TestStatsPublisherEnhanced throws NPE on JDBC connection failure Key: HIVE-2920 URL: https://issues.apache.org/jira/browse/HIVE-2920 Project: Hive Issue Type: Bug Components: Statistics, Testing Infrastructure Reporter: Carl Steinbach Assignee: Carl Steinbach Attachments: HIVE-2920.D2577.1.patch, HIVE-2920.D2577.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-0.8.1-SNAPSHOT-h0.21 - Build # 241 - Failure
Changes for Build #241 1 tests failed. REGRESSION: org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1 Error Message: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. Stack Trace: junit.framework.AssertionFailedError: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. at junit.framework.Assert.fail(Assert.java:50) at org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1(TestNegativeCliDriver.java:9440) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) The Apache Jenkins build system has built Hive-0.8.1-SNAPSHOT-h0.21 (build #241) Status: Failure Check console output at https://builds.apache.org/job/Hive-0.8.1-SNAPSHOT-h0.21/241/ to view the results.
[jira] [Updated] (HIVE-2185) extend table statistics to store the size of uncompressed data (+extend interfaces for collecting other types of statistics)
[ https://issues.apache.org/jira/browse/HIVE-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-2185: - Release Note: This patch added getSerDeStats() methods to the Serializer and Deserializer interfaces. Consequently, any SerDes which were compiled against the old interfaces will need to be recompiled against the new interfaces in order to work against Hive 0.8.0. extend table statistics to store the size of uncompressed data (+extend interfaces for collecting other types of statistics) Key: HIVE-2185 URL: https://issues.apache.org/jira/browse/HIVE-2185 Project: Hive Issue Type: New Feature Components: Serializers/Deserializers, Statistics Reporter: Tomasz Nykiel Assignee: Tomasz Nykiel Fix For: 0.8.0 Attachments: HIVE-2185.1.patch, HIVE-2185.2.patch, HIVE-2185.patch Currently, when executing INSERT OVERWRITE and ANALYZE TABLE commands we collect statistics about the number of rows per partition/table. Other statistics (e.g., total table/partition size) are derived from the file system. Here, we want to collect information about the sizes of uncompressed data, to be able to determine the efficiency of compression. Currently, a large part of statistics collection mechanism is hardcoded and not-easily extensible for other statistics. On top of adding the new statistic collected, it would be desirable to extend the collection mechanism, so any new statistics could be added easily. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2504) Warehouse table subdirectories should inherit the group permissions of the warehouse parent directory
[ https://issues.apache.org/jira/browse/HIVE-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244756#comment-13244756 ] Rohini Palaniswamy commented on HIVE-2504: -- Haven't gotten any response on this for more than a week. So I am assuming no one is looking at it. Chinna, If you are not working on this, I can pick this one up. Warehouse table subdirectories should inherit the group permissions of the warehouse parent directory - Key: HIVE-2504 URL: https://issues.apache.org/jira/browse/HIVE-2504 Project: Hive Issue Type: Bug Components: Metastore Reporter: Carl Steinbach Assignee: Chinna Rao Lalam Fix For: 0.9.0 Attachments: HIVE-2504.patch When the Hive Metastore creates a subdirectory in the Hive warehouse for a new table it does so with the default HDFS permissions. Since the default dfs.umask value is 022, this means that the new subdirectory will not inherit the group write permissions of the hive warehouse directory. We should make the umask used by Warehouse.mkdirs() configurable, and set it to use a default value of 002. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2848) Add validation to HiveConf ConfVars
[ https://issues.apache.org/jira/browse/HIVE-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2848: -- Attachment: HIVE-2848.D2313.3.patch navis updated the revision HIVE-2848 [jira] Add validation to HiveConf ConfVars. Reviewers: JIRA 1. Enum-ed configuration type. 2. Removed checking existence of hive.~ configurations. 3. Fixed test-case bugs in udaf_context_ngrams.q, udaf_ngrams.q and udaf_percentile_approx.q after applying validation. except 3, passed all tests. REVISION DETAIL https://reviews.facebook.net/D2313 AFFECTED FILES cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java ql/src/test/queries/clientnegative/set_hiveconf_validation0.q ql/src/test/queries/clientnegative/set_hiveconf_validation1.q ql/src/test/queries/clientpositive/udaf_context_ngrams.q ql/src/test/queries/clientpositive/udaf_ngrams.q ql/src/test/queries/clientpositive/udaf_percentile_approx.q ql/src/test/results/clientnegative/set_hiveconf_validation0.q.out ql/src/test/results/clientnegative/set_hiveconf_validation1.q.out ql/src/test/results/clientpositive/udaf_context_ngrams.q.out ql/src/test/results/clientpositive/udaf_ngrams.q.out ql/src/test/results/clientpositive/udaf_percentile_approx.q.out Add validation to HiveConf ConfVars --- Key: HIVE-2848 URL: https://issues.apache.org/jira/browse/HIVE-2848 Project: Hive Issue Type: Improvement Affects Versions: 0.9.0 Reporter: Kevin Wilfong Assignee: Navis Attachments: HIVE-2848.D2313.1.patch, HIVE-2848.D2313.2.patch, HIVE-2848.D2313.3.patch It would be good if we could add some validation to ConfVars in the HiveConf when they are set. At least if a variable is supposed to be boolean, numeric, or strict/nonstrict, it shouldn't be too difficult to check for these. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2922) PartitionSpec should have a API for getting partition filter string instead of using toString()
PartitionSpec should have a API for getting partition filter string instead of using toString() --- Key: HIVE-2922 URL: https://issues.apache.org/jira/browse/HIVE-2922 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.9.0 Reporter: Rohini Palaniswamy Priority: Minor -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2920) TestStatsPublisherEnhanced throws NPE on JDBC connection failure
[ https://issues.apache.org/jira/browse/HIVE-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244876#comment-13244876 ] Phabricator commented on HIVE-2920: --- zhenxiao has commented on the revision HIVE-2920 [jira] TestStatsPublisherEnhanced throws NPE on JDBC connection failure. + 1 REVISION DETAIL https://reviews.facebook.net/D2577 TestStatsPublisherEnhanced throws NPE on JDBC connection failure Key: HIVE-2920 URL: https://issues.apache.org/jira/browse/HIVE-2920 Project: Hive Issue Type: Bug Components: Statistics, Testing Infrastructure Reporter: Carl Steinbach Assignee: Carl Steinbach Attachments: HIVE-2920.D2577.1.patch, HIVE-2920.D2577.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2923) testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac
testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac Key: HIVE-2923 URL: https://issues.apache.org/jira/browse/HIVE-2923 Project: Hive Issue Type: Bug Environment: Mac OSX Lion Reporter: Kevin Wilfong Assignee: Ashutosh Chauhan Priority: Blocker When running testAclPositive in TestZooKeeperTokenStore in a clean checkout, it fails with the error: Failed to validate token path. org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Failed to validate token path. at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:207) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.setConf(ZooKeeperTokenStore.java:225) at org.apache.hadoop.hive.thrift.TestZooKeeperTokenStore.testAclPositive(TestZooKeeperTokenStore.java:170) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:232) at junit.framework.TestSuite.run(TestSuite.java:227) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /zktokenstore-testAcl at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:119) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:204) ... 17 more This message is also printed to standard out: Unable to load realm mapping info from SCDynamicStore The test seems to run fine in Linux, but more than one developer has reported this on a Mac. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2585) Collapse hive.metastore.uris and hive.metastore.local
[
https://issues.apache.org/jira/browse/HIVE-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244895#comment-13244895
]
Ashutosh Chauhan commented on HIVE-2585:
{code}
BUILD SUCCESSFUL
Total time: 339 minutes 51 seconds
{code}
All the tests passed with this patch.
Collapse hive.metastore.uris and hive.metastore.local
-
Key: HIVE-2585
URL: https://issues.apache.org/jira/browse/HIVE-2585
Project: Hive
Issue Type: Improvement
Components: Metastore
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Attachments: HIVE-2585.D2559.1.patch
We should just have hive.metastore.uris. If it is empty, we shall assume
local mode, if non-empty we shall use that string to connect to remote
metastore. Having two different keys for same information is confusing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive 0.9 release
Hi all, Branch for 0.8-r2 was created on Dec 7, almost four months ago. Between then and now lots of cool stuff has landed in trunk waiting to be released and get to users. I think its a good time now to get the ball rolling for 0.9 release. If this sounds good, I would propose to cut a branch for 0.9 later this week. Then we can focus on stabilizing the branch and subsequent release from it. Thoughts? Thanks, Ashutosh
[jira] [Updated] (HIVE-2908) Hive: Extend ALTER TABLE DROP PARTITION syntax to use all comparators
[ https://issues.apache.org/jira/browse/HIVE-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-2908: --- Fix Version/s: 0.9.0 Hive: Extend ALTER TABLE DROP PARTITION syntax to use all comparators - Key: HIVE-2908 URL: https://issues.apache.org/jira/browse/HIVE-2908 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.9.0 Reporter: Sambavi Muthukrishnan Assignee: Sambavi Muthukrishnan Priority: Minor Fix For: 0.9.0 Attachments: HIVE-2908.D2523.1.patch, HIVE-2908.D2523.2.patch, HIVE-2908.final.patch.txt Original Estimate: 96h Remaining Estimate: 96h To drop a partition from a Hive table, this works: ALTER TABLE foo DROP PARTITION(ds = 'date') ...but it should also work to drop all partitions prior to date. ALTER TABLE foo DROP PARTITION(ds 'date') This task is to implement ALTER TABLE DROP PARTITION for all of the comparators, = = = != instead of just for =. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2860) TestNegativeCliDriver autolocal1.q fails on 0.23
[ https://issues.apache.org/jira/browse/HIVE-2860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244909#comment-13244909 ] Ashutosh Chauhan commented on HIVE-2860: Latest patch results in 5 failed tests in TestNegativeCliDriver TestNegativeCliDriver autolocal1.q fails on 0.23 Key: HIVE-2860 URL: https://issues.apache.org/jira/browse/HIVE-2860 Project: Hive Issue Type: Bug Components: Testing Infrastructure Affects Versions: 0.9.0 Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.9.0 Attachments: HIVE-2860.D2253.1.patch, HIVE-2860.D2253.1.patch, HIVE-2860.D2565.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2881) Remove redundant key comparing in SMBMapJoinOperator
[ https://issues.apache.org/jira/browse/HIVE-2881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-2881: --- Fix Version/s: 0.9.0 Remove redundant key comparing in SMBMapJoinOperator Key: HIVE-2881 URL: https://issues.apache.org/jira/browse/HIVE-2881 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Fix For: 0.9.0 Attachments: HIVE-2881.D2379.1.patch Currently, SMBJoin compares keys twice in #findSmallestKey and #joinObject. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2860) TestNegativeCliDriver autolocal1.q fails on 0.23
[ https://issues.apache.org/jira/browse/HIVE-2860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244912#comment-13244912 ] Carl Steinbach commented on HIVE-2860: -- @Ashutosh: Which tests failed? TestNegativeCliDriver autolocal1.q fails on 0.23 Key: HIVE-2860 URL: https://issues.apache.org/jira/browse/HIVE-2860 Project: Hive Issue Type: Bug Components: Testing Infrastructure Affects Versions: 0.9.0 Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.9.0 Attachments: HIVE-2860.D2253.1.patch, HIVE-2860.D2253.1.patch, HIVE-2860.D2565.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2646) Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
[ https://issues.apache.org/jira/browse/HIVE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2646: --- Fix Version/s: 0.9.0 Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs Key: HIVE-2646 URL: https://issues.apache.org/jira/browse/HIVE-2646 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.8.0 Reporter: Andrew Bayer Assignee: Andrew Bayer Priority: Critical Fix For: 0.9.0 Attachments: HIVE-2646.D2133.1.patch, HIVE-2646.D2133.10.patch, HIVE-2646.D2133.11.patch, HIVE-2646.D2133.12.patch, HIVE-2646.D2133.2.patch, HIVE-2646.D2133.3.patch, HIVE-2646.D2133.4.patch, HIVE-2646.D2133.5.patch, HIVE-2646.D2133.6.patch, HIVE-2646.D2133.7.patch, HIVE-2646.D2133.8.patch, HIVE-2646.D2133.9.patch, HIVE-2646.diff.txt The current Hive Ivy dependency logic for its Hadoop dependencies is problematic - depending on the tarball and extracting the jars from there, rather than depending on the jars directly. It'd be great if this was fixed to actually have the jar dependencies defined directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Hive 0.9 release
Here is a list of jiras which I plan to get in 0.9. HIVE-2084 HIVE-2822 HIVE-2764 HIVE-538 I will work with authors of these patches to see these can get in. Others, please feel free to add this list. Thanks, Ashutosh On Mon, Apr 2, 2012 at 18:39, Carl Steinbach c...@cloudera.com wrote: I'm +1 on doing an 0.9.0 release, but would also like to suggest that we put together a list of 0.9.0 blockers before cutting the release branch. In the past we have frequently underestimated the amount of work required to get trunk into a releasable state, with the consequence that we up wasting time doing a lot of backports from trunk to the release branch. It would be great if we could avoid all of that this time around. Thanks. Carl On Mon, Apr 2, 2012 at 6:33 PM, Ashutosh Chauhan hashut...@apache.org wrote: Hi all, Branch for 0.8-r2 was created on Dec 7, almost four months ago. Between then and now lots of cool stuff has landed in trunk waiting to be released and get to users. I think its a good time now to get the ball rolling for 0.9 release. If this sounds good, I would propose to cut a branch for 0.9 later this week. Then we can focus on stabilizing the branch and subsequent release from it. Thoughts? Thanks, Ashutosh
[jira] [Updated] (HIVE-2585) Collapse hive.metastore.uris and hive.metastore.local
[ https://issues.apache.org/jira/browse/HIVE-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-2585: - Status: Open (was: Patch Available) Collapse hive.metastore.uris and hive.metastore.local - Key: HIVE-2585 URL: https://issues.apache.org/jira/browse/HIVE-2585 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-2585.D2559.1.patch We should just have hive.metastore.uris. If it is empty, we shall assume local mode, if non-empty we shall use that string to connect to remote metastore. Having two different keys for same information is confusing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2585) Collapse hive.metastore.uris and hive.metastore.local
[ https://issues.apache.org/jira/browse/HIVE-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244916#comment-13244916 ] Phabricator commented on HIVE-2585: --- cwsteinbach has requested changes to the revision HIVE-2585 [jira] Collapse hive.metastore.uris and hive.metastore.local. INLINE COMMENTS conf/hive-default.xml.template:102 We should throw an error at startup if both hive.metastore.uris and javax.jdo.option.ConnectionURL are set. Giving silent preference to one of these properties is going to cause a lot of confusion for people who are using hive.metastore.local to quickly swap between local and remote metastores. data/conf/hive-site.xml:71 It would be nice to add a deprecation warning at startup if this is set. REVISION DETAIL https://reviews.facebook.net/D2559 BRANCH svn Collapse hive.metastore.uris and hive.metastore.local - Key: HIVE-2585 URL: https://issues.apache.org/jira/browse/HIVE-2585 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-2585.D2559.1.patch We should just have hive.metastore.uris. If it is empty, we shall assume local mode, if non-empty we shall use that string to connect to remote metastore. Having two different keys for same information is confusing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Hive 0.9 release
Would be great to get HIVE-2646 included. Thanks, Thomas On 4/2/12 6:59 PM, Ashutosh Chauhan hashut...@apache.org wrote: Here is a list of jiras which I plan to get in 0.9. HIVE-2084 HIVE-2822 HIVE-2764 HIVE-538 I will work with authors of these patches to see these can get in. Others, please feel free to add this list. Thanks, Ashutosh On Mon, Apr 2, 2012 at 18:39, Carl Steinbach c...@cloudera.com wrote: I'm +1 on doing an 0.9.0 release, but would also like to suggest that we put together a list of 0.9.0 blockers before cutting the release branch. In the past we have frequently underestimated the amount of work required to get trunk into a releasable state, with the consequence that we up wasting time doing a lot of backports from trunk to the release branch. It would be great if we could avoid all of that this time around. Thanks. Carl On Mon, Apr 2, 2012 at 6:33 PM, Ashutosh Chauhan hashut...@apache.org wrote: Hi all, Branch for 0.8-r2 was created on Dec 7, almost four months ago. Between then and now lots of cool stuff has landed in trunk waiting to be released and get to users. I think its a good time now to get the ball rolling for 0.9 release. If this sounds good, I would propose to cut a branch for 0.9 later this week. Then we can focus on stabilizing the branch and subsequent release from it. Thoughts? Thanks, Ashutosh
[jira] [Commented] (HIVE-2860) TestNegativeCliDriver autolocal1.q fails on 0.23
[ https://issues.apache.org/jira/browse/HIVE-2860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244919#comment-13244919 ] Ashutosh Chauhan commented on HIVE-2860: * external1.q * external2.q * fetchtask_ioexception.q * fs_default_name1.q * fs_default_name1.q TestNegativeCliDriver autolocal1.q fails on 0.23 Key: HIVE-2860 URL: https://issues.apache.org/jira/browse/HIVE-2860 Project: Hive Issue Type: Bug Components: Testing Infrastructure Affects Versions: 0.9.0 Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.9.0 Attachments: HIVE-2860.D2253.1.patch, HIVE-2860.D2253.1.patch, HIVE-2860.D2565.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2741) Single binary built against 0.20 and 0.23, does not work against 0.23 clusters.
[ https://issues.apache.org/jira/browse/HIVE-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2741: --- Fix Version/s: 0.9.0 Requires build against 0.23.1 Single binary built against 0.20 and 0.23, does not work against 0.23 clusters. --- Key: HIVE-2741 URL: https://issues.apache.org/jira/browse/HIVE-2741 Project: Hive Issue Type: Bug Components: Shims Affects Versions: 0.8.1 Reporter: Amareshwari Sriramadasu Fix For: 0.9.0 After HIVE-2629, if single binary is built for 0.20 and 0.23, it results in following exception in 0.23 clusters: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapred.Counters$Counter, but class was expected at org.apache.hadoop.hive.ql.exec.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:341) at org.apache.hadoop.hive.ql.exec.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:685) at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:458) at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:200) FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.MapRedTask If we have to make single binary work against both 0.20 and 0.23, we need to move all such in-compatibilities to Shims. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2866) Cache local map reduce job errors for additional logging
[ https://issues.apache.org/jira/browse/HIVE-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-2866: --- Fix Version/s: 0.9.0 Cache local map reduce job errors for additional logging Key: HIVE-2866 URL: https://issues.apache.org/jira/browse/HIVE-2866 Project: Hive Issue Type: Improvement Components: Logging Reporter: Kevin Wilfong Assignee: Kevin Wilfong Fix For: 0.9.0 Attachments: HIVE-2866.D2277.1.patch Using the CachingPrintStream we are storing errors that occur locally in Hive, but because local map reduce jobs are run in a separate JVM we are not storing that occur for these. We can use this same construct to store errors written to the subprocesses error stream. This way, when we log failed queries, these will give us a decent idea of why those queries failed. See related issues: https://issues.apache.org/jira/browse/HIVE-2832 https://issues.apache.org/jira/browse/HIVE-2858 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2676) The row count that loaded to a table may not right
[ https://issues.apache.org/jira/browse/HIVE-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-2676: --- Fix Version/s: 0.9.0 The row count that loaded to a table may not right - Key: HIVE-2676 URL: https://issues.apache.org/jira/browse/HIVE-2676 Project: Hive Issue Type: Improvement Reporter: binlijin Priority: Minor Labels: patch Fix For: 0.9.0 Attachments: HIVE-2676.patch create table tablename as SELECT *** At the end hive will print a number that show how many Rows loaded to the tablename, but sometimes the number is not right. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2875) Renaming partition changes partition location prefix
[ https://issues.apache.org/jira/browse/HIVE-2875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-2875: --- Fix Version/s: 0.9.0 Renaming partition changes partition location prefix Key: HIVE-2875 URL: https://issues.apache.org/jira/browse/HIVE-2875 Project: Hive Issue Type: Bug Reporter: Kevin Wilfong Assignee: Kevin Wilfong Fix For: 0.9.0 Attachments: HIVE-2875.D2349.1.patch, HIVE-2875.D2349.2.patch Renaming a partition changes the location of the partition to the default location of the table, followed by the partition specification. It should just change the partition specification of the path. If the path does not end with the old partition specification, we should probably throw an exception because renaming a partition should not change the path so dramatically, and not changing the path to reflect the new partition name could leave the partition in a very confusing state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2908) Hive: Extend ALTER TABLE DROP PARTITION syntax to use all comparators
[ https://issues.apache.org/jira/browse/HIVE-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244924#comment-13244924 ] Hudson commented on HIVE-2908: -- Integrated in Hive-trunk-h0.21 #1348 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1348/]) HIVE-2908. Hive: Extend ALTER TABLE DROP PARTITION syntax to use all comparators. (sambavim via kevinwilfong) (Revision 1308427) Result = SUCCESS kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1308427 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/DropTableDesc.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionSpec.java * /hive/trunk/ql/src/test/queries/clientnegative/drop_partition_filter_failure.q * /hive/trunk/ql/src/test/queries/clientpositive/drop_partitions_filter.q * /hive/trunk/ql/src/test/results/clientnegative/drop_partition_failure.q.out * /hive/trunk/ql/src/test/results/clientnegative/drop_partition_filter_failure.q.out * /hive/trunk/ql/src/test/results/clientpositive/drop_multi_partitions.q.out * /hive/trunk/ql/src/test/results/clientpositive/drop_partitions_filter.q.out * /hive/trunk/ql/src/test/results/clientpositive/escape1.q.out Hive: Extend ALTER TABLE DROP PARTITION syntax to use all comparators - Key: HIVE-2908 URL: https://issues.apache.org/jira/browse/HIVE-2908 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.9.0 Reporter: Sambavi Muthukrishnan Assignee: Sambavi Muthukrishnan Priority: Minor Fix For: 0.9.0 Attachments: HIVE-2908.D2523.1.patch, HIVE-2908.D2523.2.patch, HIVE-2908.final.patch.txt Original Estimate: 96h Remaining Estimate: 96h To drop a partition from a Hive table, this works: ALTER TABLE foo DROP PARTITION(ds = 'date') ...but it should also work to drop all partitions prior to date. ALTER TABLE foo DROP PARTITION(ds 'date') This task is to implement ALTER TABLE DROP PARTITION for all of the comparators, = = = != instead of just for =. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2767) Optionally use framed transport with metastore
[
https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-2767:
---
Status: Open (was: Patch Available)
Patch needs to be rebased.
Optionally use framed transport with metastore
--
Key: HIVE-2767
URL: https://issues.apache.org/jira/browse/HIVE-2767
Project: Hive
Issue Type: New Feature
Components: Metastore
Reporter: Travis Crawford
Assignee: Julien Le Dem
Attachments: HIVE-2767.patch.txt, HIVE-2767_a.patch.txt
Users may want/need to use thrift's framed transport when communicating with
the Hive MetaStore. This patch adds a new property
{{hive.metastore.thrift.framed.transport.enabled}} that enables the framed
transport (defaults to off, aka no change from before the patch). This
property must be set for both clients and the HMS server.
It wasn't immediately clear how to use the framed transport with SASL, so as
written an exception is thrown if you try starting the server with both
options. If SASL and the framed transport will indeed work together I can
update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2332) If all of the parameters of distinct functions are exists in group by columns, query fails in runtime
[
https://issues.apache.org/jira/browse/HIVE-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-2332:
---
Status: Open (was: Patch Available)
Canceling patch as it first needs to address Namit's comments.
If all of the parameters of distinct functions are exists in group by
columns, query fails in runtime
-
Key: HIVE-2332
URL: https://issues.apache.org/jira/browse/HIVE-2332
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Critical
Fix For: 0.9.0
Attachments: HIVE-2332.1.patch.txt, HIVE-2332.2.patch.txt,
HIVE-2332.D663.1.patch
select sum(key_int1), sum(distinct key_int1) from t1 group by key_int1;
fails with message..
{code}
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
{code}
hadoop says..
{code}
Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.RangeCheck(ArrayList.java:547)
at java.util.ArrayList.get(ArrayList.java:322)
at
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.init(StandardStructObjectInspector.java:95)
at
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.(StandardStructObjectInspector.java:86)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:252)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.initEvaluatorsAndReturnStruct(ReduceSinkOperator.java:188)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:197)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:85)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:532)
{code}
I think the deficient number of key expression, compared to number of key
column, is the problem, which should be equal or more.
Would it be solved if add some key expression? I'll try.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2332) If all of the parameters of distinct functions are exists in group by columns, query fails in runtime
[
https://issues.apache.org/jira/browse/HIVE-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244945#comment-13244945
]
Navis commented on HIVE-2332:
-
It's already on phabricator by Namit.
If all of the parameters of distinct functions are exists in group by
columns, query fails in runtime
-
Key: HIVE-2332
URL: https://issues.apache.org/jira/browse/HIVE-2332
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Critical
Fix For: 0.9.0
Attachments: HIVE-2332.1.patch.txt, HIVE-2332.2.patch.txt,
HIVE-2332.D663.1.patch
select sum(key_int1), sum(distinct key_int1) from t1 group by key_int1;
fails with message..
{code}
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
{code}
hadoop says..
{code}
Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.RangeCheck(ArrayList.java:547)
at java.util.ArrayList.get(ArrayList.java:322)
at
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.init(StandardStructObjectInspector.java:95)
at
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.(StandardStructObjectInspector.java:86)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:252)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.initEvaluatorsAndReturnStruct(ReduceSinkOperator.java:188)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:197)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:85)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:532)
{code}
I think the deficient number of key expression, compared to number of key
column, is the problem, which should be equal or more.
Would it be solved if add some key expression? I'll try.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2861) Support eventual constant expression for filter pushdown for key ranges in hbase
[
https://issues.apache.org/jira/browse/HIVE-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244950#comment-13244950
]
Phabricator commented on HIVE-2861:
---
ashutoshc has accepted the revision HIVE-2861 [jira] Support eventual constant
expression for filter pushdown for key ranges in hbase.
+1 will commit if tests pass.
REVISION DETAIL
https://reviews.facebook.net/D2259
BRANCH
DPAL-950
Support eventual constant expression for filter pushdown for key ranges in
hbase
Key: HIVE-2861
URL: https://issues.apache.org/jira/browse/HIVE-2861
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-2861.D2259.1.patch
Minor upgrade from HIVE-2771, which supports simple eventual constant
expression as a filter (especially 'cast').
For example,
{noformat}
select * from hbase_pushdown where key cast(20 + 30 as string);
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2332) If all of the parameters of distinct functions are exists in group by columns, query fails in runtime
[
https://issues.apache.org/jira/browse/HIVE-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-2332:
---
Status: Patch Available (was: Open)
Oh, I missed it. Marking Patch Available then.
If all of the parameters of distinct functions are exists in group by
columns, query fails in runtime
-
Key: HIVE-2332
URL: https://issues.apache.org/jira/browse/HIVE-2332
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Critical
Fix For: 0.9.0
Attachments: HIVE-2332.1.patch.txt, HIVE-2332.2.patch.txt,
HIVE-2332.D663.1.patch
select sum(key_int1), sum(distinct key_int1) from t1 group by key_int1;
fails with message..
{code}
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
{code}
hadoop says..
{code}
Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.RangeCheck(ArrayList.java:547)
at java.util.ArrayList.get(ArrayList.java:322)
at
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.init(StandardStructObjectInspector.java:95)
at
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.(StandardStructObjectInspector.java:86)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:252)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.initEvaluatorsAndReturnStruct(ReduceSinkOperator.java:188)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:197)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:85)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:532)
{code}
I think the deficient number of key expression, compared to number of key
column, is the problem, which should be equal or more.
Would it be solved if add some key expression? I'll try.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2738) NPE in ExprNodeGenericFuncEvaluator
[
https://issues.apache.org/jira/browse/HIVE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244954#comment-13244954
]
Ashutosh Chauhan commented on HIVE-2738:
+1 will commit if tests pass.
NPE in ExprNodeGenericFuncEvaluator
---
Key: HIVE-2738
URL: https://issues.apache.org/jira/browse/HIVE-2738
Project: Hive
Issue Type: Bug
Affects Versions: 0.8.0
Reporter: Nicolas Lalevée
Assignee: Nicolas Lalevée
Attachments: 750c8966-6402-465a-b011-903469fe56da.xml,
HIVE-2738-r1237763.patch, MapMaxUDF.java, MapToJsonUDF.java, hive_job_logs.txt
Here is the query:
bq. {{SELECT t.lid, '2011-12-12',
s_map2json(s_maxmap(UNION_MAP(t.categoryCount), 100)) FROM ( SELECT theme_lid
AS theme_lid, MAP(s_host(referer), COUNT( * )) AS categoryCount FROM
PageViewEvent WHERE day = '20130104' AND day = '20130112' AND date_ =
'2012-01-04' AND date_ '2012-01-13' AND lid IS NOT NULL GROUP BY lid,
s_host(referer) ) t GROUP BY t.lid}}
Removing the call s_map2json make it work but not by removing s_maxmap, but I
don't understand what could be wrong with the implementation of my udf. And I
don't know how to debug remote hadoop jobs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2576) HiveDataSource doesn't get a proper connection
[ https://issues.apache.org/jira/browse/HIVE-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-2576: --- Status: Open (was: Patch Available) HiveDataSource doesn't get a proper connection -- Key: HIVE-2576 URL: https://issues.apache.org/jira/browse/HIVE-2576 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.8.0 Reporter: Nicolas Lalevée Assignee: Nicolas Lalevée Attachments: HIVE-2576-r1201637.patch The HiveDataSource is creating a HiveConnection with as an URL , but the connection expects to start with jdbc:hive:// -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2576) HiveDataSource doesn't get a proper connection
[
https://issues.apache.org/jira/browse/HIVE-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244958#comment-13244958
]
Ashutosh Chauhan commented on HIVE-2576:
@Nicolas,
Patch looks good. Instead of hardcoding string, can you make {{URI_PREFIX}} in
HiveConnection public and reference that.
HiveDataSource doesn't get a proper connection
--
Key: HIVE-2576
URL: https://issues.apache.org/jira/browse/HIVE-2576
Project: Hive
Issue Type: Bug
Components: JDBC
Affects Versions: 0.8.0
Reporter: Nicolas Lalevée
Assignee: Nicolas Lalevée
Attachments: HIVE-2576-r1201637.patch
The HiveDataSource is creating a HiveConnection with as an URL , but the
connection expects to start with jdbc:hive://
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2913) BlockMergeTask Doesn't Honor Job Configuration Properties when used directly
[ https://issues.apache.org/jira/browse/HIVE-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244971#comment-13244971 ] Amareshwari Sriramadasu commented on HIVE-2913: --- +1. Running tests. Will commit once tests pass. BlockMergeTask Doesn't Honor Job Configuration Properties when used directly Key: HIVE-2913 URL: https://issues.apache.org/jira/browse/HIVE-2913 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.8.1 Reporter: Thiruvel Thirumoolan Assignee: Thiruvel Thirumoolan Priority: Minor Fix For: 0.9.0 Attachments: HIVE-2913.D2547.1.patch, HIVE-2913_1.patch, blockmerge.sh BlockMergeTask has a main() and when used directly (instead of say partition concatenate feature), the -jobconf arguments are not honored. This is not something most people directly use. Usage: BlockMergeTask -input colon seperated input paths -outputDir outputDir [-jobconf k1=v1 [-jobconf k2=v2] ...] To reproduce: Run BlockMergeTask with say -jobconf mapred.job.name=test and launched job will have a different name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2913) BlockMergeTask Doesn't Honor Job Configuration Properties when used directly
[ https://issues.apache.org/jira/browse/HIVE-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244973#comment-13244973 ] Phabricator commented on HIVE-2913: --- amareshwarisr has accepted the revision HIVE-2913 [jira] BlockMergeTask Doesn't Honor Job Configuration Properties when used directly. +1 REVISION DETAIL https://reviews.facebook.net/D2547 BRANCH svn BlockMergeTask Doesn't Honor Job Configuration Properties when used directly Key: HIVE-2913 URL: https://issues.apache.org/jira/browse/HIVE-2913 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.8.1 Reporter: Thiruvel Thirumoolan Assignee: Thiruvel Thirumoolan Priority: Minor Fix For: 0.9.0 Attachments: HIVE-2913.D2547.1.patch, HIVE-2913_1.patch, blockmerge.sh BlockMergeTask has a main() and when used directly (instead of say partition concatenate feature), the -jobconf arguments are not honored. This is not something most people directly use. Usage: BlockMergeTask -input colon seperated input paths -outputDir outputDir [-jobconf k1=v1 [-jobconf k2=v2] ...] To reproduce: Run BlockMergeTask with say -jobconf mapred.job.name=test and launched job will have a different name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2711) Make the header of RCFile unique
[ https://issues.apache.org/jira/browse/HIVE-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244981#comment-13244981 ] Ashutosh Chauhan commented on HIVE-2711: Patch results failures in TestCliDriver in following queries: * alter_concatenate_indexed_table.q * alter_merge.q * alter_merge_stats.q * create_merge_compressed.q * ctas.q * partition_wise_fileformat.q * partition_wise_fileformat3.q * sample10.q Make the header of RCFile unique Key: HIVE-2711 URL: https://issues.apache.org/jira/browse/HIVE-2711 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-2711.D2115.1.patch, HIVE-2711.D2115.2.patch, HIVE-2711.D2571.1.patch The RCFile implementation was copied from Hadoop's SequenceFile and copied the 'magic' string in the header. This means that you can't use the header to distinguish between RCFiles and SequenceFiles. I'd propose that we create a new header for RCFiles (RCF?) to replace the current SEQ. To maintain compatibility, we'll need to continue to accept the current 'SEQ\06' and just make new files contain the new header. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2848) Add validation to HiveConf ConfVars
[ https://issues.apache.org/jira/browse/HIVE-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244989#comment-13244989 ] Phabricator commented on HIVE-2848: --- njain has commented on the revision HIVE-2848 [jira] Add validation to HiveConf ConfVars. INLINE COMMENTS common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:534 Not done yet REVISION DETAIL https://reviews.facebook.net/D2313 Add validation to HiveConf ConfVars --- Key: HIVE-2848 URL: https://issues.apache.org/jira/browse/HIVE-2848 Project: Hive Issue Type: Improvement Affects Versions: 0.9.0 Reporter: Kevin Wilfong Assignee: Navis Attachments: HIVE-2848.D2313.1.patch, HIVE-2848.D2313.2.patch, HIVE-2848.D2313.3.patch It would be good if we could add some validation to ConfVars in the HiveConf when they are set. At least if a variable is supposed to be boolean, numeric, or strict/nonstrict, it shouldn't be too difficult to check for these. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2848) Add validation to HiveConf ConfVars
[ https://issues.apache.org/jira/browse/HIVE-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245000#comment-13245000 ] Phabricator commented on HIVE-2848: --- navis has commented on the revision HIVE-2848 [jira] Add validation to HiveConf ConfVars. INLINE COMMENTS common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:534 I've tested with this value 'true' and changed back to 'false', which seemed not to be you're intent. Patch will be updated shortly. REVISION DETAIL https://reviews.facebook.net/D2313 Add validation to HiveConf ConfVars --- Key: HIVE-2848 URL: https://issues.apache.org/jira/browse/HIVE-2848 Project: Hive Issue Type: Improvement Affects Versions: 0.9.0 Reporter: Kevin Wilfong Assignee: Navis Attachments: HIVE-2848.D2313.1.patch, HIVE-2848.D2313.2.patch, HIVE-2848.D2313.3.patch It would be good if we could add some validation to ConfVars in the HiveConf when they are set. At least if a variable is supposed to be boolean, numeric, or strict/nonstrict, it shouldn't be too difficult to check for these. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2848) Add validation to HiveConf ConfVars
[ https://issues.apache.org/jira/browse/HIVE-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2848: -- Attachment: HIVE-2848.D2313.4.patch navis updated the revision HIVE-2848 [jira] Add validation to HiveConf ConfVars. Reviewers: JIRA Changed default value to 'true' REVISION DETAIL https://reviews.facebook.net/D2313 AFFECTED FILES cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java ql/src/test/queries/clientnegative/set_hiveconf_validation0.q ql/src/test/queries/clientnegative/set_hiveconf_validation1.q ql/src/test/queries/clientpositive/udaf_context_ngrams.q ql/src/test/queries/clientpositive/udaf_ngrams.q ql/src/test/queries/clientpositive/udaf_percentile_approx.q ql/src/test/results/clientnegative/set_hiveconf_validation0.q.out ql/src/test/results/clientnegative/set_hiveconf_validation1.q.out ql/src/test/results/clientpositive/udaf_context_ngrams.q.out ql/src/test/results/clientpositive/udaf_ngrams.q.out ql/src/test/results/clientpositive/udaf_percentile_approx.q.out Add validation to HiveConf ConfVars --- Key: HIVE-2848 URL: https://issues.apache.org/jira/browse/HIVE-2848 Project: Hive Issue Type: Improvement Affects Versions: 0.9.0 Reporter: Kevin Wilfong Assignee: Navis Attachments: HIVE-2848.D2313.1.patch, HIVE-2848.D2313.2.patch, HIVE-2848.D2313.3.patch, HIVE-2848.D2313.4.patch It would be good if we could add some validation to ConfVars in the HiveConf when they are set. At least if a variable is supposed to be boolean, numeric, or strict/nonstrict, it shouldn't be too difficult to check for these. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
