[jira] Commented: (HIVE-1746) Support for using ALTER to set IDXPROPERTIES
[ https://issues.apache.org/jira/browse/HIVE-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932680#action_12932680 ] John Sichi commented on HIVE-1746: -- My suggestion is to push the user-get completely inside of updateModifiedParameters (which would then return int, and the caller would check). > Support for using ALTER to set IDXPROPERTIES > > > Key: HIVE-1746 > URL: https://issues.apache.org/jira/browse/HIVE-1746 > Project: Hive > Issue Type: Improvement > Components: Indexing >Affects Versions: 0.7.0 >Reporter: Marquis Wang >Assignee: Marquis Wang > Fix For: 0.7.0 > > Attachments: 1746.prelim.patch, HIVE-1746.2.patch, HIVE-1746.3.patch > > > Hive-1498 has support for IDXPROPERTIES on index creation, so now we want to > support ALTERing those properties. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1785) change Pre/Post Query Hooks to take in 1 parameter: HookContext
[ https://issues.apache.org/jira/browse/HIVE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933081#action_12933081 ] John Sichi commented on HIVE-1785: -- The way the new interface is defined, people writing the hooks will need to keep implementing the old Pre/PostExec interfaces, even though we will no longer be calling those methods. This will make it harder to remove cruft. > change Pre/Post Query Hooks to take in 1 parameter: HookContext > --- > > Key: HIVE-1785 > URL: https://issues.apache.org/jira/browse/HIVE-1785 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Namit Jain >Assignee: Liyin Tang > Attachments: hive-1785_3.patch, hive_1785_1.patch, hive_1785_2.patch > > > This way, it would be possible to add new parameters to the hooks without > changing the existing hooks. > This will be a incompatible change, and all the hooks need to change to the > new API -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1785) change Pre/Post Query Hooks to take in 1 parameter: HookContext
[ https://issues.apache.org/jira/browse/HIVE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933230#action_12933230 ] John Sichi commented on HIVE-1785: -- +1. > change Pre/Post Query Hooks to take in 1 parameter: HookContext > --- > > Key: HIVE-1785 > URL: https://issues.apache.org/jira/browse/HIVE-1785 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Namit Jain >Assignee: Liyin Tang > Attachments: hive-1785_3.patch, hive-1785_4.patch, hive_1785_1.patch, > hive_1785_2.patch > > > This way, it would be possible to add new parameters to the hooks without > changing the existing hooks. > This will be a incompatible change, and all the hooks need to change to the > new API -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1785) change Pre/Post Query Hooks to take in 1 parameter: HookContext
[ https://issues.apache.org/jira/browse/HIVE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1785: - Status: Open (was: Patch Available) Discussed with Liyin, and there are a few more things we should do here: * add Javadoc for Hook, HookContext, and ExecuteWithHookContext * add deprecation annotation for old PreExecute/PostExecute * rewrite existing hook implementations in Hive codebase to implement new interface * run Facebook-specific tests to validate backwards-compatibility during the transition > change Pre/Post Query Hooks to take in 1 parameter: HookContext > --- > > Key: HIVE-1785 > URL: https://issues.apache.org/jira/browse/HIVE-1785 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Namit Jain >Assignee: Liyin Tang > Attachments: hive-1785_3.patch, hive-1785_4.patch, hive_1785_1.patch, > hive_1785_2.patch > > > This way, it would be possible to add new parameters to the hooks without > changing the existing hooks. > This will be a incompatible change, and all the hooks need to change to the > new API -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1746) Support for using ALTER to set IDXPROPERTIES
[ https://issues.apache.org/jira/browse/HIVE-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933634#action_12933634 ] John Sichi commented on HIVE-1746: -- +1. Will commit when tests pass. > Support for using ALTER to set IDXPROPERTIES > > > Key: HIVE-1746 > URL: https://issues.apache.org/jira/browse/HIVE-1746 > Project: Hive > Issue Type: Improvement > Components: Indexing >Affects Versions: 0.7.0 >Reporter: Marquis Wang >Assignee: Marquis Wang > Fix For: 0.7.0 > > Attachments: 1746.prelim.patch, HIVE-1746.2.patch, HIVE-1746.3.patch, > HIVE-1746.4.patch > > > Hive-1498 has support for IDXPROPERTIES on index creation, so now we want to > support ALTERing those properties. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1746) Support for using ALTER to set IDXPROPERTIES
[ https://issues.apache.org/jira/browse/HIVE-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1746: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed. Thanks Marquis! > Support for using ALTER to set IDXPROPERTIES > > > Key: HIVE-1746 > URL: https://issues.apache.org/jira/browse/HIVE-1746 > Project: Hive > Issue Type: Improvement > Components: Indexing >Affects Versions: 0.7.0 >Reporter: Marquis Wang >Assignee: Marquis Wang > Fix For: 0.7.0 > > Attachments: 1746.prelim.patch, HIVE-1746.2.patch, HIVE-1746.3.patch, > HIVE-1746.4.patch > > > Hive-1498 has support for IDXPROPERTIES on index creation, so now we want to > support ALTERing those properties. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1785) change Pre/Post Query Hooks to take in 1 parameter: HookContext
[ https://issues.apache.org/jira/browse/HIVE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1785: - Fix Version/s: 0.7.0 Affects Version/s: 0.7.0 Status: Open (was: Patch Available) * It looks like you mixed in some changes from other patches? (And the patch name is HIVE-1758 instead of HIVE-1785.) Please resubmit with only the changes relevant to HIVE-1785. Also, I don't see where you addressed two of my previous comments: * rewrite existing hook implementations in Hive codebase to implement new interface * add Javadoc for Hook, HookContext, and ExecuteWithHookContext One new one while we're at it: * The old hook interface provided SessionState, Set, Set, LineageInfo and UserGroupInformation. HookContext should provide getters for these as well. I realize that some of these are obtainable via QueryPlan and HiveConf, but we should not force users to become dependent on internals like QueryPlan unless they want to be; also, providing the equivalent getters makes upgrade to the new interface easier. You can make the HookContext getters call through to QueryPlan getters etc. > change Pre/Post Query Hooks to take in 1 parameter: HookContext > --- > > Key: HIVE-1785 > URL: https://issues.apache.org/jira/browse/HIVE-1785 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 0.7.0 >Reporter: Namit Jain >Assignee: Liyin Tang > Fix For: 0.7.0 > > Attachments: hive-1785_3.patch, hive-1785_4.patch, hive_1758_5.patch, > hive_1785_1.patch, hive_1785_2.patch > > > This way, it would be possible to add new parameters to the hooks without > changing the existing hooks. > This will be a incompatible change, and all the hooks need to change to the > new API -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1785) change Pre/Post Query Hooks to take in 1 parameter: HookContext
[ https://issues.apache.org/jira/browse/HIVE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1785: - Attachment: (was: hive_1758_5.patch) > change Pre/Post Query Hooks to take in 1 parameter: HookContext > --- > > Key: HIVE-1785 > URL: https://issues.apache.org/jira/browse/HIVE-1785 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 0.7.0 >Reporter: Namit Jain >Assignee: Liyin Tang > Fix For: 0.7.0 > > Attachments: hive-1785_3.patch, hive-1785_4.patch, hive-1785_6.patch, > hive_1785_1.patch, hive_1785_2.patch > > > This way, it would be possible to add new parameters to the hooks without > changing the existing hooks. > This will be a incompatible change, and all the hooks need to change to the > new API -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1785) change Pre/Post Query Hooks to take in 1 parameter: HookContext
[ https://issues.apache.org/jira/browse/HIVE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935023#action_12935023 ] John Sichi commented on HIVE-1785: -- +1 on latest. Will commit when tests pass. > change Pre/Post Query Hooks to take in 1 parameter: HookContext > --- > > Key: HIVE-1785 > URL: https://issues.apache.org/jira/browse/HIVE-1785 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 0.7.0 >Reporter: Namit Jain >Assignee: Liyin Tang > Fix For: 0.7.0 > > Attachments: hive-1785_3.patch, hive-1785_4.patch, hive-1785_6.patch, > hive_1785_1.patch, hive_1785_2.patch > > > This way, it would be possible to add new parameters to the hooks without > changing the existing hooks. > This will be a incompatible change, and all the hooks need to change to the > new API -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-1785) change Pre/Post Query Hooks to take in 1 parameter: HookContext
[ https://issues.apache.org/jira/browse/HIVE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi resolved HIVE-1785. -- Resolution: Fixed Release Note: PreExecute and PostExecute have been deprecated in favor of ExecuteWithHookContext. Committed. Thanks Liyin! Could you explain this change on the user mailing list? Also, we need a followup patch for changing the description of hive.exec.pre/post.hooks in conf/hive-default.xml (I just remembered that). > change Pre/Post Query Hooks to take in 1 parameter: HookContext > --- > > Key: HIVE-1785 > URL: https://issues.apache.org/jira/browse/HIVE-1785 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 0.7.0 >Reporter: Namit Jain >Assignee: Liyin Tang > Fix For: 0.7.0 > > Attachments: hive-1785_3.patch, hive-1785_4.patch, hive-1785_6.patch, > hive_1785_1.patch, hive_1785_2.patch > > > This way, it would be possible to add new parameters to the hooks without > changing the existing hooks. > This will be a incompatible change, and all the hooks need to change to the > new API -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1810) a followup patch for changing the description of hive.exec.pre/post.hooks in conf/hive-default.xml
[ https://issues.apache.org/jira/browse/HIVE-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1810: - Resolution: Fixed Fix Version/s: 0.7.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed. Thanks Liyin! > a followup patch for changing the description of hive.exec.pre/post.hooks in > conf/hive-default.xml > -- > > Key: HIVE-1810 > URL: https://issues.apache.org/jira/browse/HIVE-1810 > Project: Hive > Issue Type: Sub-task >Reporter: Liyin Tang >Assignee: Liyin Tang > Fix For: 0.7.0 > > Attachments: hive-1810-1.patch > > > a followup patch for changing the description of hive.exec.pre/post.hooks in > conf/hive-default.xml -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1812) spurious rmr failure messages when inserting with dynamic partitioning
spurious rmr failure messages when inserting with dynamic partitioning -- Key: HIVE-1812 URL: https://issues.apache.org/jira/browse/HIVE-1812 Project: Hive Issue Type: Bug Components: Logging Affects Versions: 0.6.0 Reporter: John Sichi Assignee: Ning Zhang Fix For: 0.7.0 Running a test such as load_dyn_part1.q, there is a lot of noise like this: test: [junit] Copying data from file:/Users/jsichi/open/hive-trunk/data/files/kv1.txt [junit] Loading data to table srcpart partition (ds=2008-04-08, hr=11) [junit] rmr: cannot remove pfile:/Users/jsichi/open/hive-trunk/build/ql/test/data/warehouse/srcpart/ds=2008-04-08/hr=11: No such file or directory. This shows up in production logs as well, which confuses users, making them thinking that the job is encountering problems when it isn't. Is there a way we can squelch these? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1539) Concurrent metastore threading problem
[ https://issues.apache.org/jira/browse/HIVE-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12965515#action_12965515 ] John Sichi commented on HIVE-1539: -- According to the datanucleus website, 2.2.0 is scheduled for 10-Dec, so let's wait for it. > Concurrent metastore threading problem > --- > > Key: HIVE-1539 > URL: https://issues.apache.org/jira/browse/HIVE-1539 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.7.0 >Reporter: Bennie Schut >Assignee: Bennie Schut > Attachments: ClassLoaderResolver.patch, HIVE-1539.patch, > thread_dump_hanging.txt > > > When running hive as a service and running a high number of queries > concurrently I end up with multiple threads running at 100% cpu without any > progress. > Looking at these threads I notice this thread(484e): > at > org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:598) > But on a different thread(63a2): > at > org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoReplaceField(MStorageDescriptor.java) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1079) CREATE VIEW followup: derive dependencies on underlying base table partitions from view definition
[ https://issues.apache.org/jira/browse/HIVE-1079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi reassigned HIVE-1079: Assignee: John Sichi (was: Carl Steinbach) > CREATE VIEW followup: derive dependencies on underlying base table > partitions from view definition > --- > > Key: HIVE-1079 > URL: https://issues.apache.org/jira/browse/HIVE-1079 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 0.6.0 >Reporter: John Sichi >Assignee: John Sichi > > When querying a view, it would be useful to know which underlying base table > partitions it depends on in order to know how fresh the result is (or to be > able to wait until all of those partitions have been loaded consistently). > The task is to come up with a way to perform this analysis automatically > (possibly overconservatively), or alternately to let the view creator > annotate the view definition with this dependency information, or some > combination of the two. > Note that this would be useful for any complex query which directly accesses > base tables (not just view definitions). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1763) drop table (or view) should issue warning if table doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12968855#action_12968855 ] John Sichi commented on HIVE-1763: -- See HIVE-1542 for my suggested approach. > drop table (or view) should issue warning if table doesn't exist > > > Key: HIVE-1763 > URL: https://issues.apache.org/jira/browse/HIVE-1763 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: dan f >Assignee: Paul Butler >Priority: Minor > Attachments: HIVE-1763.patch > > > drop table reports "OK" even if the table doesn't exist. Better to report > something like mysql's "Unknown table 'foo'" so that, e.g., unwanted tables > (especially ones with names prone to typos) don't persist. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1415) add CLI command for executing a SQL script
[ https://issues.apache.org/jira/browse/HIVE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969513#action_12969513 ] John Sichi commented on HIVE-1415: -- One review comment here: https://reviews.apache.org/r/151/ > add CLI command for executing a SQL script > -- > > Key: HIVE-1415 > URL: https://issues.apache.org/jira/browse/HIVE-1415 > Project: Hive > Issue Type: Improvement > Components: Clients >Affects Versions: 0.5.0 >Reporter: John Sichi >Assignee: Edward Capriolo > Fix For: 0.7.0 > > Attachments: hive-1415-1-patch.txt > > > Suggestion in HIVE-1405 was "source", e.g. > source somescript.sql; -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1415) add CLI command for executing a SQL script
[ https://issues.apache.org/jira/browse/HIVE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969539#action_12969539 ] John Sichi commented on HIVE-1415: -- +1. Will commit when tests pass. > add CLI command for executing a SQL script > -- > > Key: HIVE-1415 > URL: https://issues.apache.org/jira/browse/HIVE-1415 > Project: Hive > Issue Type: Improvement > Components: Clients >Affects Versions: 0.5.0 >Reporter: John Sichi >Assignee: Edward Capriolo > Fix For: 0.7.0 > > Attachments: hive-1415-1-patch.txt, hive-1415-2-patch.txt > > > Suggestion in HIVE-1405 was "source", e.g. > source somescript.sql; -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1415) add CLI command for executing a SQL script
[ https://issues.apache.org/jira/browse/HIVE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1415: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed. Thanks Ed! Since docs still aren't converted, can you update the wiki too (noting it as only available starting with 0.7)? > add CLI command for executing a SQL script > -- > > Key: HIVE-1415 > URL: https://issues.apache.org/jira/browse/HIVE-1415 > Project: Hive > Issue Type: Improvement > Components: Clients >Affects Versions: 0.5.0 >Reporter: John Sichi >Assignee: Edward Capriolo > Fix For: 0.7.0 > > Attachments: hive-1415-1-patch.txt, hive-1415-2-patch.txt > > > Suggestion in HIVE-1405 was "source", e.g. > source somescript.sql; -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-842) Authentication Infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969866#action_12969866 ] John Sichi commented on HIVE-842: - Click "Submit Patch" to get it into the review queue. > Authentication Infrastructure for Hive > -- > > Key: HIVE-842 > URL: https://issues.apache.org/jira/browse/HIVE-842 > Project: Hive > Issue Type: New Feature > Components: Security, Server Infrastructure >Reporter: Edward Capriolo >Assignee: Todd Lipcon > Attachments: hive-842.txt, hive-842_2.patch, HiveSecurityThoughts.pdf > > > This issue deals with the authentication (user name,password) infrastructure. > Not the authorization components that specify what a user should be able to > do. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-78) Authorization infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-78?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969890#action_12969890 ] John Sichi commented on HIVE-78: Taking a first look at this one; I will have a number of suggestions on naming/structure for thrift and JDO. I think you accidentally omitted the org.apache.hadoop.hive.ql.security.authorization package since I see references to it but no code. > Authorization infrastructure for Hive > - > > Key: HIVE-78 > URL: https://issues.apache.org/jira/browse/HIVE-78 > Project: Hive > Issue Type: New Feature > Components: Metastore, Query Processor, Server Infrastructure >Reporter: Ashish Thusoo >Assignee: He Yongqiang > Attachments: createuser-v1.patch, hive-78-metadata-v1.patch, > hive-78-syntax-v1.patch, HIVE-78.1.nothrift.patch, HIVE-78.1.thrift.patch, > HIVE-78.2.nothrift.patch, HIVE-78.2.thrift.patch, HIVE-78.4.complete.patch, > HIVE-78.4.no_thrift.patch, HIVE-78.5.complete.patch, > HIVE-78.5.no_thrift.patch, hive-78.diff > > > Allow hive to integrate with existing user repositories for authentication > and authorization infromation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1731) Improve miscellaneous error messages
[ https://issues.apache.org/jira/browse/HIVE-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1731: - Comment: was deleted (was: hive> select actiontype, sourcetype, destinationtype, get_json_object(payloaddata, '$.product'), regexp_extract(fullurl, '^.*?php') from nectar_contentaction_merged_partitioned where ds = '2010-11-30' and payloadtype = 'comment' and get_json_object(payloaddata, '$.product') = 'groups' limit 50; Total MapReduce jobs = 1 Launching Job 1 out of 1 ... Ended Job = job_20101201_220674 with errors FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask The actual error was something to do with my regexp_extract syntax: index out of bounds error, no group 1 I didn't specify a group in the regexp, and when I made the adjustment everything worked as expected. It seems like this should be caught earlier, or at least the error trace for the task with the most failures (or a random task with max_failures if there's more than one) should be dumped to the CLI. This one might belong in a different task, since it's a much harder problem to deal with the error traces for badly specified functions (or, worse, custom scripts). However, in general "return code 2" is not useful. It only tells me I need to do a bunch of work to find out what's broken, and the only reason I know that is because I've been using Hive for a long time.) > Improve miscellaneous error messages > > > Key: HIVE-1731 > URL: https://issues.apache.org/jira/browse/HIVE-1731 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: John Sichi > Fix For: 0.7.0 > > > This is a place for accumulating error message improvements so that we can > update a bunch in batch. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-78) Authorization infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-78?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12970002#action_12970002 ] John Sichi commented on HIVE-78: HIVE-78.6.no_thrift.patch has the thrift-generated code. > Authorization infrastructure for Hive > - > > Key: HIVE-78 > URL: https://issues.apache.org/jira/browse/HIVE-78 > Project: Hive > Issue Type: New Feature > Components: Metastore, Query Processor, Server Infrastructure >Reporter: Ashish Thusoo >Assignee: He Yongqiang > Attachments: createuser-v1.patch, hive-78-metadata-v1.patch, > hive-78-syntax-v1.patch, HIVE-78.1.nothrift.patch, HIVE-78.1.thrift.patch, > HIVE-78.2.nothrift.patch, HIVE-78.2.thrift.patch, HIVE-78.4.complete.patch, > HIVE-78.4.no_thrift.patch, HIVE-78.5.complete.patch, > HIVE-78.5.no_thrift.patch, HIVE-78.6.complete.patch, > HIVE-78.6.no_thrift.patch, hive-78.diff > > > Allow hive to integrate with existing user repositories for authentication > and authorization infromation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-78) Authorization infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-78?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-78: --- Comment: was deleted (was: HIVE-78.6.no_thrift.patch has the thrift-generated code. ) > Authorization infrastructure for Hive > - > > Key: HIVE-78 > URL: https://issues.apache.org/jira/browse/HIVE-78 > Project: Hive > Issue Type: New Feature > Components: Metastore, Query Processor, Server Infrastructure >Reporter: Ashish Thusoo >Assignee: He Yongqiang > Attachments: createuser-v1.patch, hive-78-metadata-v1.patch, > hive-78-syntax-v1.patch, HIVE-78.1.nothrift.patch, HIVE-78.1.thrift.patch, > HIVE-78.2.nothrift.patch, HIVE-78.2.thrift.patch, HIVE-78.4.complete.patch, > HIVE-78.4.no_thrift.patch, HIVE-78.5.complete.patch, > HIVE-78.5.no_thrift.patch, HIVE-78.6.complete.patch, > HIVE-78.6.no_thrift.patch, hive-78.diff > > > Allow hive to integrate with existing user repositories for authentication > and authorization infromation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1694) Accelerate query execution using indexes
[ https://issues.apache.org/jira/browse/HIVE-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12970058#action_12970058 ] John Sichi commented on HIVE-1694: -- I talked to Namit, and he thinks there should be no relevant dependencies on the QB once we start on optimization, so letting it get out of sync with the operator DAG may not be an issue. (I scanned the code in optimizer, and it seems like a few dependencies have crept in, but only for special cases like ANALYZE.) For issue #1, you are proposing what I'll call the "internal SQL" approach, which is to construct an internal SQL expression (either in string or ASTNode form) and then partially analyze that (via SemanticAnalyzer), producing an operator DAG to be spliced into the main one. For this approach, we would need to figure out how to make the relevant phases of SemanticAnalyzer modularized and invocable. Alternately, the "direct construction" approach would be to attempt to construct the new operator subgraph directly via custom code targeted to the specific patterns you generate, and then splice that in. I'm not sure which approach is better; Namit, any opinions? The internal SQL approach definitely seems the most appropriate for the WHERE clause work being done by the Harvey Mudd team, since it produces a self-contained job to be run to produce the temp table containing the filtered block list. But for GROUP BY, the direct construction approach may be cleaner. For issue #2, it seems like this would happen automatically for the internal SQL approach (but this could also pollute the SemanticAnalyzer state to some extent). The direct construction approach is the opposite: it avoids polluting SemanticAnalyzer, but still might require modularizing some SemanticAnalyzer calls, e.g. for generating and registering the necessary aliases for index tables. Regarding issue #3, that's already true for other optimizations such as projection pushdown (ColumnPruner), which modifies operator row schemas/resolvers; see for example ColumnPrunerProcFactory.pruneJoinOperator. So there shouldn't be anything new here. Regarding the need to run your transformation first, it would be best to avoid this since a more advanced optimizer may want freedom in reordering transformations. So instead of relying on information from the QB, analyze the relevant operator subgraph to decide whether your transformation is applicable. This is the approach we expect to require for cost-based optimization. Also, note that from a lineage perspective, it makes more sense for lineage to be derived prior to index transformation rather than subsequently. Someone examining the lineage associated with an ETL job would typically be more interested in the logical source table from which it pulls (rather than from a physical index). > Accelerate query execution using indexes > > > Key: HIVE-1694 > URL: https://issues.apache.org/jira/browse/HIVE-1694 > Project: Hive > Issue Type: New Feature > Components: Indexing, Query Processor >Affects Versions: 0.7.0 >Reporter: Nikhil Deshpande >Assignee: Nikhil Deshpande > Attachments: demo_q1.hql, demo_q2.hql, HIVE-1694_2010-10-28.diff > > > The index building patch (Hive-417) is checked into trunk, this JIRA issue > tracks supporting indexes in Hive compiler & execution engine for SELECT > queries. > This is in ref. to John's comment at > https://issues.apache.org/jira/browse/HIVE-417?focusedCommentId=12884869&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12884869 > on creating separate JIRA issue for tracking index usage in optimizer & query > execution. > The aim of this effort is to use indexes to accelerate query execution (for > certain class of queries). E.g. > - Filters and range scans (already being worked on by He Yongqiang as part of > HIVE-417?) > - Joins (index based joins) > - Group By, Order By and other misc cases > The proposal is multi-step: > 1. Building index based operators, compiler and execution engine changes > 2. Optimizer enhancements (e.g. cost-based optimizer to compare and choose > between index scans, full table scans etc.) > This JIRA initially focuses on the first step. This JIRA is expected to hold > the information about index based plans & operator implementations for above > mentioned cases. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-78) Authorization infrastructure for Hive
[
https://issues.apache.org/jira/browse/HIVE-78?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12970662#action_12970662
]
John Sichi commented on HIVE-78:
First batch of review comments.
JDO:
* Do we want roles to be contained by databases? Let's discuss this
at next design review.
* Instead of two separate flags (IS_ROLE/IS_GROUP) should we instead use
an enum for principal type { USER, GROUP, ROLE }?
* Naming suggestions (if accepted, propagate to Thrift API also):
** SECURITYROLE -> ROLES
** SECURITYROLEMAP -> ROLE_MAP
** SECURITYUSER -> GLOBAL_PRIVS
** SECURITYDB -> DB_PRIVS
** SECURITYTBLPART -> TBLPART_PRIVS
** SECURITYCOLUMN -> COL_PRIVS
* VARCHAR precision for "privileges" fields should be 4000
* Since we're going to need to record GRANT OPTION eventually, maybe
we should add it now so that we don't have to ALTER TABLE later?
Thrift API:
* Avoid embedding objects inside of other objects except where
necessary. For example, in the definition of struct Role, use
dbName instead of a Database object (assuming we keep roles as
contained by databases). Likewise, in PrivilegeBag, the map keys
should be identifiers, not objects. This applies to quite a few of
the new structs.
* Can we reduce the number of new structs and API calls by
consolidating different object types? For example, for the
get_XXX_privilege_set calls, just have one, and take object
type+identifier.
* Add comments for all new methods.
Config:
* Why is hive.exec.security used for some config params instead of
hive.security? Also, those parameter names should make it clear
that they are default grants. Also, do we really need owner grants
(don't owners automatically have full privileges implicitly)?
* Looks like hive.variable.substitute crept in from some other patch.
* Comments for plugin-loading parameters should make it explicit
exactly which interface they are supposed to implement.
* Comment for role grants says "to some groups" instead.
Pluggable Interfaces:
* I don't think we need the factory classes; just add new methods to
HiveUtils (and follow the classloading pattern used there)
* Rename AuthorizationProvider to HiveAuthorizationProvider
and make it extend Configurable
* Rename AuthorizationProviderManager to AbstractAuthorizationProvider
* All outside references should be to the interface (HiveAuthorizationProvider)
not the abstract class.
* Rename Authenticator to HiveAuthenticationProvider and make it
extend Configurable
* Javadoc?
Typos:
* principla
* Authrization
* GrantInfor
* privielges
* "Table is partitioned, but partition spec found"
* DummpyAuthenticator
* detroy
* wheenve
Implementation:
* why does doAuthorization return a boolean when it just throws
anyway?
* more coming...
> Authorization infrastructure for Hive
> -
>
> Key: HIVE-78
> URL: https://issues.apache.org/jira/browse/HIVE-78
> Project: Hive
> Issue Type: New Feature
> Components: Metastore, Query Processor, Server Infrastructure
>Reporter: Ashish Thusoo
>Assignee: He Yongqiang
> Attachments: createuser-v1.patch, hive-78-metadata-v1.patch,
> hive-78-syntax-v1.patch, HIVE-78.1.nothrift.patch, HIVE-78.1.thrift.patch,
> HIVE-78.2.nothrift.patch, HIVE-78.2.thrift.patch, HIVE-78.4.complete.patch,
> HIVE-78.4.no_thrift.patch, HIVE-78.5.complete.patch,
> HIVE-78.5.no_thrift.patch, HIVE-78.6.complete.patch,
> HIVE-78.6.no_thrift.patch, hive-78.diff
>
>
> Allow hive to integrate with existing user repositories for authentication
> and authorization infromation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-78) Authorization infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-78?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12971161#action_12971161 ] John Sichi commented on HIVE-78: Some more from me: * There's a bug when attempting to grant multiple privileges at once; only one of them is getting granted (what I showed you in CLI) * Multiple grants from the same grantor to the same grantee should not result in duplicates (verify against Oracle), and we should collapse everything into one row no matter whether the grants were made at the same or different times (sort privilege names for determinism) * revokeAllPrivileges should revoke role grants as well * Role cycle is not being prevented * try/finally around transactions in ObjectStore should be used consistently (I know there are some cases which were already missing them, but we shouldn't make it worse) * Don't use printStackTrace * show [role] grant role unknown should fail (even though we have to tolerate unknown for user/group since we don't have a table for those) Some additional points noted at code review session: * Need many many negative tests * Provide a way to make partitions inherit from table (and make it the default) * Define a UNIQUE key for the priv tables in JDO * GRANT should mark WriteEntity for replication etc More Typos: * candicate * anaylze I have some more code-level comments but not all of them may be relevant after the issues above have been resolved, so I'll do another pass after the next patch. > Authorization infrastructure for Hive > - > > Key: HIVE-78 > URL: https://issues.apache.org/jira/browse/HIVE-78 > Project: Hive > Issue Type: New Feature > Components: Metastore, Query Processor, Server Infrastructure >Reporter: Ashish Thusoo >Assignee: He Yongqiang > Attachments: createuser-v1.patch, hive-78-metadata-v1.patch, > hive-78-syntax-v1.patch, HIVE-78.1.nothrift.patch, HIVE-78.1.thrift.patch, > HIVE-78.2.nothrift.patch, HIVE-78.2.thrift.patch, HIVE-78.4.complete.patch, > HIVE-78.4.no_thrift.patch, HIVE-78.5.complete.patch, > HIVE-78.5.no_thrift.patch, HIVE-78.6.complete.patch, > HIVE-78.6.no_thrift.patch, hive-78.diff > > > Allow hive to integrate with existing user repositories for authentication > and authorization infromation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-78) Authorization infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-78?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12971442#action_12971442 ] John Sichi commented on HIVE-78: We can't take the size of a patch as a justification for checking in code which doesn't pass review, especially for things like JDO and Thrift API's which are going to be there forever. I discussed it with Namit and his suggestion was to break it down into smaller patches to be committed in sequence so that we can divide-and-conquer the review process. For future projects, it would be great if we can do the same for the design process itself so that the coding doesn't get too far ahead of the design (which is how we end up with giant patches). The items below are OK for followups 2. revokeAllPrivileges should revoke role grants as well 3. Role cycle is not being prevented 6. GRANT should mark WriteEntity for replication etc For this one, we should at least work out the metastore model as part of the JDO changes: 7. Provide a way to make partitions inherit from table (and make it the default) The rest need to be addressed up front as part of the relevant patches. Separately, maybe using git for branch+merge would help make development of a feature of this size more manageable? (If you're not already.) > Authorization infrastructure for Hive > - > > Key: HIVE-78 > URL: https://issues.apache.org/jira/browse/HIVE-78 > Project: Hive > Issue Type: New Feature > Components: Metastore, Query Processor, Server Infrastructure >Reporter: Ashish Thusoo >Assignee: He Yongqiang > Attachments: createuser-v1.patch, hive-78-metadata-v1.patch, > hive-78-syntax-v1.patch, HIVE-78.1.nothrift.patch, HIVE-78.1.thrift.patch, > HIVE-78.2.nothrift.patch, HIVE-78.2.thrift.patch, HIVE-78.4.complete.patch, > HIVE-78.4.no_thrift.patch, HIVE-78.5.complete.patch, > HIVE-78.5.no_thrift.patch, HIVE-78.6.complete.patch, > HIVE-78.6.no_thrift.patch, HIVE-78.7.no_thrift.patch, HIVE-78.7.patch, > hive-78.diff > > > Allow hive to integrate with existing user repositories for authentication > and authorization infromation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-78) Authorization infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-78?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12971555#action_12971555 ] John Sichi commented on HIVE-78: Regarding pass-by-name vs pass-by-value for object references in the Thrift API, take a look at how drop table works. We already fetch the table descriptor in DDLTask (so that we can include its info in the posthook). But then, when we drop the table, we pass dbname+tblname (not the actual table object). So I don't see the need to invent a new pattern here. For dealing with compound names, it's fine to define a new struct ObjectReference with object type plus various optional components, then pass that. (In the future, we could also decide to hide an ID in there for the lookup-skipping optimization you mention if it turns out to be warranted.) > Authorization infrastructure for Hive > - > > Key: HIVE-78 > URL: https://issues.apache.org/jira/browse/HIVE-78 > Project: Hive > Issue Type: New Feature > Components: Metastore, Query Processor, Server Infrastructure >Reporter: Ashish Thusoo >Assignee: He Yongqiang > Attachments: createuser-v1.patch, hive-78-metadata-v1.patch, > hive-78-syntax-v1.patch, HIVE-78.1.nothrift.patch, HIVE-78.1.thrift.patch, > HIVE-78.2.nothrift.patch, HIVE-78.2.thrift.patch, HIVE-78.4.complete.patch, > HIVE-78.4.no_thrift.patch, HIVE-78.5.complete.patch, > HIVE-78.5.no_thrift.patch, HIVE-78.6.complete.patch, > HIVE-78.6.no_thrift.patch, HIVE-78.7.no_thrift.patch, HIVE-78.7.patch, > hive-78.diff > > > Allow hive to integrate with existing user repositories for authentication > and authorization infromation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1853) downgrade JDO version
[ https://issues.apache.org/jira/browse/HIVE-1853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12972191#action_12972191 ] John Sichi commented on HIVE-1853: -- This user report has similar symptoms: http://mail-archives.apache.org/mod_mbox/hive-user/201011.mbox/%3caanlktik6m1juxvx0lerre-cnn5qnfwi6_4ay0pq6z...@mail.gmail.com%3e > downgrade JDO version > - > > Key: HIVE-1853 > URL: https://issues.apache.org/jira/browse/HIVE-1853 > Project: Hive > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Namit Jain >Assignee: Paul Yang > Attachments: HIVE-1853.1.patch > > > After HIVE-1609, we are seeing some table not found errors intermittently. > We have a test case where 5 processes are concurrently issueing the same > query - > explain extended insert .. select from > and once in a while, we get a error not found - > When we revert back the JDO version, the error is gone. > We can investigate later to find the JDO bug, but for now this is a > show-stopper for facebook, and needs > to be reverted back immediately. > This also means, that the filters will not be pushed to mysql. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1542) DROP TABLE should raise an error when does not exist
[ https://issues.apache.org/jira/browse/HIVE-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12972269#action_12972269 ] John Sichi commented on HIVE-1542: -- Hi Marcel :) > DROP TABLE should raise an error when does not exist > > > Key: HIVE-1542 > URL: https://issues.apache.org/jira/browse/HIVE-1542 > Project: Hive > Issue Type: Bug > Components: Query Processor >Reporter: Carl Steinbach >Assignee: Marcel Kornacker > > See http://dev.mysql.com/doc/refman/5.1/en/drop-table.html -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-78) Authorization infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-78?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12972626#action_12972626 ] John Sichi commented on HIVE-78: @Alan: we discussed this in depth at the last Hive contributor meeting: http://wiki.apache.org/hadoop/Hive/Development/ContributorsMeetings/HiveContributorsMinutes101025 Let's talk to Carl about scheduling the next one and make sure we find a timeslot where you can make it. > Authorization infrastructure for Hive > - > > Key: HIVE-78 > URL: https://issues.apache.org/jira/browse/HIVE-78 > Project: Hive > Issue Type: New Feature > Components: Metastore, Query Processor, Server Infrastructure >Reporter: Ashish Thusoo >Assignee: He Yongqiang > Attachments: createuser-v1.patch, hive-78-metadata-v1.patch, > hive-78-syntax-v1.patch, HIVE-78.1.nothrift.patch, HIVE-78.1.thrift.patch, > HIVE-78.2.nothrift.patch, HIVE-78.2.thrift.patch, HIVE-78.4.complete.patch, > HIVE-78.4.no_thrift.patch, HIVE-78.5.complete.patch, > HIVE-78.5.no_thrift.patch, HIVE-78.6.complete.patch, > HIVE-78.6.no_thrift.patch, HIVE-78.7.no_thrift.patch, HIVE-78.7.patch, > HIVE-78.9.no_thrift.patch, HIVE-78.9.patch, hive-78.diff > > > Allow hive to integrate with existing user repositories for authentication > and authorization infromation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-78) Authorization infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-78?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12972643#action_12972643 ] John Sichi commented on HIVE-78: @Yongqiang: New review comments in https://reviews.apache.org/r/183/ The patch is applying cleanly for me now (I must have forgotten to svn up), so I'll do some testing later. > Authorization infrastructure for Hive > - > > Key: HIVE-78 > URL: https://issues.apache.org/jira/browse/HIVE-78 > Project: Hive > Issue Type: New Feature > Components: Metastore, Query Processor, Server Infrastructure >Reporter: Ashish Thusoo >Assignee: He Yongqiang > Attachments: createuser-v1.patch, hive-78-metadata-v1.patch, > hive-78-syntax-v1.patch, HIVE-78.1.nothrift.patch, HIVE-78.1.thrift.patch, > HIVE-78.2.nothrift.patch, HIVE-78.2.thrift.patch, HIVE-78.4.complete.patch, > HIVE-78.4.no_thrift.patch, HIVE-78.5.complete.patch, > HIVE-78.5.no_thrift.patch, HIVE-78.6.complete.patch, > HIVE-78.6.no_thrift.patch, HIVE-78.7.no_thrift.patch, HIVE-78.7.patch, > HIVE-78.9.no_thrift.patch, HIVE-78.9.patch, hive-78.diff > > > Allow hive to integrate with existing user repositories for authentication > and authorization infromation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-78) Authorization infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-78?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12972644#action_12972644 ] John Sichi commented on HIVE-78: I added one comment about referring to "grantee" instead of "principal" in some of the API's, but I did not do it consistently. I think this would be clearer across thrift/JDO to distinguish the grantor from the grantee in all cases, but if you want to leave it as is, just ignore that comment. > Authorization infrastructure for Hive > - > > Key: HIVE-78 > URL: https://issues.apache.org/jira/browse/HIVE-78 > Project: Hive > Issue Type: New Feature > Components: Metastore, Query Processor, Server Infrastructure >Reporter: Ashish Thusoo >Assignee: He Yongqiang > Attachments: createuser-v1.patch, hive-78-metadata-v1.patch, > hive-78-syntax-v1.patch, HIVE-78.1.nothrift.patch, HIVE-78.1.thrift.patch, > HIVE-78.2.nothrift.patch, HIVE-78.2.thrift.patch, HIVE-78.4.complete.patch, > HIVE-78.4.no_thrift.patch, HIVE-78.5.complete.patch, > HIVE-78.5.no_thrift.patch, HIVE-78.6.complete.patch, > HIVE-78.6.no_thrift.patch, HIVE-78.7.no_thrift.patch, HIVE-78.7.patch, > HIVE-78.9.no_thrift.patch, HIVE-78.9.patch, hive-78.diff > > > Allow hive to integrate with existing user repositories for authentication > and authorization infromation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1853) downgrade JDO version
[ https://issues.apache.org/jira/browse/HIVE-1853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12972669#action_12972669 ] John Sichi commented on HIVE-1853: -- Yeah, Hudson agrees with Ashutosh: https://hudson.apache.org/hudson/job/Hive-trunk-h0.20/445/testReport/junit/org.apache.hadoop.hive.metastore/TestEmbeddedHiveMetaStore/testPartitionFilter/ > downgrade JDO version > - > > Key: HIVE-1853 > URL: https://issues.apache.org/jira/browse/HIVE-1853 > Project: Hive > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Namit Jain >Assignee: Paul Yang > Fix For: 0.7.0 > > Attachments: HIVE-1853.1.patch, HIVE-1853.2.patch > > > After HIVE-1609, we are seeing some table not found errors intermittently. > We have a test case where 5 processes are concurrently issueing the same > query - > explain extended insert .. select from > and once in a while, we get a error not found - > When we revert back the JDO version, the error is gone. > We can investigate later to find the JDO bug, but for now this is a > show-stopper for facebook, and needs > to be reverted back immediately. > This also means, that the filters will not be pushed to mysql. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-78) Authorization infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-78?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12973388#action_12973388 ] John Sichi commented on HIVE-78: A few more comments on patch 10 in https://reviews.apache.org/r/187/ > Authorization infrastructure for Hive > - > > Key: HIVE-78 > URL: https://issues.apache.org/jira/browse/HIVE-78 > Project: Hive > Issue Type: New Feature > Components: Metastore, Query Processor, Server Infrastructure >Reporter: Ashish Thusoo >Assignee: He Yongqiang > Attachments: createuser-v1.patch, hive-78-metadata-v1.patch, > hive-78-syntax-v1.patch, HIVE-78.1.nothrift.patch, HIVE-78.1.thrift.patch, > HIVE-78.10.no_thrift.patch, HIVE-78.2.nothrift.patch, HIVE-78.2.thrift.patch, > HIVE-78.4.complete.patch, HIVE-78.4.no_thrift.patch, > HIVE-78.5.complete.patch, HIVE-78.5.no_thrift.patch, > HIVE-78.6.complete.patch, HIVE-78.6.no_thrift.patch, > HIVE-78.7.no_thrift.patch, HIVE-78.7.patch, HIVE-78.9.no_thrift.patch, > HIVE-78.9.patch, hive-78.diff > > > Allow hive to integrate with existing user repositories for authentication > and authorization infromation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1857) mixed case tablename on lefthand side of LATERAL VIEW results in query failing with confusing error message
mixed case tablename on lefthand side of LATERAL VIEW results in query failing with confusing error message --- Key: HIVE-1857 URL: https://issues.apache.org/jira/browse/HIVE-1857 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi Assignee: John Sichi Fix For: 0.7.0 Attachments: HIVE-1857.1.patch For the modified query below in lateral_view.q, the exception "org.apache.hadoop.hive.ql.parse.SemanticException: line 3:7 Invalid Table Alias or Column Reference myCol" is thrown. The query should succeed. SELECT myCol from tmp_PYANG_lv LATERAL VIEW explode(array(1,2,3)) myTab as myCol limit 3; -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1857) mixed case tablename on lefthand side of LATERAL VIEW results in query failing with confusing error message
[ https://issues.apache.org/jira/browse/HIVE-1857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1857: - Attachment: HIVE-1857.1.patch > mixed case tablename on lefthand side of LATERAL VIEW results in query > failing with confusing error message > --- > > Key: HIVE-1857 > URL: https://issues.apache.org/jira/browse/HIVE-1857 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.6.0 >Reporter: John Sichi >Assignee: John Sichi > Fix For: 0.7.0 > > Attachments: HIVE-1857.1.patch > > > For the modified query below in lateral_view.q, the exception > "org.apache.hadoop.hive.ql.parse.SemanticException: line 3:7 Invalid Table > Alias or Column Reference myCol" is thrown. The query should succeed. > SELECT myCol from tmp_PYANG_lv LATERAL VIEW explode(array(1,2,3)) myTab as > myCol limit 3; -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1857) mixed case tablename on lefthand side of LATERAL VIEW results in query failing with confusing error message
[ https://issues.apache.org/jira/browse/HIVE-1857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1857: - Status: Patch Available (was: Open) Fix is in HIVE-1857.1.patch. > mixed case tablename on lefthand side of LATERAL VIEW results in query > failing with confusing error message > --- > > Key: HIVE-1857 > URL: https://issues.apache.org/jira/browse/HIVE-1857 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.6.0 >Reporter: John Sichi >Assignee: John Sichi > Fix For: 0.7.0 > > Attachments: HIVE-1857.1.patch > > > For the modified query below in lateral_view.q, the exception > "org.apache.hadoop.hive.ql.parse.SemanticException: line 3:7 Invalid Table > Alias or Column Reference myCol" is thrown. The query should succeed. > SELECT myCol from tmp_PYANG_lv LATERAL VIEW explode(array(1,2,3)) myTab as > myCol limit 3; -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1856) Implement DROP TABLE/VIEW ... IF EXISTS
[ https://issues.apache.org/jira/browse/HIVE-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1856: - Status: Open (was: Patch Available) Review comments in https://reviews.apache.org/r/188/ Note that there are also DROP commands for databases, indexes, partitions, and temporary functions. This config property should cover those too, so we should either address those as well in this patch, or open a followup JIRA. > Implement DROP TABLE/VIEW ... IF EXISTS > > > Key: HIVE-1856 > URL: https://issues.apache.org/jira/browse/HIVE-1856 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.7.0 >Reporter: Marcel Kornacker >Assignee: Marcel Kornacker > Attachments: hive-1856.patch > > > This issue combines issues HIVE-1550/1165/1542/1551: > - augment DROP TABLE/VIEW with IF EXISTS > - signal an error if the table/view doesn't exist and IF EXISTS wasn't > specified > - introduce a flag in the configuration that allows you to turn off the new > behavior -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1326) RowContainer uses hard-coded '/tmp/' path for temporary files
[ https://issues.apache.org/jira/browse/HIVE-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi reassigned HIVE-1326: Assignee: Michael Klatt > RowContainer uses hard-coded '/tmp/' path for temporary files > - > > Key: HIVE-1326 > URL: https://issues.apache.org/jira/browse/HIVE-1326 > Project: Hive > Issue Type: Bug > Components: Query Processor > Environment: Hadoop 0.19.2 with Hive trunk. We're using FreeBSD 7.0, > but that doesn't seem relevant. >Reporter: Michael Klatt >Assignee: Michael Klatt > Fix For: 0.6.0 > > Attachments: rowcontainer.patch, rowcontainer_v2.patch > > > In our production hadoop environment, the "/tmp/" is actually pretty small, > and we encountered a problem when a query used the RowContainer class and > filled up the /tmp/ partition. I tracked down the cause to the RowContainer > class putting temporary files in the '/tmp/' path instead of using the > configured Hadoop temporary path. I've attached a patch to fix this. > Here's the traceback: > 2010-04-25 12:05:05,120 INFO > org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created > temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp > 2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 1000 > rows: used memory = 385520312 > 2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 1100 > rows: used memory = 341780472 > 2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 1200 > rows: used memory = 301446768 > 2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 1300 > rows: used memory = 399208768 > 2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 1400 > rows: used memory = 364507216 > 2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 1500 > rows: used memory = 332907280 > 2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 1600 > rows: used memory = 298774096 > 2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 1700 > rows: used memory = 396505408 > 2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 1800 > rows: used memory = 362477288 > 2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 1900 > rows: used memory = 327229744 > 2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 2000 > rows: used memory = 296051904 > 2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: > java.io.IOException: No space left on device > at > org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346) > at > org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150) > at > org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132) > at > org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121) > at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112) > at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at > org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013) > at > org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977) > at > org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70) > at > org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343) > at > org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:163) > at > org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:118) > at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:456) > at > org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:244) > at org.apache.h
[jira] Commented: (HIVE-1856) Implement DROP TABLE/VIEW ... IF EXISTS
[ https://issues.apache.org/jira/browse/HIVE-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12973809#action_12973809 ] John Sichi commented on HIVE-1856: -- +1. Will commit when tests pass. Note for next time: for patch updates, our convention is to number them like HIVE-1856.1.patch, HIVE-1856.2.patch, etc. And then click the "Submit Patch" button again when a new one is uploaded; this makes sure it gets back into the review queue. > Implement DROP TABLE/VIEW ... IF EXISTS > > > Key: HIVE-1856 > URL: https://issues.apache.org/jira/browse/HIVE-1856 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.7.0 >Reporter: Marcel Kornacker >Assignee: Marcel Kornacker > Attachments: hive-1856.patch, hive-1856.patch > > > This issue combines issues HIVE-1550/1165/1542/1551: > - augment DROP TABLE/VIEW with IF EXISTS > - signal an error if the table/view doesn't exist and IF EXISTS wasn't > specified > - introduce a flag in the configuration that allows you to turn off the new > behavior -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-1856) Implement DROP TABLE/VIEW ... IF EXISTS
[ https://issues.apache.org/jira/browse/HIVE-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi resolved HIVE-1856. -- Resolution: Fixed Fix Version/s: 0.7.0 Release Note: Backwards-compatible DROP behavior is controlled via new configuration parameter hive.exec.drop.ignorenonexistent Hadoop Flags: [Reviewed] Committed. Thanks Marcel! > Implement DROP TABLE/VIEW ... IF EXISTS > > > Key: HIVE-1856 > URL: https://issues.apache.org/jira/browse/HIVE-1856 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.7.0 >Reporter: Marcel Kornacker >Assignee: Marcel Kornacker > Fix For: 0.7.0 > > Attachments: hive-1856.patch, hive-1856.patch > > > This issue combines issues HIVE-1550/1165/1542/1551: > - augment DROP TABLE/VIEW with IF EXISTS > - signal an error if the table/view doesn't exist and IF EXISTS wasn't > specified > - introduce a flag in the configuration that allows you to turn off the new > behavior -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1611) Add alternative search-provider to Hive site
[ https://issues.apache.org/jira/browse/HIVE-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1611: - Status: Open (was: Patch Available) > Add alternative search-provider to Hive site > > > Key: HIVE-1611 > URL: https://issues.apache.org/jira/browse/HIVE-1611 > Project: Hive > Issue Type: Improvement >Reporter: Alex Baranau >Assignee: Edward Capriolo >Priority: Minor > Attachments: HIVE-1611.patch > > > Use search-hadoop.com service to make available search in Hive sources, MLs, > wiki, etc. > This was initially proposed on user mailing list. The search service was > already added in site's skin (common for all Hadoop related projects) before > so this issue is about enabling it for Hive. The ultimate goal is to use it > at all Hadoop's sub-projects' sites. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1262) Add security/checksum UDFs sha,crc32,md5,aes_encrypt, and aes_decrypt
[ https://issues.apache.org/jira/browse/HIVE-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1262: - Status: Open (was: Patch Available) > Add security/checksum UDFs sha,crc32,md5,aes_encrypt, and aes_decrypt > - > > Key: HIVE-1262 > URL: https://issues.apache.org/jira/browse/HIVE-1262 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.6.0 >Reporter: Edward Capriolo >Assignee: Edward Capriolo > Fix For: 0.7.0 > > Attachments: hive-1262-1.patch.txt > > > Add security/checksum UDFs sha,crc32,md5,aes_encrypt, and aes_decrypt -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1262) Add security/checksum UDFs sha,crc32,md5,aes_encrypt, and aes_decrypt
[ https://issues.apache.org/jira/browse/HIVE-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12974795#action_12974795 ] John Sichi commented on HIVE-1262: -- Review comments added in https://reviews.apache.org/r/192/ > Add security/checksum UDFs sha,crc32,md5,aes_encrypt, and aes_decrypt > - > > Key: HIVE-1262 > URL: https://issues.apache.org/jira/browse/HIVE-1262 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.6.0 >Reporter: Edward Capriolo >Assignee: Edward Capriolo > Fix For: 0.7.0 > > Attachments: hive-1262-1.patch.txt > > > Add security/checksum UDFs sha,crc32,md5,aes_encrypt, and aes_decrypt -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1858) Implement DROP {PARTITION, INDEX, TEMPORARY FUNCTION} IF EXISTS
[
https://issues.apache.org/jira/browse/HIVE-1858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12975308#action_12975308
]
John Sichi commented on HIVE-1858:
--
+1. Will commit if tests pass.
The wiki is going to need to be updated for this plus HIVE-1856. Be sure to
include a note that the new syntax + conf is only available starting from Hive
0.7.
> Implement DROP {PARTITION, INDEX, TEMPORARY FUNCTION} IF EXISTS
> ---
>
> Key: HIVE-1858
> URL: https://issues.apache.org/jira/browse/HIVE-1858
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 0.7.0
>Reporter: Marcel Kornacker
>Assignee: Marcel Kornacker
> Attachments: hive-1858.1.patch
>
>
> Extend HIVE-1856 to support IF EXISTS for {DROP TABLE, VIEW} and ALTER TABLE
> DROP PARTITION
> - signal an error if the to-be-dropped entity doesn't exist and IF EXISTS
> isn't specified
> - this behavior can be disabled by setting hive.exec.drop.ignorenonexistent
> to true
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HIVE-1858) Implement DROP {PARTITION, INDEX, TEMPORARY FUNCTION} IF EXISTS
[
https://issues.apache.org/jira/browse/HIVE-1858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12975377#action_12975377
]
John Sichi edited comment on HIVE-1858 at 12/27/10 6:28 PM:
Committed. Thanks Marcel!
was (Author: jvs):
Commited. Thanks Marcel!
> Implement DROP {PARTITION, INDEX, TEMPORARY FUNCTION} IF EXISTS
> ---
>
> Key: HIVE-1858
> URL: https://issues.apache.org/jira/browse/HIVE-1858
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 0.7.0
>Reporter: Marcel Kornacker
>Assignee: Marcel Kornacker
> Fix For: 0.7.0
>
> Attachments: hive-1858.1.patch
>
>
> Extend HIVE-1856 to support IF EXISTS for {DROP TABLE, VIEW} and ALTER TABLE
> DROP PARTITION
> - signal an error if the to-be-dropped entity doesn't exist and IF EXISTS
> isn't specified
> - this behavior can be disabled by setting hive.exec.drop.ignorenonexistent
> to true
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1858) Implement DROP {PARTITION, INDEX, TEMPORARY FUNCTION} IF EXISTS
[
https://issues.apache.org/jira/browse/HIVE-1858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sichi updated HIVE-1858:
-
Resolution: Fixed
Fix Version/s: 0.7.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Commited. Thanks Marcel!
> Implement DROP {PARTITION, INDEX, TEMPORARY FUNCTION} IF EXISTS
> ---
>
> Key: HIVE-1858
> URL: https://issues.apache.org/jira/browse/HIVE-1858
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 0.7.0
>Reporter: Marcel Kornacker
>Assignee: Marcel Kornacker
> Fix For: 0.7.0
>
> Attachments: hive-1858.1.patch
>
>
> Extend HIVE-1856 to support IF EXISTS for {DROP TABLE, VIEW} and ALTER TABLE
> DROP PARTITION
> - signal an error if the to-be-dropped entity doesn't exist and IF EXISTS
> isn't specified
> - this behavior can be disabled by setting hive.exec.drop.ignorenonexistent
> to true
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1858) Implement DROP {PARTITION, INDEX, TEMPORARY FUNCTION} IF EXISTS
[
https://issues.apache.org/jira/browse/HIVE-1858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12975684#action_12975684
]
John Sichi commented on HIVE-1858:
--
Looks good.
> Implement DROP {PARTITION, INDEX, TEMPORARY FUNCTION} IF EXISTS
> ---
>
> Key: HIVE-1858
> URL: https://issues.apache.org/jira/browse/HIVE-1858
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 0.7.0
>Reporter: Marcel Kornacker
>Assignee: Marcel Kornacker
> Fix For: 0.7.0
>
> Attachments: hive-1858.1.patch
>
>
> Extend HIVE-1856 to support IF EXISTS for {DROP TABLE, VIEW} and ALTER TABLE
> DROP PARTITION
> - signal an error if the to-be-dropped entity doesn't exist and IF EXISTS
> isn't specified
> - this behavior can be disabled by setting hive.exec.drop.ignorenonexistent
> to true
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1660) Change get_partitions_ps to pass partition filter to database
[ https://issues.apache.org/jira/browse/HIVE-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12976369#action_12976369 ] John Sichi commented on HIVE-1660: -- Namit, HIVE-1660.4.patch contains no reference to hbase_pushdown.q.out. However, your commit did change that file: http://svn.apache.org/viewvc?view=revision&revision=1024341 I am guessing that the commit for HIVE-1638 (close in time) broke this test, and then you updated the log file as part of this commit. > Change get_partitions_ps to pass partition filter to database > - > > Key: HIVE-1660 > URL: https://issues.apache.org/jira/browse/HIVE-1660 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 0.7.0 >Reporter: Ajay Kidave >Assignee: Paul Yang > Fix For: 0.7.0 > > Attachments: HIVE-1660.1.patch, HIVE-1660.2.patch, HIVE-1660.3.patch, > HIVE-1660.4.patch, HIVE-1660_regex.patch > > > Support for doing partition pruning by passing the partition filter to the > database is added by HIVE-1609. Changing get_partitions_ps to use this could > result in performance improvement for tables having large number of > partitions. A listPartitionNamesByFilter API might be required for > implementing this for use from Hive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1874) fix HBase filter pushdown broken by HIVE-1638
fix HBase filter pushdown broken by HIVE-1638 - Key: HIVE-1874 URL: https://issues.apache.org/jira/browse/HIVE-1874 Project: Hive Issue Type: Bug Components: HBase Handler Affects Versions: 0.7.0 Reporter: John Sichi Fix For: 0.7.0 See comments at end of HIVE-1660 for what happened. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1660) Change get_partitions_ps to pass partition filter to database
[ https://issues.apache.org/jira/browse/HIVE-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12976605#action_12976605 ] John Sichi commented on HIVE-1660: -- Amendment is in HIVE-1874. > Change get_partitions_ps to pass partition filter to database > - > > Key: HIVE-1660 > URL: https://issues.apache.org/jira/browse/HIVE-1660 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 0.7.0 >Reporter: Ajay Kidave >Assignee: Paul Yang > Fix For: 0.7.0 > > Attachments: HIVE-1660.1.patch, HIVE-1660.2.patch, HIVE-1660.3.patch, > HIVE-1660.4.patch, HIVE-1660_regex.patch > > > Support for doing partition pruning by passing the partition filter to the > database is added by HIVE-1609. Changing get_partitions_ps to use this could > result in performance improvement for tables having large number of > partitions. A listPartitionNamesByFilter API might be required for > implementing this for use from Hive. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1874) fix HBase filter pushdown broken by HIVE-1638
[ https://issues.apache.org/jira/browse/HIVE-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1874: - Attachment: HIVE-1874.1.patch > fix HBase filter pushdown broken by HIVE-1638 > - > > Key: HIVE-1874 > URL: https://issues.apache.org/jira/browse/HIVE-1874 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Affects Versions: 0.7.0 > Reporter: John Sichi > Fix For: 0.7.0 > > Attachments: HIVE-1874.1.patch > > > See comments at end of HIVE-1660 for what happened. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1874) fix HBase filter pushdown broken by HIVE-1638
[ https://issues.apache.org/jira/browse/HIVE-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi reassigned HIVE-1874: Assignee: John Sichi > fix HBase filter pushdown broken by HIVE-1638 > - > > Key: HIVE-1874 > URL: https://issues.apache.org/jira/browse/HIVE-1874 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Affects Versions: 0.7.0 > Reporter: John Sichi >Assignee: John Sichi > Fix For: 0.7.0 > > Attachments: HIVE-1874.1.patch > > > See comments at end of HIVE-1660 for what happened. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1874) fix HBase filter pushdown broken by HIVE-1638
[ https://issues.apache.org/jira/browse/HIVE-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1874: - Status: Patch Available (was: Open) > fix HBase filter pushdown broken by HIVE-1638 > - > > Key: HIVE-1874 > URL: https://issues.apache.org/jira/browse/HIVE-1874 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Affects Versions: 0.7.0 > Reporter: John Sichi >Assignee: John Sichi > Fix For: 0.7.0 > > Attachments: HIVE-1874.1.patch > > > See comments at end of HIVE-1660 for what happened. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1876) Update Metastore upgrade scripts to handle schema changes introduced in HIVE-1413
[ https://issues.apache.org/jira/browse/HIVE-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12976822#action_12976822 ] John Sichi commented on HIVE-1876: -- It ended up in HIVE-1413 that Siying used the generic TBLPROPERTIES instead of adding a new field to the metastore. So no upgrade script needed. > Update Metastore upgrade scripts to handle schema changes introduced in > HIVE-1413 > - > > Key: HIVE-1876 > URL: https://issues.apache.org/jira/browse/HIVE-1876 > Project: Hive > Issue Type: Task > Components: Metastore >Affects Versions: 0.7.0 >Reporter: Carl Steinbach >Priority: Blocker > Fix For: 0.7.0 > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1694) Accelerate query execution using indexes
[ https://issues.apache.org/jira/browse/HIVE-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12976882#action_12976882 ] John Sichi commented on HIVE-1694: -- Thanks Prajakta. Let us know once you have new code ready to review. HIVE-1644 is going to need the internal SQL support too, so I'd like to make sure that as much as possible is reusable there. > Accelerate query execution using indexes > > > Key: HIVE-1694 > URL: https://issues.apache.org/jira/browse/HIVE-1694 > Project: Hive > Issue Type: New Feature > Components: Indexing, Query Processor >Affects Versions: 0.7.0 >Reporter: Nikhil Deshpande >Assignee: Nikhil Deshpande > Attachments: demo_q1.hql, demo_q2.hql, HIVE-1694_2010-10-28.diff > > > The index building patch (Hive-417) is checked into trunk, this JIRA issue > tracks supporting indexes in Hive compiler & execution engine for SELECT > queries. > This is in ref. to John's comment at > https://issues.apache.org/jira/browse/HIVE-417?focusedCommentId=12884869&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12884869 > on creating separate JIRA issue for tracking index usage in optimizer & query > execution. > The aim of this effort is to use indexes to accelerate query execution (for > certain class of queries). E.g. > - Filters and range scans (already being worked on by He Yongqiang as part of > HIVE-417?) > - Joins (index based joins) > - Group By, Order By and other misc cases > The proposal is multi-step: > 1. Building index based operators, compiler and execution engine changes > 2. Optimizer enhancements (e.g. cost-based optimizer to compare and choose > between index scans, full table scans etc.) > This JIRA initially focuses on the first step. This JIRA is expected to hold > the information about index based plans & operator implementations for above > mentioned cases. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1878) Set the version of Hive trunk to '0.7.0-SNAPSHOT' to avoid confusing it with a release
[ https://issues.apache.org/jira/browse/HIVE-1878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977411#action_12977411 ] John Sichi commented on HIVE-1878: -- +1. Will commit when tests pass. Can you update this page accordingly? http://wiki.apache.org/hadoop/Hive/HowToRelease > Set the version of Hive trunk to '0.7.0-SNAPSHOT' to avoid confusing it with > a release > -- > > Key: HIVE-1878 > URL: https://issues.apache.org/jira/browse/HIVE-1878 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 0.7.0 >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Fix For: 0.7.0 > > Attachments: HIVE-1878.1.patch.txt > > > The build.properties file currently sets version=0.7.0, which results in > artifacts named > hive-xxx-0.7.0.jar and a tarball with the name hive-0.7.0.tar.gz. Only the > actual 0.7.0 release > should generate JARs and tarballs with this name. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1643) support range scans and non-key columns in HBase filter pushdown
[ https://issues.apache.org/jira/browse/HIVE-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977472#action_12977472 ] John Sichi commented on HIVE-1643: -- Notes for working on this: Background is in http://wiki.apache.org/hadoop/Hive/FilterPushdownDev * In HiveHBaseTableInputFormat, newIndexPredicateAnalyzer needs to add additional operators (and stop restricting the allowed column names). And then convertFilter needs to set up corresponding HBase filter conditions based on the predicates it finds. Note that for inequality conditions on the key, it's necessary to muck with startRow/stopRow (not just the filter evaluator). * See also the comment in HBaseStorageHandler.decomposePredicate. Currently, it can only accept a single predicate. If you want to be able to support AND of multiple predicates (using HBase's FilterList) then this will need to be relaxed. * Beware of the fact that until HIVE-1538 gets committed, it is more difficult to make sure that the HBase-level filtering is working as expected. The reason is that without HIVE-1538, a second copy of the filter gets applied within Hive (regardless of how the filter was decomposed when it was pushed down to HBase). So even if HBase doesn't filter out everything you're expecting it to, you won't notice in the results since Hive will do the filtering again. > support range scans and non-key columns in HBase filter pushdown > > > Key: HIVE-1643 > URL: https://issues.apache.org/jira/browse/HIVE-1643 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.7.0 > Reporter: John Sichi >Assignee: John Sichi > Fix For: 0.7.0 > > > HIVE-1226 added support for WHERE rowkey=3. We would like to support WHERE > rowkey BETWEEN 10 and 20, as well as predicates on non-rowkeys (plus > conjunctions etc). Non-rowkey conditions can't be used to filter out entire > ranges, but they can be used to push the per-row filter processing as far > down as possible. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1878) Set the version of Hive trunk to '0.7.0-SNAPSHOT' to avoid confusing it with a release
[ https://issues.apache.org/jira/browse/HIVE-1878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1878: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed. Thanks Carl! > Set the version of Hive trunk to '0.7.0-SNAPSHOT' to avoid confusing it with > a release > -- > > Key: HIVE-1878 > URL: https://issues.apache.org/jira/browse/HIVE-1878 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 0.7.0 >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Fix For: 0.7.0 > > Attachments: HIVE-1878.1.patch.txt > > > The build.properties file currently sets version=0.7.0, which results in > artifacts named > hive-xxx-0.7.0.jar and a tarball with the name hive-0.7.0.tar.gz. Only the > actual 0.7.0 release > should generate JARs and tarballs with this name. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1634) Allow access to Primitive types stored in binary format in HBase
[
https://issues.apache.org/jira/browse/HIVE-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sichi updated HIVE-1634:
-
Attachment: HIVE-1634.1.patch
Not sure what happened to the last patch I was reviewing, but here it is again.
> Allow access to Primitive types stored in binary format in HBase
>
>
> Key: HIVE-1634
> URL: https://issues.apache.org/jira/browse/HIVE-1634
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler
>Affects Versions: 0.7.0
>Reporter: Basab Maulik
>Assignee: Basab Maulik
> Attachments: HIVE-1634.0.patch, HIVE-1634.1.patch,
> TestHiveHBaseExternalTable.java
>
>
> This addresses HIVE-1245 in part, for atomic or primitive types.
> The serde property "hbase.columns.storage.types" = "-,b,b,b,b,b,b,b,b" is a
> specification of the storage option for the corresponding column in the serde
> property "hbase.columns.mapping". Allowed values are '-' for table default,
> 's' for standard string storage, and 'b' for binary storage as would be
> obtained from o.a.h.hbase.utils.Bytes. Map types for HBase column families
> use a colon separated pair such as 's:b' for the key and value part
> specifiers respectively. See the test cases and queries for HBase handler for
> additional examples.
> There is also a table property "hbase.table.default.storage.type" = "string"
> to specify a table level default storage type. The other valid specification
> is "binary". The table level default is overridden by a column level
> specification.
> This control is available for the boolean, tinyint, smallint, int, bigint,
> float, and double primitive types. The attached patch also relaxes the
> mapping of map types to HBase column families to allow any primitive type to
> be the map key.
> Attached is a program for creating a table and populating it in HBase. The
> external table in Hive can access the data as shown in the example below.
> hive> create external table TestHiveHBaseExternalTable
> > (key string, c_bool boolean, c_byte tinyint, c_short smallint,
> > c_int int, c_long bigint, c_string string, c_float float, c_double
> double)
> > stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> > with serdeproperties ("hbase.columns.mapping" =
> ":key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double")
> > tblproperties ("hbase.table.name" = "TestHiveHBaseExternalTable");
> OK
> Time taken: 0.691 seconds
> hive> select * from TestHiveHBaseExternalTable;
> OK
> key-1 NULLNULLNULLNULLNULLTest-String NULLNULL
> Time taken: 0.346 seconds
> hive> drop table TestHiveHBaseExternalTable;
> OK
> Time taken: 0.139 seconds
> hive> create external table TestHiveHBaseExternalTable
> > (key string, c_bool boolean, c_byte tinyint, c_short smallint,
> > c_int int, c_long bigint, c_string string, c_float float, c_double
> double)
> > stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> > with serdeproperties (
> > "hbase.columns.mapping" =
> ":key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double",
> > "hbase.columns.storage.types" = "-,b,b,b,b,b,b,b,b" )
> > tblproperties (
> > "hbase.table.name" = "TestHiveHBaseExternalTable",
> > "hbase.table.default.storage.type" = "string");
> OK
> Time taken: 0.139 seconds
> hive> select * from TestHiveHBaseExternalTable;
> OK
> key-1 true-128-32768 -2147483648 -9223372036854775808
> Test-String -2.1793132E-11 2.01345E291
> Time taken: 0.151 seconds
> hive> drop table TestHiveHBaseExternalTable;
> OK
> Time taken: 0.154 seconds
> hive> create external table TestHiveHBaseExternalTable
> > (key string, c_bool boolean, c_byte tinyint, c_short smallint,
> > c_int int, c_long bigint, c_string string, c_float float, c_double
> double)
> > stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> > with serdeproperties (
> > "hbase.columns.mapping" =
> ":key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double",
> > "hbase.columns.storage.types" = "-,b,b,b,b,b,-,b,b" )
> > tblproperties ("hbase.table.name" = "TestHiveHBaseExternalTable");
> OK
> Time taken: 0.347 seconds
> hive> select * from TestHiveHBaseExternalTable;
> OK
> key-1 true-128-32768 -2147483648 -9223372036854775808
> Test-String -2.1793132E-11 2.01345E291
> Time taken: 0.245 seconds
> hive>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1878) Set the version of Hive trunk to '0.7.0-SNAPSHOT' to avoid confusing it with a release
[ https://issues.apache.org/jira/browse/HIVE-1878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978040#action_12978040 ] John Sichi commented on HIVE-1878: -- SNAPSHOT follows the Maven convention for unreleased binaries. We recently ran into a case where Amazon referred to their binaries as 0.7, even though it hadn't been released yet, so this should help avoid that confusion in the future. When we cut a release branch, we'll remove the SNAPSHOT designation as part of publishing release candidates. > Set the version of Hive trunk to '0.7.0-SNAPSHOT' to avoid confusing it with > a release > -- > > Key: HIVE-1878 > URL: https://issues.apache.org/jira/browse/HIVE-1878 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 0.7.0 >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Fix For: 0.7.0 > > Attachments: HIVE-1878.1.patch.txt > > > The build.properties file currently sets version=0.7.0, which results in > artifacts named > hive-xxx-0.7.0.jar and a tarball with the name hive-0.7.0.tar.gz. Only the > actual 0.7.0 release > should generate JARs and tarballs with this name. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1878) Set the version of Hive trunk to '0.7.0-SNAPSHOT' to avoid confusing it with a release
[ https://issues.apache.org/jira/browse/HIVE-1878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978049#action_12978049 ] John Sichi commented on HIVE-1878: -- It's added as a best practice to avoid similar confusion in the future. For the specific case of Amazon, we're in communication with them via email to get them to correct their web pages (specifying SNAPSHOT and the specific revision they based their build on). > Set the version of Hive trunk to '0.7.0-SNAPSHOT' to avoid confusing it with > a release > -- > > Key: HIVE-1878 > URL: https://issues.apache.org/jira/browse/HIVE-1878 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 0.7.0 >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Fix For: 0.7.0 > > Attachments: HIVE-1878.1.patch.txt > > > The build.properties file currently sets version=0.7.0, which results in > artifacts named > hive-xxx-0.7.0.jar and a tarball with the name hive-0.7.0.tar.gz. Only the > actual 0.7.0 release > should generate JARs and tarballs with this name. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1696) Add delegation token support to metastore
[ https://issues.apache.org/jira/browse/HIVE-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978176#action_12978176 ] John Sichi commented on HIVE-1696: -- HIVE-78 (which is really large, and conflicts with this one due to the metastore codegen) has been wending its way through review for quite some time; I think Namit is going to try to get it committed tomorrow, and then we'll need an update on this one. A few comments from me: * HIVE-78's metastore API additions also references "principal", but it has a different meaning, so we should find a way to distinguish them. * The new conf variable should be named hive.metastore.token.signature * In HadoopShims.java, the overload for getTokenStrForm needs Javadoc for the tokenSignature parameter > Add delegation token support to metastore > - > > Key: HIVE-1696 > URL: https://issues.apache.org/jira/browse/HIVE-1696 > Project: Hive > Issue Type: Sub-task > Components: Metastore, Security, Server Infrastructure >Reporter: Todd Lipcon > Fix For: 0.7.0 > > Attachments: hive-1696-1-with-gen-code.patch, hive-1696-1.patch, > hive-1696-3-with-gen-code.patch, hive-1696-3.patch, hive_1696.patch, > hive_1696.patch, hive_1696_no-thrift.patch > > > As discussed in HIVE-842, kerberos authentication is only sufficient for > authentication of a hive user client to the metastore. There are other cases > where thrift calls need to be authenticated when the caller is running in an > environment without kerberos credentials. For example, an MR task running as > part of a hive job may want to report statistics to the metastore, or a job > may be running within the context of Oozie or Hive Server. > This JIRA is to implement support of delegation tokens for the metastore. The > concept of a delegation token is borrowed from the Hadoop security design - > the quick summary is that a kerberos-authenticated client may retrieve a > binary token from the server. This token can then be passed to other clients > which can use it to achieve authentication as the original user in lieu of a > kerberos ticket. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1696) Add delegation token support to metastore
[ https://issues.apache.org/jira/browse/HIVE-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi reassigned HIVE-1696: Assignee: Devaraj Das > Add delegation token support to metastore > - > > Key: HIVE-1696 > URL: https://issues.apache.org/jira/browse/HIVE-1696 > Project: Hive > Issue Type: Sub-task > Components: Metastore, Security, Server Infrastructure >Reporter: Todd Lipcon >Assignee: Devaraj Das > Fix For: 0.7.0 > > Attachments: hive-1696-1-with-gen-code.patch, hive-1696-1.patch, > hive-1696-3-with-gen-code.patch, hive-1696-3.patch, hive_1696.patch, > hive_1696.patch, hive_1696_no-thrift.patch > > > As discussed in HIVE-842, kerberos authentication is only sufficient for > authentication of a hive user client to the metastore. There are other cases > where thrift calls need to be authenticated when the caller is running in an > environment without kerberos credentials. For example, an MR task running as > part of a hive job may want to report statistics to the metastore, or a job > may be running within the context of Oozie or Hive Server. > This JIRA is to implement support of delegation tokens for the metastore. The > concept of a delegation token is borrowed from the Hadoop security design - > the quick summary is that a kerberos-authenticated client may retrieve a > binary token from the server. This token can then be passed to other clients > which can use it to achieve authentication as the original user in lieu of a > kerberos ticket. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1696) Add delegation token support to metastore
[ https://issues.apache.org/jira/browse/HIVE-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1696: - Status: Open (was: Patch Available) > Add delegation token support to metastore > - > > Key: HIVE-1696 > URL: https://issues.apache.org/jira/browse/HIVE-1696 > Project: Hive > Issue Type: Sub-task > Components: Metastore, Security, Server Infrastructure >Reporter: Todd Lipcon > Fix For: 0.7.0 > > Attachments: hive-1696-1-with-gen-code.patch, hive-1696-1.patch, > hive-1696-3-with-gen-code.patch, hive-1696-3.patch, hive_1696.patch, > hive_1696.patch, hive_1696_no-thrift.patch > > > As discussed in HIVE-842, kerberos authentication is only sufficient for > authentication of a hive user client to the metastore. There are other cases > where thrift calls need to be authenticated when the caller is running in an > environment without kerberos credentials. For example, an MR task running as > part of a hive job may want to report statistics to the metastore, or a job > may be running within the context of Oozie or Hive Server. > This JIRA is to implement support of delegation tokens for the metastore. The > concept of a delegation token is borrowed from the Hadoop security design - > the quick summary is that a kerberos-authenticated client may retrieve a > binary token from the server. This token can then be passed to other clients > which can use it to achieve authentication as the original user in lieu of a > kerberos ticket. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1896) HBase and Contrib JAR names are missing version numbers
[
https://issues.apache.org/jira/browse/HIVE-1896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978488#action_12978488
]
John Sichi commented on HIVE-1896:
--
I think this happened because the HBase code was originally in contrib, so when
I moved it out to top-level, I copied the contrib pattern. As for contrib, I
don't know why it was that way in the first place. I'm fine with bringing both
of these into conformance with the convention set by the others.
> HBase and Contrib JAR names are missing version numbers
> ---
>
> Key: HIVE-1896
> URL: https://issues.apache.org/jira/browse/HIVE-1896
> Project: Hive
> Issue Type: Bug
> Components: Build Infrastructure
>Affects Versions: 0.7.0
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
>
> Also, does anyone know why the hbase and contrib JARs use underscores
> instead of dashes in their names? Can I change this or will it break
> something?
> {code}
> ./build/dist/lib/hive-anttasks-0.7.0-SNAPSHOT.jar
> ./build/dist/lib/hive-cli-0.7.0-SNAPSHOT.jar
> ./build/dist/lib/hive-common-0.7.0-SNAPSHOT.jar
> ./build/dist/lib/hive-exec-0.7.0-SNAPSHOT.jar
> ./build/dist/lib/hive-hwi-0.7.0-SNAPSHOT.jar
> ./build/dist/lib/hive-jdbc-0.7.0-SNAPSHOT.jar
> ./build/dist/lib/hive-metastore-0.7.0-SNAPSHOT.jar
> ./build/dist/lib/hive-serde-0.7.0-SNAPSHOT.jar
> ./build/dist/lib/hive-service-0.7.0-SNAPSHOT.jar
> ./build/dist/lib/hive-shims-0.7.0-SNAPSHOT.jar
> ./build/dist/lib/hive_contrib.jar <--
> ./build/dist/lib/hive_hbase-handler.jar <--
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1899) add a factory method for creating a synchronized wrapper for IMetaStoreClient
[ https://issues.apache.org/jira/browse/HIVE-1899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1899: - Attachment: HIVE-1899.1.patch > add a factory method for creating a synchronized wrapper for IMetaStoreClient > - > > Key: HIVE-1899 > URL: https://issues.apache.org/jira/browse/HIVE-1899 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 0.7.0 > Reporter: John Sichi >Assignee: John Sichi > Fix For: 0.7.0 > > Attachments: HIVE-1899.1.patch > > > There are currently some HiveMetaStoreClient multithreading bugs. This patch > adds an (optional) synchronized wrapper for IMetaStoreClient using a dynamic > proxy. This can be used for thread safety by multithreaded apps until all > reentrancy bugs are fixed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1899) add a factory method for creating a synchronized wrapper for IMetaStoreClient
add a factory method for creating a synchronized wrapper for IMetaStoreClient - Key: HIVE-1899 URL: https://issues.apache.org/jira/browse/HIVE-1899 Project: Hive Issue Type: Improvement Components: Metastore Affects Versions: 0.7.0 Reporter: John Sichi Assignee: John Sichi Fix For: 0.7.0 Attachments: HIVE-1899.1.patch There are currently some HiveMetaStoreClient multithreading bugs. This patch adds an (optional) synchronized wrapper for IMetaStoreClient using a dynamic proxy. This can be used for thread safety by multithreaded apps until all reentrancy bugs are fixed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1899) add a factory method for creating a synchronized wrapper for IMetaStoreClient
[ https://issues.apache.org/jira/browse/HIVE-1899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1899: - Status: Patch Available (was: Open) > add a factory method for creating a synchronized wrapper for IMetaStoreClient > - > > Key: HIVE-1899 > URL: https://issues.apache.org/jira/browse/HIVE-1899 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 0.7.0 > Reporter: John Sichi >Assignee: John Sichi > Fix For: 0.7.0 > > Attachments: HIVE-1899.1.patch > > > There are currently some HiveMetaStoreClient multithreading bugs. This patch > adds an (optional) synchronized wrapper for IMetaStoreClient using a dynamic > proxy. This can be used for thread safety by multithreaded apps until all > reentrancy bugs are fixed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1482) Not all jdbc calls are threadsafe.
[ https://issues.apache.org/jira/browse/HIVE-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978943#action_12978943 ] John Sichi commented on HIVE-1482: -- In HIVE-1899, I added a special-case sync proxy, but it would be better to use the generic version Bennie has here. If someone works on this one more, maybe update HiveMetaStoreClient.newSynchronizedClient as well. (But note that my special-case needs a global lock, not a per-object lock.) > Not all jdbc calls are threadsafe. > -- > > Key: HIVE-1482 > URL: https://issues.apache.org/jira/browse/HIVE-1482 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 0.7.0 >Reporter: Bennie Schut >Assignee: Bennie Schut > Fix For: 0.7.0 > > Attachments: HIVE-1482-1.patch, HIVE-1482-2.patch > > > As per jdbc spec they should be threadsafe: > http://download.oracle.com/docs/cd/E17476_01/javase/1.3/docs/guide/jdbc/spec/jdbc-spec.frame9.html -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1903) Can't join HBase tables if one's name is the beginning of the other
[
https://issues.apache.org/jira/browse/HIVE-1903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sichi reassigned HIVE-1903:
Assignee: John Sichi
> Can't join HBase tables if one's name is the beginning of the other
> ---
>
> Key: HIVE-1903
> URL: https://issues.apache.org/jira/browse/HIVE-1903
> Project: Hive
> Issue Type: Bug
>Reporter: Jean-Daniel Cryans
>Assignee: John Sichi
> Fix For: 0.7.0
>
>
> I tried joining two tables, let's call them "table" and "table_a", but I'm
> seeing an array of errors such as this:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
> at java.util.ArrayList.get(ArrayList.java:322)
> at
> org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getRecordReader(HiveHBaseTableInputFormat.java:118)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:231)
> {noformat}
> The reason is that HiveInputFormat.pushProjectionsAndFilters matches the
> aliases with startsWith so in my case the mappers for "table_a" were getting
> the columns from "table" as well as its own (and since it had less column, it
> was trying to get one too far in the array).
> I don't know if just changing it to "equals" fill fix it, my guess is it
> won't, since it may break RCFiles.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1903) Can't join HBase tables if one's name is the beginning of the other
[
https://issues.apache.org/jira/browse/HIVE-1903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12979029#action_12979029
]
John Sichi commented on HIVE-1903:
--
Can you attach the CREATE TABLE stmts plus SELECT which fails?
> Can't join HBase tables if one's name is the beginning of the other
> ---
>
> Key: HIVE-1903
> URL: https://issues.apache.org/jira/browse/HIVE-1903
> Project: Hive
> Issue Type: Bug
>Reporter: Jean-Daniel Cryans
> Fix For: 0.7.0
>
>
> I tried joining two tables, let's call them "table" and "table_a", but I'm
> seeing an array of errors such as this:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
> at java.util.ArrayList.get(ArrayList.java:322)
> at
> org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getRecordReader(HiveHBaseTableInputFormat.java:118)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:231)
> {noformat}
> The reason is that HiveInputFormat.pushProjectionsAndFilters matches the
> aliases with startsWith so in my case the mappers for "table_a" were getting
> the columns from "table" as well as its own (and since it had less column, it
> was trying to get one too far in the array).
> I don't know if just changing it to "equals" fill fix it, my guess is it
> won't, since it may break RCFiles.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1829) Fix intermittent failures in TestRemoteMetaStore
[
https://issues.apache.org/jira/browse/HIVE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12979773#action_12979773
]
John Sichi commented on HIVE-1829:
--
+1. Will commit when tests pass.
> Fix intermittent failures in TestRemoteMetaStore
>
>
> Key: HIVE-1829
> URL: https://issues.apache.org/jira/browse/HIVE-1829
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.6.0
>Reporter: Edward Capriolo
>Assignee: Carl Steinbach
> Fix For: 0.7.0
>
> Attachments: HIVE-1829.1.patch.txt
>
>
> Notice how Running metastore! appears twice.
> {noformat}
> test:
> [junit] Running org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore
> [junit] BR.recoverFromMismatchedToken
> [junit] Tests run: 11, Failures: 0, Errors: 0, Time elapsed: 36.697 sec
> [junit] Running org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore
> [junit] Running metastore!
> [junit] Running metastore!
> [junit] org.apache.thrift.transport.TTransportException: Could not create
> ServerSocket on address 0.0.0.0/0.0.0.0:29083.
> [junit] at
> org.apache.thrift.transport.TServerSocket.(TServerSocket.java:98)
> [junit] at
> org.apache.thrift.transport.TServerSocket.(TServerSocket.java:79)
> [junit] at
> org.apache.hadoop.hive.metastore.TServerSocketKeepAlive.(TServerSocketKeepAlive.java:34)
> [junit] at
> org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:2189)
> [junit] at
> org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore$RunMS.run(TestRemoteHiveMetaStore.java:35)
> [junit] at java.lang.Thread.run(Thread.java:619)
> [junit] Running org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore
> [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
> [junit] Test org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore
> FAILED (crashed)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1903) Can't join HBase tables if one's name is the beginning of the other
[
https://issues.apache.org/jira/browse/HIVE-1903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sichi updated HIVE-1903:
-
Attachment: HIVE-1903.1.patch
> Can't join HBase tables if one's name is the beginning of the other
> ---
>
> Key: HIVE-1903
> URL: https://issues.apache.org/jira/browse/HIVE-1903
> Project: Hive
> Issue Type: Bug
>Reporter: Jean-Daniel Cryans
>Assignee: John Sichi
> Fix For: 0.7.0
>
> Attachments: HIVE-1903.1.patch
>
>
> I tried joining two tables, let's call them "table" and "table_a", but I'm
> seeing an array of errors such as this:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
> at java.util.ArrayList.get(ArrayList.java:322)
> at
> org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getRecordReader(HiveHBaseTableInputFormat.java:118)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:231)
> {noformat}
> The reason is that HiveInputFormat.pushProjectionsAndFilters matches the
> aliases with startsWith so in my case the mappers for "table_a" were getting
> the columns from "table" as well as its own (and since it had less column, it
> was trying to get one too far in the array).
> I don't know if just changing it to "equals" fill fix it, my guess is it
> won't, since it may break RCFiles.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1903) Can't join HBase tables if one's name is the beginning of the other
[
https://issues.apache.org/jira/browse/HIVE-1903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sichi updated HIVE-1903:
-
Status: Patch Available (was: Open)
This should fix it generically for all non-native tables (without disturbing
the prefix-matching logic for native tables). I'm still running through
regression tests though.
> Can't join HBase tables if one's name is the beginning of the other
> ---
>
> Key: HIVE-1903
> URL: https://issues.apache.org/jira/browse/HIVE-1903
> Project: Hive
> Issue Type: Bug
>Reporter: Jean-Daniel Cryans
>Assignee: John Sichi
> Fix For: 0.7.0
>
> Attachments: HIVE-1903.1.patch
>
>
> I tried joining two tables, let's call them "table" and "table_a", but I'm
> seeing an array of errors such as this:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
> at java.util.ArrayList.get(ArrayList.java:322)
> at
> org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getRecordReader(HiveHBaseTableInputFormat.java:118)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:231)
> {noformat}
> The reason is that HiveInputFormat.pushProjectionsAndFilters matches the
> aliases with startsWith so in my case the mappers for "table_a" were getting
> the columns from "table" as well as its own (and since it had less column, it
> was trying to get one too far in the array).
> I don't know if just changing it to "equals" fill fix it, my guess is it
> won't, since it may break RCFiles.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the Hive MetaStore
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12979926#action_12979926 ] John Sichi commented on HIVE-1862: -- I checked our server; it's MySQL 5.0.84. Also, we set the following hive-site.xml properties: datanucleus.transactionIsolation=repeatable-read datanucleus.valuegeneration.transactionIsolation=repeatable-read datanucleus.connectionPoolingType=none > Revive partition filtering in the Hive MetaStore > > > Key: HIVE-1862 > URL: https://issues.apache.org/jira/browse/HIVE-1862 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.7.0 >Reporter: Devaraj Das > Fix For: 0.7.0 > > Attachments: HIVE-1862.1.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, > runqry > > > HIVE-1853 downgraded the JDO version. This makes the feature of partition > filtering in the metastore unusable. This jira is to keep track of the lost > feature and discussing approaches to bring it back. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1829) Fix intermittent failures in TestRemoteMetaStore
[
https://issues.apache.org/jira/browse/HIVE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sichi updated HIVE-1829:
-
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed. Thanks Carl!
> Fix intermittent failures in TestRemoteMetaStore
>
>
> Key: HIVE-1829
> URL: https://issues.apache.org/jira/browse/HIVE-1829
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.6.0
>Reporter: Edward Capriolo
>Assignee: Carl Steinbach
> Fix For: 0.7.0
>
> Attachments: HIVE-1829.1.patch.txt
>
>
> Notice how Running metastore! appears twice.
> {noformat}
> test:
> [junit] Running org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore
> [junit] BR.recoverFromMismatchedToken
> [junit] Tests run: 11, Failures: 0, Errors: 0, Time elapsed: 36.697 sec
> [junit] Running org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore
> [junit] Running metastore!
> [junit] Running metastore!
> [junit] org.apache.thrift.transport.TTransportException: Could not create
> ServerSocket on address 0.0.0.0/0.0.0.0:29083.
> [junit] at
> org.apache.thrift.transport.TServerSocket.(TServerSocket.java:98)
> [junit] at
> org.apache.thrift.transport.TServerSocket.(TServerSocket.java:79)
> [junit] at
> org.apache.hadoop.hive.metastore.TServerSocketKeepAlive.(TServerSocketKeepAlive.java:34)
> [junit] at
> org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:2189)
> [junit] at
> org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore$RunMS.run(TestRemoteHiveMetaStore.java:35)
> [junit] at java.lang.Thread.run(Thread.java:619)
> [junit] Running org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore
> [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
> [junit] Test org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore
> FAILED (crashed)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the Hive MetaStore
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980202#action_12980202 ] John Sichi commented on HIVE-1862: -- Many of our clients connect to the DB directly (not through the metastore Thrift server). As far as I know, there are no issues with DBCP. > Revive partition filtering in the Hive MetaStore > > > Key: HIVE-1862 > URL: https://issues.apache.org/jira/browse/HIVE-1862 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.7.0 >Reporter: Devaraj Das > Fix For: 0.7.0 > > Attachments: HIVE-1862.1.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, > runqry > > > HIVE-1853 downgraded the JDO version. This makes the feature of partition > filtering in the metastore unusable. This jira is to keep track of the lost > feature and discussing approaches to bring it back. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1716) TestHBaseCliDriver failed in current trunk
[ https://issues.apache.org/jira/browse/HIVE-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi reassigned HIVE-1716: Assignee: John Sichi > TestHBaseCliDriver failed in current trunk > -- > > Key: HIVE-1716 > URL: https://issues.apache.org/jira/browse/HIVE-1716 > Project: Hive > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Ning Zhang > Assignee: John Sichi > Fix For: 0.7.0 > > > ant test -Dhadoop.version=0.20.0 -Dtestcase=TestHBaseCliDriver: > >[junit] org.apache.hadoop.hbase.client.NoServerForRegionException: Timed > out trying to locate root region > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:976) > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:625) > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:607) > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:738) > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:634) > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601) > [junit] at > org.apache.hadoop.hbase.client.HTable.(HTable.java:128) > [junit] at > org.apache.hadoop.hive.hbase.HBaseTestSetup.setUpFixtures(HBaseTestSetup.java:87) > [junit] at > org.apache.hadoop.hive.hbase.HBaseTestSetup.preTest(HBaseTestSetup.java:59) > [junit] at > org.apache.hadoop.hive.hbase.HBaseQTestUtil.(HBaseQTestUtil.java:31) > [junit] at > org.apache.hadoop.hive.cli.TestHBaseCliDriver.setUp(TestHBaseCliDriver.java:43) > [junit] at junit.framework.TestCase.runBare(TestCase.java:125) > [junit] at junit.framework.TestResult$1.protect(TestResult.java:106) > [junit] at > junit.framework.TestResult.runProtected(TestResult.java:124) > [junit] at junit.framework.TestResult.run(TestResult.java:109) > [junit] at junit.framework.TestCase.run(TestCase.java:118) > [junit] at junit.framework.TestSuite.runTest(TestSuite.java:208) > [junit] at junit.framework.TestSuite.run(TestSuite.java:203) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-1911) TestHBaseCliDriver is broken
[ https://issues.apache.org/jira/browse/HIVE-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi resolved HIVE-1911. -- Resolution: Duplicate Dupe of HIVE-1716. It's a sporadic issue, probably due to a port lingering. I'll see if I can add a sleep somewhere to squelch it. > TestHBaseCliDriver is broken > > > Key: HIVE-1911 > URL: https://issues.apache.org/jira/browse/HIVE-1911 > Project: Hive > Issue Type: Bug >Reporter: Ning Zhang >Assignee: John Sichi > > It broken on the current trunk: > $ ant test -Dtestcase=TestHBaseCliDriver > > > test: > [junit] Running org.apache.hadoop.hive.cli.TestHBaseCliDriver > [junit] Exception: Timed out trying to locate root region > [junit] org.apache.hadoop.hbase.client.NoServerForRegionException: Timed > out trying to locate root region > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:976) > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:625) > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:607) > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:738) > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:634) > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601) > [junit] at > org.apache.hadoop.hbase.client.HTable.(HTable.java:128) > [junit] at > org.apache.hadoop.hive.hbase.HBaseTestSetup.setUpFixtures(HBaseTestSetup.java:87) > [junit] at > org.apache.hadoop.hive.hbase.HBaseTestSetup.preTest(HBaseTestSetup.java:59) > [junit] at > org.apache.hadoop.hive.hbase.HBaseQTestUtil.(HBaseQTestUtil.java:31) > [junit] at > org.apache.hadoop.hive.cli.TestHBaseCliDriver.setUp(TestHBaseCliDriver.java:43) > [junit] at junit.framework.TestCase.runBare(TestCase.java:125) > [junit] at junit.framework.TestResult$1.protect(TestResult.java:106) > [junit] at > junit.framework.TestResult.runProtected(TestResult.java:124) > [junit] at junit.framework.TestResult.run(TestResult.java:109) > [junit] at junit.framework.TestCase.run(TestCase.java:118) > [junit] at junit.framework.TestSuite.runTest(TestSuite.java:208) > [junit] at junit.framework.TestSuite.run(TestSuite.java:203) > [junit] at > junit.extensions.TestDecorator.basicRun(TestDecorator.java:22) > [junit] at junit.extensions.TestSetup$1.protect(TestSetup.java:19) > [junit] at > junit.framework.TestResult.runProtected(TestResult.java:124) > [junit] at junit.extensions.TestSetup.run(TestSetup.java:23) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:422) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:931) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:785) > [junit] Hive history > file=/data/users/nzhang/work/2/apache-hive/build/hbase-handler/tmp/hive_job_log_nzhang_20110 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1911) TestHBaseCliDriver is broken
[ https://issues.apache.org/jira/browse/HIVE-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12981438#action_12981438 ] John Sichi commented on HIVE-1911: -- Yeah, it seems to have gotten much worse since the concurrency patch with zookeeper got committed, so it may be some interference between the different instances of zookeeper. > TestHBaseCliDriver is broken > > > Key: HIVE-1911 > URL: https://issues.apache.org/jira/browse/HIVE-1911 > Project: Hive > Issue Type: Bug >Reporter: Ning Zhang > Assignee: John Sichi > > It broken on the current trunk: > $ ant test -Dtestcase=TestHBaseCliDriver > > > test: > [junit] Running org.apache.hadoop.hive.cli.TestHBaseCliDriver > [junit] Exception: Timed out trying to locate root region > [junit] org.apache.hadoop.hbase.client.NoServerForRegionException: Timed > out trying to locate root region > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:976) > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:625) > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:607) > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:738) > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:634) > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601) > [junit] at > org.apache.hadoop.hbase.client.HTable.(HTable.java:128) > [junit] at > org.apache.hadoop.hive.hbase.HBaseTestSetup.setUpFixtures(HBaseTestSetup.java:87) > [junit] at > org.apache.hadoop.hive.hbase.HBaseTestSetup.preTest(HBaseTestSetup.java:59) > [junit] at > org.apache.hadoop.hive.hbase.HBaseQTestUtil.(HBaseQTestUtil.java:31) > [junit] at > org.apache.hadoop.hive.cli.TestHBaseCliDriver.setUp(TestHBaseCliDriver.java:43) > [junit] at junit.framework.TestCase.runBare(TestCase.java:125) > [junit] at junit.framework.TestResult$1.protect(TestResult.java:106) > [junit] at > junit.framework.TestResult.runProtected(TestResult.java:124) > [junit] at junit.framework.TestResult.run(TestResult.java:109) > [junit] at junit.framework.TestCase.run(TestCase.java:118) > [junit] at junit.framework.TestSuite.runTest(TestSuite.java:208) > [junit] at junit.framework.TestSuite.run(TestSuite.java:203) > [junit] at > junit.extensions.TestDecorator.basicRun(TestDecorator.java:22) > [junit] at junit.extensions.TestSetup$1.protect(TestSetup.java:19) > [junit] at > junit.framework.TestResult.runProtected(TestResult.java:124) > [junit] at junit.extensions.TestSetup.run(TestSetup.java:23) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:422) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:931) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:785) > [junit] Hive history > file=/data/users/nzhang/work/2/apache-hive/build/hbase-handler/tmp/hive_job_log_nzhang_20110 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1696) Add delegation token support to metastore
[ https://issues.apache.org/jira/browse/HIVE-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12981577#action_12981577 ] John Sichi commented on HIVE-1696: -- Looks fine to me. Can you post the patch with generated code so I can do some testing? > Add delegation token support to metastore > - > > Key: HIVE-1696 > URL: https://issues.apache.org/jira/browse/HIVE-1696 > Project: Hive > Issue Type: Sub-task > Components: Metastore, Security, Server Infrastructure >Reporter: Todd Lipcon >Assignee: Devaraj Das > Fix For: 0.7.0 > > Attachments: hive-1696-1-with-gen-code.patch, hive-1696-1.patch, > hive-1696-3-with-gen-code.patch, hive-1696-3.patch, hive-1696-4.patch, > hive-1696-4.patch, hive_1696.patch, hive_1696.patch, hive_1696_no-thrift.patch > > > As discussed in HIVE-842, kerberos authentication is only sufficient for > authentication of a hive user client to the metastore. There are other cases > where thrift calls need to be authenticated when the caller is running in an > environment without kerberos credentials. For example, an MR task running as > part of a hive job may want to report statistics to the metastore, or a job > may be running within the context of Oozie or Hive Server. > This JIRA is to implement support of delegation tokens for the metastore. The > concept of a delegation token is borrowed from the Hadoop security design - > the quick summary is that a kerberos-authenticated client may retrieve a > binary token from the server. This token can then be passed to other clients > which can use it to achieve authentication as the original user in lieu of a > kerberos ticket. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-1914) failures in testhbaseclidriver
[ https://issues.apache.org/jira/browse/HIVE-1914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi resolved HIVE-1914. -- Resolution: Duplicate Dup of HIVE-1716. > failures in testhbaseclidriver > -- > > Key: HIVE-1914 > URL: https://issues.apache.org/jira/browse/HIVE-1914 > Project: Hive > Issue Type: Bug >Reporter: Namit Jain > Fix For: 0.7.0 > > > i didnt debug it -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1716) TestHBaseCliDriver failed in current trunk
[ https://issues.apache.org/jira/browse/HIVE-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12981997#action_12981997 ] John Sichi commented on HIVE-1716: -- I tried debugging this one yesterday and noticed that if I kill the scribed daemon, then the test works. Is it possible that a zookeeper-enabled scribe running on the machines in question is causing port conflicts? Try running on a freshly restarted machine to confirm. > TestHBaseCliDriver failed in current trunk > -- > > Key: HIVE-1716 > URL: https://issues.apache.org/jira/browse/HIVE-1716 > Project: Hive > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Ning Zhang >Assignee: John Sichi > Fix For: 0.7.0 > > > ant test -Dhadoop.version=0.20.0 -Dtestcase=TestHBaseCliDriver: > >[junit] org.apache.hadoop.hbase.client.NoServerForRegionException: Timed > out trying to locate root region > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:976) > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:625) > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:607) > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:738) > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:634) > [junit] at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601) > [junit] at > org.apache.hadoop.hbase.client.HTable.(HTable.java:128) > [junit] at > org.apache.hadoop.hive.hbase.HBaseTestSetup.setUpFixtures(HBaseTestSetup.java:87) > [junit] at > org.apache.hadoop.hive.hbase.HBaseTestSetup.preTest(HBaseTestSetup.java:59) > [junit] at > org.apache.hadoop.hive.hbase.HBaseQTestUtil.(HBaseQTestUtil.java:31) > [junit] at > org.apache.hadoop.hive.cli.TestHBaseCliDriver.setUp(TestHBaseCliDriver.java:43) > [junit] at junit.framework.TestCase.runBare(TestCase.java:125) > [junit] at junit.framework.TestResult$1.protect(TestResult.java:106) > [junit] at > junit.framework.TestResult.runProtected(TestResult.java:124) > [junit] at junit.framework.TestResult.run(TestResult.java:109) > [junit] at junit.framework.TestCase.run(TestCase.java:118) > [junit] at junit.framework.TestSuite.runTest(TestSuite.java:208) > [junit] at junit.framework.TestSuite.run(TestSuite.java:203) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1694) Accelerate query execution using indexes
[ https://issues.apache.org/jira/browse/HIVE-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12982999#action_12982999 ] John Sichi commented on HIVE-1694: -- Is there an updated tree for this? I checked github but didn't see it. HIVE-1644 needs support for compiling internally-generated SQL into operators, so if you have that working, I'd like to point the Harvey Mudd folks at it when I talk to them tomorrow. > Accelerate query execution using indexes > > > Key: HIVE-1694 > URL: https://issues.apache.org/jira/browse/HIVE-1694 > Project: Hive > Issue Type: New Feature > Components: Indexing, Query Processor >Affects Versions: 0.7.0 >Reporter: Nikhil Deshpande >Assignee: Nikhil Deshpande > Attachments: demo_q1.hql, demo_q2.hql, HIVE-1694_2010-10-28.diff > > > The index building patch (Hive-417) is checked into trunk, this JIRA issue > tracks supporting indexes in Hive compiler & execution engine for SELECT > queries. > This is in ref. to John's comment at > https://issues.apache.org/jira/browse/HIVE-417?focusedCommentId=12884869&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12884869 > on creating separate JIRA issue for tracking index usage in optimizer & query > execution. > The aim of this effort is to use indexes to accelerate query execution (for > certain class of queries). E.g. > - Filters and range scans (already being worked on by He Yongqiang as part of > HIVE-417?) > - Joins (index based joins) > - Group By, Order By and other misc cases > The proposal is multi-step: > 1. Building index based operators, compiler and execution engine changes > 2. Optimizer enhancements (e.g. cost-based optimizer to compare and choose > between index scans, full table scans etc.) > This JIRA initially focuses on the first step. This JIRA is expected to hold > the information about index based plans & operator implementations for above > mentioned cases. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Work started: (HIVE-1235) use Ivy for fetching HBase dependencies
[ https://issues.apache.org/jira/browse/HIVE-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-1235 started by John Sichi. > use Ivy for fetching HBase dependencies > --- > > Key: HIVE-1235 > URL: https://issues.apache.org/jira/browse/HIVE-1235 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.6.0 > Reporter: John Sichi >Assignee: John Sichi > > For now I just checked in the .jars under hbase-handler/lib to avoid having > to pull down the huge tarballs. If individual jars were available > (HBASE-1933), we could use Ivy to pull those. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1235) use Ivy for fetching HBase dependencies
[ https://issues.apache.org/jira/browse/HIVE-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986123#action_12986123 ] John Sichi commented on HIVE-1235: -- Starting on this one with HBase 0.89 and zookeeper 3.3.1 as part of addressing HIVE-1716. > use Ivy for fetching HBase dependencies > --- > > Key: HIVE-1235 > URL: https://issues.apache.org/jira/browse/HIVE-1235 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.6.0 >Reporter: John Sichi >Assignee: John Sichi > > For now I just checked in the .jars under hbase-handler/lib to avoid having > to pull down the huge tarballs. If individual jars were available > (HBASE-1933), we could use Ivy to pull those. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1927) Fix TestHadoop20SAuthBridge failure on Hudson
[ https://issues.apache.org/jira/browse/HIVE-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986253#action_12986253 ] John Sichi commented on HIVE-1927: -- As a related point, it would also be good if we could set up another Hudson config running tests against 0.20S. We used to have instances for 0.17 etc when we were still supporting them. > Fix TestHadoop20SAuthBridge failure on Hudson > - > > Key: HIVE-1927 > URL: https://issues.apache.org/jira/browse/HIVE-1927 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Carl Steinbach >Assignee: Devaraj Das >Priority: Blocker > Fix For: 0.7.0 > > > The patch for HIVE-1696 is the source a test failure on Hudson. The likely > cause of > this failure is that the classpath is not being set correctly for the > TestHadoop20SAuthBridge > test, which depends on the security enhanced version of Hadoop. > A couple things to note: > * Hive tests are supposed to run against Hadoop 0.20.0 by default. This value > is set in the build.properties file in the project top-level. Altering the > Hudson job to set hadoop.version to some other value is not a fix for this > issue since the Hudson job will then cease to reflect the default behavior of > Hive tests. > * HIVE-1696 added new secure compile targets to the shims/build.xml file. > These targets explicitly set the classpath to include the secure version of > Hadoop. The fix for this issue likely involves overriding the "test" target > in shims/build.xml and explicitly setting the classpath in this target to > also use the secure version of Hadoop. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1235) use Ivy for fetching HBase dependencies
[ https://issues.apache.org/jira/browse/HIVE-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1235: - Attachment: HIVE-1235.1.patch This patch (HIVE-1235.1.patch) is not ready for review yet, and ant test is hitting an OOM in one of the ivy calls right before running the HBase tests; just posting it here as a checkpoint. To apply, you also need: svn remove lib/hbase-0.20.3-test.jar svn remove lib/hbase-0.20.3.jar svn remove lib/zookeeper-3.2.2.jar > use Ivy for fetching HBase dependencies > --- > > Key: HIVE-1235 > URL: https://issues.apache.org/jira/browse/HIVE-1235 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.6.0 > Reporter: John Sichi >Assignee: John Sichi > Attachments: HIVE-1235.1.patch > > > For now I just checked in the .jars under hbase-handler/lib to avoid having > to pull down the huge tarballs. If individual jars were available > (HBASE-1933), we could use Ivy to pull those. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1235) use Ivy for fetching HBase dependencies
[ https://issues.apache.org/jira/browse/HIVE-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986693#action_12986693 ] John Sichi commented on HIVE-1235: -- Note that transitive dependencies were pulling down all kinds of odds and ends we don't need, so I had to turn them off and then get the only hard dependency (guava) explicitly. Slightly defeating the purpose of ivy, but... > use Ivy for fetching HBase dependencies > --- > > Key: HIVE-1235 > URL: https://issues.apache.org/jira/browse/HIVE-1235 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.6.0 >Reporter: John Sichi >Assignee: John Sichi > Attachments: HIVE-1235.1.patch > > > For now I just checked in the .jars under hbase-handler/lib to avoid having > to pull down the huge tarballs. If individual jars were available > (HBASE-1933), we could use Ivy to pull those. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1867) Fix intermittent failures in TestNegativeCliDriver/dyn_part_empty.q
[
https://issues.apache.org/jira/browse/HIVE-1867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987221#action_12987221
]
John Sichi commented on HIVE-1867:
--
I think the cause here may be the same as in HIVE-1872; calling System.exit
causes ant to freak out.
> Fix intermittent failures in TestNegativeCliDriver/dyn_part_empty.q
> ---
>
> Key: HIVE-1867
> URL: https://issues.apache.org/jira/browse/HIVE-1867
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Carl Steinbach
>
> {code}
> [junit] Begin query: dyn_part_empty.q
> [junit] Running org.apache.hadoop.hive.cli.TestNegativeCliDriver
> [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
> [junit] Test org.apache.hadoop.hive.cli.TestNegativeCliDriver FAILED
> (crashed)
> {code}
> dyn_part_empty.q has been intermittently failing on Hudson. I was able to
> reproduce locally,
> and with different versions of JUnit (3.8.1, 4.5, 4.8.2).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1872) Hive process is exiting on executing ALTER query
[
https://issues.apache.org/jira/browse/HIVE-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987222#action_12987222
]
John Sichi commented on HIVE-1872:
--
I don't know about the solution, but I think the System.exit is also the cause
of HIVE-1867, which has been intermittently preventing ant test from passing.
> Hive process is exiting on executing ALTER query
>
>
> Key: HIVE-1872
> URL: https://issues.apache.org/jira/browse/HIVE-1872
> Project: Hive
> Issue Type: Bug
> Components: CLI, Server Infrastructure
>Affects Versions: 0.6.0
> Environment: SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel
> 2.6.16.60-0.21-smp (3)
> Hadoop 0.20.1
> Hive 0.6.0
>Reporter: Bharath R
>Assignee: Bharath R
> Attachments: HIVE-1872.1.patch
>
>
> Hive process is exiting on executing the below queries in the same order as
> mentioned
> 1) CREATE TABLE SAMPLETABLE(IP STRING , showtime BIGINT ) partitioned by (ds
> string,ipz int) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\040'
> 2) ALTER TABLE SAMPLETABLE add Partition(ds='sf') location
> '/user/hive/warehouse' Partition(ipz=100) location '/user/hive/warehouse'
> After the second query execution , the hive throws the below exception and
> exiting the process
> 10:09:03 ERROR exec.DDLTask: FAILED: Error in metadata: table is partitioned
> but partition spec is not specified or tab: {ipz=100}
> org.apache.hadoop.hive.ql.metadata.HiveException: table is partitioned but
> partition spec is not specified or tab: {ipz=100}
> at
> org.apache.hadoop.hive.ql.metadata.Table.isValidSpec(Table.java:341)
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:902)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:282)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:191)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
> at
> org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:114)
> at
> org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(ThriftHive.java:378)
> at
> org.apache.hadoop.hive.service.ThriftHive$Processor.process(ThriftHive.java:366)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> As the "alter" query is incorrect the exception was thrown, ideally it should
> be "ALTER TABLE SAMPLETABLE add Partition(ds='sf',ipz=100) location
> '/user/hive/warehouse'".
> It is not good to exit the HIVE process when the query is incorrect.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1872) Hive process is exiting on executing ALTER query
[
https://issues.apache.org/jira/browse/HIVE-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987236#action_12987236
]
John Sichi commented on HIVE-1872:
--
Note that the System.exit was introduced in HIVE-549; maybe worth revisiting
some of the discussion at the end of the comments there regarding refactoring
to use JobControl.
> Hive process is exiting on executing ALTER query
>
>
> Key: HIVE-1872
> URL: https://issues.apache.org/jira/browse/HIVE-1872
> Project: Hive
> Issue Type: Bug
> Components: CLI, Server Infrastructure
>Affects Versions: 0.6.0
> Environment: SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel
> 2.6.16.60-0.21-smp (3)
> Hadoop 0.20.1
> Hive 0.6.0
>Reporter: Bharath R
>Assignee: Bharath R
> Attachments: HIVE-1872.1.patch
>
>
> Hive process is exiting on executing the below queries in the same order as
> mentioned
> 1) CREATE TABLE SAMPLETABLE(IP STRING , showtime BIGINT ) partitioned by (ds
> string,ipz int) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\040'
> 2) ALTER TABLE SAMPLETABLE add Partition(ds='sf') location
> '/user/hive/warehouse' Partition(ipz=100) location '/user/hive/warehouse'
> After the second query execution , the hive throws the below exception and
> exiting the process
> 10:09:03 ERROR exec.DDLTask: FAILED: Error in metadata: table is partitioned
> but partition spec is not specified or tab: {ipz=100}
> org.apache.hadoop.hive.ql.metadata.HiveException: table is partitioned but
> partition spec is not specified or tab: {ipz=100}
> at
> org.apache.hadoop.hive.ql.metadata.Table.isValidSpec(Table.java:341)
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:902)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:282)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:191)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
> at
> org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:114)
> at
> org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(ThriftHive.java:378)
> at
> org.apache.hadoop.hive.service.ThriftHive$Processor.process(ThriftHive.java:366)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> As the "alter" query is incorrect the exception was thrown, ideally it should
> be "ALTER TABLE SAMPLETABLE add Partition(ds='sf',ipz=100) location
> '/user/hive/warehouse'".
> It is not good to exit the HIVE process when the query is incorrect.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1872) Hive process is exiting on executing ALTER query
[
https://issues.apache.org/jira/browse/HIVE-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987712#action_12987712
]
John Sichi commented on HIVE-1872:
--
Calling System.exit is definitely a problem since for HIVE-549, this causes
JUnit to treat the JVM as crashed. It seems like just returning 9 should be
good enough, since all the same System.exit processing (thread-killing and
shutdown hook execution) will take place as soon as the main thread returns;
the only difference is that JUnit gets a chance to clean up as the stack
unwinds.
interrupt isn't really guaranteed to do much more, e.g. threads can catch and
ignore InterruptedException, so I don't think we should bother adding it.
If there's a reason we need to leave the explicit System.exit call there, then
maybe we need a conf to tell it not to do that during tests.
> Hive process is exiting on executing ALTER query
>
>
> Key: HIVE-1872
> URL: https://issues.apache.org/jira/browse/HIVE-1872
> Project: Hive
> Issue Type: Bug
> Components: CLI, Server Infrastructure
>Affects Versions: 0.6.0
> Environment: SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel
> 2.6.16.60-0.21-smp (3)
> Hadoop 0.20.1
> Hive 0.6.0
>Reporter: Bharath R
>Assignee: Bharath R
> Attachments: HIVE-1872.1.patch
>
>
> Hive process is exiting on executing the below queries in the same order as
> mentioned
> 1) CREATE TABLE SAMPLETABLE(IP STRING , showtime BIGINT ) partitioned by (ds
> string,ipz int) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\040'
> 2) ALTER TABLE SAMPLETABLE add Partition(ds='sf') location
> '/user/hive/warehouse' Partition(ipz=100) location '/user/hive/warehouse'
> After the second query execution , the hive throws the below exception and
> exiting the process
> 10:09:03 ERROR exec.DDLTask: FAILED: Error in metadata: table is partitioned
> but partition spec is not specified or tab: {ipz=100}
> org.apache.hadoop.hive.ql.metadata.HiveException: table is partitioned but
> partition spec is not specified or tab: {ipz=100}
> at
> org.apache.hadoop.hive.ql.metadata.Table.isValidSpec(Table.java:341)
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:902)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:282)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:191)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
> at
> org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:114)
> at
> org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(ThriftHive.java:378)
> at
> org.apache.hadoop.hive.service.ThriftHive$Processor.process(ThriftHive.java:366)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> As the "alter" query is incorrect the exception was thrown, ideally it should
> be "ALTER TABLE SAMPLETABLE add Partition(ds='sf',ipz=100) location
> '/user/hive/warehouse'".
> It is not good to exit the HIVE process when the query is incorrect.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1235) use Ivy for fetching HBase dependencies
[ https://issues.apache.org/jira/browse/HIVE-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1235: - Attachment: HIVE-1235.2.patch HIVE-1235.2.patch passes tests. A couple of the HBase tests had broken due to recent changes, so this fixes those. It also gets rid of redundant ivy-retrieve for hadoopcore (which is what was causing the OOM's). > use Ivy for fetching HBase dependencies > --- > > Key: HIVE-1235 > URL: https://issues.apache.org/jira/browse/HIVE-1235 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.6.0 >Reporter: John Sichi >Assignee: John Sichi > Attachments: HIVE-1235.1.patch, HIVE-1235.2.patch > > > For now I just checked in the .jars under hbase-handler/lib to avoid having > to pull down the huge tarballs. If individual jars were available > (HBASE-1933), we could use Ivy to pull those. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1235) use Ivy for fetching HBase dependencies
[ https://issues.apache.org/jira/browse/HIVE-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1235: - Fix Version/s: 0.7.0 Status: Patch Available (was: In Progress) Review board: https://reviews.apache.org/r/367/ Note this also makes the hbase-handler build non-conditional since we've dropped support for pre-0.20. > use Ivy for fetching HBase dependencies > --- > > Key: HIVE-1235 > URL: https://issues.apache.org/jira/browse/HIVE-1235 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.6.0 >Reporter: John Sichi >Assignee: John Sichi > Fix For: 0.7.0 > > Attachments: HIVE-1235.1.patch, HIVE-1235.2.patch > > > For now I just checked in the .jars under hbase-handler/lib to avoid having > to pull down the huge tarballs. If individual jars were available > (HBASE-1933), we could use Ivy to pull those. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1235) use Ivy for fetching HBase dependencies
[ https://issues.apache.org/jira/browse/HIVE-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988366#action_12988366 ] John Sichi commented on HIVE-1235: -- Would be nice to get this one in for 0.7. > use Ivy for fetching HBase dependencies > --- > > Key: HIVE-1235 > URL: https://issues.apache.org/jira/browse/HIVE-1235 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.6.0 >Reporter: John Sichi >Assignee: John Sichi > Fix For: 0.7.0 > > Attachments: HIVE-1235.1.patch, HIVE-1235.2.patch > > > For now I just checked in the .jars under hbase-handler/lib to avoid having > to pull down the huge tarballs. If individual jars were available > (HBASE-1933), we could use Ivy to pull those. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1235) use Ivy for fetching HBase dependencies
[ https://issues.apache.org/jira/browse/HIVE-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1235: - Status: Open (was: Patch Available) Oops, got one failure in TestHBaseSerDe due to a change in the Result constructor contract (it now expects the kv's to be presorted). I'll uploaded another patch. > use Ivy for fetching HBase dependencies > --- > > Key: HIVE-1235 > URL: https://issues.apache.org/jira/browse/HIVE-1235 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.6.0 >Reporter: John Sichi >Assignee: John Sichi > Fix For: 0.7.0 > > Attachments: HIVE-1235.1.patch, HIVE-1235.2.patch > > > For now I just checked in the .jars under hbase-handler/lib to avoid having > to pull down the huge tarballs. If individual jars were available > (HBASE-1933), we could use Ivy to pull those. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1235) use Ivy for fetching HBase dependencies
[ https://issues.apache.org/jira/browse/HIVE-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1235: - Attachment: HIVE-1235.3.patch I updated review board as well. > use Ivy for fetching HBase dependencies > --- > > Key: HIVE-1235 > URL: https://issues.apache.org/jira/browse/HIVE-1235 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.6.0 > Reporter: John Sichi >Assignee: John Sichi > Fix For: 0.7.0 > > Attachments: HIVE-1235.1.patch, HIVE-1235.2.patch, HIVE-1235.3.patch > > > For now I just checked in the .jars under hbase-handler/lib to avoid having > to pull down the huge tarballs. If individual jars were available > (HBASE-1933), we could use Ivy to pull those. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
