RE: [jira] Updated: (HIVE-1609) Support partition filtering in metastore
John, I patched .classpath, but the build still fails, with other errors. Please look at https://issues.apache.org/jira/browse/HIVE-1664. Steven -Original Message- From: John Sichi [mailto:jsi...@facebook.com] Sent: Tuesday, September 21, 2010 7:04 PM To: Cc: Steven Wong Subject: Re: [jira] Updated: (HIVE-1609) Support partition filtering in metastore Oops, yeah, this happened the last time we upgraded datanucleus also (see HIVE-1373 where we fixed it). If someone posts a patch which fixes .classpath, I'll commit it. JVS On Sep 21, 2010, at 6:51 PM, Steven Wong wrote: > Did this check-in break the Eclipse build? > > > -Original Message- > From: John Sichi (JIRA) [mailto:j...@apache.org] > Sent: Tuesday, September 21, 2010 2:16 PM > To: hive-dev@hadoop.apache.org > Subject: [jira] Updated: (HIVE-1609) Support partition filtering in metastore > > > [ > https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel > ] > > John Sichi updated HIVE-1609: > - > > Status: Resolved (was: Patch Available) >Hadoop Flags: [Reviewed] > Resolution: Fixed > > Committed. Thanks Ajay! > > >> Support partition filtering in metastore >> >> >>Key: HIVE-1609 >>URL: https://issues.apache.org/jira/browse/HIVE-1609 >>Project: Hadoop Hive >> Issue Type: New Feature >> Components: Metastore >> Reporter: Ajay Kidave >> Assignee: Ajay Kidave >>Fix For: 0.7.0 >> >>Attachments: hive_1609.patch, hive_1609_2.patch, hive_1609_3.patch >> >> >> The metastore needs to have support for returning a list of partitions based >> on user specified filter conditions. This will be useful for tools which >> need to do partition pruning. Howl is one such use case. The way partition >> pruning is done during hive query execution need not be changed. > > -- > This message is automatically generated by JIRA. > - > You can reply to this email to add a comment to the issue online. > >
Re: [jira] Updated: (HIVE-1609) Support partition filtering in metastore
Oops, yeah, this happened the last time we upgraded datanucleus also (see HIVE-1373 where we fixed it). If someone posts a patch which fixes .classpath, I'll commit it. JVS On Sep 21, 2010, at 6:51 PM, Steven Wong wrote: > Did this check-in break the Eclipse build? > > > -Original Message- > From: John Sichi (JIRA) [mailto:j...@apache.org] > Sent: Tuesday, September 21, 2010 2:16 PM > To: hive-dev@hadoop.apache.org > Subject: [jira] Updated: (HIVE-1609) Support partition filtering in metastore > > > [ > https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel > ] > > John Sichi updated HIVE-1609: > - > > Status: Resolved (was: Patch Available) >Hadoop Flags: [Reviewed] > Resolution: Fixed > > Committed. Thanks Ajay! > > >> Support partition filtering in metastore >> >> >>Key: HIVE-1609 >>URL: https://issues.apache.org/jira/browse/HIVE-1609 >>Project: Hadoop Hive >> Issue Type: New Feature >> Components: Metastore >> Reporter: Ajay Kidave >> Assignee: Ajay Kidave >>Fix For: 0.7.0 >> >>Attachments: hive_1609.patch, hive_1609_2.patch, hive_1609_3.patch >> >> >> The metastore needs to have support for returning a list of partitions based >> on user specified filter conditions. This will be useful for tools which >> need to do partition pruning. Howl is one such use case. The way partition >> pruning is done during hive query execution need not be changed. > > -- > This message is automatically generated by JIRA. > - > You can reply to this email to add a comment to the issue online. > >
RE: [jira] Updated: (HIVE-1609) Support partition filtering in metastore
Did this check-in break the Eclipse build? -Original Message- From: John Sichi (JIRA) [mailto:j...@apache.org] Sent: Tuesday, September 21, 2010 2:16 PM To: hive-dev@hadoop.apache.org Subject: [jira] Updated: (HIVE-1609) Support partition filtering in metastore [ https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1609: - Status: Resolved (was: Patch Available) Hadoop Flags: [Reviewed] Resolution: Fixed Committed. Thanks Ajay! > Support partition filtering in metastore > > > Key: HIVE-1609 > URL: https://issues.apache.org/jira/browse/HIVE-1609 > Project: Hadoop Hive > Issue Type: New Feature > Components: Metastore >Reporter: Ajay Kidave >Assignee: Ajay Kidave > Fix For: 0.7.0 > > Attachments: hive_1609.patch, hive_1609_2.patch, hive_1609_3.patch > > > The metastore needs to have support for returning a list of partitions based > on user specified filter conditions. This will be useful for tools which need > to do partition pruning. Howl is one such use case. The way partition pruning > is done during hive query execution need not be changed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1609) Support partition filtering in metastore
[ https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1609: - Status: Resolved (was: Patch Available) Hadoop Flags: [Reviewed] Resolution: Fixed Committed. Thanks Ajay! > Support partition filtering in metastore > > > Key: HIVE-1609 > URL: https://issues.apache.org/jira/browse/HIVE-1609 > Project: Hadoop Hive > Issue Type: New Feature > Components: Metastore >Reporter: Ajay Kidave >Assignee: Ajay Kidave > Fix For: 0.7.0 > > Attachments: hive_1609.patch, hive_1609_2.patch, hive_1609_3.patch > > > The metastore needs to have support for returning a list of partitions based > on user specified filter conditions. This will be useful for tools which need > to do partition pruning. Howl is one such use case. The way partition pruning > is done during hive query execution need not be changed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1609) Support partition filtering in metastore
[ https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kidave updated HIVE-1609: -- Status: Patch Available (was: Open) > Support partition filtering in metastore > > > Key: HIVE-1609 > URL: https://issues.apache.org/jira/browse/HIVE-1609 > Project: Hadoop Hive > Issue Type: New Feature > Components: Metastore >Reporter: Ajay Kidave >Assignee: Ajay Kidave > Fix For: 0.7.0 > > Attachments: hive_1609.patch, hive_1609_2.patch, hive_1609_3.patch > > > The metastore needs to have support for returning a list of partitions based > on user specified filter conditions. This will be useful for tools which need > to do partition pruning. Howl is one such use case. The way partition pruning > is done during hive query execution need not be changed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1609) Support partition filtering in metastore
[ https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kidave updated HIVE-1609: -- Attachment: hive_1609_3.patch Thanks for the review comments. I was on vacation for few days, sorry for late response. I have attached a new patch changing the parser to ANTLR. Added few more tests to check for parser issues. @John, * With new DataNucleus version, the default select query for a object fetch is a select distinct on all the columns. This does not work on LONGVARCHAR columns since they are not comparable (in derby). Setting default-fetch-group="false" menas that the distinct would not be applied for the viewtext columns. These columns would be fetched lazily when required. * I would like to let HIVE-1539 make the change as required, this current feature requires only DataNucleus 2.1.1. Since 2.2 is not released yet, we can wait for the 2.2 release before moving to that or HIVE-1539 can do the required testing before using the unreleased version. * When HIVE-1626 changes to the custom Stack implementation, it would be a one line import change for this patch. * I have run checkstyle and fixed the issues. Only one new issue reported is "Got an exception - java.lang.RuntimeException: Unable to get class information for MetaException." This is because checkstyle is not resolving generated classes, this issue is already present for other files referencing generated classes. @Paul * get_partitions_ps can be updated to use the new filtering method. This can be done separately since I am not sure of the existing use cases of get_partitions_ps and the howl use cases do not require this. Same applies for adding a new listPartitionNamesByFilter function. I can open a JIRA for these if required. * Test cases are present for the function added in this patch. If we add the above two, tests would be required for that along with the patch. > Support partition filtering in metastore > > > Key: HIVE-1609 > URL: https://issues.apache.org/jira/browse/HIVE-1609 > Project: Hadoop Hive > Issue Type: New Feature > Components: Metastore >Reporter: Ajay Kidave >Assignee: Ajay Kidave > Fix For: 0.7.0 > > Attachments: hive_1609.patch, hive_1609_2.patch, hive_1609_3.patch > > > The metastore needs to have support for returning a list of partitions based > on user specified filter conditions. This will be useful for tools which need > to do partition pruning. Howl is one such use case. The way partition pruning > is done during hive query execution need not be changed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1609) Support partition filtering in metastore
[ https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1609: - Status: Open (was: Patch Available) > Support partition filtering in metastore > > > Key: HIVE-1609 > URL: https://issues.apache.org/jira/browse/HIVE-1609 > Project: Hadoop Hive > Issue Type: New Feature > Components: Metastore >Reporter: Ajay Kidave > Fix For: 0.7.0 > > Attachments: hive_1609.patch, hive_1609_2.patch > > > The metastore needs to have support for returning a list of partitions based > on user specified filter conditions. This will be useful for tools which need > to do partition pruning. Howl is one such use case. The way partition pruning > is done during hive query execution need not be changed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1609) Support partition filtering in metastore
[ https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kidave updated HIVE-1609: -- Status: Patch Available (was: Open) Release Note: Added support for a new listPartitionsByFilter API in HiveMetaStoreClient. This returns the list of partitions matching a specified partition filter. The filter supports "=", "!=", ">", "<", ">=", "<=" and "LIKE" operations on partition keys of type string. "AND" and "OR" logical operations are supported in the filter. So for example, for a table having partition keys country and state, the filter can be 'country = "USA" AND (state = "CA" OR state = "AZ")' > Support partition filtering in metastore > > > Key: HIVE-1609 > URL: https://issues.apache.org/jira/browse/HIVE-1609 > Project: Hadoop Hive > Issue Type: New Feature > Components: Metastore >Reporter: Ajay Kidave > Fix For: 0.7.0 > > Attachments: hive_1609.patch, hive_1609_2.patch > > > The metastore needs to have support for returning a list of partitions based > on user specified filter conditions. This will be useful for tools which need > to do partition pruning. Howl is one such use case. The way partition pruning > is done during hive query execution need not be changed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1609) Support partition filtering in metastore
[ https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kidave updated HIVE-1609: -- Attachment: hive_1609_2.patch Thanks for the review Carl. Javacc is already used in the hive serde code, so it is not a completely new dependency for hive. Javacc has issues with generating proper errors for multi-line inputs, since we are using it for a small filter string only, this issue should not be seen. The build approach is same as taken in serde, i.e the code is regenerated only if javacc.home is defined. Regarding throwing Unknown[DB|Table]Exception, it would require an extra database call to first check whether the database is valid. So I have changed it to throw a NoSuchObjectException saying db.table does not exist if the getMTable operation fails. I have attached a patch which addresses the other issues. > Support partition filtering in metastore > > > Key: HIVE-1609 > URL: https://issues.apache.org/jira/browse/HIVE-1609 > Project: Hadoop Hive > Issue Type: New Feature > Components: Metastore >Reporter: Ajay Kidave > Fix For: 0.7.0 > > Attachments: hive_1609.patch, hive_1609_2.patch > > > The metastore needs to have support for returning a list of partitions based > on user specified filter conditions. This will be useful for tools which need > to do partition pruning. Howl is one such use case. The way partition pruning > is done during hive query execution need not be changed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1609) Support partition filtering in metastore
[ https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-1609: - Status: Open (was: Patch Available) > Support partition filtering in metastore > > > Key: HIVE-1609 > URL: https://issues.apache.org/jira/browse/HIVE-1609 > Project: Hadoop Hive > Issue Type: New Feature > Components: Metastore >Reporter: Ajay Kidave > Fix For: 0.7.0 > > Attachments: hive_1609.patch > > > The metastore needs to have support for returning a list of partitions based > on user specified filter conditions. This will be useful for tools which need > to do partition pruning. Howl is one such use case. The way partition pruning > is done during hive query execution need not be changed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1609) Support partition filtering in metastore
[ https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-1609: - Fix Version/s: 0.7.0 (was: 0.6.0) Affects Version/s: (was: 0.5.0) > Support partition filtering in metastore > > > Key: HIVE-1609 > URL: https://issues.apache.org/jira/browse/HIVE-1609 > Project: Hadoop Hive > Issue Type: New Feature > Components: Metastore >Reporter: Ajay Kidave > Fix For: 0.7.0 > > Attachments: hive_1609.patch > > > The metastore needs to have support for returning a list of partitions based > on user specified filter conditions. This will be useful for tools which need > to do partition pruning. Howl is one such use case. The way partition pruning > is done during hive query execution need not be changed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1609) Support partition filtering in metastore
[ https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kidave updated HIVE-1609: -- Status: Patch Available (was: Open) Fix Version/s: 0.6.0 > Support partition filtering in metastore > > > Key: HIVE-1609 > URL: https://issues.apache.org/jira/browse/HIVE-1609 > Project: Hadoop Hive > Issue Type: New Feature > Components: Metastore >Affects Versions: 0.5.0 >Reporter: Ajay Kidave > Fix For: 0.6.0 > > Attachments: hive_1609.patch > > > The metastore needs to have support for returning a list of partitions based > on user specified filter conditions. This will be useful for tools which need > to do partition pruning. Howl is one such use case. The way partition pruning > is done during hive query execution need not be changed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1609) Support partition filtering in metastore
[ https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kidave updated HIVE-1609: -- Attachment: hive_1609.patch Attached patch with support for a new metastore API which returns list of partitions matching specified string filter. Thrift does not support recursive nested structures, so the filter is specified as a string instead of an expression object. The datanucleus jar version is upgraded to get support for the JDOQL operations needed (a clean build is required at the root level to remove the older version of the datanucleus jars from build/ivy/lib). > Support partition filtering in metastore > > > Key: HIVE-1609 > URL: https://issues.apache.org/jira/browse/HIVE-1609 > Project: Hadoop Hive > Issue Type: New Feature > Components: Metastore >Affects Versions: 0.5.0 >Reporter: Ajay Kidave > Attachments: hive_1609.patch > > > The metastore needs to have support for returning a list of partitions based > on user specified filter conditions. This will be useful for tools which need > to do partition pruning. Howl is one such use case. The way partition pruning > is done during hive query execution need not be changed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
