[jira] Updated: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar
[ https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-1546: --- Status: Patch Available (was: Open) Ability to plug custom Semantic Analyzers for Hive Grammar -- Key: HIVE-1546 URL: https://issues.apache.org/jira/browse/HIVE-1546 Project: Hadoop Hive Issue Type: Improvement Components: Metastore Affects Versions: 0.7.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.7.0 Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, hive-1546_2.patch, hooks.patch, Howl_Semantic_Analysis.txt It will be useful if Semantic Analysis phase is made pluggable such that other projects can do custom analysis of hive queries before doing metastore operations on them. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar
[ https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-1546: --- Attachment: hooks.patch @John, Neat work with hook interfaces. I was able to use the proposed one as it is without any modifications. Attaching the patch for incorporating these interfaces based on hooks in Hive. Ability to plug custom Semantic Analyzers for Hive Grammar -- Key: HIVE-1546 URL: https://issues.apache.org/jira/browse/HIVE-1546 Project: Hadoop Hive Issue Type: Improvement Components: Metastore Affects Versions: 0.7.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.7.0 Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, hive-1546_2.patch, hooks.patch, Howl_Semantic_Analysis.txt It will be useful if Semantic Analysis phase is made pluggable such that other projects can do custom analysis of hive queries before doing metastore operations on them. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar
[ https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-1546: --- Attachment: Howl_Semantic_Analysis.txt @Namit, As far as I understand, Hooks were designed to run before or after the query executes, not while query is getting executing. By executing a query I mean (generating AST, doing semantic analysis of that AST which includes necessary metadata operations, building a query plan and running the plan). If that was how hook were intended to be used I am not clear how they will be used in this scenario. In Howl its not just a list of metadata operations that we want to allow or disallow, there is a bit more semantic analysis that we do which includes doing metadata operation on metadata repository. Its not really before or after query is executed, it is one of the parts of query processing itself. You also hinted that hooks can be modified and more information can be provided in those methods. If thats what we do then I think we are loosing semantics on hooks which is something which is fired before and after the query executes, as hook will now be running while query is executing. Its entirely possible that I have misunderstood hooks and their intended usage. You obviously know more then so feel free to correct me. I am also attaching what semantic analysis that we are doing for everyone's reference for better context. Ability to plug custom Semantic Analyzers for Hive Grammar -- Key: HIVE-1546 URL: https://issues.apache.org/jira/browse/HIVE-1546 Project: Hadoop Hive Issue Type: Improvement Components: Metastore Affects Versions: 0.7.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.7.0 Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, hive-1546_2.patch, Howl_Semantic_Analysis.txt It will be useful if Semantic Analysis phase is made pluggable such that other projects can do custom analysis of hive queries before doing metastore operations on them. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar
[
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-1546:
---
Attachment: hive-1546-4.patch
After some recent changes, It seems its necessary to do
{code}
conf.set(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY.varname, false);
{code}
in the test-case, otherwise test-case just hangs. Added it in my test-case. No
other changes in the patch (as compared to previous patch) otherwise.
Ability to plug custom Semantic Analyzers for Hive Grammar
--
Key: HIVE-1546
URL: https://issues.apache.org/jira/browse/HIVE-1546
Project: Hadoop Hive
Issue Type: Improvement
Components: Metastore
Affects Versions: 0.7.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Fix For: 0.7.0
Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch,
hive-1546_2.patch
It will be useful if Semantic Analysis phase is made pluggable such that
other projects can do custom analysis of hive queries before doing metastore
operations on them.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar
[ https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-1546: --- Attachment: hive-1546-3.patch Thanks for quick feedback, John. New patch with the approach in first patch incorporating review comments that followed it. For method signature, I took the first suggestion you had and made it void and figured out a way how to deal with it in Howl. Test cases added. Ability to plug custom Semantic Analyzers for Hive Grammar -- Key: HIVE-1546 URL: https://issues.apache.org/jira/browse/HIVE-1546 Project: Hadoop Hive Issue Type: Improvement Components: Metastore Affects Versions: 0.7.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.7.0 Attachments: hive-1546-3.patch, hive-1546.patch, hive-1546_2.patch It will be useful if Semantic Analysis phase is made pluggable such that other projects can do custom analysis of hive queries before doing metastore operations on them. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar
[ https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-1546: --- Attachment: (was: hive-1546-3.patch) Ability to plug custom Semantic Analyzers for Hive Grammar -- Key: HIVE-1546 URL: https://issues.apache.org/jira/browse/HIVE-1546 Project: Hadoop Hive Issue Type: Improvement Components: Metastore Affects Versions: 0.7.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.7.0 Attachments: hive-1546-3.patch, hive-1546.patch, hive-1546_2.patch It will be useful if Semantic Analysis phase is made pluggable such that other projects can do custom analysis of hive queries before doing metastore operations on them. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar
[ https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-1546: --- Attachment: hive-1546_2.patch Attaching a new patch. bq. In addition, I think we should introduce a new interface HiveSemanticAnalyzer and have the factory produce that, since factories should generally produce interfaces rather than classes (e.g. so that dynamic proxies can be used). This got me thinking and I have redesigned the patch, so now we have factory producing factory enabling dynamic proxying (abstractions and extensibility !) Your other comments got affected by it because of it but I have taken care of them. Signature for method handleGenricFileFormat() was String[] because description for task and task itself is created in analyzeCreateTable() and analyzeTableFileFormat() so that method cant be void, as it needs to return whatever information it gathered to its calling method. Since at present we have no such file format, return value is ignored by calling methods. However I changed it to MapString,String because that seems more useful then String[]. Added test-cases for all three cases you suggested. Ability to plug custom Semantic Analyzers for Hive Grammar -- Key: HIVE-1546 URL: https://issues.apache.org/jira/browse/HIVE-1546 Project: Hadoop Hive Issue Type: Improvement Components: Metastore Affects Versions: 0.7.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.7.0 Attachments: hive-1546.patch, hive-1546_2.patch It will be useful if Semantic Analysis phase is made pluggable such that other projects can do custom analysis of hive queries before doing metastore operations on them. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar
[ https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-1546: --- Attachment: hive-1546.patch Attached patch adds the capability to Hive so that custom semantic analysis of query is possible before it is handed over to Hive. Plus there are few other miscellaneous refactoring around it. Changes include: * Addition of SemanticAnalyzerFactoryInterface. If conf has a particular variable specified, a custom analyzer will be loaded and used, otherwise existing Hive Semantic Analyzer will be used. So, default behavior is preserved. * Changed visibility of few methods in DDLSemanticAnalyzer and SemanticAnalyzer from private to protected as I wanted to override them in my custom analyzer. * Changed file format specification in grammar, so that it can optionally take two more parameters (InputDriver and OutputDriver) in addition to InputFormat and OutputFormat. These are optional, so preserves the default behavior. * In file format specification, currently SequenceFile, TextFile and RCFile are supported through keyword. Expanded that production so to accept an identifier so that its possible to provide support for more file formats without needing to change Hive grammar every time. Currently, that token results in exception since there are none, but when we add support for other file formats that could be changed. This preserves current behavior. Note that there are no new test cases since its mostly code restructuring and doesnt add/modify current behavior, thus passing existing tests should suffice. I should point out most of these changes are driven by Howl and would like to thank John for suggesting the initial approach for these changes. Ability to plug custom Semantic Analyzers for Hive Grammar -- Key: HIVE-1546 URL: https://issues.apache.org/jira/browse/HIVE-1546 Project: Hadoop Hive Issue Type: Improvement Components: Metastore Reporter: Ashutosh Chauhan Attachments: hive-1546.patch It will be useful if Semantic Analysis phase is made pluggable such that other projects can do custom analysis of hive queries before doing metastore operations on them. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar
[ https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1546: - Fix Version/s: 0.7.0 Affects Version/s: 0.7.0 Ability to plug custom Semantic Analyzers for Hive Grammar -- Key: HIVE-1546 URL: https://issues.apache.org/jira/browse/HIVE-1546 Project: Hadoop Hive Issue Type: Improvement Components: Metastore Affects Versions: 0.7.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.7.0 Attachments: hive-1546.patch It will be useful if Semantic Analysis phase is made pluggable such that other projects can do custom analysis of hive queries before doing metastore operations on them. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
