[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-10-04 Thread Ashutosh Chauhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917851#action_12917851
 ] 

Ashutosh Chauhan commented on HIVE-1546:


I did it in junit form because John suggested it that way in his earlier 
comment:

{quote}
* We need a test for loading a variation on the default semantic analyzer 
in order to exercise the pluggable configuration. You can create a subclass of 
the default analyzer (under ql/src/test/org/apache/hadoop/hive/ql/parse) to 
inject some mock behavior change.
{quote}

I also feel junit test is better suited for this kind of behavioral testing of 
code paths (which exercises interface points) rather then forcing through 
string comparison ways of test/queries/*  which are more end-to-end tests for 
hive. Further if we add dummy hook name in data/conf/hive-site.xml then that 
dummy hook will get loaded and all the subsequent tests will have it too. Do we 
want it that way?

> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch, hooks.patch, Howl_Semantic_Analysis.txt
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-10-04 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917677#action_12917677
 ] 

Namit Jain commented on HIVE-1546:
--

I will take a look in more detail, but overall it looks good. I had the 
following comments:

1. Instead of TestSemanticAnalyzerHookLoading.java, add tests in 
test/queries/clientpositive and test/queries/clientnegative
2. Do you want to set the value of hive.semantic.analyzer.hook to a dummy value 
in data/conf/hive-site.xml for the unit tests ?
Can something meaningful be printed here, which can be used for comparing ?


> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch, hooks.patch, Howl_Semantic_Analysis.txt
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-30 Thread Ashutosh Chauhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12916758#action_12916758
 ] 

Ashutosh Chauhan commented on HIVE-1546:


Hey Namit,

Would you like to review it?

> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch, hooks.patch, Howl_Semantic_Analysis.txt
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-30 Thread John Sichi (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12916756#action_12916756
 ] 

John Sichi commented on HIVE-1546:
--

Great.  I'm actually out on vacation until the end of next week, so if you want 
to get this one committed before then, ping Namit (njain at facebook dot com) 
and get him to review.  Otherwise, I'll push it through once I get back.


> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch, hooks.patch, Howl_Semantic_Analysis.txt
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-15 Thread John Sichi (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12909822#action_12909822
 ] 

John Sichi commented on HIVE-1546:
--

For the Hive part, it will be good if you can keep the same coverage as you 
have in the new unit tests from your last patch--the same cases can be 
validated regardless of hook vs factory.


> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch, Howl_Semantic_Analysis.txt
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-15 Thread Ashutosh Chauhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12909799#action_12909799
 ] 

Ashutosh Chauhan commented on HIVE-1546:


@John,
I took a look at the proposed new interface. I didn't go full in-depth into it 
yet to ascertain whether this works for Howl as desired or not, but I got the 
high level idea. It is a major redesign from the current patch. It will require 
some amount of design and dev work. This also implies all our testing (both 
unit and integration) which we have done on this patch till now is redundant 
and we need to redo that again as well. 

> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch, Howl_Semantic_Analysis.txt
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-14 Thread John Sichi (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12909549#action_12909549
 ] 

John Sichi commented on HIVE-1546:
--

I think we can start off with this interface:

{noformat}
/**
 * HiveSemanticAnalyzerHook allows Hive to be extended with custom
 * logic for semantic analysis of QL statements.  This interface
 * and any Hive internals it exposes are currently 
 * "limited private and evolving" (unless otherwise stated elsewhere)
 * and intended mainly for use by the Howl project.
 *
 *
 *
 * Note that the lifetime of an instantiated hook object is scoped to
 * the analysis of a single statement; hook instances are never reused.
 */
public interface HiveSemanticAnalyzerHook {
  /**
   * Invoked before Hive performs its own semantic analysis on
   * a statement.  The implementation may inspect the statement AST and
   * prevent its execution by throwing a SemanticException.
   * Optionally, it may also augment/rewrite the AST, but must produce
   * a form equivalent to one which could have
   * been returned directly from Hive's own parser.
   *
   * @param context context information for semantic analysis
   *
   * @param ast AST being analyzed and optionally rewritten
   *
   * @return replacement AST (typically the same as the original AST unless the
   * entire tree had to be replaced; must not be null)
   */
  public ASTNode preAnalyze(
HiveSemanticAnalyzerHookContext context,
ASTNode ast) throws SemanticException;

  /**
   * Invoked after Hive performs its own semantic analysis on a
   * statement (including optimization).  
   * Hive calls postAnalyze on the same hook object
   * as preAnalyze, so the hook can maintain state across the calls.
   *
   * @param context context information for semantic analysis
   *
   * @param rootTasks root tasks produced by semantic analysis;
   * the hook is free to modify this list or its contents
   */
  public void postAnalyze(
HiveSemanticAnalyzerHookContext context,
List> rootTasks) throws SemanticException;
}
{noformat}

Plus companion context interface to be passed in from Hive:

{noformat}
/**
 * Context information provided by Hive to implementations of 
HiveSemanticAnalyzerHook.
 */
public interface HiveSemanticAnalyzerHookContext {
  /**
   * @return the Hive db instance; hook implementations can use this for 
purposes such as getting configuration information or making metastore calls
   */
  public Hive getHive();
}
{noformat}

The reason for the context is so that later, if we need to make more information
available to the hook, we just add new getters to the context object
(rather than adding new parameters to methods such as preAnalyze,
which would break existing hook implementations).

Unlike pre/post exec hooks, we will only allow one hook to be
configured (rather than a list).  If someone really wants to run
more than one, they can write a list hook implementation which delegates
to multiple.  The conf variable used to load the hook class will be
hive.semantic.analyzer.hook since it's no longer a factory.

We also need an insulation class:

{noformat}
public abstract class AbstractSemanticAnalyzerHook 
  implements HiveSemanticAnalyzerHook {

  public ASTNode preAnalyze(
HiveSemanticAnalyzerHookContext context,
ASTNode ast) throws SemanticException
  {
return ast;
  }

  public void postAnalyze(
HiveSemanticAnalyzerHookContext context,
List> rootTasks) throws SemanticException
  {
  }
}
{noformat}

Hook implementations should extend this to avoid breaking when we add
new methods to the HiveSemanticAnalyzerHook interface later.

By using the hook approach, we limit the dependency exposure: only ASTNodes,
Tasks, and the org.apache.hadoop.hive.ql.metadata.Hive class can be accessed
for now.  If we decide to open it up more later, that will be via an
agreed-upon decision in a new patch (rather than via ad hoc dependency
creep).

Instead of invoking the hook from many different places inside of
SemanticAnalyzer, we'll start with just invoking it only pre- and post-
the call to sem.analyze.  More invocation points
(e.g. pre-optimization) can be added later on an as-needed basis.

Howl's table property additions can be saved during the preAnalyze
call and then applied to the Task during the postAnalyze call (since
the hook is allowed to maintain state); this stays very close to the
approach in the current patch.  Another way to do this is to
not use a postAnalyze at all, and instead just rewrite the AST to
splice in the additional TBLPROPERTIES up front.  Both ways are a
little messy, but either way should be fine.

For handleGenericFileFormat, I think we can deal with it by having
Howl edit the AST to delete the STORED AS clause entirely during
preAnalyze (in cases where it accepts the storage format).  That way
Hive can continue to reject it unconditionally, and we can avoid
adding a third in

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-14 Thread John Sichi (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12909432#action_12909432
 ] 

John Sichi commented on HIVE-1546:
--

@Ashutosh:  I will get back to you soon with a proposal.  Based on what I see 
in your Howl_Semantic_Analysis.txt code, I think the interface can be very 
small.


> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch, Howl_Semantic_Analysis.txt
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-14 Thread Ashutosh Chauhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12909318#action_12909318
 ] 

Ashutosh Chauhan commented on HIVE-1546:


@Namit,

Continuing from discussion yesterday w.r.t new interface/hooks that you were 
suggesting. As far as I understood you have two primary concerns:
 * You do not want to have config variable enabled SemanticAnalyzer factory 
producing different Analyzers. Rather, you want to have only one  
HiveSemanticAnalyzer.
 * You also dont want others to extend SemanticAnalyzer, DDLSemanticAnalyzer 
and other Hive classes.

As a solution to this, there will be new hooks introduced. These pre and post 
hooks will be called by Hive at different times during Semantic Analysis phase. 
To me, it seems there are few potential problems: 
 * Hive will need to call these hooks from each and every method of its 
Analyzer which opens up whole lot more integration points to external tools 
(like Howl) and burden on those Analyzers then the current approach of factory 
enabled instantiation and extending of those classes.
 * Either there will be a one generic hook or many different hooks. If there is 
only one generic hook, then passed in information also need to be generic and 
as a result it will be a root of AST Node. If so, in any particular hook I am 
using I need to traverse the whole AST to get the node I am interested in. 
 * If the hooks are more specific, then more specific information can be passed 
in which is more useful, but then you will have long list of such hooks.
 * Even if these concerns are taken care of, it seems that all of Howl's 
requirements may not get satisfied (like getting hold of DDLTask and modifying 
it, doing metadata operations etc.) I can elaborate on them further once I can 
see what you are proposing.

Clearly from my description, you can see that I didnt get what you were 
envisioning and how is it better then current design. I must be missing 
something. Would you like to propose what you were suggesting? The attached 
patch and Howl classes on this jira should be sufficient to spec out Howl's 
requirements, if you need them.  

> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch, Howl_Semantic_Analysis.txt
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-03 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12906185#action_12906185
 ] 

Namit Jain commented on HIVE-1546:
--

I agree it would be good to have a meeting.

btw, the hooks I was referrring to and Pre/Post Execution Statement level 
hooks, with the following signature:

public interface PostExecute {

  /**
   * The run command that is called just before the execution of the query.
   *
   * @param sess
   *  The session state.
   * @param inputs
   *  The set of input tables and partitions.
   * @param outputs
   *  The set of output tables, partitions, local and hdfs directories.
   * @param lInfo
   *   The column level lineage information.
   * @param ugi
   *  The user group security information.
   */
  void run(SessionState sess, Set inputs,
  Set outputs, LineageInfo lInfo,
  UserGroupInformation ugi) throws Exception;
}


Looking at the spec., it looks like a subset of DDLs need to be supported, 
which can be easily accomplished via the hook.
If need be, we can pass more info. in the hook - there was a plan to add job 
level hook also, which is not checked in,
but via which the configuration etc. can be changed.

> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-03 Thread John Sichi (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12906041#action_12906041
 ] 

John Sichi commented on HIVE-1546:
--

@Namit:  the option of putting Howl directly into Hive was previously proposed 
but dropped for the same reasons Alan mentioned above.  Regarding hooks, could 
you point me to the hook you're referring to?  I don't believe Pre/Post have 
enough information currently, do they?

@Carl: I don't think Howl cares about the query processing stuff like {Task and 
FetchTask,QB,QBParseInfo,QBMetaData,QBJoinTree}.  For the others, it's not any 
time we touch them; it's only when we make breaking changes.  And since Howl is 
also open source, it's not like these are opaque dependencies.  We would need 
to do the same impact analysis if we used the contrib approach, right?  I don't 
see a big difference between the two except with contrib we get the convenience 
of immediate compilation errors to tell us something broke.  A continuous 
integration setup for Howl would take us close to that.

@All:  maybe we should set up a f2f meeting to hash this out?


> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-02 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905767#action_12905767
 ] 

Namit Jain commented on HIVE-1546:
--

Sorry on jumping on this late.

I quickly reviewed http://wiki.apache.org/pig/Howl/HowlCliFuncSpec, and it 
seems like most of the functionality is already
present in hive. So, we need a way to restrict other types of statements - is 
that a fair statement ?

If there is a slight change needed in hive (for some howl behavior), we can add 
it to hive ? 
Why do we need a brand new client ?


> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-02 Thread Carl Steinbach (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905757#action_12905757
 ] 

Carl Steinbach commented on HIVE-1546:
--

I gather from Ashutosh's latest patch that you want to do the following:

* Provide your own implementation of HiveSemanticAnalyzerFactory.
* Subclass SemanticAnalyzer
* Subclass DDLSemanticAnalzyer

I looked at the public and protected members in these classes and think
that at a minimum we would have to mark the following classes as limited
private and evolving:

* HiveSemanticAnalyzerFactory
* BaseSemanticAnalyzer
* SemanticAnalyzer
* DDLSemanticAnalyzer
* ASTNode
* HiveParser (i.e. Hive's ANTLR grammar)
* SemanticAnalyzer Context (org.apache.hadoop.hive.ql.Context)
* Task and FetchTask
* QB
* QBParseInfo
* QBMetaData
* QBJoinTree
* CreateTableDesc

So anytime we touch one of these classes we would need to coordinate with the 
Howl folks to make sure we aren't breaking one of their plugins? I don't think 
this is a good tradeoff if the main benefit we can expect is a simpler build 
and release process for Howl.

> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-02 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905752#action_12905752
 ] 

Namit Jain commented on HIVE-1546:
--

Would it be possible to do it via a hook ?

Do you want to allow a subset of operations ? The hook is not very advanced 
right now, and you cannot change the query plan etc.
But, it might be good enough for disallowing a class of statements. We can add 
more parameters to the hook if need be.

That way, the change will be completely outside hive, and we will be able to 
use the existing client, but with a limited functionality.

> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-02 Thread Alan Gates (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905711#action_12905711
 ] 

Alan Gates commented on HIVE-1546:
--

Using the definitions given in HADOOP-5073, can we call this interface limited 
private and evolving?  We (the Howl team) know it will continue to change, and 
we understand Hive's desire not to make this a public API.  But checking Howl 
code into Hive just muddles things and makes our build and release process 
harder.  

> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-02 Thread Carl Steinbach (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905707#action_12905707
 ] 

Carl Steinbach commented on HIVE-1546:
--

I'm +1 on the approach outlined by John.

> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-02 Thread John Sichi (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905706#action_12905706
 ] 

John Sichi commented on HIVE-1546:
--

That's fine with me if it doesn't drag in unrelated dependencies.  I would vote 
for contrib, with the plugin mechanism remaining the same as Ashutosh has 
defined it, but with the config parameter explicitly defining it as intended 
for internal use only for now.

Ashutosh, could you run this proposal by the Howl team and see if that is 
acceptable?


> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-02 Thread Carl Steinbach (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905692#action_12905692
 ] 

Carl Steinbach commented on HIVE-1546:
--

What do you think of this option: we check the Howl SemanticAnalyzer into the 
Hive source tree and provide a config option that optionally enables it? This 
gives Howl the features they need without making the SemanticAnalyzer API 
public.


> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-02 Thread John Sichi (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905687#action_12905687
 ] 

John Sichi commented on HIVE-1546:
--

It's the usual tradeoffs on copy-and-paste vs factoring.  There's a significant 
amount of DDL processing code which can be shared, and that will continue to 
grow as we add new features (e.g. GRANT/REVOKE) which are applicable to both.


> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-02 Thread Carl Steinbach (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905686#action_12905686
 ] 

Carl Steinbach commented on HIVE-1546:
--

bq. we've agreed at the high level on the approach of creating Howl as a 
wrapper around Hive

I thought Howl was supposed to be a wrapper around (and replacement for) the 
Hive metastore, not all of Hive.

I think there are clear advantages to Hive and Howl sharing the same metastore 
code as long as they access this facility through the public API, but can't say 
the same for the two projects using the same CLI code if it means allowing 
external projects to depend on loosely defined set of internal APIs. What 
benefits are we hoping to achieve by having Howl and Hive share the same CLI 
code, especially if Howl is only interested in a small part of it? What are the 
drawbacks of instead encouraging the Howl project to copy the CLI code and 
maintain their own version?


> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-02 Thread John Sichi (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905675#action_12905675
 ] 

John Sichi commented on HIVE-1546:
--

New dependencies:   we don't prevent anyone from using it, but we can Javadoc 
it as unstable.  We can work out the language now in an updated patch since 
there's currently no Javadoc on the factory interface.

Dependencies on AST/ANTLR:  it does make such changes more expensive in terms 
of impact analysis and migration, but it doesn't really prevent us in any way, 
does it?

Given that we've agreed at the high level on the approach of creating Howl as a 
wrapper around Hive (reusing as much as possible of what's already there), can 
you suggest an alternative mechanism that addresses the requirements while 
minimizing the injection of Howl behavior directly into Hive itself?  If it 
were something generic like a bitmask of allowed operations, I could kind of 
see it, but the validation logic is more involved than that (and may become 
even more so over time).  I wasn't able to come up with anything clean on that 
front myself, which is why I suggested the factoring approach to Pradeep 
originally.  Apologies for not getting stuff aired out sooner.


> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-02 Thread Carl Steinbach (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905665#action_12905665
 ] 

Carl Steinbach commented on HIVE-1546:
--

bq. Can we get agreement from the Howl team that even though we're introducing 
this dependency now, we will not let its existence hinder future semantic 
analyzer refactoring within Hive?

What about other projects that use this feature? How do we get them to agree to 
this, or how do we prevent them from using it? The new configuration property 
is documented in hive-default.xml, which implies that it's open to everyone.

bq. one possible refinement would be to limit the public interface to just 
validation (as opposed to full semantic analysis). In that case, we would have 
HiveStmtValidatorFactory producing HiveStmtValidator with just a single method 
validate().

This reduces the scope of the dependency, but doesn't eliminate it. Plugins 
would presumably depend on the structure of the AST that they are trying to 
validate, which in turn would limit our ability to refactor the grammar or to 
replace ANTLR with another parser generator.

> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-02 Thread John Sichi (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905656#action_12905656
 ] 

John Sichi commented on HIVE-1546:
--

For the last sentence, I meant "If Howl's CLI customized behavior is going to 
need to influence more than just validation"

> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-02 Thread John Sichi (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905655#action_12905655
 ] 

John Sichi commented on HIVE-1546:
--

@Carl:  I understand your concern, but this seemed like the least intrusive 
approach as opposed to continually patching Hive to refine what Howl's CLI 
wants to support at a given point in time (which really has nothing to do with 
Hive).  The override approach allows that behavior to be factored completely 
out into Howl.  A number of our existing extensibility interfaces (e.g. 
StorageHandler) already have similar issues regarding impact from continual 
refactoring, so I expect an across-the-board SPI stabilization effort to be 
required in the future (with corresponding migrations from old to new).  This 
will need to be part of that effort.

@Ashutosh:  I hit the hang you mentioned, so I can retry tests with your latest 
patch.  But let's resolve the approach with Carl first.  In particular, can we 
get agreement from the Howl team that even though we're introducing this 
dependency now, we will not let its existence hinder future semantic analyzer 
refactoring within Hive?  As long as we all stay in frequent communication, we 
can make that work.

@Both:  one possible refinement would be to limit the public interface to just 
validation (as opposed to full semantic analysis).  In that case, we would have 
HiveStmtValidatorFactory producing HiveStmtValidator with just a single method 
validate().  This would also remove the unpleasantness of having a factory 
returning a base class rather than an interface.  However, if CLI is going to 
need to do more than just validation, then this isn't good enough.


> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-02 Thread Ashutosh Chauhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905640#action_12905640
 ] 

Ashutosh Chauhan commented on HIVE-1546:


@Carl,

* Ya, the main motivating use case is to provide an alternate DDL CLI tool 
(hopefully not crippled *smiles*). Reason for that is to enforce certain 
use-cases on DDL commands in Howl CLI. More details on that are here: 
http://wiki.apache.org/pig/Howl/HowlCliFuncSpec If you have questions why we 
are making such decisions in Howl, I will encourage you to post it on howl-dev 
list and we can discuss it there. howl...@yahoogroups.com
* I dont understand what do you mean by making "SemanticAnalyzer a public API". 
This patch is just letting other tools to do some semantic analysis of the 
query and then use Hive to do further processing (if tool chooses to do so). 
Important point here is *other tools*. This in no way enforcing any changes to 
any Hive behavior. Hive can continue to have its own semantic analyzer and do 
any sort of semantic analysis of the query. Hive is making no guarantees to any 
tool.
* Hive doesnt care about INPUTDRIVER and OUTPUTDRIVER and neither this patch is 
asking it to. I dont see any way that its providing any mechanism for defining 
tables in MetaStore that Hive cant read or write to.

@John,
Do you want to make me to any further changes or are we good to go?

> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-01 Thread Carl Steinbach (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905411#action_12905411
 ] 

Carl Steinbach commented on HIVE-1546:
--

@ John: Good to know, but what's the motivation for this change? Was is it 
covered in the back-channel discussions you mentioned above? And is making 
SemanticAnalyzer a public API really a good idea?

> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-01 Thread John Sichi (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905410#action_12905410
 ] 

John Sichi commented on HIVE-1546:
--

Hi Carl,

As mentioned above, this patch originated with some of the early discussions on 
Howl before we had mailing lists set up.


> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-01 Thread Carl Steinbach (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905406#action_12905406
 ] 

Carl Steinbach commented on HIVE-1546:
--

If the main motivation for this ticket is the ability to produce a crippled 
version of the HiveCLI that is only capable of executing DDL, then I think we 
should consider simpler approaches that don't involve making SemanticAnalyzer a 
public API. 

SemanticAnalyzer is in serious need of refactoring. Making this API public will 
severely restrict our ability to do this work in the future.






> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-01 Thread Carl Steinbach (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905390#action_12905390
 ] 

Carl Steinbach commented on HIVE-1546:
--

@Ashutosh: Can you provide some background on what you hope to accomplish with 
this? What is the motivating use case, i.e. what custom SemanticAnalyzer do you 
plan to write?

Also, how are the new INPUTDRIVER and OUTPUTDRIVER properties used? By adding 
these to the Hive grammar it seems like we may be providing a mechanism for 
defining tables in the MetaStore that Hive can't read or write to. If that's 
the case what are your plans for adding this support to Hive?



> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-01 Thread Ashutosh Chauhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905353#action_12905353
 ] 

Ashutosh Chauhan commented on HIVE-1546:


It is the new test that I wrote for this patch TestSemanticAnalyzerLoading.java 
If I dont have this line my test hangs but if I have it, testcase completes 
successfully. How should I go about debugging it?  Do I need to bring up 
zookeeper externally or something? 

> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-09-01 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905342#action_12905342
 ] 

Namit Jain commented on HIVE-1546:
--

We need to debug this. 
For tests, we should be able to turn concurrency on.

Which test is hanging ?

> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546-4.patch, hive-1546.patch, 
> hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-08-31 Thread John Sichi (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904834#action_12904834
 ] 

John Sichi commented on HIVE-1546:
--

+1.  Will commit when tests pass.


> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546.patch, hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-08-30 Thread HBase Review Board (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904397#action_12904397
 ] 

HBase Review Board commented on HIVE-1546:
--

Message from: "Carl Steinbach" 

---
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/713/
---

(Updated 2010-08-30 15:42:15.768310)


Review request for Hive Developers.


Changes
---

Updated diff with 
https://issues.apache.org/jira/secure/attachment/12453466/hive-1546-3.patch


Summary
---

review by JVS


This addresses bug HIVE-1546.
http://issues.apache.org/jira/browse/HIVE-1546


Diffs (updated)
-

  trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 990934 
  trunk/conf/hive-default.xml 990934 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 990934 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveUtils.java 990934 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
990934 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
990934 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java 
990934 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 990934 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveSemanticAnalyzerFactory.java
 PRE-CREATION 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
990934 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
990934 
  
trunk/ql/src/test/org/apache/hadoop/hive/ql/metadata/DummySemanticAnalyzerFactory.java
 PRE-CREATION 
  
trunk/ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSemanticAnalyzerLoading.java
 PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/genericFileFormat.q PRE-CREATION 
  trunk/ql/src/test/queries/clientpositive/inoutdriver.q PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/genericFileFormat.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientpositive/inoutdriver.q.out PRE-CREATION 

Diff: http://review.cloudera.org/r/713/diff


Testing
---


Thanks,

John




> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546-3.patch, hive-1546.patch, hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-08-27 Thread John Sichi (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903746#action_12903746
 ] 

John Sichi commented on HIVE-1546:
--

More specifically, for the last suggestion, where we currently have

{noformat}
  case HiveParser.TOK_TBLRCFILE:
inputFormat = RCFILE_INPUT;
outputFormat = RCFILE_OUTPUT;
shared.serde = COLUMNAR_SERDE;
storageFormat = true;
{noformat}

instead do

{noformat}
  case HiveParser.TOK_TBLRCFILE:
  processGenericFileFormat("RCFILE");
  break;
  case HiveParser.TOK_FILEFORMAT_GENERIC:
  processGenericFileFormat(child.getChild(0).getText());
  break;

...

void processGenericFileFormat(String formatName) {
Map props = handleGenericFileFormat(formatName);
inputFormat = props.get(Constants.FILE_INPUT_FORMAT);
outputFormat = props.get(Constants.FILE_OUTPUT_FORMAT);
shared.serde = props.get(Constants.META_TABLE_SERDE);
   ...
}
{noformat}

Then in Hive's version of handleGenericFileFormat, make it return the right 
info for SEQUENCEFILE/TEXTFILE/RCFILE.

This is just a sketch, not the real code, but I hope it makes sense.


> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546.patch, hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-08-27 Thread John Sichi (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903738#action_12903738
 ] 

John Sichi commented on HIVE-1546:
--

Yes, that is what I was envisioning.  I think the interface as you've specified 
it looks close to the abstract functionality of the semantic analyzer, which is 
what we want (although we should use interfaces such as Set in preference to 
concrete classes such as HashSet, something which is currently crufty 
throughout Hive).

I agree that this could be too involved for your first patch, and we would 
probably need to evolve the HiveSemanticAnalyzer interface anyway.  So, if 
you're not comfortable going there, let's scale it back to the approach in the 
first patch (with HiveSemanticAnalyzerFactory returning BaseSemanticAnalyzer).

handleGenericFileFormat:  if it's only a hook for Howl, then you can have a 
separate method inside of Howl which returns whatever you want, but wrap it 
with a void method which overrides a void one in Hive (and discards the return 
values).  Or, if the idea is to have Hive use this too, then go ahead and add 
Javadoc specifying exactly what the return map is supposed to contain, and then 
convert the existing SEQUENCEFILE/TEXTFILE/RCFILE cases so they go through the 
generic path.  (But still keep them as reserved words rather than literal 
strings for backwards compatibility.)


> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546.patch, hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-08-27 Thread Ashutosh Chauhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903660#action_12903660
 ] 

Ashutosh Chauhan commented on HIVE-1546:


bq. What I was thinking in my previous comment was as follows:
* only one level of factory
* define new interface HiveSemanticAnalyzer (not an abstract class): copy 
signatures of interface methods from public methods on BaseSemanticAnalyzer; 
add Javadoc (I can help with that if any of them are non-obvious)
* when callers get new analyzer instance from factory, they refer to it via 
the HiveSemanticAnalyzer interface only

Are you envisioning something like this in Driver.java
{code}
HiveSemanticAnalyzer sem = HiveSemanticAnalyzerFactory.get(conf);

// Do semantic analysis and plan generation
sem.analyze(tree, ctx);

// validate the plan
sem.validate();

plan = new QueryPlan(command, sem);

// get the output schema
schema = getSchema(sem, conf);
{code} 

Note where {sem} is used. If so, HiveSemanticAnalyzer interface needs to look 
like this:
{code}
public interface HiveSemanticAnalyzer extends Configurable{

  public void analyze(ASTNode ast, Context ctx) throws SemanticException;

  public void validate() throws SemanticException;

  public List getResultSchema();

  public FetchTask getFetchTask();

  public List> getRootTasks();

  public HashSet getInputs();

  public HashSet getOutputs();

  public LineageInfo getLineageInfo();

  public HashMap getIdToTableNameMap();
}

{code}

This to me look like an awkward interface, which has to expose internal details 
of the very class it is trying to hide.
But even if we make an effort to improve on it and try to come up with a  
better interface, this will mean that I need to touch upon lot of critical code 
paths in BaseSemanticAnalyzer, SemanticAnalyzer and DDLSemanticAnalyzer which I 
am not sure I know enough of them to make changes. If this is not what you were 
envisioning then I have missed something in what you were suggesting.

Assuming we dont go this route, I liked the approach in my second patch better 
then from my first patch. I think it provides better abstractions. But if you 
have different opinion, I am fine with the first one as well.

Regarding handleGenericFileFormat, its the point (a). It cant be void, because 
in Howl where we override this it needs to return information to its caller 
which will work on that information. handleGenericFileFormat() can only make 
use of the token and provide back some information. Caller works on that 
information and modifies the task it has created which usually happens to be of 
type DDLWork. And even Hive will need to go through this same code flow if and 
when it decides to add more FileFormats.

> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546.patch, hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-08-26 Thread John Sichi (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903035#action_12903035
 ] 

John Sichi commented on HIVE-1546:
--

Thanks, I took a look.  But factory producing factory seems like overkill here.

What I was thinking in my previous comment was as follows:

* only one level of factory
* define new interface HiveSemanticAnalyzer (not an abstract class):  copy 
signatures of interface methods from public methods on BaseSemanticAnalyzer; 
add Javadoc (I can help with that if any of them are non-obvious)
* when callers get new analyzer instance from factory, they refer to it via the 
HiveSemanticAnalyzer interface only

If that doesn't work for some reason, then we can just use your original 
pattern where the factory returns a class (BaseSemanticAnalyzer) instead of an 
interface.

Regarding handleGenericFileFormat:  I still don't understand.  Your code in 
Hive is ignoring the return value (and the only implementation doesn't return 
anything, it just throws).   So, either

(a) you're planning to do something inside of Howl with it; in that case, the 
Hive method is just a hook for you to intercept, and it should return void

or

(b) you're planning to add some more code inside of Hive which actually does 
something with the return value (e.g. sets the serde+inputformat+outputformat); 
in that case, you need to keep working on the patch to make this happen


> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546.patch, hive-1546_2.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-08-23 Thread HBase Review Board (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12901654#action_12901654
 ] 

HBase Review Board commented on HIVE-1546:
--

Message from: "John Sichi" 

---
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/713/#review1008
---



http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java


Can you break this out into several statements for readability?



http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveUtils.java


This does not belong in HiveUtils.  Can't we just leave it where it is (in 
SemanticAnalyzerFactory) and add a public accessor method for it so that 
extensions can reuse it?




http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java


Why does this return a String []?  The callers don't use it.



http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactoryInterface.java


Putting "Interface" into the interface name is clunky.

Let's call the interface HiveSemanticAnalyzerFactory instead (matching the 
convention for other plugin interfaces such as HiveStorageHandler and 
HiveIndexHandler).

In addition, I think we should introduce a new interface 
HiveSemanticAnalyzer and have the factory produce that, since factories should 
generally produce interfaces rather than classes (e.g. so that dynamic proxies 
can be used).




http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactoryInterface.java


This configuration parameter needs to be defined in HiveConf.java following 
the pattern already in use there.

Also, it needs a new entry in conf/hive-default.xml




http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactoryInterface.java


Our convention for existing Hive plugins is to make them extend 
org.apache.hadoop.conf.Configurable and pass in the configuration only once via 
setConf (rather than to individual methods such as get).

Inside a Configurable, you can get back to a HiveConf with:

new HiveConf(getConf(), YourClass.class)



- John





> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-08-23 Thread HBase Review Board (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12901646#action_12901646
 ] 

HBase Review Board commented on HIVE-1546:
--

Message from: "John Sichi" 

---
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/713/
---

Review request for Hive Developers.


Summary
---

review by JVS


This addresses bug HIVE-1546.
http://issues.apache.org/jira/browse/HIVE-1546


Diffs
-

  
http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
 988318 
  
http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveUtils.java
 988318 
  
http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java
 988318 
  
http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
 988318 
  
http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java
 988318 
  
http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
 988318 
  
http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
 988318 
  
http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java
 988318 
  
http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactoryInterface.java
 PRE-CREATION 

Diff: http://review.cloudera.org/r/713/diff


Testing
---


Thanks,

John




> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-08-23 Thread John Sichi (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12901641#action_12901641
 ] 

John Sichi commented on HIVE-1546:
--

I have published some review comments here:

http://review.cloudera.org/r/713/

In addition:

* We need a test for loading a variation on the default semantic analyzer in 
order to exercise the pluggable configuration.  You can create a subclass of 
the default analyzer (under ql/src/test/org/apache/hadoop/hive/ql/parse) to 
inject some mock behavior change.

* We need a test for the new INPUTDRIVER/OUTPUTDRIVER clauses.  Otherwise, 
people will wonder why they're in the grammar but not referenced elsewhere in 
Hive; the test will give you a chance to explain the Howl dependency.

* We need a negative test for unrecognized generic format.

Let me know if you need help with how to script new positive+negative test 
cases.

(BTW, your patch file header had a glitch in it; we usually use svn diff 
format.)


> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-08-19 Thread John Sichi (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900428#action_12900428
 ] 

John Sichi commented on HIVE-1546:
--

Thanks Ashutosh, I've reassigned this one to you and will get back to you with 
some review comments.


> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: hive-1546.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-08-18 Thread Ashutosh Chauhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900137#action_12900137
 ] 

Ashutosh Chauhan commented on HIVE-1546:


Btw, can someone assign this jira to me and add me to the list of contributors 
so that in future I can do that myself.

> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Ashutosh Chauhan
> Attachments: hive-1546.patch
>
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-08-16 Thread John Sichi (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899206#action_12899206
 ] 

John Sichi commented on HIVE-1546:
--

(Note for other Hive committers:  this JIRA originated with some back-channel 
discussions between me and Pradeep about how to make it possible for Howl to 
reuse Hive CLI functionality.)


> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Ashutosh Chauhan
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1546) Ability to plug custom Semantic Analyzers for Hive Grammar

2010-08-16 Thread Ashutosh Chauhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899204#action_12899204
 ] 

Ashutosh Chauhan commented on HIVE-1546:


This will involve a bit of refactoring and adding interface points which other 
systems can latch on.

> Ability to plug custom Semantic Analyzers for Hive Grammar
> --
>
> Key: HIVE-1546
> URL: https://issues.apache.org/jira/browse/HIVE-1546
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Ashutosh Chauhan
>
> It will be useful if Semantic Analysis phase is made pluggable such that 
> other projects can do custom analysis of hive queries before doing metastore 
> operations on them. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.