[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383431#comment-15383431 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-trafodion/pull/577 > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15382720#comment-15382720 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r71197478 --- Diff: core/sql/parser/sqlparser.y --- @@ -6953,13 +6968,22 @@ del_stmt_w_acc_type_rtn_list_and_as_clause_col_list : '(' delete_statement acce table_name_as_clause_and_hint : table_name as_clause optimizer_hint hbase_access_options { - $1->setCorrName(*$2); - $$ = new (PARSERHEAP()) Scan(*$1); - if ($3) -$$->setHint($3); - if ($4) -((Scan*)$$)->setHbaseAccessOptions($4); - + NAString tmp = ((*$1).getQualifiedNameAsString()); + if(SqlParser_CurrentParser->hasWithDefinition(&tmp) ) + { + RelExpr *re = SqlParser_CurrentParser->getWithDefinition(&tmp); + RenameTable *rt = new (PARSERHEAP()) RenameTable(re, *$2); + $$=rt->copyTree(PARSERHEAP()); --- End diff -- Same as above, could give an error if we see a hint or an HBase access option. > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15382718#comment-15382718 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r71197264 --- Diff: core/sql/parser/sqlparser.y --- @@ -6647,15 +6651,26 @@ table_as_tmudf_function : TOK_UDF '(' table_mapping_function_invocation ')' table_name_and_hint : table_name optimizer_hint hbase_access_options { -$$ = new (PARSERHEAP()) Scan(*$1); -if ($2) - $$->setHint($2); +NAString tmp = ((*$1).getQualifiedNameAsString()); + if(SqlParser_CurrentParser->hasWithDefinition(&tmp) ) +{ + RelExpr *re = SqlParser_CurrentParser->getWithDefinition(&tmp); + $$=re->copyTree(PARSERHEAP()); + delete $1; --- End diff -- One thing we could do here is to raise an error if we have an optimizer hint or HBase access options at this point. Those are allowed in the grammar, but we wouldn't be able to apply them to a more complicated WITH clause. > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15381367#comment-15381367 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user traflm commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r71083286 --- Diff: core/sql/parser/sqlparser.y --- @@ -14684,6 +14759,11 @@ optional_limit_spec : TOK_LIMIT NUMERIC_LITERAL_EXACT_NO_SCALE dml_statement : dml_query { $$ = $1; } + | with_clause_list dml_query --- End diff -- sorry, I made mistake, by running TESTOK2, it shows the change still introduce one more reduce/reduce conflict. Still looking at it. > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380400#comment-15380400 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user traflm commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r71058116 --- Diff: core/sql/parser/sqlparser.y --- @@ -7000,6 +7024,46 @@ rel_subquery_and_as_clause : rel_subquery as_clause CheckModeSpecial1(); $$ = $2; } +with_clause : + TOK_WITH with_clause_elements + { + $$ = NULL; + } + | TOK_WITH TOK_RECURSIVE with_clause_elements + { +*SqlParser_Diags << DgSqlCode(-3022) << DgString0("WITH RECURSIVE"); +YYERROR; + } + +with_clause_elements : with_clause_element + { + $$ = NULL; + } + | with_clause_elements ',' with_clause_element + { + $$ = NULL; + } + +with_clause_element : correlation_name TOK_AS '(' query_expression ')' + { + RelRoot *root = new (PARSERHEAP()) +RelRoot($4, REL_ROOT); + $$= new (PARSERHEAP()) RenameTable(root, *$1); + + //Duplicated definition of WITH + if(SqlParser_CurrentParser->hasWithDefinition($1) ) + { +*SqlParser_Diags << DgSqlCode(-3288) + << DgString0((*$1).toCharStar()); + delete $1; + delete $4; + YYERROR; + } + + SqlParser_CurrentParser->insertWithDefinition($1 , $$); + delete $1; + delete $4; --- End diff -- I agree, will change it. > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380398#comment-15380398 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user traflm commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r71058095 --- Diff: core/sql/parser/sqlparser.y --- @@ -1,14 +13396,41 @@ query_specification : select_token set_quantifier query_spec_body ColReference(new (PARSERHEAP()) ColRefName(TRUE, PARSERHEAP())) ) ); - assert($3->getOperatorType() == REL_ROOT); + assert($3->getOperatorType() == REL_ROOT); + RelRoot *root1 = (RelRoot *) $$; + RelRoot *root2 = (RelRoot *) $3; + root1->assignmentStTree() = root2->assignmentStTree(); + root2->assignmentStTree() = NULL; + } + else + $$ = $3; + + if (CmpCommon::getDefault(MVQR_LOG_QUERY_DESCRIPTORS) == DF_DUMP || + CmpCommon::getDefault(MVQR_LOG_QUERY_DESCRIPTORS) == DF_DUMP_MV) + ((RelRoot*)$$)->setAnalyzeOnly(); + +} + +/* type relx */ +query_specification :with_clause select_token set_quantifier query_spec_body + { + if ($3) { + $$ = new (PARSERHEAP()) + RelRoot(new (PARSERHEAP()) + GroupByAgg($4 --- End diff -- if ($3) means there is a set_quantifier token, which is DISTINCT for example, so need a GroupByAgg node here. > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380394#comment-15380394 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user traflm commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r71057841 --- Diff: core/sql/parser/sqlparser.y --- @@ -13418,6 +13507,55 @@ query_specification : select_token '[' firstn_sorted NUMERIC_LITERAL_EXACT_NO_SC if (CmpCommon::getDefault(MVQR_LOG_QUERY_DESCRIPTORS) == DF_DUMP || CmpCommon::getDefault(MVQR_LOG_QUERY_DESCRIPTORS) == DF_DUMP_MV) ((RelRoot*)$$)->setAnalyzeOnly(); + } +/* type relx */ +query_specification : with_clause select_token '[' firstn_sorted NUMERIC_LITERAL_EXACT_NO_SCALE ']' set_quantifier query_spec_body + { + if ($7) { + $$ = new (PARSERHEAP()) + RelRoot(new (PARSERHEAP()) + GroupByAgg($8 --- End diff -- if($7) means there is a set_quantifier like (DISTINCT), so it will add a GroupByAgg node here. > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15379745#comment-15379745 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user DaveBirdsall commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r71010941 --- Diff: core/sql/parser/sqlparser.y --- @@ -7000,6 +7024,46 @@ rel_subquery_and_as_clause : rel_subquery as_clause CheckModeSpecial1(); $$ = $2; } +with_clause : + TOK_WITH with_clause_elements + { + $$ = NULL; + } + | TOK_WITH TOK_RECURSIVE with_clause_elements + { +*SqlParser_Diags << DgSqlCode(-3022) << DgString0("WITH RECURSIVE"); +YYERROR; + } + +with_clause_elements : with_clause_element + { + $$ = NULL; + } + | with_clause_elements ',' with_clause_element + { + $$ = NULL; + } + +with_clause_element : correlation_name TOK_AS '(' query_expression ')' + { + RelRoot *root = new (PARSERHEAP()) +RelRoot($4, REL_ROOT); + $$= new (PARSERHEAP()) RenameTable(root, *$1); + + //Duplicated definition of WITH + if(SqlParser_CurrentParser->hasWithDefinition($1) ) + { +*SqlParser_Diags << DgSqlCode(-3288) + << DgString0((*$1).toCharStar()); + delete $1; + delete $4; + YYERROR; + } + + SqlParser_CurrentParser->insertWithDefinition($1 , $$); + delete $1; + delete $4; --- End diff -- I don't think we want to delete $1 and $4 here, since the RelRoot node points to $4 and the RenameTable node points to $1. > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15379735#comment-15379735 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user DaveBirdsall commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r71010585 --- Diff: core/sql/parser/sqlparser.y --- @@ -13418,6 +13507,55 @@ query_specification : select_token '[' firstn_sorted NUMERIC_LITERAL_EXACT_NO_SC if (CmpCommon::getDefault(MVQR_LOG_QUERY_DESCRIPTORS) == DF_DUMP || CmpCommon::getDefault(MVQR_LOG_QUERY_DESCRIPTORS) == DF_DUMP_MV) ((RelRoot*)$$)->setAnalyzeOnly(); + } +/* type relx */ +query_specification : with_clause select_token '[' firstn_sorted NUMERIC_LITERAL_EXACT_NO_SCALE ']' set_quantifier query_spec_body + { + if ($7) { + $$ = new (PARSERHEAP()) + RelRoot(new (PARSERHEAP()) + GroupByAgg($8 --- End diff -- I don't understand why we add a GroupByAgg node here. > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15379733#comment-15379733 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user DaveBirdsall commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r71010430 --- Diff: core/sql/parser/sqlparser.y --- @@ -1,14 +13396,41 @@ query_specification : select_token set_quantifier query_spec_body ColReference(new (PARSERHEAP()) ColRefName(TRUE, PARSERHEAP())) ) ); - assert($3->getOperatorType() == REL_ROOT); + assert($3->getOperatorType() == REL_ROOT); + RelRoot *root1 = (RelRoot *) $$; + RelRoot *root2 = (RelRoot *) $3; + root1->assignmentStTree() = root2->assignmentStTree(); + root2->assignmentStTree() = NULL; + } + else + $$ = $3; + + if (CmpCommon::getDefault(MVQR_LOG_QUERY_DESCRIPTORS) == DF_DUMP || + CmpCommon::getDefault(MVQR_LOG_QUERY_DESCRIPTORS) == DF_DUMP_MV) + ((RelRoot*)$$)->setAnalyzeOnly(); + +} + +/* type relx */ +query_specification :with_clause select_token set_quantifier query_spec_body + { + if ($3) { + $$ = new (PARSERHEAP()) + RelRoot(new (PARSERHEAP()) + GroupByAgg($4 --- End diff -- I don't understand why we stick a GroupByAgg node here. > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378533#comment-15378533 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r70897000 --- Diff: core/sql/parser/sqlparser.y --- @@ -14684,6 +14759,11 @@ optional_limit_spec : TOK_LIMIT NUMERIC_LITERAL_EXACT_NO_SCALE dml_statement : dml_query { $$ = $1; } + | with_clause_list dml_query --- End diff -- Thanks, that's excellent news! > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376435#comment-15376435 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user traflm commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r70756198 --- Diff: core/sql/parser/sqlparser.y --- @@ -6647,6 +6652,26 @@ table_as_tmudf_function : TOK_UDF '(' table_mapping_function_invocation ')' table_name_and_hint : table_name optimizer_hint hbase_access_options { +NAString tmp = ((*$1).getQualifiedNameAsString()); + if(SqlParser_CurrentParser->with_clauses_->contains(&tmp) ) +{ + RelExpr *re = SqlParser_CurrentParser->with_clauses_->getFirstValue(&tmp); + $$=re->copyTree(PARSERHEAP()); --- End diff -- yes, I am not quite sure how memory was managed in the parser, I changed to delete $1 > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376429#comment-15376429 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user traflm commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r70755998 --- Diff: core/sql/sqlcomp/parser.h --- @@ -261,6 +261,8 @@ ItemExpr *get_w_ItemExprTree(const NAWchar * str, NABoolean isHQCCacheable() { return HQCKey_?HQCKey_->isCacheable():FALSE; } + NAHashDictionary *with_clauses_; --- End diff -- yes, changed > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376434#comment-15376434 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user traflm commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r70756042 --- Diff: core/sql/bin/SqlciErrors.txt --- @@ -410,6 +410,7 @@ 1430 3F000 9 BEGINNER MAJOR DBADMIN A schema name that starts and ends with an "_"(underscore) is reserved for internal usage. It cannot be used to create a user schema. 1431 Z 9 BEGINNER MINOR DBADMIN Object $0~String0 exists in HBase. This could be due to a concurrent transactional ddl operation in progress on this table. 1432 Z 9 BEGINNER MINOR DBADMIN Input LOB type $0~Int0 does not match column's storage type: $1~Int1 Column name: $0~String0 . +1433 Z 9 BEGINNER MINOR DBADMIN WITH clause redefined. WITH name $0~String0 . --- End diff -- yes, changed to 3288 > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376157#comment-15376157 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user traflm commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r70736959 --- Diff: core/sql/parser/sqlparser.y --- @@ -14684,6 +14759,11 @@ optional_limit_spec : TOK_LIMIT NUMERIC_LITERAL_EXACT_NO_SCALE dml_statement : dml_query { $$ = $1; } + | with_clause_list dml_query --- End diff -- I found a way to avoid the reduce/shift and reduce/reduce error by add with_clause down to the underlying rules of non_join_query_expression, instead of add it at same level of non_join_query_expression. non_join_query_expression can represent many different styles of query expression, including utilities, which we don't want a with_clause as prefix, so in the underlying rules, only add with_clause before those SELECT syntax, and it solve the problem. And it seems if use redundant rule explicitly (looks annoying) , we can avoid the error. For example: A : b | c is better to change to A : b A : c I am still trying to understand this. I checked with PostgreSQL parser, they are doing similar things, comments from postgresql parser: * This rule parses the equivalent of the standard's . * The duplicative productions are annoying, but hard to get rid of without * creating shift/reduce conflicts. > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375270#comment-15375270 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r70656018 --- Diff: core/sql/parser/sqlparser.y --- @@ -14684,6 +14759,11 @@ optional_limit_spec : TOK_LIMIT NUMERIC_LITERAL_EXACT_NO_SCALE dml_statement : dml_query { $$ = $1; } + | with_clause_list dml_query --- End diff -- Yes, I got a similar result. I tried this: query_expression : with_clause non_join_query_expression | non_join_query_expression That gives me three additional reduce/reduce conflicts which I don't understand. Here are two of them: State 1941 1870 query_expression: with_clause non_join_query_expression . 1871 | non_join_query_expression . TOK_EXCEPT reduce using rule 1870 (query_expression) TOK_EXCEPT [reduce using rule 1871 (query_expression)] TOK_UNION reduce using rule 1870 (query_expression) TOK_UNION [reduce using rule 1871 (query_expression)] $defaultreduce using rule 1870 (query_expression) I also tried this: query_expression : optional_with_clause non_join_query_expression That doesn't cause additional reduce/reduce conflicts, but we get a very large number of additional shift/reduce conflicts, something in the neighborhood of 50-100. If we can't get the "correct" parser rules to work, then maybe we can stick with the approach you chose (assuming it accepts all the valid WITH clauses) and handle any illegal syntax in the semantic actions. > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374956#comment-15374956 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user traflm commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r70620967 --- Diff: core/sql/parser/sqlparser.y --- @@ -14684,6 +14759,11 @@ optional_limit_spec : TOK_LIMIT NUMERIC_LITERAL_EXACT_NO_SCALE dml_statement : dml_query { $$ = $1; } + | with_clause_list dml_query --- End diff -- hi, Hans, Dave, the current change doesn't get higher number of conflicts, it is still 73/12, but if I tried to move the rule into query_expression, it will increase the shift/reduce conflicts, I still cannot find a good way to write a correct syntax. It is rather difficult to find the root cause of shift/reduce conflict > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15373646#comment-15373646 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user DaveBirdsall commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r70517961 --- Diff: core/sql/parser/sqlparser.y --- @@ -6579,6 +6581,9 @@ table_reference : table_name_and_hint | rel_subquery_and_as_clause { $$ = $1; } + | with_clause_list --- End diff -- I agree with Hans; it would be better to move the wish_clause non-terminal to the query_expression production. Right now, that production reads: query_expression : non_join_query_expression You could change it to: query_expression: with_clause non_join_query_expression | non_join_query_expression, or alternatively, query_expression : with_clause non_join_query_expression And then code with_clause to allow an empty production, for example: with_clause : { $$ = NULL; /* empty with_clause case */ } | TOK_WITH with_clause_elements > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15373629#comment-15373629 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user DaveBirdsall commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r70514345 --- Diff: core/sql/parser/sqlparser.y --- @@ -6579,6 +6581,9 @@ table_reference : table_name_and_hint | rel_subquery_and_as_clause { $$ = $1; } + | with_clause_list --- End diff -- I think this is incorrect. Making "with_clause_list" an alternative for table_reference would allow things like: "select * from with c1 as (select * from t)". > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15373528#comment-15373528 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user DaveBirdsall commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r70505167 --- Diff: core/sql/parser/sqlparser.y --- @@ -7001,6 +7036,46 @@ rel_subquery_and_as_clause : rel_subquery as_clause $$ = $2; } +with_clause_list : with_clause + { $$ = $1 ; } + + | with_clause_list ',' correlation_name TOK_AS rel_subquery --- End diff -- One way would be: with_clause : WITH with_clause_elements with_clause_elements : with_clause element | with_clause_elements with_clause_element with_clause_element : correlation_name TOK_AS rel_subquery By the way, I notice the way this is coded, the "with_clause" non-terminal is a single correlation_name TOK_AS rel_subquery element. It seems more natural (to me anyway) to think of the with_clause as being everything from the WITH keyword through the entire list of subqueries. > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15373507#comment-15373507 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user DaveBirdsall commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r70504097 --- Diff: core/sql/parser/sqlparser.y --- @@ -6647,6 +6652,26 @@ table_as_tmudf_function : TOK_UDF '(' table_mapping_function_invocation ')' table_name_and_hint : table_name optimizer_hint hbase_access_options { +NAString tmp = ((*$1).getQualifiedNameAsString()); + if(SqlParser_CurrentParser->with_clauses_->contains(&tmp) ) +{ + RelExpr *re = SqlParser_CurrentParser->with_clauses_->getFirstValue(&tmp); + $$=re->copyTree(PARSERHEAP()); --- End diff -- Should we delete $1 in this code path as well? > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15372910#comment-15372910 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user traflm commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r70441662 --- Diff: core/sql/parser/sqlparser.y --- @@ -7001,6 +7036,46 @@ rel_subquery_and_as_clause : rel_subquery as_clause $$ = $2; } +with_clause_list : with_clause + { $$ = $1 ; } + + | with_clause_list ',' correlation_name TOK_AS rel_subquery --- End diff -- @zellerh , I cannot figure out how to do this. Could you please give me an example? I am very new to the parser. > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366302#comment-15366302 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r69932659 --- Diff: core/sql/parser/sqlparser.y --- @@ -14684,6 +14759,11 @@ optional_limit_spec : TOK_LIMIT NUMERIC_LITERAL_EXACT_NO_SCALE dml_statement : dml_query { $$ = $1; } + | with_clause_list dml_query --- End diff -- That is the number of conflicts encountered by the bison parser when it processes file core/sql/parser/sqlparser.y. Here is how I get this information: ``` touch core/sql/parser/sqlparser.y cd core/sql/nskgmake gmake -j 4 -ks linuxdebug ``` Here is the current output: ``` ../parser/sqlparser.y: warning: 73 shift/reduce conflicts [-Wconflicts-sr] ../parser/sqlparser.y: warning: 12 reduce/reduce conflicts [-Wconflicts-rr] ``` If you get a different (higher) number, then we would have to investigate whether there are any potential problems with the new syntax. > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365622#comment-15365622 ] Anoop Sharma commented on TRAFODION-1673: - when a make is done, the parser component shows shift/reduce and reduce/reduce conflicts. Something like this will be part of make output: ../parser/sqlparser.y: warning: 73 shift/reduce conflicts [-Wconflicts-sr] ../parser/sqlparser.y: warning: 12 reduce/reduce conflicts [-Wconflicts-rr] It is currently at 73/12. This will also be caught by regress/compGeneral/TESTTOK2. If that test passes, then conflicts have not changed. Check for that. > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365573#comment-15365573 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user traflm commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r69848822 --- Diff: core/sql/parser/sqlparser.y --- @@ -14684,6 +14759,11 @@ optional_limit_spec : TOK_LIMIT NUMERIC_LITERAL_EXACT_NO_SCALE dml_statement : dml_query { $$ = $1; } + | with_clause_list dml_query --- End diff -- @zellerh , I don't understand what is "number of shift/reduce and reduce/reduce conflicts". How to check this number? What is this? > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365454#comment-15365454 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r69839237 --- Diff: core/sql/bin/SqlciErrors.txt --- @@ -410,6 +410,7 @@ 1430 3F000 9 BEGINNER MAJOR DBADMIN A schema name that starts and ends with an "_"(underscore) is reserved for internal usage. It cannot be used to create a user schema. 1431 Z 9 BEGINNER MINOR DBADMIN Object $0~String0 exists in HBase. This could be due to a concurrent transactional ddl operation in progress on this table. 1432 Z 9 BEGINNER MINOR DBADMIN Input LOB type $0~Int0 does not match column's storage type: $1~Int1 Column name: $0~String0 . +1433 Z 9 BEGINNER MINOR DBADMIN WITH clause redefined. WITH name $0~String0 . --- End diff -- Parser errors should be in the range 3000-3999. Also, since this is a syntax error, use 42000 (the SQLSTATE value for syntax errors) instead of "Z". > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365423#comment-15365423 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r69837244 --- Diff: core/sql/sqlcomp/parser.h --- @@ -261,6 +261,8 @@ ItemExpr *get_w_ItemExprTree(const NAWchar * str, NABoolean isHQCCacheable() { return HQCKey_?HQCKey_->isCacheable():FALSE; } + NAHashDictionary *with_clauses_; --- End diff -- Can you make this a private data member? > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365421#comment-15365421 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r69837060 --- Diff: core/sql/parser/sqlparser.y --- @@ -14684,6 +14759,11 @@ optional_limit_spec : TOK_LIMIT NUMERIC_LITERAL_EXACT_NO_SCALE dml_statement : dml_query { $$ = $1; } + | with_clause_list dml_query --- End diff -- Related question: When you made this change, did the number of shift/reduce and reduce/reduce conflicts change? > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365413#comment-15365413 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r69836802 --- Diff: core/sql/parser/sqlparser.y --- @@ -14684,6 +14759,11 @@ optional_limit_spec : TOK_LIMIT NUMERIC_LITERAL_EXACT_NO_SCALE dml_statement : dml_query { $$ = $1; } + | with_clause_list dml_query --- End diff -- Same comment as a above about where the parser rule for the WITH clause should be. > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365409#comment-15365409 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r69836669 --- Diff: core/sql/parser/sqlparser.y --- @@ -7001,6 +7036,46 @@ rel_subquery_and_as_clause : rel_subquery as_clause $$ = $2; } +with_clause_list : with_clause + { $$ = $1 ; } + + | with_clause_list ',' correlation_name TOK_AS rel_subquery --- End diff -- Can we do this in a way that doesn't have two different rules for the first and the subsequent parts of the WITH clause? See comment above for how the ANSI standard handles the WITH clause. > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365402#comment-15365402 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r69836226 --- Diff: core/sql/parser/sqlparser.y --- @@ -6579,6 +6581,9 @@ table_reference : table_name_and_hint | rel_subquery_and_as_clause { $$ = $1; } + | with_clause_list --- End diff -- Here is the syntax rule from the ANSI standard: ::= [ ] > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365383#comment-15365383 ] ASF GitHub Bot commented on TRAFODION-1673: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/577#discussion_r69835003 --- Diff: core/sql/parser/sqlparser.y --- @@ -6579,6 +6581,9 @@ table_reference : table_name_and_hint | rel_subquery_and_as_clause { $$ = $1; } + | with_clause_list --- End diff -- The ANSI SQL standard defines the "with" clause as part of a query_expression, so I would recommend doing the same here (non_join_query_expression in our case). Also, it would be good to accept the WITH RECURSIVE syntax and to give an error that this is not yet supported. > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15364404#comment-15364404 ] liu ming commented on TRAFODION-1673: - To fully support WITH, I agree with both Hans and QiFan. Especially the idea of evaluating the CTE only once. However, we can do this step by step. The very first step is as Hans' suggest, Trafodion support WITH function in the parser. The 'Temp View' can be simply be a 'subquery', which is a RenameTable in the parse tree. As initial testing, it can support recursive reference as well, but with limitation that the WITH definition must occur before it is referred , for example: with w1 as (select * from witht1), w2 as (select * from w1) select * from w2; is good, since w1 is defined before it is used in w2. But with w1 as (select * from w2), w2 as (select * from t1) select * from w1; will not work, however, I don't see the second usage in all TPCDS queries. So I feel it should be good enough to support the first syntax. This first iteration can implement the functionality, and parser change can be reused later of a 'common CTE', and test case can be used also. Later, we can implement a new CTE type of node, which can feed multiple data consumers, but I feel that is not easy in current architecture. > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359961#comment-15359961 ] liu ming commented on TRAFODION-1673: - It can be first implemented using sub-query, that is, to replace each WITH clause by a sub query. (RenamedTable) CTE mentioned by QiFan is a great idea, and we can set it as a final goal. To meet the functional requirement, we can simply use subquery in the first iteration. > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller >Assignee: liu ming > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1673) Implement the WITH clause in Trafodion SQL for simple use cases
[ https://issues.apache.org/jira/browse/TRAFODION-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15044992#comment-15044992 ] Qifan Chen commented on TRAFODION-1673: --- On the scope of WITH support. In the following CTE = common table expression defined by in a WITH clause. By looking TPCDS, I can see the following. 1. Multiple CTEs can be defined within one query 2. One CTE can be defined based on another CTE Q2: wswscs is dependent on wscs Q47: v2 is based on v1 Q54: my_revenue is dependent on my_customers, segments is dependent on my_revenue Q57: v2 is based on v1 Q65: cross_sales is based on cs_ui 3. Repeated references of the same CTE in a single query Q2: wswscs is used in two separate joins Q39: inv is used in join with itself as inv1 and inv2 Q4: year_total is self joined 6 times Q59: wss is used in separate sub-queries Q65: cross_sales is self joined Q74: year_total is self-joined 3 times Q95: ws_wh is used in two sub-queries 4. CTEs can be used just like a table expression joined with another table joined with another CTE Union-ed with another CTE 5. CTEs can require extensive computations So it looks like we need to deal with all above aspects of WITH. In addition, to efficiently handle repeated references of the same CET and CTEs require extensive computations to compute, I feel we should consider computing CTE only once per query. Here is one idea to achieve it. For example, to self join a CBE C, we could have the following query tree. S is the single computation of C. R is a reference to C. S "smartly" broadcasts data to two Rs. join / \ RR | / S > Implement the WITH clause in Trafodion SQL for simple use cases > --- > > Key: TRAFODION-1673 > URL: https://issues.apache.org/jira/browse/TRAFODION-1673 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-cmp >Reporter: Hans Zeller > > We keep running into queries that use a WITH clause to define a temporary > view that can be used once or multiple times in a FROM clause in the query. > For non-recursive queries, the WITH clause could probably be handled very > similar to a view. When it is defined, we create an in-memory view > descriptor, containing the name and the definition. When it is used in a FROM > clause, we could go through a code path similar to that of a view - bind the > (temporary) view text and substitute it in the query. The fix could probably > be handled entirely in the binder. > This JIRA is *not* about recursive queries, those would require a lot more > effort, involving many components in addition to the binder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)