Re: Dynamic Parameters Handling in RelNode
Hi Walaa, Adding onto the last mail: 1) I already have the value of *table_name* parameter with schema and table object to help calcite resolving it while creating RelNode. Is there any way to pass this? But In the translated query the value should be replaced with the key itself. Or Do we always need to pass this in the sql query as in the static query itself? On Fri, May 3, 2024 at 11:04 AM Walaa Eldin Moustafa wrote: > Hi Ravi, > > RelNodes depend on specific table objects with known schemas. Unless we > know that table's schema there is no way to generate the RelNode in the > first place. Maybe if you know the schema (data types of the table > columns), it is possible to get around the lack of a name, but I do not see > how this works if there is no table object or schema in the first place. > > Thanks, > Walaa. > > > On Thu, May 2, 2024 at 10:23 PM Ravi Kapoor > wrote: > > > Hello Team, > > Is there any way in the calcite framework to use runtime variables in Rel > > Node in order to get the same variable back in the translated query? > > Basically I want the same variable to be used back as the runtime > parameter > > after query translation. > > > > Here I am talking specifically about using dynamic parameters in > > table/column identifiers and not dynamic rex nodes. > > > > Let's say in Teradata I can write below query > > *TD*: select employee_id, TRIM(employee_name) from :*table_name* > > > > The translated sql should give back to me the same parameter in the > query. > > *say Trim is mapped to LTRIM* > > *BigQuery*: select employee_id, LTRIM(employee_name) from :*table_name* > > > > I believe we can only have static values in schema plus to > > create tables/columns. But what about such cases? > > Need to handle this gracefully with no string manipulation. > > > > -- > > Thanks, > > Ravi > > > -- Thanks, Ravi Kapoor +91-9818764564 kapoorrav...@gmail.com
Dynamic Parameters Handling in RelNode
Hello Team, Is there any way in the calcite framework to use runtime variables in Rel Node in order to get the same variable back in the translated query? Basically I want the same variable to be used back as the runtime parameter after query translation. Here I am talking specifically about using dynamic parameters in table/column identifiers and not dynamic rex nodes. Let's say in Teradata I can write below query *TD*: select employee_id, TRIM(employee_name) from :*table_name* The translated sql should give back to me the same parameter in the query. *say Trim is mapped to LTRIM* *BigQuery*: select employee_id, LTRIM(employee_name) from :*table_name* I believe we can only have static values in schema plus to create tables/columns. But what about such cases? Need to handle this gracefully with no string manipulation. -- Thanks, Ravi
CALCITE-3147 | Not Supported
Hi Team,
As per https://issues.apache.org/jira/browse/CALCITE-3147
the triple quoted string for BigQuerySqlDialect should also be
supported in case of query falls in multiple lines.
However, I cannot see it working. Neither do we have a test case
supporting the use case.
Can you please provide some insights on that?
I have a use case where RexLiteral value falls under multiple new
lines in BQ dialect and the
unparsed string generated as shown:
'SELECT wuaht_.report_access_brch_cd as branch_cd,
wuaht_.report_access_brch_nm AS branch_name
FROM BRCH_INCTV_PROD_DATA.wire_user_access_hier_1 as wuaht_
WHERE wuaht_.system_access_swb_prsn_id = ?
GROUP BY 1,2;'; -> RexLiteral
which fails in validation.
I can set it as .withLiteralQuoteString(" ''' ") to the context
But that would make all literals wrapped in triple quotes.
Can we wrap it in triple quotes only when drills down to multiple lines?
Thanks,
Ravi
Link4j AST Node | ExpressionWriter Runtime Exception
Hi Team, The Java AST Node interface in the calcite - linq4j project has below API for converting expressions to java code *void accept(ExpressionWriter expressionWriter)* However, ExpressionWriter class is declared as package protected and not part of exported API. Could not understand the rationale behind this as the tree itself is public I need to create custom Node by extending a Statement or Expression from Linq4j and override the accept of writer in order to covert my own Node expression. In this case it goes to the parent accept() method which is of AbstractNode and throws RuntimeException - unparse not supported. Can you please check on this? -- Thanks, Ravi Kapoor +91-9818764564 kapoorrav...@gmail.com
Re: Stored Proc to Relational Expression
Yeah Stamatis, I know that as I can see the github page for linq4j that lots of task is in Backlog and Link-To-SQL Provider is one of them But the intention of the project is to make java objects as Queryable talk to DB. Expressions and various Statement used in linq4j is to generate the different constructs like where filter, join to talk the back-end systems as Sql Queries. But what about the normal While loop and If then else statements which is not used in the Query-able objects . Example could be below: LOOP SET heads = RAND() < 0.5; IF heads THEN SELECT 'Heads!'; SET heads_count = heads_count + 1; ELSE SELECT 'Tails!'; BREAK; END IF; END LOOP; SELECT CONCAT(CAST(heads_count AS STRING), ' heads in a row'); How linq4j going to convert normal while and If then else java expressions to sql query if they are not involved in computation of sql constructs like in lambda expression of filter. In one of the task in backlog I could see: *In the prototype LINQ-to-SQL provider, write a simple rule to recognize a select list and where clause and push them down to SQL.* I believe it was never intended for procedural code right? Thanks, Ravi On Wed, Apr 22, 2020 at 2:43 AM Stamatis Zampetakis wrote: > Hi Ravi, > > As Julian already mentioned, we are quite far from what you would like to > achieve. > > Nevertheless, I would like to mention that you can model certain kind of > loops and recursion using the RepeatUnion [1] and Spool [2] relational > expressions. Note that these APIs are experimental. > > Moreover, if your end goal is to push the computation in a DBMS, I have to > warn you that the code to translate these expressions back to SQL is not > there yet. > > Best, > Stamatis > > [1] > > https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/core/RepeatUnion.java > [2] > > https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/core/Spool.java > > On Mon, Apr 20, 2020 at 8:08 PM Julian Hyde wrote: > > > Calcite relational expressions can represent SELECT, INSERT etc. but not > > procedural code. It’s a direction we could consider going. > > > > RexProgram is the closest thing we currently have to procedural code in > > the algebra - single assignment of variables, use of variables in > > expressions assigning to other variables - but it is a long way short > > because there are no loops. > > > > > On Apr 19, 2020, at 12:47 PM, Ravi Kapoor > > wrote: > > > > > > Hi Team, > > > > > > I have my use where I need to convert my dialect specific stored > > procedure > > > constructs like while loop, If then else to Rel expression > > > > > > Basically this can contain control flow statements like below > > > > > > DECLARE heads BOOL; > > > DECLARE heads_count INT64 DEFAULT 0; > > > LOOP > > > SET heads = RAND() < 0.5; > > > IF heads THEN > > >SELECT 'Heads!'; > > >SET heads_count = heads_count + 1; > > > ELSE > > >SELECT 'Tails!'; > > >BREAK; > > > END IF; > > > END LOOP; > > > SELECT CONCAT(CAST(heads_count AS STRING), ' heads in a row'); > > > > > > > > > I can create a Java AST model from the linq4j provided by calcite > however > > > this is only going to generate Java Result and I believe its only used > by > > > the calcite for relational expressions of enumerable calling convention > > > which is used by adapters which does not support core relational > > operations > > > right? > > > > > > Is there a way I can convert the stored proc constructs into some > > canonical > > > form like Rel Tree and back to Stored proc of target dialect. > > > -- > > > > > > Thanks, > > > Ravi > > > > > -- Thanks, Ravi Kapoor +91-9818764564 kapoorrav...@gmail.com
Stored Proc to Relational Expression
Hi Team, I have my use where I need to convert my dialect specific stored procedure constructs like while loop, If then else to Rel expression Basically this can contain control flow statements like below DECLARE heads BOOL; DECLARE heads_count INT64 DEFAULT 0; LOOP SET heads = RAND() < 0.5; IF heads THEN SELECT 'Heads!'; SET heads_count = heads_count + 1; ELSE SELECT 'Tails!'; BREAK; END IF; END LOOP; SELECT CONCAT(CAST(heads_count AS STRING), ' heads in a row'); I can create a Java AST model from the linq4j provided by calcite however this is only going to generate Java Result and I believe its only used by the calcite for relational expressions of enumerable calling convention which is used by adapters which does not support core relational operations right? Is there a way I can convert the stored proc constructs into some canonical form like Rel Tree and back to Stored proc of target dialect. -- Thanks, Ravi
[jira] [Created] (CALCITE-3784) RexCall toString digest gives OOM while huge expression is evaluated
Ravi Kapoor created CALCITE-3784:
Summary: RexCall toString digest gives OOM while huge expression
is evaluated
Key: CALCITE-3784
URL: https://issues.apache.org/jira/browse/CALCITE-3784
Project: Calcite
Issue Type: Bug
Components: core
Reporter: Ravi Kapoor
I have some complex query which has tens of thousands of rex expressions and
this expression is used in filter expression in the query.
On creating a filter below code gets called:
public RelBuilder filter(Iterable variablesSet,
Iterable predicates) {
final RexNode simplifiedPredicates =
simplifier.simplifyFilterPredicates(predicates);
if (simplifiedPredicates == null) {
return empty();
}
RexSimplify further adds the rexnode in the Set calling hashcode()
internally which calls toString()
Is there any way to avoid this computeDigest Call which creates complex string
object and blows up the RAM about 14GB?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
