Sorry to bug with another question. I was saving a data set with below schema
(this dataset comes from sql query). Saw the SqlCharType issue. Did anyone see
this issue before?
[main] INFO com.zillow.pipeler.core.transform.DatasetFlattenerCore - Fields:
Field{name=id, description=, type=LOGICAL_TYPE NOT NULL, options={{}}}
Field{name=user_tmp, description=, type=LOGICAL_TYPE NOT NULL, options={{}}}
Field{name=market_name, description=, type=LOGICAL_TYPE NOT NULL, options={{}}}
Field{name=market_transactionManagement_transactionManagers_email,
description=, type=LOGICAL_TYPE NOT NULL, options={{}}}
Field{name=market_transactionManagement_transactionManagers_name, description=,
type=LOGICAL_TYPE NOT NULL, options={{}}}
Field{name=market_transactionManagement_transactionProfileId, description=,
type=LOGICAL_TYPE NOT NULL, options={{}}}
Options:{{}}
Exception in thread "main" java.lang.RuntimeException: Unhandled logical type
SqlCharType
at
org.apache.beam.sdk.schemas.utils.AvroUtils.getFieldSchema(AvroUtils.java:911)
at
org.apache.beam.sdk.schemas.utils.AvroUtils.toAvroField(AvroUtils.java:306)
at
org.apache.beam.sdk.schemas.utils.AvroUtils.toAvroSchema(AvroUtils.java:341)
at
org.apache.beam.sdk.schemas.utils.AvroUtils.toAvroSchema(AvroUtils.java:348)
From: Tao Li <[email protected]>
Reply-To: "[email protected]" <[email protected]>
Date: Monday, May 10, 2021 at 7:19 PM
To: "[email protected]" <[email protected]>
Cc: Yuan Feng <[email protected]>
Subject: Re: A problem with calcite sql
Never mind. Looks like “user” is a reserved name.
From: Tao Li <[email protected]>
Reply-To: "[email protected]" <[email protected]>
Date: Monday, May 10, 2021 at 7:10 PM
To: "[email protected]" <[email protected]>
Cc: Yuan Feng <[email protected]>
Subject: A problem with calcite sql
Hi Beam community,
I am seeing a weird issue by using calcite sql. I don’t understand why it’s
complaining my query is not valid. Once I removed “user AS user”, it worked
fine. Please advise. Thanks.
Exception in thread "main"
org.apache.beam.sdk.extensions.sql.impl.ParseException: Unable to parse query
SELECT id AS id, user AS user, market_name AS market_name,
market_transactionManagement_transactionManagers.email AS
market_transactionManagement_transactionManagers_email,
market_transactionManagement_transactionManagers.name AS
market_transactionManagement_transactionManagers_name,
market_transactionManagement_transactionProfileId AS
market_transactionManagement_transactionProfileId FROM PCOLLECTION
at
org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:214)
at
org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:111)
at
org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:171)
at
org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:109)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:547)
at
org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:498)
at
org.apache.beam.sdk.values.PCollection.apply(PCollection.java:370)
at
com.zillow.pipeler.core.transform.DatasetFlattenerCore.updateSchemaBasedOnAvroSchema(DatasetFlattenerCore.java:85)
at
com.zillow.pipeler.core.transform.DatasetFlattenerCore.execute(DatasetFlattenerCore.java:61)
at
com.zillow.pipeler.core.transform.DatasetFlattenerCore.execute(DatasetFlattenerCore.java:29)
at
com.zillow.pipeler.orchestrator.BaseOrchestrator.run(BaseOrchestrator.java:61)
at
com.zillow.pipeler.orchestrator.transform.DatasetFlattenerOrchestrator.main(DatasetFlattenerOrchestrator.java:71)
Caused by:
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParseException:
Encountered "AS user" at line 1, column 23.
Was expecting one of:
<EOF>
"ORDER" ...
"LIMIT" ...
