[jira] [Commented] (CALCITE-6046) QuoteStringLiteralUnicode returns unparsed string with u&' prefix, which will cause the SqlLiteral
[
https://issues.apache.org/jira/browse/CALCITE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775755#comment-17775755
]
xiaogang zhou commented on CALCITE-6046:
Hi [~julianhyde] ,
The behavior I thought was wrong is when I use below code
{code:java}
// code placeholder
SqlParser.Config parserConfig = getCurrentSqlParserConfig(sqlDialect);
SqlParser sqlParser = SqlParser.create(sqlContent, parserConfig);
SqlNodeList sqlNodeList = sqlParser.parseStmtList();
sqlParser.parse(sqlNodeList.get(0)); {code}
to parse
{code:java}
// code placeholder
CREATE TABLE source (
a BIGINT
) comment '测试test'
WITH (
'connector' = 'test'
); {code}
then unparse it , I will get
{code:java}
// code placeholder
CREATE TABLE `source` (
`a` BIGINT
)
COMMENT u&'\5218\51eftest' WITH (
'connector' = 'test'
) {code}
which is not parsable by FLINK sql template
{code:java}
// code placeholder
[ {
String p = SqlParserUtil.parseString(token.image);
comment = SqlLiteral.createCharString(p, getPos());
}] {code}
Since you mentioned '' is Standard SQL DIALECT, I think there is nothing
wrong in CALCITE. If the statement above makes sense to you, we can just close
this CALCITE issue, and I will follow it in FLINK issue with FLINK TEAM.
> QuoteStringLiteralUnicode returns unparsed string with u&' prefix, which will
> cause the SqlLiteral
> --
>
> Key: CALCITE-6046
> URL: https://issues.apache.org/jira/browse/CALCITE-6046
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.35.0
>Reporter: xiaogang zhou
>Priority: Major
> Fix For: 1.36.0
>
>
> quoteStringLiteralUnicode returns unparsed string with u&' prefix, which will
> cause the SqlLiteral
>
> for example with a SQL
>
> {code:java}
> // code placeholder
> CREATE TABLE source (
> a BIGINT
> ) comment '测试test'
> WITH (
> 'connector' = 'test'
> ); {code}
> with a parsed Sqlnode, the toString will create a SQL like below, which is
> not parsable again.
>
> {code:java}
> // code placeholder
> CREATE TABLE `source` (
> `a` BIGINT
> )
> COMMENT u&'\5218\51eftest' WITH (
> 'connector' = 'test'
> ) {code}
> I think this is caused by
> {code:java}
> // code placeholder
> public void quoteStringLiteralUnicode(StringBuilder buf, String val) {
> buf.append("u&'"); {code}
> not sure if I misconfigured something. Is it possiable to remove the
> buf.append("u&'"); ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6046) QuoteStringLiteralUnicode returns unparsed string with u&' prefix, which will cause the SqlLiteral
[
https://issues.apache.org/jira/browse/CALCITE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775533#comment-17775533
]
Julian Hyde commented on CALCITE-6046:
--
I'll ask again. Can you change the summary to talk about the behavior that you
think is wrong. Your current summary starts with a method name, which means
that you are describing the solution, not the problem.
> QuoteStringLiteralUnicode returns unparsed string with u&' prefix, which will
> cause the SqlLiteral
> --
>
> Key: CALCITE-6046
> URL: https://issues.apache.org/jira/browse/CALCITE-6046
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.35.0
>Reporter: xiaogang zhou
>Priority: Major
> Fix For: 1.36.0
>
>
> quoteStringLiteralUnicode returns unparsed string with u&' prefix, which will
> cause the SqlLiteral
>
> for example with a SQL
>
> {code:java}
> // code placeholder
> CREATE TABLE source (
> a BIGINT
> ) comment '测试test'
> WITH (
> 'connector' = 'test'
> ); {code}
> with a parsed Sqlnode, the toString will create a SQL like below, which is
> not parsable again.
>
> {code:java}
> // code placeholder
> CREATE TABLE `source` (
> `a` BIGINT
> )
> COMMENT u&'\5218\51eftest' WITH (
> 'connector' = 'test'
> ) {code}
> I think this is caused by
> {code:java}
> // code placeholder
> public void quoteStringLiteralUnicode(StringBuilder buf, String val) {
> buf.append("u&'"); {code}
> not sure if I misconfigured something. Is it possiable to remove the
> buf.append("u&'"); ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6046) quoteStringLiteralUnicode returns unparsed string with u&' prefix, which will cause the SqlLiteral
[
https://issues.apache.org/jira/browse/CALCITE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775356#comment-17775356
]
LakeShen commented on CALCITE-6046:
---
Hi [~zhoujira86] ,please capitalize the first letter of the JIRA
title.Otherwise, you may not be able to pass the Pipeline Job inspection.
> quoteStringLiteralUnicode returns unparsed string with u&' prefix, which will
> cause the SqlLiteral
> --
>
> Key: CALCITE-6046
> URL: https://issues.apache.org/jira/browse/CALCITE-6046
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.35.0
>Reporter: xiaogang zhou
>Priority: Major
> Fix For: 1.36.0
>
>
> quoteStringLiteralUnicode returns unparsed string with u&' prefix, which will
> cause the SqlLiteral
>
> for example with a SQL
>
> {code:java}
> // code placeholder
> CREATE TABLE source (
> a BIGINT
> ) comment '测试test'
> WITH (
> 'connector' = 'test'
> ); {code}
> with a parsed Sqlnode, the toString will create a SQL like below, which is
> not parsable again.
>
> {code:java}
> // code placeholder
> CREATE TABLE `source` (
> `a` BIGINT
> )
> COMMENT u&'\5218\51eftest' WITH (
> 'connector' = 'test'
> ) {code}
> I think this is caused by
> {code:java}
> // code placeholder
> public void quoteStringLiteralUnicode(StringBuilder buf, String val) {
> buf.append("u&'"); {code}
> not sure if I misconfigured something. Is it possiable to remove the
> buf.append("u&'"); ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6046) quoteStringLiteralUnicode returns unparsed string with u&' prefix, which will cause the SqlLiteral
[
https://issues.apache.org/jira/browse/CALCITE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775096#comment-17775096
]
Julian Hyde commented on CALCITE-6046:
--
Standard SQL
> quoteStringLiteralUnicode returns unparsed string with u&' prefix, which will
> cause the SqlLiteral
> --
>
> Key: CALCITE-6046
> URL: https://issues.apache.org/jira/browse/CALCITE-6046
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.35.0
>Reporter: xiaogang zhou
>Priority: Major
> Fix For: 1.36.0
>
>
> quoteStringLiteralUnicode returns unparsed string with u&' prefix, which will
> cause the SqlLiteral
>
> for example with a SQL
>
> {code:java}
> // code placeholder
> CREATE TABLE source (
> a BIGINT
> ) comment '测试test'
> WITH (
> 'connector' = 'test'
> ); {code}
> with a parsed Sqlnode, the toString will create a SQL like below, which is
> not parsable again.
>
> {code:java}
> // code placeholder
> CREATE TABLE `source` (
> `a` BIGINT
> )
> COMMENT u&'\5218\51eftest' WITH (
> 'connector' = 'test'
> ) {code}
> I think this is caused by
> {code:java}
> // code placeholder
> public void quoteStringLiteralUnicode(StringBuilder buf, String val) {
> buf.append("u&'"); {code}
> not sure if I misconfigured something. Is it possiable to remove the
> buf.append("u&'"); ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6046) quoteStringLiteralUnicode returns unparsed string with u&' prefix, which will cause the SqlLiteral
[
https://issues.apache.org/jira/browse/CALCITE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774763#comment-17774763
]
xiaogang zhou commented on CALCITE-6046:
[~julianhyde]
Although I think it should be more proper to replace the with
StringLiteral(), I still want to consult you on the It seems not found in
[https://cloud.google.com/bigquery/docs/reference/standard-sql/lexical]
Literal part. May I ask which DIALECT we use this prefix?
> quoteStringLiteralUnicode returns unparsed string with u&' prefix, which will
> cause the SqlLiteral
> --
>
> Key: CALCITE-6046
> URL: https://issues.apache.org/jira/browse/CALCITE-6046
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.35.0
>Reporter: xiaogang zhou
>Priority: Major
> Fix For: 1.36.0
>
>
> quoteStringLiteralUnicode returns unparsed string with u&' prefix, which will
> cause the SqlLiteral
>
> for example with a SQL
>
> {code:java}
> // code placeholder
> CREATE TABLE source (
> a BIGINT
> ) comment '测试test'
> WITH (
> 'connector' = 'test'
> ); {code}
> with a parsed Sqlnode, the toString will create a SQL like below, which is
> not parsable again.
>
> {code:java}
> // code placeholder
> CREATE TABLE `source` (
> `a` BIGINT
> )
> COMMENT u&'\5218\51eftest' WITH (
> 'connector' = 'test'
> ) {code}
> I think this is caused by
> {code:java}
> // code placeholder
> public void quoteStringLiteralUnicode(StringBuilder buf, String val) {
> buf.append("u&'"); {code}
> not sure if I misconfigured something. Is it possiable to remove the
> buf.append("u&'"); ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6046) quoteStringLiteralUnicode returns unparsed string with u&' prefix, which will cause the SqlLiteral
[
https://issues.apache.org/jira/browse/CALCITE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774311#comment-17774311
]
xiaogang zhou commented on CALCITE-6046:
I understood, the problem is the parser should not parse the comment as a
[
and it should be parsed by
StringLiteral() :
> quoteStringLiteralUnicode returns unparsed string with u&' prefix, which will
> cause the SqlLiteral
> --
>
> Key: CALCITE-6046
> URL: https://issues.apache.org/jira/browse/CALCITE-6046
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.35.0
>Reporter: xiaogang zhou
>Priority: Major
> Fix For: 1.36.0
>
>
> quoteStringLiteralUnicode returns unparsed string with u&' prefix, which will
> cause the SqlLiteral
>
> for example with a SQL
>
> {code:java}
> // code placeholder
> CREATE TABLE source (
> a BIGINT
> ) comment '测试test'
> WITH (
> 'connector' = 'test'
> ); {code}
> with a parsed Sqlnode, the toString will create a SQL like below, which is
> not parsable again.
>
> {code:java}
> // code placeholder
> CREATE TABLE `source` (
> `a` BIGINT
> )
> COMMENT u&'\5218\51eftest' WITH (
> 'connector' = 'test'
> ) {code}
> I think this is caused by
> {code:java}
> // code placeholder
> public void quoteStringLiteralUnicode(StringBuilder buf, String val) {
> buf.append("u&'"); {code}
> not sure if I misconfigured something. Is it possiable to remove the
> buf.append("u&'"); ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6046) quoteStringLiteralUnicode returns unparsed string with u&' prefix, which will cause the SqlLiteral
[
https://issues.apache.org/jira/browse/CALCITE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774304#comment-17774304
]
xiaogang zhou commented on CALCITE-6046:
[~julianhyde]
Hi, I found this problem when I used below code to split SQL statements. the
process is SQL string -> SqlNode -> SQL String
{code:java}
// code placeholder
SqlParser.Config parserConfig = getCurrentSqlParserConfig(sqlDialect);
SqlParser sqlParser = SqlParser.create(sqlContent, parserConfig);
SqlNodeList sqlNodeList = sqlParser.parseStmtList();
sqlParser.parse(sqlNodeList.get(0));{code}
the Dialect/ SqlConformance is a costumed one:
[https://github.com/apache/flink/blob/master/flink-table/flink-sql-parser/src/main/java/org/apache/flink/sql/parser/validate/FlinkSqlConformance.java]
then I found below SQL
{code:java}
// code placeholder
CREATE TABLE source (
a BIGINT
) comment '测试test'
WITH (
'connector' = 'test'
); {code}
transformed to
{code:java}
// code placeholder
CREATE TABLE `source` (
`a` BIGINT
)
COMMENT u&'\5218\51eftest' WITH (
'connector' = 'test'
) {code}
and the SQL parser template is like
{code:java}
// code placeholder
SqlCreate SqlCreateTable(Span s, boolean replace, boolean isTemporary) :
{
final SqlParserPos startPos = s.pos();
boolean ifNotExists = false;
SqlIdentifier tableName;
List constraints = new ArrayList();
SqlWatermark watermark = null;
SqlNodeList columnList = SqlNodeList.EMPTY;
SqlCharStringLiteral comment = null;
SqlTableLike tableLike = null;
SqlNode asQuery = null;
SqlNodeList propertyList = SqlNodeList.EMPTY;
SqlNodeList partitionColumns = SqlNodeList.EMPTY;
SqlParserPos pos = startPos;
}
{
ifNotExists = IfNotExistsOpt()
tableName = CompoundIdentifier()
[
{ pos = getPos(); TableCreationContext ctx = new
TableCreationContext();}
TableColumn(ctx)
(
TableColumn(ctx)
)*
{
pos = pos.plus(getPos());
columnList = new SqlNodeList(ctx.columnList, pos);
constraints = ctx.constraints;
watermark = ctx.watermark;
}
]
[ {
String p = SqlParserUtil.parseString(token.image);
comment = SqlLiteral.createCharString(p, getPos());
}]
[
partitionColumns = ParenthesizedSimpleIdentifierList()
]
[
propertyList = TableProperties()
]
[
tableLike = SqlTableLike(getPos())
{
return new SqlCreateTableLike(startPos.plus(getPos()),
tableName,
columnList,
constraints,
propertyList,
partitionColumns,
watermark,
comment,
tableLike,
isTemporary,
ifNotExists);
}
|
asQuery = OrderedQueryOrExpr(ExprContext.ACCEPT_QUERY)
{
return new SqlCreateTableAs(startPos.plus(getPos()),
tableName,
columnList,
constraints,
propertyList,
partitionColumns,
watermark,
comment,
asQuery,
isTemporary,
ifNotExists);
}
]
{
return new SqlCreateTable(startPos.plus(getPos()),
tableName,
columnList,
constraints,
propertyList,
partitionColumns,
watermark,
comment,
isTemporary,
ifNotExists);
}
} {code}
will give a exception :
Caused by: org.apache.calcite.sql.parser.SqlParseException: Encountered
"u&\'\\5218\\51eftest\'" at line 4, column 9.
Was expecting:
...
> quoteStringLiteralUnicode returns unparsed string with u&' prefix, which will
> cause the SqlLiteral
> --
>
> Key: CALCITE-6046
> URL: https://issues.apache.org/jira/browse/CALCITE-6046
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.35.0
>Reporter: xiaogang zhou
>Priority: Major
> Fix For: 1.36.0
>
>
> quoteStringLiteralUnicode returns unparsed string with u&' prefix, which will
> cause the SqlLiteral
>
> for example with a SQL
>
> {code:java}
> // code placeholder
> CREATE TABLE source (
> a BIGINT
> ) comment '测试test'
> WITH (
> 'connector' = 'test'
> ); {code}
> with a parsed Sqlnode, the toString will create a SQL like below, which is
> not parsable again.
>
> {code:java}
> // code placeholder
> CREATE TABLE `source` (
> `a` BIGINT
> )
> COMMENT u&'\5218\51eftest' WITH (
> 'connector' = 'test'
> ) {code}
> I think this is caused by
>
[jira] [Commented] (CALCITE-6046) quoteStringLiteralUnicode returns unparsed string with u&' prefix, which will cause the SqlLiteral
[
https://issues.apache.org/jira/browse/CALCITE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774227#comment-17774227
]
Julian Hyde commented on CALCITE-6046:
--
Can you talk about the behavior rather than the code. What SQL does it
generate? On which dialect(s) is that SQL invalid? The summary in particular
should talk about behavior.
Only when you have defined the problem, you can start talking about the fix.
> quoteStringLiteralUnicode returns unparsed string with u&' prefix, which will
> cause the SqlLiteral
> --
>
> Key: CALCITE-6046
> URL: https://issues.apache.org/jira/browse/CALCITE-6046
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.35.0
>Reporter: xiaogang zhou
>Priority: Major
> Fix For: 1.36.0
>
>
> quoteStringLiteralUnicode returns unparsed string with u&' prefix, which will
> cause the SqlLiteral
>
> for example with a SQL
>
> {code:java}
> // code placeholder
> CREATE TABLE source (
> a BIGINT
> ) comment '测试test'
> WITH (
> 'connector' = 'test'
> ); {code}
> with a parsed Sqlnode, the toString will create a SQL like below, which is
> not parsable again.
>
> {code:java}
> // code placeholder
> CREATE TABLE `source` (
> `a` BIGINT
> )
> COMMENT u&'\5218\51eftest' WITH (
> 'connector' = 'test'
> ) {code}
> I think this is caused by
> {code:java}
> // code placeholder
> public void quoteStringLiteralUnicode(StringBuilder buf, String val) {
> buf.append("u&'"); {code}
> not sure if I misconfigured something. Is it possiable to remove the
> buf.append("u&'"); ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
