Re: 关于Flink SQL语句内的函数名和内建函数名称对应不上的问题
感谢大佬,我找到了。 所以说SQL类的内建函数实际上使用的是calcite的能力,而flink自己的内建函数是在table api中使用 > 2023年11月24日 17:07,Xuyang 写道: > > Hi, > 关于你举的例子,如果编译了源码的话,可以在FlinkSqlParserImpl这个被动态生成的词法解析器类中找到PostfixRowOperator方法,大致是通过识别到IS > NOT NULL这三个关键字,转化为Calcite的这个内置函数SqlStdOperatorTable.IS_NOT_NULL > > > > > -- > >Best! >Xuyang > > > > > > 在 2023-11-24 15:15:04,"jinzhuguang" 写道: >> flink 1.18.0 >> >> >> 例如我写下一条SQL: >> select * from KafkaTable where id is not null; >> >> IS NOT NULL应该属于系统内建函数,于是我找到相关代码: >> >> public static final BuiltInFunctionDefinition IS_NOT_NULL = >> BuiltInFunctionDefinition.newBuilder() >> .name("isNotNull") >> .kind(SCALAR) >> >> .inputTypeStrategy(wildcardWithCount(ConstantArgumentCount.of(1))) >> >> .outputTypeStrategy(explicit(DataTypes.BOOLEAN().notNull())) >> .build(); >> >> 发现他的name是“ isNotNull”,和“is not null”对应不上。并且经过实际测验,确实证实了我的猜想: >> >> DEBUG org.apache.flink.table.module.ModuleManager [] - >> Cannot find FunctionDefinition 'is not null' from any loaded modules. >> >> >> 所以我很疑惑,SQL到底是在哪里找到了”is not null”这个函数的呢? >> >> 以下是调用栈: >> @org.apache.flink.table.planner.catalog.FunctionCatalogOperatorTable.lookupOperatorOverloads() >> at >> org.apache.calcite.sql.util.ChainedSqlOperatorTable.lookupOperatorOverloads(ChainedSqlOperatorTable.java:69) >> at >> org.apache.calcite.sql.SqlUtil.lookupSubjectRoutinesByName(SqlUtil.java:609) >> at >> org.apache.calcite.sql.SqlUtil.lookupSubjectRoutines(SqlUtil.java:535) >> at org.apache.calcite.sql.SqlUtil.lookupRoutine(SqlUtil.java:486) >> at org.apache.calcite.sql.SqlOperator.deriveType(SqlOperator.java:595) >> at >> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6302) >> at >> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6287) >> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:161) >> at >> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1869) >> at >> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1860) >> at >> org.apache.calcite.sql.validate.SqlValidatorImpl.validateWhereOrOn(SqlValidatorImpl.java:4341) >> at >> org.apache.calcite.sql.validate.SqlValidatorImpl.validateWhereClause(SqlValidatorImpl.java:4333) >> at >> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3606) >> at >> org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:64) >> at >> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:89) >> at >> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1050) >> at >> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:1025) >> at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:248) >> at >> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:1000) >> at >> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:749) >> at >> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:196) >> at >> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:117) >> at >> org.apache.flink.table.planner.operations.SqlNodeToOperationConversion.convert(SqlNodeToOperationConversion.java:261) >> at >> org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:106) >> at >> org.apache.flink.table.gateway.service.operation.OperationExecutor.executeStatement(OperationExecutor.java:187) >> at >> org.apache.flink.table.gateway.service.SqlGatewayServiceImpl.lambda$executeStatement$1(SqlGatewayServiceImpl.java:212) >> at >> org.apache.flink.table.gateway.service.operation.OperationManager.lambda$submitOperation$1(OperationManager.java:119) >> at >> org.apache.flink.table.gateway.service.operation.OperationManager$Operation.lambda$run$0(OperationManager.java:258) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> at java.lang.Thread.run(Thread.java:750)
Re:关于Flink SQL语句内的函数名和内建函数名称对应不上的问题
Hi, 关于你举的例子,如果编译了源码的话,可以在FlinkSqlParserImpl这个被动态生成的词法解析器类中找到PostfixRowOperator方法,大致是通过识别到IS NOT NULL这三个关键字,转化为Calcite的这个内置函数SqlStdOperatorTable.IS_NOT_NULL -- Best! Xuyang 在 2023-11-24 15:15:04,"jinzhuguang" 写道: >flink 1.18.0 > > >例如我写下一条SQL: > select * from KafkaTable where id is not null; > >IS NOT NULL应该属于系统内建函数,于是我找到相关代码: > >public static final BuiltInFunctionDefinition IS_NOT_NULL = >BuiltInFunctionDefinition.newBuilder() >.name("isNotNull") >.kind(SCALAR) > > .inputTypeStrategy(wildcardWithCount(ConstantArgumentCount.of(1))) > > .outputTypeStrategy(explicit(DataTypes.BOOLEAN().notNull())) >.build(); > >发现他的name是“ isNotNull”,和“is not null”对应不上。并且经过实际测验,确实证实了我的猜想: > >DEBUG org.apache.flink.table.module.ModuleManager [] - Cannot >find FunctionDefinition 'is not null' from any loaded modules. > > >所以我很疑惑,SQL到底是在哪里找到了”is not null”这个函数的呢? > >以下是调用栈: >@org.apache.flink.table.planner.catalog.FunctionCatalogOperatorTable.lookupOperatorOverloads() >at > org.apache.calcite.sql.util.ChainedSqlOperatorTable.lookupOperatorOverloads(ChainedSqlOperatorTable.java:69) >at > org.apache.calcite.sql.SqlUtil.lookupSubjectRoutinesByName(SqlUtil.java:609) >at > org.apache.calcite.sql.SqlUtil.lookupSubjectRoutines(SqlUtil.java:535) >at org.apache.calcite.sql.SqlUtil.lookupRoutine(SqlUtil.java:486) >at org.apache.calcite.sql.SqlOperator.deriveType(SqlOperator.java:595) >at > org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6302) >at > org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6287) >at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:161) >at > org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1869) >at > org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1860) >at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateWhereOrOn(SqlValidatorImpl.java:4341) >at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateWhereClause(SqlValidatorImpl.java:4333) >at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3606) >at > org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:64) >at > org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:89) >at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1050) >at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:1025) >at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:248) >at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:1000) >at > org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:749) >at > org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:196) >at > org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:117) >at > org.apache.flink.table.planner.operations.SqlNodeToOperationConversion.convert(SqlNodeToOperationConversion.java:261) >at > org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:106) >at > org.apache.flink.table.gateway.service.operation.OperationExecutor.executeStatement(OperationExecutor.java:187) >at > org.apache.flink.table.gateway.service.SqlGatewayServiceImpl.lambda$executeStatement$1(SqlGatewayServiceImpl.java:212) >at > org.apache.flink.table.gateway.service.operation.OperationManager.lambda$submitOperation$1(OperationManager.java:119) >at > org.apache.flink.table.gateway.service.operation.OperationManager$Operation.lambda$run$0(OperationManager.java:258) >at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >at java.util.concurrent.FutureTask.run(FutureTask.java:266) >at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >at java.util.concurrent.FutureTask.run(FutureTask.java:266) >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >at java.lang.Thread.run(Thread.java:750)
关于Flink SQL语句内的函数名和内建函数名称对应不上的问题
flink 1.18.0 例如我写下一条SQL: select * from KafkaTable where id is not null; IS NOT NULL应该属于系统内建函数,于是我找到相关代码: public static final BuiltInFunctionDefinition IS_NOT_NULL = BuiltInFunctionDefinition.newBuilder() .name("isNotNull") .kind(SCALAR) .inputTypeStrategy(wildcardWithCount(ConstantArgumentCount.of(1))) .outputTypeStrategy(explicit(DataTypes.BOOLEAN().notNull())) .build(); 发现他的name是“ isNotNull”,和“is not null”对应不上。并且经过实际测验,确实证实了我的猜想: DEBUG org.apache.flink.table.module.ModuleManager [] - Cannot find FunctionDefinition 'is not null' from any loaded modules. 所以我很疑惑,SQL到底是在哪里找到了”is not null”这个函数的呢? 以下是调用栈: @org.apache.flink.table.planner.catalog.FunctionCatalogOperatorTable.lookupOperatorOverloads() at org.apache.calcite.sql.util.ChainedSqlOperatorTable.lookupOperatorOverloads(ChainedSqlOperatorTable.java:69) at org.apache.calcite.sql.SqlUtil.lookupSubjectRoutinesByName(SqlUtil.java:609) at org.apache.calcite.sql.SqlUtil.lookupSubjectRoutines(SqlUtil.java:535) at org.apache.calcite.sql.SqlUtil.lookupRoutine(SqlUtil.java:486) at org.apache.calcite.sql.SqlOperator.deriveType(SqlOperator.java:595) at org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6302) at org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6287) at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:161) at org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1869) at org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1860) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateWhereOrOn(SqlValidatorImpl.java:4341) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateWhereClause(SqlValidatorImpl.java:4333) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3606) at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:64) at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:89) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1050) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:1025) at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:248) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:1000) at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:749) at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:196) at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:117) at org.apache.flink.table.planner.operations.SqlNodeToOperationConversion.convert(SqlNodeToOperationConversion.java:261) at org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:106) at org.apache.flink.table.gateway.service.operation.OperationExecutor.executeStatement(OperationExecutor.java:187) at org.apache.flink.table.gateway.service.SqlGatewayServiceImpl.lambda$executeStatement$1(SqlGatewayServiceImpl.java:212) at org.apache.flink.table.gateway.service.operation.OperationManager.lambda$submitOperation$1(OperationManager.java:119) at org.apache.flink.table.gateway.service.operation.OperationManager$Operation.lambda$run$0(OperationManager.java:258) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750)