To answer your initial question, the parser factory can be replaced here: https://github.com/apache/beam/blob/7c43ab6a8df9b23caa7321fddff9a032a71908f6/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcFactory.java#L98
The support from DDL in Beam is a copy of the calcite/server module (as recommended by Calcite). It looks like this code path has been completely rewritten in Calcite since 1.20. We copied files from https://github.com/apache/calcite/tree/branch-1.20/server/src/main/java/org/apache/calcite/sql/ddl To here: https://github.com/apache/beam/tree/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser Looks like all of that has been replaced with https://github.com/apache/calcite/blob/master/server/src/main/java/org/apache/calcite/server/ServerDdlExecutor.java in Calcite and Beam will need to be updated and the parser factory from that file used. Also of note: BeamSqlParserImpl is generated from this source file: https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/codegen/config.fmpp Which was copied from https://github.com/apache/calcite/blob/master/server/src/main/codegen/config.fmpp Thanks for working on this, Andrew On Sun, Dec 6, 2020 at 11:58 AM Niels Basjes <ni...@basjes.nl> wrote: > Hi, > > I'm trying to update the version of Calcite used in Beam to the latest > version available. > https://issues.apache.org/jira/browse/BEAM-9379 > https://github.com/apache/beam/pull/12962 > > I've hit a bit of a roadblock that I'm asking your help for. > > What I ran into is that some tests in the current state of this pull > request fail over a change in Calcite. > I am specifically talking about > https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineTest.java#L63 > and many other tests in this file that do a CREATE EXTERNAL TABLE or a DROP > TABLE (i.e. do DDL stuff). > > I tracked the source of this exception back to Calcite where a few months > ago a DdlExecutor was added and this default method for `DdlExecutor > getDdlExecutor()` was added in Calcite: > https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/parser/SqlParserImplFactory.java#L58 > > This default implementation simply returns a DdlExecutor (a new interface) > that always fails with a UnsupportedOperationException if no valid > implementation has been provided. > This is marked as experimental and looks good to me so far. > > I found in the generated code of BeamSqlParserImpl this code (I shortened > it a bit) > > public static final SqlParserImplFactory FACTORY = new > SqlParserImplFactory() { > public SqlAbstractParserImpl getParser(Reader reader) { > final BeamSqlParserImpl parser = new BeamSqlParserImpl(reader); > ... > return parser; > } > }; > > which is generated from a template in Calcite itself and (as far as I have > been able to find so far) does not have a way of implementing a non-default > method for DdlExecutor getDdlExecutor(). > > How do we fix this? > or ... can we fix this in Beam or is actually a change in Calcite needed? > > -- > Best regards / Met vriendelijke groeten, > > Niels Basjes >
