Hi, @Benchao you are right, no, the question did not intend to block this FLIP.
@Shammon It looks quite interesting, if we imagine how the Flink jdbc connector could work with Flink jdbc driver. It makes "on-the-fly" cross-flink-applications data consumption possible, i.e. bounded source scan, lookup source from another flink job, and sink to another flink job. The streaming append mode of the sink could sort of turn the data pipeline into streaming mode. All of these could not be done in Flink jobs directly, if I am not mistaken. Best regards, Jing On Mon, Mar 13, 2023 at 3:38 AM Benchao Li <libenc...@apache.org> wrote: > Hi Jing & Shammon, > > IMO, Jdbc is a Java standard and Flink-JDBC-Driver is an implementation to > interact with Flink, so in theory, it would be doable to use > Flink-JDBC-Driver in Flink-JDBC-Connector if we add it as a dialect. > > However, I don't see a scenario which requires this yet. All in all, it > would be an orthogonal topic, which does not block current FLIP discussion. > > Shammon FY <zjur...@gmail.com> 于2023年3月13日周一 08:47写道： > > > Hi Jing > > > > Flink-Jdbc-Driver is mainly used to facilitate users to submit flink sql > > jobs. I think we will not use Flink-Jdbc-Driver in Flink-Jdbc-Connector > to > > read/write data in general. If users can read/write specific tables by > > Flink-Jdbc-Driver, they can use these tables in their flink jobs > directly. > > Thanks > > > > Best, > > Shammon > > > > > > On Mon, Mar 13, 2023 at 6:21 AM Jing Ge <j...@ververica.com.invalid> > > wrote: > > > > > Hi, Shammon, > > > > > > This FLIP is very interesting. Thanks for driving it! Just out of > > > curiosity, Could it be used in the Flink JDBC connector[1] after > > > doing the appropriate implementation there? > > > > > > Best regards, > > > Jing > > > > > > [1] > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/jdbc/ > > > > > > On Fri, Mar 3, 2023 at 8:56 AM Shammon FY <zjur...@gmail.com> wrote: > > > > > > > Hi, > > > > > > > > Thanks jinsong. I think implementing JDBC for batch mode first sounds > > > good. > > > > This will simplify the implementation and we can also remove the row > > kind > > > > first. We can claim this in the FLIP and docs, I will update the > FLIP. > > > > > > > > Best, > > > > Shammon > > > > > > > > > > > > On Fri, Mar 3, 2023 at 2:36 PM Jingsong Li <jingsongl...@gmail.com> > > > wrote: > > > > > > > > > Hi, Shammon, > > > > > > > > > > I took a look at JDBC `ResultSet` and `Statement`. They are > > > > > complicated and have many interfaces. Some of the interfaces may > not > > > > > be very suitable for streaming. > > > > > > > > > > I think maybe we can just implement JDBC for batch/olap only. It is > > > > > hard to have an integration for JDBC and streaming... > > > > > > > > > > Do you need to use JDBC in streaming mode? Or do we just implement > > > > > JDBC for batch only first? > > > > > > > > > > Best, > > > > > Jingsong > > > > > > > > > > > > > > > On Thu, Mar 2, 2023 at 6:22 PM Shammon FY <zjur...@gmail.com> > wrote: > > > > > > > > > > > > Hi > > > > > > > > > > > > Thanks for the feedback from Jingsong and Benchao. > > > > > > > > > > > > For @Jingsong > > > > > > > If the user does not cast into a FlinkResultSet, will there be > > > > > > serious consequences here (RowKind is ignored)? > > > > > > > > > > > > I agree with you that it's indeed a big deal if users ignore the > > row > > > > kind > > > > > > when they must know it. One idea that comes to my mind is we can > > add > > > an > > > > > > option such as `table.result.changelog-mode`, users can set it > > > through > > > > > > connection properties or set dynamic parameters. The option value > > can > > > > be > > > > > > `insert-only`, `upset` or `all` and the default value is > > > `insert-only`. > > > > > > > > > > > > If the result does not conform to the changelog mode, the jdbc > > driver > > > > > > throws an exception. What do you think? > > > > > > > > > > > > > > > > > > For @Benchao > > > > > > > Besides `java.sql.Driver`, have you considered also adding > > support > > > > for > > > > > > `javax.sql.DataSource` interface? > > > > > > > > > > > > I missed the `javax.sql.DataSource` and I have added it to the > > FLIP, > > > > > thanks > > > > > > Benchao > > > > > > > > > > > > > > > > > > Best, > > > > > > Shammon > > > > > > > > > > > > On Wed, Mar 1, 2023 at 7:57 PM Benchao Li <libenc...@apache.org> > > > > wrote: > > > > > > > > > > > > > +1 for the FLIP, thanks Shammon for driving this. > > > > > > > > > > > > > > JDBC is quite useful in OLAP scenarios, supporting JDBC would > > > enable > > > > > Flink > > > > > > > to be used with existing tools, such as Tableau. > > > > > > > > > > > > > > Regarding the JDBC interfaces listed in the FLIP, I think they > > > looks > > > > > good > > > > > > > already. Besides `java.sql.Driver`, have you considered also > > adding > > > > > support > > > > > > > for `javax.sql.DataSource` interface? > > > > > > > > > > > > > > Jingsong Li <jingsongl...@gmail.com> 于2023年3月1日周三 17:53写道： > > > > > > > > > > > > > > > Thanks Shammon for driving. > > > > > > > > > > > > > > > > Big +1 for this. > > > > > > > > > > > > > > > > I heard that many users want to use FlinkGateway + JDBC to do > > > some > > > > > > > > queries, but at present, only Hive JDBC can be used. It is > Hive > > > > > > > > dialect by default, and the experience is also different from > > > > > > > > FlinkSQL. We need to have our own JDBC. > > > > > > > > > > > > > > > > I took a look at your `Public Interface` part, only > > > > > > > > `FlinkResultSet.getRowKind` is a true new interface, others > are > > > > just > > > > > > > > implementations. > > > > > > > > > > > > > > > > If the user does not cast into a FlinkResultSet, will there > be > > > > > serious > > > > > > > > consequences here (RowKind is ignored)? > > > > > > > > > > > > > > > > Best, > > > > > > > > Jingsong > > > > > > > > > > > > > > > > On Wed, Mar 1, 2023 at 4:59 PM Shammon FY <zjur...@gmail.com > > > > > > wrote: > > > > > > > > > > > > > > > > > > Hi devs, > > > > > > > > > > > > > > > > > > I'd like to start a discussion about FLIP-293: Introduce > > Flink > > > > Jdbc > > > > > > > > Driver > > > > > > > > > For Sql Gateway[1]. > > > > > > > > > > > > > > > > > > FLIP-275[2] supports remote sql client based on gateway, > > users > > > > can > > > > > > > > interact > > > > > > > > > with gateway by flink console. However, for users who > create > > > > > session > > > > > > > > > clusters with Flink, they'd like to use Jdbc Driver to > > interact > > > > > with > > > > > > > the > > > > > > > > > gateway in their applications, such as olap queries.. > > > > > > > > > > > > > > > > > > I have discussed this proposal with @shengkaifang and > > > > @jinsonglee. > > > > > In > > > > > > > > this > > > > > > > > > FLIP, we'd like to introduce Jdbc Driver for gateway. Users > > can > > > > use > > > > > > > Jdbc > > > > > > > > > Driver to submit their queries and get results like a > > database > > > in > > > > > their > > > > > > > > > applications. > > > > > > > > > > > > > > > > > > Looking forward to your feedback, thanks. > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-293%3A+Introduce+Flink+Jdbc+Driver+For+Sql+Gateway > > > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-275%3A+Support+Remote+SQL+Client+Based+on+SQL+Gateway > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Shammon > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Best, > > > > > > > Benchao Li > > > > > > > > > > > > > > > > > > > > > > > > -- > > Best, > Benchao Li >
