Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway
Hi Jing I think your thoughts are interesting. We can consider a flink cluster with Jdbc-Driver as a warehouse and flink jobs can interact with it by Flink-Jdbc-Connector. This may have higher requirements for the throughput and latency of the warehouse (flink cluster), but we can try it after this FLIP Best, Shammon On Mon, Mar 13, 2023 at 3:52 PM Jing Ge wrote: > Hi, > > @Benchao > you are right, no, the question did not intend to block this FLIP. > > @Shammon > It looks quite interesting, if we imagine how the Flink jdbc connector > could work with Flink jdbc driver. It makes "on-the-fly" > cross-flink-applications data consumption possible, i.e. bounded source > scan, lookup source from another flink job, and sink to another flink job. > The streaming append mode of the sink could sort of turn the data pipeline > into streaming mode. All of these could not be done in Flink jobs directly, > if I am not mistaken. > > Best regards, > Jing > > > On Mon, Mar 13, 2023 at 3:38 AM Benchao Li wrote: > > > Hi Jing & Shammon, > > > > IMO, Jdbc is a Java standard and Flink-JDBC-Driver is an implementation > to > > interact with Flink, so in theory, it would be doable to use > > Flink-JDBC-Driver in Flink-JDBC-Connector if we add it as a dialect. > > > > However, I don't see a scenario which requires this yet. All in all, it > > would be an orthogonal topic, which does not block current FLIP > discussion. > > > > Shammon FY 于2023年3月13日周一 08:47写道： > > > > > Hi Jing > > > > > > Flink-Jdbc-Driver is mainly used to facilitate users to submit flink > sql > > > jobs. I think we will not use Flink-Jdbc-Driver in Flink-Jdbc-Connector > > to > > > read/write data in general. If users can read/write specific tables by > > > Flink-Jdbc-Driver, they can use these tables in their flink jobs > > directly. > > > Thanks > > > > > > Best, > > > Shammon > > > > > > > > > On Mon, Mar 13, 2023 at 6:21 AM Jing Ge > > > wrote: > > > > > > > Hi, Shammon, > > > > > > > > This FLIP is very interesting. Thanks for driving it! Just out of > > > > curiosity, Could it be used in the Flink JDBC connector[1] after > > > > doing the appropriate implementation there? > > > > > > > > Best regards, > > > > Jing > > > > > > > > [1] > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/jdbc/ > > > > > > > > On Fri, Mar 3, 2023 at 8:56 AM Shammon FY wrote: > > > > > > > > > Hi, > > > > > > > > > > Thanks jinsong. I think implementing JDBC for batch mode first > sounds > > > > good. > > > > > This will simplify the implementation and we can also remove the > row > > > kind > > > > > first. We can claim this in the FLIP and docs, I will update the > > FLIP. > > > > > > > > > > Best, > > > > > Shammon > > > > > > > > > > > > > > > On Fri, Mar 3, 2023 at 2:36 PM Jingsong Li > > > > > wrote: > > > > > > > > > > > Hi, Shammon, > > > > > > > > > > > > I took a look at JDBC `ResultSet` and `Statement`. They are > > > > > > complicated and have many interfaces. Some of the interfaces may > > not > > > > > > be very suitable for streaming. > > > > > > > > > > > > I think maybe we can just implement JDBC for batch/olap only. It > is > > > > > > hard to have an integration for JDBC and streaming... > > > > > > > > > > > > Do you need to use JDBC in streaming mode? Or do we just > implement > > > > > > JDBC for batch only first? > > > > > > > > > > > > Best, > > > > > > Jingsong > > > > > > > > > > > > > > > > > > On Thu, Mar 2, 2023 at 6:22 PM Shammon FY > > wrote: > > > > > > > > > > > > > > Hi > > > > > > > > > > > > > > Thanks for the feedback from Jingsong and Benchao. > > > > > > > > > > > > > > For @Jingsong > > > > > > > > If the user does not cast into a FlinkResultSet, will there > be > > > > > > > serious consequences here (RowKind is ignored)? > > > > > > > > > > > > > > I agree with you that it's indeed a big deal if users ignore > the > > > row > > > > > kind > > > > > > > when they must know it. One idea that comes to my mind is we > can > > > add > > > > an > > > > > > > option such as `table.result.changelog-mode`, users can set it > > > > through > > > > > > > connection properties or set dynamic parameters. The option > value > > > can > > > > > be > > > > > > > `insert-only`, `upset` or `all` and the default value is > > > > `insert-only`. > > > > > > > > > > > > > > If the result does not conform to the changelog mode, the jdbc > > > driver > > > > > > > throws an exception. What do you think? > > > > > > > > > > > > > > > > > > > > > For @Benchao > > > > > > > > Besides `java.sql.Driver`, have you considered also adding > > > support > > > > > for > > > > > > > `javax.sql.DataSource` interface? > > > > > > > > > > > > > > I missed the `javax.sql.DataSource` and I have added it to the > > > FLIP, > > > > > > thanks > > > > > > > Benchao > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > Shammon > > > > > > > > > > > > > > On Wed, Mar 1, 2023 at 7:57 PM Benchao Li < >
Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway
Hi, @Benchao you are right, no, the question did not intend to block this FLIP. @Shammon It looks quite interesting, if we imagine how the Flink jdbc connector could work with Flink jdbc driver. It makes "on-the-fly" cross-flink-applications data consumption possible, i.e. bounded source scan, lookup source from another flink job, and sink to another flink job. The streaming append mode of the sink could sort of turn the data pipeline into streaming mode. All of these could not be done in Flink jobs directly, if I am not mistaken. Best regards, Jing On Mon, Mar 13, 2023 at 3:38 AM Benchao Li wrote: > Hi Jing & Shammon, > > IMO, Jdbc is a Java standard and Flink-JDBC-Driver is an implementation to > interact with Flink, so in theory, it would be doable to use > Flink-JDBC-Driver in Flink-JDBC-Connector if we add it as a dialect. > > However, I don't see a scenario which requires this yet. All in all, it > would be an orthogonal topic, which does not block current FLIP discussion. > > Shammon FY 于2023年3月13日周一 08:47写道： > > > Hi Jing > > > > Flink-Jdbc-Driver is mainly used to facilitate users to submit flink sql > > jobs. I think we will not use Flink-Jdbc-Driver in Flink-Jdbc-Connector > to > > read/write data in general. If users can read/write specific tables by > > Flink-Jdbc-Driver, they can use these tables in their flink jobs > directly. > > Thanks > > > > Best, > > Shammon > > > > > > On Mon, Mar 13, 2023 at 6:21 AM Jing Ge > > wrote: > > > > > Hi, Shammon, > > > > > > This FLIP is very interesting. Thanks for driving it! Just out of > > > curiosity, Could it be used in the Flink JDBC connector[1] after > > > doing the appropriate implementation there? > > > > > > Best regards, > > > Jing > > > > > > [1] > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/jdbc/ > > > > > > On Fri, Mar 3, 2023 at 8:56 AM Shammon FY wrote: > > > > > > > Hi, > > > > > > > > Thanks jinsong. I think implementing JDBC for batch mode first sounds > > > good. > > > > This will simplify the implementation and we can also remove the row > > kind > > > > first. We can claim this in the FLIP and docs, I will update the > FLIP. > > > > > > > > Best, > > > > Shammon > > > > > > > > > > > > On Fri, Mar 3, 2023 at 2:36 PM Jingsong Li > > > wrote: > > > > > > > > > Hi, Shammon, > > > > > > > > > > I took a look at JDBC `ResultSet` and `Statement`. They are > > > > > complicated and have many interfaces. Some of the interfaces may > not > > > > > be very suitable for streaming. > > > > > > > > > > I think maybe we can just implement JDBC for batch/olap only. It is > > > > > hard to have an integration for JDBC and streaming... > > > > > > > > > > Do you need to use JDBC in streaming mode? Or do we just implement > > > > > JDBC for batch only first? > > > > > > > > > > Best, > > > > > Jingsong > > > > > > > > > > > > > > > On Thu, Mar 2, 2023 at 6:22 PM Shammon FY > wrote: > > > > > > > > > > > > Hi > > > > > > > > > > > > Thanks for the feedback from Jingsong and Benchao. > > > > > > > > > > > > For @Jingsong > > > > > > > If the user does not cast into a FlinkResultSet, will there be > > > > > > serious consequences here (RowKind is ignored)? > > > > > > > > > > > > I agree with you that it's indeed a big deal if users ignore the > > row > > > > kind > > > > > > when they must know it. One idea that comes to my mind is we can > > add > > > an > > > > > > option such as `table.result.changelog-mode`, users can set it > > > through > > > > > > connection properties or set dynamic parameters. The option value > > can > > > > be > > > > > > `insert-only`, `upset` or `all` and the default value is > > > `insert-only`. > > > > > > > > > > > > If the result does not conform to the changelog mode, the jdbc > > driver > > > > > > throws an exception. What do you think? > > > > > > > > > > > > > > > > > > For @Benchao > > > > > > > Besides `java.sql.Driver`, have you considered also adding > > support > > > > for > > > > > > `javax.sql.DataSource` interface? > > > > > > > > > > > > I missed the `javax.sql.DataSource` and I have added it to the > > FLIP, > > > > > thanks > > > > > > Benchao > > > > > > > > > > > > > > > > > > Best, > > > > > > Shammon > > > > > > > > > > > > On Wed, Mar 1, 2023 at 7:57 PM Benchao Li > > > > wrote: > > > > > > > > > > > > > +1 for the FLIP, thanks Shammon for driving this. > > > > > > > > > > > > > > JDBC is quite useful in OLAP scenarios, supporting JDBC would > > > enable > > > > > Flink > > > > > > > to be used with existing tools, such as Tableau. > > > > > > > > > > > > > > Regarding the JDBC interfaces listed in the FLIP, I think they > > > looks > > > > > good > > > > > > > already. Besides `java.sql.Driver`, have you considered also > > adding > > > > > support > > > > > > > for `javax.sql.DataSource` interface? > > > > > > > > > > > > > > Jingsong Li 于2023年3月1日周三 17:53写道： > > > > > > > > > > > > > > > Thanks Shammon for driving. > > > > > > > > > >
Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway
Hi Jing & Shammon, IMO, Jdbc is a Java standard and Flink-JDBC-Driver is an implementation to interact with Flink, so in theory, it would be doable to use Flink-JDBC-Driver in Flink-JDBC-Connector if we add it as a dialect. However, I don't see a scenario which requires this yet. All in all, it would be an orthogonal topic, which does not block current FLIP discussion. Shammon FY 于2023年3月13日周一 08:47写道： > Hi Jing > > Flink-Jdbc-Driver is mainly used to facilitate users to submit flink sql > jobs. I think we will not use Flink-Jdbc-Driver in Flink-Jdbc-Connector to > read/write data in general. If users can read/write specific tables by > Flink-Jdbc-Driver, they can use these tables in their flink jobs directly. > Thanks > > Best, > Shammon > > > On Mon, Mar 13, 2023 at 6:21 AM Jing Ge > wrote: > > > Hi, Shammon, > > > > This FLIP is very interesting. Thanks for driving it! Just out of > > curiosity, Could it be used in the Flink JDBC connector[1] after > > doing the appropriate implementation there? > > > > Best regards, > > Jing > > > > [1] > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/jdbc/ > > > > On Fri, Mar 3, 2023 at 8:56 AM Shammon FY wrote: > > > > > Hi, > > > > > > Thanks jinsong. I think implementing JDBC for batch mode first sounds > > good. > > > This will simplify the implementation and we can also remove the row > kind > > > first. We can claim this in the FLIP and docs, I will update the FLIP. > > > > > > Best, > > > Shammon > > > > > > > > > On Fri, Mar 3, 2023 at 2:36 PM Jingsong Li > > wrote: > > > > > > > Hi, Shammon, > > > > > > > > I took a look at JDBC `ResultSet` and `Statement`. They are > > > > complicated and have many interfaces. Some of the interfaces may not > > > > be very suitable for streaming. > > > > > > > > I think maybe we can just implement JDBC for batch/olap only. It is > > > > hard to have an integration for JDBC and streaming... > > > > > > > > Do you need to use JDBC in streaming mode? Or do we just implement > > > > JDBC for batch only first? > > > > > > > > Best, > > > > Jingsong > > > > > > > > > > > > On Thu, Mar 2, 2023 at 6:22 PM Shammon FY wrote: > > > > > > > > > > Hi > > > > > > > > > > Thanks for the feedback from Jingsong and Benchao. > > > > > > > > > > For @Jingsong > > > > > > If the user does not cast into a FlinkResultSet, will there be > > > > > serious consequences here (RowKind is ignored)? > > > > > > > > > > I agree with you that it's indeed a big deal if users ignore the > row > > > kind > > > > > when they must know it. One idea that comes to my mind is we can > add > > an > > > > > option such as `table.result.changelog-mode`, users can set it > > through > > > > > connection properties or set dynamic parameters. The option value > can > > > be > > > > > `insert-only`, `upset` or `all` and the default value is > > `insert-only`. > > > > > > > > > > If the result does not conform to the changelog mode, the jdbc > driver > > > > > throws an exception. What do you think? > > > > > > > > > > > > > > > For @Benchao > > > > > > Besides `java.sql.Driver`, have you considered also adding > support > > > for > > > > > `javax.sql.DataSource` interface? > > > > > > > > > > I missed the `javax.sql.DataSource` and I have added it to the > FLIP, > > > > thanks > > > > > Benchao > > > > > > > > > > > > > > > Best, > > > > > Shammon > > > > > > > > > > On Wed, Mar 1, 2023 at 7:57 PM Benchao Li > > > wrote: > > > > > > > > > > > +1 for the FLIP, thanks Shammon for driving this. > > > > > > > > > > > > JDBC is quite useful in OLAP scenarios, supporting JDBC would > > enable > > > > Flink > > > > > > to be used with existing tools, such as Tableau. > > > > > > > > > > > > Regarding the JDBC interfaces listed in the FLIP, I think they > > looks > > > > good > > > > > > already. Besides `java.sql.Driver`, have you considered also > adding > > > > support > > > > > > for `javax.sql.DataSource` interface? > > > > > > > > > > > > Jingsong Li 于2023年3月1日周三 17:53写道： > > > > > > > > > > > > > Thanks Shammon for driving. > > > > > > > > > > > > > > Big +1 for this. > > > > > > > > > > > > > > I heard that many users want to use FlinkGateway + JDBC to do > > some > > > > > > > queries, but at present, only Hive JDBC can be used. It is Hive > > > > > > > dialect by default, and the experience is also different from > > > > > > > FlinkSQL. We need to have our own JDBC. > > > > > > > > > > > > > > I took a look at your `Public Interface` part, only > > > > > > > `FlinkResultSet.getRowKind` is a true new interface, others are > > > just > > > > > > > implementations. > > > > > > > > > > > > > > If the user does not cast into a FlinkResultSet, will there be > > > > serious > > > > > > > consequences here (RowKind is ignored)? > > > > > > > > > > > > > > Best, > > > > > > > Jingsong > > > > > > > > > > > > > > On Wed, Mar 1, 2023 at 4:59 PM Shammon FY > > > wrote: > > > > > > > > > > > > > > > > Hi devs, > > > > > > > > > > >
Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway
Hi Jing Flink-Jdbc-Driver is mainly used to facilitate users to submit flink sql jobs. I think we will not use Flink-Jdbc-Driver in Flink-Jdbc-Connector to read/write data in general. If users can read/write specific tables by Flink-Jdbc-Driver, they can use these tables in their flink jobs directly. Thanks Best, Shammon On Mon, Mar 13, 2023 at 6:21 AM Jing Ge wrote: > Hi, Shammon, > > This FLIP is very interesting. Thanks for driving it! Just out of > curiosity, Could it be used in the Flink JDBC connector[1] after > doing the appropriate implementation there? > > Best regards, > Jing > > [1] > > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/jdbc/ > > On Fri, Mar 3, 2023 at 8:56 AM Shammon FY wrote: > > > Hi, > > > > Thanks jinsong. I think implementing JDBC for batch mode first sounds > good. > > This will simplify the implementation and we can also remove the row kind > > first. We can claim this in the FLIP and docs, I will update the FLIP. > > > > Best, > > Shammon > > > > > > On Fri, Mar 3, 2023 at 2:36 PM Jingsong Li > wrote: > > > > > Hi, Shammon, > > > > > > I took a look at JDBC `ResultSet` and `Statement`. They are > > > complicated and have many interfaces. Some of the interfaces may not > > > be very suitable for streaming. > > > > > > I think maybe we can just implement JDBC for batch/olap only. It is > > > hard to have an integration for JDBC and streaming... > > > > > > Do you need to use JDBC in streaming mode? Or do we just implement > > > JDBC for batch only first? > > > > > > Best, > > > Jingsong > > > > > > > > > On Thu, Mar 2, 2023 at 6:22 PM Shammon FY wrote: > > > > > > > > Hi > > > > > > > > Thanks for the feedback from Jingsong and Benchao. > > > > > > > > For @Jingsong > > > > > If the user does not cast into a FlinkResultSet, will there be > > > > serious consequences here (RowKind is ignored)? > > > > > > > > I agree with you that it's indeed a big deal if users ignore the row > > kind > > > > when they must know it. One idea that comes to my mind is we can add > an > > > > option such as `table.result.changelog-mode`, users can set it > through > > > > connection properties or set dynamic parameters. The option value can > > be > > > > `insert-only`, `upset` or `all` and the default value is > `insert-only`. > > > > > > > > If the result does not conform to the changelog mode, the jdbc driver > > > > throws an exception. What do you think? > > > > > > > > > > > > For @Benchao > > > > > Besides `java.sql.Driver`, have you considered also adding support > > for > > > > `javax.sql.DataSource` interface? > > > > > > > > I missed the `javax.sql.DataSource` and I have added it to the FLIP, > > > thanks > > > > Benchao > > > > > > > > > > > > Best, > > > > Shammon > > > > > > > > On Wed, Mar 1, 2023 at 7:57 PM Benchao Li > > wrote: > > > > > > > > > +1 for the FLIP, thanks Shammon for driving this. > > > > > > > > > > JDBC is quite useful in OLAP scenarios, supporting JDBC would > enable > > > Flink > > > > > to be used with existing tools, such as Tableau. > > > > > > > > > > Regarding the JDBC interfaces listed in the FLIP, I think they > looks > > > good > > > > > already. Besides `java.sql.Driver`, have you considered also adding > > > support > > > > > for `javax.sql.DataSource` interface? > > > > > > > > > > Jingsong Li 于2023年3月1日周三 17:53写道： > > > > > > > > > > > Thanks Shammon for driving. > > > > > > > > > > > > Big +1 for this. > > > > > > > > > > > > I heard that many users want to use FlinkGateway + JDBC to do > some > > > > > > queries, but at present, only Hive JDBC can be used. It is Hive > > > > > > dialect by default, and the experience is also different from > > > > > > FlinkSQL. We need to have our own JDBC. > > > > > > > > > > > > I took a look at your `Public Interface` part, only > > > > > > `FlinkResultSet.getRowKind` is a true new interface, others are > > just > > > > > > implementations. > > > > > > > > > > > > If the user does not cast into a FlinkResultSet, will there be > > > serious > > > > > > consequences here (RowKind is ignored)? > > > > > > > > > > > > Best, > > > > > > Jingsong > > > > > > > > > > > > On Wed, Mar 1, 2023 at 4:59 PM Shammon FY > > wrote: > > > > > > > > > > > > > > Hi devs, > > > > > > > > > > > > > > I'd like to start a discussion about FLIP-293: Introduce Flink > > Jdbc > > > > > > Driver > > > > > > > For Sql Gateway[1]. > > > > > > > > > > > > > > FLIP-275[2] supports remote sql client based on gateway, users > > can > > > > > > interact > > > > > > > with gateway by flink console. However, for users who create > > > session > > > > > > > clusters with Flink, they'd like to use Jdbc Driver to interact > > > with > > > > > the > > > > > > > gateway in their applications, such as olap queries.. > > > > > > > > > > > > > > I have discussed this proposal with @shengkaifang and > > @jinsonglee. > > > In > > > > > > this > > > > > > > FLIP, we'd like to introduce Jdbc Driver for gateway. Users
Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway
Hi, Shammon, This FLIP is very interesting. Thanks for driving it! Just out of curiosity, Could it be used in the Flink JDBC connector[1] after doing the appropriate implementation there? Best regards, Jing [1] https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/jdbc/ On Fri, Mar 3, 2023 at 8:56 AM Shammon FY wrote: > Hi, > > Thanks jinsong. I think implementing JDBC for batch mode first sounds good. > This will simplify the implementation and we can also remove the row kind > first. We can claim this in the FLIP and docs, I will update the FLIP. > > Best, > Shammon > > > On Fri, Mar 3, 2023 at 2:36 PM Jingsong Li wrote: > > > Hi, Shammon, > > > > I took a look at JDBC `ResultSet` and `Statement`. They are > > complicated and have many interfaces. Some of the interfaces may not > > be very suitable for streaming. > > > > I think maybe we can just implement JDBC for batch/olap only. It is > > hard to have an integration for JDBC and streaming... > > > > Do you need to use JDBC in streaming mode? Or do we just implement > > JDBC for batch only first? > > > > Best, > > Jingsong > > > > > > On Thu, Mar 2, 2023 at 6:22 PM Shammon FY wrote: > > > > > > Hi > > > > > > Thanks for the feedback from Jingsong and Benchao. > > > > > > For @Jingsong > > > > If the user does not cast into a FlinkResultSet, will there be > > > serious consequences here (RowKind is ignored)? > > > > > > I agree with you that it's indeed a big deal if users ignore the row > kind > > > when they must know it. One idea that comes to my mind is we can add an > > > option such as `table.result.changelog-mode`, users can set it through > > > connection properties or set dynamic parameters. The option value can > be > > > `insert-only`, `upset` or `all` and the default value is `insert-only`. > > > > > > If the result does not conform to the changelog mode, the jdbc driver > > > throws an exception. What do you think? > > > > > > > > > For @Benchao > > > > Besides `java.sql.Driver`, have you considered also adding support > for > > > `javax.sql.DataSource` interface? > > > > > > I missed the `javax.sql.DataSource` and I have added it to the FLIP, > > thanks > > > Benchao > > > > > > > > > Best, > > > Shammon > > > > > > On Wed, Mar 1, 2023 at 7:57 PM Benchao Li > wrote: > > > > > > > +1 for the FLIP, thanks Shammon for driving this. > > > > > > > > JDBC is quite useful in OLAP scenarios, supporting JDBC would enable > > Flink > > > > to be used with existing tools, such as Tableau. > > > > > > > > Regarding the JDBC interfaces listed in the FLIP, I think they looks > > good > > > > already. Besides `java.sql.Driver`, have you considered also adding > > support > > > > for `javax.sql.DataSource` interface? > > > > > > > > Jingsong Li 于2023年3月1日周三 17:53写道： > > > > > > > > > Thanks Shammon for driving. > > > > > > > > > > Big +1 for this. > > > > > > > > > > I heard that many users want to use FlinkGateway + JDBC to do some > > > > > queries, but at present, only Hive JDBC can be used. It is Hive > > > > > dialect by default, and the experience is also different from > > > > > FlinkSQL. We need to have our own JDBC. > > > > > > > > > > I took a look at your `Public Interface` part, only > > > > > `FlinkResultSet.getRowKind` is a true new interface, others are > just > > > > > implementations. > > > > > > > > > > If the user does not cast into a FlinkResultSet, will there be > > serious > > > > > consequences here (RowKind is ignored)? > > > > > > > > > > Best, > > > > > Jingsong > > > > > > > > > > On Wed, Mar 1, 2023 at 4:59 PM Shammon FY > wrote: > > > > > > > > > > > > Hi devs, > > > > > > > > > > > > I'd like to start a discussion about FLIP-293: Introduce Flink > Jdbc > > > > > Driver > > > > > > For Sql Gateway[1]. > > > > > > > > > > > > FLIP-275[2] supports remote sql client based on gateway, users > can > > > > > interact > > > > > > with gateway by flink console. However, for users who create > > session > > > > > > clusters with Flink, they'd like to use Jdbc Driver to interact > > with > > > > the > > > > > > gateway in their applications, such as olap queries.. > > > > > > > > > > > > I have discussed this proposal with @shengkaifang and > @jinsonglee. > > In > > > > > this > > > > > > FLIP, we'd like to introduce Jdbc Driver for gateway. Users can > use > > > > Jdbc > > > > > > Driver to submit their queries and get results like a database in > > their > > > > > > applications. > > > > > > > > > > > > Looking forward to your feedback, thanks. > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-293%3A+Introduce+Flink+Jdbc+Driver+For+Sql+Gateway > > > > > > [2] > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-275%3A+Support+Remote+SQL+Client+Based+on+SQL+Gateway > > > > > > > > > > > > > > > > > > Best, > > > > > > Shammon > > > > > > > > > > > > > > > > > -- > > > > > > > > Best, >
Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway
Hi, Thanks jinsong. I think implementing JDBC for batch mode first sounds good. This will simplify the implementation and we can also remove the row kind first. We can claim this in the FLIP and docs, I will update the FLIP. Best, Shammon On Fri, Mar 3, 2023 at 2:36 PM Jingsong Li wrote: > Hi, Shammon, > > I took a look at JDBC `ResultSet` and `Statement`. They are > complicated and have many interfaces. Some of the interfaces may not > be very suitable for streaming. > > I think maybe we can just implement JDBC for batch/olap only. It is > hard to have an integration for JDBC and streaming... > > Do you need to use JDBC in streaming mode? Or do we just implement > JDBC for batch only first? > > Best, > Jingsong > > > On Thu, Mar 2, 2023 at 6:22 PM Shammon FY wrote: > > > > Hi > > > > Thanks for the feedback from Jingsong and Benchao. > > > > For @Jingsong > > > If the user does not cast into a FlinkResultSet, will there be > > serious consequences here (RowKind is ignored)? > > > > I agree with you that it's indeed a big deal if users ignore the row kind > > when they must know it. One idea that comes to my mind is we can add an > > option such as `table.result.changelog-mode`, users can set it through > > connection properties or set dynamic parameters. The option value can be > > `insert-only`, `upset` or `all` and the default value is `insert-only`. > > > > If the result does not conform to the changelog mode, the jdbc driver > > throws an exception. What do you think? > > > > > > For @Benchao > > > Besides `java.sql.Driver`, have you considered also adding support for > > `javax.sql.DataSource` interface? > > > > I missed the `javax.sql.DataSource` and I have added it to the FLIP, > thanks > > Benchao > > > > > > Best, > > Shammon > > > > On Wed, Mar 1, 2023 at 7:57 PM Benchao Li wrote: > > > > > +1 for the FLIP, thanks Shammon for driving this. > > > > > > JDBC is quite useful in OLAP scenarios, supporting JDBC would enable > Flink > > > to be used with existing tools, such as Tableau. > > > > > > Regarding the JDBC interfaces listed in the FLIP, I think they looks > good > > > already. Besides `java.sql.Driver`, have you considered also adding > support > > > for `javax.sql.DataSource` interface? > > > > > > Jingsong Li 于2023年3月1日周三 17:53写道： > > > > > > > Thanks Shammon for driving. > > > > > > > > Big +1 for this. > > > > > > > > I heard that many users want to use FlinkGateway + JDBC to do some > > > > queries, but at present, only Hive JDBC can be used. It is Hive > > > > dialect by default, and the experience is also different from > > > > FlinkSQL. We need to have our own JDBC. > > > > > > > > I took a look at your `Public Interface` part, only > > > > `FlinkResultSet.getRowKind` is a true new interface, others are just > > > > implementations. > > > > > > > > If the user does not cast into a FlinkResultSet, will there be > serious > > > > consequences here (RowKind is ignored)? > > > > > > > > Best, > > > > Jingsong > > > > > > > > On Wed, Mar 1, 2023 at 4:59 PM Shammon FY wrote: > > > > > > > > > > Hi devs, > > > > > > > > > > I'd like to start a discussion about FLIP-293: Introduce Flink Jdbc > > > > Driver > > > > > For Sql Gateway[1]. > > > > > > > > > > FLIP-275[2] supports remote sql client based on gateway, users can > > > > interact > > > > > with gateway by flink console. However, for users who create > session > > > > > clusters with Flink, they'd like to use Jdbc Driver to interact > with > > > the > > > > > gateway in their applications, such as olap queries.. > > > > > > > > > > I have discussed this proposal with @shengkaifang and @jinsonglee. > In > > > > this > > > > > FLIP, we'd like to introduce Jdbc Driver for gateway. Users can use > > > Jdbc > > > > > Driver to submit their queries and get results like a database in > their > > > > > applications. > > > > > > > > > > Looking forward to your feedback, thanks. > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-293%3A+Introduce+Flink+Jdbc+Driver+For+Sql+Gateway > > > > > [2] > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-275%3A+Support+Remote+SQL+Client+Based+on+SQL+Gateway > > > > > > > > > > > > > > > Best, > > > > > Shammon > > > > > > > > > > > > > -- > > > > > > Best, > > > Benchao Li > > > >
Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway
Hi, Shammon, I took a look at JDBC `ResultSet` and `Statement`. They are complicated and have many interfaces. Some of the interfaces may not be very suitable for streaming. I think maybe we can just implement JDBC for batch/olap only. It is hard to have an integration for JDBC and streaming... Do you need to use JDBC in streaming mode? Or do we just implement JDBC for batch only first? Best, Jingsong On Thu, Mar 2, 2023 at 6:22 PM Shammon FY wrote: > > Hi > > Thanks for the feedback from Jingsong and Benchao. > > For @Jingsong > > If the user does not cast into a FlinkResultSet, will there be > serious consequences here (RowKind is ignored)? > > I agree with you that it's indeed a big deal if users ignore the row kind > when they must know it. One idea that comes to my mind is we can add an > option such as `table.result.changelog-mode`, users can set it through > connection properties or set dynamic parameters. The option value can be > `insert-only`, `upset` or `all` and the default value is `insert-only`. > > If the result does not conform to the changelog mode, the jdbc driver > throws an exception. What do you think? > > > For @Benchao > > Besides `java.sql.Driver`, have you considered also adding support for > `javax.sql.DataSource` interface? > > I missed the `javax.sql.DataSource` and I have added it to the FLIP, thanks > Benchao > > > Best, > Shammon > > On Wed, Mar 1, 2023 at 7:57 PM Benchao Li wrote: > > > +1 for the FLIP, thanks Shammon for driving this. > > > > JDBC is quite useful in OLAP scenarios, supporting JDBC would enable Flink > > to be used with existing tools, such as Tableau. > > > > Regarding the JDBC interfaces listed in the FLIP, I think they looks good > > already. Besides `java.sql.Driver`, have you considered also adding support > > for `javax.sql.DataSource` interface? > > > > Jingsong Li 于2023年3月1日周三 17:53写道： > > > > > Thanks Shammon for driving. > > > > > > Big +1 for this. > > > > > > I heard that many users want to use FlinkGateway + JDBC to do some > > > queries, but at present, only Hive JDBC can be used. It is Hive > > > dialect by default, and the experience is also different from > > > FlinkSQL. We need to have our own JDBC. > > > > > > I took a look at your `Public Interface` part, only > > > `FlinkResultSet.getRowKind` is a true new interface, others are just > > > implementations. > > > > > > If the user does not cast into a FlinkResultSet, will there be serious > > > consequences here (RowKind is ignored)? > > > > > > Best, > > > Jingsong > > > > > > On Wed, Mar 1, 2023 at 4:59 PM Shammon FY wrote: > > > > > > > > Hi devs, > > > > > > > > I'd like to start a discussion about FLIP-293: Introduce Flink Jdbc > > > Driver > > > > For Sql Gateway[1]. > > > > > > > > FLIP-275[2] supports remote sql client based on gateway, users can > > > interact > > > > with gateway by flink console. However, for users who create session > > > > clusters with Flink, they'd like to use Jdbc Driver to interact with > > the > > > > gateway in their applications, such as olap queries.. > > > > > > > > I have discussed this proposal with @shengkaifang and @jinsonglee. In > > > this > > > > FLIP, we'd like to introduce Jdbc Driver for gateway. Users can use > > Jdbc > > > > Driver to submit their queries and get results like a database in their > > > > applications. > > > > > > > > Looking forward to your feedback, thanks. > > > > > > > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-293%3A+Introduce+Flink+Jdbc+Driver+For+Sql+Gateway > > > > [2] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-275%3A+Support+Remote+SQL+Client+Based+on+SQL+Gateway > > > > > > > > > > > > Best, > > > > Shammon > > > > > > > > > -- > > > > Best, > > Benchao Li > >
Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway
Hi Thanks for the feedback from Jingsong and Benchao. For @Jingsong > If the user does not cast into a FlinkResultSet, will there be serious consequences here (RowKind is ignored)? I agree with you that it's indeed a big deal if users ignore the row kind when they must know it. One idea that comes to my mind is we can add an option such as `table.result.changelog-mode`, users can set it through connection properties or set dynamic parameters. The option value can be `insert-only`, `upset` or `all` and the default value is `insert-only`. If the result does not conform to the changelog mode, the jdbc driver throws an exception. What do you think? For @Benchao > Besides `java.sql.Driver`, have you considered also adding support for `javax.sql.DataSource` interface? I missed the `javax.sql.DataSource` and I have added it to the FLIP, thanks Benchao Best, Shammon On Wed, Mar 1, 2023 at 7:57 PM Benchao Li wrote: > +1 for the FLIP, thanks Shammon for driving this. > > JDBC is quite useful in OLAP scenarios, supporting JDBC would enable Flink > to be used with existing tools, such as Tableau. > > Regarding the JDBC interfaces listed in the FLIP, I think they looks good > already. Besides `java.sql.Driver`, have you considered also adding support > for `javax.sql.DataSource` interface? > > Jingsong Li 于2023年3月1日周三 17:53写道： > > > Thanks Shammon for driving. > > > > Big +1 for this. > > > > I heard that many users want to use FlinkGateway + JDBC to do some > > queries, but at present, only Hive JDBC can be used. It is Hive > > dialect by default, and the experience is also different from > > FlinkSQL. We need to have our own JDBC. > > > > I took a look at your `Public Interface` part, only > > `FlinkResultSet.getRowKind` is a true new interface, others are just > > implementations. > > > > If the user does not cast into a FlinkResultSet, will there be serious > > consequences here (RowKind is ignored)? > > > > Best, > > Jingsong > > > > On Wed, Mar 1, 2023 at 4:59 PM Shammon FY wrote: > > > > > > Hi devs, > > > > > > I'd like to start a discussion about FLIP-293: Introduce Flink Jdbc > > Driver > > > For Sql Gateway[1]. > > > > > > FLIP-275[2] supports remote sql client based on gateway, users can > > interact > > > with gateway by flink console. However, for users who create session > > > clusters with Flink, they'd like to use Jdbc Driver to interact with > the > > > gateway in their applications, such as olap queries.. > > > > > > I have discussed this proposal with @shengkaifang and @jinsonglee. In > > this > > > FLIP, we'd like to introduce Jdbc Driver for gateway. Users can use > Jdbc > > > Driver to submit their queries and get results like a database in their > > > applications. > > > > > > Looking forward to your feedback, thanks. > > > > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-293%3A+Introduce+Flink+Jdbc+Driver+For+Sql+Gateway > > > [2] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-275%3A+Support+Remote+SQL+Client+Based+on+SQL+Gateway > > > > > > > > > Best, > > > Shammon > > > > > -- > > Best, > Benchao Li >
Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway
+1 for the FLIP, thanks Shammon for driving this. JDBC is quite useful in OLAP scenarios, supporting JDBC would enable Flink to be used with existing tools, such as Tableau. Regarding the JDBC interfaces listed in the FLIP, I think they looks good already. Besides `java.sql.Driver`, have you considered also adding support for `javax.sql.DataSource` interface? Jingsong Li 于2023年3月1日周三 17:53写道： > Thanks Shammon for driving. > > Big +1 for this. > > I heard that many users want to use FlinkGateway + JDBC to do some > queries, but at present, only Hive JDBC can be used. It is Hive > dialect by default, and the experience is also different from > FlinkSQL. We need to have our own JDBC. > > I took a look at your `Public Interface` part, only > `FlinkResultSet.getRowKind` is a true new interface, others are just > implementations. > > If the user does not cast into a FlinkResultSet, will there be serious > consequences here (RowKind is ignored)? > > Best, > Jingsong > > On Wed, Mar 1, 2023 at 4:59 PM Shammon FY wrote: > > > > Hi devs, > > > > I'd like to start a discussion about FLIP-293: Introduce Flink Jdbc > Driver > > For Sql Gateway[1]. > > > > FLIP-275[2] supports remote sql client based on gateway, users can > interact > > with gateway by flink console. However, for users who create session > > clusters with Flink, they'd like to use Jdbc Driver to interact with the > > gateway in their applications, such as olap queries.. > > > > I have discussed this proposal with @shengkaifang and @jinsonglee. In > this > > FLIP, we'd like to introduce Jdbc Driver for gateway. Users can use Jdbc > > Driver to submit their queries and get results like a database in their > > applications. > > > > Looking forward to your feedback, thanks. > > > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-293%3A+Introduce+Flink+Jdbc+Driver+For+Sql+Gateway > > [2] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-275%3A+Support+Remote+SQL+Client+Based+on+SQL+Gateway > > > > > > Best, > > Shammon > -- Best, Benchao Li
Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway
Thanks Shammon for driving. Big +1 for this. I heard that many users want to use FlinkGateway + JDBC to do some queries, but at present, only Hive JDBC can be used. It is Hive dialect by default, and the experience is also different from FlinkSQL. We need to have our own JDBC. I took a look at your `Public Interface` part, only `FlinkResultSet.getRowKind` is a true new interface, others are just implementations. If the user does not cast into a FlinkResultSet, will there be serious consequences here (RowKind is ignored)? Best, Jingsong On Wed, Mar 1, 2023 at 4:59 PM Shammon FY wrote: > > Hi devs, > > I'd like to start a discussion about FLIP-293: Introduce Flink Jdbc Driver > For Sql Gateway[1]. > > FLIP-275[2] supports remote sql client based on gateway, users can interact > with gateway by flink console. However, for users who create session > clusters with Flink, they'd like to use Jdbc Driver to interact with the > gateway in their applications, such as olap queries.. > > I have discussed this proposal with @shengkaifang and @jinsonglee. In this > FLIP, we'd like to introduce Jdbc Driver for gateway. Users can use Jdbc > Driver to submit their queries and get results like a database in their > applications. > > Looking forward to your feedback, thanks. > > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-293%3A+Introduce+Flink+Jdbc+Driver+For+Sql+Gateway > [2] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-275%3A+Support+Remote+SQL+Client+Based+on+SQL+Gateway > > > Best, > Shammon
[DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway
Hi devs, I'd like to start a discussion about FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway[1]. FLIP-275[2] supports remote sql client based on gateway, users can interact with gateway by flink console. However, for users who create session clusters with Flink, they'd like to use Jdbc Driver to interact with the gateway in their applications, such as olap queries.. I have discussed this proposal with @shengkaifang and @jinsonglee. In this FLIP, we'd like to introduce Jdbc Driver for gateway. Users can use Jdbc Driver to submit their queries and get results like a database in their applications. Looking forward to your feedback, thanks. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-293%3A+Introduce+Flink+Jdbc+Driver+For+Sql+Gateway [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-275%3A+Support+Remote+SQL+Client+Based+on+SQL+Gateway Best, Shammon
