Hello devs, Is any active work happening on this FLIP? As far as I see there are blockers that needs to happen first to implement regarding artifact distribution.
Is this work in halt completetly or some efforts are going into resolve the blockers first or something? Our platform would benefit this feature a lot, we have a kind of working custom implementation at the moment, but it is uniquely adapted to our app and platform. I could help out to move this forward. Best, Ferenc On Friday, June 30th, 2023 at 04:53, Paul Lam <paullin3...@gmail.com> wrote: > > > Hi Jing, > > Thanks for your input! > > > Would you like to add > > one section to describe(better with script/code example) how to use it in > > these two scenarios from users' perspective? > > > OK. I’ll update the FLIP with the code snippet after I get the POC branch > done. > > > NIT: the pictures have transparent background when readers click on it. It > > would be great if you can replace them with pictures with white background. > > > Fixed. Thanks for pointing that out :) > > Best, > Paul Lam > > > 2023年6月27日 06:51，Jing Ge j...@ververica.com.INVALID 写道： > > > > Hi Paul, > > > > Thanks for driving it and thank you all for the informative discussion! The > > FLIP is in good shape now. As described in the FLIP, SQL Driver will be > > mainly used to run Flink SQLs in two scenarios: 1. SQL client/gateway in > > application mode and 2. external system integration. Would you like to add > > one section to describe(better with script/code example) how to use it in > > these two scenarios from users' perspective? > > > > NIT: the pictures have transparent background when readers click on it. It > > would be great if you can replace them with pictures with white background. > > > > Best regards, > > Jing > > > > On Mon, Jun 26, 2023 at 1:31 PM Paul Lam <paullin3...@gmail.com > > mailto:paullin3...@gmail.com> wrote: > > > > > Hi Shengkai, > > > > > > > * How can we ship the json plan to the JobManager? > > > > > > The Flink K8s module should be responsible for file distribution. We could > > > introduce > > > an option like `kubernetes.storage.dir`. For each flink cluster, there > > > would be a > > > dedicated subdirectory, with the pattern like > > > `${kubernetes.storage.dir}/${cluster-id}`. > > > > > > All resources-related options (e.g. pipeline jars, json plans) that are > > > configured with > > > scheme `file://` <file://`/> <file:///%60 <file:///%60>> would be > > > uploaded to the resource directory > > > and downloaded to the > > > jobmanager, before SQL Driver accesses the files with the original > > > filenames. > > > > > > > * Classloading strategy > > > > > > We could directly specify the SQL Gateway jar as the jar file in > > > PackagedProgram. > > > It would be treated like a normal user jar and the SQL Driver is loaded > > > into the user > > > classloader. WDYT? > > > > > > > * Option `$internal.sql-gateway.driver.sql-config` is string type > > > > I think it's better to use Map type here > > > > > > By Map type configuration, do you mean a nested map that contains all > > > configurations? > > > > > > I hope I've explained myself well, it’s a file that contains the extra SQL > > > configurations, which would be shipped to the jobmanager. > > > > > > > * PoC branch > > > > > > Sure. I’ll let you know once I get the job done. > > > > > > Best, > > > Paul Lam > > > > > > > 2023年6月26日 14:27，Shengkai Fang <fskm...@gmail.com > > > > mailto:fskm...@gmail.com> 写道： > > > > > > > > Hi, Paul. > > > > > > > > Thanks for your update. I have a few questions about the new design: > > > > > > > > * How can we ship the json plan to the JobManager? > > > > > > > > The current design only exposes an option about the URL of the json > > > > plan. It seems the gateway is responsible to upload to an external > > > > stroage. > > > > Can we reuse the PipelineOptions.JARS to ship to the remote filesystem? > > > > > > > > * Classloading strategy > > > > > > > > Currently, the Driver is in the sql-gateway package. It means the Driver > > > > is not in the JM's classpath directly. Because the sql-gateway jar is > > > > now > > > > in the opt directory rather than lib directory. It may need to add the > > > > external dependencies as Python does[1]. BTW, I think it's better to > > > > move > > > > the Driver into the flink-table-runtime package, which is much easier to > > > > find(Sorry for the wrong opinion before). > > > > > > > > * Option `$internal.sql-gateway.driver.sql-config` is string type > > > > > > > > I think it's better to use Map type here > > > > > > > > * PoC branch > > > > > > > > Because this FLIP involves many modules, do you have a PoC branch to > > > > verify it does work? > > > > > > > > Best, > > > > Shengkai > > > > > > > > [1] > > > > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940 > > > > > > > > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940 > > > > < > > > > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940 > > > > > > > > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940 > > > > > > > > Paul Lam <paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com>> > > > > 于2023年6月19日周一 14:09写道： > > > > Hi Shengkai, > > > > > > > > Sorry for my late reply. It took me some time to update the FLIP. > > > > > > > > In the latest FLIP design, SQL Driver is placed in flink-sql-gateway > > > > module. PTAL. > > > > > > > > The FLIP does not cover details about the K8s file distribution, but its > > > > general usage would > > > > be very much the same as YARN setups. We could make follow-up > > > > discussions in the jira > > > > tickets. > > > > > > > > Best, > > > > Paul Lam > > > > > > > > > 2023年6月12日 15:29，Shengkai Fang <fskm...@gmail.com > > > > > mailto:fskm...@gmail.com <mailto: > > > > > fskm...@gmail.com mailto:fskm...@gmail.com>> 写道： > > > > > > > > > > > If it’s the case, I’m good with introducing a new module and making > > > > > > SQL Driver > > > > > > an internal class and accepts JSON plans only. > > > > > > > > > > I rethink this again and again. I think it's better to move the > > > > > SqlDriver into the sql-gateway module because the sql client relies > > > > > on the > > > > > sql-gateway to submit the sql and the sql-gateway has the ability to > > > > > generate the ExecNodeGraph now. +1 to support accepting JSON plans > > > > > only. > > > > > > > > > > * Upload configuration through command line parameter > > > > > > > > > > ExecNodeGraph only contains the job's information but it doesn't > > > > > contain the checkpoint dir, checkpoint interval, execution mode and > > > > > so on. > > > > > So I think we should also upload the configuration. > > > > > > > > > > * KubernetesClusterDescripter and > > > > > KubernetesApplicationClusterEntrypoint are responsible for the jar > > > > > upload/download > > > > > > > > > > +1 for the change. > > > > > > > > > > Could you update the FLIP about the current discussion? > > > > > > > > > > Best, > > > > > Shengkai > > > > > > > > > > Yang Wang <wangyang0...@apache.org mailto:wangyang0...@apache.org > > > > > <mailto:wangyang0...@apache.org mailto:wangyang0...@apache.org>> > > > > > 于2023年6月12日周一 11:41写道： > > > > > Sorry for the late reply. I am in favor of introducing such a built-in > > > > > resource localization mechanism > > > > > based on Flink FileSystem. Then FLINK-28915[1] could be the second > > > > > step > > > > > which will download > > > > > the jars and dependencies to the JobManager/TaskManager local > > > > > directory > > > > > before working. > > > > > > > > > > The first step could be done in another ticket in Flink. Or some > > > > > external > > > > > Flink jobs management system > > > > > could also take care of this. > > > > > > > > > > [1]. https://issues.apache.org/jira/browse/FLINK-28915 > > > > > https://issues.apache.org/jira/browse/FLINK-28915 < > > > > > https://issues.apache.org/jira/browse/FLINK-28915 > > > > > https://issues.apache.org/jira/browse/FLINK-28915> > > > > > > > > > > Best, > > > > > Yang > > > > > > > > > > Paul Lam <paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com>> > > > > > 于2023年6月9日周五 17:39写道： > > > > > > > > > > > Hi Mason, > > > > > > > > > > > > I get your point. I'm increasingly feeling the need to introduce a > > > > > > built-in > > > > > > file distribution mechanism for flink-kubernetes module, just like > > > > > > Spark > > > > > > does with `spark.kubernetes.file.upload.path` [1]. > > > > > > > > > > > > I’m assuming the workflow is as follows: > > > > > > > > > > > > - KubernetesClusterDescripter uploads all local resources to a > > > > > > remote > > > > > > storage via Flink filesystem (skips if the resources are already > > > > > > remote). > > > > > > - KubernetesApplicationClusterEntrypoint downloads the resources > > > > > > and put them in the classpath during startup. > > > > > > > > > > > > I wouldn't mind splitting it into another FLIP to ensure that > > > > > > everything is > > > > > > done correctly. > > > > > > > > > > > > cc'ed @Yang to gather more opinions. > > > > > > > > > > > > [1] > > > > > > https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management > > > > > > https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management > > > < > > > https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management > > > > > > https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management > > > > > > > > > Best, > > > > > > Paul Lam > > > > > > > > > > > > 2023年6月8日 12:15，Mason Chen <mas.chen6...@gmail.com > > > > > > mailto:mas.chen6...@gmail.com <mailto: > > > > > > mas.chen6...@gmail.com mailto:mas.chen6...@gmail.com>> 写道： > > > > > > > > > > > > Hi Paul, > > > > > > > > > > > > Thanks for your response! > > > > > > > > > > > > I agree that utilizing SQL Drivers in Java applications is equally > > > > > > important > > > > > > > > > > > > as employing them in SQL Gateway. WRT init containers, I think most > > > > > > users use them just as a workaround. For example, wget a jar from > > > > > > the > > > > > > maven repo. > > > > > > > > > > > > We could implement the functionality in SQL Driver in a more > > > > > > graceful > > > > > > way and the flink-supported filesystem approach seems to be a > > > > > > good choice. > > > > > > > > > > > > My main point is: can we solve the problem with a design agnostic of > > > > > > SQL > > > > > > and Stream API? I mentioned a use case where this ability is useful > > > > > > for > > > > > > Java or Stream API applications. Maybe this is even a non-goal to > > > > > > your FLIP > > > > > > since you are focusing on the driver entrypoint. > > > > > > > > > > > > Jark mentioned some optimizations: > > > > > > > > > > > > This allows SQLGateway to leverage some metadata caching and UDF JAR > > > > > > caching for better compiling performance. > > > > > > > > > > > > It would be great to see this even outside the SQLGateway (i.e. UDF > > > > > > JAR > > > > > > caching). > > > > > > > > > > > > Best, > > > > > > Mason > > > > > > > > > > > > On Wed, Jun 7, 2023 at 2:26 AM Shengkai Fang <fskm...@gmail.com > > > > > > mailto:fskm...@gmail.com > > > > > > <mailto:fskm...@gmail.com mailto:fskm...@gmail.com>> wrote: > > > > > > > > > > > > Hi. Paul. Thanks for your update and the update makes me understand > > > > > > the > > > > > > design much better. > > > > > > > > > > > > But I still have some questions about the FLIP. > > > > > > > > > > > > For SQL Gateway, only DMLs need to be delegated to the SQL server > > > > > > Driver. I would think about the details and update the FLIP. Do you > > > > > > have > > > > > > > > > > > > some > > > > > > > > > > > > ideas already? > > > > > > > > > > > > If the applicaiton mode can not support library mode, I think we > > > > > > should > > > > > > only execute INSERT INTO and UPDATE/ DELETE statement in the > > > > > > application > > > > > > mode. AFAIK, we can not support ANALYZE TABLE and CALL PROCEDURE > > > > > > statements. The ANALYZE TABLE syntax need to register the statistic > > > > > > to the > > > > > > catalog after job finishes and the CALL PROCEDURE statement doesn't > > > > > > generate the ExecNodeGraph. > > > > > > > > > > > > * Introduce storage via option `sql-gateway.application.storage-dir` > > > > > > > > > > > > If we can not support to submit the jars through web submission, +1 > > > > > > to > > > > > > introduce the options to upload the files. While I think the > > > > > > uploader > > > > > > should be responsible to remove the uploaded jars. Can we remove the > > > > > > jars > > > > > > if the job is running or gateway exits? > > > > > > > > > > > > * JobID is not avaliable > > > > > > > > > > > > Can we use the returned rest client by ApplicationDeployer to query > > > > > > the job > > > > > > id? I am concerned that users don't know which job is related to the > > > > > > submitted SQL. > > > > > > > > > > > > * Do we need to introduce a new module named flink-table-sql-runner? > > > > > > > > > > > > It seems we need to introduce a new module. Will the new module is > > > > > > available in the distribution package? I agree with Jark that we > > > > > > don't need > > > > > > to introduce this for table-API users and these users have their > > > > > > main > > > > > > class. If we want to make users write the k8s operator more easily, > > > > > > I > > > > > > think > > > > > > we should modify the k8s operator repo. If we don't need to support > > > > > > SQL > > > > > > files, can we make this jar only visible in the sql-gateway like we > > > > > > do in > > > > > > the planner loader?[1] > > > > > > > > > > > > [1] > > > > > > https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95 > > > > > > https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95 > > > < > > > https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95 > > > > > > https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95 > > > > > > > > > Best, > > > > > > Shengkai > > > > > > > > > > > > Weihua Hu <huweihua....@gmail.com mailto:huweihua....@gmail.com > > > > > > <mailto:huweihua....@gmail.com mailto:huweihua....@gmail.com>> > > > > > > 于2023年6月7日周三 10:52写道： > > > > > > > > > > > > Hi, > > > > > > > > > > > > Thanks for updating the FLIP. > > > > > > > > > > > > I have two cents on the distribution of SQLs and resources. > > > > > > 1. Should we support a common file distribution mechanism for k8s > > > > > > application mode? > > > > > > I have seen some issues and requirements on the mailing list. > > > > > > In our production environment, we implement the download command in > > > > > > the > > > > > > CliFrontend. > > > > > > And automatically add an init container to the POD for file > > > > > > > > > > > > downloading. > > > > > > > > > > > > The advantage of this > > > > > > is that we can use all Flink-supported file systems to store files. > > > > > > > > > > > > This need more discussion. I would appreciate hearing more opinions. > > > > > > > > > > > > 2. In this FLIP, we distribute files in two different ways in YARN > > > > > > and > > > > > > Kubernetes. Can we combine it in one way? > > > > > > If we don't want to implement a common file distribution for k8s > > > > > > application mode. Could we use the SQLDriver > > > > > > to download the files both in YARN and K8S? IMO, this can reduce the > > > > > > > > > > > > cost > > > > > > > > > > > > of code maintenance. > > > > > > > > > > > > Best, > > > > > > Weihua > > > > > > > > > > > > On Wed, Jun 7, 2023 at 10:18 AM Paul Lam <paullin3...@gmail.com > > > > > > mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com>> wrote: > > > > > > > > > > > > Hi Mason, > > > > > > > > > > > > Thanks for your input! > > > > > > > > > > > > +1 for init containers or a more generalized way of obtaining > > > > > > > > > > > > arbitrary > > > > > > > > > > > > files. File fetching isn't specific to just SQL--it also matters for > > > > > > > > > > > > Java > > > > > > > > > > > > applications if the user doesn't want to rebuild a Flink image and > > > > > > > > > > > > just > > > > > > > > > > > > wants to modify the user application fat jar. > > > > > > > > > > > > I agree that utilizing SQL Drivers in Java applications is equally > > > > > > important > > > > > > as employing them in SQL Gateway. WRT init containers, I think most > > > > > > users use them just as a workaround. For example, wget a jar from > > > > > > the > > > > > > maven repo. > > > > > > > > > > > > We could implement the functionality in SQL Driver in a more > > > > > > graceful > > > > > > way and the flink-supported filesystem approach seems to be a > > > > > > good choice. > > > > > > > > > > > > Also, what do you think about prefixing the config options with > > > > > > `sql-driver` instead of just `sql` to be more specific? > > > > > > > > > > > > LGTM, since SQL Driver is a public interface and the options are > > > > > > specific to it. > > > > > > > > > > > > Best, > > > > > > Paul Lam > > > > > > > > > > > > 2023年6月6日 06:30，Mason Chen <mas.chen6...@gmail.com > > > > > > mailto:mas.chen6...@gmail.com <mailto: > > > > > > mas.chen6...@gmail.com mailto:mas.chen6...@gmail.com>> 写道： > > > > > > > > > > > > Hi Paul, > > > > > > > > > > > > +1 for this feature and supporting SQL file + JSON plans. We get a > > > > > > > > > > > > lot > > > > > > > > > > > > of > > > > > > > > > > > > requests to just be able to submit a SQL file, but the JSON plan > > > > > > optimizations make sense. > > > > > > > > > > > > +1 for init containers or a more generalized way of obtaining > > > > > > > > > > > > arbitrary > > > > > > > > > > > > files. File fetching isn't specific to just SQL--it also matters for > > > > > > > > > > > > Java > > > > > > > > > > > > applications if the user doesn't want to rebuild a Flink image and > > > > > > > > > > > > just > > > > > > > > > > > > wants to modify the user application fat jar. > > > > > > > > > > > > Please note that we could reuse the checkpoint storage like S3/HDFS, > > > > > > > > > > > > which > > > > > > > > > > > > should > > > > > > > > > > > > be required to run Flink in production, so I guess that would be > > > > > > > > > > > > acceptable > > > > > > > > > > > > for most > > > > > > > > > > > > users. WDYT? > > > > > > > > > > > > If you do go this route, it would be nice to support writing these > > > > > > > > > > > > files > > > > > > > > > > > > to > > > > > > > > > > > > S3/HDFS via Flink. This makes access control and policy management > > > > > > > > > > > > simpler. > > > > > > > > > > > > Also, what do you think about prefixing the config options with > > > > > > `sql-driver` instead of just `sql` to be more specific? > > > > > > > > > > > > Best, > > > > > > Mason > > > > > > > > > > > > On Mon, Jun 5, 2023 at 2:28 AM Paul Lam <paullin3...@gmail.com > > > > > > mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com> > > > > > > > > > > > > <mailto: > > > > > > > > > > > > paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com>>> wrote: > > > > > > > > > > > > Hi Jark, > > > > > > > > > > > > Thanks for your input! Please see my comments inline. > > > > > > > > > > > > Isn't Table API the same way as DataSream jobs to submit Flink SQL? > > > > > > DataStream API also doesn't provide a default main class for users, > > > > > > why do we need to provide such one for SQL? > > > > > > > > > > > > Sorry for the confusion I caused. By DataStream jobs, I mean jobs > > > > > > > > > > > > submitted > > > > > > > > > > > > via Flink CLI which actually could be DataStream/Table jobs. > > > > > > > > > > > > I think a default main class would be user-friendly which eliminates > > > > > > > > > > > > the > > > > > > > > > > > > need > > > > > > for users to write a main class as SQLRunner in Flink K8s operator > > > > > > > > > > > > [1]. > > > > > > > > > > > > I thought the proposed SqlDriver was a dedicated main class > > > > > > > > > > > > accepting > > > > > > > > > > > > SQL files, is > > > > > > > > > > > > that correct? > > > > > > > > > > > > Both JSON plans and SQL files are accepted. SQL Gateway should use > > > > > > > > > > > > JSON > > > > > > > > > > > > plans, > > > > > > while CLI users may use either JSON plans or SQL files. > > > > > > > > > > > > Please see the updated FLIP[2] for more details. > > > > > > > > > > > > Personally, I prefer the way of init containers which doesn't > > > > > > > > > > > > depend > > > > > > > > > > > > on > > > > > > > > > > > > additional components. > > > > > > This can reduce the moving parts of a production environment. > > > > > > Depending on a distributed file system makes the testing, demo, and > > > > > > > > > > > > local > > > > > > > > > > > > setup harder than init containers. > > > > > > > > > > > > Please note that we could reuse the checkpoint storage like S3/HDFS, > > > > > > > > > > > > which > > > > > > > > > > > > should > > > > > > be required to run Flink in production, so I guess that would be > > > > > > acceptable for most > > > > > > users. WDYT? > > > > > > > > > > > > WRT testing, demo, and local setups, I think we could support the > > > > > > > > > > > > local > > > > > > > > > > > > filesystem > > > > > > scheme i.e. file://** <file:///**> <> as the state backends do. It > > > > > > works as long as > > > > > > > > > > > > SQL > > > > > > > > > > > > Gateway > > > > > > and JobManager(or SQL Driver) can access the resource directory > > > > > > > > > > > > (specified > > > > > > > > > > > > via > > > > > > `sql-gateway.application.storage-dir`). > > > > > > > > > > > > Thanks! > > > > > > > > > > > > [1] > > > > > > https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java > > > > > > https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java > > > < > > > https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java > > > > > > https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java > > > > > > > > > [2] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver > > > < > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver > > > > > > > > > [3] > > > > > > https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161 > > > > > > https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161 > > > < > > > https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161 > > > > > > https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161 > > > > > > > > > Best, > > > > > > Paul Lam > > > > > > > > > > > > 2023年6月3日 12:21，Jark Wu <imj...@gmail.com mailto:imj...@gmail.com > > > > > > <mailto:imj...@gmail.com mailto:imj...@gmail.com>> > > > > > > 写道： > > > > > > > > > > > > Hi Paul, > > > > > > > > > > > > Thanks for your reply. I left my comments inline. > > > > > > > > > > > > As the FLIP said, it’s good to have a default main class for Flink > > > > > > > > > > > > SQLs, > > > > > > > > > > > > which allows users to submit Flink SQLs in the same way as > > > > > > > > > > > > DataStream > > > > > > > > > > > > jobs, or else users need to write their own main class. > > > > > > > > > > > > Isn't Table API the same way as DataSream jobs to submit Flink SQL? > > > > > > DataStream API also doesn't provide a default main class for users, > > > > > > why do we need to provide such one for SQL? > > > > > > > > > > > > With the help of ExecNodeGraph, do we still need the serialized > > > > > > SessionState? If not, we could make SQL Driver accepts two > > > > > > > > > > > > serialized > > > > > > > > > > > > formats: > > > > > > > > > > > > No, ExecNodeGraph doesn't need to serialize SessionState. I thought > > > > > > > > > > > > the > > > > > > > > > > > > proposed SqlDriver was a dedicated main class accepting SQL files, > > > > > > > > > > > > is > > > > > > > > > > > > that correct? > > > > > > If true, we have to ship the SessionState for this case which is a > > > > > > > > > > > > large > > > > > > > > > > > > work. > > > > > > I think we just need a JsonPlanDriver which is a main class that > > > > > > > > > > > > accepts > > > > > > > > > > > > JsonPlan as the parameter. > > > > > > > > > > > > The common solutions I know is to use distributed file systems or > > > > > > > > > > > > use > > > > > > > > > > > > init containers to localize the resources. > > > > > > > > > > > > Personally, I prefer the way of init containers which doesn't > > > > > > > > > > > > depend > > > > > > > > > > > > on > > > > > > > > > > > > additional components. > > > > > > This can reduce the moving parts of a production environment. > > > > > > Depending on a distributed file system makes the testing, demo, and > > > > > > > > > > > > local > > > > > > > > > > > > setup harder than init containers. > > > > > > > > > > > > Best, > > > > > > Jark > > > > > > > > > > > > On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3...@gmail.com > > > > > > mailto:paullin3...@gmail.com <mailto: > > > > > > paullin3...@gmail.com mailto:paullin3...@gmail.com> > > > > > > > > > > > > <mailto: > > > > > > > > > > > > paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com>> > > > > > > <mailto: > > > > > > > > > > > > paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com> <mailto: > > > > > > paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com>>>> > > > > > > wrote: > > > > > > > > > > > > The FLIP is in the early phase and some details are not included, > > > > > > > > > > > > but > > > > > > > > > > > > fortunately, we got lots of valuable ideas from the discussion. > > > > > > > > > > > > Thanks to everyone who joined the dissuasion! > > > > > > @Weihua @Shanmon @Shengkai @Biao @Jark > > > > > > > > > > > > This weekend I’m gonna revisit and update the FLIP, adding more > > > > > > details. Hopefully, we can further align our opinions. > > > > > > > > > > > > Best, > > > > > > Paul Lam > > > > > > > > > > > > 2023年6月2日 18:02，Paul Lam <paullin3...@gmail.com > > > > > > mailto:paullin3...@gmail.com <mailto: > > > > > > paullin3...@gmail.com mailto:paullin3...@gmail.com> <mailto: > > > > > > > > > > > > paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com>>> 写道： > > > > > > > > > > > > Hi Jark, > > > > > > > > > > > > Thanks a lot for your input! > > > > > > > > > > > > If we decide to submit ExecNodeGraph instead of SQL file, is it > > > > > > > > > > > > still > > > > > > > > > > > > necessary to support SQL Driver? > > > > > > > > > > > > I think so. Apart from usage in SQL Gateway, SQL Driver could > > > > > > > > > > > > simplify > > > > > > > > > > > > Flink SQL execution with Flink CLI. > > > > > > > > > > > > As the FLIP said, it’s good to have a default main class for > > > > > > > > > > > > Flink > > > > > > > > > > > > SQLs, > > > > > > > > > > > > which allows users to submit Flink SQLs in the same way as > > > > > > > > > > > > DataStream > > > > > > > > > > > > jobs, or else users need to write their own main class. > > > > > > > > > > > > SQL Driver needs to serialize SessionState which is very > > > > > > > > > > > > challenging > > > > > > > > > > > > but not detailed covered in the FLIP. > > > > > > > > > > > > With the help of ExecNodeGraph, do we still need the serialized > > > > > > SessionState? If not, we could make SQL Driver accepts two > > > > > > > > > > > > serialized > > > > > > > > > > > > formats: > > > > > > > > > > > > - SQL files for user-facing public usage > > > > > > - ExecNodeGraph for internal usage > > > > > > > > > > > > It’s kind of similar to the relationship between job jars and > > > > > > > > > > > > jobgraphs. > > > > > > > > > > > > Regarding "K8S doesn't support shipping multiple jars", is that > > > > > > > > > > > > true? > > > > > > > > > > > > Is it > > > > > > > > > > > > possible to support it? > > > > > > > > > > > > Yes, K8s doesn’t distribute any files. It’s the users’ > > > > > > > > > > > > responsibility > > > > > > > > > > > > to > > > > > > > > > > > > make > > > > > > > > > > > > sure the resources are accessible in the containers. The common > > > > > > > > > > > > solutions > > > > > > > > > > > > I know is to use distributed file systems or use init containers > > > > > > > > > > > > to > > > > > > > > > > > > localize the > > > > > > > > > > > > resources. > > > > > > > > > > > > Now I lean toward introducing a fs to do the distribution job. > > > > > > > > > > > > WDYT? > > > > > > > > > > > > Best, > > > > > > Paul Lam > > > > > > > > > > > > 2023年6月1日 20:33，Jark Wu <imj...@gmail.com mailto:imj...@gmail.com > > > > > > <mailto:imj...@gmail.com mailto:imj...@gmail.com> > > > > > > <mailto: > > > > > > > > > > > > imj...@gmail.com mailto:imj...@gmail.com <mailto:imj...@gmail.com > > > > > > mailto:imj...@gmail.com>> > > > > > > > > > > > > <mailto:imj...@gmail.com mailto:imj...@gmail.com > > > > > > <mailto:imj...@gmail.com mailto:imj...@gmail.com> <mailto: > > > > > > imj...@gmail.com mailto:imj...@gmail.com <mailto:imj...@gmail.com > > > > > > mailto:imj...@gmail.com>>> > > > > > > > > > > > > <mailto:imj...@gmail.com mailto:imj...@gmail.com > > > > > > <mailto:imj...@gmail.com mailto:imj...@gmail.com> <mailto: > > > > > > imj...@gmail.com mailto:imj...@gmail.com <mailto:imj...@gmail.com > > > > > > mailto:imj...@gmail.com>> <mailto: > > > > > > > > > > > > imj...@gmail.com mailto:imj...@gmail.com <mailto:imj...@gmail.com > > > > > > mailto:imj...@gmail.com> <mailto:imj...@gmail.com > > > > > > mailto:imj...@gmail.com > > > > > > <mailto:imj...@gmail.com mailto:imj...@gmail.com>>>>> > > > > > > > > > > > > 写道： > > > > > > > > > > > > Hi Paul, > > > > > > > > > > > > Thanks for starting this discussion. I like the proposal! This > > > > > > > > > > > > is > > > > > > > > > > > > a > > > > > > > > > > > > frequently requested feature! > > > > > > > > > > > > I agree with Shengkai that ExecNodeGraph as the submission > > > > > > > > > > > > object > > > > > > > > > > > > is a > > > > > > > > > > > > better idea than SQL file. To be more specific, it should be > > > > > > > > > > > > JsonPlanGraph > > > > > > > > > > > > or CompiledPlan which is the serializable representation. > > > > > > > > > > > > CompiledPlan > > > > > > > > > > > > is a > > > > > > > > > > > > clear separation between compiling/optimization/validation and > > > > > > > > > > > > execution. > > > > > > > > > > > > This can keep the validation and metadata accessing still on the > > > > > > > > > > > > SQLGateway > > > > > > > > > > > > side. This allows SQLGateway to leverage some metadata caching > > > > > > > > > > > > and > > > > > > > > > > > > UDF > > > > > > > > > > > > JAR > > > > > > > > > > > > caching for better compiling performance. > > > > > > > > > > > > If we decide to submit ExecNodeGraph instead of SQL file, is it > > > > > > > > > > > > still > > > > > > > > > > > > necessary to support SQL Driver? Regarding non-interactive SQL > > > > > > > > > > > > jobs, > > > > > > > > > > > > users > > > > > > > > > > > > can use the Table API program for application mode. SQL Driver > > > > > > > > > > > > needs > > > > > > > > > > > > to > > > > > > > > > > > > serialize SessionState which is very challenging but not > > > > > > > > > > > > detailed > > > > > > > > > > > > covered > > > > > > > > > > > > in the FLIP. > > > > > > > > > > > > Regarding "K8S doesn't support shipping multiple jars", is that > > > > > > > > > > > > true? > > > > > > > > > > > > Is it > > > > > > > > > > > > possible to support it? > > > > > > > > > > > > Best, > > > > > > Jark > > > > > > > > > > > > On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3...@gmail.com > > > > > > mailto:paullin3...@gmail.com <mailto: > > > > > > paullin3...@gmail.com mailto:paullin3...@gmail.com> > > > > > > > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com>> > > > > > > <mailto: > > > > > > > > > > > > paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com> <mailto: > > > > > > paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com>>> > > > > > > <mailto: > > > > > > > > > > > > paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com> <mailto: > > > > > > paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com>> > > > > > > <mailto: > > > > > > > > > > > > paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com> <mailto: > > > > > > paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com>>>>> > > > > > > wrote: > > > > > > > > > > > > Hi Weihua, > > > > > > > > > > > > You’re right. Distributing the SQLs to the TMs is one of the > > > > > > > > > > > > challenging > > > > > > > > > > > > parts of this FLIP. > > > > > > > > > > > > Web submission is not enabled in application mode currently as > > > > > > > > > > > > you > > > > > > > > > > > > said, > > > > > > > > > > > > but it could be changed if we have good reasons. > > > > > > > > > > > > What do you think about introducing a distributed storage for > > > > > > > > > > > > SQL > > > > > > > > > > > > Gateway? > > > > > > > > > > > > We could make use of Flink file systems [1] to distribute the > > > > > > > > > > > > SQL > > > > > > > > > > > > Gateway > > > > > > > > > > > > generated resources, that should solve the problem at its root > > > > > > > > > > > > cause. > > > > > > > > > > > > Users could specify Flink-supported file systems to ship files. > > > > > > > > > > > > It’s > > > > > > > > > > > > only > > > > > > > > > > > > required when using SQL Gateway with K8s application mode. > > > > > > > > > > > > [1] > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > < > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > > > > < > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > < > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > > > > < > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > < > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > > > > < > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > < > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > > > > < > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > < > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > > > > < > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > < > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > > > > < > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > < > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > > > > < > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > < > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ > > > > > > > > > Best, > > > > > > Paul Lam > > > > > > > > > > > > 2023年6月1日 13:55，Weihua Hu <huweihua....@gmail.com > > > > > > mailto:huweihua....@gmail.com <mailto: > > > > > > huweihua....@gmail.com mailto:huweihua....@gmail.com> <mailto: > > > > > > > > > > > > huweihua....@gmail.com mailto:huweihua....@gmail.com > > > > > > <mailto:huweihua....@gmail.com mailto:huweihua....@gmail.com>> > > > > > > <mailto: > > > > > > > > > > > > huweihua....@gmail.com mailto:huweihua....@gmail.com > > > > > > <mailto:huweihua....@gmail.com mailto:huweihua....@gmail.com> > > > > > > <mailto: > > > > > > huweihua....@gmail.com mailto:huweihua....@gmail.com > > > > > > <mailto:huweihua....@gmail.com mailto:huweihua....@gmail.com>>>> 写道： > > > > > > > > > > > > Thanks Paul for your reply. > > > > > > > > > > > > SQLDriver looks good to me. > > > > > > > > > > > > 2. Do you mean a pass the SQL string a configuration or a > > > > > > > > > > > > program > > > > > > > > > > > > argument? > > > > > > > > > > > > I brought this up because we were unable to pass the SQL file > > > > > > > > > > > > to > > > > > > > > > > > > Flink > > > > > > > > > > > > using Kubernetes mode. > > > > > > For DataStream/Python users, they need to prepare their images > > > > > > > > > > > > for > > > > > > > > > > > > the > > > > > > > > > > > > jars > > > > > > > > > > > > and dependencies. > > > > > > But for SQL users, they can use a common image to run > > > > > > > > > > > > different > > > > > > > > > > > > SQL > > > > > > > > > > > > queries > > > > > > > > > > > > if there are no other udf requirements. > > > > > > It would be great if the SQL query and image were not bound. > > > > > > > > > > > > Using strings is a way to decouple these, but just as you > > > > > > > > > > > > mentioned, > > > > > > > > > > > > it's > > > > > > > > > > > > not easy to pass complex SQL. > > > > > > > > > > > > use web submission > > > > > > > > > > > > AFAIK, we can not use web submission in the Application mode. > > > > > > > > > > > > Please > > > > > > > > > > > > correct me if I'm wrong. > > > > > > > > > > > > Best, > > > > > > Weihua > > > > > > > > > > > > On Wed, May 31, 2023 at 9:37 PM Paul Lam < > > > > > > > > > > > > paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com> > > > > > > > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com>> > > > > > > > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com> <mailto: > > > > > > paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com>>>> > > > > > > > > > > > > wrote: > > > > > > > > > > > > Hi Biao, > > > > > > > > > > > > Thanks for your comments! > > > > > > > > > > > > 1. Scope: is this FLIP only targeted for non-interactive > > > > > > > > > > > > Flink > > > > > > > > > > > > SQL > > > > > > > > > > > > jobs > > > > > > > > > > > > in > > > > > > > > > > > > Application mode? More specifically, if we use SQL > > > > > > > > > > > > client/gateway > > > > > > > > > > > > to > > > > > > > > > > > > execute some interactive SQLs like a SELECT query, can we > > > > > > > > > > > > ask > > > > > > > > > > > > flink > > > > > > > > > > > > to > > > > > > > > > > > > use > > > > > > > > > > > > Application mode to execute those queries after this FLIP? > > > > > > > > > > > > Thanks for pointing it out. I think only DMLs would be > > > > > > > > > > > > executed > > > > > > > > > > > > via > > > > > > > > > > > > SQL > > > > > > > > > > > > Driver. > > > > > > I'll add the scope to the FLIP. > > > > > > > > > > > > 2. Deployment: I believe in YARN mode, the implementation is > > > > > > > > > > > > trivial as > > > > > > > > > > > > we > > > > > > > > > > > > can ship files via YARN's tool easily but for K8s, things > > > > > > > > > > > > can > > > > > > > > > > > > be > > > > > > > > > > > > more > > > > > > > > > > > > complicated as Shengkai said. > > > > > > > > > > > > Your input is very informative. I’m thinking about using web > > > > > > > > > > > > submission, > > > > > > > > > > > > but it requires exposing the JobManager port which could also > > > > > > > > > > > > be > > > > > > > > > > > > a > > > > > > > > > > > > problem > > > > > > > > > > > > on K8s. > > > > > > > > > > > > Another approach is to explicitly require a distributed > > > > > > > > > > > > storage > > > > > > > > > > > > to > > > > > > > > > > > > ship > > > > > > > > > > > > files, > > > > > > but we may need a new deployment executor for that. > > > > > > > > > > > > What do you think of these two approaches? > > > > > > > > > > > > 3. Serialization of SessionState: in SessionState, there are > > > > > > > > > > > > some > > > > > > > > > > > > unserializable fields > > > > > > like > > > > > > > > > > > > org.apache.flink.table.resource.ResourceManager#userClassLoader. > > > > > > > > > > > > It > > > > > > > > > > > > may be worthwhile to add more details about the > > > > > > > > > > > > serialization > > > > > > > > > > > > part. > > > > > > > > > > > > I agree. That’s a missing part. But if we use ExecNodeGraph > > > > > > > > > > > > as > > > > > > > > > > > > Shengkai > > > > > > > > > > > > mentioned, do we eliminate the need for serialization of > > > > > > > > > > > > SessionState? > > > > > > > > > > > > Best, > > > > > > Paul Lam > > > > > > > > > > > > 2023年5月31日 13:07，Biao Geng <biaoge...@gmail.com > > > > > > mailto:biaoge...@gmail.com <mailto: > > > > > > biaoge...@gmail.com mailto:biaoge...@gmail.com> <mailto: > > > > > > > > > > > > biaoge...@gmail.com mailto:biaoge...@gmail.com > > > > > > <mailto:biaoge...@gmail.com mailto:biaoge...@gmail.com>> <mailto: > > > > > > > > > > > > biaoge...@gmail.com mailto:biaoge...@gmail.com > > > > > > <mailto:biaoge...@gmail.com mailto:biaoge...@gmail.com> <mailto: > > > > > > biaoge...@gmail.com mailto:biaoge...@gmail.com > > > > > > <mailto:biaoge...@gmail.com mailto:biaoge...@gmail.com>>>> 写道： > > > > > > > > > > > > Thanks Paul for the proposal!I believe it would be very > > > > > > > > > > > > useful > > > > > > > > > > > > for > > > > > > > > > > > > flink > > > > > > > > > > > > users. > > > > > > After reading the FLIP, I have some questions: > > > > > > 1. Scope: is this FLIP only targeted for non-interactive > > > > > > > > > > > > Flink > > > > > > > > > > > > SQL > > > > > > > > > > > > jobs > > > > > > > > > > > > in > > > > > > > > > > > > Application mode? More specifically, if we use SQL > > > > > > > > > > > > client/gateway > > > > > > > > > > > > to > > > > > > > > > > > > execute some interactive SQLs like a SELECT query, can we > > > > > > > > > > > > ask > > > > > > > > > > > > flink > > > > > > > > > > > > to > > > > > > > > > > > > use > > > > > > > > > > > > Application mode to execute those queries after this FLIP? > > > > > > 2. Deployment: I believe in YARN mode, the implementation is > > > > > > > > > > > > trivial as > > > > > > > > > > > > we > > > > > > > > > > > > can ship files via YARN's tool easily but for K8s, things > > > > > > > > > > > > can > > > > > > > > > > > > be > > > > > > > > > > > > more > > > > > > > > > > > > complicated as Shengkai said. I have implemented a simple > > > > > > > > > > > > POC > > > > > > > > > > > > < > > > > > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 > > > > > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 > > > < > > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 > > > > > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 > > > > > > > > > < > > > > > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 > > > > > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 > > > < > > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 > > > > > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 > > > > > > > > > < > > > > > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 > > > > > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 > > > < > > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 > > > > > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 > > > > > > > > > < > > > > > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 > > > > > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 > > > < > > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 > > > > > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 > > > > > > > > > based on SQL client before(i.e. consider the SQL client > > > > > > > > > > > > which > > > > > > > > > > > > supports > > > > > > > > > > > > executing a SQL file as the SQL driver in this FLIP). One > > > > > > > > > > > > problem > > > > > > > > > > > > I > > > > > > > > > > > > have > > > > > > > > > > > > met is how do we ship SQL files ( or Job Graph) to the k8s > > > > > > > > > > > > side. > > > > > > > > > > > > Without > > > > > > > > > > > > such support, users have to modify the initContainer or > > > > > > > > > > > > rebuild > > > > > > > > > > > > a > > > > > > > > > > > > new > > > > > > > > > > > > K8s > > > > > > > > > > > > image every time to fetch the SQL file. Like the flink k8s > > > > > > > > > > > > operator, > > > > > > > > > > > > one > > > > > > > > > > > > workaround is to utilize the flink config(transforming the > > > > > > > > > > > > SQL > > > > > > > > > > > > file > > > > > > > > > > > > to > > > > > > > > > > > > a > > > > > > > > > > > > escaped string like Weihua mentioned) which will be > > > > > > > > > > > > converted > > > > > > > > > > > > to a > > > > > > > > > > > > ConfigMap but K8s has size limit of ConfigMaps(no larger > > > > > > > > > > > > than > > > > > > > > > > > > 1MB > > > > > > > > > > > > < > > > > > > > > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/ > > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/ < > > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/ > > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/> > > > > > > > > > > > > < > > > > > > > > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/ > > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/ < > > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/ > > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/>> < > > > > > > > > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/ > > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/ < > > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/ > > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/> < > > > > > > > > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/ > > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/ < > > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/ > > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/>>>>). > > > > > > > > > > > > Not > > > > > > > > > > > > sure > > > > > > > > > > > > if we have better solutions. > > > > > > 3. Serialization of SessionState: in SessionState, there are > > > > > > > > > > > > some > > > > > > > > > > > > unserializable fields > > > > > > like > > > > > > > > > > > > org.apache.flink.table.resource.ResourceManager#userClassLoader. > > > > > > > > > > > > It > > > > > > > > > > > > may be worthwhile to add more details about the > > > > > > > > > > > > serialization > > > > > > > > > > > > part. > > > > > > > > > > > > Best, > > > > > > Biao Geng > > > > > > > > > > > > Paul Lam <paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com> > > > > > > <mailto: > > > > > > > > > > > > paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com> > > > > > > > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com> <mailto: > > > > > > paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com>>>> > > > > > > > > > > > > 于2023年5月31日周三 11:49写道： > > > > > > > > > > > > Hi Weihua, > > > > > > > > > > > > Thanks a lot for your input! Please see my comments inline. > > > > > > > > > > > > - Is SQLRunner the better name? We use this to run a SQL > > > > > > > > > > > > Job. > > > > > > > > > > > > (Not > > > > > > > > > > > > strong, > > > > > > > > > > > > the SQLDriver is fine for me) > > > > > > > > > > > > I’ve thought about SQL Runner but picked SQL Driver for the > > > > > > > > > > > > following > > > > > > > > > > > > reasons FYI: > > > > > > > > > > > > 1. I have a PythonDriver doing the same job for PyFlink [1] > > > > > > 2. Flink program's main class is sort of like Driver in > > > > > > > > > > > > JDBC > > > > > > > > > > > > which > > > > > > > > > > > > translates SQLs into > > > > > > databases specific languages. > > > > > > > > > > > > In general, I’m +1 for SQL Driver and +0 for SQL Runner. > > > > > > > > > > > > - Could we run SQL jobs using SQL in strings? Otherwise, > > > > > > > > > > > > we > > > > > > > > > > > > need > > > > > > > > > > > > to > > > > > > > > > > > > prepare > > > > > > > > > > > > a SQL file in an image for Kubernetes application mode, > > > > > > > > > > > > which > > > > > > > > > > > > may > > > > > > > > > > > > be > > > > > > > > > > > > a > > > > > > > > > > > > bit > > > > > > > > > > > > cumbersome. > > > > > > > > > > > > Do you mean a pass the SQL string a configuration or a > > > > > > > > > > > > program > > > > > > > > > > > > argument? > > > > > > > > > > > > I thought it might be convenient for testing propose, but > > > > > > > > > > > > not > > > > > > > > > > > > recommended > > > > > > > > > > > > for production, > > > > > > cause Flink SQLs could be complicated and involves lots of > > > > > > > > > > > > characters > > > > > > > > > > > > that > > > > > > > > > > > > need to escape. > > > > > > > > > > > > WDYT? > > > > > > > > > > > > - I noticed that we don't specify the SQLDriver jar in the > > > > > > > > > > > > "run-application" > > > > > > > > > > > > command. Does that mean we need to perform automatic > > > > > > > > > > > > detection > > > > > > > > > > > > in > > > > > > > > > > > > Flink? > > > > > > > > > > > > Yes! It’s like running a PyFlink job with the following > > > > > > > > > > > > command: > > > > > > > > > > > > `./bin/flink run \\ --pyModule table.word_count \\ --pyFiles > > > > > > examples/python/table` > > > > > > > > > > > > The CLI determines if it’s a SQL job, if yes apply the SQL > > > > > > > > > > > > Driver > > > > > > > > > > > > automatically. > > > > > > > > > > > > [1] > > > > > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java > > > > > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java > > > < > > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java > > > > > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java > > > > > > > > > < > > > > > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java > > > > > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java > > > < > > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java > > > > > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java > > > > > > > > > < > > > > > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java > > > > > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java > > > < > > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java > > > > > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java > > > > > > > > > < > > > > > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java > > > > > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java > > > < > > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java > > > > > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java > > > > > > > > > Best, > > > > > > Paul Lam > > > > > > > > > > > > 2023年5月30日 21:56，Weihua Hu <huweihua....@gmail.com > > > > > > mailto:huweihua....@gmail.com <mailto: > > > > > > huweihua....@gmail.com mailto:huweihua....@gmail.com> > > > > > > > > > > > > <mailto: > > > > > > > > > > > > huweihua....@gmail.com mailto:huweihua....@gmail.com > > > > > > <mailto:huweihua....@gmail.com mailto:huweihua....@gmail.com>> > > > > > > <mailto: > > > > > > > > > > > > huweihua....@gmail.com mailto:huweihua....@gmail.com > > > > > > <mailto:huweihua....@gmail.com mailto:huweihua....@gmail.com>>> 写道： > > > > > > > > > > > > Thanks Paul for the proposal. > > > > > > > > > > > > +1 for this. It is valuable in improving ease of use. > > > > > > > > > > > > I have a few questions. > > > > > > - Is SQLRunner the better name? We use this to run a SQL > > > > > > > > > > > > Job. > > > > > > > > > > > > (Not > > > > > > > > > > > > strong, > > > > > > > > > > > > the SQLDriver is fine for me) > > > > > > - Could we run SQL jobs using SQL in strings? Otherwise, > > > > > > > > > > > > we > > > > > > > > > > > > need > > > > > > > > > > > > to > > > > > > > > > > > > prepare > > > > > > > > > > > > a SQL file in an image for Kubernetes application mode, > > > > > > > > > > > > which > > > > > > > > > > > > may > > > > > > > > > > > > be > > > > > > > > > > > > a > > > > > > > > > > > > bit > > > > > > > > > > > > cumbersome. > > > > > > - I noticed that we don't specify the SQLDriver jar in the > > > > > > > > > > > > "run-application" > > > > > > > > > > > > command. Does that mean we need to perform automatic > > > > > > > > > > > > detection > > > > > > > > > > > > in > > > > > > > > > > > > Flink? > > > > > > > > > > > > Best, > > > > > > Weihua > > > > > > > > > > > > On Mon, May 29, 2023 at 7:24 PM Paul Lam < > > > > > > > > > > > > paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com> > > > > > > > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com> <mailto: > > > > > > paullin3...@gmail.com mailto:paullin3...@gmail.com > > > > > > <mailto:paullin3...@gmail.com mailto:paullin3...@gmail.com>>>> > > > > > > > > > > > > wrote: > > > > > > > > > > > > Hi team, > > > > > > > > > > > > I’d like to start a discussion about FLIP-316 [1], which > > > > > > > > > > > > introduces > > > > > > > > > > > > a > > > > > > > > > > > > SQL > > > > > > > > > > > > driver as the > > > > > > default main class for Flink SQL jobs. > > > > > > > > > > > > Currently, Flink SQL could be executed out of the box > > > > > > > > > > > > either > > > > > > > > > > > > via > > > > > > > > > > > > SQL > > > > > > > > > > > > Client/Gateway > > > > > > or embedded in a Flink Java/Python program. > > > > > > > > > > > > However, each one has its drawback: > > > > > > > > > > > > - SQL Client/Gateway doesn’t support the application > > > > > > > > > > > > deployment > > > > > > > > > > > > mode > > > > > > > > > > > > [2] > > > > > > > > > > > > - Flink Java/Python program requires extra work to write > > > > > > > > > > > > a > > > > > > > > > > > > non-SQL > > > > > > > > > > > > program > > > > > > > > > > > > Therefore, I propose adding a SQL driver to act as the > > > > > > > > > > > > default > > > > > > > > > > > > main > > > > > > > > > > > > class > > > > > > > > > > > > for SQL jobs. > > > > > > Please see the FLIP docs for details and feel free to > > > > > > > > > > > > comment. > > > > > > > > > > > > Thanks! > > > > > > > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver > > > < > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver > > > > > > > > > < > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver > > > < > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver > > > > > > > > > [2] https://issues.apache.org/jira/browse/FLINK-26541 > > > > > > https://issues.apache.org/jira/browse/FLINK-26541 < > > > > > > https://issues.apache.org/jira/browse/FLINK-26541 > > > > > > https://issues.apache.org/jira/browse/FLINK-26541> < > > > > > > https://issues.apache.org/jira/browse/FLINK-26541 > > > > > > https://issues.apache.org/jira/browse/FLINK-26541 < > > > > > > https://issues.apache.org/jira/browse/FLINK-26541 > > > > > > https://issues.apache.org/jira/browse/FLINK-26541>> > > > > > > > > > > > > Best, > > > > > > Paul Lam
