As I wrote in my previous answer, this could be done as a helm chart or as part of the operator easily. Both would work. My main concern for adding this into the operator is that the SQL Gateway itself is not properly integrated with the Operator Custom resources.
Gyula On Mon, Sep 18, 2023 at 4:24 AM Shammon FY <zjur...@gmail.com> wrote: > Thanks @Gyula, I would like to share our use of sql-gateway with the Flink > session cluster and I hope that it could help you to have a clearer > understanding of our needs :) > > As @Yangze mentioned, currently we use flink as an olap platform by the > following steps > 1. Setup a flink session cluster by flink k8s session with k8s or zk > highavailable. > 2. Write a Helm chart for Sql-Gateway image and launch multiple gateway > instances to submit jobs to the same flink session cluster. > > As we mentioned in docs[1], we hope that users can easily launch > sql-gateway instances in k8s. Does it only need to add a Helm chart for > sql-gateway, or should we need to add this feature to the flink > operator? Can you help give the conclusion? Thank you very much @Gyula > > [1] > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/olap_quickstart/ > > Best, > Shammon FY > > > > On Sun, Sep 17, 2023 at 2:02 PM Gyula Fóra <gyula.f...@gmail.com> wrote: > > > Hi! > > It sounds pretty easy to deploy the gateway automatically with session > > cluster deployments from the operator , but there is a major limitation > > currently. The SQL gateway itself doesn't really support any operator > > integration so jobs submitted through the SQL gateway would not be > > manageable by the operator (they won't show up as session jobs). > > > > Without that, this is a very strange feature. We would make something > much > > easier for users that is not well supported by the operator in the first > > place. The operator is designed to manage clusters and jobs > > (FlinkDeployment / FlinkSessionJob). It would be good to understand if we > > could make the SQL Gateway create a FlinkSessionJob / Deployment (that > > would require application cluster support) and basically submit the job > > through the operator. > > > > Cheers, > > Gyula > > > > On Sun, Sep 17, 2023 at 1:26 AM Yangze Guo <karma...@gmail.com> wrote: > > > > > > There would be many different ways of doing this. One gateway per > > > session cluster, one gateway shared across different clusters... > > > > > > Currently, sql gateway cannot be shared across multiple clusters. > > > > > > > understand the tradeoff and the simplest way of accomplishing this. > > > > > > I'm not familiar with the Flink operator codebase, it would be > > > appreciated if you could elaborate more on the cost of adding this > > > feature. I agree that deploying a gateway using the native Kubernetes > > > Deployment can be a simple way and straightforward for users. However, > > > integrating it into an operator can provide additional benefits and be > > > more user-friendly, especially for users who are less familiar with > > > Kubernetes. By using an operator, users can benefit from consistent > > > version management with the session cluster and upgrade capabilities. > > > > > > > > > Best, > > > Yangze Guo > > > > > > On Fri, Sep 15, 2023 at 5:38 PM Gyula Fóra <gyula.f...@gmail.com> > wrote: > > > > > > > > There would be many different ways of doing this. One gateway per > > session > > > > cluster, one gateway shared across different clusters... > > > > I would not rush to add anything anywhere until we understand the > > > tradeoff > > > > and the simplest way of accomplishing this. > > > > > > > > The operator already supports ingresses for session clusters so we > > could > > > > have a gateway sitting somewhere else simply using it. > > > > > > > > Gyula > > > > > > > > On Fri, Sep 15, 2023 at 10:18 AM Yangze Guo <karma...@gmail.com> > > wrote: > > > > > > > > > Thanks for bringing this up, Dongwoo. Flink SQL Gateway is also a > key > > > > > component for OLAP scenarios. > > > > > > > > > > @Gyula > > > > > How about add sql gateway as an optional component to Session > Cluster > > > > > Deployments. User can specify the resource / instance number and > > ports > > > > > of the sql gateway. I think that would help a lot for OLAP and > batch > > > > > user. > > > > > > > > > > > > > > > Best, > > > > > Yangze Guo > > > > > > > > > > On Fri, Sep 15, 2023 at 3:19 PM ConradJam <jam.gz...@gmail.com> > > wrote: > > > > > > > > > > > > If we start from the crd direction, I think this mode is more > like > > a > > > > > > sidecar of the session cluster, which is submitted to the session > > > cluster > > > > > > by sending sql commands to the sql gateway. I don't know if my > > > statement > > > > > is > > > > > > accurate. > > > > > > > > > > > > Xiaolong Wang <xiaolong.w...@smartnews.com.invalid> > 于2023年9月15日周五 > > > > > 13:27写道： > > > > > > > > > > > > > Hi, Dongwoo, > > > > > > > > > > > > > > Since Flink SQL gateway should run upon a Flink session > cluster, > > I > > > > > think > > > > > > > it'd be easier to add more fields to the CRD of > > `FlinkSessionJob`. > > > > > > > > > > > > > > e.g. > > > > > > > > > > > > > > apiVersion: flink.apache.org/v1beta1 > > > > > > > kind: FlinkSessionJob > > > > > > > metadata: > > > > > > > name: sql-gateway > > > > > > > spec: > > > > > > > sqlGateway: > > > > > > > endpoint: "hiveserver2" > > > > > > > mode: "streaming" > > > > > > > hiveConf: > > > > > > > configMap: > > > > > > > name: hive-config > > > > > > > items: > > > > > > > - key: hive-site.xml > > > > > > > path: hive-site.xml > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 15, 2023 at 12:56 PM Dongwoo Kim < > > > dongwoo7....@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > *@Gyula* > > > > > > > > Thanks for the consideration Gyula. My initial idea for the > CR > > > was > > > > > > > roughly > > > > > > > > like below. > > > > > > > > I focused on simplifying the setup in k8s environment, but I > > > agree > > > > > with > > > > > > > > your opinion that for the sql gateway > > > > > > > > we don't need custom operator logic to handle and most of the > > > > > > > requirements > > > > > > > > can be met by existing k8s resources. > > > > > > > > So maybe helm chart that bundles all resources needed should > be > > > > > enough. > > > > > > > > > > > > > > > > apiVersion: flink.apache.org/v1beta1 > > > > > > > > kind: FlinkSqlGateway > > > > > > > > metadata: > > > > > > > > name: flink-sql-gateway-example > > > > > > > > namespace: default > > > > > > > > spec: > > > > > > > > clusterName: flink-session-cluster-example > > > > > > > > exposeServiceType: LoadBalancer > > > > > > > > flinkSqlGatewayConfiguration: > > > > > > > > sql-gateway.endpoint.type: "hiveserver2" > > > > > > > > sql-gateway.endpoint.hiveserver2.catalog.name: "hive" > > > > > > > > hiveConf: > > > > > > > > configMap: > > > > > > > > name: hive-config > > > > > > > > items: > > > > > > > > - key: hive-site.xml > > > > > > > > path: hive-site.xml > > > > > > > > > > > > > > > > > > > > > > > > *@xiaolong, @Shammon* > > > > > > > > Hi xiaolong and Shammon. > > > > > > > > Thanks for taking the time to share. > > > > > > > > I'd also like to add my experience with setting up flink sql > > > gateway > > > > > on > > > > > > > > k8s. > > > > > > > > Without building a new Docker image, I've added a separate > > > container > > > > > to > > > > > > > the > > > > > > > > existing JobManager pod and started the sql gateway using the > > > > > > > > "sql-gateway.sh start-foreground" command. > > > > > > > > I haven't explored deploying the sql gateway as an > independent > > > > > deployment > > > > > > > > yet, but that's something I'm considering after modifying > JM's > > > > > address to > > > > > > > > desired session cluster. > > > > > > > > > > > > > > > > Thanks all > > > > > > > > > > > > > > > > Best > > > > > > > > Dongwoo > > > > > > > > > > > > > > > > 2023년 9월 15일 (금) 오전 11:55, Xiaolong Wang > > > > > > > > <xiaolong.w...@smartnews.com.invalid>님이 작성: > > > > > > > > > > > > > > > > > Hi, Shammon, > > > > > > > > > > > > > > > > > > Yes, I want to create a Flink SQL-gateway in a job-manager. > > > > > > > > > > > > > > > > > > Currently, the above script is generally a work-around and > > > allows > > > > > me to > > > > > > > > > start a Flink session job manager with a SQL gateway > running > > > upon. > > > > > > > > > > > > > > > > > > I agree that it'd be more elegant that we create a new job > > > type and > > > > > > > > write a > > > > > > > > > script, which is much easier for the user to use (since > they > > > do not > > > > > > > need > > > > > > > > to > > > > > > > > > build a separate Flink image any more). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 15, 2023 at 10:29 AM Shammon FY < > > zjur...@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > Currently `sql-gateway` can be started with the script > > > > > > > `sql-gateway.sh` > > > > > > > > > in > > > > > > > > > > an existing node, it is more like a simple "standalone" > > > node. I > > > > > think > > > > > > > > > it's > > > > > > > > > > valuable if we can do more work to start it in k8s. > > > > > > > > > > > > > > > > > > > > For xiaolong: > > > > > > > > > > Do you want to start a sql-gateway instance in the > > jobmanager > > > > > pod? I > > > > > > > > > think > > > > > > > > > > maybe we need a script like `kubernetes-sql-gatewah.sh` > to > > > start > > > > > > > > > > `sql-gateway` pods with a flink image, what do you think? > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Shammon FY > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 15, 2023 at 10:02 AM Xiaolong Wang > > > > > > > > > > <xiaolong.w...@smartnews.com.invalid> wrote: > > > > > > > > > > > > > > > > > > > > > Hi, I've experiment this feature on K8S recently, here > is > > > some > > > > > of > > > > > > > my > > > > > > > > > > trial: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Create a new kubernetes-jobmanager.sh script with > the > > > > > following > > > > > > > > > > content > > > > > > > > > > > > > > > > > > > > > > #!/usr/bin/env bash > > > > > > > > > > > $FLINK_HOME/bin/sql-gateway.sh start > > > > > > > > > > > $FLINK_HOME/bin/kubernetes-jobmanager1.sh > > > kubernetes-session > > > > > > > > > > > > > > > > > > > > > > 2. Build your own Flink docker image something like > this > > > > > > > > > > > FROM flink:1.17.1-scala_2.12-java11 > > > > > > > > > > > > > > > > > > > > > > RUN mv $FLINK_HOME/bin/kubernetes-jobmanager.sh > > > > > $FLINK_HOME/bin/ > > > > > > > > > > > kubernetes-jobmanager1.sh > > > > > > > > > > > COPY ./kubernetes-jobmanager.sh > > > > > > > > > $FLINK_HOME/bin/kubernetes-jobmanager.sh > > > > > > > > > > > > > > > > > > > > > > RUN chmod +x $FLINK_HOME/bin/*.sh > > > > > > > > > > > USER flink > > > > > > > > > > > > > > > > > > > > > > 3. Create a Flink session job with the operator using > the > > > above > > > > > > > > image. > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 14, 2023 at 9:49 PM Gyula Fóra < > > > > > gyula.f...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi! > > > > > > > > > > > > > > > > > > > > > > > > I don't completely understand what would be a content > > of > > > such > > > > > > > CRD, > > > > > > > > > > could > > > > > > > > > > > > you give a minimal example how the Flink SQL Gateway > CR > > > yaml > > > > > > > would > > > > > > > > > look > > > > > > > > > > > > like? > > > > > > > > > > > > > > > > > > > > > > > > Adding a CRD would mean you need to add some > > > > > operator/controller > > > > > > > > > logic > > > > > > > > > > as > > > > > > > > > > > > well. Why not simply use a Deployment / StatefulSet > in > > > > > > > Kubernetes? > > > > > > > > > > > > > > > > > > > > > > > > Or a Helm chart if you want to make it more user > > > friendly? > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Gyula > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 14, 2023 at 12:57 PM Dongwoo Kim < > > > > > > > > dongwoo7....@gmail.com > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > I've been working on setting up a flink SQL gateway > > in > > > a > > > > > k8s > > > > > > > > > > > environment > > > > > > > > > > > > > and it got me thinking — what if we had a CRD for > > this? > > > > > > > > > > > > > > > > > > > > > > > > > > So I have quick questions below. > > > > > > > > > > > > > 1. Is there ongoing work to create a CRD for the > > Flink > > > SQL > > > > > > > > Gateway? > > > > > > > > > > > > > 2. If not, would the community be open to > > considering a > > > > > CRD for > > > > > > > > > this? > > > > > > > > > > > > > > > > > > > > > > > > > > I've noticed a growing demand for simplified setup > of > > > the > > > > > flink > > > > > > > > sql > > > > > > > > > > > > gateway > > > > > > > > > > > > > in flink's slack channel. > > > > > > > > > > > > > Implementing a CRD could make deployments easier > and > > > offer > > > > > > > better > > > > > > > > > > > > > integration with k8s. > > > > > > > > > > > > > > > > > > > > > > > > > > If this idea is accepted, I'm open to drafting a > FLIP > > > for > > > > > > > further > > > > > > > > > > > > > discussion > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your time and looking forward to your > > > thoughts! > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > Dongwoo > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Best > > > > > > > > > > > > ConradJam > > > > > > > > > > >
