Hi! It sounds pretty easy to deploy the gateway automatically with session cluster deployments from the operator , but there is a major limitation currently. The SQL gateway itself doesn't really support any operator integration so jobs submitted through the SQL gateway would not be manageable by the operator (they won't show up as session jobs).
Without that, this is a very strange feature. We would make something much easier for users that is not well supported by the operator in the first place. The operator is designed to manage clusters and jobs (FlinkDeployment / FlinkSessionJob). It would be good to understand if we could make the SQL Gateway create a FlinkSessionJob / Deployment (that would require application cluster support) and basically submit the job through the operator. Cheers, Gyula On Sun, Sep 17, 2023 at 1:26 AM Yangze Guo <karma...@gmail.com> wrote: > > There would be many different ways of doing this. One gateway per > session cluster, one gateway shared across different clusters... > > Currently, sql gateway cannot be shared across multiple clusters. > > > understand the tradeoff and the simplest way of accomplishing this. > > I'm not familiar with the Flink operator codebase, it would be > appreciated if you could elaborate more on the cost of adding this > feature. I agree that deploying a gateway using the native Kubernetes > Deployment can be a simple way and straightforward for users. However, > integrating it into an operator can provide additional benefits and be > more user-friendly, especially for users who are less familiar with > Kubernetes. By using an operator, users can benefit from consistent > version management with the session cluster and upgrade capabilities. > > > Best, > Yangze Guo > > On Fri, Sep 15, 2023 at 5:38 PM Gyula Fóra <gyula.f...@gmail.com> wrote: > > > > There would be many different ways of doing this. One gateway per session > > cluster, one gateway shared across different clusters... > > I would not rush to add anything anywhere until we understand the > tradeoff > > and the simplest way of accomplishing this. > > > > The operator already supports ingresses for session clusters so we could > > have a gateway sitting somewhere else simply using it. > > > > Gyula > > > > On Fri, Sep 15, 2023 at 10:18 AM Yangze Guo <karma...@gmail.com> wrote: > > > > > Thanks for bringing this up, Dongwoo. Flink SQL Gateway is also a key > > > component for OLAP scenarios. > > > > > > @Gyula > > > How about add sql gateway as an optional component to Session Cluster > > > Deployments. User can specify the resource / instance number and ports > > > of the sql gateway. I think that would help a lot for OLAP and batch > > > user. > > > > > > > > > Best, > > > Yangze Guo > > > > > > On Fri, Sep 15, 2023 at 3:19 PM ConradJam <jam.gz...@gmail.com> wrote: > > > > > > > > If we start from the crd direction, I think this mode is more like a > > > > sidecar of the session cluster, which is submitted to the session > cluster > > > > by sending sql commands to the sql gateway. I don't know if my > statement > > > is > > > > accurate. > > > > > > > > Xiaolong Wang <xiaolong.w...@smartnews.com.invalid> 于2023年9月15日周五 > > > 13:27写道： > > > > > > > > > Hi, Dongwoo, > > > > > > > > > > Since Flink SQL gateway should run upon a Flink session cluster, I > > > think > > > > > it'd be easier to add more fields to the CRD of `FlinkSessionJob`. > > > > > > > > > > e.g. > > > > > > > > > > apiVersion: flink.apache.org/v1beta1 > > > > > kind: FlinkSessionJob > > > > > metadata: > > > > > name: sql-gateway > > > > > spec: > > > > > sqlGateway: > > > > > endpoint: "hiveserver2" > > > > > mode: "streaming" > > > > > hiveConf: > > > > > configMap: > > > > > name: hive-config > > > > > items: > > > > > - key: hive-site.xml > > > > > path: hive-site.xml > > > > > > > > > > > > > > > On Fri, Sep 15, 2023 at 12:56 PM Dongwoo Kim < > dongwoo7....@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > *@Gyula* > > > > > > Thanks for the consideration Gyula. My initial idea for the CR > was > > > > > roughly > > > > > > like below. > > > > > > I focused on simplifying the setup in k8s environment, but I > agree > > > with > > > > > > your opinion that for the sql gateway > > > > > > we don't need custom operator logic to handle and most of the > > > > > requirements > > > > > > can be met by existing k8s resources. > > > > > > So maybe helm chart that bundles all resources needed should be > > > enough. > > > > > > > > > > > > apiVersion: flink.apache.org/v1beta1 > > > > > > kind: FlinkSqlGateway > > > > > > metadata: > > > > > > name: flink-sql-gateway-example > > > > > > namespace: default > > > > > > spec: > > > > > > clusterName: flink-session-cluster-example > > > > > > exposeServiceType: LoadBalancer > > > > > > flinkSqlGatewayConfiguration: > > > > > > sql-gateway.endpoint.type: "hiveserver2" > > > > > > sql-gateway.endpoint.hiveserver2.catalog.name: "hive" > > > > > > hiveConf: > > > > > > configMap: > > > > > > name: hive-config > > > > > > items: > > > > > > - key: hive-site.xml > > > > > > path: hive-site.xml > > > > > > > > > > > > > > > > > > *@xiaolong, @Shammon* > > > > > > Hi xiaolong and Shammon. > > > > > > Thanks for taking the time to share. > > > > > > I'd also like to add my experience with setting up flink sql > gateway > > > on > > > > > > k8s. > > > > > > Without building a new Docker image, I've added a separate > container > > > to > > > > > the > > > > > > existing JobManager pod and started the sql gateway using the > > > > > > "sql-gateway.sh start-foreground" command. > > > > > > I haven't explored deploying the sql gateway as an independent > > > deployment > > > > > > yet, but that's something I'm considering after modifying JM's > > > address to > > > > > > desired session cluster. > > > > > > > > > > > > Thanks all > > > > > > > > > > > > Best > > > > > > Dongwoo > > > > > > > > > > > > 2023년 9월 15일 (금) 오전 11:55, Xiaolong Wang > > > > > > <xiaolong.w...@smartnews.com.invalid>님이 작성: > > > > > > > > > > > > > Hi, Shammon, > > > > > > > > > > > > > > Yes, I want to create a Flink SQL-gateway in a job-manager. > > > > > > > > > > > > > > Currently, the above script is generally a work-around and > allows > > > me to > > > > > > > start a Flink session job manager with a SQL gateway running > upon. > > > > > > > > > > > > > > I agree that it'd be more elegant that we create a new job > type and > > > > > > write a > > > > > > > script, which is much easier for the user to use (since they > do not > > > > > need > > > > > > to > > > > > > > build a separate Flink image any more). > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 15, 2023 at 10:29 AM Shammon FY <zjur...@gmail.com > > > > > wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > Currently `sql-gateway` can be started with the script > > > > > `sql-gateway.sh` > > > > > > > in > > > > > > > > an existing node, it is more like a simple "standalone" > node. I > > > think > > > > > > > it's > > > > > > > > valuable if we can do more work to start it in k8s. > > > > > > > > > > > > > > > > For xiaolong: > > > > > > > > Do you want to start a sql-gateway instance in the jobmanager > > > pod? I > > > > > > > think > > > > > > > > maybe we need a script like `kubernetes-sql-gatewah.sh` to > start > > > > > > > > `sql-gateway` pods with a flink image, what do you think? > > > > > > > > > > > > > > > > Best, > > > > > > > > Shammon FY > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 15, 2023 at 10:02 AM Xiaolong Wang > > > > > > > > <xiaolong.w...@smartnews.com.invalid> wrote: > > > > > > > > > > > > > > > > > Hi, I've experiment this feature on K8S recently, here is > some > > > of > > > > > my > > > > > > > > trial: > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Create a new kubernetes-jobmanager.sh script with the > > > following > > > > > > > > content > > > > > > > > > > > > > > > > > > #!/usr/bin/env bash > > > > > > > > > $FLINK_HOME/bin/sql-gateway.sh start > > > > > > > > > $FLINK_HOME/bin/kubernetes-jobmanager1.sh > kubernetes-session > > > > > > > > > > > > > > > > > > 2. Build your own Flink docker image something like this > > > > > > > > > FROM flink:1.17.1-scala_2.12-java11 > > > > > > > > > > > > > > > > > > RUN mv $FLINK_HOME/bin/kubernetes-jobmanager.sh > > > $FLINK_HOME/bin/ > > > > > > > > > kubernetes-jobmanager1.sh > > > > > > > > > COPY ./kubernetes-jobmanager.sh > > > > > > > $FLINK_HOME/bin/kubernetes-jobmanager.sh > > > > > > > > > > > > > > > > > > RUN chmod +x $FLINK_HOME/bin/*.sh > > > > > > > > > USER flink > > > > > > > > > > > > > > > > > > 3. Create a Flink session job with the operator using the > above > > > > > > image. > > > > > > > > > > > > > > > > > > On Thu, Sep 14, 2023 at 9:49 PM Gyula Fóra < > > > gyula.f...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi! > > > > > > > > > > > > > > > > > > > > I don't completely understand what would be a content of > such > > > > > CRD, > > > > > > > > could > > > > > > > > > > you give a minimal example how the Flink SQL Gateway CR > yaml > > > > > would > > > > > > > look > > > > > > > > > > like? > > > > > > > > > > > > > > > > > > > > Adding a CRD would mean you need to add some > > > operator/controller > > > > > > > logic > > > > > > > > as > > > > > > > > > > well. Why not simply use a Deployment / StatefulSet in > > > > > Kubernetes? > > > > > > > > > > > > > > > > > > > > Or a Helm chart if you want to make it more user > friendly? > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Gyula > > > > > > > > > > > > > > > > > > > > On Thu, Sep 14, 2023 at 12:57 PM Dongwoo Kim < > > > > > > dongwoo7....@gmail.com > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > I've been working on setting up a flink SQL gateway in > a > > > k8s > > > > > > > > > environment > > > > > > > > > > > and it got me thinking — what if we had a CRD for this? > > > > > > > > > > > > > > > > > > > > > > So I have quick questions below. > > > > > > > > > > > 1. Is there ongoing work to create a CRD for the Flink > SQL > > > > > > Gateway? > > > > > > > > > > > 2. If not, would the community be open to considering a > > > CRD for > > > > > > > this? > > > > > > > > > > > > > > > > > > > > > > I've noticed a growing demand for simplified setup of > the > > > flink > > > > > > sql > > > > > > > > > > gateway > > > > > > > > > > > in flink's slack channel. > > > > > > > > > > > Implementing a CRD could make deployments easier and > offer > > > > > better > > > > > > > > > > > integration with k8s. > > > > > > > > > > > > > > > > > > > > > > If this idea is accepted, I'm open to drafting a FLIP > for > > > > > further > > > > > > > > > > > discussion > > > > > > > > > > > > > > > > > > > > > > Thanks for your time and looking forward to your > thoughts! > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > Dongwoo > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Best > > > > > > > > ConradJam > > > >
