There would be many different ways of doing this. One gateway per session cluster, one gateway shared across different clusters... I would not rush to add anything anywhere until we understand the tradeoff and the simplest way of accomplishing this.
The operator already supports ingresses for session clusters so we could have a gateway sitting somewhere else simply using it. Gyula On Fri, Sep 15, 2023 at 10:18 AM Yangze Guo <karma...@gmail.com> wrote: > Thanks for bringing this up, Dongwoo. Flink SQL Gateway is also a key > component for OLAP scenarios. > > @Gyula > How about add sql gateway as an optional component to Session Cluster > Deployments. User can specify the resource / instance number and ports > of the sql gateway. I think that would help a lot for OLAP and batch > user. > > > Best, > Yangze Guo > > On Fri, Sep 15, 2023 at 3:19 PM ConradJam <jam.gz...@gmail.com> wrote: > > > > If we start from the crd direction, I think this mode is more like a > > sidecar of the session cluster, which is submitted to the session cluster > > by sending sql commands to the sql gateway. I don't know if my statement > is > > accurate. > > > > Xiaolong Wang <xiaolong.w...@smartnews.com.invalid> 于2023年9月15日周五 > 13:27写道： > > > > > Hi, Dongwoo, > > > > > > Since Flink SQL gateway should run upon a Flink session cluster, I > think > > > it'd be easier to add more fields to the CRD of `FlinkSessionJob`. > > > > > > e.g. > > > > > > apiVersion: flink.apache.org/v1beta1 > > > kind: FlinkSessionJob > > > metadata: > > > name: sql-gateway > > > spec: > > > sqlGateway: > > > endpoint: "hiveserver2" > > > mode: "streaming" > > > hiveConf: > > > configMap: > > > name: hive-config > > > items: > > > - key: hive-site.xml > > > path: hive-site.xml > > > > > > > > > On Fri, Sep 15, 2023 at 12:56 PM Dongwoo Kim <dongwoo7....@gmail.com> > > > wrote: > > > > > > > Hi all, > > > > > > > > *@Gyula* > > > > Thanks for the consideration Gyula. My initial idea for the CR was > > > roughly > > > > like below. > > > > I focused on simplifying the setup in k8s environment, but I agree > with > > > > your opinion that for the sql gateway > > > > we don't need custom operator logic to handle and most of the > > > requirements > > > > can be met by existing k8s resources. > > > > So maybe helm chart that bundles all resources needed should be > enough. > > > > > > > > apiVersion: flink.apache.org/v1beta1 > > > > kind: FlinkSqlGateway > > > > metadata: > > > > name: flink-sql-gateway-example > > > > namespace: default > > > > spec: > > > > clusterName: flink-session-cluster-example > > > > exposeServiceType: LoadBalancer > > > > flinkSqlGatewayConfiguration: > > > > sql-gateway.endpoint.type: "hiveserver2" > > > > sql-gateway.endpoint.hiveserver2.catalog.name: "hive" > > > > hiveConf: > > > > configMap: > > > > name: hive-config > > > > items: > > > > - key: hive-site.xml > > > > path: hive-site.xml > > > > > > > > > > > > *@xiaolong, @Shammon* > > > > Hi xiaolong and Shammon. > > > > Thanks for taking the time to share. > > > > I'd also like to add my experience with setting up flink sql gateway > on > > > > k8s. > > > > Without building a new Docker image, I've added a separate container > to > > > the > > > > existing JobManager pod and started the sql gateway using the > > > > "sql-gateway.sh start-foreground" command. > > > > I haven't explored deploying the sql gateway as an independent > deployment > > > > yet, but that's something I'm considering after modifying JM's > address to > > > > desired session cluster. > > > > > > > > Thanks all > > > > > > > > Best > > > > Dongwoo > > > > > > > > 2023년 9월 15일 (금) 오전 11:55, Xiaolong Wang > > > > <xiaolong.w...@smartnews.com.invalid>님이 작성: > > > > > > > > > Hi, Shammon, > > > > > > > > > > Yes, I want to create a Flink SQL-gateway in a job-manager. > > > > > > > > > > Currently, the above script is generally a work-around and allows > me to > > > > > start a Flink session job manager with a SQL gateway running upon. > > > > > > > > > > I agree that it'd be more elegant that we create a new job type and > > > > write a > > > > > script, which is much easier for the user to use (since they do not > > > need > > > > to > > > > > build a separate Flink image any more). > > > > > > > > > > > > > > > > > > > > On Fri, Sep 15, 2023 at 10:29 AM Shammon FY <zjur...@gmail.com> > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > Currently `sql-gateway` can be started with the script > > > `sql-gateway.sh` > > > > > in > > > > > > an existing node, it is more like a simple "standalone" node. I > think > > > > > it's > > > > > > valuable if we can do more work to start it in k8s. > > > > > > > > > > > > For xiaolong: > > > > > > Do you want to start a sql-gateway instance in the jobmanager > pod? I > > > > > think > > > > > > maybe we need a script like `kubernetes-sql-gatewah.sh` to start > > > > > > `sql-gateway` pods with a flink image, what do you think? > > > > > > > > > > > > Best, > > > > > > Shammon FY > > > > > > > > > > > > > > > > > > On Fri, Sep 15, 2023 at 10:02 AM Xiaolong Wang > > > > > > <xiaolong.w...@smartnews.com.invalid> wrote: > > > > > > > > > > > > > Hi, I've experiment this feature on K8S recently, here is some > of > > > my > > > > > > trial: > > > > > > > > > > > > > > > > > > > > > 1. Create a new kubernetes-jobmanager.sh script with the > following > > > > > > content > > > > > > > > > > > > > > #!/usr/bin/env bash > > > > > > > $FLINK_HOME/bin/sql-gateway.sh start > > > > > > > $FLINK_HOME/bin/kubernetes-jobmanager1.sh kubernetes-session > > > > > > > > > > > > > > 2. Build your own Flink docker image something like this > > > > > > > FROM flink:1.17.1-scala_2.12-java11 > > > > > > > > > > > > > > RUN mv $FLINK_HOME/bin/kubernetes-jobmanager.sh > $FLINK_HOME/bin/ > > > > > > > kubernetes-jobmanager1.sh > > > > > > > COPY ./kubernetes-jobmanager.sh > > > > > $FLINK_HOME/bin/kubernetes-jobmanager.sh > > > > > > > > > > > > > > RUN chmod +x $FLINK_HOME/bin/*.sh > > > > > > > USER flink > > > > > > > > > > > > > > 3. Create a Flink session job with the operator using the above > > > > image. > > > > > > > > > > > > > > On Thu, Sep 14, 2023 at 9:49 PM Gyula Fóra < > gyula.f...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > > Hi! > > > > > > > > > > > > > > > > I don't completely understand what would be a content of such > > > CRD, > > > > > > could > > > > > > > > you give a minimal example how the Flink SQL Gateway CR yaml > > > would > > > > > look > > > > > > > > like? > > > > > > > > > > > > > > > > Adding a CRD would mean you need to add some > operator/controller > > > > > logic > > > > > > as > > > > > > > > well. Why not simply use a Deployment / StatefulSet in > > > Kubernetes? > > > > > > > > > > > > > > > > Or a Helm chart if you want to make it more user friendly? > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Gyula > > > > > > > > > > > > > > > > On Thu, Sep 14, 2023 at 12:57 PM Dongwoo Kim < > > > > dongwoo7....@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > I've been working on setting up a flink SQL gateway in a > k8s > > > > > > > environment > > > > > > > > > and it got me thinking — what if we had a CRD for this? > > > > > > > > > > > > > > > > > > So I have quick questions below. > > > > > > > > > 1. Is there ongoing work to create a CRD for the Flink SQL > > > > Gateway? > > > > > > > > > 2. If not, would the community be open to considering a > CRD for > > > > > this? > > > > > > > > > > > > > > > > > > I've noticed a growing demand for simplified setup of the > flink > > > > sql > > > > > > > > gateway > > > > > > > > > in flink's slack channel. > > > > > > > > > Implementing a CRD could make deployments easier and offer > > > better > > > > > > > > > integration with k8s. > > > > > > > > > > > > > > > > > > If this idea is accepted, I'm open to drafting a FLIP for > > > further > > > > > > > > > discussion > > > > > > > > > > > > > > > > > > Thanks for your time and looking forward to your thoughts! > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > Dongwoo > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Best > > > > ConradJam >
