Re: [Discuss] CRD for flink sql gateway in the flink k8s operator

[ConradJam]

If we start from the crd direction, I think this mode is more like a
sidecar of the session cluster, which is submitted to the session cluster
by sending sql commands to the sql gateway. I don't know if my statement is
accurate.

Xiaolong Wang  于2023年9月15日周五 13:27写道：

> Hi, Dongwoo,
>
> Since Flink SQL gateway should run upon a Flink session cluster, I think
> it'd be easier to add more fields to the CRD of `FlinkSessionJob`.
>
> e.g.
>
> apiVersion: flink.apache.org/v1beta1
> kind: FlinkSessionJob
> metadata:
>   name: sql-gateway
> spec:
>   sqlGateway:
> endpoint: "hiveserver2"
> mode: "streaming"
> hiveConf:
>   configMap:
> name: hive-config
> items:
>   - key: hive-site.xml
> path: hive-site.xml
>
>
> On Fri, Sep 15, 2023 at 12:56 PM Dongwoo Kim 
> wrote:
>
> > Hi all,
> >
> > *@Gyula*
> > Thanks for the consideration Gyula. My initial idea for the CR was
> roughly
> > like below.
> > I focused on simplifying the setup in k8s environment, but I agree with
> > your opinion that for the sql gateway
> > we don't need custom operator logic to handle and most of the
> requirements
> > can be met by existing k8s resources.
> > So maybe helm chart that bundles all resources needed should be enough.
> >
> > apiVersion: flink.apache.org/v1beta1
> > kind: FlinkSqlGateway
> > metadata:
> >   name: flink-sql-gateway-example
> >   namespace: default
> > spec:
> >   clusterName: flink-session-cluster-example
> >   exposeServiceType: LoadBalancer
> >   flinkSqlGatewayConfiguration:
> > sql-gateway.endpoint.type: "hiveserver2"
> > sql-gateway.endpoint.hiveserver2.catalog.name: "hive"
> >   hiveConf:
> > configMap:
> >   name: hive-config
> >   items:
> > - key: hive-site.xml
> >   path: hive-site.xml
> >
> >
> > *@xiaolong, @Shammon*
> > Hi xiaolong and Shammon.
> > Thanks for taking the time to share.
> > I'd also like to add my experience with setting up flink sql gateway on
> > k8s.
> > Without building a new Docker image, I've added a separate container to
> the
> > existing JobManager pod and started the sql gateway using the
> > "sql-gateway.sh start-foreground" command.
> > I haven't explored deploying the sql gateway as an independent deployment
> > yet, but that's something I'm considering after modifying JM's address to
> > desired session cluster.
> >
> > Thanks all
> >
> > Best
> > Dongwoo
> >
> > 2023년 9월 15일 (금) 오전 11:55, Xiaolong Wang
> > 님이 작성:
> >
> > > Hi, Shammon,
> > >
> > > Yes, I want to create a Flink SQL-gateway in a job-manager.
> > >
> > > Currently, the above script is generally a work-around and allows me to
> > > start a Flink session job manager with a SQL gateway running upon.
> > >
> > > I agree that it'd be more elegant that we create a new job type and
> > write a
> > > script, which is much easier for the user to use (since they do not
> need
> > to
> > > build a separate Flink image any more).
> > >
> > >
> > >
> > > On Fri, Sep 15, 2023 at 10:29 AM Shammon FY  wrote:
> > >
> > > > Hi,
> > > >
> > > > Currently `sql-gateway` can be started with the script
> `sql-gateway.sh`
> > > in
> > > > an existing node, it is more like a simple "standalone" node. I think
> > > it's
> > > > valuable if we can do more work to start it in k8s.
> > > >
> > > > For xiaolong:
> > > > Do you want to start a sql-gateway instance in the jobmanager pod? I
> > > think
> > > > maybe we need a script like `kubernetes-sql-gatewah.sh` to start
> > > > `sql-gateway` pods with a flink image, what do you think?
> > > >
> > > > Best,
> > > > Shammon FY
> > > >
> > > >
> > > > On Fri, Sep 15, 2023 at 10:02 AM Xiaolong Wang
> > > >  wrote:
> > > >
> > > > > Hi, I've experiment this feature on K8S recently, here is some of
> my
> > > > trial:
> > > > >
> > > > >
> > > > > 1. Create a new kubernetes-jobmanager.sh script with the following
> > > > content
> > > > >
> > > > > #!/usr/bin/env bash
> > > > > $FLINK_HOME/bin/sql-gateway.sh start
> > > > > $FLINK_HOME/bin/kubernetes-jobmanager1.sh kubernetes-session
> > > > >
> > > > > 2. Build your own Flink docker image something like this
> > > > > FROM flink:1.17.1-scala_2.12-java11
> > > > >
> > > > > RUN mv $FLINK_HOME/bin/kubernetes-jobmanager.sh $FLINK_HOME/bin/
> > > > > kubernetes-jobmanager1.sh
> > > > > COPY ./kubernetes-jobmanager.sh
> > > $FLINK_HOME/bin/kubernetes-jobmanager.sh
> > > > >
> > > > > RUN chmod +x $FLINK_HOME/bin/*.sh
> > > > > USER flink
> > > > >
> > > > > 3. Create a Flink session job with the operator using the above
> > image.
> > > > >
> > > > > On Thu, Sep 14, 2023 at 9:49 PM Gyula Fóra 
> > > wrote:
> > > > >
> > > > > > Hi!
> > > > > >
> > > > > > I don't completely understand what would be a content of such
> CRD,
> > > > could
> > > > > > you give a minimal example how the Flink SQL Gateway CR yaml
> would
> > > look
> > > > > > like?
> > > > > >
> > > > > > Adding a CRD would mean you need to add some operator/controller
> > > logic
> > > > as
> 

[jira] [Created] (FLINK-33092) Improve the resource-stabilization-timeout mechanism when rescale a job for Adaptive Scheduler

[Rui Fan (Jira)]

Rui Fan created FLINK-33092:
---

 Summary: Improve the resource-stabilization-timeout mechanism when 
rescale a job for Adaptive Scheduler
 Key: FLINK-33092
 URL: https://issues.apache.org/jira/browse/FLINK-33092
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Reporter: Rui Fan
Assignee: Rui Fan
 Attachments: image-2023-09-15-14-43-35-104.png

!image-2023-09-15-14-43-35-104.png|width=776,height=548!
h1. 1. Propose

The above is the state transition graph when rescale a job in Adaptive 
Scheduler.

In brief, when we trigger a rescale, the job will wait 
_*resource-stabilization-timeout*_ in WaitingForResources State when it has 
sufficient resources and it doesn't have the desired resource.

If the _*resource-stabilization-timeout mechanism*_ is moved into the Executing 
State, the rescale downtime will be significantly reduced.
h1. 2. Why the downtime is long?can be significantly reduced

Currently, when rescale a job:
 * The Executing will transition to Restarting
 * The Restarting will cancel this job first.
 * The Restarting will transition to WaitingForResources after the whole job is 
terminal.
 * When this job has sufficient resources and it doesn't have the desired 
resource, the WaitingForResources needs to wait  
_*resource-stabilization-timeout*_ .
 * WaitingForResources will transition to CreatingExecutionGraph after  
resource-stabilization-timeout.

The problem is the job isn't running during the resource-stabilization-timeout 
phase.
h1. 3. How to reduce the downtime?

We can move the _*resource-stabilization-timeout mechanism*_ into the Executing 
State when trigger a rescale. It means:
 * When this job has desired resources, the Executing can rescale directly.
 * When this job has sufficient resources and it doesn't have the desired 
resource, we can rescale after _*resource-stabilization-timeout.*_
 * The WaitingForResources will ignore the resource-stabilization-timeout after 
this improvement.

The resource-stabilization-timeout works before cancel job, so the rescale 
downtime will be significantly reduced.

 

Note: the resource-stabilization-timeout still works in WaitingForResources 
when start a job. It's just changed when rescale a job.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] [FLINK-32873] Add a config to allow disabling Query hints

[Martijn Visser]

Hi all,

I think Jark has a valid point with:

> Does this mean that in the future we might add an option to disable each
feature?

I don't think that's a reasonable outcome indeed, but we are currently in a
situation where we don't have clear guidelines on when to add a
configuration option, and when not to add one. From a platform perspective,
there might not be an imminent or obvious security implication, but you
want to minimize a potential attack surface.

> We should try to remove the unnecessary configuration from the list in
Flink 2.0.

I agree with that too.

With these things in mind, my proposal would be the following:

* Add a configuration option for this situation, given that we don't have
clear guidelines on when to add/not add a new config option.
* Since we want to overhaul the configuration layer anyway in Flink 2.0, we
clean-up the configuration list by not having an option for each item, but
either a generic option that allows you to disable one or more features (by
providing a list as the configuration option), or we already bundle
multiple configuration options into a specific category, e.g. so that you
can have a default Flink without any hardening, a read-only Flink, a
fully-hardened Flink etc)

Best regards,

Martijn


On Mon, Sep 11, 2023 at 4:51 PM Jim Hughes 
wrote:

> Hi Jing and Jark!
>
> I can definitely appreciate the desire to have fewer configurations.
>
> Do you have a suggested alternative for platform providers to limit or
> restrict the hints that Bonnie is talking about?
>
> As one possibility, maybe one configuration could be set to control all
> hints.
>
> Cheers,
>
> Jim
>
> On Sat, Sep 9, 2023 at 6:16 AM Jark Wu  wrote:
>
> > I agree with Jing,
> >
> > My biggest concern is this makes the boundary of adding an option very
> > unclear.
> > It's not a strong reason to add a config just because of it doesn't
> affect
> > existing
> > users. Does this mean that in the future we might add an option to
> disable
> > each feature?
> >
> > Flink already has a very long list of configurations [1][2] and this is
> > very scary
> > and not easy to use. We should try to remove the unnecessary
> configuration
> > from
> > the list in Flink 2.0. However, from my perspective, adding this option
> > makes us far
> > away from this direction.
> >
> > Best,
> > Jark
> >
> > [1]
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/config/
> > [2]
> >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/
> >
> > On Sat, 9 Sept 2023 at 17:33, Jing Ge 
> wrote:
> >
> > > Hi,
> > >
> > > Thanks for bringing this to our attention. At the first glance, it
> looks
> > > reasonable to offer a new configuration to enable/disable SQL hints
> > > globally. However, IMHO, it is not the right timing to do it now,
> because
> > > we should not only think as platform providers but also as end
> users(the
> > > number of end users are much bigger than platform providers):
> > >
> > > 1. Users don't need it because users have the choice to use hints or
> not,
> > > just like Jark pointed out. With this configuration, there will be a
> > fight
> > > between platform providers and users which will cause more confusions
> and
> > > conflicts. And users will probably win, IMHO, because they are the end
> > > customers that use Flink to create business values.
> > > 2. SQL hints could be considered as an additional feature for users to
> > > control, to optimize the execution plan without touching the internal
> > > logic, i.e. features for advanced use cases and i.e. don't use it if
> you
> > > don't understand it.
> > > 3. Before the system is smart enough to take over(where we are now,
> > > fortunately and unfortunately :-))), there should be a way for users to
> > do
> > > such tuning, even if it is a temporary phase from a
> > > long-term's perspective, i.e. just because it is a temporary solution,
> > does
> > > not mean it is not necessary for now.
> > > 4. What if users write wrong hints? Well, the code review process is
> > > recommended. Someone who truly understands hints should double check it
> > > before hints are merged to the master or submitted to the production
> env.
> > > Just like a common software development process.
> > >
> > > Just my two cents.
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Thu, Sep 7, 2023 at 10:02 PM Bonnie Arogyam Varghese
> > >  wrote:
> > >
> > > > Hi Liu,
> > > >  The default will be set to enabled which is the current behavior.
> The
> > > > option will allow users/platform providers to disable it if they want
> > to.
> > > >
> > > > On Wed, Sep 6, 2023 at 6:39 PM liu ron  wrote:
> > > >
> > > > > Hi, Boonie
> > > > >
> > > > > I'm with Jark on why disable hint is needed if it won't affect
> > > security.
> > > > If
> > > > > users don't need to use hint, then they won't care about it and I
> > don't
> > > > > think it's going to be a nuisance. On top of that, Lookup Join Hint
> > is
> > > > very
> >

Re: [Discuss] CRD for flink sql gateway in the flink k8s operator

[Yangze Guo]

Thanks for bringing this up, Dongwoo. Flink SQL Gateway is also a key
component for OLAP scenarios.

@Gyula
How about add sql gateway as an optional component to Session Cluster
Deployments. User can specify the resource / instance number and ports
of the sql gateway. I think that would help a lot for OLAP and batch
user.


Best,
Yangze Guo

On Fri, Sep 15, 2023 at 3:19 PM ConradJam  wrote:
>
> If we start from the crd direction, I think this mode is more like a
> sidecar of the session cluster, which is submitted to the session cluster
> by sending sql commands to the sql gateway. I don't know if my statement is
> accurate.
>
> Xiaolong Wang  于2023年9月15日周五 13:27写道：
>
> > Hi, Dongwoo,
> >
> > Since Flink SQL gateway should run upon a Flink session cluster, I think
> > it'd be easier to add more fields to the CRD of `FlinkSessionJob`.
> >
> > e.g.
> >
> > apiVersion: flink.apache.org/v1beta1
> > kind: FlinkSessionJob
> > metadata:
> >   name: sql-gateway
> > spec:
> >   sqlGateway:
> > endpoint: "hiveserver2"
> > mode: "streaming"
> > hiveConf:
> >   configMap:
> > name: hive-config
> > items:
> >   - key: hive-site.xml
> > path: hive-site.xml
> >
> >
> > On Fri, Sep 15, 2023 at 12:56 PM Dongwoo Kim 
> > wrote:
> >
> > > Hi all,
> > >
> > > *@Gyula*
> > > Thanks for the consideration Gyula. My initial idea for the CR was
> > roughly
> > > like below.
> > > I focused on simplifying the setup in k8s environment, but I agree with
> > > your opinion that for the sql gateway
> > > we don't need custom operator logic to handle and most of the
> > requirements
> > > can be met by existing k8s resources.
> > > So maybe helm chart that bundles all resources needed should be enough.
> > >
> > > apiVersion: flink.apache.org/v1beta1
> > > kind: FlinkSqlGateway
> > > metadata:
> > >   name: flink-sql-gateway-example
> > >   namespace: default
> > > spec:
> > >   clusterName: flink-session-cluster-example
> > >   exposeServiceType: LoadBalancer
> > >   flinkSqlGatewayConfiguration:
> > > sql-gateway.endpoint.type: "hiveserver2"
> > > sql-gateway.endpoint.hiveserver2.catalog.name: "hive"
> > >   hiveConf:
> > > configMap:
> > >   name: hive-config
> > >   items:
> > > - key: hive-site.xml
> > >   path: hive-site.xml
> > >
> > >
> > > *@xiaolong, @Shammon*
> > > Hi xiaolong and Shammon.
> > > Thanks for taking the time to share.
> > > I'd also like to add my experience with setting up flink sql gateway on
> > > k8s.
> > > Without building a new Docker image, I've added a separate container to
> > the
> > > existing JobManager pod and started the sql gateway using the
> > > "sql-gateway.sh start-foreground" command.
> > > I haven't explored deploying the sql gateway as an independent deployment
> > > yet, but that's something I'm considering after modifying JM's address to
> > > desired session cluster.
> > >
> > > Thanks all
> > >
> > > Best
> > > Dongwoo
> > >
> > > 2023년 9월 15일 (금) 오전 11:55, Xiaolong Wang
> > > 님이 작성:
> > >
> > > > Hi, Shammon,
> > > >
> > > > Yes, I want to create a Flink SQL-gateway in a job-manager.
> > > >
> > > > Currently, the above script is generally a work-around and allows me to
> > > > start a Flink session job manager with a SQL gateway running upon.
> > > >
> > > > I agree that it'd be more elegant that we create a new job type and
> > > write a
> > > > script, which is much easier for the user to use (since they do not
> > need
> > > to
> > > > build a separate Flink image any more).
> > > >
> > > >
> > > >
> > > > On Fri, Sep 15, 2023 at 10:29 AM Shammon FY  wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Currently `sql-gateway` can be started with the script
> > `sql-gateway.sh`
> > > > in
> > > > > an existing node, it is more like a simple "standalone" node. I think
> > > > it's
> > > > > valuable if we can do more work to start it in k8s.
> > > > >
> > > > > For xiaolong:
> > > > > Do you want to start a sql-gateway instance in the jobmanager pod? I
> > > > think
> > > > > maybe we need a script like `kubernetes-sql-gatewah.sh` to start
> > > > > `sql-gateway` pods with a flink image, what do you think?
> > > > >
> > > > > Best,
> > > > > Shammon FY
> > > > >
> > > > >
> > > > > On Fri, Sep 15, 2023 at 10:02 AM Xiaolong Wang
> > > > >  wrote:
> > > > >
> > > > > > Hi, I've experiment this feature on K8S recently, here is some of
> > my
> > > > > trial:
> > > > > >
> > > > > >
> > > > > > 1. Create a new kubernetes-jobmanager.sh script with the following
> > > > > content
> > > > > >
> > > > > > #!/usr/bin/env bash
> > > > > > $FLINK_HOME/bin/sql-gateway.sh start
> > > > > > $FLINK_HOME/bin/kubernetes-jobmanager1.sh kubernetes-session
> > > > > >
> > > > > > 2. Build your own Flink docker image something like this
> > > > > > FROM flink:1.17.1-scala_2.12-java11
> > > > > >
> > > > > > RUN mv $FLINK_HOME/bin/kubernetes-jobmanager.sh $FLINK_HOME/bin/
> > > > > > kubernetes-jobmanager1.sh
> > > > > > C

Re: [VOTE] FLIP-361: Improve GC Metrics

[Yangze Guo]

+1 (binding)

Best,
Yangze Guo

On Thu, Sep 14, 2023 at 5:47 PM Maximilian Michels  wrote:
>
> +1 (binding)
>
> On Thu, Sep 14, 2023 at 4:26 AM Venkatakrishnan Sowrirajan
>  wrote:
> >
> > +1 (non-binding)
> >
> > On Wed, Sep 13, 2023, 6:55 PM Matt Wang  wrote:
> >
> > > +1 (non-binding)
> > >
> > >
> > > Thanks for driving this FLIP
> > >
> > >
> > >
> > >
> > > --
> > >
> > > Best,
> > > Matt Wang
> > >
> > >
> > >  Replied Message 
> > > | From | Xintong Song |
> > > | Date | 09/14/2023 09:54 |
> > > | To |  |
> > > | Subject | Re: [VOTE] FLIP-361: Improve GC Metrics |
> > > +1 (binding)
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Thu, Sep 14, 2023 at 2:40 AM Samrat Deb  wrote:
> > >
> > > +1 ( non binding)
> > >
> > > These improved GC metrics will be a great addition.
> > >
> > > Bests,
> > > Samrat
> > >
> > > On Wed, 13 Sep 2023 at 7:58 PM, ConradJam  wrote:
> > >
> > > +1 (non-binding)
> > > gc metrics help with autoscale tuning features
> > >
> > > Chen Zhanghao  于2023年9月13日周三 22:16写道：
> > >
> > > +1 (unbinding). Looking forward to it
> > >
> > > Best,
> > > Zhanghao Chen
> > > 
> > > 发件人: Gyula Fóra 
> > > 发送时间: 2023年9月13日 21:16
> > > 收件人: dev 
> > > 主题: [VOTE] FLIP-361: Improve GC Metrics
> > >
> > > Hi All!
> > >
> > > Thanks for all the feedback on FLIP-361: Improve GC Metrics [1][2]
> > >
> > > I'd like to start a vote for it. The vote will be open for at least 72
> > > hours unless there is an objection or insufficient votes.
> > >
> > > Cheers,
> > > Gyula
> > >
> > > [1]
> > >
> > >
> > >
> > >
> > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-361*3A*Improve*GC*Metrics__;JSsrKw!!IKRxdwAv5BmarQ!dpHSOqsSHlPJ5gCvZ2yxSGjcR4xA2N-mpGZ1w2jPuKb78aujNpbzENmi1e7B26d6v4UQ8bQZ7IQaUcI$
> > > [2]
> > > https://urldefense.com/v3/__https://lists.apache.org/thread/qqqv54vyr4gbp63wm2d12q78m8h95xb2__;!!IKRxdwAv5BmarQ!dpHSOqsSHlPJ5gCvZ2yxSGjcR4xA2N-mpGZ1w2jPuKb78aujNpbzENmi1e7B26d6v4UQ8bQZFdEMnAg$
> > >
> > >
> > >
> > > --
> > > Best
> > >
> > > ConradJam
> > >
> > >
> > >

Re: [DISCUSS] [FLINK-32873] Add a config to allow disabling Query hints

[Jark Wu]

Hi Martijn,

I can understand that.
Is there any database/system that supports disabling/enabling query hints
 with a configuration? This might help us to understand the use case,
and follow the approach.

Best,
Jark

On Fri, 15 Sept 2023 at 15:34, Martijn Visser 
wrote:

> Hi all,
>
> I think Jark has a valid point with:
>
> > Does this mean that in the future we might add an option to disable each
> feature?
>
> I don't think that's a reasonable outcome indeed, but we are currently in a
> situation where we don't have clear guidelines on when to add a
> configuration option, and when not to add one. From a platform perspective,
> there might not be an imminent or obvious security implication, but you
> want to minimize a potential attack surface.
>
> > We should try to remove the unnecessary configuration from the list in
> Flink 2.0.
>
> I agree with that too.
>
> With these things in mind, my proposal would be the following:
>
> * Add a configuration option for this situation, given that we don't have
> clear guidelines on when to add/not add a new config option.
> * Since we want to overhaul the configuration layer anyway in Flink 2.0, we
> clean-up the configuration list by not having an option for each item, but
> either a generic option that allows you to disable one or more features (by
> providing a list as the configuration option), or we already bundle
> multiple configuration options into a specific category, e.g. so that you
> can have a default Flink without any hardening, a read-only Flink, a
> fully-hardened Flink etc)
>
> Best regards,
>
> Martijn
>
>
> On Mon, Sep 11, 2023 at 4:51 PM Jim Hughes 
> wrote:
>
> > Hi Jing and Jark!
> >
> > I can definitely appreciate the desire to have fewer configurations.
> >
> > Do you have a suggested alternative for platform providers to limit or
> > restrict the hints that Bonnie is talking about?
> >
> > As one possibility, maybe one configuration could be set to control all
> > hints.
> >
> > Cheers,
> >
> > Jim
> >
> > On Sat, Sep 9, 2023 at 6:16 AM Jark Wu  wrote:
> >
> > > I agree with Jing,
> > >
> > > My biggest concern is this makes the boundary of adding an option very
> > > unclear.
> > > It's not a strong reason to add a config just because of it doesn't
> > affect
> > > existing
> > > users. Does this mean that in the future we might add an option to
> > disable
> > > each feature?
> > >
> > > Flink already has a very long list of configurations [1][2] and this is
> > > very scary
> > > and not easy to use. We should try to remove the unnecessary
> > configuration
> > > from
> > > the list in Flink 2.0. However, from my perspective, adding this option
> > > makes us far
> > > away from this direction.
> > >
> > > Best,
> > > Jark
> > >
> > > [1]
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/config/
> > > [2]
> > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/
> > >
> > > On Sat, 9 Sept 2023 at 17:33, Jing Ge 
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > Thanks for bringing this to our attention. At the first glance, it
> > looks
> > > > reasonable to offer a new configuration to enable/disable SQL hints
> > > > globally. However, IMHO, it is not the right timing to do it now,
> > because
> > > > we should not only think as platform providers but also as end
> > users(the
> > > > number of end users are much bigger than platform providers):
> > > >
> > > > 1. Users don't need it because users have the choice to use hints or
> > not,
> > > > just like Jark pointed out. With this configuration, there will be a
> > > fight
> > > > between platform providers and users which will cause more confusions
> > and
> > > > conflicts. And users will probably win, IMHO, because they are the
> end
> > > > customers that use Flink to create business values.
> > > > 2. SQL hints could be considered as an additional feature for users
> to
> > > > control, to optimize the execution plan without touching the internal
> > > > logic, i.e. features for advanced use cases and i.e. don't use it if
> > you
> > > > don't understand it.
> > > > 3. Before the system is smart enough to take over(where we are now,
> > > > fortunately and unfortunately :-))), there should be a way for users
> to
> > > do
> > > > such tuning, even if it is a temporary phase from a
> > > > long-term's perspective, i.e. just because it is a temporary
> solution,
> > > does
> > > > not mean it is not necessary for now.
> > > > 4. What if users write wrong hints? Well, the code review process is
> > > > recommended. Someone who truly understands hints should double check
> it
> > > > before hints are merged to the master or submitted to the production
> > env.
> > > > Just like a common software development process.
> > > >
> > > > Just my two cents.
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Thu, Sep 7, 2023 at 10:02 PM Bonnie Arogyam Varghese
> > > >  wrote:
> > > >
> > > > > Hi Liu,
> > > > >  Th

Re: [DISCUSS] FLIP-363: Unify the Representation of TaskManager Location in REST API and Web UI

[Yangze Guo]

Thanks for driving this, Zhanghao. +1 for the overall proposal.

Some cents from my side:

1. Since most of the rest api get the location from
TaskManagerLocation, we can align the string formatter in this class.
E.g. add something like toHumanRealableString to this class.

2. From my understanding of FLIP-321, if we want to deprecate the host
field, we should annotate the related field / getter / setter with
@Deprecated in this version, and keep them at least 2 minor releases.

Best,
Yangze Guo

On Wed, Sep 13, 2023 at 8:52 PM Chen Zhanghao  wrote:
>
> Hi Jing,
>
> Thanks for the suggestion. Endpoint is indeed a more professional word in the 
> networking world but I think location is more suited here for two reasons:
>
>   1.  The term here is for uniquely identifying the TaskManager where the 
> task is deployed while providing the host machine info as well to help 
> identify taskmanager- and host-aggregative problems. So strictly speaking, it 
> is not used in a pure networking context.
>   2.  The term "location" is already used widely in the codebase, e.g. 
> TaskManagerLocation and JobExceptions-related classes.
>
> WDYT?
>
> Best,
> Zhanghao Chen
> 
> 发件人: Jing Ge 
> 发送时间: 2023年9月13日 4:52
> 收件人: dev@flink.apache.org 
> 主题: Re: [DISCUSS] FLIP-363: Unify the Representation of TaskManager Location 
> in REST API and Web UI
>
> Hi Zhanghao,
>
> Thanks for bringing this to our attention. It is a good proposal to improve
> data consistency.
>
> Speaking of naming conventions of choosing location over host, how about
> "endpoint" with the following thoughts:
>
> 1. endpoint is a more professional word than location in the network
> context.
> 2. I know commonly endpoints mean the URLs of services. Using Hostname:port
> as the endpoint follows exactly the same rule, because TaskManager is the
> top level service that aligns with the top level endpoint.
>
> WDYT?
>
> Best regards,
> Jing
>
>
> On Mon, Sep 11, 2023 at 6:01 AM Weihua Hu  wrote:
>
> > Hi, Zhanghao
> >
> > Since the meaning of "host" is not aligned, it seems good for me to remove
> > it in the future release.
> >
> > Best,
> > Weihua
> >
> >
> > On Mon, Sep 11, 2023 at 11:48 AM Chen Zhanghao 
> > wrote:
> >
> > > Hi Fan Rui,
> > >
> > > Thanks for clarifying the definition of "public interfaces", that helps a
> > > lot!
> > >
> > > Best,
> > > Zhanghao Chen
> > > 
> > > 发件人: Rui Fan <1996fan...@gmail.com>
> > > 发送时间: 2023年9月11日 11:18
> > > 收件人: dev@flink.apache.org 
> > > 主题: Re: [DISCUSS] FLIP-363: Unify the Representation of TaskManager
> > > Location in REST API and Web UI
> > >
> > > Thanks Zhanghao driving this FLIP, adding the port in Web UI
> > > seems good to me.
> > >
> > > Hi Shammon and Zhanghao,
> > >
> > > I would like to clarify the difference between Public Interfaces
> > > in FLIP and @Public in code.
> > >
> > > As I understand, the `Public Interfaces in FLIP` means these
> > > changes will be used in user side, such as: @Public class,
> > > Configuration settings, User-facing scripts/command-line tools,
> > > and rest api, etc.
> > >
> > > You can refer to  "What are the "public interfaces" of the project?"
> > > part in Flink Improvement Proposals doc[1].
> > >
> > > @Public class means the user will use this class directly, and
> > > these rest classes won't be depended on directly. So I think
> > > these classes related to rest don't need to be marked @Public.
> > >
> > > Please correct me if anything is wrong, thanks~
> > >
> > > [1]
> > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > >
> > > Best,
> > > Rui
> > >
> > > On Mon, Sep 11, 2023 at 11:09 AM Weihua Hu 
> > wrote:
> > >
> > > > Hi, Zhanghao
> > > >
> > > > Thanks for bringing this proposal.
> > > >
> > > > I have a concern:
> > > >
> > > > I prefer to keep the "host" field and add a "location" field in future
> > > > versions.
> > > > Consider a scenario where a machine (host) with multiple TaskManagers
> > has
> > > > poor processing performance due to some problems.
> > > > By using a host field aggregation, I can identify the problems with
> > this
> > > > machine and take it offline.
> > > >
> > > > Best,
> > > > Weihua
> > > >
> > > >
> > > > On Mon, Sep 11, 2023 at 10:34 AM Chen Zhanghao <
> > > zhanghao.c...@outlook.com>
> > > > wrote:
> > > >
> > > > > Hi Shammon,
> > > > >
> > > > > I think all REST API response messages (e.g.
> > > > > SubtaskExecutionAttemptDetailsInfo) should be considered as part of
> > the
> > > > > public APIs and therefore be marked as @Public. It is true though
> > none
> > > of
> > > > > them are marked as @public yet. Maybe we should do that. ccing
> > > > > @chesnay for confirmation.
> > > > >
> > > > > Best,
> > > > > Zhanghao Chen
> > > > > 
> > > > > 发件人: Shammon FY 
> > > > > 发送时间: 2023年9月11日 10:22
> > > > > 收件人: dev@flink.apache.org 
> > > > > 主题: Re: [DI

Re: [DISCUSS] Flink annotation strategy/consensus

[Chesnay Schepler]

We need the @Internal annotation for cases where a user-facing class has 
methods entirely meant for internal uses.
It's not great that we have these cases to being with, but fixing that 
is another story.


The semantics for non-annotated classes is already documented.
https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/upgrading/#api-compatibility-guarantees

On 14/09/2023 02:31, Becket Qin wrote:

Does it make sense to clearly define that APIs without annotation are
internal APIs and should be used outside of Flink. And deprecate @Internal?


We can do this. Although I think it is OK to keep the @Internal annotation
in case extra clarity is needed sometimes.

Thanks,

Jiangjie (Becket) Qin

On Tue, Sep 12, 2023 at 7:11 PM Jing Ge  wrote:


Hi Becket,

Thanks for your reply with details.



2. I agree it would be too verbose to annotate every internal method /
class / interface. Currently we already treat the methods / interfaces /
classes without annotations as effectively @Internal.


Does it make sense to clearly define that APIs without annotation are
internal APIs and should be used outside of Flink. And deprecate @Internal?

Best regards,
Jing

On Mon, Sep 11, 2023 at 5:05 AM Becket Qin  wrote:


Hi Jing,

Thanks for bringing up the discussion. My two cents:

1. All the public methods / classes / interfaces MUST be annotated with

one

of the @Experimental / @PublicEvolving / @Public. In practice, all the
methods by default inherit the annotation from the containing class,

unless

annotated otherwise. e.g. an @Internal method in a @Public class.


2. I agree it would be too verbose to annotate every internal method /

class / interface. Currently we already treat the methods / interfaces /
classes without annotations as effectively @Internal.



3. Per our discussion in the other thread, @Deprecated SHOULD coexist with

one of the @Experimental / @PublicEvolving / @Public. In that
case, @Deprecated overrides the other annotation, which means that public
API will not evolve and will be removed according to the deprecation
process.

4. The internal methods / classes / interfaces SHOULD NOT be marked as
deprecated. Instead, an immediate refactor should be done to remove the
"deprecated" internal methods / classes / interfaces, and migrate the

code

to its successor. Otherwise, technical debts will build up.

Thanks,

Jiangjie (Becket) Qin



On Sat, Sep 9, 2023 at 5:29 AM Jing Ge 

wrote:

Hi devs,

While I was joining the flink-avro enhancement and cleanup discussion
driven by Becket[1], I realized that there are some issues with the

current

Flink API annotation usage in the source code.

As far as I am concerned, Flink wants to control the access/visibility

of

APIs across modules and for downstreams. Since no OSGI is used(it

should

not be used because of its complexity, IMHO), Flink decided to use a

very

lightweight but manual solution: customized annotation like @Internal,
@Experimental, @PublicEvolving,
etc. This is a Flink only concept on top of JDK annotation and is

therefore

orthogonal to @Deprecated or any other annotations offered by JDK.

After

this concept has been used, APIs without one of these annotations are

in

the kind of gray area which means they have no contract in the context

of

this new concept. Without any given metadata they could be considered
as @Internal or @Experimental, because changes are allowed to be

applied

at

any time. But there is no clear definition and therefore different

people

will understand it differently.

There are two options to improve it, as far as I could figure out:

option 1: All APIs must have one of those annotations. We should put

some

effort into going through all source code and add missing annotations.
There were discussions[2] and activities going in this direction.
option 2: the community comes to a new consensus that APIs without
annotation equals one of @Internal, @Experimental, or @PublicEvolving.

I

personally will choose @Internal, because it is the safest one. And if
@Internal is chosen as the default one, it could also be deprecated,
because no annotation equals @Internal. If it makes sense, I can

create a

FLIP and help the community reach this consensus.

Both options have their own pros and cons. I would choose option 2,

since

we will not end up with a lot of APIs marked as @Internal.

Looking forward to hearing your thoughts.

Best regards
Jing


[1] https://lists.apache.org/thread/7zsv528swbjxo5zk0bxq33hrkvd77d6f
[2] https://lists.apache.org/thread/zl2rmodsjsdb49tt4hn6wv3gdwo0m31o

Re: [Discuss] CRD for flink sql gateway in the flink k8s operator

[Gyula Fóra]

There would be many different ways of doing this. One gateway per session
cluster, one gateway shared across different clusters...
I would not rush to add anything anywhere until we understand the tradeoff
and the simplest way of accomplishing this.

The operator already supports ingresses for session clusters so we could
have a gateway sitting somewhere else simply using it.

Gyula

On Fri, Sep 15, 2023 at 10:18 AM Yangze Guo  wrote:

> Thanks for bringing this up, Dongwoo. Flink SQL Gateway is also a key
> component for OLAP scenarios.
>
> @Gyula
> How about add sql gateway as an optional component to Session Cluster
> Deployments. User can specify the resource / instance number and ports
> of the sql gateway. I think that would help a lot for OLAP and batch
> user.
>
>
> Best,
> Yangze Guo
>
> On Fri, Sep 15, 2023 at 3:19 PM ConradJam  wrote:
> >
> > If we start from the crd direction, I think this mode is more like a
> > sidecar of the session cluster, which is submitted to the session cluster
> > by sending sql commands to the sql gateway. I don't know if my statement
> is
> > accurate.
> >
> > Xiaolong Wang  于2023年9月15日周五
> 13:27写道：
> >
> > > Hi, Dongwoo,
> > >
> > > Since Flink SQL gateway should run upon a Flink session cluster, I
> think
> > > it'd be easier to add more fields to the CRD of `FlinkSessionJob`.
> > >
> > > e.g.
> > >
> > > apiVersion: flink.apache.org/v1beta1
> > > kind: FlinkSessionJob
> > > metadata:
> > >   name: sql-gateway
> > > spec:
> > >   sqlGateway:
> > > endpoint: "hiveserver2"
> > > mode: "streaming"
> > > hiveConf:
> > >   configMap:
> > > name: hive-config
> > > items:
> > >   - key: hive-site.xml
> > > path: hive-site.xml
> > >
> > >
> > > On Fri, Sep 15, 2023 at 12:56 PM Dongwoo Kim 
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > *@Gyula*
> > > > Thanks for the consideration Gyula. My initial idea for the CR was
> > > roughly
> > > > like below.
> > > > I focused on simplifying the setup in k8s environment, but I agree
> with
> > > > your opinion that for the sql gateway
> > > > we don't need custom operator logic to handle and most of the
> > > requirements
> > > > can be met by existing k8s resources.
> > > > So maybe helm chart that bundles all resources needed should be
> enough.
> > > >
> > > > apiVersion: flink.apache.org/v1beta1
> > > > kind: FlinkSqlGateway
> > > > metadata:
> > > >   name: flink-sql-gateway-example
> > > >   namespace: default
> > > > spec:
> > > >   clusterName: flink-session-cluster-example
> > > >   exposeServiceType: LoadBalancer
> > > >   flinkSqlGatewayConfiguration:
> > > > sql-gateway.endpoint.type: "hiveserver2"
> > > > sql-gateway.endpoint.hiveserver2.catalog.name: "hive"
> > > >   hiveConf:
> > > > configMap:
> > > >   name: hive-config
> > > >   items:
> > > > - key: hive-site.xml
> > > >   path: hive-site.xml
> > > >
> > > >
> > > > *@xiaolong, @Shammon*
> > > > Hi xiaolong and Shammon.
> > > > Thanks for taking the time to share.
> > > > I'd also like to add my experience with setting up flink sql gateway
> on
> > > > k8s.
> > > > Without building a new Docker image, I've added a separate container
> to
> > > the
> > > > existing JobManager pod and started the sql gateway using the
> > > > "sql-gateway.sh start-foreground" command.
> > > > I haven't explored deploying the sql gateway as an independent
> deployment
> > > > yet, but that's something I'm considering after modifying JM's
> address to
> > > > desired session cluster.
> > > >
> > > > Thanks all
> > > >
> > > > Best
> > > > Dongwoo
> > > >
> > > > 2023년 9월 15일 (금) 오전 11:55, Xiaolong Wang
> > > > 님이 작성:
> > > >
> > > > > Hi, Shammon,
> > > > >
> > > > > Yes, I want to create a Flink SQL-gateway in a job-manager.
> > > > >
> > > > > Currently, the above script is generally a work-around and allows
> me to
> > > > > start a Flink session job manager with a SQL gateway running upon.
> > > > >
> > > > > I agree that it'd be more elegant that we create a new job type and
> > > > write a
> > > > > script, which is much easier for the user to use (since they do not
> > > need
> > > > to
> > > > > build a separate Flink image any more).
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Sep 15, 2023 at 10:29 AM Shammon FY 
> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Currently `sql-gateway` can be started with the script
> > > `sql-gateway.sh`
> > > > > in
> > > > > > an existing node, it is more like a simple "standalone" node. I
> think
> > > > > it's
> > > > > > valuable if we can do more work to start it in k8s.
> > > > > >
> > > > > > For xiaolong:
> > > > > > Do you want to start a sql-gateway instance in the jobmanager
> pod? I
> > > > > think
> > > > > > maybe we need a script like `kubernetes-sql-gatewah.sh` to start
> > > > > > `sql-gateway` pods with a flink image, what do you think?
> > > > > >
> > > > > > Best,
> > > > > > Shammon FY
> > > > > >
> > > > > >
> > >

Re: [DISCUSS] [FLINK-32873] Add a config to allow disabling Query hints

[Martijn Visser]

Hi Jark,

Oracle has/had a generic "OPTIMIZER_IGNORE_HINTS" [1]. It looks like MSSQL
has configuration options to disable specific hints [2] instead of a
generic solution.

[1]
https://docs.oracle.com/en/database/oracle/oracle-database/23/refrn/OPTIMIZER_IGNORE_HINTS.html#GUID-D62CA6D8-D0D8-4A20-93EA-EEB4B3144347
[2]
https://www.mssqltips.com/sqlservertip/4175/disabling-sql-server-optimizer-rules-with-queryruleoff/

Best regards,

Martijn

On Fri, Sep 15, 2023 at 10:53 AM Jark Wu  wrote:

> Hi Martijn,
>
> I can understand that.
> Is there any database/system that supports disabling/enabling query hints
>  with a configuration? This might help us to understand the use case,
> and follow the approach.
>
> Best,
> Jark
>
> On Fri, 15 Sept 2023 at 15:34, Martijn Visser 
> wrote:
>
> > Hi all,
> >
> > I think Jark has a valid point with:
> >
> > > Does this mean that in the future we might add an option to disable
> each
> > feature?
> >
> > I don't think that's a reasonable outcome indeed, but we are currently
> in a
> > situation where we don't have clear guidelines on when to add a
> > configuration option, and when not to add one. From a platform
> perspective,
> > there might not be an imminent or obvious security implication, but you
> > want to minimize a potential attack surface.
> >
> > > We should try to remove the unnecessary configuration from the list in
> > Flink 2.0.
> >
> > I agree with that too.
> >
> > With these things in mind, my proposal would be the following:
> >
> > * Add a configuration option for this situation, given that we don't have
> > clear guidelines on when to add/not add a new config option.
> > * Since we want to overhaul the configuration layer anyway in Flink 2.0,
> we
> > clean-up the configuration list by not having an option for each item,
> but
> > either a generic option that allows you to disable one or more features
> (by
> > providing a list as the configuration option), or we already bundle
> > multiple configuration options into a specific category, e.g. so that you
> > can have a default Flink without any hardening, a read-only Flink, a
> > fully-hardened Flink etc)
> >
> > Best regards,
> >
> > Martijn
> >
> >
> > On Mon, Sep 11, 2023 at 4:51 PM Jim Hughes  >
> > wrote:
> >
> > > Hi Jing and Jark!
> > >
> > > I can definitely appreciate the desire to have fewer configurations.
> > >
> > > Do you have a suggested alternative for platform providers to limit or
> > > restrict the hints that Bonnie is talking about?
> > >
> > > As one possibility, maybe one configuration could be set to control all
> > > hints.
> > >
> > > Cheers,
> > >
> > > Jim
> > >
> > > On Sat, Sep 9, 2023 at 6:16 AM Jark Wu  wrote:
> > >
> > > > I agree with Jing,
> > > >
> > > > My biggest concern is this makes the boundary of adding an option
> very
> > > > unclear.
> > > > It's not a strong reason to add a config just because of it doesn't
> > > affect
> > > > existing
> > > > users. Does this mean that in the future we might add an option to
> > > disable
> > > > each feature?
> > > >
> > > > Flink already has a very long list of configurations [1][2] and this
> is
> > > > very scary
> > > > and not easy to use. We should try to remove the unnecessary
> > > configuration
> > > > from
> > > > the list in Flink 2.0. However, from my perspective, adding this
> option
> > > > makes us far
> > > > away from this direction.
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > > [1]
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/config/
> > > > [2]
> > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/
> > > >
> > > > On Sat, 9 Sept 2023 at 17:33, Jing Ge 
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Thanks for bringing this to our attention. At the first glance, it
> > > looks
> > > > > reasonable to offer a new configuration to enable/disable SQL hints
> > > > > globally. However, IMHO, it is not the right timing to do it now,
> > > because
> > > > > we should not only think as platform providers but also as end
> > > users(the
> > > > > number of end users are much bigger than platform providers):
> > > > >
> > > > > 1. Users don't need it because users have the choice to use hints
> or
> > > not,
> > > > > just like Jark pointed out. With this configuration, there will be
> a
> > > > fight
> > > > > between platform providers and users which will cause more
> confusions
> > > and
> > > > > conflicts. And users will probably win, IMHO, because they are the
> > end
> > > > > customers that use Flink to create business values.
> > > > > 2. SQL hints could be considered as an additional feature for users
> > to
> > > > > control, to optimize the execution plan without touching the
> internal
> > > > > logic, i.e. features for advanced use cases and i.e. don't use it
> if
> > > you
> > > > > don't understand it.
> > > > > 3. Before the system is smart enough to take over(where we are now,
> > > > > fo

RE: Inconsistent build Flink from source in readme and docs

[David Radley]

Hi Sergey,
Thanks for your response.
To summarise:
Flink 1.17 needs Maven 3.25 or 3.3.x with shading
Flink current master and  1.18 when released will require 3.8.6 (recommended or 
higher if it works).

I suggest that for the 1.18 build we include, we police the Maven version 
number in the build.
I can code this in the pom file using a maven-enforcer-plugin to police a 
minimum version of Maven 3.8.6 in master. For an example see:
https://github.com/odpi/egeria/blob/egeria-release-3.2/pom.xml

We could police java levels as well. I could put out a warning if built with 
java 8 indicating it I deprecated. Error for java levels less than 8.

If this ok I will raise a Jira to add this Maven and potentially java checking 
to the build,
   Kind regards, David.


From: Sergey Nuyanzin 
Date: Thursday, 14 September 2023 at 16:07
To: dev@flink.apache.org 
Subject: [EXTERNAL] Re: Inconsistent build Flink from source in readme and docs
Hi David

The links you have mentioned seems correct

Movement[1] towards to maven 3.8.6 happened since 1.18.x, and this is
mentioned in doc  for master branch

At the same time 1.17.x still requires 3.3.x as mentioned in the doc for
1.17 you've mentioned.

so it depends on the Flink version

[1] https://lists.apache.org/thread/jbw3lzzoq5w16ckco3fc9xokycs3f22x

On Thu, Sep 14, 2023 at 4:09 PM David Radley 
wrote:

> Hello,
> I am looking to build Flink from source I notice that the documentation
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/flinkdev/building/
> says
>
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/flinkdev/building/
>
> In addition you need Maven 3 and a JDK (Java Development Kit). Flink
> requires Java 8 (deprecated) or Java 11 to build.
>
> NOTE: Maven 3.3.x can build Flink, but will not properly shade away
> certain dependencies. Maven 3.2.5 creates the libraries properly.
> The README.md https://github.com/apache/flink/blob/master/README.md  says:
>
>
>   *   Maven (we recommend version 3.8.6 and require at least 3.1.1)
>   *   Java 8 or 11 (Java 9 or 10 may work)
> git clone https://github.com/apache/flink.git
> cd flink
> ./mvnw clean package -DskipTests # this will take up to 10 minutes
> Flink is now installed in build-target.
> NOTE: Maven 3.3.x can build Flink, but will not properly shade away
> certain dependencies. Maven 3.1.1 creates the libraries properly. To build
> unit tests with Java 8, use Java 8u51 or above to prevent failures in unit
> tests that use the PowerMock runner.
>
> Can you confirm that the readme is correct and whether there is a need to
> do the documented dependency on versions after Maven 3.3.*
>
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/flinkdev/building/#dependency-shading
>
> The recommended Maven level is 3.8.6, is this what was tested or is there
> a reason not to use the latest Maven version.
>
> I am happy to amend the docs to make them consistent,
>
> Kind regards, David
>
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>


--
Best regards,
Sergey

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

FW: RE: Inconsistent build Flink from source in readme and docs

[David Radley]

Hi again,
I have just checked and you already enforcer have the enforcer for this in 
master. I guess the only extra piece would be to put out a warning for java 8 
indicating it is deprecated – I could look at that in a Jira,
   Kind regards, David.

From: David Radley 
Date: Friday, 15 September 2023 at 11:44
To: dev@flink.apache.org 
Subject: [EXTERNAL] RE: Inconsistent build Flink from source in readme and docs
Hi Sergey,
Thanks for your response.
To summarise:
Flink 1.17 needs Maven 3.25 or 3.3.x with shading
Flink current master and  1.18 when released will require 3.8.6 (recommended or 
higher if it works).

I suggest that for the 1.18 build we include, we police the Maven version 
number in the build.
I can code this in the pom file using a maven-enforcer-plugin to police a 
minimum version of Maven 3.8.6 in master. For an example see:
https://github.com/odpi/egeria/blob/egeria-release-3.2/pom.xml

We could police java levels as well. I could put out a warning if built with 
java 8 indicating it I deprecated. Error for java levels less than 8.

If this ok I will raise a Jira to add this Maven and potentially java checking 
to the build,
   Kind regards, David.


From: Sergey Nuyanzin 
Date: Thursday, 14 September 2023 at 16:07
To: dev@flink.apache.org 
Subject: [EXTERNAL] Re: Inconsistent build Flink from source in readme and docs
Hi David

The links you have mentioned seems correct

Movement[1] towards to maven 3.8.6 happened since 1.18.x, and this is
mentioned in doc  for master branch

At the same time 1.17.x still requires 3.3.x as mentioned in the doc for
1.17 you've mentioned.

so it depends on the Flink version

[1] https://lists.apache.org/thread/jbw3lzzoq5w16ckco3fc9xokycs3f22x

On Thu, Sep 14, 2023 at 4:09 PM David Radley 
wrote:

> Hello,
> I am looking to build Flink from source I notice that the documentation
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/flinkdev/building/
> says
>
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/flinkdev/building/
>
> In addition you need Maven 3 and a JDK (Java Development Kit). Flink
> requires Java 8 (deprecated) or Java 11 to build.
>
> NOTE: Maven 3.3.x can build Flink, but will not properly shade away
> certain dependencies. Maven 3.2.5 creates the libraries properly.
> The README.md https://github.com/apache/flink/blob/master/README.md   says:
>
>
>   *   Maven (we recommend version 3.8.6 and require at least 3.1.1)
>   *   Java 8 or 11 (Java 9 or 10 may work)
> git clone https://github.com/apache/flink.git
> cd flink
> ./mvnw clean package -DskipTests # this will take up to 10 minutes
> Flink is now installed in build-target.
> NOTE: Maven 3.3.x can build Flink, but will not properly shade away
> certain dependencies. Maven 3.1.1 creates the libraries properly. To build
> unit tests with Java 8, use Java 8u51 or above to prevent failures in unit
> tests that use the PowerMock runner.
>
> Can you confirm that the readme is correct and whether there is a need to
> do the documented dependency on versions after Maven 3.3.*
>
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/flinkdev/building/#dependency-shading
>
> The recommended Maven level is 3.8.6, is this what was tested or is there
> a reason not to use the latest Maven version.
>
> I am happy to amend the docs to make them consistent,
>
> Kind regards, David
>
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>


--
Best regards,
Sergey

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Re: [DISCUSS] [FLINK-32873] Add a config to allow disabling Query hints

[Jark Wu]

Hi Martijn,

Thanks for the investigation. I found the blog[1] shows a use case
that they use "OPTIMIZER_IGNORE_HINTS" to check if embedded
hints can be removed in legacy code. I think this is a useful tool to
improve queries without complex hints strewn throughout the code.
Therefore, I'm fine to support this now.

Maybe we can follow Oracle to name the config
"table.optimizer.ignore-query-hints=false"?

Best,
Jark

[1]: https://dbaora.com/optimizer_ignore_hints-oracle-database-18c/

On Fri, 15 Sept 2023 at 17:57, Martijn Visser 
wrote:

> Hi Jark,
>
> Oracle has/had a generic "OPTIMIZER_IGNORE_HINTS" [1]. It looks like MSSQL
> has configuration options to disable specific hints [2] instead of a
> generic solution.
>
> [1]
>
> https://docs.oracle.com/en/database/oracle/oracle-database/23/refrn/OPTIMIZER_IGNORE_HINTS.html#GUID-D62CA6D8-D0D8-4A20-93EA-EEB4B3144347
> [2]
>
> https://www.mssqltips.com/sqlservertip/4175/disabling-sql-server-optimizer-rules-with-queryruleoff/
>
> Best regards,
>
> Martijn
>
> On Fri, Sep 15, 2023 at 10:53 AM Jark Wu  wrote:
>
> > Hi Martijn,
> >
> > I can understand that.
> > Is there any database/system that supports disabling/enabling query hints
> >  with a configuration? This might help us to understand the use case,
> > and follow the approach.
> >
> > Best,
> > Jark
> >
> > On Fri, 15 Sept 2023 at 15:34, Martijn Visser 
> > wrote:
> >
> > > Hi all,
> > >
> > > I think Jark has a valid point with:
> > >
> > > > Does this mean that in the future we might add an option to disable
> > each
> > > feature?
> > >
> > > I don't think that's a reasonable outcome indeed, but we are currently
> > in a
> > > situation where we don't have clear guidelines on when to add a
> > > configuration option, and when not to add one. From a platform
> > perspective,
> > > there might not be an imminent or obvious security implication, but you
> > > want to minimize a potential attack surface.
> > >
> > > > We should try to remove the unnecessary configuration from the list
> in
> > > Flink 2.0.
> > >
> > > I agree with that too.
> > >
> > > With these things in mind, my proposal would be the following:
> > >
> > > * Add a configuration option for this situation, given that we don't
> have
> > > clear guidelines on when to add/not add a new config option.
> > > * Since we want to overhaul the configuration layer anyway in Flink
> 2.0,
> > we
> > > clean-up the configuration list by not having an option for each item,
> > but
> > > either a generic option that allows you to disable one or more features
> > (by
> > > providing a list as the configuration option), or we already bundle
> > > multiple configuration options into a specific category, e.g. so that
> you
> > > can have a default Flink without any hardening, a read-only Flink, a
> > > fully-hardened Flink etc)
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > >
> > > On Mon, Sep 11, 2023 at 4:51 PM Jim Hughes
>  > >
> > > wrote:
> > >
> > > > Hi Jing and Jark!
> > > >
> > > > I can definitely appreciate the desire to have fewer configurations.
> > > >
> > > > Do you have a suggested alternative for platform providers to limit
> or
> > > > restrict the hints that Bonnie is talking about?
> > > >
> > > > As one possibility, maybe one configuration could be set to control
> all
> > > > hints.
> > > >
> > > > Cheers,
> > > >
> > > > Jim
> > > >
> > > > On Sat, Sep 9, 2023 at 6:16 AM Jark Wu  wrote:
> > > >
> > > > > I agree with Jing,
> > > > >
> > > > > My biggest concern is this makes the boundary of adding an option
> > very
> > > > > unclear.
> > > > > It's not a strong reason to add a config just because of it doesn't
> > > > affect
> > > > > existing
> > > > > users. Does this mean that in the future we might add an option to
> > > > disable
> > > > > each feature?
> > > > >
> > > > > Flink already has a very long list of configurations [1][2] and
> this
> > is
> > > > > very scary
> > > > > and not easy to use. We should try to remove the unnecessary
> > > > configuration
> > > > > from
> > > > > the list in Flink 2.0. However, from my perspective, adding this
> > option
> > > > > makes us far
> > > > > away from this direction.
> > > > >
> > > > > Best,
> > > > > Jark
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/config/
> > > > > [2]
> > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/
> > > > >
> > > > > On Sat, 9 Sept 2023 at 17:33, Jing Ge 
> > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Thanks for bringing this to our attention. At the first glance,
> it
> > > > looks
> > > > > > reasonable to offer a new configuration to enable/disable SQL
> hints
> > > > > > globally. However, IMHO, it is not the right timing to do it now,
> > > > because
> > > > > > we should not only think as platform providers but also as end
> > > > users(the
> > > > > > number of end users are mu

回复: [DISCUSS] FLIP-363: Unify the Representation of TaskManager Location in REST API and Web UI

[Chen Zhanghao]

Hi Yangze,

Thanks for the input, totally agree with you, will take the advice.

Best,
Zhanghao Chen

发件人: Yangze Guo 
发送时间: 2023年9月15日 17:26
收件人: dev@flink.apache.org 
主题: Re: [DISCUSS] FLIP-363: Unify the Representation of TaskManager Location in 
REST API and Web UI

Thanks for driving this, Zhanghao. +1 for the overall proposal.

Some cents from my side:

1. Since most of the rest api get the location from
TaskManagerLocation, we can align the string formatter in this class.
E.g. add something like toHumanRealableString to this class.

2. From my understanding of FLIP-321, if we want to deprecate the host
field, we should annotate the related field / getter / setter with
@Deprecated in this version, and keep them at least 2 minor releases.

Best,
Yangze Guo

On Wed, Sep 13, 2023 at 8:52 PM Chen Zhanghao  wrote:
>
> Hi Jing,
>
> Thanks for the suggestion. Endpoint is indeed a more professional word in the 
> networking world but I think location is more suited here for two reasons:
>
>   1.  The term here is for uniquely identifying the TaskManager where the 
> task is deployed while providing the host machine info as well to help 
> identify taskmanager- and host-aggregative problems. So strictly speaking, it 
> is not used in a pure networking context.
>   2.  The term "location" is already used widely in the codebase, e.g. 
> TaskManagerLocation and JobExceptions-related classes.
>
> WDYT?
>
> Best,
> Zhanghao Chen
> 
> 发件人: Jing Ge 
> 发送时间: 2023年9月13日 4:52
> 收件人: dev@flink.apache.org 
> 主题: Re: [DISCUSS] FLIP-363: Unify the Representation of TaskManager Location 
> in REST API and Web UI
>
> Hi Zhanghao,
>
> Thanks for bringing this to our attention. It is a good proposal to improve
> data consistency.
>
> Speaking of naming conventions of choosing location over host, how about
> "endpoint" with the following thoughts:
>
> 1. endpoint is a more professional word than location in the network
> context.
> 2. I know commonly endpoints mean the URLs of services. Using Hostname:port
> as the endpoint follows exactly the same rule, because TaskManager is the
> top level service that aligns with the top level endpoint.
>
> WDYT?
>
> Best regards,
> Jing
>
>
> On Mon, Sep 11, 2023 at 6:01 AM Weihua Hu  wrote:
>
> > Hi, Zhanghao
> >
> > Since the meaning of "host" is not aligned, it seems good for me to remove
> > it in the future release.
> >
> > Best,
> > Weihua
> >
> >
> > On Mon, Sep 11, 2023 at 11:48 AM Chen Zhanghao 
> > wrote:
> >
> > > Hi Fan Rui,
> > >
> > > Thanks for clarifying the definition of "public interfaces", that helps a
> > > lot!
> > >
> > > Best,
> > > Zhanghao Chen
> > > 
> > > 发件人: Rui Fan <1996fan...@gmail.com>
> > > 发送时间: 2023年9月11日 11:18
> > > 收件人: dev@flink.apache.org 
> > > 主题: Re: [DISCUSS] FLIP-363: Unify the Representation of TaskManager
> > > Location in REST API and Web UI
> > >
> > > Thanks Zhanghao driving this FLIP, adding the port in Web UI
> > > seems good to me.
> > >
> > > Hi Shammon and Zhanghao,
> > >
> > > I would like to clarify the difference between Public Interfaces
> > > in FLIP and @Public in code.
> > >
> > > As I understand, the `Public Interfaces in FLIP` means these
> > > changes will be used in user side, such as: @Public class,
> > > Configuration settings, User-facing scripts/command-line tools,
> > > and rest api, etc.
> > >
> > > You can refer to  "What are the "public interfaces" of the project?"
> > > part in Flink Improvement Proposals doc[1].
> > >
> > > @Public class means the user will use this class directly, and
> > > these rest classes won't be depended on directly. So I think
> > > these classes related to rest don't need to be marked @Public.
> > >
> > > Please correct me if anything is wrong, thanks~
> > >
> > > [1]
> > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > >
> > > Best,
> > > Rui
> > >
> > > On Mon, Sep 11, 2023 at 11:09 AM Weihua Hu 
> > wrote:
> > >
> > > > Hi, Zhanghao
> > > >
> > > > Thanks for bringing this proposal.
> > > >
> > > > I have a concern:
> > > >
> > > > I prefer to keep the "host" field and add a "location" field in future
> > > > versions.
> > > > Consider a scenario where a machine (host) with multiple TaskManagers
> > has
> > > > poor processing performance due to some problems.
> > > > By using a host field aggregation, I can identify the problems with
> > this
> > > > machine and take it offline.
> > > >
> > > > Best,
> > > > Weihua
> > > >
> > > >
> > > > On Mon, Sep 11, 2023 at 10:34 AM Chen Zhanghao <
> > > zhanghao.c...@outlook.com>
> > > > wrote:
> > > >
> > > > > Hi Shammon,
> > > > >
> > > > > I think all REST API response messages (e.g.
> > > > > SubtaskExecutionAttemptDetailsInfo) should be considered as part of
> > the
> > > > > public APIs and therefore be marked as @Public. It is true though
> > none
> > > of
> > > > > them are marked as @public yet. May

回复: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL Sources

[Chen Zhanghao]

Hi Yun,

Thanks for the input. Agree with that setting parallelism for each operator is 
also valuable, and compiled plan should work. However, this would significantly 
increase the complexity. I'd rather make the FLIP focused and leave it for 
future work. WDYT?

Best,
Zhanghao Chen

发件人: Yun Tang 
发送时间: 2023年9月15日 10:32
收件人: dev@flink.apache.org 
抄送: dewe...@outlook.com 
主题: Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL Sources

Thanks for creating this FLIP,

Many users have demands to configure the source parallelism just as configuring 
the sink parallelism via DDL. Look forward for this feature.

BTW, I think setting parallelism for each operator should also be valuable. And 
this shall work with compiled plan [1] instead of SQL's DDL.


[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-292%3A+Enhance+COMPILED+PLAN+to+support+operator-level+state+TTL+configuration

Best
Yun Tang

From: Benchao Li 
Sent: Thursday, September 14, 2023 19:53
To: dev@flink.apache.org 
Cc: dewe...@outlook.com 
Subject: Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL 
Sources

Thanks Zhanghao, Dewei for preparing the FLIP,

I think this is a long awaited feature, and I appreciate your effort,
especially the "Other concerns" part you listed.

Regarding the parallelism of transformations following the source
transformation, it's indeed a problem that we initially want to solve
when we introduced this feature internally. I'd like to hear more
opinions on this. Personally I'm ok to leave it out of this FLIP for
the time being.

Chen Zhanghao  于2023年9月14日周四 14:46写道：
>
> Hi Devs,
>
> Dewei (cced) and I would like to start a discussion on FLIP-367: Support 
> Setting Parallelism for Table/SQL Sources [1].
>
> Currently, Flink Table/SQL jobs do not expose fine-grained control of 
> operator parallelism to users. FLIP-146 [2] brings us support for setting 
> parallelism for sinks, but except for that, one can only set a default global 
> parallelism and all other operators share the same parallelism. However, in 
> many cases, setting parallelism for sources individually is preferable:
>
> - Many connectors have an upper bound parallelism to efficiently ingest data. 
> For example, the parallelism of a Kafka source is bound by the number of 
> partitions, any extra tasks would be idle.
> - Other operators may involve intensive computation and need a larger 
> parallelism.
>
> We propose to improve the current situation by extending the current table 
> source API to support setting parallelism for Table/SQL sources via connector 
> options.
>
> Looking forward to your feedback.
>
> [1] FLIP-367: Support Setting Parallelism for Table/SQL Sources - Apache 
> Flink - Apache Software 
> Foundation
> [2] FLIP-146: Improve new TableSource and TableSink interfaces - Apache Flink 
> - Apache Software 
> Foundation
>
> Best,
> Zhanghao Chen



--

Best,
Benchao Li

FW: FW: RE: Inconsistent build Flink from source in readme and docs

[David Radley]

Hi,
I have looked further into the pom file in master. I see:

If I run locally:

mvn clean install -DskipTests

It does not appear to run the enforcer checks.

I see that the enforcer checks for the higher versions are only in the release 
profile.
I notice that elsewhere in the pom file :


   
   [3.1.1,)


   ${target.java.version}



Is there a reason to have these 2 versions of Maven enforcers or can we use one 
Maven enforcer that will apply to all cases?
   Kind regards, David.



From: David Radley 
Date: Friday, 15 September 2023 at 11:47
To: dev@flink.apache.org 
Subject: [EXTERNAL] FW: RE: Inconsistent build Flink from source in readme and 
docs
Hi again,
I have just checked and you already enforcer have the enforcer for this in 
master. I guess the only extra piece would be to put out a warning for java 8 
indicating it is deprecated – I could look at that in a Jira,
   Kind regards, David.

From: David Radley 
Date: Friday, 15 September 2023 at 11:44
To: dev@flink.apache.org 
Subject: [EXTERNAL] RE: Inconsistent build Flink from source in readme and docs
Hi Sergey,
Thanks for your response.
To summarise:
Flink 1.17 needs Maven 3.25 or 3.3.x with shading
Flink current master and  1.18 when released will require 3.8.6 (recommended or 
higher if it works).

I suggest that for the 1.18 build we include, we police the Maven version 
number in the build.
I can code this in the pom file using a maven-enforcer-plugin to police a 
minimum version of Maven 3.8.6 in master. For an example see:
https://github.com/odpi/egeria/blob/egeria-release-3.2/pom.xml

We could police java levels as well. I could put out a warning if built with 
java 8 indicating it I deprecated. Error for java levels less than 8.

If this ok I will raise a Jira to add this Maven and potentially java checking 
to the build,
   Kind regards, David.


From: Sergey Nuyanzin 
Date: Thursday, 14 September 2023 at 16:07
To: dev@flink.apache.org 
Subject: [EXTERNAL] Re: Inconsistent build Flink from source in readme and docs
Hi David

The links you have mentioned seems correct

Movement[1] towards to maven 3.8.6 happened since 1.18.x, and this is
mentioned in doc  for master branch

At the same time 1.17.x still requires 3.3.x as mentioned in the doc for
1.17 you've mentioned.

so it depends on the Flink version

[1] https://lists.apache.org/thread/jbw3lzzoq5w16ckco3fc9xokycs3f22x

On Thu, Sep 14, 2023 at 4:09 PM David Radley 
wrote:

> Hello,
> I am looking to build Flink from source I notice that the documentation
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/flinkdev/building/
> says
>
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/flinkdev/building/
>
> In addition you need Maven 3 and a JDK (Java Development Kit). Flink
> requires Java 8 (deprecated) or Java 11 to build.
>
> NOTE: Maven 3.3.x can build Flink, but will not properly shade away
> certain dependencies. Maven 3.2.5 creates the libraries properly.
> The README.md https://github.com/apache/flink/blob/master/README.mdsays:
>
>
>   *   Maven (we recommend version 3.8.6 and require at least 3.1.1)
>   *   Java 8 or 11 (Java 9 or 10 may work)
> git clone https://github.com/apache/flink.git
> cd flink
> ./mvnw clean package -DskipTests # this will take up to 10 minutes
> Flink is now installed in build-target.
> NOTE: Maven 3.3.x can build Flink, but will not properly shade away
> certain dependencies. Maven 3.1.1 creates the libraries properly. To build
> unit tests with Java 8, use Java 8u51 or above to prevent failures in unit
> tests that use the PowerMock runner.
>
> Can you confirm that the readme is correct and whether there is a need to
> do the documented dependency on versions after Maven 3.3.*
>
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/flinkdev/building/#dependency-shading
>
> The recommended Maven level is 3.8.6, is this what was tested or is there
> a reason not to use the latest Maven version.
>
> I am happy to amend the docs to make them consistent,
>
> Kind regards, David
>
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>


--
Best regards,
Sergey

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

回复: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL Sources

[Chen Zhanghao]

Hi Jane,

Thanks for the valuable suggestions.

For Q1, it's indeed an issue. Some possible ideas include introducing a fake 
transformation after the source that takes the global default parallelism, or 
simply make exec nodes to take the global default parallelism, but both ways 
prevent potential chaining opportunity and I'm not sure if that's good to go. 
We'll need to give deeper thoughts in it and polish our proposal. We're also 
more than glad to hear your inputs on it.

For Q2, scan.parallelism will take high precedence, as the more specific config 
should take higher precedence.

Best,
Zhanghao Chen

发件人: Jane Chan 
发送时间: 2023年9月15日 11:56
收件人: dev@flink.apache.org 
抄送: dewe...@outlook.com 
主题: Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL Sources

Hi, Zhanghao, Dewei,

Thanks for initiating this discussion. This feature is valuable in
providing more flexibility for performance tuning for SQL pipelines.

Here are my two cents,

1. In the FLIP, you mentioned concerns about the parallelism of the calc
node and concluded to "leave the behavior unchanged for now."  This means
that the calc node will use the parallelism of the source operator,
regardless of whether the source parallelism is configured or not. If I
understand correctly, currently, except for the sink exec node (which has
the ability to configure its own parallelism), the rest of the exec nodes
accept its input parallelism. From the design, I didn't see the details
about coping with input and default parallelism for the rest of the exec
nodes. Can you elaborate more about the details?

2. Does the configuration `table.exec.resource.default-parallelism` take
precedence over `scan.parallelism`?

Best,
Jane

On Fri, Sep 15, 2023 at 10:43 AM Yun Tang  wrote:

> Thanks for creating this FLIP,
>
> Many users have demands to configure the source parallelism just as
> configuring the sink parallelism via DDL. Look forward for this feature.
>
> BTW, I think setting parallelism for each operator should also be
> valuable. And this shall work with compiled plan [1] instead of SQL's DDL.
>
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-292%3A+Enhance+COMPILED+PLAN+to+support+operator-level+state+TTL+configuration
>
> Best
> Yun Tang
> 
> From: Benchao Li 
> Sent: Thursday, September 14, 2023 19:53
> To: dev@flink.apache.org 
> Cc: dewe...@outlook.com 
> Subject: Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL
> Sources
>
> Thanks Zhanghao, Dewei for preparing the FLIP,
>
> I think this is a long awaited feature, and I appreciate your effort,
> especially the "Other concerns" part you listed.
>
> Regarding the parallelism of transformations following the source
> transformation, it's indeed a problem that we initially want to solve
> when we introduced this feature internally. I'd like to hear more
> opinions on this. Personally I'm ok to leave it out of this FLIP for
> the time being.
>
> Chen Zhanghao  于2023年9月14日周四 14:46写道：
> >
> > Hi Devs,
> >
> > Dewei (cced) and I would like to start a discussion on FLIP-367: Support
> Setting Parallelism for Table/SQL Sources [1].
> >
> > Currently, Flink Table/SQL jobs do not expose fine-grained control of
> operator parallelism to users. FLIP-146 [2] brings us support for setting
> parallelism for sinks, but except for that, one can only set a default
> global parallelism and all other operators share the same parallelism.
> However, in many cases, setting parallelism for sources individually is
> preferable:
> >
> > - Many connectors have an upper bound parallelism to efficiently ingest
> data. For example, the parallelism of a Kafka source is bound by the number
> of partitions, any extra tasks would be idle.
> > - Other operators may involve intensive computation and need a larger
> parallelism.
> >
> > We propose to improve the current situation by extending the current
> table source API to support setting parallelism for Table/SQL sources via
> connector options.
> >
> > Looking forward to your feedback.
> >
> > [1] FLIP-367: Support Setting Parallelism for Table/SQL Sources - Apache
> Flink - Apache Software Foundation<
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429150
> >
> > [2] FLIP-146: Improve new TableSource and TableSink interfaces - Apache
> Flink - Apache Software Foundation<
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-146%3A+Improve+new+TableSource+and+TableSink+interfaces
> >
> >
> > Best,
> > Zhanghao Chen
>
>
>
> --
>
> Best,
> Benchao Li
>

Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL Sources

[Martijn Visser]

Hi everyone,

Thanks for the FLIP and the discussion. I find it exciting. Thanks for
pushing for this.

Best regards,

Martijn

On Fri, Sep 15, 2023 at 2:25 PM Chen Zhanghao 
wrote:

> Hi Jane,
>
> Thanks for the valuable suggestions.
>
> For Q1, it's indeed an issue. Some possible ideas include introducing a
> fake transformation after the source that takes the global default
> parallelism, or simply make exec nodes to take the global default
> parallelism, but both ways prevent potential chaining opportunity and I'm
> not sure if that's good to go. We'll need to give deeper thoughts in it and
> polish our proposal. We're also more than glad to hear your inputs on it.
>
> For Q2, scan.parallelism will take high precedence, as the more specific
> config should take higher precedence.
>
> Best,
> Zhanghao Chen
> 
> 发件人: Jane Chan 
> 发送时间: 2023年9月15日 11:56
> 收件人: dev@flink.apache.org 
> 抄送: dewe...@outlook.com 
> 主题: Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL
> Sources
>
> Hi, Zhanghao, Dewei,
>
> Thanks for initiating this discussion. This feature is valuable in
> providing more flexibility for performance tuning for SQL pipelines.
>
> Here are my two cents,
>
> 1. In the FLIP, you mentioned concerns about the parallelism of the calc
> node and concluded to "leave the behavior unchanged for now."  This means
> that the calc node will use the parallelism of the source operator,
> regardless of whether the source parallelism is configured or not. If I
> understand correctly, currently, except for the sink exec node (which has
> the ability to configure its own parallelism), the rest of the exec nodes
> accept its input parallelism. From the design, I didn't see the details
> about coping with input and default parallelism for the rest of the exec
> nodes. Can you elaborate more about the details?
>
> 2. Does the configuration `table.exec.resource.default-parallelism` take
> precedence over `scan.parallelism`?
>
> Best,
> Jane
>
> On Fri, Sep 15, 2023 at 10:43 AM Yun Tang  wrote:
>
> > Thanks for creating this FLIP,
> >
> > Many users have demands to configure the source parallelism just as
> > configuring the sink parallelism via DDL. Look forward for this feature.
> >
> > BTW, I think setting parallelism for each operator should also be
> > valuable. And this shall work with compiled plan [1] instead of SQL's
> DDL.
> >
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-292%3A+Enhance+COMPILED+PLAN+to+support+operator-level+state+TTL+configuration
> >
> > Best
> > Yun Tang
> > 
> > From: Benchao Li 
> > Sent: Thursday, September 14, 2023 19:53
> > To: dev@flink.apache.org 
> > Cc: dewe...@outlook.com 
> > Subject: Re: [DISCUSS] FLIP-367: Support Setting Parallelism for
> Table/SQL
> > Sources
> >
> > Thanks Zhanghao, Dewei for preparing the FLIP,
> >
> > I think this is a long awaited feature, and I appreciate your effort,
> > especially the "Other concerns" part you listed.
> >
> > Regarding the parallelism of transformations following the source
> > transformation, it's indeed a problem that we initially want to solve
> > when we introduced this feature internally. I'd like to hear more
> > opinions on this. Personally I'm ok to leave it out of this FLIP for
> > the time being.
> >
> > Chen Zhanghao  于2023年9月14日周四 14:46写道：
> > >
> > > Hi Devs,
> > >
> > > Dewei (cced) and I would like to start a discussion on FLIP-367:
> Support
> > Setting Parallelism for Table/SQL Sources [1].
> > >
> > > Currently, Flink Table/SQL jobs do not expose fine-grained control of
> > operator parallelism to users. FLIP-146 [2] brings us support for setting
> > parallelism for sinks, but except for that, one can only set a default
> > global parallelism and all other operators share the same parallelism.
> > However, in many cases, setting parallelism for sources individually is
> > preferable:
> > >
> > > - Many connectors have an upper bound parallelism to efficiently ingest
> > data. For example, the parallelism of a Kafka source is bound by the
> number
> > of partitions, any extra tasks would be idle.
> > > - Other operators may involve intensive computation and need a larger
> > parallelism.
> > >
> > > We propose to improve the current situation by extending the current
> > table source API to support setting parallelism for Table/SQL sources via
> > connector options.
> > >
> > > Looking forward to your feedback.
> > >
> > > [1] FLIP-367: Support Setting Parallelism for Table/SQL Sources -
> Apache
> > Flink - Apache Software Foundation<
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429150
> > >
> > > [2] FLIP-146: Improve new TableSource and TableSink interfaces - Apache
> > Flink - Apache Software Foundation<
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-146%3A+Improve+new+TableSource+and+TableSink+interfaces
> > >
> > >
> > > Best,
> > > Zhanghao Chen
> >
> >
> >
> > 

Re: [DISCUSS]clean up the savepoints compatibility table

[Martijn Visser]

Hi Jing,

I think it's fine to update the table to include lesser versions, as long
as we provide a link to an older version of the table, for users who might
be trying to upgrade from an older Flink version to the latest, so that
they are aware of potential issues that could run into.

Best regards,

Martijn

On Thu, Sep 14, 2023 at 10:22 PM Jing Ge  wrote:

> Hi,
>
> According the community update policy[1] for old releases, there is another
> cleaner option:
>
> option 3: only keep the last three versions.
>
> Compatibility info of older versions could still be found in the old
> releases.
>
> If there are no concerns, the upcoming 1.18 release will choose option 3
> which aligns with the community's support policy.
>
> Best regards,
> Jing
>
> [1] https://flink.apache.org/downloads/#update-policy-for-old-releases
>
> On Thu, Sep 14, 2023 at 1:03 PM Jing Ge  wrote:
>
> > Hi folks,
> >
> > The compatibility table[1] contains old Flink versions. Does it make
> sense
> > to clean up and remove some very old versions? My proposal would be:
> >
> > option 1: keeping less than the last 10 versions in this table so that in
> > most cases reader does not need to scroll to the right to read hidden
> > columns.
> > option 2: remove versions older than 1.8.x, since their savepoints are
> not
> > compatible with 1.17.x (the last release).
> >
> > If we could reach a consensus, it will go into effect immediately with
> the
> > master branch and the upcoming 1.18 release.
> >
> > Looking forward to your feedback and votes.
> >
> >
> > Best regards,
> > Jing
> >
> > [1]
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/upgrading/#compatibility-table
> >
> >
> >
>

Re: [DISCUSS] FLIP-327: Support stream-batch unified operator to improve job throughput when processing backlog data

[Dong Lin]

Hi Xintong,

Thanks for your comments! Please see my reply inline.

On Thu, Sep 14, 2023 at 4:58 PM Xintong Song  wrote:

> Sorry to join the discussion late.
>
> Overall, I think it's a good idea to support dynamically switching the
> operator algorithms between Streaming (optimized towards low latency +
> checkpointing supports) and Batch (optimized towards throughput). This is
> indeed a big and complex topic, and I really appreciate the previous
> discussions that narrow the scope of this FLIP down to only considering
> switching from Batch to Streaming as a first step.
>
> I have several questions.
>
> 1. The FLIP discusses various behaviors under 4 scenarios: streaming mode,
> batch mode, mixed mode with checkpoint interval > 0, mixed mode with
> checkpoint interval = 0. IIUC, this is because many batch optimizations
> cannot be supported together with checkpointing. This justifies that in
> mixed mode with interval > 0, most behaviors are the same as in streaming
> mode. However, mixed mode with checkpoint interval = 0 does not always
> necessarily mean we should apply such optimization. It is possible that in
> some cases (likely with small data amounts) the cost of such optimizations
> are higher than the benefit. Therefore, I'd suggest decoupling the concept
> of applying these optimizations (i.e., the batch execution phase in the
> mixed mode) from whether checkpointing is enabled or not. In particular,
> I'd suggest removing the scenario "mixed mode with
> e.c.interval-during-backlog > 0", changing the scenario "mixed mode with
> e.c.interval-during-backlog = 0" to simply "mixed mode", and say that can
> have different strategies for deciding whether to enable the mixed mode and
> as the first step the strategy is to enable it when
> e.c.interval-during-backlog = 0.
>

Thanks for the detailed explanation!

I have updated the "Behavior changes when switching from batch mode to
stream mode" section with the following changes.

1) Remove the description of "mixed mode with interval-during-backlog > 0"
and add the statement saying that "after this FLIP, the behavior of Flink
runtime with execution.runtime-mode = streaming AND
execution.checkpointing.interval-during-backlog > 0, will be same as the
stream mode prior to this FLIP"

2) Add the statement saying that "Mixed mode refers to the behavior of
Flink runtime after this FLIP with execution.runtime-mode = streaming AND
execution.checkpointing.interval-during-backlog = 0".

3) Add the statement saying that "It is possible for mixed mode to be
slower than stream mode, particularly when there is only small amount of
input records and the overhead of buffering/sorting inputs out-weight its
benefit. This is similar to how the merge join might be slower than hash
join. This FLIP focuses on optimizing the Flink throughput when there is a
high number of input records. In the future, we might introduce more
strategies to turn on mix mode in a smart way to avoid performance
regression".

Would this address your concern?


>
> 2. According to the FLIP, before isBacklog = false, the timer service only
> keeps timers for the current key. It also says upon the end of each key, it
> fires timers of the key up to the last watermark. IIUC, that means not all
> timers are guaranteed to be fired. It is possible that some timers are left
> to be triggered after isBacklog switching to false. If the timer service
> only keeps timers for the current key, those not-fired timers may get lost
> when switching to a new key.
>

Thanks for catching this. You are right that all timers should be fired as
long as the corresponding firing condition (either processing-time or
event-time) is satisfied.

I have updated the "Timer Service" part of the "Behavior changes when
switching from batch mode to stream mode" section accordingly. Can you see
if it addresses your concern?


>
> 3. Is it possible that some sources / operators in the job switch to
> isBacklog = false, while others are still isBacklog = true? In that case,
> what is the expected behavior for checkpointing and failover?
>

Yes, it is possible. And in this case, Flink runtime will handle this
operator as if all the operator's inputs have isBacklog=false. In
particular, Flink runtime will not automatically sort inputs of this
operator.

I added the following statement in the FLIP to clarify the behavior: "For
an operator with 2+ inputs, where some inputs have isBacklog=true and some
other inputs have isBacklog=false, Flink runtime will handle this operator
as if all its inputs have isBacklog=false".

The expected behavior of checkpointing and failover depends on whether
there is any operator currently running in the job with all its inputs'
isBacklog=true. If there exists such an operator
and interval-during-backlog = 0, then checkpoint will be disabled and the
operator will have to failover in a way similar to batch mode.



>
> 4. Do we require RecordAttributes to be properly handled by all operators?
> Or do we c

[jira] [Created] (FLINK-33093) SHOW FUNCTIONS throw exception with unset catalog

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-33093:


 Summary: SHOW FUNCTIONS throw exception with unset catalog
 Key: FLINK-33093
 URL: https://issues.apache.org/jira/browse/FLINK-33093
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Affects Versions: 1.18.0
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz
 Fix For: 1.18.0


A test like this throw an exception. It should instead return only built-in 
functions
{code}
@Test
public void testUnsetCatalogWithShowFunctions() throws Exception {
TableEnvironment tEnv = TableEnvironment.create(ENVIRONMENT_SETTINGS);

tEnv.useCatalog(null);

TableResult table = tEnv.executeSql("SHOW FUNCTIONS");
final List functions = 
CollectionUtil.iteratorToList(table.collect());

// check it has some built-in functions
assertThat(functions).hasSizeGreaterThan(0);
}
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL Sources

[ConradJam]

+ 1 Thanks for the FLIP and the discussion. I would like to ask whether to
use SQL Hint syntax to set this parallelism？

Martijn Visser  于2023年9月15日周五 20:52写道：

> Hi everyone,
>
> Thanks for the FLIP and the discussion. I find it exciting. Thanks for
> pushing for this.
>
> Best regards,
>
> Martijn
>
> On Fri, Sep 15, 2023 at 2:25 PM Chen Zhanghao 
> wrote:
>
> > Hi Jane,
> >
> > Thanks for the valuable suggestions.
> >
> > For Q1, it's indeed an issue. Some possible ideas include introducing a
> > fake transformation after the source that takes the global default
> > parallelism, or simply make exec nodes to take the global default
> > parallelism, but both ways prevent potential chaining opportunity and I'm
> > not sure if that's good to go. We'll need to give deeper thoughts in it
> and
> > polish our proposal. We're also more than glad to hear your inputs on it.
> >
> > For Q2, scan.parallelism will take high precedence, as the more specific
> > config should take higher precedence.
> >
> > Best,
> > Zhanghao Chen
> > 
> > 发件人: Jane Chan 
> > 发送时间: 2023年9月15日 11:56
> > 收件人: dev@flink.apache.org 
> > 抄送: dewe...@outlook.com 
> > 主题: Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL
> > Sources
> >
> > Hi, Zhanghao, Dewei,
> >
> > Thanks for initiating this discussion. This feature is valuable in
> > providing more flexibility for performance tuning for SQL pipelines.
> >
> > Here are my two cents,
> >
> > 1. In the FLIP, you mentioned concerns about the parallelism of the calc
> > node and concluded to "leave the behavior unchanged for now."  This means
> > that the calc node will use the parallelism of the source operator,
> > regardless of whether the source parallelism is configured or not. If I
> > understand correctly, currently, except for the sink exec node (which has
> > the ability to configure its own parallelism), the rest of the exec nodes
> > accept its input parallelism. From the design, I didn't see the details
> > about coping with input and default parallelism for the rest of the exec
> > nodes. Can you elaborate more about the details?
> >
> > 2. Does the configuration `table.exec.resource.default-parallelism` take
> > precedence over `scan.parallelism`?
> >
> > Best,
> > Jane
> >
> > On Fri, Sep 15, 2023 at 10:43 AM Yun Tang  wrote:
> >
> > > Thanks for creating this FLIP,
> > >
> > > Many users have demands to configure the source parallelism just as
> > > configuring the sink parallelism via DDL. Look forward for this
> feature.
> > >
> > > BTW, I think setting parallelism for each operator should also be
> > > valuable. And this shall work with compiled plan [1] instead of SQL's
> > DDL.
> > >
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-292%3A+Enhance+COMPILED+PLAN+to+support+operator-level+state+TTL+configuration
> > >
> > > Best
> > > Yun Tang
> > > 
> > > From: Benchao Li 
> > > Sent: Thursday, September 14, 2023 19:53
> > > To: dev@flink.apache.org 
> > > Cc: dewe...@outlook.com 
> > > Subject: Re: [DISCUSS] FLIP-367: Support Setting Parallelism for
> > Table/SQL
> > > Sources
> > >
> > > Thanks Zhanghao, Dewei for preparing the FLIP,
> > >
> > > I think this is a long awaited feature, and I appreciate your effort,
> > > especially the "Other concerns" part you listed.
> > >
> > > Regarding the parallelism of transformations following the source
> > > transformation, it's indeed a problem that we initially want to solve
> > > when we introduced this feature internally. I'd like to hear more
> > > opinions on this. Personally I'm ok to leave it out of this FLIP for
> > > the time being.
> > >
> > > Chen Zhanghao  于2023年9月14日周四 14:46写道：
> > > >
> > > > Hi Devs,
> > > >
> > > > Dewei (cced) and I would like to start a discussion on FLIP-367:
> > Support
> > > Setting Parallelism for Table/SQL Sources [1].
> > > >
> > > > Currently, Flink Table/SQL jobs do not expose fine-grained control of
> > > operator parallelism to users. FLIP-146 [2] brings us support for
> setting
> > > parallelism for sinks, but except for that, one can only set a default
> > > global parallelism and all other operators share the same parallelism.
> > > However, in many cases, setting parallelism for sources individually is
> > > preferable:
> > > >
> > > > - Many connectors have an upper bound parallelism to efficiently
> ingest
> > > data. For example, the parallelism of a Kafka source is bound by the
> > number
> > > of partitions, any extra tasks would be idle.
> > > > - Other operators may involve intensive computation and need a larger
> > > parallelism.
> > > >
> > > > We propose to improve the current situation by extending the current
> > > table source API to support setting parallelism for Table/SQL sources
> via
> > > connector options.
> > > >
> > > > Looking forward to your feedback.
> > > >
> > > > [1] FLIP-367: Support Setting Parallelism for Table/SQL Sources -
> > Apache

[jira] [Created] (FLINK-33094) KUBERNETES_KERBEROS_MOUNT_DECORATOR_ENABLED(kubernetes.decorator.kerberos-mount.enabled) not works

[Lee SeungMin (Jira)]

Lee SeungMin created FLINK-33094:


 Summary: 
KUBERNETES_KERBEROS_MOUNT_DECORATOR_ENABLED(kubernetes.decorator.kerberos-mount.enabled)
 not works
 Key: FLINK-33094
 URL: https://issues.apache.org/jira/browse/FLINK-33094
 Project: Flink
  Issue Type: Bug
  Components: Deployment / Kubernetes, Kubernetes Operator
Reporter: Lee SeungMin
 Attachments: image-2023-09-16-00-08-56-204.png, 
image-2023-09-16-00-12-12-793.png

I deployed `basic-example` using flink-kubernetes-operator(v1.5.0)

([Quick Start | Apache Flink Kubernetes 
Operator)|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.5/docs/try-flink-kubernetes-operator/quick-start/]

 

I do not need to mount kerberos (keytab), so I set 
`kubernetes.decorator.kerberos-mount.enabled` as `false`

And checked that above option set as `false` in log of job manager and task 
manager.

!image-2023-09-16-00-08-56-204.png!

 

Then as I understand, when `kubernetes.decorator.kerberos-mount.enabled` is 
`false`, it should not create secret.

But It create secret.

!image-2023-09-16-00-12-12-793.png!

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL Sources

[Yun Tang]

Hi Zhanghao,

Certainly, I think we shall leave this FLIP focus on setting the source 
parallelism via DDL's properties. I just want to clarify that setting 
parallelism for individual operators is also profitable from my experience, 
which is slighted in your FLIP.

@ConradJam BTW, compared with SQL hint, I think using `scan.parallelism` is 
better to align with current `sink.parallelism`. And once we introduce such 
option, we can also use SQL hint of dynamic table options[1] to configure the 
source parallelism.

[1] 
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/hints/#dynamic-table-options


Best
Yun Tang

From: ConradJam 
Sent: Friday, September 15, 2023 22:52
To: dev@flink.apache.org 
Subject: Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL 
Sources

+ 1 Thanks for the FLIP and the discussion. I would like to ask whether to
use SQL Hint syntax to set this parallelism？

Martijn Visser  于2023年9月15日周五 20:52写道：

> Hi everyone,
>
> Thanks for the FLIP and the discussion. I find it exciting. Thanks for
> pushing for this.
>
> Best regards,
>
> Martijn
>
> On Fri, Sep 15, 2023 at 2:25 PM Chen Zhanghao 
> wrote:
>
> > Hi Jane,
> >
> > Thanks for the valuable suggestions.
> >
> > For Q1, it's indeed an issue. Some possible ideas include introducing a
> > fake transformation after the source that takes the global default
> > parallelism, or simply make exec nodes to take the global default
> > parallelism, but both ways prevent potential chaining opportunity and I'm
> > not sure if that's good to go. We'll need to give deeper thoughts in it
> and
> > polish our proposal. We're also more than glad to hear your inputs on it.
> >
> > For Q2, scan.parallelism will take high precedence, as the more specific
> > config should take higher precedence.
> >
> > Best,
> > Zhanghao Chen
> > 
> > 发件人: Jane Chan 
> > 发送时间: 2023年9月15日 11:56
> > 收件人: dev@flink.apache.org 
> > 抄送: dewe...@outlook.com 
> > 主题: Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL
> > Sources
> >
> > Hi, Zhanghao, Dewei,
> >
> > Thanks for initiating this discussion. This feature is valuable in
> > providing more flexibility for performance tuning for SQL pipelines.
> >
> > Here are my two cents,
> >
> > 1. In the FLIP, you mentioned concerns about the parallelism of the calc
> > node and concluded to "leave the behavior unchanged for now."  This means
> > that the calc node will use the parallelism of the source operator,
> > regardless of whether the source parallelism is configured or not. If I
> > understand correctly, currently, except for the sink exec node (which has
> > the ability to configure its own parallelism), the rest of the exec nodes
> > accept its input parallelism. From the design, I didn't see the details
> > about coping with input and default parallelism for the rest of the exec
> > nodes. Can you elaborate more about the details?
> >
> > 2. Does the configuration `table.exec.resource.default-parallelism` take
> > precedence over `scan.parallelism`?
> >
> > Best,
> > Jane
> >
> > On Fri, Sep 15, 2023 at 10:43 AM Yun Tang  wrote:
> >
> > > Thanks for creating this FLIP,
> > >
> > > Many users have demands to configure the source parallelism just as
> > > configuring the sink parallelism via DDL. Look forward for this
> feature.
> > >
> > > BTW, I think setting parallelism for each operator should also be
> > > valuable. And this shall work with compiled plan [1] instead of SQL's
> > DDL.
> > >
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-292%3A+Enhance+COMPILED+PLAN+to+support+operator-level+state+TTL+configuration
> > >
> > > Best
> > > Yun Tang
> > > 
> > > From: Benchao Li 
> > > Sent: Thursday, September 14, 2023 19:53
> > > To: dev@flink.apache.org 
> > > Cc: dewe...@outlook.com 
> > > Subject: Re: [DISCUSS] FLIP-367: Support Setting Parallelism for
> > Table/SQL
> > > Sources
> > >
> > > Thanks Zhanghao, Dewei for preparing the FLIP,
> > >
> > > I think this is a long awaited feature, and I appreciate your effort,
> > > especially the "Other concerns" part you listed.
> > >
> > > Regarding the parallelism of transformations following the source
> > > transformation, it's indeed a problem that we initially want to solve
> > > when we introduced this feature internally. I'd like to hear more
> > > opinions on this. Personally I'm ok to leave it out of this FLIP for
> > > the time being.
> > >
> > > Chen Zhanghao  于2023年9月14日周四 14:46写道：
> > > >
> > > > Hi Devs,
> > > >
> > > > Dewei (cced) and I would like to start a discussion on FLIP-367:
> > Support
> > > Setting Parallelism for Table/SQL Sources [1].
> > > >
> > > > Currently, Flink Table/SQL jobs do not expose fine-grained control of
> > > operator parallelism to users. FLIP-146 [2] brings us support for
> setting
> > > parallelism for sinks, but except for that, one can 

Re: [VOTE] FLIP-334: Decoupling autoscaler and kubernetes and support the Standalone Autoscaler

[Rui Fan]

+1(binding)

Best,
Rui

On Thu, Sep 14, 2023 at 10:06 AM liu ron  wrote:

> +1(non-binding)
>
> Best,
> Ron
>
> Dong Lin  于2023年9月14日周四 09:01写道：
>
> > Thank you Rui for the proposal.
> >
> > +1 (binding)
> >
> > On Wed, Sep 13, 2023 at 10:52 AM Rui Fan <1996fan...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > Thanks for all the feedback about the FLIP-334:
> > > Decoupling autoscaler and kubernetes and
> > > support the Standalone Autoscaler[1].
> > > This FLIP was discussed in [2].
> > >
> > > I'd like to start a vote for it. The vote will be open for at least 72
> > > hours (until Sep 16th 11:00 UTC+8) unless there is an objection or
> > > insufficient votes.
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-334+%3A+Decoupling+autoscaler+and+kubernetes+and+support+the+Standalone+Autoscaler
> > > [2] https://lists.apache.org/thread/kmm03gls1vw4x6vk1ypr9ny9q9522495
> > >
> > > Best,
> > > Rui
> > >
> >
>

[RESULT][VOTE] FLIP-334: Decoupling autoscaler and kubernetes and support the Standalone Autoscaler

[Rui Fan]

Hi all,

Thanks everyone for your review and the votes!

I am happy to announce that FLIP-334:
Decoupling autoscaler and kubernetes and
support the Standalone Autoscaler[1]
has been accepted.

There are 4 binding votes and 8 non-binding vote [2]:

- Gyula Fora (binding)
- Maximilian Michels (binding)
- Dong Lin (binding)
- Rui Fan(binding)
- Ahmed Hamdy
- ConradJam
- Matt Wang
- Ferenc Csaky
- Zhanghao Chen
- Feng Jin
- Samrat Deb
- Ron Liu

There is no disapproving vote.

[1] https://cwiki.apache.org/confluence/x/x4qzDw
[2] https://lists.apache.org/thread/3wmhhqgkkg1l7ncxnzwqnjqyhqz545gl

Best,
Rui

Re: [VOTE] FLIP-361: Improve GC Metrics

[Weihua Hu]

+1 (binding)

Best,
Weihua


On Fri, Sep 15, 2023 at 4:28 PM Yangze Guo  wrote:

> +1 (binding)
>
> Best,
> Yangze Guo
>
> On Thu, Sep 14, 2023 at 5:47 PM Maximilian Michels  wrote:
> >
> > +1 (binding)
> >
> > On Thu, Sep 14, 2023 at 4:26 AM Venkatakrishnan Sowrirajan
> >  wrote:
> > >
> > > +1 (non-binding)
> > >
> > > On Wed, Sep 13, 2023, 6:55 PM Matt Wang  wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > >
> > > > Thanks for driving this FLIP
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Best,
> > > > Matt Wang
> > > >
> > > >
> > > >  Replied Message 
> > > > | From | Xintong Song |
> > > > | Date | 09/14/2023 09:54 |
> > > > | To |  |
> > > > | Subject | Re: [VOTE] FLIP-361: Improve GC Metrics |
> > > > +1 (binding)
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > >
> > > > On Thu, Sep 14, 2023 at 2:40 AM Samrat Deb 
> wrote:
> > > >
> > > > +1 ( non binding)
> > > >
> > > > These improved GC metrics will be a great addition.
> > > >
> > > > Bests,
> > > > Samrat
> > > >
> > > > On Wed, 13 Sep 2023 at 7:58 PM, ConradJam 
> wrote:
> > > >
> > > > +1 (non-binding)
> > > > gc metrics help with autoscale tuning features
> > > >
> > > > Chen Zhanghao  于2023年9月13日周三 22:16写道：
> > > >
> > > > +1 (unbinding). Looking forward to it
> > > >
> > > > Best,
> > > > Zhanghao Chen
> > > > 
> > > > 发件人: Gyula Fóra 
> > > > 发送时间: 2023年9月13日 21:16
> > > > 收件人: dev 
> > > > 主题: [VOTE] FLIP-361: Improve GC Metrics
> > > >
> > > > Hi All!
> > > >
> > > > Thanks for all the feedback on FLIP-361: Improve GC Metrics [1][2]
> > > >
> > > > I'd like to start a vote for it. The vote will be open for at least
> 72
> > > > hours unless there is an objection or insufficient votes.
> > > >
> > > > Cheers,
> > > > Gyula
> > > >
> > > > [1]
> > > >
> > > >
> > > >
> > > >
> > > >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-361*3A*Improve*GC*Metrics__;JSsrKw!!IKRxdwAv5BmarQ!dpHSOqsSHlPJ5gCvZ2yxSGjcR4xA2N-mpGZ1w2jPuKb78aujNpbzENmi1e7B26d6v4UQ8bQZ7IQaUcI$
> > > > [2]
> > > >
> https://urldefense.com/v3/__https://lists.apache.org/thread/qqqv54vyr4gbp63wm2d12q78m8h95xb2__;!!IKRxdwAv5BmarQ!dpHSOqsSHlPJ5gCvZ2yxSGjcR4xA2N-mpGZ1w2jPuKb78aujNpbzENmi1e7B26d6v4UQ8bQZFdEMnAg$
> > > >
> > > >
> > > >
> > > > --
> > > > Best
> > > >
> > > > ConradJam
> > > >
> > > >
> > > >
>

Re: [DISCUSS] FLIP-307: Flink connector Redshift

[Samrat Deb]

Hello Martijn,

I apologize for the delay in responding.

Regarding your question about integrating Redshift directly into the JDBC
connector, we are planning to offer two modes: JDBC and UNLOAD. Through our
internal benchmarking, we have observed good performance in the UNLOAD
flow. Additionally, there is a need for both flows based on different user
use cases.

If we were to explicitly add the JDBC mode to the flink-connector-jdbc, we
would have two options:

1. Include flink-connector-redshift in flink-connector-jdbc: This would
involve incorporating the Redshift connector into the JDBC connector. Since
Redshift is an AWS proprietary product, some authentication utilities can
be utilized from flink-connector-aws-base. If additional utilities are
required from the Redshift connector, they could be added to
flink-connector-aws-base. In my opinion, this approach is favorable as it
keeps everything related to AWS in flink-connector-aws.

2. Implement JDBC mode for Redshift sink in flink-connector-jdbc and UNLOAD
in flink-connector-aws: This alternative is not advisable as it could lead
to maintenance challenges and complexities.


Furthermore, it's important to highlight that Redshift has its own
customized JDBC driver[1], specifically optimized for compatibility with
Redshift. While I cannot confirm this definitively, there is a possibility
that the Redshift JDBC driver [1] might have differences in compatibility
when compared to the JDBC driver used in flink-connector-jdbc. This
suggests that if flink-connector-redshift were to rely on the JDBC
connector, it could potentially lead to future compatibility issues.

Given these considerations, it seems prudent to maintain the
Redshift-related functionality within flink-connector-aws and keep the
Redshift connector independent of the JDBC connector. This approach can
help ensure that the Redshift connector remains flexible and adaptable to
any potential changes in JDBC compatibility.

I will update the FLIP[2] to remove dependencies on flink-connector-jdbc.

[1]
https://docs.aws.amazon.com/redshift/latest/mgmt/jdbc20-download-driver.html
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-307%3A++Flink+Connector+Redshift

Bests,
Samrat


On Mon, Sep 11, 2023 at 1:21 PM Martijn Visser 
wrote:

> Hi Samrat,
>
> I'm still having doubts about the dependency on the JDBC connector. When a
> user specifies 'read mode', it will use the JDBC connector under the hood.
> Why not integrate Redshift then directly in the JDBC connector itself? It
> removes the need for a dependency on the JDBC driver, especially keeping in
> mind that this driver uses the old SourceFunction/SinkFunction interfaces
> because it hasn't been migrated yet.
>
> Best regards,
>
> Martijn
>
> On Mon, Sep 11, 2023 at 8:54 AM Samrat Deb  wrote:
>
> > Hi Leonard,
> >
> > > Do we have to rely on the latest version of JDBC Connector here?
> >
> > No, there's no need for us to depend on the latest version of the JDBC
> > Connector. Redshift has its dedicated JDBC driver [1], which includes
> > custom modifications tailored to Redshift's specific implementation
> needs.
> > This driver is the most suitable choice for our purposes.
> >
> >
> > > Could you collect the APIs that Redshift generally needs to use?
> >
> > I am actively working on it and making progress towards creating the POC.
> >
> > Bests,
> > Samrat
> >
> > [1]
> >
> >
> https://docs.aws.amazon.com/redshift/latest/mgmt/jdbc20-download-driver.html
> >
> > On Mon, Sep 11, 2023 at 12:02 PM Samrat Deb 
> wrote:
> >
> > > Hello Danny,
> > >
> > > I wanted to express my gratitude for your valuable feedback and
> > insightful
> > > suggestions.
> > >
> > > I will be revising the FLIP to incorporate all of your queries and
> review
> > > suggestions. Additionally, I plan to provide a Proof of Concept (POC)
> for
> > > the connector by the end of this week. This POC will address the points
> > > you've raised and ensure that the FLIP aligns with your
> recommendations.
> > >
> > > Thank you once again for your input.
> > >
> > > Bests,
> > > Samrat
> > >
> > > On Thu, Sep 7, 2023 at 10:21 PM Danny Cranmer  >
> > > wrote:
> > >
> > >> Hello Leonard,
> > >>
> > >> > Do we have to rely on the latest version of JDBC Connector here? I
> > >> understand that as long as the version of flink minor is the same as
> the
> > >> JDBC Connector, Could you collect the APIs that Redshift generally
> needs
> > >> to
> > >> use?
> > >>
> > >> I agree we do not necessarily need to rely on the latest patch
> version,
> > >> only the same minor. The main issue for me is the dependency
> introduces
> > a
> > >> blocker following a new Flink version. For example, when Flink 1.18.0
> is
> > >> released we cannot release the AWS connectors until the JDBC is
> > complete.
> > >> But I think this is a good tradeoff.
> > >>
> > >> > Splitting a separate redshift repository does not solve this
> coupling
> > >> problem
> > >>
> > >> Arguably it solves the AWS<>JDBC co

Re: [DISCUSS] FLIP-229: Prometheus Sink Connector

[Lorenzo Nicora]

Hello

(apologies if this is a duplicate reply)

I was working with Karthi on this connector, and I have taken over the
development.
We have a working version we would like to submit to the community.

The renumbered FLIP-312 is also updated with more details [1].
Happy to answer any questions.

Regards
Lorenzo

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-312%3A+Prometheus+Sink+Connector

On Mon, 21 Aug 2023, 13:06 Ahmed Hamdy,  wrote:

> Hello Karthi
> Is this FLIP still in progress? I see the FLIP not updated & couldn't find
> open JIRAs.
> I am happy to take over if you are no longer working on this.
> Best Regards
> Ahmed Hamdy
>
>
> On Mon, 22 May 2023 at 14:49, Martijn Visser 
> wrote:
>
> > Hi all,
> >
> > > For example, a user might want to read in logs, perform some
> aggregations
> > and publish it into a metrics store for visualisation. This might be a
> > great use-case for reducing the cardinality of metrics!
> >
> > I can see that. What I would like to see in the FLIP is a proposal on the
> > boundaries of the metrics reporter vs the Prometheus sink. I think it's
> > important that we make clear when to use a metric reporter and when not.
> I
> > can imagine that there will be Flink users who think that they can get
> data
> > from the metric reporter, make aggregrations in Flink and then store it
> > using the Prometheus sink.
> >
> > Overall, I think more context must be added to the FLIP, especially on
> the
> > motivation.
> >
> > Best regards,
> >
> > Martijn
> >
> > On Fri, May 19, 2023 at 4:28 PM Karthi Thyagarajan <
> kar...@karthitect.com>
> > wrote:
> >
> > > Hi Lijie
> > >
> > > Thank you for pointing this out. I've corrected it [1]. Also, this page
> > > [2] still shows 178 and 229 as available, which is why I picked it up.
> > >
> > > Thanks
> > > Karthi
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-312%3A+Prometheus+Sink+Connector
> > > [2]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > >
> > > On May 15, 2023, at 9:37 PM, Lijie Wang 
> > wrote:
> > >
> > >
> > > Hi Karthi,
> > >
> > > I think you are using a wrong FLIP id, the FLIP-229 has already be
> > used[1].
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Introduces+Join+Hint+for+Flink+SQL+Batch+Job
> > >
> > > Best,
> > > Lijie
> > >
> > > Martijn Visser  于2023年5月16日周二 04:44写道：
> > >
> > > Hi Karthi,
> > >
> > > Thanks for the FLIP and opening up the discussion. My main question is:
> > why
> > > should we create a separate connector and not use and/or improve the
> > > existing integrations with Prometheus? I would like to understand more
> so
> > > that it can be added to the motivation of the FLIP.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > On Mon, May 15, 2023 at 6:03 PM Karthi Thyagarajan <
> > kar...@karthitect.com>
> > > wrote:
> > >
> > > > Hello all,
> > > >
> > > > We would like to start a discussion thread on FLIP-229: Prometheus
> Sink
> > > > Connector [1] where we propose to provide a sink connector for
> > Prometheus
> > > > [2] based on the Async Sink [3]. Looking forward to comments and
> > > feedback.
> > > > Thank you.
> > > >
> > > > [1]
> > > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Prometheus+Sink+Connector
> > > > [2] https://prometheus.io/
> > > > [3]
> > > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink
> > > >
> > >
> > >
> > >
> >
>