subject:"\\\?\\\?\\\?\\\?\\\?\\\? \\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?\\\?"

Re:flink sql作业如何支持配置流？

2023-11-20 文章 Xuyang

Hi, 
是否可以将这个”配置维表“换成流表，利用flink cdc，改动这个配置表的时候，监听字段cdc变化，同时下游上流join呢？




--

Best！
Xuyang





在 2023-11-20 19:24:47，"casel.chen"  写道：
>我有一个flink 
>sql作业过滤条件customer_id是需要根据用户配置来定的，类似于Apollo配置中心，是否可以通过定义一张配置维表来实现呢？设置TTL定期去获取最新配置。
>
>
>create table customer_conf_tbl (
>  customer_id STRING
>) with (
>  'connector' = 'apollo',
>  '其他属性' 
>);
>select * from biz_table where customer_id in (select string_split(customer_id, 
>',') from customer_conf_tbl)
>
>
>如果要做成配置实时更新作用于sql作业的话又该如何实现呢？

flink sql作业如何支持配置流？

2023-11-20 文章 casel.chen

我有一个flink 
sql作业过滤条件customer_id是需要根据用户配置来定的，类似于Apollo配置中心，是否可以通过定义一张配置维表来实现呢？设置TTL定期去获取最新配置。


create table customer_conf_tbl (
  customer_id STRING
) with (
  'connector' = 'apollo',
  '其他属性' 
);
select * from biz_table where customer_id in (select string_split(customer_id, 
',') from customer_conf_tbl)


如果要做成配置实时更新作用于sql作业的话又该如何实现呢？

Re: [DISCUSS] Change the default restart-strategy to exponential-delay

2023-11-19 文章 Rui Fan

Hi David and Mason,

Thanks for your feedback!

To David:

> Given that the new default feels more complex than the current behavior,
if we decide to do this I think it will be important to include the
rationale you've shared in the documentation.

Sounds make sense to me, I will add the related doc if we
update the default strategy.

To Mason:

> I suppose we could do some benchmarking on what works well for the
resource providers that Flink relies on e.g. Kubernetes. Based on
conferences and blogs,
> it seems most people are relying on Kubernetes to deploy Flink and the
restart strategy has a large dependency on how well Kubernetes can scale to
requests to redeploy the job.

Sorry, I didn't understand what type of benchmarking
we should do, could you elaborate on it? Thanks a lot.

Best,
Rui

On Sat, Nov 18, 2023 at 3:32 AM Mason Chen  wrote:

> Hi Rui,
>
> I suppose we could do some benchmarking on what works well for the
> resource providers that Flink relies on e.g. Kubernetes. Based on
> conferences and blogs, it seems most people are relying on Kubernetes to
> deploy Flink and the restart strategy has a large dependency on how well
> Kubernetes can scale to requests to redeploy the job.
>
> Best,
> Mason
>
> On Fri, Nov 17, 2023 at 10:07 AM David Anderson 
> wrote:
>
>> Rui,
>>
>> I don't have any direct experience with this topic, but given the
>> motivation you shared, the proposal makes sense to me. Given that the new
>> default feels more complex than the current behavior, if we decide to do
>> this I think it will be important to include the rationale you've shared in
>> the documentation.
>>
>> David
>>
>> On Wed, Nov 15, 2023 at 10:17 PM Rui Fan <1996fan...@gmail.com> wrote:
>>
>>> Hi dear flink users and devs:
>>>
>>> FLIP-364[1] intends to make some improvements to restart-strategy
>>> and discuss updating some of the default values of exponential-delay,
>>> and whether exponential-delay can be used as the default
>>> restart-strategy.
>>> After discussing at dev mail list[2], we hope to collect more feedback
>>> from Flink users.
>>>
>>> # Why does the default restart-strategy need to be updated?
>>>
>>> If checkpointing is enabled, the default value is fixed-delay with
>>> Integer.MAX_VALUE restart attempts and '1 s' delay[3]. It means
>>> the job will restart infinitely with high frequency when a job
>>> continues to fail.
>>>
>>> When the Kafka cluster fails, a large number of flink jobs will be
>>> restarted frequently. After the kafka cluster is recovered, a large
>>> number of high-frequency restarts of flink jobs may cause the
>>> kafka cluster to avalanche again.
>>>
>>> Considering the exponential-delay as the default strategy with
>>> a couple of reasons:
>>>
>>> - The exponential-delay can reduce the restart frequency when
>>>   a job continues to fail.
>>> - It can restart a job quickly when a job fails occasionally.
>>> - The restart-strategy.exponential-delay.jitter-factor can avoid r
>>>   estarting multiple jobs at the same time. It’s useful to prevent
>>>   avalanches.
>>>
>>> # What are the current default values[4] of exponential-delay?
>>>
>>> restart-strategy.exponential-delay.initial-backoff : 1s
>>> restart-strategy.exponential-delay.backoff-multiplier : 2.0
>>> restart-strategy.exponential-delay.jitter-factor : 0.1
>>> restart-strategy.exponential-delay.max-backoff : 5 min
>>> restart-strategy.exponential-delay.reset-backoff-threshold : 1h
>>>
>>> backoff-multiplier=2 means that the delay time of each restart
>>> will be doubled. The delay times are:
>>> 1s, 2s, 4s, 8s, 16s, 32s, 64s, 128s, 256s, 300s, 300s, etc.
>>>
>>> The delay time is increased rapidly, it will affect the recover
>>> time for flink jobs.
>>>
>>> # Option improvements
>>>
>>> We think the backoff-multiplier between 1 and 2 is more sensible,
>>> such as:
>>>
>>> restart-strategy.exponential-delay.backoff-multiplier : 1.2
>>> restart-strategy.exponential-delay.max-backoff : 1 min
>>>
>>> After updating, the delay times are:
>>>
>>> 1s, 1.2s, 1.44s, 1.728s, 2.073s, 2.488s, 2.985s, 3.583s, 4.299s,
>>> 5.159s, 6.191s, 7.430s, 8.916s, 10.699s, 12.839s, 15.407s, 18.488s,
>>> 22.186s, 26.623s, 31.948s, 38.337s, etc
>>>
>>> They achieve the following goals:
>>> - When restarts are infrequent in a short period of time, flink can
>>>   quickly restart the job. (For example: the retry delay time when
>>>   restarting 5 times is 2.073s)
>>> - When restarting frequently in a short period of time, flink can
>>>   slightly reduce the restart frequency to prevent avalanches.
>>>   (For example: the retry delay time when retrying 10 times is 5.1 s,
>>>   and the retry delay time when retrying 20 times is 38s, which is not
>>> very
>>> large.)
>>>
>>> As @Mingliang Liu   mentioned at dev mail list: the
>>> one-size-fits-all
>>> default values do not exist. So our goal is that the default values
>>> can be suitable for most jobs.
>>>
>>> Looking forward to your thoughts and feedback, thanks~
>>>
>>> [1]

Re:Re:flink的sql gateway支持自定义的UDF吗？

2023-11-19 文章 RS

Hi，
这种ADD JAR的方式测试了也可以用，谢谢了老哥


Thanks





在 2023-11-01 17:34:48，"Xuyang"  写道：
>Hi, 
>你指的是sql gateway上 ADD JAR这种方式来上传自定义UDF的jar包[1]么？
>
>
>
>
>[1] 
>https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/jar/
>
>--
>
>Best！
>Xuyang
>
>
>
>
>
>在 2023-11-01 14:21:04，"RS"  写道：
>>Hi
>>flink的sql gateway支持自定义的UDF吗？包括java和python的，有示例可以参考下吗？

Re:Re:flink的sql gateway支持自定义的UDF吗？

2023-11-19 文章 RS

Hi,
是的，自定义的UDF比较多，或者实现方式不同，所以加载的时候，想单独加载下，
sql-client有个参数就可以支持，-j 
sql gateway为什么不提供了？


Thanks





在 2023-11-01 17:34:48，"Xuyang"  写道：
>Hi, 
>你指的是sql gateway上 ADD JAR这种方式来上传自定义UDF的jar包[1]么？
>
>
>
>
>[1] 
>https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/jar/
>
>--
>
>Best！
>Xuyang
>
>
>
>
>
>在 2023-11-01 14:21:04，"RS"  写道：
>>Hi
>>flink的sql gateway支持自定义的UDF吗？包括java和python的，有示例可以参考下吗？

Re: [DISCUSS] Change the default restart-strategy to exponential-delay

2023-11-17 文章 David Anderson

Rui,

I don't have any direct experience with this topic, but given the
motivation you shared, the proposal makes sense to me. Given that the new
default feels more complex than the current behavior, if we decide to do
this I think it will be important to include the rationale you've shared in
the documentation.

David

On Wed, Nov 15, 2023 at 10:17 PM Rui Fan <1996fan...@gmail.com> wrote:

> Hi dear flink users and devs:
>
> FLIP-364[1] intends to make some improvements to restart-strategy
> and discuss updating some of the default values of exponential-delay,
> and whether exponential-delay can be used as the default restart-strategy.
> After discussing at dev mail list[2], we hope to collect more feedback
> from Flink users.
>
> # Why does the default restart-strategy need to be updated?
>
> If checkpointing is enabled, the default value is fixed-delay with
> Integer.MAX_VALUE restart attempts and '1 s' delay[3]. It means
> the job will restart infinitely with high frequency when a job
> continues to fail.
>
> When the Kafka cluster fails, a large number of flink jobs will be
> restarted frequently. After the kafka cluster is recovered, a large
> number of high-frequency restarts of flink jobs may cause the
> kafka cluster to avalanche again.
>
> Considering the exponential-delay as the default strategy with
> a couple of reasons:
>
> - The exponential-delay can reduce the restart frequency when
>   a job continues to fail.
> - It can restart a job quickly when a job fails occasionally.
> - The restart-strategy.exponential-delay.jitter-factor can avoid r
>   estarting multiple jobs at the same time. It’s useful to prevent
>   avalanches.
>
> # What are the current default values[4] of exponential-delay?
>
> restart-strategy.exponential-delay.initial-backoff : 1s
> restart-strategy.exponential-delay.backoff-multiplier : 2.0
> restart-strategy.exponential-delay.jitter-factor : 0.1
> restart-strategy.exponential-delay.max-backoff : 5 min
> restart-strategy.exponential-delay.reset-backoff-threshold : 1h
>
> backoff-multiplier=2 means that the delay time of each restart
> will be doubled. The delay times are:
> 1s, 2s, 4s, 8s, 16s, 32s, 64s, 128s, 256s, 300s, 300s, etc.
>
> The delay time is increased rapidly, it will affect the recover
> time for flink jobs.
>
> # Option improvements
>
> We think the backoff-multiplier between 1 and 2 is more sensible,
> such as:
>
> restart-strategy.exponential-delay.backoff-multiplier : 1.2
> restart-strategy.exponential-delay.max-backoff : 1 min
>
> After updating, the delay times are:
>
> 1s, 1.2s, 1.44s, 1.728s, 2.073s, 2.488s, 2.985s, 3.583s, 4.299s,
> 5.159s, 6.191s, 7.430s, 8.916s, 10.699s, 12.839s, 15.407s, 18.488s,
> 22.186s, 26.623s, 31.948s, 38.337s, etc
>
> They achieve the following goals:
> - When restarts are infrequent in a short period of time, flink can
>   quickly restart the job. (For example: the retry delay time when
>   restarting 5 times is 2.073s)
> - When restarting frequently in a short period of time, flink can
>   slightly reduce the restart frequency to prevent avalanches.
>   (For example: the retry delay time when retrying 10 times is 5.1 s,
>   and the retry delay time when retrying 20 times is 38s, which is not very
> large.)
>
> As @Mingliang Liu   mentioned at dev mail list: the
> one-size-fits-all
> default values do not exist. So our goal is that the default values
> can be suitable for most jobs.
>
> Looking forward to your thoughts and feedback, thanks~
>
> [1] https://cwiki.apache.org/confluence/x/uJqzDw
> [2] https://lists.apache.org/thread/5cgrft73kgkzkgjozf9zfk0w2oj7rjym
> [3]
>
> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/config/#restart-strategy-type
> [4]
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/task_failure_recovery/#exponential-delay-restart-strategy
>
> Best,
> Rui
>

[DISCUSS] Change the default restart-strategy to exponential-delay

2023-11-15 文章 Rui Fan

Hi dear flink users and devs:

FLIP-364[1] intends to make some improvements to restart-strategy
and discuss updating some of the default values of exponential-delay,
and whether exponential-delay can be used as the default restart-strategy.
After discussing at dev mail list[2], we hope to collect more feedback
from Flink users.

# Why does the default restart-strategy need to be updated?

If checkpointing is enabled, the default value is fixed-delay with
Integer.MAX_VALUE restart attempts and '1 s' delay[3]. It means
the job will restart infinitely with high frequency when a job
continues to fail.

When the Kafka cluster fails, a large number of flink jobs will be
restarted frequently. After the kafka cluster is recovered, a large
number of high-frequency restarts of flink jobs may cause the
kafka cluster to avalanche again.

Considering the exponential-delay as the default strategy with
a couple of reasons:

- The exponential-delay can reduce the restart frequency when
a job continues to fail.
- It can restart a job quickly when a job fails occasionally.
- The restart-strategy.exponential-delay.jitter-factor can avoid r
estarting multiple jobs at the same time. It’s useful to prevent
avalanches.

# What are the current default values[4] of exponential-delay?

restart-strategy.exponential-delay.initial-backoff : 1s
restart-strategy.exponential-delay.backoff-multiplier : 2.0
restart-strategy.exponential-delay.jitter-factor : 0.1
restart-strategy.exponential-delay.max-backoff : 5 min
restart-strategy.exponential-delay.reset-backoff-threshold : 1h

backoff-multiplier=2 means that the delay time of each restart
will be doubled. The delay times are:
1s, 2s, 4s, 8s, 16s, 32s, 64s, 128s, 256s, 300s, 300s, etc.

The delay time is increased rapidly, it will affect the recover
time for flink jobs.

# Option improvements

We think the backoff-multiplier between 1 and 2 is more sensible,
such as:

restart-strategy.exponential-delay.backoff-multiplier : 1.2
restart-strategy.exponential-delay.max-backoff : 1 min

After updating, the delay times are:

1s, 1.2s, 1.44s, 1.728s, 2.073s, 2.488s, 2.985s, 3.583s, 4.299s,
5.159s, 6.191s, 7.430s, 8.916s, 10.699s, 12.839s, 15.407s, 18.488s,
22.186s, 26.623s, 31.948s, 38.337s, etc

They achieve the following goals:
- When restarts are infrequent in a short period of time, flink can
quickly restart the job. (For example: the retry delay time when
restarting 5 times is 2.073s)
- When restarting frequently in a short period of time, flink can
slightly reduce the restart frequency to prevent avalanches.
(For example: the retry delay time when retrying 10 times is 5.1 s,
and the retry delay time when retrying 20 times is 38s, which is not very
large.)

As @Mingliang Liu mentioned at dev mail list: the
one-size-fits-all
default values do not exist. So our goal is that the default values
can be suitable for most jobs.

Looking forward to your thoughts and feedback, thanks~

[1] https://cwiki.apache.org/confluence/x/uJqzDw
[2] https://lists.apache.org/thread/5cgrft73kgkzkgjozf9zfk0w2oj7rjym
[3]
https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/config/#restart-strategy-type
[4]
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/task_failure_recovery/#exponential-delay-restart-strategy

< 1 2 3 4 5 6 7 8 9 10 >

共有 16769 项搜索結果，以下是第 401 - 500 matches

Mail list logo