-原始邮件-
发件人:"刘大龙"
发送时间:2020-05-06 17:55:25 (星期三)
收件人: "Jark Wu"
抄送:
主题: Re: Re: Re: The use of state ttl incremental cleanup strategy in sql
deduplication resulting in significant performance degradation
Thanks for your tuning ideas, I will test it later. Just to emphasize, I use
Hi Andrey,
Thanks for the tuning ideas. I will explain the design of deduplication.
The mini-batch implementation of deduplication buffers a bundle of input
data in heap (Java Map),
when the bundle size hit the trigger size or trigger time, the buffered
data will be processed together.
So we
Hi lsyldliu,
You can try to tune the StateTtlConfig. As the documentation suggests [1]
the TTL incremental cleanup can decrease the per record performance. This
is the price of the automatic cleanup.
If the only thing, which happens mostly in your operator, is working with
state then even
> -原始邮件-
> 发件人: "Jark Wu"
> 发送时间: 2020-04-29 14:09:44 (星期三)
> 收件人: dev , "Yu Li" , myas...@live.com
> 抄送: azagre...@apache.org
> 主题: Re: The use of state ttl incremental cleanup strategy in sql
> deduplication resulting in significant performance degradation
>
> Hi lsyldliu,
>
>
Hi lsyldliu,
Thanks for investigating this.
First of all, if you are using mini-batch deduplication, it doesn't support
state ttl in 1.9. That's why the tps looks the same with 1.11 disable state
ttl.
We just introduce state ttl for mini-batch deduplication recently.
Regarding to the
Hi, all!
At flink master branch, we have supported state ttl for sql mini-batch
deduplication using incremental cleanup strategy on heap backend, refer to
FLINK-16581. Because I want to test the performance of this feature, so I
compile master branch code and deploy the jar to production
