we might do it early
> to
> avoid affecting more users.
>
> Best,
> Yun
>
> --
> From:刘建刚
> Send Time:2022 Jan. 4 (Tue.) 20:43
> To:user-zh
> Cc:dev ; user
> Subject:Re: [DISCUSS] Change
we might do it early
> to
> avoid affecting more users.
>
> Best,
> Yun
>
> --
> From:刘建刚
> Send Time:2022 Jan. 4 (Tue.) 20:43
> To:user-zh
> Cc:dev ; user
> Subject:Re: [DISCUSS] Change
; user
Subject:Re: [DISCUSS] Change some default config values of blocking shuffle
Thanks for the experiment. +1 for the changes.
Yingjie Cao 于2022年1月4日周二 17:35写道：
> Hi all,
>
> After running some tests with the proposed default value (
> taskmanager.network.sort-shuffle.min-pa
; user
Subject:Re: [DISCUSS] Change some default config values of blocking shuffle
Thanks for the experiment. +1 for the changes.
Yingjie Cao 于2022年1月4日周二 17:35写道：
> Hi all,
>
> After running some tests with the proposed default value (
> taskmanager.network.sort-shuffle.min-pa
small jobs, using sort-shuffle may not bring much performance gain, this
>>>> may because that all shuffle data can be cached in memory (page cache),
>>>> this is the case if the cluster have enough resources. However, if the
>>>> whole cluster is under heavy burden or you are runnin
small jobs, using sort-shuffle may not bring much performance gain, this
>>>> may because that all shuffle data can be cached in memory (page cache),
>>>> this is the case if the cluster have enough resources. However, if the
>>>> whole cluster is under heavy burden or you are runnin
ers and
>>> taskmanager.memory.framework.off-heap.batch-shuffle.size, you can increase
>>> these values for large-scale batch jobs.
>>> >> >
>>> >> > BTW, I am still running TPCDS tests these days and I can share
>>> these results soon.
>>> >> >
>>> >> > Best,
>>> >> > Yingjie
>>> >&g
ers and
>>> taskmanager.memory.framework.off-heap.batch-shuffle.size, you can increase
>>> these values for large-scale batch jobs.
>>> >> >
>>> >> > BTW, I am still running TPCDS tests these days and I can share
>>> these results soon.
>>> >> >
>>> >> > Best,
>>> >> > Yingjie
>>> >&g
jobs
>> with the changing configs can not improve the performance much just as your
>> test. I have some suggestions:
>> >> >>
>> >> >> The config can affect the memory usage. Will the related memory
>> configs be changed?
>> >> >>
jobs
>> with the changing configs can not improve the performance much just as your
>> test. I have some suggestions:
>> >> >>
>> >> >> The config can affect the memory usage. Will the related memory
>> configs be changed?
>> >> >>
---
From:Till Rohrmann
Send Time:2021 Dec. 14 (Tue.) 18:57
To:Jingsong Li
Cc:Yingjie Cao ; 刘建刚 ; dev
; Yun Gao ; user
; user-zh
Subject:Re: [DISCUSS] Change some default config values of blocking shuffle
As part of this FLIP, does it make sense to also extend the documentation fo
---
From:Till Rohrmann
Send Time:2021 Dec. 14 (Tue.) 18:57
To:Jingsong Li
Cc:Yingjie Cao ; 刘建刚 ; dev
; Yun Gao ; user
; user-zh
Subject:Re: [DISCUSS] Change some default config values of blocking shuffle
As part of this FLIP, does it make sense to also extend the documentation fo
to change them for different
> users. In this case, the experience can help a lot.
> >> >>
> >> >> Best,
> >> >> Liu Jiangang
> >> >>
> >> >> Yun Gao 于2021年12月10日周五 17:20写道：
> >> >>>
> >> >>>
to change them for different
> users. In this case, the experience can help a lot.
> >> >>
> >> >> Best,
> >> >> Liu Jiangang
> >> >>
> >> >> Yun Gao 于2021年12月10日周五 17:20写道：
> >> >>>
> >> >>>
nt configs? Although we change
>> >> the default values, it is helpful to change them for different users. In
>> >> this case, the experience can help a lot.
>> >>
>> >> Best,
>> >> Liu Jiangang
>> >>
>> &g
nt configs? Although we change
>> >> the default values, it is helpful to change them for different users. In
>> >> this case, the experience can help a lot.
>> >>
>> >> Best,
>> >> Liu Jiangang
>> >>
>> &g
irmation for
> taskmanager.network.sort-shuffle.min-parallelism that
> >>> since other frameworks like Spark have used sort-based shuffle for all
> the cases, does our
> >>> current circumstance still have difference with them?
> >>>
> >>> Best,
&g
irmation for
> taskmanager.network.sort-shuffle.min-parallelism that
> >>> since other frameworks like Spark have used sort-based shuffle for all
> the cases, does our
> >>> current circumstance still have difference with them?
> >>>
> >>> Best,
&g
> since other frameworks like Spark have used sort-based shuffle for all the
>>> cases, does our
>>> current circumstance still have difference with them?
>>>
>>> Best,
>>> Yun
>>>
>>>
>>>
>>>
>>> --
> since other frameworks like Spark have used sort-based shuffle for all the
>>> cases, does our
>>> current circumstance still have difference with them?
>>>
>>> Best,
>>> Yun
>>>
>>>
>>>
>>>
>>> --
or all
>> the cases, does our
>> current circumstance still have difference with them?
>>
>> Best,
>> Yun
>>
>>
>>
>>
>> --------------
>> From:Yingjie Cao
>> Send Time:2021 Dec. 10 (Fri.) 16:17
>> To:dev ; user ; user-zh <
>> user-zh@flink.apache.o
or all
>> the cases, does our
>> current circumstance still have difference with them?
>>
>> Best,
>> Yun
>>
>>
>>
>>
>> --------------
>> From:Yingjie Cao
>> Send Time:2021 Dec. 10 (Fri.) 16:17
>> To:dev ; user ; user-zh <
>> user...@flink.apache.o
ve difference with them?
>
> Best,
> Yun
>
>
>
> --
> From:Yingjie Cao
> Send Time:2021 Dec. 10 (Fri.) 16:17
> To:dev ; user ; user-zh <
> user-zh@flink.apache.org>
> Subject:Re: [DISCUSS] Change some default
ve difference with them?
>
> Best,
> Yun
>
>
>
> --
> From:Yingjie Cao
> Send Time:2021 Dec. 10 (Fri.) 16:17
> To:dev ; user ; user-zh <
> user...@flink.apache.org>
> Subject:Re: [DISCUSS] Change some default
with them?
>
> Best,
> Yun
>
>
>
>
> --
> From:Yingjie Cao
> Send Time:2021 Dec. 10 (Fri.) 16:17
> To:dev ; user ; user-zh <
> user-zh@flink.apache.org>
> Subject:Re: [DISCUSS] Change some default config val
with them?
>
> Best,
> Yun
>
>
>
>
> --
> From:Yingjie Cao
> Send Time:2021 Dec. 10 (Fri.) 16:17
> To:dev ; user ; user-zh <
> user...@flink.apache.org>
> Subject:Re: [DISCUSS] Change some default config val
difference with them?
Best,
Yun
--
From:Yingjie Cao
Send Time:2021 Dec. 10 (Fri.) 16:17
To:dev ; user ; user-zh
Subject:Re: [DISCUSS] Change some default config values of blocking shuffle
Hi dev & users:
I have created a
difference with them?
Best,
Yun
--
From:Yingjie Cao
Send Time:2021 Dec. 10 (Fri.) 16:17
To:dev ; user ; user-zh
Subject:Re: [DISCUSS] Change some default config values of blocking shuffle
Hi dev & users:
I have created a
Hi dev & users:
I have created a FLIP [1] for it, feedbacks are highly appreciated.
Best,
Yingjie
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-199%3A+Change+some+default+config+values+of+blocking+shuffle+for+better+usability
Yingjie Cao 于2021年12月3日周五 17:02写道：
> Hi dev & users,
Hi dev & users:
I have created a FLIP [1] for it, feedbacks are highly appreciated.
Best,
Yingjie
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-199%3A+Change+some+default+config+values+of+blocking+shuffle+for+better+usability
Yingjie Cao 于2021年12月3日周五 17:02写道：
> Hi dev & users,
Hi Till,
Thanks for your feedback.
>>> How will our tests be affected by these changes? Will Flink require
more resources and, thus, will it risk destabilizing our testing
infrastructure?
There are some tests that need to be adjusted, for example,
BlockingShuffleITCase. For other tests,
Hi Till,
Thanks for your feedback.
>>> How will our tests be affected by these changes? Will Flink require
more resources and, thus, will it risk destabilizing our testing
infrastructure?
There are some tests that need to be adjusted, for example,
BlockingShuffleITCase. For other tests,
Thanks for starting this discussion Yingjie,
How will our tests be affected by these changes? Will Flink require more
resources and, thus, will it risk destabilizing our testing infrastructure?
I would propose to create a FLIP for these changes since you propose to
change the default behaviour.
Thanks for starting this discussion Yingjie,
How will our tests be affected by these changes? Will Flink require more
resources and, thus, will it risk destabilizing our testing infrastructure?
I would propose to create a FLIP for these changes since you propose to
change the default behaviour.
Hi dev & users,
We propose to change some default values of blocking shuffle to improve the
user out-of-box experience (not influence streaming). The default values we
want to change are as follows:
1. Data compression
(taskmanager.network.blocking-shuffle.compression.enabled): Currently, the
Hi dev & users,
We propose to change some default values of blocking shuffle to improve the
user out-of-box experience (not influence streaming). The default values we
want to change are as follows:
1. Data compression
(taskmanager.network.blocking-shuffle.compression.enabled): Currently, the
36 matches
Mail list logo
