Thanks Daisy and Kevin! The IO scheduling idea of the sequential reading and the benchmark result look really great! Looking forward to the next work.
Best, Yuxin weijie guo <guoweijieres...@gmail.com> 于2021年11月3日周三 下午5:24写道： > It's really an amazing job to fill in the defects of flink in batch > shuffle. I really appreciate the work done in io scheduling, the sequential > reading of the shuffle reader can greatly improve the disk IO performance > and stability. Sort-based shuffle realizes this feature in a concise and > efficient way. By the way, the default shuffle implementation in the batch > mode of flink is still hash-based, maybe we can consider using the new > shuffle implementation by default later. Last but not least, thank Yingjie > Cao (Kevin) and Daisy Tsang for publishing this blog. > > Lijie Wang <wangdachui9...@gmail.com> 于2021年11月3日周三 下午4:17写道： > >> Thanks Daisy and Kevin for bringing this blog, it is very helpful for >> understanding the principle of sort shuffle. >> >> >> Best, >> >> Lijie >> >> Guowei Ma <guowei....@gmail.com> 于2021年11月3日周三 下午2:57写道： >> >>> >>> Thank Daisy& Kevin much for your introduction to the improvement of TM >>> blocking shuffle, credit base+io scheduling is indeed a very interesting >>> thing. At the same time, I look forward to this as a default setting for tm >>> blocking shuffle. >>> >>> Best, >>> Guowei >>> >>> >>> On Wed, Nov 3, 2021 at 2:46 PM Gen Luo <luogen...@gmail.com> wrote: >>> >>>> Thanks Daisy and Kevin! The benchmark results look really exciting! >>>> >>>> On Tue, Nov 2, 2021 at 4:38 PM David Morávek <d...@apache.org> wrote: >>>> >>>>> Thanks Daisy and Kevin for a great write up! ;) Especially the 2nd >>>>> part was really interesting, I really like the idea of the single spill >>>>> file with a custom scheduling of read requests. >>>>> >>>>> Best, >>>>> D. >>>>> >>>>> On Mon, Nov 1, 2021 at 10:01 AM Daisy Tsang <da...@ververica.com> >>>>> wrote: >>>>> >>>>>> Hey everyone, we have a new two-part post published on the Apache >>>>>> Flink blog about the sort-based blocking shuffle implementation in Flink. >>>>>> It covers benchmark results, design and implementation details, and more! >>>>>> We hope you like it and welcome any sort of feedback on it. :) >>>>>> >>>>>> >>>>>> https://flink.apache.org/2021/10/26/sort-shuffle-part1.html >>>>>> https://flink.apache.org/2021/10/26/sort-shuffle-part2.html >>>>>> >>>>>
