--
From:Jark Wu
Send Time:2019 Aug. 27 (Tue.) 16:27
To:dev
Cc:Yun Gao
Subject:Re: [DISCUSS] Enhance Support for Multicast Communication Pattern
Hi all,
Thanks Yun for bringing this topic. I missed this discussion because of the
"multicast" title.
After reading the design, if I
untime,
> >> like broadcasting events, to have a complete design. Therefore, I think
> we
> >> may still first have the broadcasting problem settled in this thread?
> Based
> >> on the points learned in the discussion, now I think that we might be
> able
> >>
ements and more generalized
>> multicasting mechanism. :)
>>
>> Best,
>> Yun
>>
>>
>>
>> ----------
>> From:SHI Xiaogang
>> Send Time:2019 Aug. 27 (Tue.) 09:16
>> To:dev ; Yun
ossible for us to partition to multiple steps and
> supports broadcasting events first ? At the same time we could also
> continue working on other options to support more scenarios like non-key
> join and they seems to requires more thoughts.
>
> Best,
> Yun
>
>
>
&g
--
From:Zhu Zhu mailto:reed...@gmail.com>>
Send Time:2019 Aug. 23 (Fri.) 17:25
To:dev mailto:dev@flink.apache.org>>
Cc:Yun Gao mailto:yungao...@aliyun.com>>
Subject:Re: [DISCUSS] Enhance Support for Multicast Communication
Pattern
Hi Piotr,
Yes you are right it's
uired since users may create any subgraph
> for
> the
> iteration body, including the operators with key. I think a possible
> solution to this issue is to introduce another data type for
> 'broadcastEmit'. For example, for an operator Operator, it may
> broadcast
> emit another type E instead of T,
-keyed partitioner.
Best,
Yun
--
From:Piotr Nowojski
Send Time:2019 Aug. 23 (Fri.) 22:29
To:Zhu Zhu
Cc:dev ; Yun Gao
Subject:Re: [DISCUSS] Enhance Support for Multicast Communication
Pattern
Hi,
If the primary motivation is broadcasting (for the iterations) and
we
have
no immediate need for m
1:56写道：
> >>>>>
> >>>>>>Hi Piotr,
> >>>>>>
> >>>>>> Very thanks for the suggestions!
> >>>>>>
> >>>>>>Totally agree with that we could first focus on the broadcast
> >>>
t;> records to all the tasks may be confused considering the semantics
>> of
>>>> keyed
>>>>>> partitioner. However, in the iteration case supporting broadcast
>> over
>>>> keyed
>>>>>> partitioner should be required since use
; > >> iteration body, including the operators with key. I think a possible
> > > >> solution to this issue is to introduce another data type for
> > > >> 'broadcastEmit'. For example, for an operator Operator, it may
> > > broadcast
> > > >>
solving this issue somehow. But before we start discussing
> > how
> > >> to solve it one last control question:
> > >>>
> > >>> I guess this multicast is intended to be used in blink planner,
> right?
> > >> Assuming that we im
should result in the design
> to
> >> introduce customized operator event (option 1 in the document). The
> cost of
> >> this method is that we need to introduce a new type of StreamElement and
> >> new interface for this type, but it should be suitable for both keyed or
> >> non-ke
e for both keyed or
> non-keyed partitioner.
>
> Best,
> Yun
>
>
>
> --
> From:Piotr Nowojski
> Send Time:2019 Aug. 23 (Fri.) 22:29
> To:Zhu Zhu
> Cc:dev ; Yun Gao
> Subject:Re: [DISCUSS] Enhance Support for Multicast Communication Pattern
>
&g
--
From:Piotr Nowojski
Send Time:2019 Aug. 23 (Fri.) 22:29
To:Zhu Zhu
Cc:dev ; Yun Gao
Subject:Re: [DISCUSS] Enhance Support for Multicast Communication Pattern
Hi,
If the primary motivation is broadcasting (for the iterations) and we have no
immediate need for multicast (cross join
ectChannels()`, I'm wondering
> >> if we introduce a new MulticastRecordWriter and left the current
> >> RecordWriter untouched, could we avoid the performance degradation ? Since
> >> with such a modification the normal RecordWriter does not need to iterate
> >
---------------------
> > From:Zhu Zhu
> > Send Time:2019 Aug. 23 (Fri.) 17:25
> > To:dev
> > Cc:Yun Gao
> > Subject:Re: [DISCUSS] Enhance Support for Multicast Communication Pattern
> >
> > Hi Piotr,
> &
e cross join case!
> Best,
> Yun
>
>
>
> --
> From:Zhu Zhu
> Send Time:2019 Aug. 23 (Fri.) 17:25
> To:dev
> Cc:Yun Gao
> Subject:Re: [DISCUSS] Enhance Support for Multicast Communication Pattern
>
> Hi Piotr,
>
&
end Time:2019 Aug. 23 (Fri.) 15:20
> To:dev
> Cc:Yun Gao
> Subject:Re: [DISCUSS] Enhance Support for Multicast Communication Pattern
>
> Hi,
>
> Yun:
>
> Thanks for proposing the idea. I have checked the document and left couple
> of questions there, but
dification the normal RecordWriter does not need to iterate
> the return array by ChannelSelector, and the only difference will be
> returning an array instead of an integer, and accessing the first element
> of the returned array instead of reading the integer directly.
> >
> > Best
cessing the first element of the returned array
> instead of reading the integer directly.
>
> Best,
> Yun
>
> --
> From:Piotr Nowojski
> Send Time:2019 Aug. 23 (Fri.) 15:20
> To:dev
> Cc:Yun
,
Yun
--
From:Piotr Nowojski
Send Time:2019 Aug. 23 (Fri.) 15:20
To:dev
Cc:Yun Gao
Subject:Re: [DISCUSS] Enhance Support for Multicast Communication Pattern
Hi,
Yun:
Thanks for proposing the idea. I have checked the document
Hi Piotr,
The case is about a broadcast join:
A--\
+--(join)--> C
B--/
Usually we can broadcast A(the result that JobVertex A produces) to all
subtasks of C.
But in this case the size of A is too large to fit in one subtask of C.
Thus we have to partition A to (A_0, A_1, A_2, ..., A_m-1).
Hi,
Yun:
Thanks for proposing the idea. I have checked the document and left couple of
questions there, but it might be better to answer them here.
What is the exact motivation and what problems do you want to solve? We have
dropped multicast support from the network stack [1] for two
Thanks Yun for starting this discussion.
I think the multicasting can be very helpful in certain cases.
I have received requirements from users that they want to do broadcast
join, while the data set to broadcast is too large to fit in one task.
Thus the requirement turned out to be to support
Hi everyone,
In some scenarios we met a requirement that some operators want to send
records to theirs downstream operators with an multicast communication pattern.
In detail, for some records, the operators want to send them according to the
partitioner (for example, Rebalance), and for
25 matches
Mail list logo
