gt; >
> > Thanks,
> > Gary Li
> >
> > From: Danny Chan
> > Sent: Thursday, January 7, 2021 10:19 AM
> > To: dev@hudi.apache.org
> > Subject: Re: Re: [DISCUSS] New Flink Writer Proposal
>
rent flink writer is like an app, just like the delta
>> > streamer. If we want to build another Flink writer, we can still share
>> the
>> > same flink client right? Does the flink client also have to use the new
>> > feature only available on Flink 1.12?
>> >
>
share
> the
> > same flink client right? Does the flink client also have to use the new
> > feature only available on Flink 1.12?
> >
> > Thanks,
> > Gary Li
> > ________
> > From: Danny Chan
> > Sent: Thursday, January 7, 20
> Sent: Thursday, January 7, 2021 10:19 AM
> To: dev@hudi.apache.org
> Subject: Re: Re: [DISCUSS] New Flink Writer Proposal
>
> Thanks vino yang ~
>
> IMO, we should not put much put too much energy for current Flink writer,
> it is not production-r
ature only available on Flink 1.12?
>
> Thanks,
> Gary Li
>
> From: Danny Chan
> Sent: Thursday, January 7, 2021 10:19 AM
> To: dev@hudi.apache.org
> Subject: Re: Re: [DISCUSS] New Flink Writer Proposal
>
> Thanks vino yang ~
>
> IMO, we
1.12?
>
> Thanks,
> Gary Li
>
> From: Danny Chan
> Sent: Thursday, January 7, 2021 10:19 AM
> To: dev@hudi.apache.org
> Subject: Re: Re: [DISCUSS] New Flink Writer Proposal
>
> Thanks vino yang ~
>
> IMO, we should not pu
From: Danny Chan
Sent: Thursday, January 7, 2021 10:19 AM
To: dev@hudi.apache.org
Subject: Re: Re: [DISCUSS] New Flink Writer Proposal
Thanks vino yang ~
IMO, we should not put much put too much energy for current Flink writer,
it is not production-ready in the long run. There are
> >> >> >> This feature is very helpful for improving the implementation
> >> >> mechanism of
> >> >> >> Flink write-client.
> >> >> >>
> >> >> >> But this feature is only available after Flink 1.11. Before t
fter Flink 1.11. Before that,
>> there
>> >> >> was no good way to realize the mechanism of task upstream and
>> >> downstream
>> >> >> coordination through the public API provided by Flink.
>> >> >> I just have a concern,
ownstream
> >> >> coordination through the public API provided by Flink.
> >> >> I just have a concern, whether we need to take into account the users
> >> of
> >> >> earlier versions (less than Flink 1.11).
> >> >>
> >> &
sers
>> of
>> >> earlier versions (less than Flink 1.11).
>> >>
>> >> [1]: https://issues.apache.org/jira/browse/FLINK-15099
>> >>
>> >> Best,
>> >> Vino
>> >>
>> >> Gary Li 于2021年1月5日周二 上午10:40写道：
>> [1]: https://issues.apache.org/jira/browse/FLINK-15099
> >>
> >> Best,
> >> Vino
> >>
> >> Gary Li 于2021年1月5日周二 上午10:40写道：
> >>
> >>> Hi Danny,
> >>>
> >>> Thanks for the proposal. I'd recommend s
>> Gary Li 于2021年1月5日周二 上午10:40写道：
>>
>>> Hi Danny,
>>>
>>> Thanks for the proposal. I'd recommend starting a new RFC. RFC-13 was
>>> done and including some work about the refactoring so we should mark it as
>>> completed. Looking forwa
____________
>> From: Danny Chan
>> Sent: Tuesday, January 5, 2021 10:22 AM
>> To: dev@hudi.apache.org
>> Subject: Re: [DISCUSS] New Flink Writer Proposal
>>
>> Sure, i can update the RFC-13 cwiki if you agree with that.
>>
>> Vin
>
> Best,
> Gary Li
>
> From: Danny Chan
> Sent: Tuesday, January 5, 2021 10:22 AM
> To: dev@hudi.apache.org
> Subject: Re: [DISCUSS] New Flink Writer Proposal
>
> Sure, i can update the RFC-13 cwiki if you agree with that.
>
> Vinoth
Sent: Tuesday, January 5, 2021 10:22 AM
To: dev@hudi.apache.org
Subject: Re: [DISCUSS] New Flink Writer Proposal
Sure, i can update the RFC-13 cwiki if you agree with that.
Vinoth Chandar 于2021年1月5日周二 上午2:58写道：
> Overall +1 on the idea.
>
> Danny, could we move this to the apache cwik
Sure, i can update the RFC-13 cwiki if you agree with that.
Vinoth Chandar 于2021年1月5日周二 上午2:58写道：
> Overall +1 on the idea.
>
> Danny, could we move this to the apache cwiki if you don't mind?
> That's what we have been using for other RFC discussions.
>
> On Mon, Jan 4, 2021 at 1:22 AM Danny Ch
+1 on the idea as well, these changes could be super useful. Let’s collaborate
more on the cwiki.
-Nishith
> On Jan 4, 2021, at 10:58 AM, Vinoth Chandar wrote:
>
> Overall +1 on the idea.
>
> Danny, could we move this to the apache cwiki if you don't mind?
> That's what we have been using fo
Overall +1 on the idea.
Danny, could we move this to the apache cwiki if you don't mind?
That's what we have been using for other RFC discussions.
On Mon, Jan 4, 2021 at 1:22 AM Danny Chan wrote:
> The RFC-13 Flink writer has some bottlenecks that make it hard to adapter
> to production:
>
> -
The RFC-13 Flink writer has some bottlenecks that make it hard to adapter
to production:
- The InstantGeneratorOperator is parallelism 1, which is a limit for
high-throughput consumption; because all the split inputs drain to a single
thread, the network IO would gains pressure too
- The WriteProc
20 matches
Mail list logo
