--
From:liwei li
Send Time:2022 Oct. 10 (Mon.) 13:22
To:Steven Wu
Cc:Krzysztof Chmielewski ; Yun Gao
; dev
Subject:Re: Sink V2 interface replacement for GlobalCommitter
Thanks for the discussion.
Favor schemes that allow for custom committer parallelism. This
; >> commiter (parallelism = 1)`.
> >>
> >> Do you think this would be reasonable?
> >>
> >> Also @Krzysztof I have the same question with Steven that is it possible
> >> directly write to
> >> the formal files and skip the step of renaming? Si
t possible
>> directly write to
>> the formal files and skip the step of renaming? Since before the meta is
>> written to the Delta Log
>> I guess the files are not visible to the users, thus it is safe to
>> directly write to the formal files?
>>
>> Best,
>> Yun Gao
>>
>
> [1] https://issues.apache.org/jira/browse/FLINK-29459
>
>
>
> --------------
> From:Steven Wu
> Send Time:2022 Sep. 14 (Wed.) 21:33
> To:Krzysztof Chmielewski
> Cc:dev ; Yun Gao ; hililiwei <
> hilili...@gmail.
ra/browse/FLINK-29459
<https://issues.apache.org/jira/browse/FLINK-29459 >
--
From:Steven Wu
Send Time:2022 Sep. 14 (Wed.) 21:33
To:Krzysztof Chmielewski
Cc:dev ; Yun Gao ; hililiwei
Subject:Re: Sink V2 interface replacemen
h process in FLIP-147 to
>>>> ensures all the records could be committed at the end of bounded
>>>> streaming job, we have to desert the support for the cascade commits,
>>>> which makes the cascade commit of `committer -> global committer` not
>>>>
uld this option
>>> solves
>>> the issue in the current scenarios? I'll also have a double check with if
>>> it could be implemented and the failed tests Krzysztof met.
>>> Best,
>>> Yun
>>> -
he issue in the current scenarios? I'll also have a double check with if
>> it could be implemented and the failed tests Krzysztof met.
>> Best,
>> Yun
>> ----------------------
>> From:Steven Wu
>> Send Time:2022 Sep. 10
e implemented and the failed tests Krzysztof met.
> Best,
> Yun
> --
> From:Steven Wu
> Send Time:2022 Sep. 10 (Sat.) 11:31
> To:dev
> Cc:Yun Gao ; hililiwei
> Subject:Re: Sink V2 interface replacement for GlobalCommitter
> Martjin, thanks a lot for chiming in!
>
1
To:dev
Cc:Yun Gao ; hililiwei
Subject:Re: Sink V2 interface replacement for GlobalCommitter
Martjin, thanks a lot for chiming in!
Here are my concerns with adding GlobalCommitter in the PostCommitTopology
1. when we use TwoPhaseCommittingSink. We would need to create a noop/dummy
committer
Committer in Sink V1
> > > > > interface
> > > > > onto an implementation of `WithPostCommitTopology`.
> > > > > Since `WithPostCommitTopology` supports arbitrary subgraph, thus It
> > > seems
> > > > > to
> > > > > me it could support both global committer and sm
tter and small file compaction?
> We
> > > > might
> > > > have an `WithPostCommitTopology` implementation like
> > > > DataStream ds = add global committer;
> > > > if (enable file compaction) {
> > > > build the compaction subgraph from ds
s
> > > }
> > > Best,
> > > Yun
> > > [1]
> > >
> >
> https://github.com/apache/flink/blob/a8ca381c57788cd1a1527e4ebdc19bdbcd132fc4/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkV1Adapter.java#L365
> > > <
> > >
> >
> https://github.com/apache/flink/blob/a8ca381c57788cd1a15
src/main/java/org/apache/flink/streaming/api/transformations/SinkV1Adapter.java#L365
> > <
> >
> https://github.com/apache/flink/blob/a8ca381c57788cd1a1527e4ebdc19bdbcd132fc4/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkV1Adapter.java#L365
--
> From:Steven Wu
> Send Time:2022 Aug. 17 (Wed.) 07:30
> To:dev ; hililiwei
> Subject:Re: Sink V2 interface replacement for GlobalCommitter
> > Plus, it will disable the future capability of small file compaction
> stage post commit.
> I should clarify this comment. if
iliwei
Subject:Re: Sink V2 interface replacement for GlobalCommitter
> Plus, it will disable the future capability of small file compaction
stage post commit.
I should clarify this comment. if we are using the `WithPostCommitTopology`
for global committer, we would lose the capability of using
> Plus, it will disable the future capability of small file compaction
stage post commit.
I should clarify this comment. if we are using the `WithPostCommitTopology`
for global committer, we would lose the capability of using the post commit
stage for small files compaction.
On Tue, Aug 16, 2022
In the V1 sink interface, there is a GlobalCommitter for Iceberg. With the
V2 sink interface, GlobalCommitter has been deprecated by
WithPostCommitTopology. I thought the post commit stage is mainly for async
maintenance (like compaction).
Are we supposed to do sth similar to the GlobalCommitting
18 matches
Mail list logo