gt;> >
>> https://github.com/apache/flink/blob/a8ca381c57788cd1a1527e4ebdc19bdbcd132fc4/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkV1Adapter.java#L365
>> <
>> https://github.com/apache/flink/blob/a8ca381c57788cd1a1527e4ebdc19bdbcd132fc4/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkV1Adapter.java#L365
>> >
>> > > > > >
>> > > > >
>> --
>> > > > > From:Steven Wu > stevenz...@gmail.com >>
>> > > > > Send Time:2022 Aug. 17 (Wed.) 07:30
>> > > > > To:dev mailto:dev@flink.apache.org >>;
>> hililiwei mailto:hilili...@gmail.com >>
>> > > > > Subject:Re: Sink V2 interface replacement for GlobalCommitter
>> > > > > > Plus, it will disable the future capability of small file
>> > compaction
>> > > > > stage post commit.
>> > > > > I should clarify this comment. if we are using the
>> > > > `WithPostCommitTopology`
>> > > > > for global committer, we would lose the capability of using the
>> post
>> > > > commit
>> > > > > stage for small files compaction.
>> > > > > On Tue, Aug 16, 2022 at 9:53 AM Steven Wu > <mailto:stevenz...@gmail.com >>
>> > > wrote:
>> > > > > >
>> > > > > > In the V1 sink interface, there is a GlobalCommitter for
>> Iceberg.
>> > > With
>> > > > > the
>> > > > > > V2 sink interface, GlobalCommitter has been deprecated by
>> > > > > > WithPostCommitTopology. I thought the post commit stage is
>> mainly
>> > for
>> > > > > async
>> > > > > > maintenance (like compaction).
>> > > > > >
>> > > > > > Are we supposed to do sth similar to the
>> > GlobalCommittingSinkAdapter?
>> > > > It
>> > > > > > seems like a temporary transition plan for bridging v1 sinks
>> to v2
>> > > > > > interfaces.
>> > > > > >
>> > > > > > private class GlobalCommittingSinkAdapter extends
>> > > > > TwoPhaseCommittingSinkAdapter
>> > > > > > implements WithPostCommitTopology {
>> > > > > > @Override
>> > > > > > public void
>> > > addPostCommitTopology(DataStream>
>> > > > > committables) {
>> > > > > > StandardSinkTopologies.addGlobalCommitter(
>> > > > > > committables,
>> > > > > > GlobalCommitterAdapter::new,
>> > > > > > () -> sink.getCommittableSerializer().get());
>> > > > > > }
>> > > > > > }
>> > > > > >
>> > > > > >
>> > > > > > In the Iceberg PR [1] for adopting the new sink interface,
>> Liwei
>> > used
>> > > > the
>> > > > > > "global" partitioner to force all committables go to a single
>> > > committer
>> > > > > > task 0. It will effectively force a global committer disguised
>> in
>> > the
>> > > > > > parallel committers. It is a little weird and also can lead to
>> > > > questions
>> > > > > > why other committer tasks are not getting any messages. Plus,
>> it
>> > will
>> > > > > > disable the future capability of small file compaction stage
>> post
>> > > > commit.
>> > > > > > Hence, I am asking what is the right approach to achieve global
>> > > > committer
>> > > > > > behavior.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Steven
>> > > > > >
>> > > > > > [1]
>> https://github.com/apache/iceberg/pull/4904/files#r946975047 <
>> https://github.com/apache/iceberg/pull/4904/files#r946975047 > <
>> > > > > https://github.com/apache/iceberg/pull/4904/files#r946975047 <
>> https://github.com/apache/iceberg/pull/4904/files#r946975047 > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>>
>> --
liwei li
liwei li created FLINK-29555:
Summary: GlobalStreamingCommitterHandler not call
notifyCheckpointCompleted after endOfInput
Key: FLINK-29555
URL: https://issues.apache.org/jira/browse/FLINK-29555
Project
liwei li created FLINK-27946:
Summary: Extending SQL syntax in a way similar to
'spark.sql.extensions'
Key: FLINK-27946
URL: https://issues.apache.org/jira/browse/FLINK-27946
Project: Flink
Congratulations Fabian :-)
Piotr Nowojski 于2021年11月16日周二 下午11:16写道：
> Congratulations :)
>
> wt., 16 lis 2021 o 16:04 Yang Wang napisał(a):
>
> > Congratulations Fabian!
> >
> > Best,
> > Yang
> >
> > Fabian Paul 于2021年11月16日周二 下午3:57写道：
> >
> > > Thanks for the warm welcome, I am looking
liwei li created FLINK-24351:
Summary: translate "JSON Function" pages into Chinese
Key: FLINK-24351
URL: https://issues.apache.org/jira/browse/FLINK-24351
Project: Flink
liwei li created FLINK-24247:
Summary: Uniformly limit the value of ABS(offset) in all window
Key: FLINK-24247
URL: https://issues.apache.org/jira/browse/FLINK-24247
Project: Flink
Issue Type
hi guys when our team developed the business code, we found
ContinuousProcessingTimeTrigger doesn't fire at the end of the window, then I
searched on JIRA, I found this Issue
https://issues.apache.org/jira/plugins/servlet/mobile#issue/FLINK-20443 . I
modified the firing logic so that the
liwei li created FLINK-23904:
Summary: Add a parameter to flink's startup command that points to
the directory named jobid containing all versions of checkpoint
Key: FLINK-23904
URL: https://issues.apache.org/jira
liwei li created FLINK-23775:
Summary: Provide a method to create StreamExecutionEnvironment
with Configuration in scala api
Key: FLINK-23775
URL: https://issues.apache.org/jira/browse/FLINK-23775
liwei li created FLINK-23655:
Summary: Custom transformation name displayed on the web, when
using Flink Table & SQL API, just like the name method of
SingleOutputStreamOperator , or remove the SQL content display directly.
-in function -
ASF JIRA (apache.org) <https://issues.apache.org/jira/browse/FLINK-22405>
[image: image.png]
Requet:
[FLINK-22405] [Table SQL / API] Support fixed-lengh chars in the Lead… by
hililiwei · Pull Request #16650 · apache/flink · GitHub
<https://github.com/apache/flink/pull/16650>
11 matches
Mail list logo
