> > > > [1] https://issues.apache.org/jira/browse/FLINK-17496
> >>>>>> > > > > > [2] https://issues.apache.org/jira/browse/FLINK-14881
> >>>>>> > > > > >
> >>>>>> > > > > > Best
--
>>>>>> > > > > > From:Thomas Weise
>>>>>> > > > > > Send Time:2020年7月17日(星期五) 05:29
>>>>>> > > > > > To:dev
>>>>>> > > > > > Cc:Zhijiang ; Stephan Ewen <
>
vestigate further today.
>>>>> > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > On Wed, Jul 8, 2020 at 4:42 AM Aljoscha Krettek <
>>>>> > > aljos...@apache.org
>>>>>
;>>> > > > > .invalid>
>>>> > > > > wrote:
>>>> > > > >
>>>> > > > > > Hi Thomas,
>>>> > > > > >
>>>> > > > > > Thanks for your further profiling inf
e point of #snapshotState
>>> in
>>> > > > previous
>>> > > > > > discussions since it indeed cost much time to block normal
>>> operator
>>> > > > > > processing.
>>> > > > > >
>&g
> >> >
>> > > > > > >> > I think we need to find out whether it has other symptoms
>> > > changed
>> > > > > > >> besides the performance regression to further figure out the
>> > > scop
> > > > > + dev@ for visibility
> > > > > > >
> > > > > > > I will investigate further today.
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Jul 8, 2020 at 4:42 AM Aljos
ons
> > > > > inside KinesisProducer implementation provided by amazonaws, which
> I
> > am
> > > > not
> > > > > quite familiar with.
> > > > > But I noticed there were two upgrades related to it in
> > release-1.11.
ding amazon-kinesis-producer to 0.14.0 [1] and another is
> > for
> > > > upgrading aws-sdk-version to 1.11.754 [2].
> > > > You mentioned that you already reverted the SDK upgrade to verify no
> > > > changes. Did you also revert the [1] to verify?
> &
the [1] to verify?
> > > [1] https://issues.apache.org/jira/browse/FLINK-17496
> > > [2] https://issues.apache.org/jira/browse/FLINK-14881
> > >
> > > Best,
> > > Zhijiang
> > > --
; You mentioned that you already reverted the SDK upgrade to verify no
> > changes. Did you also revert the [1] to verify?
> > [1] https://issues.apache.org/jira/browse/FLINK-17496
> > [2] https://issues.apache.org/jira/browse/FLINK-14881
> >
> > Best,
> > Zhijiang
> > --
&
From:Thomas Weise
> Send Time:2020年7月17日(星期五) 05:29
> To:dev
> Cc:Zhijiang ; Stephan Ewen ;
> Arvid Heise ; Aljoscha Krettek
> Subject:Re: Kinesis Performance Issue (was [VOTE] Release 1.11.0, release
> candidate #4)
>
> Sorry for the delay.
>
> I confirmed that the r
t:Re: Kinesis Performance Issue (was [VOTE] Release 1.11.0, release
candidate #4)
Sorry for the delay.
I confirmed that the regression is due to the sink (unsurprising, since
another job with the same consumer, but not the producer, runs as expected).
As promised I did CPU profiling on th
Zhijiang
>> >
>> >
>> > ------
>> > From:Thomas Weise
>> > Send Time:2020年7月7日(星期二) 23:01
>> > To:Stephan Ewen
>> > Cc:Aljoscha Krettek ; Arvid Heise <
>> ar...@verver
> > Zhijiang
> >
> >
> > --
> > From:Thomas Weise
> > Send Time:2020年7月7日(星期二) 23:01
> > To:Stephan Ewen
> > Cc:Aljoscha Krettek ; Arvid Heise <
> ar...@ververica.com>; Zhijiang
&g
I'm happy to announce that we have unanimously approved the 1.11.0 release.
There are 17 approving votes, 8 of which are binding:
- Stephan (binding)
- Till (binding)
- Aljoscha (binding)
- Robert (binding)
- Chesnay (binding)
- Dawid (binding)
- Jark (binding)
- Jincheng (binding)
- Leonard Xu
ote result soon in a separate email.
>
> Best,
> Zhijiang
>
>
> --
> From:Jingsong Li
> Send Time:2020年7月6日(星期一) 12:11
> To:dev
> Subject:Re: [VOTE] Release 1.11.0, release candidate #4
>
> +1
期一) 12:11
To:dev
Subject:Re: [VOTE] Release 1.11.0, release candidate #4
+1 (non-binding)
- verified signature and checksum
- build from source
- checked webui and log sanity
- played with filesystem and new connectors
- played with Hive connector
Best,
Jingsonga
On Mon, Jul 6, 2020 at 9:50 AM
> changes above, and it is only indicating the duration for calling
> > > `Operation.snapshotState`.
> > > If this duration is beyond your expectation, you can check or debug
> > > whether the source/sink operations might take more time to finish
> > > `snapshotState` in practice. E.g. you can
> > > make the impl
ion of this method as empty to further verify the
> > effect.
> >
> > Best,
> > Zhijiang
> >
> >
> > --
> > From:Thomas Weise
> > Send Time:2020年7月5日(星期日) 12:22
> > To:de
ore time to finish
> `snapshotState` in practice. E.g. you can
> make the implementation of this method as empty to further verify the
> effect.
>
> Best,
> Zhijiang
>
>
> --
> From:Thomas Weise
> Send T
ementation of this method as empty to further verify the effect.
Best,
Zhijiang
--
From:Thomas Weise
Send Time:2020年7月5日(星期日) 12:22
To:dev ; Zhijiang
Cc:Yingjie Cao
Subject:Re: [VOTE] Release 1.11.0, release candidate #4
Hi Zhijia
r case based on the current
> analysis.
> > If we really conclude anything need to be resolved after the final
> > release, then we can also make the next minor release-1.11.1 come soon.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-18433
> >
> > Bes
Best,
> Zhijiang
>
>
> ------
> From:Thomas Weise
> Send Time:2020年7月4日(星期六) 12:26
> To:dev ; Zhijiang
> Cc:Yingjie Cao
> Subject:Re: [VOTE] Release 1.11.0, release candidate #4
>
> Hi Zhijiang,
>
e-1.11.1 come soon.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-18433
> >
> > Best,
> > Zhijiang
> >
> >
> > --
> > From:Thomas Weise
> > Send Time:2020年7月4日(星期六) 12
e the next minor release-1.11.1 come soon.
>
> [1] https://issues.apache.org/jira/browse/FLINK-18433
>
> Best,
> Zhijiang
>
>
> ------
> From:Thomas Weise
> Send Time:2020年7月4日(星期六) 12:26
> To:dev ; Zh
iang
--
From:Thomas Weise
Send Time:2020年7月4日(星期六) 12:26
To:dev ; Zhijiang
Cc:Yingjie Cao
Subject:Re: [VOTE] Release 1.11.0, release candidate #4
Hi Zhijiang,
It will probably be best if we connect next week and discuss the issue
directly since this could be quite difficult to reproduce.
Be
sharing, I guess these three vertex parallelism task would
> probably be deployed into the same slot, then the data shuffle is by memory
> queue, not network stack. If so, the above [2] should no effect.
> - I also saw some Jira changes for kinesis in this release, could you
> c
; > other?
> > The checkpoint mode is exactly-once with rocksDB state backend?
> > The backpressure happened in this case?
> > How much percentage regression in this case?
> >
> > Best,
> > Zhijiang
> >
> >
> >
> > ---
+1 (non-binding)
- check wheel package consistency with the built from the source code
- test the built from the wheel package in mac os with python 3.6
- verify the performance for PyFlink UDFs including Python General UDF and
Pandas UDF
- test Python UDTF
Best,
Xingbo
Dian Fu 于2020年7月3日周五
+1 (non-binding)
- built from source with scala 2.11 successfully
- checked the signature and checksum of the binary packages
- installed PyFlink on MacOS, Windows and Linux successfully
- tested the functionality of Pandas UDF and the conversion between PyFlink
Table and Pandas DataFrame
-
+1(binding)
Checks:
- check wheel package consistency
- test the built from the wheel package
- checked the signature and checksum
- pip installed the Python package
`apache_flink-1.11.0-cp37-cp37m-macosx_10_9_x86_64.whl` successfully and
run a simple word count example successfully
+1 (non-binding)
- checked/verified signatures and hashes
- built from source sing scala 2.11 succeeded
- go through all issues which "fixVersion" property is 1.11.0, there is no
blocker.
- checked that there are no missing artifacts
- test SQL connector Elasticsearch7/JDBC/HBase/Kafka (new
; > other?
> > The checkpoint mode is exactly-once with rocksDB state backend?
> > The backpressure happened in this case?
> > How much percentage regression in this case?
> >
> > Best,
> > Zhijiang
> >
> >
> >
> > ---
By slot sharing, I guess these three vertex parallelism task would
>> probably be deployed into the same slot, then the data shuffle is by memory
>> queue, not network stack. If so, the above [2] should no effect.
>> - I also saw some Jira changes for kinesis in this release, coul
y be deployed into the same slot, then the data shuffle is by memory
> queue, not network stack. If so, the above [2] should no effect.
> - I also saw some Jira changes for kinesis in this release, could you
> confirm that these changes would not effect the performance?
>
> Bes
k stack. If so, the above [2] should no effect.
- I also saw some Jira changes for kinesis in this release, could you
confirm that these changes would not effect the performance?
Best,
Zhijiang
--
From:Thomas Weise
Send Time:2020年7月3日(
oint. Maybe we should adjust the vote template accordingly in
>>>> the respective wiki to guide the following release processes.
>>>>
>>>> Regarding the performance regression, could you provide some more details
>>>> for our better measurement
this case?
Best,
Zhijiang
--
From:Thomas Weise
Send Time:2020年7月2日(星期四) 09:54
To:dev
Subject:Re: [VOTE] Release 1.11.0, release candidate #4
Hi Till,
Yes, we don't have the setting in flink-conf.yaml.
Generally, we carry forward the existing configuration and any change to
default configuration value
gt; E.g. I guess the topology only includes two vertexes source and sink?
> > > What is the parallelism for every vertex?
> > > The upstream shuffles data to the downstream via rebalance partitioner or
> > > other?
> > > The checkpoint mode is exactly-once with rocksDB sta
x?
> > The upstream shuffles data to the downstream via rebalance partitioner or
> > other?
> > The checkpoint mode is exactly-once with rocksDB state backend?
> > The backpressure happened in this case?
> > How much percentage regression in this case?
> >
> >
int mode is exactly-once with rocksDB state backend?
> The backpressure happened in this case?
> How much percentage regression in this case?
>
> Best,
> Zhijiang
>
>
>
> --
> From:Thomas Weise
> S
ble we can reproduce to
>> >>> analysis the reason based on Thomas's feedback, then we can evaluate
>> its
>> >>> effect.
>> >>>
>> >>> Regarding the FLINK-18461, after syncing with Jark offline, the bug
>> would
>> >>> effect one of three scenarios for using CDC feature, and t
users.
> >>> My suggestion is to merge it into release-1.11 ATM since the PR already
> >>> open for review, then let's further finalize the conclusion later. If
> >> this
> >>> issue is the only one after RC4 going through, then another option is
> to
> &g
doubt to cover
all
of them in next RC5.
Best,
Zhijiang
--
From:Till Rohrmann
Send Time:2020年7月2日(星期四) 16:46
To:dev
Cc:Zhijiang
Subject:Re: [VOTE] Release 1.11.0, release candidate #4
I agree with Robert.
@Chesnay: The problem
suggested, as we can prepare
> for
> > the next minor release soon. If there are other blockers issues during
> > voting and necessary to be resolved soon, then it is no doubt to cover
> all
> > of them in next RC5.
> >
> > Best,
> > Zhijiang
> >
> >
> >
voting and necessary to be resolved soon, then it is no doubt to cover all
of them in next RC5.
Best,
Zhijiang
--
From:Till Rohrmann
Send Time:2020年7月2日(星期四) 16:46
To:dev
Cc:Zhijiang
Subject:Re: [VOTE] Release 1.11.0, release c
olved soon, then it is no doubt to cover all
> of them in next RC5.
>
> Best,
> Zhijiang
>
>
> --
> From:Till Rohrmann
> Send Time:2020年7月2日(星期四) 16:46
> To:dev
> Cc:Zhijiang
> Subject:Re: [VOTE] Rele
to
be resolved soon, then it is no doubt to cover all of them in next RC5.
Best,
Zhijiang
--
From:Till Rohrmann
Send Time:2020年7月2日(星期四) 16:46
To:dev
Cc:Zhijiang
Subject:Re: [VOTE] Release 1.11.0, release candidate #4
I agree
rce and sink?
> > > What is the parallelism for every vertex?
> > > The upstream shuffles data to the downstream via rebalance partitioner
> or
> > > other?
> > > The checkpoint mode is exactly-once with rocksDB state backend?
> > > The backpressur
er or
> > other?
> > The checkpoint mode is exactly-once with rocksDB state backend?
> > The backpressure happened in this case?
> > How much percentage regression in this case?
> >
> > Best,
> > Zhijiang
> >
> >
> >
> > -
--
> From:Thomas Weise
> Send Time:2020年7月2日(星期四) 09:54
> To:dev
> Subject:Re: [VOTE] Release 1.11.0, release candidate #4
>
> Hi Till,
>
> Yes, we don't have the setting in flink-conf.yaml.
>
> Generally, we carry forward the existing c
] Release 1.11.0, release candidate #4
Hi Till,
Yes, we don't have the setting in flink-conf.yaml.
Generally, we carry forward the existing configuration and any change to
default configuration values would impact the upgrade.
Yes, since it is an incompatible change I would state it in the release
Hi Till,
Yes, we don't have the setting in flink-conf.yaml.
Generally, we carry forward the existing configuration and any change to
default configuration values would impact the upgrade.
Yes, since it is an incompatible change I would state it in the release
notes.
Thanks,
Thomas
BTW I found
Hi Thomas,
just to confirm: When starting the image in local mode, then you don't have
any of the JobManager memory configuration settings configured in the
effective flink-conf.yaml, right? Does this mean that you have explicitly
removed `jobmanager.heap.size: 1024m` from the default
Thanks for preparing another RC!
As mentioned in the previous RC thread, it would be super helpful if the
release notes that are part of the documentation can be included [1]. It's
a significant time-saver to have read those first.
I found one more non-backward compatible change that would be
- source does not contain binaries
- started a local cluster, logs are fine, examples run
- web submission works _in general_
However, a number of batch examples fail when submitted through the
WebUI with the following error:
Caused by: org.apache.flink.api.common.InvalidProgramException:
Job
Hi everyone,
Please review and vote on the release candidate #4 for the version 1.11.0, as
follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)
The complete staging area is available for your review, which includes:
* JIRA release notes [1],
58 matches
Mail list logo
