Hi Jing and everyone, I have conducted three rounds of benchmarking with Java11, comparing release 1.18 (commit: deb07e99560[1]) with commit 6d62f9918ea[2]. The results are attached[3]. Most of the tests show no obvious regression. However, I did observe significant change in several tests. Upon reviewing the historical results from the previous pipeline, I also discovered a substantial variance in those tests, as shown in the timeline pictures included in the sheet[3]. I believe this variance has existed for a long time and requires further investigation, and fully measuring the variance requires more rounds (15 or more). I think for now it is not a blocker for release 1.18. WDYT?
Best, Zakelly [1] https://github.com/apache/flink/commit/deb07e99560b45033a629afc3f90666ad0a32feb [2] https://github.com/apache/flink/commit/6d62f9918ea2cbb8a10c705a25a4ff6deab60711 [3] https://docs.google.com/spreadsheets/d/1V0-duzNTgu7H6R7kioF-TAPhlqWl7Co6Q9ikTBuaULo/edit?usp=sharing On Sun, Sep 24, 2023 at 11:29 AM ConradJam <jam.gz...@gmail.com> wrote: > > +1 for testing with Java 17 > > Jing Ge <j...@ververica.com.invalid> 于2023年9月24日周日 09:40写道： > > > +1 for testing with Java 17 too. Thanks Zakelly for your effort! > > > > Best regards, > > Jing > > > > On Fri, Sep 22, 2023 at 1:01 PM Zakelly Lan <zakelly....@gmail.com> wrote: > > > > > Hi Jing, > > > > > > I agree we could wait for the result with Java 11. And it should be > > > available next Monday. > > > Additionally, I could also build a pipeline with Java 17 later since > > > it is supported in 1.18[1]. > > > > > > > > > Best regards, > > > Zakelly > > > > > > [1] > > > > > https://github.com/apache/flink/commit/9c1318ca7fa5b2e7b11827068ad1288483aaa464#diff-8310c97396d60e96766a936ca8680f1e2971ef486cfc2bc55ec9ca5a5333c47fR53 > > > > > > On Fri, Sep 22, 2023 at 5:57 PM Jing Ge <j...@ververica.com.invalid> > > > wrote: > > > > > > > > Hi Zakelly, > > > > > > > > Thanks for your effort and the update! Since Java 8 has been > > > deprecated[1], > > > > let's wait for the result with Java 11. It should be available after > > the > > > > weekend and there should be no big surprise. WDYT? > > > > > > > > Best regards, > > > > Jing > > > > > > > > [1] > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/release-notes/flink-1.15/#jdk-upgrade > > > > > > > > On Fri, Sep 22, 2023 at 11:26 AM Zakelly Lan <zakelly....@gmail.com> > > > wrote: > > > > > > > > > Hi everyone, > > > > > > > > > > I want to provide an update on the benchmark results that I have been > > > > > working on. After spending some time preparing the environment and > > > > > adjusting the benchmark script, I finally got a comparison between > > > > > release 1.18 (commit: 2aeb99804ba[1]) and the commit before the old > > > > > codespeed server went down (commit: 6d62f9918ea[2]) on openjdk8. The > > > > > report is attached[3]. Note that the test has only run once on jdk8, > > > > > so the impact of single-test fluctuations is not ruled out. > > > > > Additionally, I have noticed some significant fluctuations in > > specific > > > > > tests when reviewing previous benchmark scores, which I have also > > > > > noted in the report. Taking all of these factors into consideration, > > I > > > > > think there is no obvious regression in release 1.18 *for now*. More > > > > > tests including the one on openjdk11 are on the way. Hope it does not > > > > > delay the release procedure. > > > > > > > > > > Please let me know if you have any concerns. > > > > > > > > > > > > > > > Best, > > > > > Zakelly > > > > > > > > > > [1] > > > > > > > > > > https://github.com/apache/flink/commit/2aeb99804ba56c008df0a1730f3246d3fea856b9 > > > > > [2] > > > > > > > > > > https://github.com/apache/flink/commit/6d62f9918ea2cbb8a10c705a25a4ff6deab60711 > > > > > [3] > > > > > > > > > > https://docs.google.com/spreadsheets/d/1-3Y974jYq_WrQNzLN-y_6lOU-NGXaIDTQBYTZd04tJ0/edit?usp=sharing > > > > > > > > > > The new environment for benchmark: > > > > > ECS on Aliyun > > > > > CPU: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz (8 Core available) > > > > > Memory: 64GB > > > > > OS: Alibaba Cloud Linux 3.2104 LTS 64bit > > > > > Kernel: 5.10.134-15.al8.x86_64 > > > > > OpenJDK8 version: 1.8.0_372 > > > > > > > > > > On Thu, Sep 21, 2023 at 12:04 PM Yuxin Tan <tanyuxinw...@gmail.com> > > > wrote: > > > > > > > > > > > > Hi, Zakelly, > > > > > > > > > > > > No benchmark tests currently are affected by this issue. We > > > > > > may add benchmarks to guard it later. Thanks. > > > > > > > > > > > > Best, > > > > > > Yuxin > > > > > > > > > > > > > > > > > > Zakelly Lan <zakelly....@gmail.com> 于2023年9月21日周四 11:56写道： > > > > > > > > > > > > > Hi Jing, > > > > > > > > > > > > > > Sure, I will run the benchmark with this fix. > > > > > > > > > > > > > > Hi Yunxin, > > > > > > > > > > > > > > I'm not familiar with the hybrid shuffle. Is there any specific > > > > > > > benchmark test that may be affected by this issue? I will pay > > > special > > > > > > > attention to it. > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > Zakelly > > > > > > > > > > > > > > On Thu, Sep 21, 2023 at 10:08 AM Yuxin Tan < > > tanyuxinw...@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > > > > > > Hi, Jing, Qingsheng, > > > > > > > > > > > > > > > > Thanks a lot. > > > > > > > > The fix has been backported. > > > > > > > > > > > > > > > > Best, > > > > > > > > Yuxin > > > > > > > > > > > > > > > > > > > > > > > > Jing Ge <j...@ververica.com.invalid> 于2023年9月21日周四 00:42写道： > > > > > > > > > > > > > > > > > Hi Lijie, > > > > > > > > > > > > > > > > > > Thanks for reaching out. Please backport it to release-1.18. > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > Jing > > > > > > > > > > > > > > > > > > On Wed, Sep 20, 2023 at 4:35 PM Lijie Wang < > > > > > wangdachui9...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi community and release managers: > > > > > > > > > > > > > > > > > > > > We found a critical bug[1] of the rest client a few days > > ago, > > > > > which > > > > > > > may > > > > > > > > > > cause the inode to be used up. Now the fix-PR[2] is ready > > for > > > > > > > merging, I > > > > > > > > > > hope to backport it to release-1.18. > > > > > > > > > > > > > > > > > > > > Please let me know if you have any concerns. Thanks. > > > > > > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-32974 > > > > > > > > > > [2] https://github.com/apache/flink/pull/23363 > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Lijie > > > > > > > > > > > > > > > > > > > > Zakelly Lan <zakelly....@gmail.com> 于2023年9月19日周二 17:26写道： > > > > > > > > > > > > > > > > > > > > > Hi Yuan and Jing, > > > > > > > > > > > > > > > > > > > > > > Thank you for sharing your thoughts. I completely agree > > > that > > > > > it is > > > > > > > our > > > > > > > > > > > top priority to ensure that there are no regressions from > > > the > > > > > last > > > > > > > > > > > commit the previous benchmark pipeline covered to the > > final > > > > > commit > > > > > > > of > > > > > > > > > > > this release. I will try to get this result first. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > Zakelly > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 19, 2023 at 4:55 PM Jing Ge > > > > > <j...@ververica.com.invalid > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hi > > > > > > > > > > > > > > > > > > > > > > > > Thanks Zakelly and Yuan for your effort and update. > > > Since we > > > > > > > changed > > > > > > > > > > the > > > > > > > > > > > > hardware, IMHO, if we are able to reach a consensus in > > > the > > > > > > > community > > > > > > > > > > that > > > > > > > > > > > > there is no regression with the benchmarks, we could > > > consider > > > > > > > > > releasing > > > > > > > > > > > rc1 > > > > > > > > > > > > without waiting for the new baseline scores which might > > > take > > > > > > > days. > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Jing > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 19, 2023 at 10:42 AM Yuan Mei < > > > > > > > yuanmei.w...@gmail.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hey Zakelly, > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks very much for the efforts to re-build the > > entire > > > > > > > benchmark > > > > > > > > > > > > > environment. > > > > > > > > > > > > > > > > > > > > > > > > > > As long as we have > > > > > > > > > > > > > 1) the pipeline set up and ready (no need for the > > > entire > > > > > portal > > > > > > > > > > ready), > > > > > > > > > > > > > 2) get benchmark comparison numbers (comparing with > > the > > > > > commit > > > > > > > just > > > > > > > > > > > before > > > > > > > > > > > > > the benchmark pipeline is down) and > > > > > > > > > > > > > 3) confirmed no-regression, it should be good enough. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks again! > > > > > > > > > > > > > > > > > > > > > > > > > > Best > > > > > > > > > > > > > Yuan > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 19, 2023 at 4:26 PM Zakelly Lan < > > > > > > > zakelly....@gmail.com > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am working on rebuilding the benchmark pipeline > > and > > > > > it's > > > > > > > almost > > > > > > > > > > > > > > done. However, due to the change in machines for > > > > > > > benchmarking, I > > > > > > > > > > will > > > > > > > > > > > > > > need a few more days to run tests and gather the > > > baseline > > > > > > > scores > > > > > > > > > > for > > > > > > > > > > > > > > further comparison. Once the pipeline is fully > > > ready, we > > > > > will > > > > > > > > > > proceed > > > > > > > > > > > > > > with the performance test for release 1.18.0. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please let me know if you have any concerns. Thank > > > you > > > > > all > > > > > > > for > > > > > > > > > your > > > > > > > > > > > > > > patience. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > Zakelly > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 18, 2023 at 6:57 PM Jing Ge > > > > > > > > > <j...@ververica.com.invalid > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The RC0 for Apache Flink 1.18.0 has been created. > > > This > > > > > RC > > > > > > > is > > > > > > > > > > > currently > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > preview only to facilitate the integrated testing > > > > > since the > > > > > > > > > > > benchmark > > > > > > > > > > > > > > tests > > > > > > > > > > > > > > > are not available yet[1] and the release > > > announcement > > > > > is > > > > > > > still > > > > > > > > > > > under > > > > > > > > > > > > > > > review. The RC1 will be released after all > > > benchmarks > > > > > > > tests are > > > > > > > > > > > passed. > > > > > > > > > > > > > > The > > > > > > > > > > > > > > > related voting process will be triggered once the > > > > > > > announcement > > > > > > > > > is > > > > > > > > > > > > > ready. > > > > > > > > > > > > > > > The RC0 has all the artifacts that we would > > > typically > > > > > have > > > > > > > for > > > > > > > > > a > > > > > > > > > > > > > release, > > > > > > > > > > > > > > > except for the release note and the website pull > > > > > request > > > > > > > for > > > > > > > > > the > > > > > > > > > > > > > release > > > > > > > > > > > > > > > announcement. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The following contents are available for your > > > review: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - The preview source release and binary > > convenience > > > > > > > releases > > > > > > > > > [2], > > > > > > > > > > > which > > > > > > > > > > > > > > > are signed with the key with fingerprint > > > > > > > 96AE0E32CBE6E0753CE6 > > > > > > > > > > [3]. > > > > > > > > > > > > > > > - all artifacts that would normally be deployed > > to > > > the > > > > > > > Maven > > > > > > > > > > > > > > > Central Repository [4]. > > > > > > > > > > > > > > > - source code tag "release-1.18.0-rc0" [5] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Your help testing the release will be greatly > > > > > appreciated! > > > > > > > And > > > > > > > > > > > we'll > > > > > > > > > > > > > > > create the rc1 release and the voting thread as > > > soon as > > > > > > > all the > > > > > > > > > > > efforts > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > finished. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > https://issues.apache.org/jira/browse/FLINK-33052 > > > > > > > > > > > > > > > [2] > > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc0/ > > > > > > > > > > > > > > > [3] > > > > > https://dist.apache.org/repos/dist/release/flink/KEYS > > > > > > > > > > > > > > > [4] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheflink-1656/ > > > > > > > > > > > > > > > [5] > > > > > > > > > > > > > > > > https://github.com/apache/flink/releases/tag/release-1.18.0-rc0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > Qingsheng, Sergei, Konstantin and Jing > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Best > > ConradJam
