This is an automated email from the ASF dual-hosted git repository. lindong pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/flink-web.git
The following commit(s) were added to refs/heads/asf-site by this push: new acf2d4018 Rebuild website acf2d4018 is described below commit acf2d4018ff10c0d29bc5d884b41419636e80eb6 Author: zhangzp <zhangzhipe...@gmail.com> AuthorDate: Tue Jul 12 11:16:32 2022 +0800 Rebuild website --- content/blog/feed.xml | 315 ++++++++------- content/blog/index.html | 39 +- content/blog/page10/index.html | 38 +- content/blog/page11/index.html | 38 +- content/blog/page12/index.html | 38 +- content/blog/page13/index.html | 40 +- content/blog/page14/index.html | 38 +- content/blog/page15/index.html | 37 +- content/blog/page16/index.html | 39 +- content/blog/page17/index.html | 38 +- content/blog/page18/index.html | 37 +- content/blog/page19/index.html | 39 +- content/blog/page2/index.html | 39 +- content/blog/page20/index.html | 25 ++ content/blog/page3/index.html | 36 +- content/blog/page4/index.html | 36 +- content/blog/page5/index.html | 38 +- content/blog/page6/index.html | 38 +- content/blog/page7/index.html | 36 +- content/blog/page8/index.html | 36 +- content/blog/page9/index.html | 36 +- content/index.html | 6 +- .../2022/07/12/release-ml-2.1.0.html} | 437 +++++++++------------ content/zh/index.html | 6 +- 24 files changed, 849 insertions(+), 656 deletions(-) diff --git a/content/blog/feed.xml b/content/blog/feed.xml index 728e4e401..fd81c8158 100644 --- a/content/blog/feed.xml +++ b/content/blog/feed.xml @@ -6,6 +6,181 @@ <link>https://flink.apache.org/blog</link> <atom:link href="https://flink.apache.org/blog/feed.xml" rel="self" type="application/rss+xml" /> +<item> +<title>Apache Flink ML 2.1.0 Release Announcement</title> +<description><p>The Apache Flink community is excited to announce the release of Flink ML 2.1.0! +This release focuses on improving Flink ML’s infrastructure, such as Python SDK, +memory management, and benchmark framework, to facilitate the development of +performant, memory-safe, and easy-to-use algorithm libraries. We validated the +enhanced infrastructure by implementing, benchmarking, and optimizing 10 new +algorithms in Flink ML, and confirmed that Flink ML can meet or exceed the +performance of selected algorithms from alternative popular ML libraries. +In addition, this release added example Python and Java programs for each +algorithm in the library to help users learn and use Flink ML.</p> + +<p>With the improvements and performance benchmarks made in this release, we +believe Flink ML’s infrastructure is ready for use by the interested developers +in the community to build performant pythonic machine learning libraries.</p> + +<p>We encourage you to <a href="https://flink.apache.org/downloads.html">download the release</a> +and share your feedback with the community through the Flink +<a href="https://flink.apache.org/community.html#mailing-lists">mailing lists</a> or +<a href="https://issues.apache.org/jira/browse/flink">JIRA</a>! We hope you like the new +release and we’d be eager to learn about your experience with it.</p> + +<div class="page-toc"> +<ul id="markdown-toc"> + <li><a href="#notable-features" id="markdown-toc-notable-features">Notable Features</a> <ul> + <li><a href="#api-and-infrastructure" id="markdown-toc-api-and-infrastructure">API and Infrastructure</a> <ul> + <li><a href="#supporting-fine-grained-per-operator-memory-management" id="markdown-toc-supporting-fine-grained-per-operator-memory-management">Supporting fine-grained per-operator memory management</a></li> + <li><a href="#improved-infrastructure-for-developing-online-learning-algorithms" id="markdown-toc-improved-infrastructure-for-developing-online-learning-algorithms">Improved infrastructure for developing online learning algorithms</a></li> + <li><a href="#algorithm-benchmark-framework" id="markdown-toc-algorithm-benchmark-framework">Algorithm benchmark framework</a></li> + </ul> + </li> + <li><a href="#python-sdk" id="markdown-toc-python-sdk">Python SDK</a></li> + <li><a href="#algorithm-library" id="markdown-toc-algorithm-library">Algorithm Library</a></li> + </ul> + </li> + <li><a href="#upgrade-notes" id="markdown-toc-upgrade-notes">Upgrade Notes</a></li> + <li><a href="#release-notes-and-resources" id="markdown-toc-release-notes-and-resources">Release Notes and Resources</a></li> + <li><a href="#list-of-contributors" id="markdown-toc-list-of-contributors">List of Contributors</a></li> +</ul> + +</div> + +<h1 id="notable-features">Notable Features</h1> + +<h2 id="api-and-infrastructure">API and Infrastructure</h2> + +<h3 id="supporting-fine-grained-per-operator-memory-management">Supporting fine-grained per-operator memory management</h3> + +<p>Before this release, algorithm operators with internal states (e.g. the training +data to be replayed for each round of iteration) store state data using the +state-backend API (e.g. ListState). Such an operator either needs to store all +data in memory, which risks OOM, or it needs to always store data on disk. +In the latter case, it needs to read and de-serialize all data from disks +repeatedly in each round of iteration even if the data can fit in RAM, leading +to sub-optimal performance when the training data size is small. This makes it +hard for developers to write performant and memory-safe operators.</p> + +<p>This release enhances the Flink ML infrastructure with the mechanism to specify +the amount of managed memory that an operator can consume. This allows algorithm +operators to write and read data from managed memory when the data size is below +the quota, and automatically spill those data that exceeds the memory quota to +disks to avoid OOM. Algorithm developers can use this mechanism to achieve +optimal algorithm performance as input data size varies. Please feel free to +check out the implementation of the KMeans operator for example.</p> + +<h3 id="improved-infrastructure-for-developing-online-learning-algorithms">Improved infrastructure for developing online learning algorithms</h3> + +<p>A key objective of Flink ML is to facilitate the development of online learning +applications. In the last release, we enhanced the Flink ML API with +setModelData() and getModelData(), which allows users of online learning +algorithms to transmit and persist model data as unbounded data streams. +This release continues the effort by improving and validating the infrastructure +needed to develop online learning algorithms.</p> + +<p>Specifically, this release added two online learning algorithm prototypes (i.e. +OnlineKMeans and OnlineLogisticRegression) with tests covering the entire +lifecycle of using these algorithms. These two algorithms introduce concepts +such as global batch size and model version, together with metrics and APIs to +set and get those values. While the online algorithm prototypes have not been +optimized for prediction accuracy yet, this line of work is an important step +toward setting up best practices for building online learning algorithms in +Flink ML. We hope more contributors from the community can join this effort.</p> + +<h3 id="algorithm-benchmark-framework">Algorithm benchmark framework</h3> + +<p>An easy-to-use benchmark framework is critical to developing and maintaining +performant algorithm libraries in Flink ML. This release added a benchmark +framework that provides APIs to write pluggable and reusable data generators, +takes benchmark configuration in JSON format, and outputs benchmark results in +JSON format to enable custom analysis. An off-the-shelf script is provided to +visualize benchmark results using Matplotlib. Feel free to check out this +<a href="https://github.com/apache/flink-ml/blob/release-2.1/flink-ml-benchmark/README.md">README</a> +for instructions on how to use this benchmark framework.</p> + +<p>The benchmark framework currently supports evaluating algorithm throughput. +In the future release, we plan to support evaluating algorithm latency and +accuracy.</p> + +<h2 id="python-sdk">Python SDK</h2> + +<p>This release enhances the Python SDK so that operators in the Flink ML Python +library can invoke the corresponding operators in the Java library. The Python +operator is a thin-wrapper around the Java operator and delivers the same +performance as the Java operator during execution. This capability significantly +improves developer velocity by allowing algorithm developers to maintain both +the Python and the Java libraries of algorithms without having to implement +those algorithms twice.</p> + +<h2 id="algorithm-library">Algorithm Library</h2> +<p>This release continues to extend the algorithm library in Flink ML, with the +focus on validating the functionalities and the performance of Flink ML +infrastructure using representative algorithms in different categories.</p> + +<p>Below are the lists of algorithms newly supported in this release, grouped by +their categories:</p> + +<ul> + <li>Feature engineering (MinMaxScaler, StringIndexer, VectorAssembler, +StandardScaler, Bucketizer)</li> + <li>Online learning (OnlineKmeans, OnlineLogisiticRegression)</li> + <li>Regression (LinearRegression)</li> + <li>Classification (LinearSVC)</li> + <li>Evaluation (BinaryClassificationEvaluator)</li> +</ul> + +<p>Example Python and Java programs for these algorithms are provided on the Apache +Flink ML <a href="https://nightlies.apache.org/flink/flink-ml-docs-release-2.1/">website</a> to +help users learn and try out Flink ML. And we also provided example benchmark +<a href="https://github.com/apache/flink-ml/tree/release-2.1/flink-ml-benchmark/src/main/resources">configuration files</a> +in the repo for users to validate Flink ML performance. Feel free to check out +this <a href="https://github.com/apache/flink-ml/blob/release-2.1/flink-ml-benchmark/README.md">README</a> +for instructions on how to run those benchmarks.</p> + +<h1 id="upgrade-notes">Upgrade Notes</h1> + +<p>Please review this note for a list of adjustments to make and issues to check +if you plan to upgrade to Flink ML 2.1.0.</p> + +<p>This note discusses any critical information about incompatibilities and +breaking changes, performance changes, and any other changes that might impact +your production deployment of Flink ML.</p> + +<ul> + <li> + <p><strong>Flink dependency is changed from 1.14 to 1.15</strong>.</p> + + <p>This change introduces all the breaking changes listed in the Flink 1.15 +<a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/release-notes/flink-1.15/">release notes</a>.</p> + </li> +</ul> + +<h1 id="release-notes-and-resources">Release Notes and Resources</h1> + +<p>Please take a look at the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&amp;version=12351141">release notes</a> +for a detailed list of changes and new features.</p> + +<p>The source artifacts is now available on the updated +<a href="https://flink.apache.org/downloads.html">Downloads page</a> of the Flink website, +and the most recent distribution of Flink ML Python package is available on +<a href="https://pypi.org/project/apache-flink-ml">PyPI</a>.</p> + +<h1 id="list-of-contributors">List of Contributors</h1> + +<p>The Apache Flink community would like to thank each one of the contributors +that have made this release possible:</p> + +<p>Yunfeng Zhou, Zhipeng Zhang, huangxingbo, weibo, Dong Lin, Yun Gao, Jingsong Li +and mumuhhh.</p> + +</description> +<pubDate>Tue, 12 Jul 2022 10:00:00 +0200</pubDate> +<link>https://flink.apache.org/news/2022/07/12/release-ml-2.1.0.html</link> +<guid isPermaLink="true">/news/2022/07/12/release-ml-2.1.0.html</guid> +</item> + <item> <title>FLIP-147: Support Checkpoints After Tasks Finished - Part Two</title> <description><p>In the <a href="/2022/07/11/final-checkpoint-part1.html">first part</a> of this blog, @@ -20191,145 +20366,5 @@ To understand why this is the case, let us start with articulating a realistic s <guid isPermaLink="true">/news/2020/01/15/demo-fraud-detection.html</guid> </item> -<item> -<title>Apache Flink 1.8.3 Released</title> -<description><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.8 series.</p> - -<p>This release includes 45 fixes and minor improvements for Flink 1.8.2. The list below includes a detailed list of all fixes and improvements.</p> - -<p>We highly recommend all users to upgrade to Flink 1.8.3.</p> - -<p>Updated Maven dependencies:</p> - -<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span> - <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span> - <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span> - <span class="nt">&lt;version&gt;</span>1.8.3<span class="nt">&lt;/version&gt;</span> -<span class="nt">&lt;/dependency&gt;</span> -<span class="nt">&lt;dependency&gt;</span> - <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span> - <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span> - <span class="nt">&lt;version&gt;</span>1.8.3<span class="nt">&lt;/version&gt;</span> -<span class="nt">&lt;/dependency&gt;</span> -<span class="nt">&lt;dependency&gt;</span> - <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span> - <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span> - <span class="nt">&lt;version&gt;</span>1.8.3<span class="nt">&lt;/version&gt;</span> -<span class="nt">&lt;/dependency&gt;</span></code></pre></div> - -<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p> - -<p>List of resolved issues:</p> - -<h2> Sub-task -</h2> -<ul> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13723">FLINK-13723</a>] - Use liquid-c for faster doc generation -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13724">FLINK-13724</a>] - Remove unnecessary whitespace from the docs&#39; sidenav -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13725">FLINK-13725</a>] - Use sassc for faster doc generation -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13726">FLINK-13726</a>] - Build docs with jekyll 4.0.0.pre.beta1 -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13791">FLINK-13791</a>] - Speed up sidenav by using group_by -</li> -</ul> - -<h2> Bug -</h2> -<ul> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12342">FLINK-12342</a>] - Yarn Resource Manager Acquires Too Many Containers -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13184">FLINK-13184</a>] - Starting a TaskExecutor blocks the YarnResourceManager&#39;s main thread -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13728">FLINK-13728</a>] - Fix wrong closing tag order in sidenav -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13746">FLINK-13746</a>] - Elasticsearch (v2.3.5) sink end-to-end test fails on Travis -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13749">FLINK-13749</a>] - Make Flink client respect classloading policy -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13892">FLINK-13892</a>] - HistoryServerTest failed on Travis -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13936">FLINK-13936</a>] - NOTICE-binary is outdated -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13966">FLINK-13966</a>] - Jar sorting in collect_license_files.sh is locale dependent -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13995">FLINK-13995</a>] - Fix shading of the licence information of netty -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13999">FLINK-13999</a>] - Correct the documentation of MATCH_RECOGNIZE -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14009">FLINK-14009</a>] - Cron jobs broken due to verifying incorrect NOTICE-binary file -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14010">FLINK-14010</a>] - Dispatcher &amp; JobManagers don&#39;t give up leadership when AM is shut down -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14043">FLINK-14043</a>] - SavepointMigrationTestBase is super slow -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14107">FLINK-14107</a>] - Kinesis consumer record emitter deadlock under event time alignment -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14175">FLINK-14175</a>] - Upgrade KPL version in flink-connector-kinesis to fix application OOM -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14235">FLINK-14235</a>] - Kafka010ProducerITCase&gt;KafkaProducerTestBase.testOneToOneAtLeastOnceCustomOperator fails on travis -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14315">FLINK-14315</a>] - NPE with JobMaster.disconnectTaskManager -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14337">FLINK-14337</a>] - HistoryServerTest.testHistoryServerIntegration failed on Travis -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14347">FLINK-14347</a>] - YARNSessionFIFOITCase.checkForProhibitedLogContents found a log with prohibited string -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14370">FLINK-14370</a>] - KafkaProducerAtLeastOnceITCase&gt;KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink fails on Travis -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14398">FLINK-14398</a>] - Further split input unboxing code into separate methods -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14413">FLINK-14413</a>] - shade-plugin ApacheNoticeResourceTransformer uses platform-dependent encoding -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14434">FLINK-14434</a>] - Dispatcher#createJobManagerRunner should not start JobManagerRunner -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14562">FLINK-14562</a>] - RMQSource leaves idle consumer after closing -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14589">FLINK-14589</a>] - Redundant slot requests with the same AllocationID leads to inconsistent slot table -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15036">FLINK-15036</a>] - Container startup error will be handled out side of the YarnResourceManager&#39;s main thread -</li> -</ul> - -<h2> Improvement -</h2> -<ul> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12848">FLINK-12848</a>] - Method equals() in RowTypeInfo should consider fieldsNames -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13729">FLINK-13729</a>] - Update website generation dependencies -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13965">FLINK-13965</a>] - Keep hasDeprecatedKeys and deprecatedKeys methods in ConfigOption and mark it with @Deprecated annotation -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13967">FLINK-13967</a>] - Generate full binary licensing via collect_license_files.sh -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13968">FLINK-13968</a>] - Add travis check for the correctness of the binary licensing -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13991">FLINK-13991</a>] - Add git exclusion for 1.9+ features to 1.8 -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14008">FLINK-14008</a>] - Auto-generate binary licensing -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14104">FLINK-14104</a>] - Bump Jackson to 2.10.1 -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14123">FLINK-14123</a>] - Lower the default value of taskmanager.memory.fraction -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14215">FLINK-14215</a>] - Add Docs for TM and JM Environment Variable Setting -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14334">FLINK-14334</a>] - ElasticSearch docs refer to non-existent ExceptionUtils.containsThrowable -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14639">FLINK-14639</a>] - Fix the document of Metrics that has an error for `User Scope` -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14646">FLINK-14646</a>] - Check non-null for key in KeyGroupStreamPartitioner -</li> -<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14995">FLINK-14995</a>] - Kinesis NOTICE is incorrect -</li> -</ul> -</description> -<pubDate>Wed, 11 Dec 2019 13:00:00 +0100</pubDate> -<link>https://flink.apache.org/news/2019/12/11/release-1.8.3.html</link> -<guid isPermaLink="true">/news/2019/12/11/release-1.8.3.html</guid> -</item> - </channel> </rss> diff --git a/content/blog/index.html b/content/blog/index.html index 5cf6dc9fe..98b5705c5 100644 --- a/content/blog/index.html +++ b/content/blog/index.html @@ -236,6 +236,19 @@ <div class="col-sm-8"> <!-- Blog posts --> + <article> + <h2 class="blog-title"><a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></h2> + + <p>12 Jul 2022 + Zhipeng Zhang & Dong Lin </p> + + <p>The Apache Flink community is excited to announce the release of Flink ML 2.1.0! This release focuses on improving Flink ML's infrastructure, such as Python SDK, memory management, and benchmark framework, to facilitate the development of performant, memory-safe, and easy-to-use algorithm libraries. We validated the enhanced infrastructure via benchmarks and confirmed that Flink ML can meet or exceed the performance of selected algorithms from alternative popular ML libraries. I [...] + + <p><a href="/news/2022/07/12/release-ml-2.1.0.html">Continue reading »</a></p> + </article> + + <hr> + <article> <h2 class="blog-title"><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></h2> @@ -356,22 +369,6 @@ We are now proud to announce the first production ready release of the operator <hr> - <article> - <h2 class="blog-title"><a href="/news/2022/05/11/release-table-store-0.1.0.html">Apache Flink Table Store 0.1.0 Release Announcement</a></h2> - - <p>11 May 2022 - Jingsong Lee & Jiangjie (Becket) Qin </p> - - <p><p>The Apache Flink community is pleased to announce the preview release of the -<a href="https://github.com/apache/flink-table-store">Apache Flink Table Store</a> (0.1.0).</p> - -</p> - - <p><a href="/news/2022/05/11/release-table-store-0.1.0.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -404,6 +401,16 @@ We are now proud to announce the first production ready release of the operator <ul id="markdown-toc"> + <li><a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li> diff --git a/content/blog/page10/index.html b/content/blog/page10/index.html index 1fe66f6f1..6e31b511a 100644 --- a/content/blog/page10/index.html +++ b/content/blog/page10/index.html @@ -236,6 +236,19 @@ <div class="col-sm-8"> <!-- Blog posts --> + <article> + <h2 class="blog-title"><a href="/news/2020/04/01/community-update.html">Flink Community Update - April'20</a></h2> + + <p>01 Apr 2020 + Marta Paes (<a href="https://twitter.com/morsapaes">@morsapaes</a>)</p> + + <p>While things slow down around us, the Apache Flink community is privileged to remain as active as ever. This blogpost combs through the past few months to give you an update on the state of things in Flink — from core releases to Stateful Functions; from some good old community stats to a new development blog.</p> + + <p><a href="/news/2020/04/01/community-update.html">Continue reading »</a></p> + </article> + + <hr> + <article> <h2 class="blog-title"><a href="/features/2020/03/27/flink-for-data-warehouse.html">Flink as Unified Engine for Modern Data Warehousing: Production-Ready Hive Integration</a></h2> @@ -359,21 +372,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/news/2019/12/11/release-1.8.3.html">Apache Flink 1.8.3 Released</a></h2> - - <p>11 Dec 2019 - Hequn Cheng </p> - - <p><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.8 series.</p> - -</p> - - <p><a href="/news/2019/12/11/release-1.8.3.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -406,6 +404,16 @@ <ul id="markdown-toc"> + <li><a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li> diff --git a/content/blog/page11/index.html b/content/blog/page11/index.html index 8cde1aa21..b02a0ffbd 100644 --- a/content/blog/page11/index.html +++ b/content/blog/page11/index.html @@ -236,6 +236,21 @@ <div class="col-sm-8"> <!-- Blog posts --> + <article> + <h2 class="blog-title"><a href="/news/2019/12/11/release-1.8.3.html">Apache Flink 1.8.3 Released</a></h2> + + <p>11 Dec 2019 + Hequn Cheng </p> + + <p><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.8 series.</p> + +</p> + + <p><a href="/news/2019/12/11/release-1.8.3.html">Continue reading »</a></p> + </article> + + <hr> + <article> <h2 class="blog-title"><a href="/news/2019/12/09/flink-kubernetes-kudo.html">Running Apache Flink on Kubernetes with KUDO</a></h2> @@ -362,19 +377,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/2019/06/26/broadcast-state.html">A Practical Guide to Broadcast State in Apache Flink</a></h2> - - <p>26 Jun 2019 - Fabian Hueske (<a href="https://twitter.com/fhueske">@fhueske</a>)</p> - - <p>Apache Flink has multiple types of operator state, one of which is called Broadcast State. In this post, we explain what Broadcast State is, and show an example of how it can be applied to an application that evaluates dynamic patterns on an event stream.</p> - - <p><a href="/2019/06/26/broadcast-state.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -407,6 +409,16 @@ <ul id="markdown-toc"> + <li><a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li> diff --git a/content/blog/page12/index.html b/content/blog/page12/index.html index 86ebcc36f..e32e495e4 100644 --- a/content/blog/page12/index.html +++ b/content/blog/page12/index.html @@ -236,6 +236,19 @@ <div class="col-sm-8"> <!-- Blog posts --> + <article> + <h2 class="blog-title"><a href="/2019/06/26/broadcast-state.html">A Practical Guide to Broadcast State in Apache Flink</a></h2> + + <p>26 Jun 2019 + Fabian Hueske (<a href="https://twitter.com/fhueske">@fhueske</a>)</p> + + <p>Apache Flink has multiple types of operator state, one of which is called Broadcast State. In this post, we explain what Broadcast State is, and show an example of how it can be applied to an application that evaluates dynamic patterns on an event stream.</p> + + <p><a href="/2019/06/26/broadcast-state.html">Continue reading »</a></p> + </article> + + <hr> + <article> <h2 class="blog-title"><a href="/2019/06/05/flink-network-stack.html">A Deep-Dive into Flink's Network Stack</a></h2> @@ -361,21 +374,6 @@ for more details.</p> <hr> - <article> - <h2 class="blog-title"><a href="/news/2019/02/25/release-1.6.4.html">Apache Flink 1.6.4 Released</a></h2> - - <p>25 Feb 2019 - </p> - - <p><p>The Apache Flink community released the fourth bugfix version of the Apache Flink 1.6 series.</p> - -</p> - - <p><a href="/news/2019/02/25/release-1.6.4.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -408,6 +406,16 @@ for more details.</p> <ul id="markdown-toc"> + <li><a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li> diff --git a/content/blog/page13/index.html b/content/blog/page13/index.html index 04776eff4..36540a1c7 100644 --- a/content/blog/page13/index.html +++ b/content/blog/page13/index.html @@ -236,6 +236,21 @@ <div class="col-sm-8"> <!-- Blog posts --> + <article> + <h2 class="blog-title"><a href="/news/2019/02/25/release-1.6.4.html">Apache Flink 1.6.4 Released</a></h2> + + <p>25 Feb 2019 + </p> + + <p><p>The Apache Flink community released the fourth bugfix version of the Apache Flink 1.6 series.</p> + +</p> + + <p><a href="/news/2019/02/25/release-1.6.4.html">Continue reading »</a></p> + </article> + + <hr> + <article> <h2 class="blog-title"><a href="/news/2019/02/15/release-1.7.2.html">Apache Flink 1.7.2 Released</a></h2> @@ -371,21 +386,6 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa <hr> - <article> - <h2 class="blog-title"><a href="/news/2018/09/20/release-1.5.4.html">Apache Flink 1.5.4 Released</a></h2> - - <p>20 Sep 2018 - </p> - - <p><p>The Apache Flink community released the fourth bugfix version of the Apache Flink 1.5 series.</p> - -</p> - - <p><a href="/news/2018/09/20/release-1.5.4.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -418,6 +418,16 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa <ul id="markdown-toc"> + <li><a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li> diff --git a/content/blog/page14/index.html b/content/blog/page14/index.html index 0a78a343a..23bf25df7 100644 --- a/content/blog/page14/index.html +++ b/content/blog/page14/index.html @@ -236,6 +236,21 @@ <div class="col-sm-8"> <!-- Blog posts --> + <article> + <h2 class="blog-title"><a href="/news/2018/09/20/release-1.5.4.html">Apache Flink 1.5.4 Released</a></h2> + + <p>20 Sep 2018 + </p> + + <p><p>The Apache Flink community released the fourth bugfix version of the Apache Flink 1.5 series.</p> + +</p> + + <p><a href="/news/2018/09/20/release-1.5.4.html">Continue reading »</a></p> + </article> + + <hr> + <article> <h2 class="blog-title"><a href="/news/2018/08/21/release-1.5.3.html">Apache Flink 1.5.3 Released</a></h2> @@ -369,19 +384,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/features/2018/01/30/incremental-checkpointing.html">Managing Large State in Apache Flink: An Intro to Incremental Checkpointing</a></h2> - - <p>30 Jan 2018 - Stefan Ricther (<a href="https://twitter.com/StefanRRicther">@StefanRRicther</a>) & Chris Ward (<a href="https://twitter.com/chrischinch">@chrischinch</a>)</p> - - <p>Flink 1.3.0 introduced incremental checkpointing, making it possible for applications with large state to generate checkpoints more efficiently.</p> - - <p><a href="/features/2018/01/30/incremental-checkpointing.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -414,6 +416,16 @@ <ul id="markdown-toc"> + <li><a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li> diff --git a/content/blog/page15/index.html b/content/blog/page15/index.html index cbbe6d1e9..0ee4a1f48 100644 --- a/content/blog/page15/index.html +++ b/content/blog/page15/index.html @@ -236,6 +236,19 @@ <div class="col-sm-8"> <!-- Blog posts --> + <article> + <h2 class="blog-title"><a href="/features/2018/01/30/incremental-checkpointing.html">Managing Large State in Apache Flink: An Intro to Incremental Checkpointing</a></h2> + + <p>30 Jan 2018 + Stefan Ricther (<a href="https://twitter.com/StefanRRicther">@StefanRRicther</a>) & Chris Ward (<a href="https://twitter.com/chrischinch">@chrischinch</a>)</p> + + <p>Flink 1.3.0 introduced incremental checkpointing, making it possible for applications with large state to generate checkpoints more efficiently.</p> + + <p><a href="/features/2018/01/30/incremental-checkpointing.html">Continue reading »</a></p> + </article> + + <hr> + <article> <h2 class="blog-title"><a href="/news/2017/12/21/2017-year-in-review.html">Apache Flink in 2017: Year in Review</a></h2> @@ -372,20 +385,6 @@ what’s coming in Flink 1.4.0 as well as a preview of what the Flink community <hr> - <article> - <h2 class="blog-title"><a href="/news/2017/04/04/dynamic-tables.html">Continuous Queries on Dynamic Tables</a></h2> - - <p>04 Apr 2017 by Fabian Hueske, Shaoxuan Wang, and Xiaowei Jiang - </p> - - <p><p>Flink's relational APIs, the Table API and SQL, are unified APIs for stream and batch processing, meaning that a query produces the same result when being evaluated on streaming or static data.</p> -<p>In this blog post we discuss the future of these APIs and introduce the concept of Dynamic Tables. Dynamic tables will significantly expand the scope of the Table API and SQL on streams and enable many more advanced use cases. We discuss how streams and dynamic tables relate to each other and explain the semantics of continuously evaluating queries on dynamic tables.</p></p> - - <p><a href="/news/2017/04/04/dynamic-tables.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -418,6 +417,16 @@ what’s coming in Flink 1.4.0 as well as a preview of what the Flink community <ul id="markdown-toc"> + <li><a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li> diff --git a/content/blog/page16/index.html b/content/blog/page16/index.html index b28e3acb3..64d74c744 100644 --- a/content/blog/page16/index.html +++ b/content/blog/page16/index.html @@ -236,6 +236,20 @@ <div class="col-sm-8"> <!-- Blog posts --> + <article> + <h2 class="blog-title"><a href="/news/2017/04/04/dynamic-tables.html">Continuous Queries on Dynamic Tables</a></h2> + + <p>04 Apr 2017 by Fabian Hueske, Shaoxuan Wang, and Xiaowei Jiang + </p> + + <p><p>Flink's relational APIs, the Table API and SQL, are unified APIs for stream and batch processing, meaning that a query produces the same result when being evaluated on streaming or static data.</p> +<p>In this blog post we discuss the future of these APIs and introduce the concept of Dynamic Tables. Dynamic tables will significantly expand the scope of the Table API and SQL on streams and enable many more advanced use cases. We discuss how streams and dynamic tables relate to each other and explain the semantics of continuously evaluating queries on dynamic tables.</p></p> + + <p><a href="/news/2017/04/04/dynamic-tables.html">Continue reading »</a></p> + </article> + + <hr> + <article> <h2 class="blog-title"><a href="/news/2017/03/29/table-sql-api-update.html">From Streams to Tables and Back Again: An Update on Flink's Table & SQL API</a></h2> @@ -365,21 +379,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/news/2016/08/08/release-1.1.0.html">Announcing Apache Flink 1.1.0</a></h2> - - <p>08 Aug 2016 - </p> - - <p><div class="alert alert-success"><strong>Important</strong>: The Maven artifacts published with version 1.1.0 on Maven central have a Hadoop dependency issue. It is highly recommended to use <strong>1.1.1</strong> or <strong>1.1.1-hadoop1</strong> as the Flink version.</div> - -</p> - - <p><a href="/news/2016/08/08/release-1.1.0.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -412,6 +411,16 @@ <ul id="markdown-toc"> + <li><a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li> diff --git a/content/blog/page17/index.html b/content/blog/page17/index.html index f899f993c..2806e7327 100644 --- a/content/blog/page17/index.html +++ b/content/blog/page17/index.html @@ -236,6 +236,21 @@ <div class="col-sm-8"> <!-- Blog posts --> + <article> + <h2 class="blog-title"><a href="/news/2016/08/08/release-1.1.0.html">Announcing Apache Flink 1.1.0</a></h2> + + <p>08 Aug 2016 + </p> + + <p><div class="alert alert-success"><strong>Important</strong>: The Maven artifacts published with version 1.1.0 on Maven central have a Hadoop dependency issue. It is highly recommended to use <strong>1.1.1</strong> or <strong>1.1.1-hadoop1</strong> as the Flink version.</div> + +</p> + + <p><a href="/news/2016/08/08/release-1.1.0.html">Continue reading »</a></p> + </article> + + <hr> + <article> <h2 class="blog-title"><a href="/news/2016/05/24/stream-sql.html">Stream Processing for Everyone with SQL and Apache Flink</a></h2> @@ -366,19 +381,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/news/2015/12/11/storm-compatibility.html">Storm Compatibility in Apache Flink: How to run existing Storm topologies on Flink</a></h2> - - <p>11 Dec 2015 by Matthias J. Sax (<a href="https://twitter.com/">@MatthiasJSax</a>) - </p> - - <p>In this blog post, we describe Flink's compatibility package for <a href="https://storm.apache.org">Apache Storm</a> that allows to embed Spouts (sources) and Bolts (operators) in a regular Flink streaming job. Furthermore, the compatibility package provides a Storm compatible API in order to execute whole Storm topologies with (almost) no code adaption.</p> - - <p><a href="/news/2015/12/11/storm-compatibility.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -411,6 +413,16 @@ <ul id="markdown-toc"> + <li><a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li> diff --git a/content/blog/page18/index.html b/content/blog/page18/index.html index 6e8dfdea0..3a340a0ee 100644 --- a/content/blog/page18/index.html +++ b/content/blog/page18/index.html @@ -236,6 +236,19 @@ <div class="col-sm-8"> <!-- Blog posts --> + <article> + <h2 class="blog-title"><a href="/news/2015/12/11/storm-compatibility.html">Storm Compatibility in Apache Flink: How to run existing Storm topologies on Flink</a></h2> + + <p>11 Dec 2015 by Matthias J. Sax (<a href="https://twitter.com/">@MatthiasJSax</a>) + </p> + + <p>In this blog post, we describe Flink's compatibility package for <a href="https://storm.apache.org">Apache Storm</a> that allows to embed Spouts (sources) and Bolts (operators) in a regular Flink streaming job. Furthermore, the compatibility package provides a Storm compatible API in order to execute whole Storm topologies with (almost) no code adaption.</p> + + <p><a href="/news/2015/12/11/storm-compatibility.html">Continue reading »</a></p> + </article> + + <hr> + <article> <h2 class="blog-title"><a href="/news/2015/12/04/Introducing-windows.html">Introducing Stream Windows in Apache Flink</a></h2> @@ -374,20 +387,6 @@ vertex-centric or gather-sum-apply to Flink dataflows.</p> <hr> - <article> - <h2 class="blog-title"><a href="/news/2015/05/11/Juggling-with-Bits-and-Bytes.html">Juggling with Bits and Bytes</a></h2> - - <p>11 May 2015 by Fabian Hüske (<a href="https://twitter.com/">@fhueske</a>) - </p> - - <p><p>Nowadays, a lot of open-source systems for analyzing large data sets are implemented in Java or other JVM-based programming languages. The most well-known example is Apache Hadoop, but also newer frameworks such as Apache Spark, Apache Drill, and also Apache Flink run on JVMs. A common challenge that JVM-based data analysis engines face is to store large amounts of data in memory - both for caching and for efficient processing such as sorting and joining of data. Managing the [...] -<p>In this blog post we discuss how Apache Flink manages memory, talk about its custom data de/serialization stack, and show how it operates on binary data.</p></p> - - <p><a href="/news/2015/05/11/Juggling-with-Bits-and-Bytes.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -420,6 +419,16 @@ vertex-centric or gather-sum-apply to Flink dataflows.</p> <ul id="markdown-toc"> + <li><a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li> diff --git a/content/blog/page19/index.html b/content/blog/page19/index.html index b0f84bd30..1a1288def 100644 --- a/content/blog/page19/index.html +++ b/content/blog/page19/index.html @@ -236,6 +236,20 @@ <div class="col-sm-8"> <!-- Blog posts --> + <article> + <h2 class="blog-title"><a href="/news/2015/05/11/Juggling-with-Bits-and-Bytes.html">Juggling with Bits and Bytes</a></h2> + + <p>11 May 2015 by Fabian Hüske (<a href="https://twitter.com/">@fhueske</a>) + </p> + + <p><p>Nowadays, a lot of open-source systems for analyzing large data sets are implemented in Java or other JVM-based programming languages. The most well-known example is Apache Hadoop, but also newer frameworks such as Apache Spark, Apache Drill, and also Apache Flink run on JVMs. A common challenge that JVM-based data analysis engines face is to store large amounts of data in memory - both for caching and for efficient processing such as sorting and joining of data. Managing the [...] +<p>In this blog post we discuss how Apache Flink manages memory, talk about its custom data de/serialization stack, and show how it operates on binary data.</p></p> + + <p><a href="/news/2015/05/11/Juggling-with-Bits-and-Bytes.html">Continue reading »</a></p> + </article> + + <hr> + <article> <h2 class="blog-title"><a href="/news/2015/04/13/release-0.9.0-milestone1.html">Announcing Flink 0.9.0-milestone1 preview release</a></h2> @@ -381,21 +395,6 @@ and offers a new API including definition of flexible windows.</p> <hr> - <article> - <h2 class="blog-title"><a href="/news/2014/11/04/release-0.7.0.html">Apache Flink 0.7.0 available</a></h2> - - <p>04 Nov 2014 - </p> - - <p><p>We are pleased to announce the availability of Flink 0.7.0. This release includes new user-facing features as well as performance and bug fixes, brings the Scala and Java APIs in sync, and introduces Flink Streaming. A total of 34 people have contributed to this release, a big thanks to all of them!</p> - -</p> - - <p><a href="/news/2014/11/04/release-0.7.0.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -428,6 +427,16 @@ and offers a new API including definition of flexible windows.</p> <ul id="markdown-toc"> + <li><a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li> diff --git a/content/blog/page2/index.html b/content/blog/page2/index.html index 22d3dc865..8f80f5edb 100644 --- a/content/blog/page2/index.html +++ b/content/blog/page2/index.html @@ -236,6 +236,22 @@ <div class="col-sm-8"> <!-- Blog posts --> + <article> + <h2 class="blog-title"><a href="/news/2022/05/11/release-table-store-0.1.0.html">Apache Flink Table Store 0.1.0 Release Announcement</a></h2> + + <p>11 May 2022 + Jingsong Lee & Jiangjie (Becket) Qin </p> + + <p><p>The Apache Flink community is pleased to announce the preview release of the +<a href="https://github.com/apache/flink-table-store">Apache Flink Table Store</a> (0.1.0).</p> + +</p> + + <p><a href="/news/2022/05/11/release-table-store-0.1.0.html">Continue reading »</a></p> + </article> + + <hr> + <article> <h2 class="blog-title"><a href="/2022/05/06/async-sink-base.html">The Generic Asynchronous Base Sink</a></h2> @@ -364,19 +380,6 @@ This new release brings various improvements to the StateFun runtime, a leaner w <hr> - <article> - <h2 class="blog-title"><a href="/2022/01/20/pravega-connector-101.html">Pravega Flink Connector 101</a></h2> - - <p>20 Jan 2022 - Yumin Zhou (Brian) (<a href="https://twitter.com/crazy__zhou">@crazy__zhou</a>)</p> - - <p>A brief introduction to the Pravega Flink Connector</p> - - <p><a href="/2022/01/20/pravega-connector-101.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -409,6 +412,16 @@ This new release brings various improvements to the StateFun runtime, a leaner w <ul id="markdown-toc"> + <li><a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li> diff --git a/content/blog/page20/index.html b/content/blog/page20/index.html index c03fed09c..58553cc62 100644 --- a/content/blog/page20/index.html +++ b/content/blog/page20/index.html @@ -236,6 +236,21 @@ <div class="col-sm-8"> <!-- Blog posts --> + <article> + <h2 class="blog-title"><a href="/news/2014/11/04/release-0.7.0.html">Apache Flink 0.7.0 available</a></h2> + + <p>04 Nov 2014 + </p> + + <p><p>We are pleased to announce the availability of Flink 0.7.0. This release includes new user-facing features as well as performance and bug fixes, brings the Scala and Java APIs in sync, and introduces Flink Streaming. A total of 34 people have contributed to this release, a big thanks to all of them!</p> + +</p> + + <p><a href="/news/2014/11/04/release-0.7.0.html">Continue reading »</a></p> + </article> + + <hr> + <article> <h2 class="blog-title"><a href="/news/2014/10/03/upcoming_events.html">Upcoming Events</a></h2> @@ -316,6 +331,16 @@ academic and open source project that Flink originates from.</p> <ul id="markdown-toc"> + <li><a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li> diff --git a/content/blog/page3/index.html b/content/blog/page3/index.html index aecdb160b..ebc7afbec 100644 --- a/content/blog/page3/index.html +++ b/content/blog/page3/index.html @@ -236,6 +236,19 @@ <div class="col-sm-8"> <!-- Blog posts --> + <article> + <h2 class="blog-title"><a href="/2022/01/20/pravega-connector-101.html">Pravega Flink Connector 101</a></h2> + + <p>20 Jan 2022 + Yumin Zhou (Brian) (<a href="https://twitter.com/crazy__zhou">@crazy__zhou</a>)</p> + + <p>A brief introduction to the Pravega Flink Connector</p> + + <p><a href="/2022/01/20/pravega-connector-101.html">Continue reading »</a></p> + </article> + + <hr> + <article> <h2 class="blog-title"><a href="/news/2022/01/17/release-1.14.3.html">Apache Flink 1.14.3 Release Announcement</a></h2> @@ -357,19 +370,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/2021/10/26/sort-shuffle-part1.html">Sort-Based Blocking Shuffle Implementation in Flink - Part One</a></h2> - - <p>26 Oct 2021 - Yingjie Cao (Kevin) & Daisy Tsang </p> - - <p>Flink has implemented the sort-based blocking shuffle (FLIP-148) for batch data processing. In this blog post, we will take a close look at the design & implementation details and see what we can gain from it.</p> - - <p><a href="/2021/10/26/sort-shuffle-part1.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -402,6 +402,16 @@ <ul id="markdown-toc"> + <li><a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li> diff --git a/content/blog/page4/index.html b/content/blog/page4/index.html index 2aa2e7aed..1c5ad1dfd 100644 --- a/content/blog/page4/index.html +++ b/content/blog/page4/index.html @@ -236,6 +236,19 @@ <div class="col-sm-8"> <!-- Blog posts --> + <article> + <h2 class="blog-title"><a href="/2021/10/26/sort-shuffle-part1.html">Sort-Based Blocking Shuffle Implementation in Flink - Part One</a></h2> + + <p>26 Oct 2021 + Yingjie Cao (Kevin) & Daisy Tsang </p> + + <p>Flink has implemented the sort-based blocking shuffle (FLIP-148) for batch data processing. In this blog post, we will take a close look at the design & implementation details and see what we can gain from it.</p> + + <p><a href="/2021/10/26/sort-shuffle-part1.html">Continue reading »</a></p> + </article> + + <hr> + <article> <h2 class="blog-title"><a href="/news/2021/10/19/release-1.13.3.html">Apache Flink 1.13.3 Released</a></h2> @@ -377,19 +390,6 @@ This new release brings various improvements to the StateFun runtime, a leaner w <hr> - <article> - <h2 class="blog-title"><a href="/2021/07/07/backpressure.html">How to identify the source of backpressure?</a></h2> - - <p>07 Jul 2021 - Piotr Nowojski (<a href="https://twitter.com/PiotrNowojski">@PiotrNowojski</a>)</p> - - <p>Apache Flink 1.13 introduced a couple of important changes in the area of backpressure monitoring and performance analysis of Flink Jobs. This blog post aims to introduce those changes and explain how to use them.</p> - - <p><a href="/2021/07/07/backpressure.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -422,6 +422,16 @@ This new release brings various improvements to the StateFun runtime, a leaner w <ul id="markdown-toc"> + <li><a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li> diff --git a/content/blog/page5/index.html b/content/blog/page5/index.html index 233bf67ff..8eaf9abad 100644 --- a/content/blog/page5/index.html +++ b/content/blog/page5/index.html @@ -236,6 +236,19 @@ <div class="col-sm-8"> <!-- Blog posts --> + <article> + <h2 class="blog-title"><a href="/2021/07/07/backpressure.html">How to identify the source of backpressure?</a></h2> + + <p>07 Jul 2021 + Piotr Nowojski (<a href="https://twitter.com/PiotrNowojski">@PiotrNowojski</a>)</p> + + <p>Apache Flink 1.13 introduced a couple of important changes in the area of backpressure monitoring and performance analysis of Flink Jobs. This blog post aims to introduce those changes and explain how to use them.</p> + + <p><a href="/2021/07/07/backpressure.html">Continue reading »</a></p> + </article> + + <hr> + <article> <h2 class="blog-title"><a href="/news/2021/05/28/release-1.13.1.html">Apache Flink 1.13.1 Released</a></h2> @@ -365,21 +378,6 @@ to develop scalable, consistent, and elastic distributed applications.</p> <hr> - <article> - <h2 class="blog-title"><a href="/news/2021/01/29/release-1.10.3.html">Apache Flink 1.10.3 Released</a></h2> - - <p>29 Jan 2021 - Xintong Song </p> - - <p><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.10 series.</p> - -</p> - - <p><a href="/news/2021/01/29/release-1.10.3.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -412,6 +410,16 @@ to develop scalable, consistent, and elastic distributed applications.</p> <ul id="markdown-toc"> + <li><a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li> diff --git a/content/blog/page6/index.html b/content/blog/page6/index.html index f916b36b9..d497c1c4e 100644 --- a/content/blog/page6/index.html +++ b/content/blog/page6/index.html @@ -236,6 +236,21 @@ <div class="col-sm-8"> <!-- Blog posts --> + <article> + <h2 class="blog-title"><a href="/news/2021/01/29/release-1.10.3.html">Apache Flink 1.10.3 Released</a></h2> + + <p>29 Jan 2021 + Xintong Song </p> + + <p><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.10 series.</p> + +</p> + + <p><a href="/news/2021/01/29/release-1.10.3.html">Continue reading »</a></p> + </article> + + <hr> + <article> <h2 class="blog-title"><a href="/news/2021/01/19/release-1.12.1.html">Apache Flink 1.12.1 Released</a></h2> @@ -361,19 +376,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/2020/10/15/from-aligned-to-unaligned-checkpoints-part-1.html">From Aligned to Unaligned Checkpoints - Part 1: Checkpoints, Alignment, and Backpressure</a></h2> - - <p>15 Oct 2020 - Arvid Heise & Stephan Ewen </p> - - <p>Apache Flink’s checkpoint-based fault tolerance mechanism is one of its defining features. Because of that design, Flink unifies batch and stream processing, can easily scale to both very small and extremely large scenarios and provides support for many operational features. In this post we recap the original checkpointing process in Flink, its core properties and issues under backpressure.</p> - - <p><a href="/2020/10/15/from-aligned-to-unaligned-checkpoints-part-1.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -406,6 +408,16 @@ <ul id="markdown-toc"> + <li><a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li> diff --git a/content/blog/page7/index.html b/content/blog/page7/index.html index 6dee27caf..b69436623 100644 --- a/content/blog/page7/index.html +++ b/content/blog/page7/index.html @@ -236,6 +236,19 @@ <div class="col-sm-8"> <!-- Blog posts --> + <article> + <h2 class="blog-title"><a href="/2020/10/15/from-aligned-to-unaligned-checkpoints-part-1.html">From Aligned to Unaligned Checkpoints - Part 1: Checkpoints, Alignment, and Backpressure</a></h2> + + <p>15 Oct 2020 + Arvid Heise & Stephan Ewen </p> + + <p>Apache Flink’s checkpoint-based fault tolerance mechanism is one of its defining features. Because of that design, Flink unifies batch and stream processing, can easily scale to both very small and extremely large scenarios and provides support for many operational features. In this post we recap the original checkpointing process in Flink, its core properties and issues under backpressure.</p> + + <p><a href="/2020/10/15/from-aligned-to-unaligned-checkpoints-part-1.html">Continue reading »</a></p> + </article> + + <hr> + <article> <h2 class="blog-title"><a href="/news/2020/10/13/stateful-serverless-internals.html">Stateful Functions Internals: Behind the scenes of Stateful Serverless</a></h2> @@ -365,19 +378,6 @@ as well as increased observability for operational purposes.</p> <hr> - <article> - <h2 class="blog-title"><a href="/2020/08/04/pyflink-pandas-udf-support-flink.html">PyFlink: The integration of Pandas into PyFlink</a></h2> - - <p>04 Aug 2020 - Jincheng Sun (<a href="https://twitter.com/sunjincheng121">@sunjincheng121</a>) & Markos Sfikas (<a href="https://twitter.com/MarkSfik">@MarkSfik</a>)</p> - - <p>The Apache Flink community put some great effort into integrating Pandas with PyFlink in the latest Flink version 1.11. Some of the added features include support for Pandas UDF and the conversion between Pandas DataFrame and Table. In this article, we will introduce how these functionalities work and how to use them with a step-by-step example.</p> - - <p><a href="/2020/08/04/pyflink-pandas-udf-support-flink.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -410,6 +410,16 @@ as well as increased observability for operational purposes.</p> <ul id="markdown-toc"> + <li><a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li> diff --git a/content/blog/page8/index.html b/content/blog/page8/index.html index f7491c4f6..06584f373 100644 --- a/content/blog/page8/index.html +++ b/content/blog/page8/index.html @@ -236,6 +236,19 @@ <div class="col-sm-8"> <!-- Blog posts --> + <article> + <h2 class="blog-title"><a href="/2020/08/04/pyflink-pandas-udf-support-flink.html">PyFlink: The integration of Pandas into PyFlink</a></h2> + + <p>04 Aug 2020 + Jincheng Sun (<a href="https://twitter.com/sunjincheng121">@sunjincheng121</a>) & Markos Sfikas (<a href="https://twitter.com/MarkSfik">@MarkSfik</a>)</p> + + <p>The Apache Flink community put some great effort into integrating Pandas with PyFlink in the latest Flink version 1.11. Some of the added features include support for Pandas UDF and the conversion between Pandas DataFrame and Table. In this article, we will introduce how these functionalities work and how to use them with a step-by-step example.</p> + + <p><a href="/2020/08/04/pyflink-pandas-udf-support-flink.html">Continue reading »</a></p> + </article> + + <hr> + <article> <h2 class="blog-title"><a href="/news/2020/07/30/demo-fraud-detection-3.html">Advanced Flink Application Patterns Vol.3: Custom Window Processing</a></h2> @@ -371,19 +384,6 @@ and provide a tutorial for running Streaming ETL with Flink on Zeppelin.</p> <hr> - <article> - <h2 class="blog-title"><a href="/news/2020/06/11/community-update.html">Flink Community Update - June'20</a></h2> - - <p>11 Jun 2020 - Marta Paes (<a href="https://twitter.com/morsapaes">@morsapaes</a>)</p> - - <p>And suddenly it’s June. The previous month has been calm on the surface, but quite hectic underneath — the final testing phase for Flink 1.11 is moving at full speed, Stateful Functions 2.1 is out in the wild and Flink has made it into Google Season of Docs 2020.</p> - - <p><a href="/news/2020/06/11/community-update.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -416,6 +416,16 @@ and provide a tutorial for running Streaming ETL with Flink on Zeppelin.</p> <ul id="markdown-toc"> + <li><a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li> diff --git a/content/blog/page9/index.html b/content/blog/page9/index.html index 802dc3bb3..73aa6f1c4 100644 --- a/content/blog/page9/index.html +++ b/content/blog/page9/index.html @@ -236,6 +236,19 @@ <div class="col-sm-8"> <!-- Blog posts --> + <article> + <h2 class="blog-title"><a href="/news/2020/06/11/community-update.html">Flink Community Update - June'20</a></h2> + + <p>11 Jun 2020 + Marta Paes (<a href="https://twitter.com/morsapaes">@morsapaes</a>)</p> + + <p>And suddenly it’s June. The previous month has been calm on the surface, but quite hectic underneath — the final testing phase for Flink 1.11 is moving at full speed, Stateful Functions 2.1 is out in the wild and Flink has made it into Google Season of Docs 2020.</p> + + <p><a href="/news/2020/06/11/community-update.html">Continue reading »</a></p> + </article> + + <hr> + <article> <h2 class="blog-title"><a href="/news/2020/06/09/release-statefun-2.1.0.html">Stateful Functions 2.1.0 Release Announcement</a></h2> @@ -362,19 +375,6 @@ This release marks a big milestone: Stateful Functions 2.0 is not only an API up <hr> - <article> - <h2 class="blog-title"><a href="/news/2020/04/01/community-update.html">Flink Community Update - April'20</a></h2> - - <p>01 Apr 2020 - Marta Paes (<a href="https://twitter.com/morsapaes">@morsapaes</a>)</p> - - <p>While things slow down around us, the Apache Flink community is privileged to remain as active as ever. This blogpost combs through the past few months to give you an update on the state of things in Flink — from core releases to Stateful Functions; from some good old community stats to a new development blog.</p> - - <p><a href="/news/2020/04/01/community-update.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -407,6 +407,16 @@ This release marks a big milestone: Stateful Functions 2.0 is not only an API up <ul id="markdown-toc"> + <li><a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></li> + + + + + + + + + <li><a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></li> diff --git a/content/index.html b/content/index.html index 9659e791d..c40650b74 100644 --- a/content/index.html +++ b/content/index.html @@ -401,6 +401,9 @@ <dl> + <dt> <a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></dt> + <dd>The Apache Flink community is excited to announce the release of Flink ML 2.1.0! This release focuses on improving Flink ML's infrastructure, such as Python SDK, memory management, and benchmark framework, to facilitate the development of performant, memory-safe, and easy-to-use algorithm libraries. We validated the enhanced infrastructure via benchmarks and confirmed that Flink ML can meet or exceed the performance of selected algorithms from alternative popular ML libraries [...] + <dt> <a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></dt> <dd>This post presents more details on the changes on the checkpoint procedure and task finish process made by the final checkpoint mechanism.</dd> @@ -412,9 +415,6 @@ <dt> <a href="/news/2022/06/22/release-1.14.5.html">Apache Flink 1.14.5 Release Announcement</a></dt> <dd>The Apache Flink Community is pleased to announce another bug fix release for Flink 1.14.</dd> - - <dt> <a href="/2022/06/17/adaptive-batch-scheduler.html">Adaptive Batch Scheduler: Automatically Decide Parallelism of Flink Batch Jobs</a></dt> - <dd>To automatically decide parallelism of Flink batch jobs, we introduced adaptive batch scheduler in Flink 1.15. In this post, we'll take a close look at the design & implementation details.</dd> </dl> diff --git a/content/index.html b/content/news/2022/07/12/release-ml-2.1.0.html similarity index 54% copy from content/index.html copy to content/news/2022/07/12/release-ml-2.1.0.html index 9659e791d..e8b15bcd4 100644 --- a/content/index.html +++ b/content/news/2022/07/12/release-ml-2.1.0.html @@ -5,7 +5,7 @@ <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1"> <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags --> - <title>Apache Flink: Stateful Computations over Data Streams</title> + <title>Apache Flink: Apache Flink ML 2.1.0 Release Announcement</title> <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"> <link rel="icon" href="/favicon.ico" type="image/x-icon"> @@ -149,7 +149,7 @@ <li><a href="/gettinghelp.html">Getting Help</a></li> <!-- Blog --> - <li><a href="/blog/"><b>Flink Blog</b></a></li> + <li class="active"><a href="/blog/"><b>Flink Blog</b></a></li> <!-- Flink-packages --> @@ -181,7 +181,8 @@ <li> - <a href="/zh/">中文版</a> + <!-- link to the Chinese home page when current is blog page --> + <a href="/zh">中文版</a> </li> @@ -229,257 +230,201 @@ </div> <div class="col-sm-9"> <div class="row-fluid"> - <div class="col-sm-12"> - <p class="lead"> - <strong>Apache Flink<sup>®</sup> — Stateful Computations over Data Streams</strong> - </p> - </div> - -<div class="col-sm-12"> - <hr /> -</div> - -</div> - -<!-- High-level architecture figure --> + <div class="row"> + <h1>Apache Flink ML 2.1.0 Release Announcement</h1> + <p><i></i></p> + + <article> + <p>12 Jul 2022 Zhipeng Zhang & Dong Lin </p> + +<p>The Apache Flink community is excited to announce the release of Flink ML 2.1.0! +This release focuses on improving Flink ML’s infrastructure, such as Python SDK, +memory management, and benchmark framework, to facilitate the development of +performant, memory-safe, and easy-to-use algorithm libraries. We validated the +enhanced infrastructure by implementing, benchmarking, and optimizing 10 new +algorithms in Flink ML, and confirmed that Flink ML can meet or exceed the +performance of selected algorithms from alternative popular ML libraries. +In addition, this release added example Python and Java programs for each +algorithm in the library to help users learn and use Flink ML.</p> + +<p>With the improvements and performance benchmarks made in this release, we +believe Flink ML’s infrastructure is ready for use by the interested developers +in the community to build performant pythonic machine learning libraries.</p> + +<p>We encourage you to <a href="https://flink.apache.org/downloads.html">download the release</a> +and share your feedback with the community through the Flink +<a href="https://flink.apache.org/community.html#mailing-lists">mailing lists</a> or +<a href="https://issues.apache.org/jira/browse/flink">JIRA</a>! We hope you like the new +release and we’d be eager to learn about your experience with it.</p> + +<div class="page-toc"> +<ul id="markdown-toc"> + <li><a href="#notable-features" id="markdown-toc-notable-features">Notable Features</a> <ul> + <li><a href="#api-and-infrastructure" id="markdown-toc-api-and-infrastructure">API and Infrastructure</a> <ul> + <li><a href="#supporting-fine-grained-per-operator-memory-management" id="markdown-toc-supporting-fine-grained-per-operator-memory-management">Supporting fine-grained per-operator memory management</a></li> + <li><a href="#improved-infrastructure-for-developing-online-learning-algorithms" id="markdown-toc-improved-infrastructure-for-developing-online-learning-algorithms">Improved infrastructure for developing online learning algorithms</a></li> + <li><a href="#algorithm-benchmark-framework" id="markdown-toc-algorithm-benchmark-framework">Algorithm benchmark framework</a></li> + </ul> + </li> + <li><a href="#python-sdk" id="markdown-toc-python-sdk">Python SDK</a></li> + <li><a href="#algorithm-library" id="markdown-toc-algorithm-library">Algorithm Library</a></li> + </ul> + </li> + <li><a href="#upgrade-notes" id="markdown-toc-upgrade-notes">Upgrade Notes</a></li> + <li><a href="#release-notes-and-resources" id="markdown-toc-release-notes-and-resources">Release Notes and Resources</a></li> + <li><a href="#list-of-contributors" id="markdown-toc-list-of-contributors">List of Contributors</a></li> +</ul> -<div class="row front-graphic"> - <hr /> - <img src="/img/flink-home-graphic.png" width="800px" /> </div> -<!-- Feature grid --> - -<!-- -<div class="row"> - <div class="col-sm-12"> - <hr /> - <h2><a href="/features.html">Features</a></h2> - </div> -</div> ---> -<div class="row"> - <div class="col-sm-4"> - <div class="panel panel-default"> - <div class="panel-heading"> - <span class="glyphicon glyphicon-th"></span> <b>All streaming use cases</b> - </div> - <div class="panel-body"> - <ul style="font-size: small;"> - <li>Event-driven Applications</li> - <li>Stream & Batch Analytics</li> - <li>Data Pipelines & ETL</li> - </ul> - <a href="/usecases.html">Learn more</a> - </div> - </div> - </div> - <div class="col-sm-4"> - <div class="panel panel-default"> - <div class="panel-heading"> - <span class="glyphicon glyphicon-ok"></span> <b>Guaranteed correctness</b> - </div> - <div class="panel-body"> - <ul style="font-size: small;"> - <li>Exactly-once state consistency</li> - <li>Event-time processing</li> - <li>Sophisticated late data handling</li> - </ul> - <a href="/flink-applications.html#building-blocks-for-streaming-applications">Learn more</a> - </div> - </div> - </div> - <div class="col-sm-4"> - <div class="panel panel-default"> - <div class="panel-heading"> - <span class="glyphicon glyphicon glyphicon-sort-by-attributes"></span> <b>Layered APIs</b> - </div> - <div class="panel-body"> - <ul style="font-size: small;"> - <li>SQL on Stream & Batch Data</li> - <li>DataStream API & DataSet API</li> - <li>ProcessFunction (Time & State)</li> - </ul> - <a href="/flink-applications.html#layered-apis">Learn more</a> - </div> - </div> - </div> -</div> -<div class="row"> - <div class="col-sm-4"> - <div class="panel panel-default"> - <div class="panel-heading"> - <span class="glyphicon glyphicon-dashboard"></span> <b>Operational Focus</b> - </div> - <div class="panel-body"> - <ul style="font-size: small;"> - <li>Flexible deployment</li> - <li>High-availability setup</li> - <li>Savepoints</li> - </ul> - <a href="/flink-operations.html">Learn more</a> - </div> - </div> - </div> - <div class="col-sm-4"> - <div class="panel panel-default"> - <div class="panel-heading"> - <span class="glyphicon glyphicon-fullscreen"></span> <b>Scales to any use case</b> - </div> - <div class="panel-body"> - <ul style="font-size: small;"> - <li>Scale-out architecture</li> - <li>Support for very large state</li> - <li>Incremental checkpointing</li> - </ul> - <a href="/flink-architecture.html#run-applications-at-any-scale">Learn more</a> - </div> +<h1 id="notable-features">Notable Features</h1> + +<h2 id="api-and-infrastructure">API and Infrastructure</h2> + +<h3 id="supporting-fine-grained-per-operator-memory-management">Supporting fine-grained per-operator memory management</h3> + +<p>Before this release, algorithm operators with internal states (e.g. the training +data to be replayed for each round of iteration) store state data using the +state-backend API (e.g. ListState). Such an operator either needs to store all +data in memory, which risks OOM, or it needs to always store data on disk. +In the latter case, it needs to read and de-serialize all data from disks +repeatedly in each round of iteration even if the data can fit in RAM, leading +to sub-optimal performance when the training data size is small. This makes it +hard for developers to write performant and memory-safe operators.</p> + +<p>This release enhances the Flink ML infrastructure with the mechanism to specify +the amount of managed memory that an operator can consume. This allows algorithm +operators to write and read data from managed memory when the data size is below +the quota, and automatically spill those data that exceeds the memory quota to +disks to avoid OOM. Algorithm developers can use this mechanism to achieve +optimal algorithm performance as input data size varies. Please feel free to +check out the implementation of the KMeans operator for example.</p> + +<h3 id="improved-infrastructure-for-developing-online-learning-algorithms">Improved infrastructure for developing online learning algorithms</h3> + +<p>A key objective of Flink ML is to facilitate the development of online learning +applications. In the last release, we enhanced the Flink ML API with +setModelData() and getModelData(), which allows users of online learning +algorithms to transmit and persist model data as unbounded data streams. +This release continues the effort by improving and validating the infrastructure +needed to develop online learning algorithms.</p> + +<p>Specifically, this release added two online learning algorithm prototypes (i.e. +OnlineKMeans and OnlineLogisticRegression) with tests covering the entire +lifecycle of using these algorithms. These two algorithms introduce concepts +such as global batch size and model version, together with metrics and APIs to +set and get those values. While the online algorithm prototypes have not been +optimized for prediction accuracy yet, this line of work is an important step +toward setting up best practices for building online learning algorithms in +Flink ML. We hope more contributors from the community can join this effort.</p> + +<h3 id="algorithm-benchmark-framework">Algorithm benchmark framework</h3> + +<p>An easy-to-use benchmark framework is critical to developing and maintaining +performant algorithm libraries in Flink ML. This release added a benchmark +framework that provides APIs to write pluggable and reusable data generators, +takes benchmark configuration in JSON format, and outputs benchmark results in +JSON format to enable custom analysis. An off-the-shelf script is provided to +visualize benchmark results using Matplotlib. Feel free to check out this +<a href="https://github.com/apache/flink-ml/blob/release-2.1/flink-ml-benchmark/README.md">README</a> +for instructions on how to use this benchmark framework.</p> + +<p>The benchmark framework currently supports evaluating algorithm throughput. +In the future release, we plan to support evaluating algorithm latency and +accuracy.</p> + +<h2 id="python-sdk">Python SDK</h2> + +<p>This release enhances the Python SDK so that operators in the Flink ML Python +library can invoke the corresponding operators in the Java library. The Python +operator is a thin-wrapper around the Java operator and delivers the same +performance as the Java operator during execution. This capability significantly +improves developer velocity by allowing algorithm developers to maintain both +the Python and the Java libraries of algorithms without having to implement +those algorithms twice.</p> + +<h2 id="algorithm-library">Algorithm Library</h2> +<p>This release continues to extend the algorithm library in Flink ML, with the +focus on validating the functionalities and the performance of Flink ML +infrastructure using representative algorithms in different categories.</p> + +<p>Below are the lists of algorithms newly supported in this release, grouped by +their categories:</p> + +<ul> + <li>Feature engineering (MinMaxScaler, StringIndexer, VectorAssembler, +StandardScaler, Bucketizer)</li> + <li>Online learning (OnlineKmeans, OnlineLogisiticRegression)</li> + <li>Regression (LinearRegression)</li> + <li>Classification (LinearSVC)</li> + <li>Evaluation (BinaryClassificationEvaluator)</li> +</ul> + +<p>Example Python and Java programs for these algorithms are provided on the Apache +Flink ML <a href="https://nightlies.apache.org/flink/flink-ml-docs-release-2.1/">website</a> to +help users learn and try out Flink ML. And we also provided example benchmark +<a href="https://github.com/apache/flink-ml/tree/release-2.1/flink-ml-benchmark/src/main/resources">configuration files</a> +in the repo for users to validate Flink ML performance. Feel free to check out +this <a href="https://github.com/apache/flink-ml/blob/release-2.1/flink-ml-benchmark/README.md">README</a> +for instructions on how to run those benchmarks.</p> + +<h1 id="upgrade-notes">Upgrade Notes</h1> + +<p>Please review this note for a list of adjustments to make and issues to check +if you plan to upgrade to Flink ML 2.1.0.</p> + +<p>This note discusses any critical information about incompatibilities and +breaking changes, performance changes, and any other changes that might impact +your production deployment of Flink ML.</p> + +<ul> + <li> + <p><strong>Flink dependency is changed from 1.14 to 1.15</strong>.</p> + + <p>This change introduces all the breaking changes listed in the Flink 1.15 +<a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/release-notes/flink-1.15/">release notes</a>.</p> + </li> +</ul> + +<h1 id="release-notes-and-resources">Release Notes and Resources</h1> + +<p>Please take a look at the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351141">release notes</a> +for a detailed list of changes and new features.</p> + +<p>The source artifacts is now available on the updated +<a href="https://flink.apache.org/downloads.html">Downloads page</a> of the Flink website, +and the most recent distribution of Flink ML Python package is available on +<a href="https://pypi.org/project/apache-flink-ml">PyPI</a>.</p> + +<h1 id="list-of-contributors">List of Contributors</h1> + +<p>The Apache Flink community would like to thank each one of the contributors +that have made this release possible:</p> + +<p>Yunfeng Zhou, Zhipeng Zhang, huangxingbo, weibo, Dong Lin, Yun Gao, Jingsong Li +and mumuhhh.</p> + + + </article> </div> - </div> - <div class="col-sm-4"> - <div class="panel panel-default"> - <div class="panel-heading"> - <span class="glyphicon glyphicon-flash"></span> <b>Excellent Performance</b> - </div> - <div class="panel-body"> - <ul style="font-size: small;"> - <li>Low latency</li> - <li>High throughput</li> - <li>In-Memory computing</li> - </ul> - <a href="/flink-architecture.html#leverage-in-memory-performance">Learn more</a> - </div> + + <div class="row"> + <div id="disqus_thread"></div> + <script type="text/javascript"> + /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */ + var disqus_shortname = 'stratosphere-eu'; // required: replace example with your forum shortname + + /* * * DON'T EDIT BELOW THIS LINE * * */ + (function() { + var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; + dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js'; + (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); + })(); + </script> </div> </div> </div> - -<!-- Events section --> -<div class="row"> - -<div class="col-sm-12"> - <hr /> -</div> - -<div class="col-sm-3"> - - <h2><a>Upcoming Events</a></h2> - -</div> -<div class="col-sm-9"> - <!-- Flink Forward --> - <a href="https://flink-forward.org" target="_blank"> - <img style="width: 180px; padding-right: 10px" src="/img/flink-forward.png" alt="Flink Forward" /> - </a> - <!-- ApacheCon --> - <a href="https://www.apache.org/events/current-event" target="_blank"> - <img style="width: 200px; padding-right: 10px" src="https://www.apache.org/events/current-event-234x60.png" alt="ApacheCon" /> - </a> - <!-- Flink Forward Asia --> - <a href="https://flink-forward.org.cn/" target="_blank"> - <img style="width: 230px" src="/img/flink-forward-asia.png" alt="Flink Forward Asia" /> - </a> -</div> - -</div> - -<!-- Updates section --> - -<div class="row"> - -<div class="col-sm-12"> - <hr /> -</div> - -<div class="col-sm-3"> - - <h2><a href="/blog">Latest Blog Posts</a></h2> - -</div> - -<div class="col-sm-9"> - - <dl> - - <dt> <a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></dt> - <dd>This post presents more details on the changes on the checkpoint procedure and task finish process made by the final checkpoint mechanism.</dd> - - <dt> <a href="/2022/07/11/final-checkpoint-part1.html">FLIP-147: Support Checkpoints After Tasks Finished - Part One</a></dt> - <dd>This post briefly describes the motivation and changes made by the final checkpoint mechanism, including the changes to the checkpoint procedure and how tasks finish.</dd> - - <dt> <a href="/news/2022/07/06/release-1.15.1.html">Apache Flink 1.15.1 Release Announcement</a></dt> - <dd>The Apache Flink Community is pleased to announce a bug fix release for Flink 1.15.</dd> - - <dt> <a href="/news/2022/06/22/release-1.14.5.html">Apache Flink 1.14.5 Release Announcement</a></dt> - <dd>The Apache Flink Community is pleased to announce another bug fix release for Flink 1.14.</dd> - - <dt> <a href="/2022/06/17/adaptive-batch-scheduler.html">Adaptive Batch Scheduler: Automatically Decide Parallelism of Flink Batch Jobs</a></dt> - <dd>To automatically decide parallelism of Flink batch jobs, we introduced adaptive batch scheduler in Flink 1.15. In this post, we'll take a close look at the design & implementation details.</dd> - - </dl> - -</div> - -<!-- Scripts section --> - -<script type="text/javascript" src="/js/jquery.jcarousel.min.js"></script> - -<script type="text/javascript"> - - $(window).load(function(){ - $(function() { - var jcarousel = $('.jcarousel'); - - jcarousel - .on('jcarousel:reload jcarousel:create', function () { - var carousel = $(this), - width = carousel.innerWidth(); - - if (width >= 600) { - width = width / 4; - } else if (width >= 350) { - width = width / 3; - } - - carousel.jcarousel('items').css('width', Math.ceil(width) + 'px'); - }) - .jcarousel({ - wrap: 'circular', - autostart: true - }); - - $('.jcarousel-control-prev') - .jcarouselControl({ - target: '-=1' - }); - - $('.jcarousel-control-next') - .jcarouselControl({ - target: '+=1' - }); - - $('.jcarousel-pagination') - .on('jcarouselpagination:active', 'a', function() { - $(this).addClass('active'); - }) - .on('jcarouselpagination:inactive', 'a', function() { - $(this).removeClass('active'); - }) - .on('click', function(e) { - e.preventDefault(); - }) - .jcarouselPagination({ - perPage: 1, - item: function(page) { - return '<a href="#' + page + '">' + page + '</a>'; - } - }); - }); - }); - -</script> -</div> - </div> </div> diff --git a/content/zh/index.html b/content/zh/index.html index a5bb8e0b3..19927fd19 100644 --- a/content/zh/index.html +++ b/content/zh/index.html @@ -398,6 +398,9 @@ <dl> + <dt> <a href="/news/2022/07/12/release-ml-2.1.0.html">Apache Flink ML 2.1.0 Release Announcement</a></dt> + <dd>The Apache Flink community is excited to announce the release of Flink ML 2.1.0! This release focuses on improving Flink ML's infrastructure, such as Python SDK, memory management, and benchmark framework, to facilitate the development of performant, memory-safe, and easy-to-use algorithm libraries. We validated the enhanced infrastructure via benchmarks and confirmed that Flink ML can meet or exceed the performance of selected algorithms from alternative popular ML libraries [...] + <dt> <a href="/2022/07/11/final-checkpoint-part2.html">FLIP-147: Support Checkpoints After Tasks Finished - Part Two</a></dt> <dd>This post presents more details on the changes on the checkpoint procedure and task finish process made by the final checkpoint mechanism.</dd> @@ -409,9 +412,6 @@ <dt> <a href="/news/2022/06/22/release-1.14.5.html">Apache Flink 1.14.5 Release Announcement</a></dt> <dd>The Apache Flink Community is pleased to announce another bug fix release for Flink 1.14.</dd> - - <dt> <a href="/2022/06/17/adaptive-batch-scheduler.html">Adaptive Batch Scheduler: Automatically Decide Parallelism of Flink Batch Jobs</a></dt> - <dd>To automatically decide parallelism of Flink batch jobs, we introduced adaptive batch scheduler in Flink 1.15. In this post, we'll take a close look at the design & implementation details.</dd> </dl>