This is an automated email from the ASF dual-hosted git repository. nkruber pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit 6a50d296127bf0fdc2cae7863cfc8b801343414f Author: Nico Kruber <nico.kru...@gmail.com> AuthorDate: Tue Apr 14 17:32:53 2020 +0200 Rebuild website --- content/blog/feed.xml | 387 ++++++++++---- content/blog/index.html | 38 +- content/blog/page10/index.html | 43 +- content/blog/page11/index.html | 25 + content/blog/page2/index.html | 41 +- content/blog/page3/index.html | 39 +- content/blog/page4/index.html | 38 +- content/blog/page5/index.html | 40 +- content/blog/page6/index.html | 40 +- content/blog/page7/index.html | 40 +- content/blog/page8/index.html | 40 +- content/blog/page9/index.html | 43 +- ...-15-flink-serialization-performance-results.svg | 1 + content/index.html | 6 +- .../04/15/flink-serialization-tuning-vol-1.html | 555 +++++++++++++++++++++ content/zh/index.html | 6 +- 16 files changed, 1138 insertions(+), 244 deletions(-) diff --git a/content/blog/feed.xml b/content/blog/feed.xml index 02f6ac5..126b62b 100644 --- a/content/blog/feed.xml +++ b/content/blog/feed.xml @@ -7,6 +7,307 @@ <atom:link href="https://flink.apache.org/blog/feed.xml" rel="self" type="application/rss+xml" /> <item> +<title>Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</title> +<description><p>Almost every Flink job has to exchange data between its operators and since these records may not only be sent to another instance in the same JVM but instead to a separate process, records need to be serialized to bytes first. Similarly, Flink’s off-heap state-backend is based on a local embedded RocksDB instance which is implemented in native C++ code and thus also needs transformation into bytes on every state access. Wire and state serialization alone can easily [...] + +<p>Since serialization is so crucial to your Flink job, we would like to highlight Flink’s serialization stack in a series of blog posts starting with looking at the different ways Flink can serialize your data types.</p> + +<div class="page-toc"> +<ul id="markdown-toc"> + <li><a href="#recap-flink-serialization" id="markdown-toc-recap-flink-serialization">Recap: Flink Serialization</a></li> + <li><a href="#choice-of-serializer" id="markdown-toc-choice-of-serializer">Choice of Serializer</a> <ul> + <li><a href="#pojoserializer" id="markdown-toc-pojoserializer">PojoSerializer</a></li> + <li><a href="#tuple-data-types" id="markdown-toc-tuple-data-types">Tuple Data Types</a></li> + <li><a href="#row-data-types" id="markdown-toc-row-data-types">Row Data Types</a></li> + <li><a href="#avro" id="markdown-toc-avro">Avro</a> <ul> + <li><a href="#avro-specific" id="markdown-toc-avro-specific">Avro Specific</a></li> + <li><a href="#avro-generic" id="markdown-toc-avro-generic">Avro Generic</a></li> + <li><a href="#avro-reflect" id="markdown-toc-avro-reflect">Avro Reflect</a></li> + </ul> + </li> + <li><a href="#kryo" id="markdown-toc-kryo">Kryo</a> <ul> + <li><a href="#disabling-kryo" id="markdown-toc-disabling-kryo">Disabling Kryo</a></li> + </ul> + </li> + <li><a href="#apache-thrift-via-kryo" id="markdown-toc-apache-thrift-via-kryo">Apache Thrift (via Kryo)</a></li> + <li><a href="#protobuf-via-kryo" id="markdown-toc-protobuf-via-kryo">Protobuf (via Kryo)</a></li> + </ul> + </li> + <li><a href="#state-schema-evolution" id="markdown-toc-state-schema-evolution">State Schema Evolution</a></li> + <li><a href="#performance-comparison" id="markdown-toc-performance-comparison">Performance Comparison</a></li> + <li><a href="#conclusion" id="markdown-toc-conclusion">Conclusion</a></li> +</ul> + +</div> + +<h1 id="recap-flink-serialization">Recap: Flink Serialization</h1> + +<p>Flink handles <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/types_serialization.html">data types and serialization</a> with its own type descriptors, generic type extraction, and type serialization framework. We recommend reading through the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/types_serialization.html">documentation</a> first in order to be able to follow the arguments w [...] +<code>stream.keyBy(“ruleId”)</code> or +<code>dataSet.join(another).where("name").equalTo("personName")</code>. It also allows optimizations in the serialization format as well as reducing unnecessary de/serializations (mainly in certain Batch operations as well as in the SQL/Table APIs).</p> + +<h1 id="choice-of-serializer">Choice of Serializer</h1> + +<p>Apache Flink’s out-of-the-box serialization can be roughly divided into the following groups:</p> + +<ul> + <li> + <p><strong>Flink-provided special serializers</strong> for basic types (Java primitives and their boxed form), arrays, composite types (tuples, Scala case classes, Rows), and a few auxiliary types (Option, Either, Lists, Maps, …),</p> + </li> + <li> + <p><strong>POJOs</strong>; a public, standalone class with a public no-argument constructor and all non-static, non-transient fields in the class hierarchy either public or with a public getter- and a setter-method; see <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/types_serialization.html#rules-for-pojo-types">POJO Rules</a>,</p> + </li> + <li> + <p><strong>Generic types</strong>; user-defined data types that are not recognized as a POJO and then serialized via <a href="https://github.com/EsotericSoftware/kryo">Kryo</a>.</p> + </li> +</ul> + +<p>Alternatively, you can also register <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/custom_serializers.html">custom serializers</a> for user-defined data types. This includes writing your own serializers or integrating other serialization systems like <a href="https://developers.google.com/protocol-buffers/">Google Protobuf</a> or <a href="https://thrift.apache.org/">Apache Thrift</a> vi [...] + +<h2 id="pojoserializer">PojoSerializer</h2> + +<p>As outlined above, if your data type is not covered by a specialized serializer but follows the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/types_serialization.html#rules-for-pojo-types">POJO Rules</a>, it will be serialized with the <a href="https://github.com/apache/flink/blob/release-1.10.0/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/PojoSerializer.java">PojoSerializer</a> which [...] + +<blockquote> + <p>15:45:51,460 INFO org.apache.flink.api.java.typeutils.TypeExtractor - Class … cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on “Data Types &amp; Serialization” for details of the effect on performance.</p> +</blockquote> + +<p>This means, that the PojoSerializer will not be used, but instead Flink will fall back to Kryo for serialization (see below). We will have a more detailed look into a few (more) situations that can lead to unexpected Kryo fallbacks in the second part of this blog post series.</p> + +<h2 id="tuple-data-types">Tuple Data Types</h2> + +<p>Flink comes with a predefined set of tuple types which all have a fixed length and contain a set of strongly-typed fields of potentially different types. There are implementations for <code>Tuple0</code>, <code>Tuple1&lt;T0&gt;</code>, …, <code>Tuple25&lt;T0, T1, ..., T24&gt;</code> and they may serve as easy-to-use wrappers that spare the creation of POJOs for each and every combination of objects you need to pass between comp [...] + +<div class="alert alert-info"> + <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span> + Since <code>Tuple0</code> does not contain any data and therefore is probably a bit special anyway, it will use a special serializer implementation: <a href="https://github.com/apache/flink/blob/release-1.10.0/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/Tuple0Serializer.java">Tuple0Serializer</a>.</p> +</div> + +<h2 id="row-data-types">Row Data Types</h2> + +<p>Row types are mainly used by the Table and SQL APIs of Flink. A <code>Row</code> groups an arbitrary number of objects together similar to the tuples above. These fields are not strongly typed and may all be of different types. Because field types are missing, Flink’s type extraction cannot automatically extract type information and users of a <code>Row</code> need to manually tell Flink about the row’s field types. The <a href="https://github.com [...] + +<p>Row type information can be provided in two ways:</p> + +<ul> + <li>you can have your source or operator implement <code>ResultTypeQueryable&lt;Row&gt;</code>:</li> +</ul> + +<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">RowSource</span> <span class="kd">implements</span> <span class="n">SourceFunction</span><span class="o">&lt;</span><span class= [...] + <span class="c1">// ...</span> + + <span class="nd">@Override</span> + <span class="kd">public</span> <span class="n">TypeInformation</span><span class="o">&lt;</span><span class="n">Row</span><span class="o">&gt;</span> <span class="nf">getProducedType</span><span class="o">()</span> <span class="o">{</span> + <span class="k">return</span> <span class="n">Types</span><span class="o">.</span><span class="na">ROW</span><span class="o">(</span><span class="n">Types</span><span class="o">.</span><span class="na">INT</span><span class="o">,</span> <span class="n">Types</span>< [...] + <span class="o">}</span> +<span class="o">}</span></code></pre></div> + +<ul> + <li>you can provide the types when building the job graph by using <code>SingleOutputStreamOperator#returns()</code></li> +</ul> + +<div class="highlight"><pre><code class="language-java"><span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Row</span><span class="o">&gt;</span> <span class="n">sourceStream</span> <span class="o">=</span> + <span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o">(</span><span class="k">new</span> <span class="nf">RowSource</span><span class="o">())</span> + <span class="o">.</span><span class="na">returns</span><span class="o">(</span><span class="n">Types</span><span class="o">.</span><span class="na">ROW</span><span class="o">(</span><span class="n">Types</span><span class="o">.</span><span class="na">INT</span>< [...] + +<div class="alert alert-warning"> + <p><span class="label label-warning" style="display: inline-block"><span class="glyphicon glyphicon-warning-sign" aria-hidden="true"></span> Warning</span> +If you fail to provide the type information for a <code>Row</code>, Flink identifies that <code>Row</code> is not a valid POJO type according to the rules above and falls back to Kryo serialization (see below) which you will also see in the logs as:</p> + + <p><code>13:10:11,148 INFO org.apache.flink.api.java.typeutils.TypeExtractor - Class class org.apache.flink.types.Row cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types &amp; Serialization" for details of the effect on performance.</code></p> +</div> + +<h2 id="avro">Avro</h2> + +<p>Flink offers built-in support for the <a href="http://avro.apache.org/">Apache Avro</a> serialization framework (currently using version 1.8.2) by adding the <code>org.apache.flink:flink-avro</code> dependency into your job. Flink’s <a href="https://github.com/apache/flink/blob/release-1.10.0/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/AvroSerializer.java">AvroSerializer</a> can then use A [...] + +<h3 id="avro-specific">Avro Specific</h3> + +<p>Avro specific records will be automatically detected by checking that the given type’s type hierarchy contains the <code>SpecificRecordBase</code> class. You can either specify your concrete Avro type, or—if you want to be more generic and allow different types in your operator—use the <code>SpecificRecordBase</code> type (or a subtype) in your user functions, in <code>ResultTypeQueryable#getProducedType()</code>, or in <code>SingleOutpu [...] + +<div class="alert alert-warning"> + <p><span class="label label-warning" style="display: inline-block"><span class="glyphicon glyphicon-warning-sign" aria-hidden="true"></span> Warning</span> If you specify the Flink type as <code>SpecificRecord</code> and not <code>SpecificRecordBase</code>, Flink will not see this as an Avro type. Instead, it will use Kryo to de/serialize any objects which may be considerably slower.</p> +</div> + +<h3 id="avro-generic">Avro Generic</h3> + +<p>Avro’s <code>GenericRecord</code> types cannot, unfortunately, be used automatically since they require the user to <a href="https://avro.apache.org/docs/1.8.2/gettingstartedjava.html#Serializing+and+deserializing+without+code+generation">specify a schema</a> (either manually or by retrieving it from some schema registry). With that schema, you can provide the right type information by either of the following options just like for the Row Types [...] + +<ul> + <li>implement <code>ResultTypeQueryable&lt;GenericRecord&gt;</code>:</li> +</ul> + +<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">AvroGenericSource</span> <span class="kd">implements</span> <span class="n">SourceFunction</span><span class="o">&lt;</span><spa [...] + <span class="kd">private</span> <span class="kd">final</span> <span class="n">GenericRecordAvroTypeInfo</span> <span class="n">producedType</span><span class="o">;</span> + + <span class="kd">public</span> <span class="nf">AvroGenericSource</span><span class="o">(</span><span class="n">Schema</span> <span class="n">schema</span><span class="o">)</span> <span class="o">{</span> + <span class="k">this</span><span class="o">.</span><span class="na">producedType</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">GenericRecordAvroTypeInfo</span><span class="o">(</span><span class="n">schema</span><span class="o">);</span> + <span class="o">}</span> + + <span class="nd">@Override</span> + <span class="kd">public</span> <span class="n">TypeInformation</span><span class="o">&lt;</span><span class="n">GenericRecord</span><span class="o">&gt;</span> <span class="nf">getProducedType</span><span class="o">()</span> <span class="o">{</span> + <span class="k">return</span> <span class="n">producedType</span><span class="o">;</span> + <span class="o">}</span> +<span class="o">}</span></code></pre></div> +<ul> + <li>provide type information when building the job graph by using <code>SingleOutputStreamOperator#returns()</code></li> +</ul> + +<div class="highlight"><pre><code class="language-java"><span class="n">DataStream</span><span class="o">&lt;</span><span class="n">GenericRecord</span><span class="o">&gt;</span> <span class="n">sourceStream</span> <span class="o">=</span> + <span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o">(</span><span class="k">new</span> <span class="nf">AvroGenericSource</span><span class="o">())</span> + <span class="o">.</span><span class="na">returns</span><span class="o">(</span><span class="k">new</span> <span class="nf">GenericRecordAvroTypeInfo</span><span class="o">(</span><span class="n">schema</span><span class="o">));</span></code></pre></div> +<p>Without this type information, Flink will fall back to Kryo for serialization which would serialize the schema into every record, over and over again. As a result, the serialized form will be bigger and more costly to create.</p> + +<div class="alert alert-info"> + <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span> + Since Avro’s <code>Schema</code> class is not serializable, it can not be sent around as is. You can work around this by converting it to a String and parsing it back when needed. If you only do this once on initialization, there is practically no difference to sending it directly.</p> +</div> + +<h3 id="avro-reflect">Avro Reflect</h3> + +<p>The third way of using Avro is to exchange Flink’s PojoSerializer (for POJOs according to the rules above) for Avro’s reflection-based serializer. This can be enabled by calling</p> + +<div class="highlight"><pre><code class="language-java"><span class="n">env</span><span class="o">.</span><span class="na">getConfig</span><span class="o">().</span><span class="na">enableForceAvro</span><span class="o">();</span></code></pre></div> + +<h2 id="kryo">Kryo</h2> + +<p>Any class or object which does not fall into the categories above or is covered by a Flink-provided special serializer is de/serialized with a fallback to <a href="https://github.com/EsotericSoftware/kryo">Kryo</a> (currently version 2.24.0) which is a powerful and generic serialization framework in Java. Flink calls such a type a <em>generic type</em> and you may stumble upon <code>GenericTypeInfo</code> when debugging code. If you [...] + +<div class="highlight"><pre><code class="language-java"><span class="n">env</span><span class="o">.</span><span class="na">getConfig</span><span class="o">().</span><span class="na">registerKryoType</span><span class="o">(</span><span class="n">MyCustomType</span><span class="o">.< [...] +<p>Registering types adds them to an internal map of classes to tags so that, during serialization, Kryo does not have to add the fully qualified class names as a prefix into the serialized form. Instead, Kryo uses these (integer) tags to identify the underlying classes and reduce serialization overhead.</p> + +<div class="alert alert-info"> + <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span> +Flink will store Kryo serializer mappings from type registrations in its checkpoints and savepoints and will retain them across job (re)starts.</p> +</div> + +<h3 id="disabling-kryo">Disabling Kryo</h3> + +<p>If desired, you can disable the Kryo fallback, i.e. the ability to serialize generic types, by calling</p> + +<div class="highlight"><pre><code class="language-java"><span class="n">env</span><span class="o">.</span><span class="na">getConfig</span><span class="o">().</span><span class="na">disableGenericTypes</span><span class="o">();</span></code></pre></div> + +<p>This is mostly useful for finding out where these fallbacks are applied and replacing them with better serializers. If your job has any generic types with this configuration, it will fail with</p> + +<blockquote> + <p>Exception in thread “main” java.lang.UnsupportedOperationException: Generic types have been disabled in the ExecutionConfig and type … is treated as a generic type.</p> +</blockquote> + +<p>If you cannot immediately see from the type where it is being used, this log message also gives you a stacktrace that can be used to set breakpoints and find out more details in your IDE.</p> + +<h2 id="apache-thrift-via-kryo">Apache Thrift (via Kryo)</h2> + +<p>In addition to the variants above, Flink also allows you to <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/custom_serializers.html#register-a-custom-serializer-for-your-flink-program">register other type serialization frameworks</a> with Kryo. After adding the appropriate dependencies from the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/custom_serializers.html#register-a-custom-serializer-for- [...] + +<div class="highlight"><pre><code class="language-java"><span class="n">env</span><span class="o">.</span><span class="na">getConfig</span><span class="o">().</span><span class="na">addDefaultKryoSerializer</span><span class="o">(</span><span class="n">MyCustomType</span><span class="o" [...] + +<p>This only works if generic types are not disabled and <code>MyCustomType</code> is a Thrift-generated data type. If the data type is not generated by Thrift, Flink will fail at runtime with an exception like this:</p> + +<blockquote> + <p>java.lang.ClassCastException: class MyCustomType cannot be cast to class org.apache.thrift.TBase (MyCustomType and org.apache.thrift.TBase are in unnamed module of loader ‘app’)</p> +</blockquote> + +<div class="alert alert-info"> + <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span> +Please note that <code>TBaseSerializer</code> can be registered as a default Kryo serializer as above (and as specified in <a href="https://github.com/twitter/chill/blob/v0.7.6/chill-thrift/src/main/java/com/twitter/chill/thrift/TBaseSerializer.java">its documentation</a>) or via <code>registerTypeWithKryoSerializer</code>. In practice, we found both ways working. We also saw no difference between registering Thrift classes in addition to the [...] +</div> + +<h2 id="protobuf-via-kryo">Protobuf (via Kryo)</h2> + +<p>In a way similar to Apache Thrift, <a href="https://developers.google.com/protocol-buffers/">Google Protobuf</a> may be <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/custom_serializers.html#register-a-custom-serializer-for-your-flink-program">registered as a custom serializer</a> after adding the right dependencies (<code>com.twitter:chill-protobuf</code> and <code>com.google.protobuf:proto [...] + +<div class="highlight"><pre><code class="language-java"><span class="n">env</span><span class="o">.</span><span class="na">getConfig</span><span class="o">().</span><span class="na">registerTypeWithKryoSerializer</span><span class="o">(</span><span class="n">MyCustomType</span><span class="o [...] +<p>This will work as long as generic types have not been disabled (this would disable Kryo for good). If <code>MyCustomType</code> is not a Protobuf-generated class, your Flink job will fail at runtime with the following exception:</p> + +<blockquote> + <p>java.lang.ClassCastException: class <code>MyCustomType</code> cannot be cast to class com.google.protobuf.Message (<code>MyCustomType</code> and com.google.protobuf.Message are in unnamed module of loader ‘app’)</p> +</blockquote> + +<div class="alert alert-info"> + <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span> +Please note that <code>ProtobufSerializer</code> can be registered as a default Kryo serializer (as specified in the <a href="https://github.com/twitter/chill/blob/v0.7.6/chill-thrift/src/main/java/com/twitter/chill/thrift/TBaseSerializer.java">Protobuf documentation</a>) or via <code>registerTypeWithKryoSerializer</code> (as presented here). In practice, we found both ways working. We also saw no difference between registering your Protobuf [...] +</div> + +<h1 id="state-schema-evolution">State Schema Evolution</h1> + +<p>Before taking a closer look at the performance of each of the serializers described above, we would like to emphasize that performance is not everything that counts inside a real-world Flink job. Types for storing state, for example, should be able to evolve their schema (add/remove/change fields) throughout the lifetime of the job without losing previous state. This is what Flink calls <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/sche [...] + +<h1 id="performance-comparison">Performance Comparison</h1> + +<p>With so many options for serialization, it is actually not easy to make the right choice. We already saw some technical advantages and disadvantages of each of them outlined above. Since serializers are at the core of your Flink jobs and usually also sit on the hot path (per record invocations), let us actually take a deeper look into their performance with the help of the Flink benchmarks project at <a href="https://github.com/dataArtisans/flink-benchmarks">http [...] + +<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyPojo</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">int</span> <span class="n">id</span><span class="o">;</span> + <span class="kd">private</span> <span class="n">String</span> <span class="n">name</span><span class="o">;</span> + <span class="kd">private</span> <span class="n">String</span><span class="o">[]</span> <span class="n">operationNames</span><span class="o">;</span> + <span class="kd">private</span> <span class="n">MyOperation</span><span class="o">[]</span> <span class="n">operations</span><span class="o">;</span> + <span class="kd">private</span> <span class="kt">int</span> <span class="n">otherId1</span><span class="o">;</span> + <span class="kd">private</span> <span class="kt">int</span> <span class="n">otherId2</span><span class="o">;</span> + <span class="kd">private</span> <span class="kt">int</span> <span class="n">otherId3</span><span class="o">;</span> + <span class="kd">private</span> <span class="n">Object</span> <span class="n">someObject</span><span class="o">;</span> +<span class="o">}</span> +<span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyOperation</span> <span class="o">{</span> + <span class="kt">int</span> <span class="n">id</span><span class="o">;</span> + <span class="kd">protected</span> <span class="n">String</span> <span class="n">name</span><span class="o">;</span> +<span class="o">}</span></code></pre></div> + +<p>This is mapped to tuples, rows, Avro specific records, Thrift and Protobuf representations appropriately and sent through a simple Flink job at parallelism 4 where the data type is used during network communication like this:</p> + +<div class="highlight"><pre><code class="language-java"><span class="n">env</span><span class="o">.</span><span class="na">setParallelism</span><span class="o">(</span><span class="mi">4</span><span class="o">);</span> +<span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o">(</span><span class="k">new</span> <span class="nf">PojoSource</span><span class="o">(</span><span class="n">RECORDS_PER_INVOCATION</span><span class="o">,</span> <span class="mi"> [...] + <span class="o">.</span><span class="na">rebalance</span><span class="o">()</span> + <span class="o">.</span><span class="na">addSink</span><span class="o">(</span><span class="k">new</span> <span class="n">DiscardingSink</span><span class="o">&lt;&gt;());</span></code></pre></div> +<p>After running this through the <a href="http://openjdk.java.net/projects/code-tools/jmh/">jmh</a> micro-benchmarks defined in <a href="https://github.com/dataArtisans/flink-benchmarks/blob/master/src/main/java/org/apache/flink/benchmark/full/SerializationFrameworkAllBenchmarks.java">SerializationFrameworkAllBenchmarks.java</a>, I retrieved the following performance results for Flink 1.10 on my machine (in number of operations per milli [...] +<br /></p> + +<center> +<img src="/img/blog/2020-04-15-flink-serialization-performance-results.svg" width="800px" alt="Communication between the Flink operator and the Python execution environment" /> +</center> +<p><br /></p> + +<p>A few takeaways from these numbers:</p> + +<ul> + <li> + <p>The default fallback from POJO to Kryo reduces performance by 75%.<br /> +Registering types with Kryo significantly improves its performance with only 64% fewer operations than by using a POJO.</p> + </li> + <li> + <p>Avro GenericRecord and SpecificRecord are roughly serialized at the same speed.</p> + </li> + <li> + <p>Avro Reflect serialization is even slower than Kryo default (-45%).</p> + </li> + <li> + <p>Tuples are the fastest, closely followed by Rows. Both leverage fast specialized serialization code based on direct access without Java reflection.</p> + </li> + <li> + <p>Using a (nested) Tuple instead of a POJO may speed up your job by 42% (but is less flexible!). + Having code-generation for the PojoSerializer (<a href="https://jira.apache.org/jira/browse/FLINK-3599">FLINK-3599</a>) may actually close that gap (or at least move closer to the RowSerializer). If you feel like giving the implementation a go, please give the Flink community a note and we will see whether we can make that happen.</p> + </li> + <li> + <p>If you cannot use POJOs, try to define your data type with one of the serialization frameworks that generate specific code for it: Protobuf, Avro, Thrift (in that order, performance-wise).</p> + </li> +</ul> + +<div class="alert alert-info"> + <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span> As with all benchmarks, please bear in mind that these numbers only give a hint on Flink’s serializer performance in a specific scenario. They may be different with your data types but the rough classification is probably the same. If you want to be sure, please verify the [...] +</div> + +<h1 id="conclusion">Conclusion</h1> + +<p>In the sections above, we looked at how Flink performs serialization for different sorts of data types and elaborated the technical advantages and disadvantages. For data types used in Flink state, you probably want to leverage either POJO or Avro types which, currently, are the only ones supporting state evolution out of the box and allow your stateful application to develop over time. POJOs are usually faster in the de/serialization while Avro may support more flexible schema [...] + +<p>The fastest de/serialization is achieved with Flink’s internal tuple and row serializers which can access these types’ fields directly without going via reflection. With roughly 30% decreased throughput as compared to tuples, Protobuf and POJO types do not perform too badly on their own and are more flexible and maintainable. Avro (specific and generic) records as well as Thrift data types further reduce performance by 20% and 30%, respectively. You definitely want to avoid Kryo [...] + +<p>The next article in this series will use this finding as a starting point to look into a few common pitfalls and obstacles of avoiding Kryo, how to get the most out of the PojoSerializer, and a few more tuning techniques with respect to serialization. Stay tuned for more.</p> +</description> +<pubDate>Wed, 15 Apr 2020 10:00:00 +0200</pubDate> +<link>https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html</link> +<guid isPermaLink="true">/news/2020/04/15/flink-serialization-tuning-vol-1.html</guid> +</item> + +<item> <title>PyFlink: Introducing Python Support for UDFs in Flink's Table API</title> <description><p>Flink 1.9 introduced the Python Table API, allowing developers and data engineers to write Python Table API jobs for Table transformations and analysis, such as Python ETL or aggregate jobs. However, Python users faced some limitations when it came to support for Python UDFs in Flink 1.9, preventing them from extending the system’s built-in functionality.</p> @@ -17002,91 +17303,5 @@ internally, fault tolerance, and performance measurements!</p> <guid isPermaLink="true">/news/2015/02/04/january-in-flink.html</guid> </item> -<item> -<title>Apache Flink 0.8.0 available</title> -<description><p>We are pleased to announce the availability of Flink 0.8.0. This release includes new user-facing features as well as performance and bug fixes, extends the support for filesystems and introduces the Scala API and flexible windowing semantics for Flink Streaming. A total of 33 people have contributed to this release, a big thanks to all of them!</p> - -<p><a href="http://www.apache.org/dyn/closer.cgi/flink/flink-0.8.0/flink-0.8.0-bin-hadoop2.tgz">Download Flink 0.8.0</a></p> - -<p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&amp;version=12328699">See the release changelog</a></p> - -<h2 id="overview-of-major-new-features">Overview of major new features</h2> - -<ul> - <li> - <p><strong>Extended filesystem support</strong>: The former <code>DistributedFileSystem</code> interface has been generalized to <code>HadoopFileSystem</code> now supporting all sub classes of <code>org.apache.hadoop.fs.FileSystem</code>. This allows users to use all file systems supported by Hadoop with Apache Flink. -<a href="http://ci.apache.org/projects/flink/flink-docs-release-0.8/example_connectors.html">See connecting to other systems</a></p> - </li> - <li> - <p><strong>Streaming Scala API</strong>: As an alternative to the existing Java API Streaming is now also programmable in Scala. The Java and Scala APIs have now the same syntax and transformations and will be kept from now on in sync in every future release.</p> - </li> - <li> - <p><strong>Streaming windowing semantics</strong>: The new windowing api offers an expressive way to define custom logic for triggering the execution of a stream window and removing elements. The new features include out-of-the-box support for windows based in logical or physical time and data-driven properties on the events themselves among others. <a href="http://ci.apache.org/projects/flink/flink-docs-release-0.8/streaming_guide.html#window-operators" [...] - </li> - <li> - <p><strong>Mutable and immutable objects in runtime</strong> All Flink versions before 0.8.0 were always passing the same objects to functions written by users. This is a common performance optimization, also used in other systems such as Hadoop. - However, this is error-prone for new users because one has to carefully check that references to the object aren’t kept in the user function. Starting from 0.8.0, Flink allows to configure a mode which is disabling that mechanism.</p> - </li> - <li><strong>Performance and usability improvements</strong>: The new Apache Flink 0.8.0 release brings several new features which will significantly improve the performance and the usability of the system. Amongst others, these features include: - <ul> - <li>Improved input split assignment which maximizes computation locality</li> - <li>Smart broadcasting mechanism which minimizes network I/O</li> - <li>Custom partitioners which let the user control how the data is partitioned within the cluster. This helps to prevent data skewness and allows to implement highly efficient algorithms.</li> - <li>coGroup operator now supports group sorting for its inputs</li> - </ul> - </li> - <li> - <p><strong>Kryo is the new fallback serializer</strong>: Apache Flink has a sophisticated type analysis and serialization framework that is able to handle commonly used types very efficiently. - In addition to that, there is a fallback serializer for types which are not supported. Older versions of Flink used the reflective <a href="http://avro.apache.org/">Avro</a> serializer for that purpose. With this release, Flink is using the powerful <a href="https://github.com/EsotericSoftware/kryo">Kryo</a> and twitter-chill library for support of types such as Java Collections and Scala specifc types.</p> - </li> - <li> - <p><strong>Hadoop 2.2.0+ is now the default Hadoop dependency</strong>: With Flink 0.8.0 we made the “hadoop2” build profile the default build for Flink. This means that all users using Hadoop 1 (0.2X or 1.2.X versions) have to specify version “0.8.0-hadoop1” in their pom files.</p> - </li> - <li><strong>HBase module updated</strong> The HBase version has been updated to 0.98.6.1. Also, Hbase is now available to the Hadoop1 and Hadoop2 profile of Flink.</li> -</ul> - -<h2 id="contributors">Contributors</h2> - -<ul> - <li>Marton Balassi</li> - <li>Daniel Bali</li> - <li>Carsten Brandt</li> - <li>Moritz Borgmann</li> - <li>Stefan Bunk</li> - <li>Paris Carbone</li> - <li>Ufuk Celebi</li> - <li>Nils Engelbach</li> - <li>Stephan Ewen</li> - <li>Gyula Fora</li> - <li>Gabor Hermann</li> - <li>Fabian Hueske</li> - <li>Vasiliki Kalavri</li> - <li>Johannes Kirschnick</li> - <li>Aljoscha Krettek</li> - <li>Suneel Marthi</li> - <li>Robert Metzger</li> - <li>Felix Neutatz</li> - <li>Chiwan Park</li> - <li>Flavio Pompermaier</li> - <li>Mingliang Qi</li> - <li>Shiva Teja Reddy</li> - <li>Till Rohrmann</li> - <li>Henry Saputra</li> - <li>Kousuke Saruta</li> - <li>Chesney Schepler</li> - <li>Erich Schubert</li> - <li>Peter Szabo</li> - <li>Jonas Traub</li> - <li>Kostas Tzoumas</li> - <li>Timo Walther</li> - <li>Daniel Warneke</li> - <li>Chen Xu</li> -</ul> -</description> -<pubDate>Wed, 21 Jan 2015 11:00:00 +0100</pubDate> -<link>https://flink.apache.org/news/2015/01/21/release-0.8.html</link> -<guid isPermaLink="true">/news/2015/01/21/release-0.8.html</guid> -</item> - </channel> </rss> diff --git a/content/blog/index.html b/content/blog/index.html index 586a499..aa24919 100644 --- a/content/blog/index.html +++ b/content/blog/index.html @@ -195,6 +195,19 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</a></h2> + + <p>15 Apr 2020 + Nico Kruber </p> + + <p>Serialization is a crucial element of your Flink job. This article is the first in a series of posts that will highlight Flink’s serialization stack, and looks at the different ways Flink can serialize your data types.</p> + + <p><a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/2020/04/09/pyflink-udf-support-flink.html">PyFlink: Introducing Python Support for UDFs in Flink's Table API</a></h2> <p>09 Apr 2020 @@ -318,21 +331,6 @@ This release marks a big milestone: Stateful Functions 2.0 is not only an API up <hr> - <article> - <h2 class="blog-title"><a href="/news/2020/01/30/release-1.9.2.html">Apache Flink 1.9.2 Released</a></h2> - - <p>30 Jan 2020 - Hequn Cheng (<a href="https://twitter.com/HequnC">@HequnC</a>)</p> - - <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.9 series.</p> - -</p> - - <p><a href="/news/2020/01/30/release-1.9.2.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -365,6 +363,16 @@ This release marks a big milestone: Stateful Functions 2.0 is not only an API up <ul id="markdown-toc"> + <li><a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</a></li> + + + + + + + + + <li><a href="/2020/04/09/pyflink-udf-support-flink.html">PyFlink: Introducing Python Support for UDFs in Flink's Table API</a></li> diff --git a/content/blog/page10/index.html b/content/blog/page10/index.html index d08d083..21caa67 100644 --- a/content/blog/page10/index.html +++ b/content/blog/page10/index.html @@ -195,6 +195,24 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2015/08/24/introducing-flink-gelly.html">Introducing Gelly: Graph Processing with Apache Flink</a></h2> + + <p>24 Aug 2015 + </p> + + <p><p>This blog post introduces <strong>Gelly</strong>, Apache Flink’s <em>graph-processing API and library</em>. Flink’s native support +for iterations makes it a suitable platform for large-scale graph analytics. +By leveraging delta iterations, Gelly is able to map various graph processing models such as +vertex-centric or gather-sum-apply to Flink dataflows.</p> + +</p> + + <p><a href="/news/2015/08/24/introducing-flink-gelly.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2015/06/24/announcing-apache-flink-0.9.0-release.html">Announcing Apache Flink 0.9.0</a></h2> <p>24 Jun 2015 @@ -336,21 +354,6 @@ and offers a new API including definition of flexible windows.</p> <hr> - <article> - <h2 class="blog-title"><a href="/news/2015/01/21/release-0.8.html">Apache Flink 0.8.0 available</a></h2> - - <p>21 Jan 2015 - </p> - - <p><p>We are pleased to announce the availability of Flink 0.8.0. This release includes new user-facing features as well as performance and bug fixes, extends the support for filesystems and introduces the Scala API and flexible windowing semantics for Flink Streaming. A total of 33 people have contributed to this release, a big thanks to all of them!</p> - -</p> - - <p><a href="/news/2015/01/21/release-0.8.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -383,6 +386,16 @@ and offers a new API including definition of flexible windows.</p> <ul id="markdown-toc"> + <li><a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</a></li> + + + + + + + + + <li><a href="/2020/04/09/pyflink-udf-support-flink.html">PyFlink: Introducing Python Support for UDFs in Flink's Table API</a></li> diff --git a/content/blog/page11/index.html b/content/blog/page11/index.html index ff672cd..ed7f3aa 100644 --- a/content/blog/page11/index.html +++ b/content/blog/page11/index.html @@ -195,6 +195,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2015/01/21/release-0.8.html">Apache Flink 0.8.0 available</a></h2> + + <p>21 Jan 2015 + </p> + + <p><p>We are pleased to announce the availability of Flink 0.8.0. This release includes new user-facing features as well as performance and bug fixes, extends the support for filesystems and introduces the Scala API and flexible windowing semantics for Flink Streaming. A total of 33 people have contributed to this release, a big thanks to all of them!</p> + +</p> + + <p><a href="/news/2015/01/21/release-0.8.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2015/01/06/december-in-flink.html">December 2014 in the Flink community</a></h2> <p>06 Jan 2015 @@ -319,6 +334,16 @@ academic and open source project that Flink originates from.</p> <ul id="markdown-toc"> + <li><a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</a></li> + + + + + + + + + <li><a href="/2020/04/09/pyflink-udf-support-flink.html">PyFlink: Introducing Python Support for UDFs in Flink's Table API</a></li> diff --git a/content/blog/page2/index.html b/content/blog/page2/index.html index 79dd8aa..42535b0 100644 --- a/content/blog/page2/index.html +++ b/content/blog/page2/index.html @@ -195,6 +195,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2020/01/30/release-1.9.2.html">Apache Flink 1.9.2 Released</a></h2> + + <p>30 Jan 2020 + Hequn Cheng (<a href="https://twitter.com/HequnC">@HequnC</a>)</p> + + <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.9 series.</p> + +</p> + + <p><a href="/news/2020/01/30/release-1.9.2.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html">State Unlocked: Interacting with State in Apache Flink</a></h2> <p>29 Jan 2020 @@ -317,22 +332,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/news/2019/08/22/release-1.9.0.html">Apache Flink 1.9.0 Release Announcement</a></h2> - - <p>22 Aug 2019 - </p> - - <p><p>The Apache Flink community is proud to announce the release of Apache Flink -1.9.0.</p> - -</p> - - <p><a href="/news/2019/08/22/release-1.9.0.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -365,6 +364,16 @@ <ul id="markdown-toc"> + <li><a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</a></li> + + + + + + + + + <li><a href="/2020/04/09/pyflink-udf-support-flink.html">PyFlink: Introducing Python Support for UDFs in Flink's Table API</a></li> diff --git a/content/blog/page3/index.html b/content/blog/page3/index.html index da9d3fc..9abb37e 100644 --- a/content/blog/page3/index.html +++ b/content/blog/page3/index.html @@ -195,6 +195,22 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2019/08/22/release-1.9.0.html">Apache Flink 1.9.0 Release Announcement</a></h2> + + <p>22 Aug 2019 + </p> + + <p><p>The Apache Flink community is proud to announce the release of Apache Flink +1.9.0.</p> + +</p> + + <p><a href="/news/2019/08/22/release-1.9.0.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/2019/07/23/flink-network-stack-2.html">Flink Network Stack Vol. 2: Monitoring, Metrics, and that Backpressure Thing</a></h2> <p>23 Jul 2019 @@ -321,19 +337,6 @@ for more details.</p> <hr> - <article> - <h2 class="blog-title"><a href="/features/2019/03/11/prometheus-monitoring.html">Flink and Prometheus: Cloud-native monitoring of streaming applications</a></h2> - - <p>11 Mar 2019 - Maximilian Bode, TNG Technology Consulting (<a href="https://twitter.com/mxpbode">@mxpbode</a>)</p> - - <p>This blog post describes how developers can leverage Apache Flink's built-in metrics system together with Prometheus to observe and monitor streaming applications in an effective way.</p> - - <p><a href="/features/2019/03/11/prometheus-monitoring.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -366,6 +369,16 @@ for more details.</p> <ul id="markdown-toc"> + <li><a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</a></li> + + + + + + + + + <li><a href="/2020/04/09/pyflink-udf-support-flink.html">PyFlink: Introducing Python Support for UDFs in Flink's Table API</a></li> diff --git a/content/blog/page4/index.html b/content/blog/page4/index.html index 4663315..4e0e20d 100644 --- a/content/blog/page4/index.html +++ b/content/blog/page4/index.html @@ -195,6 +195,19 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/features/2019/03/11/prometheus-monitoring.html">Flink and Prometheus: Cloud-native monitoring of streaming applications</a></h2> + + <p>11 Mar 2019 + Maximilian Bode, TNG Technology Consulting (<a href="https://twitter.com/mxpbode">@mxpbode</a>)</p> + + <p>This blog post describes how developers can leverage Apache Flink's built-in metrics system together with Prometheus to observe and monitor streaming applications in an effective way.</p> + + <p><a href="/features/2019/03/11/prometheus-monitoring.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2019/03/06/ffsf-preview.html">What to expect from Flink Forward San Francisco 2019</a></h2> <p>06 Mar 2019 @@ -325,21 +338,6 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa <hr> - <article> - <h2 class="blog-title"><a href="/news/2018/10/29/release-1.6.2.html">Apache Flink 1.6.2 Released</a></h2> - - <p>29 Oct 2018 - </p> - - <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.6 series.</p> - -</p> - - <p><a href="/news/2018/10/29/release-1.6.2.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -372,6 +370,16 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa <ul id="markdown-toc"> + <li><a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</a></li> + + + + + + + + + <li><a href="/2020/04/09/pyflink-udf-support-flink.html">PyFlink: Introducing Python Support for UDFs in Flink's Table API</a></li> diff --git a/content/blog/page5/index.html b/content/blog/page5/index.html index 4d43c0f..fb48d96 100644 --- a/content/blog/page5/index.html +++ b/content/blog/page5/index.html @@ -195,6 +195,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2018/10/29/release-1.6.2.html">Apache Flink 1.6.2 Released</a></h2> + + <p>29 Oct 2018 + </p> + + <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.6 series.</p> + +</p> + + <p><a href="/news/2018/10/29/release-1.6.2.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2018/10/29/release-1.5.5.html">Apache Flink 1.5.5 Released</a></h2> <p>29 Oct 2018 @@ -329,21 +344,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/news/2018/03/08/release-1.4.2.html">Apache Flink 1.4.2 Released</a></h2> - - <p>08 Mar 2018 - </p> - - <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.4 series.</p> - -</p> - - <p><a href="/news/2018/03/08/release-1.4.2.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -376,6 +376,16 @@ <ul id="markdown-toc"> + <li><a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</a></li> + + + + + + + + + <li><a href="/2020/04/09/pyflink-udf-support-flink.html">PyFlink: Introducing Python Support for UDFs in Flink's Table API</a></li> diff --git a/content/blog/page6/index.html b/content/blog/page6/index.html index e812215..0a294fe 100644 --- a/content/blog/page6/index.html +++ b/content/blog/page6/index.html @@ -195,6 +195,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2018/03/08/release-1.4.2.html">Apache Flink 1.4.2 Released</a></h2> + + <p>08 Mar 2018 + </p> + + <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.4 series.</p> + +</p> + + <p><a href="/news/2018/03/08/release-1.4.2.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/features/2018/03/01/end-to-end-exactly-once-apache-flink.html">An Overview of End-to-End Exactly-Once Processing in Apache Flink (with Apache Kafka, too!)</a></h2> <p>01 Mar 2018 @@ -326,21 +341,6 @@ what’s coming in Flink 1.4.0 as well as a preview of what the Flink community <hr> - <article> - <h2 class="blog-title"><a href="/news/2017/06/01/release-1.3.0.html">Apache Flink 1.3.0 Release Announcement</a></h2> - - <p>01 Jun 2017 by Robert Metzger (<a href="https://twitter.com/">@rmetzger_</a>) - </p> - - <p><p>The Apache Flink community is pleased to announce the 1.3.0 release. Over the past 4 months, the Flink community has been working hard to resolve more than 680 issues. See the <a href="/blog/release_1.3.0-changelog.html">complete changelog</a> for more detail.</p> - -</p> - - <p><a href="/news/2017/06/01/release-1.3.0.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -373,6 +373,16 @@ what’s coming in Flink 1.4.0 as well as a preview of what the Flink community <ul id="markdown-toc"> + <li><a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</a></li> + + + + + + + + + <li><a href="/2020/04/09/pyflink-udf-support-flink.html">PyFlink: Introducing Python Support for UDFs in Flink's Table API</a></li> diff --git a/content/blog/page7/index.html b/content/blog/page7/index.html index 8b31cba..8f00f06 100644 --- a/content/blog/page7/index.html +++ b/content/blog/page7/index.html @@ -195,6 +195,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2017/06/01/release-1.3.0.html">Apache Flink 1.3.0 Release Announcement</a></h2> + + <p>01 Jun 2017 by Robert Metzger (<a href="https://twitter.com/">@rmetzger_</a>) + </p> + + <p><p>The Apache Flink community is pleased to announce the 1.3.0 release. Over the past 4 months, the Flink community has been working hard to resolve more than 680 issues. See the <a href="/blog/release_1.3.0-changelog.html">complete changelog</a> for more detail.</p> + +</p> + + <p><a href="/news/2017/06/01/release-1.3.0.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2017/05/16/official-docker-image.html">Introducing Docker Images for Apache Flink</a></h2> <p>16 May 2017 by Patrick Lucas (Data Artisans) and Ismaël Mejía (Talend) (<a href="https://twitter.com/">@iemejia</a>) @@ -322,21 +337,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/news/2016/09/05/release-1.1.2.html">Apache Flink 1.1.2 Released</a></h2> - - <p>05 Sep 2016 - </p> - - <p><p>The Apache Flink community released another bugfix version of the Apache Flink 1.1. series.</p> - -</p> - - <p><a href="/news/2016/09/05/release-1.1.2.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -369,6 +369,16 @@ <ul id="markdown-toc"> + <li><a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</a></li> + + + + + + + + + <li><a href="/2020/04/09/pyflink-udf-support-flink.html">PyFlink: Introducing Python Support for UDFs in Flink's Table API</a></li> diff --git a/content/blog/page8/index.html b/content/blog/page8/index.html index e5e719a..251627f 100644 --- a/content/blog/page8/index.html +++ b/content/blog/page8/index.html @@ -195,6 +195,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2016/09/05/release-1.1.2.html">Apache Flink 1.1.2 Released</a></h2> + + <p>05 Sep 2016 + </p> + + <p><p>The Apache Flink community released another bugfix version of the Apache Flink 1.1. series.</p> + +</p> + + <p><a href="/news/2016/09/05/release-1.1.2.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2016/08/24/ff16-keynotes-panels.html">Flink Forward 2016: Announcing Schedule, Keynotes, and Panel Discussion</a></h2> <p>24 Aug 2016 @@ -326,21 +341,6 @@ <hr> - <article> - <h2 class="blog-title"><a href="/news/2016/03/08/release-1.0.0.html">Announcing Apache Flink 1.0.0</a></h2> - - <p>08 Mar 2016 - </p> - - <p><p>The Apache Flink community is pleased to announce the availability of the 1.0.0 release. The community put significant effort into improving and extending Apache Flink since the last release, focusing on improving the experience of writing and executing data stream processing pipelines in production.</p> - -</p> - - <p><a href="/news/2016/03/08/release-1.0.0.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -373,6 +373,16 @@ <ul id="markdown-toc"> + <li><a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</a></li> + + + + + + + + + <li><a href="/2020/04/09/pyflink-udf-support-flink.html">PyFlink: Introducing Python Support for UDFs in Flink's Table API</a></li> diff --git a/content/blog/page9/index.html b/content/blog/page9/index.html index 4b348a7..7029cab 100644 --- a/content/blog/page9/index.html +++ b/content/blog/page9/index.html @@ -195,6 +195,21 @@ <!-- Blog posts --> <article> + <h2 class="blog-title"><a href="/news/2016/03/08/release-1.0.0.html">Announcing Apache Flink 1.0.0</a></h2> + + <p>08 Mar 2016 + </p> + + <p><p>The Apache Flink community is pleased to announce the availability of the 1.0.0 release. The community put significant effort into improving and extending Apache Flink since the last release, focusing on improving the experience of writing and executing data stream processing pipelines in production.</p> + +</p> + + <p><a href="/news/2016/03/08/release-1.0.0.html">Continue reading »</a></p> + </article> + + <hr> + + <article> <h2 class="blog-title"><a href="/news/2016/02/11/release-0.10.2.html">Flink 0.10.2 Released</a></h2> <p>11 Feb 2016 @@ -327,24 +342,6 @@ Apache Flink started.</p> <hr> - <article> - <h2 class="blog-title"><a href="/news/2015/08/24/introducing-flink-gelly.html">Introducing Gelly: Graph Processing with Apache Flink</a></h2> - - <p>24 Aug 2015 - </p> - - <p><p>This blog post introduces <strong>Gelly</strong>, Apache Flink’s <em>graph-processing API and library</em>. Flink’s native support -for iterations makes it a suitable platform for large-scale graph analytics. -By leveraging delta iterations, Gelly is able to map various graph processing models such as -vertex-centric or gather-sum-apply to Flink dataflows.</p> - -</p> - - <p><a href="/news/2015/08/24/introducing-flink-gelly.html">Continue reading »</a></p> - </article> - - <hr> - <!-- Pagination links --> @@ -377,6 +374,16 @@ vertex-centric or gather-sum-apply to Flink dataflows.</p> <ul id="markdown-toc"> + <li><a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</a></li> + + + + + + + + + <li><a href="/2020/04/09/pyflink-udf-support-flink.html">PyFlink: Introducing Python Support for UDFs in Flink's Table API</a></li> diff --git a/content/img/blog/2020-04-15-flink-serialization-performance-results.svg b/content/img/blog/2020-04-15-flink-serialization-performance-results.svg new file mode 100644 index 0000000..314ae7d --- /dev/null +++ b/content/img/blog/2020-04-15-flink-serialization-performance-results.svg @@ -0,0 +1 @@ +<svg version="1.1" viewBox="0.0 0.0 1020.0 587.0" fill="none" stroke="none" stroke-linecap="square" stroke-miterlimit="10" width="1020" height="587" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg"><path fill="#ffffff" d="M0 0L1020.0 0L1020.0 587.0L0 587.0L0 0Z" fill-rule="nonzero"/><path stroke="#666666" stroke-width="1.0" stroke-linecap="butt" d="M379.5 72.5L379.5 500.5" fill-rule="nonzero"/><path stroke="#666666" stroke-width="1.0" stroke-linecap="butt" d= [...] \ No newline at end of file diff --git a/content/index.html b/content/index.html index 94571bf..a374861 100644 --- a/content/index.html +++ b/content/index.html @@ -567,6 +567,9 @@ <dl> + <dt> <a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</a></dt> + <dd>Serialization is a crucial element of your Flink job. This article is the first in a series of posts that will highlight Flink’s serialization stack, and looks at the different ways Flink can serialize your data types.</dd> + <dt> <a href="/2020/04/09/pyflink-udf-support-flink.html">PyFlink: Introducing Python Support for UDFs in Flink's Table API</a></dt> <dd>Flink 1.10 extends its support for Python by adding Python UDFs in PyFlink. This post explains how UDFs work in PyFlink and gives some practical examples of how to use UDFs in PyFlink.</dd> @@ -583,9 +586,6 @@ This release marks a big milestone: Stateful Functions 2.0 is not only an API up <dd><p>In this blog post, you will learn our motivation behind the Flink-Hive integration, and how Flink 1.10 can help modernize your data warehouse.</p> </dd> - - <dt> <a href="/news/2020/03/24/demo-fraud-detection-2.html">Advanced Flink Application Patterns Vol.2: Dynamic Updates of Application Logic</a></dt> - <dd>In this series of blog posts you will learn about powerful Flink patterns for building streaming applications.</dd> </dl> diff --git a/content/news/2020/04/15/flink-serialization-tuning-vol-1.html b/content/news/2020/04/15/flink-serialization-tuning-vol-1.html new file mode 100644 index 0000000..92e3e36 --- /dev/null +++ b/content/news/2020/04/15/flink-serialization-tuning-vol-1.html @@ -0,0 +1,555 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags --> + <title>Apache Flink: Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</title> + <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"> + <link rel="icon" href="/favicon.ico" type="image/x-icon"> + + <!-- Bootstrap --> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.4.1/css/bootstrap.min.css"> + <link rel="stylesheet" href="/css/flink.css"> + <link rel="stylesheet" href="/css/syntax.css"> + + <!-- Blog RSS feed --> + <link href="/blog/feed.xml" rel="alternate" type="application/rss+xml" title="Apache Flink Blog: RSS feed" /> + + <!-- jQuery (necessary for Bootstrap's JavaScript plugins) --> + <!-- We need to load Jquery in the header for custom google analytics event tracking--> + <script src="/js/jquery.min.js"></script> + + <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> + <!-- WARNING: Respond.js doesn't work if you view the page via file:// --> + <!--[if lt IE 9]> + <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script> + <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script> + <![endif]--> + </head> + <body> + + + <!-- Main content. --> + <div class="container"> + <div class="row"> + + + <div id="sidebar" class="col-sm-3"> + + +<!-- Top navbar. --> + <nav class="navbar navbar-default"> + <!-- The logo. --> + <div class="navbar-header"> + <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <div class="navbar-logo"> + <a href="/"> + <img alt="Apache Flink" src="/img/flink-header-logo.svg" width="147px" height="73px"> + </a> + </div> + </div><!-- /.navbar-header --> + + <!-- The navigation links. --> + <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1"> + <ul class="nav navbar-nav navbar-main"> + + <!-- First menu section explains visitors what Flink is --> + + <!-- What is Stream Processing? --> + <!-- + <li><a href="/streamprocessing1.html">What is Stream Processing?</a></li> + --> + + <!-- What is Flink? --> + <li><a href="/flink-architecture.html">What is Apache Flink?</a></li> + + + <ul class="nav navbar-nav navbar-subnav"> + <li > + <a href="/flink-architecture.html">Architecture</a> + </li> + <li > + <a href="/flink-applications.html">Applications</a> + </li> + <li > + <a href="/flink-operations.html">Operations</a> + </li> + </ul> + + + <!-- What is Stateful Functions? --> + + <li><a href="/stateful-functions.html">What is Stateful Functions?</a></li> + + <!-- Use cases --> + <li><a href="/usecases.html">Use Cases</a></li> + + <!-- Powered by --> + <li><a href="/poweredby.html">Powered By</a></li> + + + + <!-- Second menu section aims to support Flink users --> + + <!-- Downloads --> + <li><a href="/downloads.html">Downloads</a></li> + + <!-- Getting Started --> + <li class="dropdown"> + <a class="dropdown-toggle" data-toggle="dropdown" href="#">Getting Started<span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/getting-started/index.html" target="_blank">With Flink <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> + <li><a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/getting-started/project-setup.html" target="_blank">With Flink Stateful Functions <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> + </ul> + </li> + + <!-- Documentation --> + <li class="dropdown"> + <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10" target="_blank">Flink 1.10 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> + <li><a href="https://ci.apache.org/projects/flink/flink-docs-master" target="_blank">Flink Master (Latest Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> + <li><a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0" target="_blank">Flink Stateful Functions 2.0 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> + <li><a href="https://ci.apache.org/projects/flink/flink-statefun-docs-master" target="_blank">Flink Stateful Functions Master (Latest Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> + </ul> + </li> + + <!-- getting help --> + <li><a href="/gettinghelp.html">Getting Help</a></li> + + <!-- Blog --> + <li class="active"><a href="/blog/"><b>Flink Blog</b></a></li> + + + <!-- Flink-packages --> + <li> + <a href="https://flink-packages.org" target="_blank">flink-packages.org <small><span class="glyphicon glyphicon-new-window"></span></small></a> + </li> + + + <!-- Third menu section aim to support community and contributors --> + + <!-- Community --> + <li><a href="/community.html">Community & Project Info</a></li> + + <!-- Roadmap --> + <li><a href="/roadmap.html">Roadmap</a></li> + + <!-- Contribute --> + <li><a href="/contributing/how-to-contribute.html">How to Contribute</a></li> + + + <!-- GitHub --> + <li> + <a href="https://github.com/apache/flink" target="_blank">Flink on GitHub <small><span class="glyphicon glyphicon-new-window"></span></small></a> + </li> + + + + <!-- Language Switcher --> + <li> + + + <!-- link to the Chinese home page when current is blog page --> + <a href="/zh">中文版</a> + + + </li> + + </ul> + + <ul class="nav navbar-nav navbar-bottom"> + <hr /> + + <!-- Twitter --> + <li><a href="https://twitter.com/apacheflink" target="_blank">@ApacheFlink <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> + + <!-- Visualizer --> + <li class=" hidden-md hidden-sm"><a href="/visualizer/" target="_blank">Plan Visualizer <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> + + <hr /> + + <li><a href="https://apache.org" target="_blank">Apache Software Foundation <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> + + <li> + <style> + .smalllinks:link { + display: inline-block !important; background: none; padding-top: 0px; padding-bottom: 0px; padding-right: 0px; min-width: 75px; + } + </style> + + <a class="smalllinks" href="https://www.apache.org/licenses/" target="_blank">License</a> <small><span class="glyphicon glyphicon-new-window"></span></small> + + <a class="smalllinks" href="https://www.apache.org/security/" target="_blank">Security</a> <small><span class="glyphicon glyphicon-new-window"></span></small> + + <a class="smalllinks" href="https://www.apache.org/foundation/sponsorship.html" target="_blank">Donate</a> <small><span class="glyphicon glyphicon-new-window"></span></small> + + <a class="smalllinks" href="https://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a> <small><span class="glyphicon glyphicon-new-window"></span></small> + </li> + + </ul> + </div><!-- /.navbar-collapse --> + </nav> + + </div> + <div class="col-sm-9"> + <div class="row-fluid"> + <div class="col-sm-12"> + <div class="row"> + <h1>Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</h1> + <p><i></i></p> + + <article> + <p>15 Apr 2020 Nico Kruber </p> + +<p>Almost every Flink job has to exchange data between its operators and since these records may not only be sent to another instance in the same JVM but instead to a separate process, records need to be serialized to bytes first. Similarly, Flink’s off-heap state-backend is based on a local embedded RocksDB instance which is implemented in native C++ code and thus also needs transformation into bytes on every state access. Wire and state serialization alone can easily cost a lot of your [...] + +<p>Since serialization is so crucial to your Flink job, we would like to highlight Flink’s serialization stack in a series of blog posts starting with looking at the different ways Flink can serialize your data types.</p> + +<div class="page-toc"> +<ul id="markdown-toc"> + <li><a href="#recap-flink-serialization" id="markdown-toc-recap-flink-serialization">Recap: Flink Serialization</a></li> + <li><a href="#choice-of-serializer" id="markdown-toc-choice-of-serializer">Choice of Serializer</a> <ul> + <li><a href="#pojoserializer" id="markdown-toc-pojoserializer">PojoSerializer</a></li> + <li><a href="#tuple-data-types" id="markdown-toc-tuple-data-types">Tuple Data Types</a></li> + <li><a href="#row-data-types" id="markdown-toc-row-data-types">Row Data Types</a></li> + <li><a href="#avro" id="markdown-toc-avro">Avro</a> <ul> + <li><a href="#avro-specific" id="markdown-toc-avro-specific">Avro Specific</a></li> + <li><a href="#avro-generic" id="markdown-toc-avro-generic">Avro Generic</a></li> + <li><a href="#avro-reflect" id="markdown-toc-avro-reflect">Avro Reflect</a></li> + </ul> + </li> + <li><a href="#kryo" id="markdown-toc-kryo">Kryo</a> <ul> + <li><a href="#disabling-kryo" id="markdown-toc-disabling-kryo">Disabling Kryo</a></li> + </ul> + </li> + <li><a href="#apache-thrift-via-kryo" id="markdown-toc-apache-thrift-via-kryo">Apache Thrift (via Kryo)</a></li> + <li><a href="#protobuf-via-kryo" id="markdown-toc-protobuf-via-kryo">Protobuf (via Kryo)</a></li> + </ul> + </li> + <li><a href="#state-schema-evolution" id="markdown-toc-state-schema-evolution">State Schema Evolution</a></li> + <li><a href="#performance-comparison" id="markdown-toc-performance-comparison">Performance Comparison</a></li> + <li><a href="#conclusion" id="markdown-toc-conclusion">Conclusion</a></li> +</ul> + +</div> + +<h1 id="recap-flink-serialization">Recap: Flink Serialization</h1> + +<p>Flink handles <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/types_serialization.html">data types and serialization</a> with its own type descriptors, generic type extraction, and type serialization framework. We recommend reading through the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/types_serialization.html">documentation</a> first in order to be able to follow the arguments we present below. In essence, Flink tries to infer [...] +<code>stream.keyBy(“ruleId”)</code> or +<code>dataSet.join(another).where("name").equalTo("personName")</code>. It also allows optimizations in the serialization format as well as reducing unnecessary de/serializations (mainly in certain Batch operations as well as in the SQL/Table APIs).</p> + +<h1 id="choice-of-serializer">Choice of Serializer</h1> + +<p>Apache Flink’s out-of-the-box serialization can be roughly divided into the following groups:</p> + +<ul> + <li> + <p><strong>Flink-provided special serializers</strong> for basic types (Java primitives and their boxed form), arrays, composite types (tuples, Scala case classes, Rows), and a few auxiliary types (Option, Either, Lists, Maps, …),</p> + </li> + <li> + <p><strong>POJOs</strong>; a public, standalone class with a public no-argument constructor and all non-static, non-transient fields in the class hierarchy either public or with a public getter- and a setter-method; see <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/types_serialization.html#rules-for-pojo-types">POJO Rules</a>,</p> + </li> + <li> + <p><strong>Generic types</strong>; user-defined data types that are not recognized as a POJO and then serialized via <a href="https://github.com/EsotericSoftware/kryo">Kryo</a>.</p> + </li> +</ul> + +<p>Alternatively, you can also register <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/custom_serializers.html">custom serializers</a> for user-defined data types. This includes writing your own serializers or integrating other serialization systems like <a href="https://developers.google.com/protocol-buffers/">Google Protobuf</a> or <a href="https://thrift.apache.org/">Apache Thrift</a> via <a href="https://github.com/EsotericSoftware/kryo">Kryo</a>. Overall, [...] + +<h2 id="pojoserializer">PojoSerializer</h2> + +<p>As outlined above, if your data type is not covered by a specialized serializer but follows the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/types_serialization.html#rules-for-pojo-types">POJO Rules</a>, it will be serialized with the <a href="https://github.com/apache/flink/blob/release-1.10.0/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/PojoSerializer.java">PojoSerializer</a> which uses Java reflection to access an object’s fields [...] + +<blockquote> + <p>15:45:51,460 INFO org.apache.flink.api.java.typeutils.TypeExtractor - Class … cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on “Data Types & Serialization” for details of the effect on performance.</p> +</blockquote> + +<p>This means, that the PojoSerializer will not be used, but instead Flink will fall back to Kryo for serialization (see below). We will have a more detailed look into a few (more) situations that can lead to unexpected Kryo fallbacks in the second part of this blog post series.</p> + +<h2 id="tuple-data-types">Tuple Data Types</h2> + +<p>Flink comes with a predefined set of tuple types which all have a fixed length and contain a set of strongly-typed fields of potentially different types. There are implementations for <code>Tuple0</code>, <code>Tuple1<T0></code>, …, <code>Tuple25<T0, T1, ..., T24></code> and they may serve as easy-to-use wrappers that spare the creation of POJOs for each and every combination of objects you need to pass between computations. With the exception of <code>Tuple0</code>, these [...] + +<div class="alert alert-info"> + <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span> + Since <code>Tuple0</code> does not contain any data and therefore is probably a bit special anyway, it will use a special serializer implementation: <a href="https://github.com/apache/flink/blob/release-1.10.0/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/Tuple0Serializer.java">Tuple0Serializer</a>.</p> +</div> + +<h2 id="row-data-types">Row Data Types</h2> + +<p>Row types are mainly used by the Table and SQL APIs of Flink. A <code>Row</code> groups an arbitrary number of objects together similar to the tuples above. These fields are not strongly typed and may all be of different types. Because field types are missing, Flink’s type extraction cannot automatically extract type information and users of a <code>Row</code> need to manually tell Flink about the row’s field types. The <a href="https://github.com/apache/flink/blob/release-1.10.0/flin [...] + +<p>Row type information can be provided in two ways:</p> + +<ul> + <li>you can have your source or operator implement <code>ResultTypeQueryable<Row></code>:</li> +</ul> + +<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">RowSource</span> <span class="kd">implements</span> <span class="n">SourceFunction</span><span class="o"><</span><span class="n">Row</span><span class="o">>,</span> <span class="n">ResultTypeQueryable</span><span class="o"><</span><span class="n">Row</span><span class="o">></span> <span class="o">{</span> + <span class="c1">// ...</span> + + <span class="nd">@Override</span> + <span class="kd">public</span> <span class="n">TypeInformation</span><span class="o"><</span><span class="n">Row</span><span class="o">></span> <span class="nf">getProducedType</span><span class="o">()</span> <span class="o">{</span> + <span class="k">return</span> <span class="n">Types</span><span class="o">.</span><span class="na">ROW</span><span class="o">(</span><span class="n">Types</span><span class="o">.</span><span class="na">INT</span><span class="o">,</span> <span class="n">Types</span><span class="o">.</span><span class="na">STRING</span><span class="o">,</span> <span class="n">Types</span><span class="o">.</span><span class="na">OBJECT_ARRAY</span><span class="o">(</span><span class="n">Types</span><spa [...] + <span class="o">}</span> +<span class="o">}</span></code></pre></div> + +<ul> + <li>you can provide the types when building the job graph by using <code>SingleOutputStreamOperator#returns()</code></li> +</ul> + +<div class="highlight"><pre><code class="language-java"><span class="n">DataStream</span><span class="o"><</span><span class="n">Row</span><span class="o">></span> <span class="n">sourceStream</span> <span class="o">=</span> + <span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o">(</span><span class="k">new</span> <span class="nf">RowSource</span><span class="o">())</span> + <span class="o">.</span><span class="na">returns</span><span class="o">(</span><span class="n">Types</span><span class="o">.</span><span class="na">ROW</span><span class="o">(</span><span class="n">Types</span><span class="o">.</span><span class="na">INT</span><span class="o">,</span> <span class="n">Types</span><span class="o">.</span><span class="na">STRING</span><span class="o">,</span> <span class="n">Types</span><span class="o">.</span><span class="na">OBJECT_ARRAY</span><sp [...] + +<div class="alert alert-warning"> + <p><span class="label label-warning" style="display: inline-block"><span class="glyphicon glyphicon-warning-sign" aria-hidden="true"></span> Warning</span> +If you fail to provide the type information for a <code>Row</code>, Flink identifies that <code>Row</code> is not a valid POJO type according to the rules above and falls back to Kryo serialization (see below) which you will also see in the logs as:</p> + + <p><code>13:10:11,148 INFO org.apache.flink.api.java.typeutils.TypeExtractor - Class class org.apache.flink.types.Row cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance.</code></p> +</div> + +<h2 id="avro">Avro</h2> + +<p>Flink offers built-in support for the <a href="http://avro.apache.org/">Apache Avro</a> serialization framework (currently using version 1.8.2) by adding the <code>org.apache.flink:flink-avro</code> dependency into your job. Flink’s <a href="https://github.com/apache/flink/blob/release-1.10.0/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/AvroSerializer.java">AvroSerializer</a> can then use Avro’s specific, generic, and reflective data serialization and [...] + +<h3 id="avro-specific">Avro Specific</h3> + +<p>Avro specific records will be automatically detected by checking that the given type’s type hierarchy contains the <code>SpecificRecordBase</code> class. You can either specify your concrete Avro type, or—if you want to be more generic and allow different types in your operator—use the <code>SpecificRecordBase</code> type (or a subtype) in your user functions, in <code>ResultTypeQueryable#getProducedType()</code>, or in <code>SingleOutputStreamOperator#returns()</code>. Since specific [...] + +<div class="alert alert-warning"> + <p><span class="label label-warning" style="display: inline-block"><span class="glyphicon glyphicon-warning-sign" aria-hidden="true"></span> Warning</span> If you specify the Flink type as <code>SpecificRecord</code> and not <code>SpecificRecordBase</code>, Flink will not see this as an Avro type. Instead, it will use Kryo to de/serialize any objects which may be considerably slower.</p> +</div> + +<h3 id="avro-generic">Avro Generic</h3> + +<p>Avro’s <code>GenericRecord</code> types cannot, unfortunately, be used automatically since they require the user to <a href="https://avro.apache.org/docs/1.8.2/gettingstartedjava.html#Serializing+and+deserializing+without+code+generation">specify a schema</a> (either manually or by retrieving it from some schema registry). With that schema, you can provide the right type information by either of the following options just like for the Row Types above:</p> + +<ul> + <li>implement <code>ResultTypeQueryable<GenericRecord></code>:</li> +</ul> + +<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">AvroGenericSource</span> <span class="kd">implements</span> <span class="n">SourceFunction</span><span class="o"><</span><span class="n">GenericRecord</span><span class="o">>,</span> <span class="n">ResultTypeQueryable</span><span class="o"><</span><span class="n">GenericRecord</span><span class="o">></span> <span [...] + <span class="kd">private</span> <span class="kd">final</span> <span class="n">GenericRecordAvroTypeInfo</span> <span class="n">producedType</span><span class="o">;</span> + + <span class="kd">public</span> <span class="nf">AvroGenericSource</span><span class="o">(</span><span class="n">Schema</span> <span class="n">schema</span><span class="o">)</span> <span class="o">{</span> + <span class="k">this</span><span class="o">.</span><span class="na">producedType</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">GenericRecordAvroTypeInfo</span><span class="o">(</span><span class="n">schema</span><span class="o">);</span> + <span class="o">}</span> + + <span class="nd">@Override</span> + <span class="kd">public</span> <span class="n">TypeInformation</span><span class="o"><</span><span class="n">GenericRecord</span><span class="o">></span> <span class="nf">getProducedType</span><span class="o">()</span> <span class="o">{</span> + <span class="k">return</span> <span class="n">producedType</span><span class="o">;</span> + <span class="o">}</span> +<span class="o">}</span></code></pre></div> +<ul> + <li>provide type information when building the job graph by using <code>SingleOutputStreamOperator#returns()</code></li> +</ul> + +<div class="highlight"><pre><code class="language-java"><span class="n">DataStream</span><span class="o"><</span><span class="n">GenericRecord</span><span class="o">></span> <span class="n">sourceStream</span> <span class="o">=</span> + <span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o">(</span><span class="k">new</span> <span class="nf">AvroGenericSource</span><span class="o">())</span> + <span class="o">.</span><span class="na">returns</span><span class="o">(</span><span class="k">new</span> <span class="nf">GenericRecordAvroTypeInfo</span><span class="o">(</span><span class="n">schema</span><span class="o">));</span></code></pre></div> +<p>Without this type information, Flink will fall back to Kryo for serialization which would serialize the schema into every record, over and over again. As a result, the serialized form will be bigger and more costly to create.</p> + +<div class="alert alert-info"> + <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span> + Since Avro’s <code>Schema</code> class is not serializable, it can not be sent around as is. You can work around this by converting it to a String and parsing it back when needed. If you only do this once on initialization, there is practically no difference to sending it directly.</p> +</div> + +<h3 id="avro-reflect">Avro Reflect</h3> + +<p>The third way of using Avro is to exchange Flink’s PojoSerializer (for POJOs according to the rules above) for Avro’s reflection-based serializer. This can be enabled by calling</p> + +<div class="highlight"><pre><code class="language-java"><span class="n">env</span><span class="o">.</span><span class="na">getConfig</span><span class="o">().</span><span class="na">enableForceAvro</span><span class="o">();</span></code></pre></div> + +<h2 id="kryo">Kryo</h2> + +<p>Any class or object which does not fall into the categories above or is covered by a Flink-provided special serializer is de/serialized with a fallback to <a href="https://github.com/EsotericSoftware/kryo">Kryo</a> (currently version 2.24.0) which is a powerful and generic serialization framework in Java. Flink calls such a type a <em>generic type</em> and you may stumble upon <code>GenericTypeInfo</code> when debugging code. If you are using Kryo serialization, make sure to register [...] + +<div class="highlight"><pre><code class="language-java"><span class="n">env</span><span class="o">.</span><span class="na">getConfig</span><span class="o">().</span><span class="na">registerKryoType</span><span class="o">(</span><span class="n">MyCustomType</span><span class="o">.</span><span class="na">class</span><span class="o">);</span></code></pre></div> +<p>Registering types adds them to an internal map of classes to tags so that, during serialization, Kryo does not have to add the fully qualified class names as a prefix into the serialized form. Instead, Kryo uses these (integer) tags to identify the underlying classes and reduce serialization overhead.</p> + +<div class="alert alert-info"> + <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span> +Flink will store Kryo serializer mappings from type registrations in its checkpoints and savepoints and will retain them across job (re)starts.</p> +</div> + +<h3 id="disabling-kryo">Disabling Kryo</h3> + +<p>If desired, you can disable the Kryo fallback, i.e. the ability to serialize generic types, by calling</p> + +<div class="highlight"><pre><code class="language-java"><span class="n">env</span><span class="o">.</span><span class="na">getConfig</span><span class="o">().</span><span class="na">disableGenericTypes</span><span class="o">();</span></code></pre></div> + +<p>This is mostly useful for finding out where these fallbacks are applied and replacing them with better serializers. If your job has any generic types with this configuration, it will fail with</p> + +<blockquote> + <p>Exception in thread “main” java.lang.UnsupportedOperationException: Generic types have been disabled in the ExecutionConfig and type … is treated as a generic type.</p> +</blockquote> + +<p>If you cannot immediately see from the type where it is being used, this log message also gives you a stacktrace that can be used to set breakpoints and find out more details in your IDE.</p> + +<h2 id="apache-thrift-via-kryo">Apache Thrift (via Kryo)</h2> + +<p>In addition to the variants above, Flink also allows you to <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/custom_serializers.html#register-a-custom-serializer-for-your-flink-program">register other type serialization frameworks</a> with Kryo. After adding the appropriate dependencies from the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/custom_serializers.html#register-a-custom-serializer-for-your-flink-program">documentation</a [...] + +<div class="highlight"><pre><code class="language-java"><span class="n">env</span><span class="o">.</span><span class="na">getConfig</span><span class="o">().</span><span class="na">addDefaultKryoSerializer</span><span class="o">(</span><span class="n">MyCustomType</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="n">TBaseSerializer</span><span class="o">.</span><span class="na">class</span><span class="o">);</span></code></pre></div> + +<p>This only works if generic types are not disabled and <code>MyCustomType</code> is a Thrift-generated data type. If the data type is not generated by Thrift, Flink will fail at runtime with an exception like this:</p> + +<blockquote> + <p>java.lang.ClassCastException: class MyCustomType cannot be cast to class org.apache.thrift.TBase (MyCustomType and org.apache.thrift.TBase are in unnamed module of loader ‘app’)</p> +</blockquote> + +<div class="alert alert-info"> + <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span> +Please note that <code>TBaseSerializer</code> can be registered as a default Kryo serializer as above (and as specified in <a href="https://github.com/twitter/chill/blob/v0.7.6/chill-thrift/src/main/java/com/twitter/chill/thrift/TBaseSerializer.java">its documentation</a>) or via <code>registerTypeWithKryoSerializer</code>. In practice, we found both ways working. We also saw no difference between registering Thrift classes in addition to the call above. Both may be different in your sce [...] +</div> + +<h2 id="protobuf-via-kryo">Protobuf (via Kryo)</h2> + +<p>In a way similar to Apache Thrift, <a href="https://developers.google.com/protocol-buffers/">Google Protobuf</a> may be <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/custom_serializers.html#register-a-custom-serializer-for-your-flink-program">registered as a custom serializer</a> after adding the right dependencies (<code>com.twitter:chill-protobuf</code> and <code>com.google.protobuf:protobuf-java</code>):</p> + +<div class="highlight"><pre><code class="language-java"><span class="n">env</span><span class="o">.</span><span class="na">getConfig</span><span class="o">().</span><span class="na">registerTypeWithKryoSerializer</span><span class="o">(</span><span class="n">MyCustomType</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="n">ProtobufSerializer</span><span class="o">.</span><span class="na">class</span><span class="o">);</span></code></pre></div> +<p>This will work as long as generic types have not been disabled (this would disable Kryo for good). If <code>MyCustomType</code> is not a Protobuf-generated class, your Flink job will fail at runtime with the following exception:</p> + +<blockquote> + <p>java.lang.ClassCastException: class <code>MyCustomType</code> cannot be cast to class com.google.protobuf.Message (<code>MyCustomType</code> and com.google.protobuf.Message are in unnamed module of loader ‘app’)</p> +</blockquote> + +<div class="alert alert-info"> + <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span> +Please note that <code>ProtobufSerializer</code> can be registered as a default Kryo serializer (as specified in the <a href="https://github.com/twitter/chill/blob/v0.7.6/chill-thrift/src/main/java/com/twitter/chill/thrift/TBaseSerializer.java">Protobuf documentation</a>) or via <code>registerTypeWithKryoSerializer</code> (as presented here). In practice, we found both ways working. We also saw no difference between registering your Protobuf classes in addition to the call above. Both ma [...] +</div> + +<h1 id="state-schema-evolution">State Schema Evolution</h1> + +<p>Before taking a closer look at the performance of each of the serializers described above, we would like to emphasize that performance is not everything that counts inside a real-world Flink job. Types for storing state, for example, should be able to evolve their schema (add/remove/change fields) throughout the lifetime of the job without losing previous state. This is what Flink calls <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/schema_evolution.h [...] + +<h1 id="performance-comparison">Performance Comparison</h1> + +<p>With so many options for serialization, it is actually not easy to make the right choice. We already saw some technical advantages and disadvantages of each of them outlined above. Since serializers are at the core of your Flink jobs and usually also sit on the hot path (per record invocations), let us actually take a deeper look into their performance with the help of the Flink benchmarks project at <a href="https://github.com/dataArtisans/flink-benchmarks">https://github.com/dataArt [...] + +<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyPojo</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">int</span> <span class="n">id</span><span class="o">;</span> + <span class="kd">private</span> <span class="n">String</span> <span class="n">name</span><span class="o">;</span> + <span class="kd">private</span> <span class="n">String</span><span class="o">[]</span> <span class="n">operationNames</span><span class="o">;</span> + <span class="kd">private</span> <span class="n">MyOperation</span><span class="o">[]</span> <span class="n">operations</span><span class="o">;</span> + <span class="kd">private</span> <span class="kt">int</span> <span class="n">otherId1</span><span class="o">;</span> + <span class="kd">private</span> <span class="kt">int</span> <span class="n">otherId2</span><span class="o">;</span> + <span class="kd">private</span> <span class="kt">int</span> <span class="n">otherId3</span><span class="o">;</span> + <span class="kd">private</span> <span class="n">Object</span> <span class="n">someObject</span><span class="o">;</span> +<span class="o">}</span> +<span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyOperation</span> <span class="o">{</span> + <span class="kt">int</span> <span class="n">id</span><span class="o">;</span> + <span class="kd">protected</span> <span class="n">String</span> <span class="n">name</span><span class="o">;</span> +<span class="o">}</span></code></pre></div> + +<p>This is mapped to tuples, rows, Avro specific records, Thrift and Protobuf representations appropriately and sent through a simple Flink job at parallelism 4 where the data type is used during network communication like this:</p> + +<div class="highlight"><pre><code class="language-java"><span class="n">env</span><span class="o">.</span><span class="na">setParallelism</span><span class="o">(</span><span class="mi">4</span><span class="o">);</span> +<span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o">(</span><span class="k">new</span> <span class="nf">PojoSource</span><span class="o">(</span><span class="n">RECORDS_PER_INVOCATION</span><span class="o">,</span> <span class="mi">10</span><span class="o">))</span> + <span class="o">.</span><span class="na">rebalance</span><span class="o">()</span> + <span class="o">.</span><span class="na">addSink</span><span class="o">(</span><span class="k">new</span> <span class="n">DiscardingSink</span><span class="o"><>());</span></code></pre></div> +<p>After running this through the <a href="http://openjdk.java.net/projects/code-tools/jmh/">jmh</a> micro-benchmarks defined in <a href="https://github.com/dataArtisans/flink-benchmarks/blob/master/src/main/java/org/apache/flink/benchmark/full/SerializationFrameworkAllBenchmarks.java">SerializationFrameworkAllBenchmarks.java</a>, I retrieved the following performance results for Flink 1.10 on my machine (in number of operations per millisecond): +<br /></p> + +<center> +<img src="/img/blog/2020-04-15-flink-serialization-performance-results.svg" width="800px" alt="Communication between the Flink operator and the Python execution environment" /> +</center> +<p><br /></p> + +<p>A few takeaways from these numbers:</p> + +<ul> + <li> + <p>The default fallback from POJO to Kryo reduces performance by 75%.<br /> +Registering types with Kryo significantly improves its performance with only 64% fewer operations than by using a POJO.</p> + </li> + <li> + <p>Avro GenericRecord and SpecificRecord are roughly serialized at the same speed.</p> + </li> + <li> + <p>Avro Reflect serialization is even slower than Kryo default (-45%).</p> + </li> + <li> + <p>Tuples are the fastest, closely followed by Rows. Both leverage fast specialized serialization code based on direct access without Java reflection.</p> + </li> + <li> + <p>Using a (nested) Tuple instead of a POJO may speed up your job by 42% (but is less flexible!). + Having code-generation for the PojoSerializer (<a href="https://jira.apache.org/jira/browse/FLINK-3599">FLINK-3599</a>) may actually close that gap (or at least move closer to the RowSerializer). If you feel like giving the implementation a go, please give the Flink community a note and we will see whether we can make that happen.</p> + </li> + <li> + <p>If you cannot use POJOs, try to define your data type with one of the serialization frameworks that generate specific code for it: Protobuf, Avro, Thrift (in that order, performance-wise).</p> + </li> +</ul> + +<div class="alert alert-info"> + <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span> As with all benchmarks, please bear in mind that these numbers only give a hint on Flink’s serializer performance in a specific scenario. They may be different with your data types but the rough classification is probably the same. If you want to be sure, please verify the results with your data types. You should be able to copy from <code>S [...] +</div> + +<h1 id="conclusion">Conclusion</h1> + +<p>In the sections above, we looked at how Flink performs serialization for different sorts of data types and elaborated the technical advantages and disadvantages. For data types used in Flink state, you probably want to leverage either POJO or Avro types which, currently, are the only ones supporting state evolution out of the box and allow your stateful application to develop over time. POJOs are usually faster in the de/serialization while Avro may support more flexible schema evolut [...] + +<p>The fastest de/serialization is achieved with Flink’s internal tuple and row serializers which can access these types’ fields directly without going via reflection. With roughly 30% decreased throughput as compared to tuples, Protobuf and POJO types do not perform too badly on their own and are more flexible and maintainable. Avro (specific and generic) records as well as Thrift data types further reduce performance by 20% and 30%, respectively. You definitely want to avoid Kryo as th [...] + +<p>The next article in this series will use this finding as a starting point to look into a few common pitfalls and obstacles of avoiding Kryo, how to get the most out of the PojoSerializer, and a few more tuning techniques with respect to serialization. Stay tuned for more.</p> + + </article> + </div> + + <div class="row"> + <div id="disqus_thread"></div> + <script type="text/javascript"> + /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */ + var disqus_shortname = 'stratosphere-eu'; // required: replace example with your forum shortname + + /* * * DON'T EDIT BELOW THIS LINE * * */ + (function() { + var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; + dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js'; + (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); + })(); + </script> + </div> + </div> +</div> + </div> + </div> + + <hr /> + + <div class="row"> + <div class="footer text-center col-sm-12"> + <p>Copyright © 2014-2019 <a href="http://apache.org">The Apache Software Foundation</a>. All Rights Reserved.</p> + <p>Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.</p> + <p><a href="/privacy-policy.html">Privacy Policy</a> · <a href="/blog/feed.xml">RSS feed</a></p> + </div> + </div> + </div><!-- /.container --> + + <!-- Include all compiled plugins (below), or include individual files as needed --> + <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js"></script> + <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery.matchHeight/0.7.0/jquery.matchHeight-min.js"></script> + <script src="/js/codetabs.js"></script> + <script src="/js/stickysidebar.js"></script> + + <!-- Google Analytics --> + <script> + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + + ga('create', 'UA-52545728-1', 'auto'); + ga('send', 'pageview'); + </script> + </body> +</html> diff --git a/content/zh/index.html b/content/zh/index.html index 2ba5e74..c67dadd 100644 --- a/content/zh/index.html +++ b/content/zh/index.html @@ -564,6 +564,9 @@ <dl> + <dt> <a href="/news/2020/04/15/flink-serialization-tuning-vol-1.html">Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</a></dt> + <dd>Serialization is a crucial element of your Flink job. This article is the first in a series of posts that will highlight Flink’s serialization stack, and looks at the different ways Flink can serialize your data types.</dd> + <dt> <a href="/2020/04/09/pyflink-udf-support-flink.html">PyFlink: Introducing Python Support for UDFs in Flink's Table API</a></dt> <dd>Flink 1.10 extends its support for Python by adding Python UDFs in PyFlink. This post explains how UDFs work in PyFlink and gives some practical examples of how to use UDFs in PyFlink.</dd> @@ -580,9 +583,6 @@ This release marks a big milestone: Stateful Functions 2.0 is not only an API up <dd><p>In this blog post, you will learn our motivation behind the Flink-Hive integration, and how Flink 1.10 can help modernize your data warehouse.</p> </dd> - - <dt> <a href="/news/2020/03/24/demo-fraud-detection-2.html">Advanced Flink Application Patterns Vol.2: Dynamic Updates of Application Logic</a></dt> - <dd>In this series of blog posts you will learn about powerful Flink patterns for building streaming applications.</dd> </dl>