This is an automated email from the ASF dual-hosted git repository.
hequn pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit c1c91ecb7dcd23355b58585a5d154e1d05d52765
Author: hequn8128 <chenghe...@gmail.com>
AuthorDate: Mon Jun 1 20:09:06 2020 +0800
Rebuild website
---
content/blog/feed.xml | 883 ++++++++++++++++++--------
content/news/2020/05/07/community-update.html | 2 +-
2 files changed, 630 insertions(+), 255 deletions(-)
diff --git a/content/blog/feed.xml b/content/blog/feed.xml
index 042b7bd..53492a2 100644
--- a/content/blog/feed.xml
+++ b/content/blog/feed.xml
@@ -7,6 +7,629 @@
<atom:link href="https://flink.apache.org/blog/feed.xml"; rel="self"
type="application/rss+xml" />
<item>
+<title>Apache Flink 1.10.1 Released</title>
+<description><p>The Apache Flink community released the first bugfix
version of the Apache Flink 1.10 series.</p>
+
+<p>This release includes 158 fixes and minor improvements for Flink
1.10.0. The list below includes a detailed list of all fixes and
improvements.</p>
+
+<p>We highly recommend all users to upgrade to Flink 1.10.1.</p>
+
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display:
inline-block"><span class="glyphicon glyphicon-info-sign"
aria-hidden="true"></span> Note</span>
+FLINK-16684 changed the builders of the StreamingFileSink to make them
compilable in Scala. This change is source compatible but binary incompatible.
If using the StreamingFileSink, please recompile your user code against 1.10.1
before upgrading.</p>
+</div>
+
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display:
inline-block"><span class="glyphicon glyphicon-info-sign"
aria-hidden="true"></span> Note</span>
+FLINK-16683 Flink no longer supports starting clusters with .bat scripts.
Users should instead use environments like WSL or Cygwin and work with the .sh
scripts.</p>
+</div>
+
+<p>Updated Maven dependencies:</p>
+
+<div class="highlight"><pre><code
class="language-xml"><span
class="nt">&lt;dependency&gt;</span>
+ <span
class="nt">&lt;groupId&gt;</span>org.apache.flink<span
class="nt">&lt;/groupId&gt;</span>
+ <span
class="nt">&lt;artifactId&gt;</span>flink-java<span
class="nt">&lt;/artifactId&gt;</span>
+ <span
class="nt">&lt;version&gt;</span>1.10.1<span
class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span
class="nt">&lt;groupId&gt;</span>org.apache.flink<span
class="nt">&lt;/groupId&gt;</span>
+ <span
class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span
class="nt">&lt;/artifactId&gt;</span>
+ <span
class="nt">&lt;version&gt;</span>1.10.1<span
class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span
class="nt">&lt;groupId&gt;</span>org.apache.flink<span
class="nt">&lt;/groupId&gt;</span>
+ <span
class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span
class="nt">&lt;/artifactId&gt;</span>
+ <span
class="nt">&lt;version&gt;</span>1.10.1<span
class="nt">&lt;/version&gt;</span>
+<span
class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>You can find the binaries on the updated <a
href="/downloads.html">Downloads page</a>.</p>
+
+<p>List of resolved issues:</p>
+
+<h2> Sub-task
+</h2>
+<ul>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-14126">FLINK-14126</a>]
- Elasticsearch Xpack Machine Learning doesn&#39;t support ARM
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-15143">FLINK-15143</a>]
- Create document for FLIP-49 TM memory model and configuration guide
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-15561">FLINK-15561</a>]
- Unify Kerberos credentials checking
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-15790">FLINK-15790</a>]
- Make FlinkKubeClient and its implementations asynchronous
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-15817">FLINK-15817</a>]
- Kubernetes Resource leak while deployment exception happens
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16049">FLINK-16049</a>]
- Remove outdated &quot;Best Practices&quot; section from
Application Development Section
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16131">FLINK-16131</a>]
- Translate &quot;Amazon S3&quot; page of &quot;File
Systems&quot; into Chinese
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16389">FLINK-16389</a>]
- Bump Kafka 0.10 to 0.10.2.2
+</li>
+</ul>
+
+<h2> Bug
+</h2>
+<ul>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-2336">FLINK-2336</a>]
- ArrayIndexOufOBoundsException in TypeExtractor when mapping
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-10918">FLINK-10918</a>]
- incremental Keyed state with RocksDB throws cannot create directory
error in windows
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-11193">FLINK-11193</a>]
- Rocksdb timer service factory configuration option is not settable
per job
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-13483">FLINK-13483</a>]
- PrestoS3FileSystemITCase.testDirectoryListing fails on Travis
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-14038">FLINK-14038</a>]
- ExecutionGraph deploy failed due to akka timeout
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-14311">FLINK-14311</a>]
- Streaming File Sink end-to-end test failed on Travis
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-14316">FLINK-14316</a>]
- Stuck in &quot;Job leader ... lost leadership&quot; error
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-15417">FLINK-15417</a>]
- Remove the docker volume or mount when starting Mesos e2e cluster
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-15669">FLINK-15669</a>]
- SQL client can&#39;t cancel flink job
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-15772">FLINK-15772</a>]
- Shaded Hadoop S3A with credentials provider end-to-end test fails on
travis
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-15811">FLINK-15811</a>]
- StreamSourceOperatorWatermarksTest.testNoMaxWatermarkOnAsyncCancel
fails on Travis
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-15812">FLINK-15812</a>]
- HistoryServer archiving is done in Dispatcher main thread
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-15838">FLINK-15838</a>]
- Dangling CountDownLatch.await(timeout)
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-15852">FLINK-15852</a>]
- Job is submitted to the wrong session cluster
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-15904">FLINK-15904</a>]
- Make Kafka Consumer work with activated
&quot;disableGenericTypes()&quot;
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-15936">FLINK-15936</a>]
- TaskExecutorTest#testSlotAcceptance deadlocks
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-15953">FLINK-15953</a>]
- Job Status is hard to read for some Statuses
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16013">FLINK-16013</a>]
- List and map config options could not be parsed correctly
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16014">FLINK-16014</a>]
- S3 plugin ClassNotFoundException SAXParser
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16025">FLINK-16025</a>]
- Service could expose blob server port mismatched with JM Container
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16026">FLINK-16026</a>]
- Travis failed due to python setup
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16047">FLINK-16047</a>]
- Blink planner produces wrong aggregate results with state clean up
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16067">FLINK-16067</a>]
- Flink&#39;s CalciteParser swallows error position information
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16068">FLINK-16068</a>]
- table with keyword-escaped columns and computed_column_expression
columns
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16070">FLINK-16070</a>]
- Blink planner can not extract correct unique key for
UpsertStreamTableSink
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16108">FLINK-16108</a>]
- StreamSQLExample is failed if running in blink planner
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16111">FLINK-16111</a>]
- Kubernetes deployment does not respect
&quot;taskmanager.cpu.cores&quot;.
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16113">FLINK-16113</a>]
- ExpressionReducer shouldn&#39;t escape the reduced string value
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16115">FLINK-16115</a>]
- Aliyun oss filesystem could not work with plugin mechanism
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16139">FLINK-16139</a>]
- Co-location constraints are not reset on task recovery in
DefaultScheduler
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16161">FLINK-16161</a>]
- Statistics zero should be unknown in HiveCatalog
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16170">FLINK-16170</a>]
- SearchTemplateRequest ClassNotFoundException when use
flink-sql-connector-elasticsearch7
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16220">FLINK-16220</a>]
- JsonRowSerializationSchema throws cast exception : NullNode cannot
be cast to ArrayNode
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16231">FLINK-16231</a>]
- Hive connector is missing jdk.tools exclusion against Hive 2.x.x
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16234">FLINK-16234</a>]
- Fix unstable cases in StreamingJobGraphGeneratorTest
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16241">FLINK-16241</a>]
- Remove the license and notice file in flink-ml-lib module on
release-1.10 branch
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16242">FLINK-16242</a>]
- BinaryGeneric serialization error cause checkpoint failure
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16262">FLINK-16262</a>]
- Class loader problem with FlinkKafkaProducer.Semantic.EXACTLY_ONCE
and usrlib directory
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16269">FLINK-16269</a>]
- Generic type can not be matched when convert table to stream.
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16281">FLINK-16281</a>]
- parameter &#39;maxRetryTimes&#39; can not work in
JDBCUpsertTableSink
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16301">FLINK-16301</a>]
- Annoying &quot;Cannot find FunctionDefinition&quot; messages
with SQL for f_proctime or =
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16308">FLINK-16308</a>]
- SQL connector download links are broken
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16313">FLINK-16313</a>]
- flink-state-processor-api: surefire execution unstable on Azure
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16331">FLINK-16331</a>]
- Remove source licenses for old WebUI
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16345">FLINK-16345</a>]
- Computed column can not refer time attribute column
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16360">FLINK-16360</a>]
- connector on hive 2.0.1 don&#39;t support type conversion from
STRING to VARCHAR
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16371">FLINK-16371</a>]
- HadoopCompressionBulkWriter fails with
&#39;java.io.NotSerializableException&#39;
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16373">FLINK-16373</a>]
- EmbeddedLeaderService: IllegalStateException: The RPC connection is
already closed
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16413">FLINK-16413</a>]
- Reduce hive source parallelism when limit push down
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16414">FLINK-16414</a>]
- create udaf/udtf function using sql casuing ValidationException: SQL
validation failed. null
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16433">FLINK-16433</a>]
- TableEnvironmentImpl doesn&#39;t clear buffered operations when
it fails to translate the operation
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16435">FLINK-16435</a>]
- Replace since decorator with versionadd to mark the version an API
was introduced
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16467">FLINK-16467</a>]
- MemorySizeTest#testToHumanReadableString() is not portable
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16526">FLINK-16526</a>]
- Fix exception when computed column expression references a keyword
column name
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16541">FLINK-16541</a>]
- Document of table.exec.shuffle-mode is incorrect
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16550">FLINK-16550</a>]
- HadoopS3* tests fail with NullPointerException exceptions
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16560">FLINK-16560</a>]
- Forward Configuration in PackagedProgramUtils#getPipelineFromProgram
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16567">FLINK-16567</a>]
- Get the API error of the StreamQueryConfig on Page &quot;Query
Configuration&quot;
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16573">FLINK-16573</a>]
- Kinesis consumer does not properly shutdown RecordFetcher threads
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16576">FLINK-16576</a>]
- State inconsistency on restore with memory state backends
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16626">FLINK-16626</a>]
- Prevent REST handler from being closed more than once
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16632">FLINK-16632</a>]
- SqlDateTimeUtils#toSqlTimestamp(String, String) may yield incorrect
result
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16635">FLINK-16635</a>]
- Incompatible okio dependency in flink-metrics-influxdb module
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16646">FLINK-16646</a>]
- flink read orc file throw a NullPointerException
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16647">FLINK-16647</a>]
- Miss file extension when inserting to hive table with compression
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16652">FLINK-16652</a>]
- BytesColumnVector should init buffer in Hive 3.x
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16662">FLINK-16662</a>]
- Blink Planner failed to generate JobGraph for POJO DataStream
converting to Table (Cannot determine simple type name)
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16664">FLINK-16664</a>]
- Unable to set DataStreamSource parallelism to default (-1)
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16675">FLINK-16675</a>]
- TableEnvironmentITCase. testClearOperation fails on travis nightly
build
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16684">FLINK-16684</a>]
- StreamingFileSink builder does not work with Scala
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16696">FLINK-16696</a>]
- Savepoint trigger documentation is insufficient
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16703">FLINK-16703</a>]
- AkkaRpcActor state machine does not record transition to terminating
state.
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16705">FLINK-16705</a>]
- LocalExecutor tears down MiniCluster before client can retrieve
JobResult
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16718">FLINK-16718</a>]
- KvStateServerHandlerTest leaks Netty ByteBufs
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16727">FLINK-16727</a>]
- Fix cast exception when having time point literal as parameters
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16732">FLINK-16732</a>]
- Failed to call Hive UDF with constant return value
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16740">FLINK-16740</a>]
- OrcSplitReaderUtil::logicalTypeToOrcType fails to create decimal
type with precision &lt; 10
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16759">FLINK-16759</a>]
- HiveModuleTest failed to compile on release-1.10
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16767">FLINK-16767</a>]
- Failed to read Hive table with RegexSerDe
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16771">FLINK-16771</a>]
- NPE when filtering by decimal column
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16821">FLINK-16821</a>]
- Run Kubernetes test failed with invalid named
&quot;minikube&quot;
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16822">FLINK-16822</a>]
- The config set by SET command does not work
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16825">FLINK-16825</a>]
- PrometheusReporterEndToEndITCase should rely on path returned by
DownloadCache
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16836">FLINK-16836</a>]
- Losing leadership does not clear rpc connection in
JobManagerLeaderListener
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16860">FLINK-16860</a>]
- Failed to push filter into OrcTableSource when upgrading to 1.9.2
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16888">FLINK-16888</a>]
- Re-add jquery license file under &quot;/licenses&quot;
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16901">FLINK-16901</a>]
- Flink Kinesis connector NOTICE should have contents of AWS
KPL&#39;s THIRD_PARTY_NOTICES file manually merged in
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16913">FLINK-16913</a>]
- ReadableConfigToConfigurationAdapter#getEnum throws
UnsupportedOperationException
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16916">FLINK-16916</a>]
- The logic of NullableSerializer#copy is wrong
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16944">FLINK-16944</a>]
- Compile error in. DumpCompiledPlanTest and PreviewPlanDumpTest
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16980">FLINK-16980</a>]
- Python UDF doesn&#39;t work with protobuf 3.6.1
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16981">FLINK-16981</a>]
- flink-runtime tests are crashing the JVM on Java11 because of
PowerMock
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17062">FLINK-17062</a>]
- Fix the conversion from Java row type to Python row type
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17066">FLINK-17066</a>]
- Update pyarrow version bounds less than 0.14.0
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17093">FLINK-17093</a>]
- Python UDF doesn&#39;t work when the input column is from
composite field
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17107">FLINK-17107</a>]
- CheckpointCoordinatorConfiguration#isExactlyOnce() is inconsistent
with StreamConfig#getCheckpointMode()
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17114">FLINK-17114</a>]
- When the pyflink job runs in local mode and the command
&quot;python&quot; points to Python 2.7, the startup of the Python UDF
worker will fail.
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17124">FLINK-17124</a>]
- The PyFlink Job runs into infinite loop if the Python UDF imports
job code
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17152">FLINK-17152</a>]
- FunctionDefinitionUtil generate wrong resultType and acc type of
AggregateFunctionDefinition
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17308">FLINK-17308</a>]
- ExecutionGraphCache cachedExecutionGraphs not cleanup cause OOM Bug
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17313">FLINK-17313</a>]
- Validation error when insert decimal/varchar with precision into
sink using TypeInformation of row
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17334">FLINK-17334</a>]
- Flink does not support HIVE UDFs with primitive return types
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17338">FLINK-17338</a>]
- LocalExecutorITCase.testBatchQueryCancel test timeout
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17359">FLINK-17359</a>]
- Entropy key is not resolved if flink-s3-fs-hadoop is added as a
plugin
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17403">FLINK-17403</a>]
- Fix invalid classpath in BashJavaUtilsITCase
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17471">FLINK-17471</a>]
- Move LICENSE and NOTICE files to root directory of python
distribution
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17483">FLINK-17483</a>]
- Update flink-sql-connector-elasticsearch7 NOTICE file to correctly
reflect bundled dependencies
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17496">FLINK-17496</a>]
- Performance regression with amazon-kinesis-producer 0.13.1 in Flink
1.10.x
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17499">FLINK-17499</a>]
- LazyTimerService used to register timers via State Processing API
incorrectly mixes event time timers with processing time timers
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17514">FLINK-17514</a>]
- TaskCancelerWatchdog does not kill TaskManager
+</li>
+</ul>
+
+<h2> New Feature
+</h2>
+<ul>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17275">FLINK-17275</a>]
- Add core training exercises
+</li>
+</ul>
+
+<h2> Improvement
+</h2>
+<ul>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-9656">FLINK-9656</a>]
- Environment java opts for flink run
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-15094">FLINK-15094</a>]
- Warning about using private constructor of java.nio.DirectByteBuffer
in Java 11
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-15584">FLINK-15584</a>]
- Give nested data type of ROWs in ValidationException
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-15616">FLINK-15616</a>]
- Move boot error messages from python-udf-boot.log to
taskmanager&#39;s log file
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-15989">FLINK-15989</a>]
- Rewrap OutOfMemoryError in allocateUnpooledOffHeap with better
message
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16018">FLINK-16018</a>]
- Improve error reporting when submitting batch job (instead of
AskTimeoutException)
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16125">FLINK-16125</a>]
- Make zookeeper.connect optional for Kafka connectors
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16167">FLINK-16167</a>]
- Update documentation about python shell execution
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16191">FLINK-16191</a>]
- Improve error message on Windows when RocksDB Paths are too long
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16280">FLINK-16280</a>]
- Fix sample code errors in the documentation about elasticsearch
connector
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16288">FLINK-16288</a>]
- Setting the TTL for discarding task pods on Kubernetes.
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16293">FLINK-16293</a>]
- Document using plugins in Kubernetes
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16343">FLINK-16343</a>]
- Improve exception message when reading an unbounded source in batch
mode
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16406">FLINK-16406</a>]
- Increase default value for JVM Metaspace to minimise its
OutOfMemoryError
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16538">FLINK-16538</a>]
- Restructure Python Table API documentation
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16604">FLINK-16604</a>]
- Column key in JM configuration is too narrow
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16683">FLINK-16683</a>]
- Remove scripts for starting Flink on Windows
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16697">FLINK-16697</a>]
- Disable JMX rebinding
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16763">FLINK-16763</a>]
- Should not use BatchTableEnvironment for Python UDF in the document
of flink-1.10
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16772">FLINK-16772</a>]
- Bump derby to 10.12.1.1+ or exclude it
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16790">FLINK-16790</a>]
- enables the interpretation of backslash escapes
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16862">FLINK-16862</a>]
- Remove example url in quickstarts
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16874">FLINK-16874</a>]
- Respect the dynamic options when calculating memory options in
taskmanager.sh
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16942">FLINK-16942</a>]
- ES 5 sink should allow users to select netty transport client
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17065">FLINK-17065</a>]
- Add documentation about the Python versions supported for PyFlink
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17125">FLINK-17125</a>]
- Add a Usage Notes Page to Answer Common Questions Encountered by
PyFlink Users
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17254">FLINK-17254</a>]
- Improve the PyFlink documentation and examples to use SQL DDL for
source/sink definition
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17276">FLINK-17276</a>]
- Add checkstyle to training exercises
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17277">FLINK-17277</a>]
- Apply IntelliJ recommendations to training exercises
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17278">FLINK-17278</a>]
- Add Travis to the training exercises
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17279">FLINK-17279</a>]
- Use gradle build scans for training exercises
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-17316">FLINK-17316</a>]
- Have HourlyTips solutions use TumblingEventTimeWindows.of
+</li>
+</ul>
+
+<h2> Task
+</h2>
+<ul>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-15741">FLINK-15741</a>]
- Fix TTL docs after enabling RocksDB compaction filter by default
(needs Chinese translation)
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-15933">FLINK-15933</a>]
- update content of how generic table schema is stored in hive via
HiveCatalog
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-15991">FLINK-15991</a>]
- Create Chinese documentation for FLIP-49 TM memory model
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16004">FLINK-16004</a>]
- Exclude flink-rocksdb-state-memory-control-test jars from the dist
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16454">FLINK-16454</a>]
- Update the copyright year in NOTICE files
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16530">FLINK-16530</a>]
- Add documentation about &quot;GROUPING SETS&quot; and
&quot;CUBE&quot; support in streaming mode
+</li>
+<li>[<a
href="https://issues.apache.org/jira/browse/FLINK-16592">FLINK-16592</a>]
- The doc of Streaming File Sink has a mistake of grammar
+</li>
+</ul>
+
+</description>
+<pubDate>Tue, 12 May 2020 14:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2020/05/12/release-1.10.1.html</link>
+<guid isPermaLink="true">/news/2020/05/12/release-1.10.1.html</guid>
+</item>
+
+<item>
+<title>Flink Community Update - May'20</title>
+<description><p>Can you smell it? It’s release month! It took a while,
but now that we’re <a
href="https://flink.apache.org/news/2020/04/01/community-update.html">all
caught up with the past</a>, the Community Update is here to stay. This
time around, we’re warming up for Flink 1.11 and peeping back to the month of
April in the Flink community — with the release of Stateful Functions 2.0, a
new self-paced Flink training and some efforts to improve the Flink do [...]
+
+<p>Last month also marked the debut of Flink Forward Virtual Conference
2020: what did you think? If you missed it altogether or just want to recap
some of the sessions, the <a
href="https://www.youtube.com/playlist?list=PLDX4T_cnKjD0ngnBSU-bYGfgVv17MiwA7">videos</a>
and <a
href="https://www.slideshare.net/FlinkForward">slides</a>
are now available!</p>
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#the-past-month-in-flink"
id="markdown-toc-the-past-month-in-flink">The Past Month in
Flink</a> <ul>
+ <li><a href="#flink-stateful-functions-20-is-out"
id="markdown-toc-flink-stateful-functions-20-is-out">Flink
Stateful Functions 2.0 is out!</a></li>
+ <li><a href="#warming-up-for-flink-111"
id="markdown-toc-warming-up-for-flink-111">Warming up for Flink
1.11</a></li>
+ <li><a href="#flink-minor-releases"
id="markdown-toc-flink-minor-releases">Flink Minor
Releases</a> <ul>
+ <li><a href="#flink-193"
id="markdown-toc-flink-193">Flink 1.9.3</a></li>
+ <li><a href="#flink-1101"
id="markdown-toc-flink-1101">Flink 1.10.1</a></li>
+ </ul>
+ </li>
+ <li><a href="#new-committers-and-pmc-members"
id="markdown-toc-new-committers-and-pmc-members">New Committers
and PMC Members</a> <ul>
+ <li><a href="#new-pmc-members"
id="markdown-toc-new-pmc-members">New PMC
Members</a></li>
+ <li><a href="#new-committers"
id="markdown-toc-new-committers">New
Committers</a></li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li><a href="#the-bigger-picture"
id="markdown-toc-the-bigger-picture">The Bigger Picture</a>
<ul>
+ <li><a href="#a-new-self-paced-apache-flink-training"
id="markdown-toc-a-new-self-paced-apache-flink-training">A new
self-paced Apache Flink training</a></li>
+ <li><a href="#google-season-of-docs-2020"
id="markdown-toc-google-season-of-docs-2020">Google Season of Docs
2020</a></li>
+ </ul>
+ </li>
+ <li><a href="#and-something-to-read"
id="markdown-toc-and-something-to-read">…and something to
read!</a></li>
+</ul>
+
+</div>
+
+<h1 id="the-past-month-in-flink">The Past Month in
Flink</h1>
+
+<h2 id="flink-stateful-functions-20-is-out">Flink Stateful
Functions 2.0 is out!</h2>
+
+<p>In the beginning of April, the Flink community announced the <a
href="https://flink.apache.org/news/2020/04/07/release-statefun-2.0.0.html">release
of Stateful Functions 2.0</a> — the first as part of the Apache Flink
project. From this release, you can use Flink as the base of a (stateful)
serverless platform with out-of-the-box consistent and scalable state, and
efficient messaging between functions. You can even run your stateful functions
on platforms l [...]
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<center>
+<img
src="/img/blog/2020-05-06-community-update/2020-05-06-community-update_2.png"
width="550px" alt="Stateful Functions" />
+</center>
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<p>It’s been encouraging to see so many questions about Stateful
Functions popping up in the <a
href="https://lists.apache.org/list.html?u...@flink.apache.org:lte=3M:statefun">mailing
list</a> and Stack Overflow! If you’d like to get involved, we’re always
<a
href="https://github.com/apache/flink-statefun#contributing">looking
for new contributors</a> — especially around SDKs for other languages
like Go, Javascript and Rust.</p>
+
+<hr />
+
+<h2 id="warming-up-for-flink-111">Warming up for Flink
1.11</h2>
+
+<p>The final preparations for the release of Flink 1.11 are well
underway, with the feature freeze scheduled for May 15th, and there’s a lot of
new features and improvements to look out for:</p>
+
+<ul>
+ <li>
+ <p>On the <strong>usability</strong> side, you can
expect a big focus on smoothing data ingestion with contributions like support
for Change Data Capture (CDC) in the Table API/SQL (<a
href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-105%3A+Support+to+Interpret+and+Emit+Changelog+in+Flink+SQL">FLIP-105</a>),
easy streaming data ingestion into Apache Hive (<a
href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-115%3A [...]
+ </li>
+ <li>
+ <p>On the <strong>operational</strong> side, the much
anticipated new Source API (<a
href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface">FLIP-27</a>)
will unify batch and streaming sources, and improve out-of-the-box event-time
behavior; while unaligned checkpoints (<a
href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints">FLIP-76</a>)
and changes [...]
+ </li>
+</ul>
+
+<p>Throw into the mix improvements around type systems, the WebUI,
metrics reporting, supported formats and…we can’t wait! To get an overview of
the ongoing developments, have a look at <a
href="http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ANNOUNCE-Development-progress-of-Apache-Flink-1-11-tp40718.html">this
thread</a>. We encourage the community to get involved in testing once
an RC (Release Candidate) is out. Keep an eye on the <a href=& [...]
+
+<hr />
+
+<h2 id="flink-minor-releases">Flink Minor Releases</h2>
+
+<h3 id="flink-193">Flink 1.9.3</h3>
+
+<p>The community released Flink 1.9.3, covering some outstanding bugs
from Flink 1.9! You can find more in the <a
href="(https://flink.apache.org/news/2020/04/24/release-1.9.3.html)">announcement
blogpost</a>.</p>
+
+<h3 id="flink-1101">Flink 1.10.1</h3>
+
+<p>Also in the pipeline is the release of Flink 1.10.1, already in the
<a
href="http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-1-10-1-release-candidate-2-td41019.html">RC
voting</a> phase. So, you can expect Flink 1.10.1 to be released
soon!</p>
+
+<hr />
+
+<h2 id="new-committers-and-pmc-members">New Committers and PMC
Members</h2>
+
+<p>The Apache Flink community has welcomed <strong>3 PMC
Members</strong> and <strong>2 new Committers</strong> since
the last update. Congratulations!</p>
+
+<h3 id="new-pmc-members">New PMC Members</h3>
+
+<div class="row">
+ <div class="col-lg-3">
+ <div class="text-center">
+ <img class="img-circle"
src="https://avatars2.githubusercontent.com/u/6242259?s=400&amp;u=6e39f4fdbabc8ce4ccde9125166f791957d3ae80&amp;v=4"
width="90" height="90" />
+ <p><a href="https://twitter.com/dwysakowicz">Dawid
Wysakowicz</a></p>
+ </div>
+ </div>
+ <div class="col-lg-3">
+ <div class="text-center">
+ <img class="img-circle"
src="https://avatars1.githubusercontent.com/u/4971479?s=400&amp;u=49d4f217e26186606ab13a17a23a038b62b86682&amp;v=4"
width="90" height="90" />
+ <p><a href="https://twitter.com/HequnC">Hequn
Cheng</a></p>
+ </div>
+ </div>
+ <div class="col-lg-3">
+ <div class="text-center">
+ <img class="img-circle"
src="https://avatars3.githubusercontent.com/u/12387855?s=400&amp;u=37edbfccb6908541f359433f420f9f1bc25bc714&amp;v=4"
width="90" height="90" />
+ <p>Zhijiang Wang</p>
+ </div>
+ </div>
+</div>
+
+<h3 id="new-committers">New Committers</h3>
+
+<div class="row">
+ <div class="col-lg-3">
+ <div class="text-center">
+ <img class="img-circle"
src="https://avatars3.githubusercontent.com/u/11538663?s=400&amp;u=f4643f1981e2a8f8a1962c34511b0d32a31d9502&amp;v=4"
width="90" height="90" />
+ <p><a
href="https://twitter.com/snntrable">Konstantin
Knauf</a></p>
+ </div>
+ </div>
+ <div class="col-lg-3">
+ <div class="text-center">
+ <img class="img-circle"
src="https://avatars1.githubusercontent.com/u/1891970?s=400&amp;u=b7718355ceb1f4a8d1e554c3ae7221e2f32cc8e0&amp;v=4"
width="90" height="90" />
+ <p><a href="https://twitter.com/sjwiesman">Seth
Wiesman</a></p>
+ </div>
+ </div>
+</div>
+
+<hr />
+
+<h1 id="the-bigger-picture">The Bigger Picture</h1>
+
+<h2 id="a-new-self-paced-apache-flink-training">A new
self-paced Apache Flink training</h2>
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<p>This week, the Flink website received the invaluable contribution of
a self-paced training course curated by David (<a
href="https://twitter.com/alpinegizmo">@alpinegizmo</a>) —
or, what used to be the entire training materials under <a
href="training.ververica.com">training.ververica.com</a>.
The new materials guide you through the very basics of Flink and the DataStream
API, and round off every concepts section with hands-on exercise [...]
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<center>
+<img
src="/img/blog/2020-05-06-community-update/2020-05-06-community-update_1.png"
width="1000px" alt="Self-paced Flink Training" />
+</center>
+
+<div style="line-height:140%;">
+ <br />
+</div>
+
+<p>Whether you’re new to Flink or just looking to strengthen your
foundations, this training is the most comprehensive way to get started and is
now completely open source: <a
href="https://flink.apache.org/training.html">https://flink.apache.org/training.html</a>.
For now, the materials are only available in English, but the community
intends to also provide a Chinese translation in the future.</p>
+
+<hr />
+
+<h2 id="google-season-of-docs-2020">Google Season of Docs
2020</h2>
+
+<p>Google Season of Docs (GSOD) is a great initiative organized by <a
href="https://opensource.google.com/">Google Open Source</a>
to pair technical writers with mentors to work on documentation for open source
projects. Last year, the Flink community submitted <a
href="https://flink.apache.org/news/2019/04/17/sod.html">an
application</a> that unfortunately didn’t make the cut — but we are
trying again! This time, with a project idea to i [...]
+
+<p><strong>1) Restructure the Table API &amp; SQL
Documentation</strong></p>
+
+<p>Reworking the current documentation structure would allow
to:</p>
+
+<ul>
+ <li>
+ <p>Lower the entry barrier to Flink for non-programmatic (i.e. SQL)
users.</p>
+ </li>
+ <li>
+ <p>Make the available features more easily discoverable.</p>
+ </li>
+ <li>
+ <p>Improve the flow and logical correlation of topics.</p>
+ </li>
+</ul>
+
+<p><a
href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=127405685">FLIP-60</a>
contains a detailed proposal on how to reorganize the existing documentation,
which can be used as a starting point.</p>
+
+<p><strong>2) Extend the Table API &amp; SQL
Documentation</strong></p>
+
+<p>Some areas of the documentation have insufficient detail or are not
<a
href="https://flink.apache.org/contributing/docs-style.html#general-guiding-principles">accessible</a>
for new Flink users. Examples of topics and sections that require attention
are: planners, built-in functions, connectors, overview and concepts sections.
There is a lot of work to be done and the technical writer could choose what
areas to focus on — these improvements could then be ad [...]
+
+<p>If you’re interested in learning more about this project idea or want
to get involved in GSoD as a technical writer, check out the <a
href="https://flink.apache.org/news/2020/05/04/season-of-docs.html">announcement
blogpost</a>.</p>
+
+<hr />
+
+<h1 id="and-something-to-read">…and something to
read!</h1>
+
+<p>Events across the globe have pretty much come to a halt, so we’ll
leave you with some interesting resources to read and explore instead. In
addition to this written content, you can also recap the sessions from the
<a
href="https://www.youtube.com/playlist?list=PLDX4T_cnKjD0ngnBSU-bYGfgVv17MiwA7">Flink
Forward Virtual Conference</a>!</p>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th>Type</th>
+ <th>Links</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><span class="glyphicon glyphicon
glyphicon-bookmark" aria-hidden="true"></span>
Blogposts</td>
+ <td><ul>
+ <li><a
href="https://medium.com/@abdelkrim.hadjidj/event-driven-supply-chain-for-crisis-with-flinksql-be80cb3ad4f9">Event-Driven
Supply Chain for Crisis with FlinkSQL and Zeppelin</a></li>
+ </ul>
+ <ul>
+ <li><a
href="https://flink.apache.org/news/2020/04/21/memory-management-improvements-flink-1.10.html">Memory
Management Improvements with Apache Flink 1.10</a></li>
+ <li><a
href="https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html">Flink
Serialization Tuning Vol. 1: Choosing your Serializer — if you
can</a></li>
+ </ul>
+ </td>
+ </tr>
+ <tr>
+ <td><span class="glyphicon glyphicon-console"
aria-hidden="true"></span> Tutorials</td>
+ <td><ul>
+ <li><a
href="https://flink.apache.org/2020/04/09/pyflink-udf-support-flink.html">PyFlink:
Introducing Python Support for UDFs in Flink's Table
API</a></li>
+ <li><a
href="https://dev.to/morsapaes/flink-stateful-functions-where-to-start-2j39">Flink
Stateful Functions: where to start?</a></li>
+ </ul>
+ </td>
+ </tr>
+ <tr>
+ <td><span class="glyphicon glyphicon
glyphicon-certificate" aria-hidden="true"></span> Flink
Packages</td>
+ <td><ul><p><a
href="https://flink-packages.org/">Flink Packages</a> is a
website where you can explore (and contribute to) the Flink <br />
ecosystem of connectors, extensions, APIs, tools and integrations. <b>New
in:</b> </p>
+ <li><a
href="https://flink-packages.org/packages/spillable-state-backend-for-flink">Spillable
State Backend for Flink</a></li>
+ <li><a
href="https://flink-packages.org/packages/flink-memory-calculator">Flink
Memory Calculator</a></li>
+ <li><a
href="https://flink-packages.org/packages/ververica-platform-community-edition">Ververica
Platform Community Edition</a></li>
+ </ul>
+ </td>
+ </tr>
+ </tbody>
+</table>
+
+<p>If you’d like to keep a closer eye on what’s happening in the
community, subscribe to the Flink <a
href="https://flink.apache.org/community.html#mailing-lists">@community
mailing list</a> to get fine-grained weekly updates, upcoming event
announcements and more.</p>
+</description>
+<pubDate>Thu, 07 May 2020 10:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2020/05/07/community-update.html</link>
+<guid isPermaLink="true">/news/2020/05/07/community-update.html</guid>
+</item>
+
+<item>
<title>Applying to Google Season of Docs 2020</title>
<description><p>The Flink community is thrilled to share that the
project is applying again to <a
href="https://developers.google.com/season-of-docs/">Google Season
of Docs</a> (GSoD) this year! If you’re unfamiliar with the program, GSoD
is a great initiative organized by <a
href="https://opensource.google.com/">Google Open Source</a>
to pair technical writers with mentors to work on documentation for open source
projects. The <a href [...]
@@ -28,6 +651,12 @@
<p>If working shoulder to shoulder with the Flink community on
documentation sounds exciting, we’d love to hear from you! You can read more
about our idea for this year’s project below and, depending on whether it is
accepted, <a
href="https://developers.google.com/season-of-docs/docs/tech-writer-guide">apply</a>
as a technical writer. If you have any questions or just want to know more
about the project idea, ping us at <a href="https://flink.apache.o [...]
+<div class="alert alert-info">
+ Please <a
href="mailto:dev-subscr...@flink.apache.org">subscribe</a>
to the Apache Flink mailing list before reaching out.
+ If you are not subscribed then responses to your message will not go
through.
+ You can always <a
href="mailto:dev-unsubscr...@flink.apache.org">unsubscribe</a>
at any time.
+</div>
+
<h2
id="project-improve-the-table-api--sql-documentation">Project:
Improve the Table API &amp; SQL Documentation</h2>
<p><a href="https://flink.apache.org/">Apache
Flink</a> is a stateful stream processor supporting a broad set of use
cases and featuring APIs at different levels of abstraction that allow users to
trade off expressiveness and usability, as well as work with their language of
choice (Java/Scala, SQL or Python). The Table API &amp; SQL are Flink’s
high-level relational abstractions and focus on data analytics use cases. A
core principle is that either API ca [...]
@@ -16543,259 +17172,5 @@ Improve usability of command line interface</p>
<guid isPermaLink="true">/news/2015/04/13/release-0.9.0-milestone1.html</guid>
</item>
-<item>
-<title>March 2015 in the Flink community</title>
-<description><p>March has been a busy month in the Flink
community.</p>
-
-<h3 id="scaling-als">Scaling ALS</h3>
-
-<p>Flink committers employed at <a
href="http://data-artisans.com">data Artisans</a> published
a <a
href="http://data-artisans.com/how-to-factorize-a-700-gb-matrix-with-apache-flink/">blog
post</a> on how they scaled matrix factorization with Flink and Google
Compute Engine to matrices with 28 billion elements.</p>
-
-<h3 id="learn-about-the-internals-of-flink">Learn about the
internals of Flink</h3>
-
-<p>The community has started an effort to better document the internals
-of Flink. Check out the first articles on the Flink wiki on <a
href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=53741525">how
Flink
-manages
-memory</a>,
-<a
href="https://cwiki.apache.org/confluence/display/FLINK/Data+exchange+between+tasks">how
tasks in Flink exchange
-data</a>,
-<a
href="https://cwiki.apache.org/confluence/display/FLINK/Type+System%2C+Type+Extraction%2C+Serialization">type
extraction and serialization in
-Flink</a>,
-as well as <a
href="https://cwiki.apache.org/confluence/display/FLINK/Akka+and+Actors">how
Flink builds on Akka for distributed
-coordination</a>.</p>
-
-<p>Check out also the <a
href="http://flink.apache.org/news/2015/03/13/peeking-into-Apache-Flinks-Engine-Room.html">new
blog
-post</a>
-on how Flink executes joins with several insights into Flink’s
runtime.</p>
-
-<h3 id="meetups-and-talks">Meetups and talks</h3>
-
-<p>Flink’s machine learning efforts were presented at the <a
href="http://www.meetup.com/Machine-Learning-Stockholm/events/221144997/">Machine
-Learning Stockholm meetup
-group</a>. The
-regular Berlin Flink meetup featured a talk on the past, present, and
-future of Flink. The talk is available on
-<a
href="https://www.youtube.com/watch?v=fw2DBE6ZiEQ&amp;feature=youtu.be">youtube</a>.</p>
-
-<h2 id="in-the-flink-master">In the Flink master</h2>
-
-<h3 id="table-api-in-scala-and-java">Table API in Scala and
Java</h3>
-
-<p>The new <a
href="https://github.com/apache/flink/tree/master/flink-libraries/flink-table">Table
-API</a>
-in Flink is now available in both Java and Scala. Check out the
-examples <a
href="https://github.com/apache/flink/blob/master/flink-libraries/flink-table/src/main/java/org/apache/flink/examples/java/JavaTableExample.java">here
(Java)</a> and <a
href="https://github.com/apache/flink/tree/master/flink-libraries/flink-table/src/main/scala/org/apache/flink/examples/scala">here
(Scala)</a>.</p>
-
-<h3 id="additions-to-the-machine-learning-library">Additions
to the Machine Learning library</h3>
-
-<p>Flink’s <a
href="https://github.com/apache/flink/tree/master/flink-libraries/flink-ml">Machine
Learning
-library</a>
-is seeing quite a bit of traction. Recent additions include the <a
href="http://arxiv.org/abs/1409.1458">CoCoA
-algorithm</a> for distributed
-optimization.</p>
-
-<h3
id="exactly-once-delivery-guarantees-for-streaming-jobs">Exactly-once
delivery guarantees for streaming jobs</h3>
-
-<p>Flink streaming jobs now provide exactly once processing guarantees
-when coupled with persistent sources (notably <a
href="http://kafka.apache.org">Apache
-Kafka</a>). Flink periodically checkpoints and
-persists the offsets of the sources and restarts from those
-checkpoints at failure recovery. This functionality is currently
-limited in that it does not yet handle large state and iterative
-programs.</p>
-
-</description>
-<pubDate>Tue, 07 Apr 2015 12:00:00 +0200</pubDate>
-<link>https://flink.apache.org/news/2015/04/07/march-in-flink.html</link>
-<guid isPermaLink="true">/news/2015/04/07/march-in-flink.html</guid>
-</item>
-
-<item>
-<title>Peeking into Apache Flink's Engine Room</title>
-<description><h3 id="join-processing-in-apache-flink">Join
Processing in Apache Flink</h3>
-
-<p>Joins are prevalent operations in many data processing applications.
Most data processing systems feature APIs that make joining data sets very
easy. However, the internal algorithms for join processing are much more
involved – especially if large data sets need to be efficiently handled.
Therefore, join processing serves as a good example to discuss the salient
design points and implementation details of a data processing system.</p>
-
-<p>In this blog post, we cut through Apache Flink’s layered architecture
and take a look at its internals with a focus on how it handles joins.
Specifically, I will</p>
-
-<ul>
- <li>show how easy it is to join data sets using Flink’s fluent
APIs,</li>
- <li>discuss basic distributed join strategies, Flink’s join
implementations, and its memory management,</li>
- <li>talk about Flink’s optimizer that automatically chooses join
strategies,</li>
- <li>show some performance numbers for joining data sets of different
sizes, and finally</li>
- <li>briefly discuss joining of co-located and pre-sorted data
sets.</li>
-</ul>
-
-<p><em>Disclaimer</em>: This blog post is exclusively about
equi-joins. Whenever I say “join” in the following, I actually mean
“equi-join”.</p>
-
-<h3 id="how-do-i-join-with-flink">How do I join with
Flink?</h3>
-
-<p>Flink provides fluent APIs in Java and Scala to write data flow
programs. Flink’s APIs are centered around parallel data collections which are
called data sets. data sets are processed by applying Transformations that
compute new data sets. Flink’s transformations include Map and Reduce as known
from MapReduce <a
href="http://research.google.com/archive/mapreduce.html">[1]</a>
but also operators for joining, co-grouping, and iterative processing. The
docume [...]
-
-<p>Joining two Scala case class data sets is very easy as the following
example shows:</p>
-
-<div class="highlight"><pre><code
class="language-scala"><span class="c1">// define
your data types</span>
-<span class="k">case</span> <span
class="k">class</span> <span
class="nc">PageVisit</span><span
class="o">(</span><span
class="n">url</span><span
class="k">:</span> <span
class="kt">String</span><span
class="o">,</span> <span
class="n">ip</span><span
class="k">:</span> &l [...]
-<span class="k">case</span> <span
class="k">class</span> <span
class="nc">User</span><span
class="o">(</span><span
class="n">id</span><span
class="k">:</span> <span
class="kt">Long</span><span
class="o">,</span> <span
class="n">name</span><span
class="k">:</span> <span [...]
-
-<span class="c1">// get your data from somewhere</span>
-<span class="k">val</span> <span
class="n">visits</span><span
class="k">:</span> <span
class="kt">DataSet</span><span
class="o">[</span><span
class="kt">PageVisit</span><span
class="o">]</span> <span
class="k">=</span> <span
class="o">...</span>
-<span class="k">val</span> <span
class="n">users</span><span
class="k">:</span> <span
class="kt">DataSet</span><span
class="o">[</span><span
class="kt">User</span><span
class="o">]</span> <span
class="k">=</span> <span
class="o">...</span>
-
-<span class="c1">// filter the users data set</span>
-<span class="k">val</span> <span
class="n">germanUsers</span> <span
class="k">=</span> <span
class="n">users</span><span
class="o">.</span><span
class="n">filter</span><span
class="o">((</span><span
class="n">u</span><span
class="o">)</span> <span
class="k">=&gt;</span&g [...]
-<span class="c1">// join data sets</span>
-<span class="k">val</span> <span
class="n">germanVisits</span><span
class="k">:</span> <span
class="kt">DataSet</span><span
class="o">[(</span><span
class="kt">PageVisit</span>, <span
class="kt">User</span><span
class="o">)]</span> <span
class="k">=</span>
- <span class="c1">// equi-join condition
(PageVisit.userId = User.id)</span>
- <span class="n">visits</span><span
class="o">.</span><span
class="n">join</span><span
class="o">(</span><span
class="n">germanUsers</span><span
class="o">).</span><span
class="n">where</span><span
class="o">(</span><span
class="s">&quot;userId&quot;</span><span
class="o&qu [...]
-
-<p>Flink’s APIs also allow to:</p>
-
-<ul>
- <li>apply a user-defined join function to each pair of joined elements
instead returning a <code>($Left, $Right)</code> tuple,</li>
- <li>select fields of pairs of joined Tuple elements (projection),
and</li>
- <li>define composite join keys such as <code>.where(“orderDate”,
“zipCode”).equalTo(“date”, “zip”)</code>.</li>
-</ul>
-
-<p>See the documentation for more details on Flink’s join features <a
href="http://ci.apache.org/projects/flink/flink-docs-release-0.8/dataset_transformations.html#join">[3]</a>.</p>
-
-<h3 id="how-does-flink-join-my-data">How does Flink join my
data?</h3>
-
-<p>Flink uses techniques which are well known from parallel database
systems to efficiently execute parallel joins. A join operator must establish
all pairs of elements from its input data sets for which the join condition
evaluates to true. In a standalone system, the most straight-forward
implementation of a join is the so-called nested-loop join which builds the
full Cartesian product and evaluates the join condition for each pair of
elements. This strategy has quadratic complex [...]
-
-<p>In a distributed system joins are commonly processed in two
steps:</p>
-
-<ol>
- <li>The data of both inputs is distributed across all parallel
instances that participate in the join and</li>
- <li>each parallel instance performs a standard stand-alone join
algorithm on its local partition of the overall data.</li>
-</ol>
-
-<p>The distribution of data across parallel instances must ensure that
each valid join pair can be locally built by exactly one instance. For both
steps, there are multiple valid strategies that can be independently picked and
which are favorable in different situations. In Flink terminology, the first
phase is called Ship Strategy and the second phase Local Strategy. In the
following I will describe Flink’s ship and local strategies to join two data
sets <em>R</em> and [...]
-
-<h4 id="ship-strategies">Ship Strategies</h4>
-<p>Flink features two ship strategies to establish a valid data
partitioning for a join:</p>
-
-<ul>
- <li>the <em>Repartition-Repartition</em> strategy (RR)
and</li>
- <li>the <em>Broadcast-Forward</em> strategy
(BF).</li>
-</ul>
-
-<p>The Repartition-Repartition strategy partitions both inputs, R and S,
on their join key attributes using the same partitioning function. Each
partition is assigned to exactly one parallel join instance and all data of
that partition is sent to its associated instance. This ensures that all
elements that share the same join key are shipped to the same parallel instance
and can be locally joined. The cost of the RR strategy is a full shuffle of
both data sets over the network.</p>
-
-<center>
-<img src="/img/blog/joins-repartition.png"
style="width:90%;margin:15px" />
-</center>
-
-<p>The Broadcast-Forward strategy sends one complete data set (R) to
each parallel instance that holds a partition of the other data set (S), i.e.,
each parallel instance receives the full data set R. Data set S remains local
and is not shipped at all. The cost of the BF strategy depends on the size of R
and the number of parallel instances it is shipped to. The size of S does not
matter because S is not moved. The figure below illustrates how both ship
strategies work.</p>
-
-<center>
-<img src="/img/blog/joins-broadcast.png"
style="width:90%;margin:15px" />
-</center>
-
-<p>The Repartition-Repartition and Broadcast-Forward ship strategies
establish suitable data distributions to execute a distributed join. Depending
on the operations that are applied before the join, one or even both inputs of
a join are already distributed in a suitable way across parallel instances. In
this case, Flink will reuse such distributions and only ship one or no input at
all.</p>
-
-<h4 id="flinks-memory-management">Flink’s Memory
Management</h4>
-<p>Before delving into the details of Flink’s local join algorithms, I
will briefly discuss Flink’s internal memory management. Data processing
algorithms such as joining, grouping, and sorting need to hold portions of
their input data in memory. While such algorithms perform best if there is
enough memory available to hold all data, it is crucial to gracefully handle
situations where the data size exceeds memory. Such situations are especially
tricky in JVM-based systems such as F [...]
-
-<p>Flink handles this challenge by actively managing its memory. When a
worker node (TaskManager) is started, it allocates a fixed portion (70% by
default) of the JVM’s heap memory that is available after initialization as
32KB byte arrays. These byte arrays are distributed as working memory to all
algorithms that need to hold significant portions of data in memory. The
algorithms receive their input data as Java data objects and serialize them
into their working memory.</p>
-
-<p>This design has several nice properties. First, the number of data
objects on the JVM heap is much lower resulting in less garbage collection
pressure. Second, objects on the heap have a certain space overhead and the
binary representation is more compact. Especially data sets of many small
elements benefit from that. Third, an algorithm knows exactly when the input
data exceeds its working memory and can react by writing some of its filled
byte arrays to the worker’s local file [...]
-
-<center>
-<img src="/img/blog/joins-memmgmt.png"
style="width:90%;margin:15px" />
-</center>
-
-<p>This active memory management makes Flink extremely robust for
processing very large data sets on limited memory resources while preserving
all benefits of in-memory processing if data is small enough to fit in-memory.
De/serializing data into and from memory has a certain cost overhead compared
to simply holding all data elements on the JVM’s heap. However, Flink features
efficient custom de/serializers which also allow to perform certain operations
such as comparisons directly [...]
-
-<h4 id="local-strategies">Local Strategies</h4>
-
-<p>After the data has been distributed across all parallel join
instances using either a Repartition-Repartition or Broadcast-Forward ship
strategy, each instance runs a local join algorithm to join the elements of its
local partition. Flink’s runtime features two common join strategies to perform
these local joins:</p>
-
-<ul>
- <li>the <em>Sort-Merge-Join</em> strategy (SM)
and</li>
- <li>the <em>Hybrid-Hash-Join</em> strategy (HH).</li>
-</ul>
-
-<p>The Sort-Merge-Join works by first sorting both input data sets on
their join key attributes (Sort Phase) and merging the sorted data sets as a
second step (Merge Phase). The sort is done in-memory if the local partition of
a data set is small enough. Otherwise, an external merge-sort is done by
collecting data until the working memory is filled, sorting it, writing the
sorted data to the local filesystem, and starting over by filling the working
memory again with more incoming [...]
-
-<center>
-<img src="/img/blog/joins-smj.png"
style="width:90%;margin:15px" />
-</center>
-
-<p>The Hybrid-Hash-Join distinguishes its inputs as build-side and
probe-side input and works in two phases, a build phase followed by a probe
phase. In the build phase, the algorithm reads the build-side input and inserts
all data elements into an in-memory hash table indexed by their join key
attributes. If the hash table outgrows the algorithm’s working memory, parts of
the hash table (ranges of hash indexes) are written to the local filesystem.
The build phase ends after the bu [...]
-
-<center>
-<img src="/img/blog/joins-hhj.png"
style="width:90%;margin:15px" />
-</center>
-
-<h3 id="how-does-flink-choose-join-strategies">How does Flink
choose join strategies?</h3>
-
-<p>Ship and local strategies do not depend on each other and can be
independently chosen. Therefore, Flink can execute a join of two data sets R
and S in nine different ways by combining any of the three ship strategies (RR,
BF with R being broadcasted, BF with S being broadcasted) with any of the three
local strategies (SM, HH with R being build-side, HH with S being build-side).
Each of these strategy combinations results in different execution performance
depending on the data s [...]
-
-<p>Flink features a cost-based optimizer which automatically chooses the
execution strategies for all operators including joins. Without going into the
details of cost-based optimization, this is done by computing cost estimates
for execution plans with different strategies and picking the plan with the
least estimated costs. Thereby, the optimizer estimates the amount of data
which is shipped over the the network and written to disk. If no reliable size
estimates for the input dat [...]
-
-<h3 id="how-is-flinks-join-performance">How is Flink’s join
performance?</h3>
-
-<p>Alright, that sounds good, but how fast are joins in Flink? Let’s
have a look. We start with a benchmark of the single-core performance of
Flink’s Hybrid-Hash-Join implementation and run a Flink program that executes a
Hybrid-Hash-Join with parallelism 1. We run the program on a n1-standard-2
Google Compute Engine instance (2 vCPUs, 7.5GB memory) with two locally
attached SSDs. We give 4GB as working memory to the join. The join program
generates 1KB records for both inputs on-t [...]
-
-<center>
-<img src="/img/blog/joins-single-perf.png"
style="width:85%;margin:15px" />
-</center>
-
-<p>The joins with 1 to 3 GB build side (blue bars) are pure in-memory
joins. The other joins partially spill data to disk (4 to 12GB, orange bars).
The results show that the performance of Flink’s Hybrid-Hash-Join remains
stable as long as the hash table completely fits into memory. As soon as the
hash table becomes larger than the working memory, parts of the hash table and
corresponding parts of the probe side are spilled to disk. The chart shows that
the performance of the Hybri [...]
-
-<p>So, Flink’s Hybrid-Hash-Join implementation performs well on a single
thread even for limited memory resources, but how good is Flink’s performance
when joining larger data sets in a distributed setting? For the next experiment
we compare the performance of the most common join strategy combinations,
namely:</p>
-
-<ul>
- <li>Broadcast-Forward, Hybrid-Hash-Join (broadcasting and building
with the smaller side),</li>
- <li>Repartition, Hybrid-Hash-Join (building with the smaller side),
and</li>
- <li>Repartition, Sort-Merge-Join</li>
-</ul>
-
-<p>for different input size ratios:</p>
-
-<ul>
- <li>1GB : 1000GB</li>
- <li>10GB : 1000GB</li>
- <li>100GB : 1000GB</li>
- <li>1000GB : 1000GB</li>
-</ul>
-
-<p>The Broadcast-Forward strategy is only executed for up to 10GB.
Building a hash table from 100GB broadcasted data in 5GB working memory would
result in spilling proximately 95GB (build input) + 950GB (probe input) in each
parallel thread and require more than 8TB local disk storage on each
machine.</p>
-
-<p>As in the single-core benchmark, we run 1:N joins, generate the data
on-the-fly, and immediately discard the result after the join. We run the
benchmark on 10 n1-highmem-8 Google Compute Engine instances. Each instance is
equipped with 8 cores, 52GB RAM, 40GB of which are configured as working memory
(5GB per core), and one local SSD for spilling to disk. All benchmarks are
performed using the same configuration, i.e., no fine tuning for the respective
data sizes is done. The pr [...]
-
-<center>
-<img src="/img/blog/joins-dist-perf.png"
style="width:70%;margin:15px" />
-</center>
-
-<p>As expected, the Broadcast-Forward strategy performs best for very
small inputs because the large probe side is not shipped over the network and
is locally joined. However, when the size of the broadcasted side grows, two
problems arise. First the amount of data which is shipped increases but also
each parallel instance has to process the full broadcasted data set. The
performance of both Repartitioning strategies behaves similar for growing input
sizes which indicates that thes [...]
-
-<h3
id="ive-got-sooo-much-data-to-join-do-i-really-need-to-ship-it">I’ve
got sooo much data to join, do I really need to ship it?</h3>
-
-<p>We have seen that off-the-shelf distributed joins work really well in
Flink. But what if your data is so huge that you do not want to shuffle it
across your cluster? We recently added some features to Flink for specifying
semantic properties (partitioning and sorting) on input splits and co-located
reading of local input files. With these tools at hand, it is possible to join
pre-partitioned data sets from your local filesystem without sending a single
byte over your cluster’s n [...]
-
-<h3 id="tldr-what-should-i-remember-from-all-of-this">tl;dr:
What should I remember from all of this?</h3>
-
-<ul>
- <li>Flink’s fluent Scala and Java APIs make joins and other data
transformations easy as cake.</li>
- <li>The optimizer does the hard choices for you, but gives you control
in case you know better.</li>
- <li>Flink’s join implementations perform very good in-memory and
gracefully degrade when going to disk.</li>
- <li>Due to Flink’s robust memory management, there is no need for job-
or data-specific memory tuning to avoid a nasty
<code>OutOfMemoryException</code>. It just runs
out-of-the-box.</li>
-</ul>
-
-<h4 id="references">References</h4>
-
-<p>[1] <a href="">“MapReduce: Simplified data processing
on large clusters”</a>, Dean, Ghemawat, 2004 <br />
-[2] <a
href="http://ci.apache.org/projects/flink/flink-docs-release-0.8/dataset_transformations.html">Flink
0.8.1 documentation: Data Transformations</a> <br />
-[3] <a
href="http://ci.apache.org/projects/flink/flink-docs-release-0.8/dataset_transformations.html#join">Flink
0.8.1 documentation: Joins</a> <br />
-[4] <a
href="https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/batch/index.html#semantic-annotations">Flink
1.0 documentation: Semantic annotations</a> <br />
-[5] <a
href="https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/batch/dataset_transformations.html#join-algorithm-hints">Flink
1.0 documentation: Optimizer join hints</a> <br /></p>
-</description>
-<pubDate>Fri, 13 Mar 2015 11:00:00 +0100</pubDate>
-<link>https://flink.apache.org/news/2015/03/13/peeking-into-Apache-Flinks-Engine-Room.html</link>
-<guid
isPermaLink="true">/news/2015/03/13/peeking-into-Apache-Flinks-Engine-Room.html</guid>
-</item>
-
</channel>
</rss>
diff --git a/content/news/2020/05/07/community-update.html
b/content/news/2020/05/07/community-update.html
index 8af9e16..7583d76 100644
--- a/content/news/2020/05/07/community-update.html
+++ b/content/news/2020/05/07/community-update.html
@@ -295,7 +295,7 @@
<div class="col-lg-3">
<div class="text-center">
<img class="img-circle"
src="https://avatars1.githubusercontent.com/u/4971479?s=400&u=49d4f217e26186606ab13a17a23a038b62b86682&v=4";
width="90" height="90" />
- <p><a href="https://twitter.com/HequnC";>Hequn Chen</a></p>
+ <p><a href="https://twitter.com/HequnC";>Hequn Cheng</a></p>
</div>
</div>
<div class="col-lg-3">
