This is an automated email from the ASF dual-hosted git repository.
trohrmann pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit d327b281962c510e3658e720503fda13a18abbfd
Author: Till Rohrmann <trohrm...@apache.org>
AuthorDate: Thu Aug 6 12:29:55 2020 +0200
Rebuild website
---
content/blog/feed.xml | 359 ++++++++-----------
content/blog/index.html | 40 ++-
content/blog/page10/index.html | 38 ++-
content/blog/page11/index.html | 36 +-
content/blog/page12/index.html | 38 ++-
content/blog/page13/index.html | 25 ++
content/blog/page2/index.html | 43 ++-
content/blog/page3/index.html | 39 ++-
content/blog/page4/index.html | 38 ++-
content/blog/page5/index.html | 38 ++-
content/blog/page6/index.html | 38 ++-
content/blog/page7/index.html | 40 ++-
content/blog/page8/index.html | 40 ++-
content/blog/page9/index.html | 40 ++-
.../ai-workflow.png | Bin 0 -> 286466 bytes
content/index.html | 8 +-
content/news/2020/08/06/external-resource.html | 378 +++++++++++++++++++++
content/zh/index.html | 8 +-
18 files changed, 837 insertions(+), 409 deletions(-)
diff --git a/content/blog/feed.xml b/content/blog/feed.xml
index d364dec..85ec175 100644
--- a/content/blog/feed.xml
+++ b/content/blog/feed.xml
@@ -7,6 +7,141 @@
<atom:link href="https://flink.apache.org/blog/feed.xml"; rel="self"
type="application/rss+xml" />
<item>
+<title>Accelerating your workload with GPU and other external resources</title>
+<description><p>Apache Flink 1.11 introduces a new <a
href="https://ci.apache.org/projects/flink/flink-docs-master/ops/external_resources.html">External
Resource Framework</a>,
+which allows you to request external resources from the underlying resource
management systems (e.g., Kubernetes) and accelerate your workload with
+those resources. As Flink provides a first-party GPU plugin at the moment, we
will take GPU as an example and show how it affects Flink applications
+in the AI field. Other external resources (e.g. RDMA and SSD) can also be
supported <a
href="https://ci.apache.org/projects/flink/flink-docs-master/ops/external_resources.html#implement-a-plugin-for-your-custom-resource-type">in
a pluggable manner</a>.</p>
+
+<h1 id="end-to-end-real-time-ai-with-gpu">End-to-end real-time
AI with GPU</h1>
+
+<p>Recently, AI and Machine Learning have gained additional popularity
and have been widely used in various scenarios, such
+as personalized recommendation and image recognition. <a
href="https://flink.apache.org/">Flink</a>, with the ability
to support GPU
+allocation, can be used to build an end-to-end real-time AI workflow.</p>
+
+<h2 id="why-flink">Why Flink</h2>
+
+<p>Typical AI workloads fall into two categories: training and
inference.</p>
+
+<center>
+<img
src="/img/blog/2020-08-06-accelerate-with-external-resources/ai-workflow.png"
width="800px" alt="Typical AI Workflow" />
+<br />
+<i><small>Typical AI Workflow</small></i>
+</center>
+<p><br /></p>
+
+<p>The training workload is usually a batch task, in which we train a
model from a bounded dataset. On the other hand, the inference
+workload tends to be a streaming job. It consumes an unbounded data stream,
which contains image data, for example, and uses a model
+to produce the output of predictions. Both workloads need to do data
preprocessing first. Flink, as a
+<a
href="https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html">unified
batch and stream processing engine</a>, can be used to build an
end-to-end AI workflow naturally.</p>
+
+<p>In many cases, the training and inference workload can benefit a lot
by leveraging GPUs. <a
href="https://azure.microsoft.com/en-us/blog/gpus-vs-cpus-for-deployment-of-deep-learning-models/">Research</a>
+shows that CPU cluster is outperformed by GPU cluster, which is of similar
cost, by about 400 percent. As training datasets
+are getting bigger and models more complex, supporting GPUs has become
mandatory for running AI workloads.</p>
+
+<p>With the <a
href="https://ci.apache.org/projects/flink/flink-docs-master/ops/external_resources.html">External
Resource Framework</a>
+and its <a
href="https://ci.apache.org/projects/flink/flink-docs-master/ops/external_resources.html#plugin-for-gpu-resources">GPU
plugin</a>, Flink
+can now request GPU resources from the external resource management system and
expose GPU information to operators. With this
+feature, users can now easily build end-to-end training and real-time
inference pipelines with GPU support on Flink.</p>
+
+<h2 id="example-mnist-inference-with-flink">Example: MNIST
Inference with Flink</h2>
+
+<p>We take the MNIST inference task as an example to show how to use the
<a
href="https://ci.apache.org/projects/flink/flink-docs-master/ops/external_resources.html">External
Resource Framework</a>
+and how to leverage GPUs in Flink. MNIST is a database of handwritten digits,
which is usually viewed as the HelloWorld of AI.
+The goal is to recognize a 28px*28px picture of a number from 0 to 9.</p>
+
+<p>First, you need to set configurations for the external resource
framework to enable GPU support:</p>
+
+<div class="highlight"><pre><code
class="language-bash">external-resources: gpu
+<span class="c"># Define the driver factory class of gpu
resource.</span>
+external-resource.gpu.driver-factory.class:
org.apache.flink.externalresource.gpu.GPUDriverFactory
+<span class="c"># Define the amount of gpu resource per
TaskManager.</span>
+external-resource.gpu.amount: 1
+<span class="c"># Enable the coordination mode if you run it
in standalone mode</span>
+external-resource.gpu.param.discovery-script.args: --enable-coordination
+
+
+<span class="c"># If you run it on Yarn</span>
+external-resource.gpu.yarn.config-key: yarn.io/gpu
+<span class="c"># If you run it on Kubernetes</span>
+external-resource.gpu.kubernetes.config-key:
nvidia.com/gpu</code></pre></div>
+
+<p>For more details of the configuration, please refer to the <a
href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/external_resources.html#configurations-1">official
documentation</a>.</p>
+
+<p>In the MNIST inference task, we first need to read the images and do
data preprocessing. You can download <a
href="http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz">training</a>
+or <a
href="http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz">testing</a>
data from <a href="http://yann.lecun.com/exdb/mnist/">this
site</a>.
+We provide a simple <a
href="https://github.com/KarmaGYZ/flink-mnist/blob/master/src/main/java/org/apache/flink/MNISTReader.java">MNISTReader</a>.
+It will read the image data located in the provided file path and transform
each image into a list of floating point numbers.</p>
+
+<p>Then, we need a classifier to recognize those images. A one-layer
pre-trained neural network, whose prediction accuracy is 92.14%,
+is used in our classify operator. To leverage GPUs in order to accelerate the
matrix-matrix multiplication, we use <a
href="https://github.com/jcuda/jcuda">JCuda</a>
+to call the native Cuda API. The prediction logic of the <a
href="https://github.com/KarmaGYZ/flink-mnist/blob/master/src/main/java/org/apache/flink/MNISTClassifier.java">MNISTClassifier</a>
is shown below.</p>
+
+<div class="highlight"><pre><code
class="language-java"><span
class="kd">class</span> <span
class="nc">MNISTClassifier</span> <span
class="kd">extends</span> <span
class="n">RichMapFunction</span><span
class="o">&lt;</span><span
class="n">List</span><span
class="o">&lt;</span><span class= [...]
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span
class="kt">void</span> <span
class="nf">open</span><span
class="o">(</span><span
class="n">Configuration</span> <span
class="n">parameters</span><span
class="o">)</span> <span
class="o">{</span>
+ <span class="c1">// Get the GPU information and select
the first GPU.</span>
+ <span class="kd">final</span> <span
class="n">Set</span><span
class="o">&lt;</span><span
class="n">ExternalResourceInfo</span><span
class="o">&gt;</span> <span
class="n">externalResourceInfos</span> <span
class="o">=</span> <span
class="n">getRuntimeContext</span><span
class="o">().& [...]
+ <span class="kd">final</span> <span
class="n">Optional</span><span
class="o">&lt;</span><span
class="n">String</span><span
class="o">&gt;</span> <span
class="n">firstIndexOptional</span> <span
class="o">=</span> <span
class="n">externalResourceInfos</span><span
class="o">.</span&g [...]
+
+ <span class="c1">// Initialize JCublas with the
selected GPU</span>
+ <span class="n">JCuda</span><span
class="o">.</span><span
class="na">cudaSetDevice</span><span
class="o">(</span><span
class="n">Integer</span><span
class="o">.</span><span
class="na">parseInt</span><span
class="o">(</span><span
class="n">firstIndexOptional</span><span class="o
[...]
+ <span class="n">JCublas</span><span
class="o">.</span><span
class="na">cublasInit</span><span
class="o">();</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span
class="n">Integer</span> <span
class="nf">map</span><span
class="o">(</span><span
class="n">List</span><span
class="o">&lt;</span><span
class="n">Float</span><span
class="o">&gt;</span> <span
class="n">value</span><span class="o"> [...]
+ <span class="c1">// Performs multiplication using
JCublas. The matrixPointer points to our pre-trained model.</span>
+ <span class="n">JCublas</span><span
class="o">.</span><span
class="na">cublasSgemv</span><span
class="o">(</span><span
class="sc">&#39;n&#39;</span><span
class="o">,</span> <span
class="n">DIMENSIONS</span><span
class="o">.</span><span
class="na">f1</span><span class="o& [...]
+ <span
class="n">matrixPointer</span><span
class="o">,</span> <span
class="n">DIMENSIONS</span><span
class="o">.</span><span
class="na">f1</span><span
class="o">,</span> <span
class="n">inputPointer</span><span
class="o">,</span> <span
class="mi">1</span><span class="o&qu [...]
+
+ <span class="c1">// Read the result back from
GPU.</span>
+ <span class="n">JCublas</span><span
class="o">.</span><span
class="na">cublasGetVector</span><span
class="o">(</span><span
class="n">DIMENSIONS</span><span
class="o">.</span><span
class="na">f1</span><span
class="o">,</span> <span
class="n">Sizeof</span><span class="o">.
[...]
+ <span class="kt">int</span> <span
class="n">result</span> <span
class="o">=</span> <span
class="mi">0</span><span
class="o">;</span>
+ <span class="k">for</span> <span
class="o">(</span><span
class="kt">int</span> <span
class="n">i</span> <span
class="o">=</span> <span
class="mi">0</span><span
class="o">;</span> <span
class="n">i</span> <span
class="o">&lt;</span> <span
class="n">DIMENSIONS</span [...]
+ <span class="n">result</span> <span
class="o">=</span> <span
class="n">output</span><span
class="o">[</span><span
class="n">i</span><span
class="o">]</span> <span
class="o">&gt;</span> <span
class="n">output</span><span
class="o">[</span><span class="n">result<
[...]
+ <span class="o">}</span>
+ <span class="k">return</span> <span
class="n">result</span><span
class="o">;</span>
+ <span class="o">}</span>
+<span
class="o">}</span></code></pre></div>
+
+<p>The complete MNIST inference project can be found <a
href="https://github.com/KarmaGYZ/flink-mnist">here</a>. In
this project, we simply
+print the inference result to <strong>STDOUT</strong>. In the
actual production environment, you could also write the result to Elasticsearch
or Kafka, for example.</p>
+
+<p>The MNIST inference task is just a simple case that shows you how the
external resource framework works and what Flink can
+do with GPU support. With Flink’s open source extension <a
href="https://github.com/alibaba/Alink">Alink</a>, which
contains a lot of
+pre-built algorithms based on Flink, and <a
href="https://github.com/alibaba/flink-ai-extended">Tensorflow on
Flink</a>, some complex
+AI workloads, e.g. online learning, real-time inference service, could be
easily implemented as well.</p>
+
+<h1 id="other-external-resources">Other external
resources</h1>
+
+<p>In addition to GPU support, there are many other external resources
that can be used to accelerate jobs in some specific scenarios.
+E.g. FPGA, for AI workloads, is supported by both Yarn and Kubernetes. Some
low-latency network devices, like RDMA and Solarflare, also
+provide their device plugin for Kubernetes. Currently, Yarn supports GPUs and
FPGAs, while the list of Kubernetes’ device plugins can be found <a
href="https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#examples">here</a>.</p>
+
+<p>With the external resource framework, you only need to implement a
plugin that enables the operator to get the information
+for these external resources; see <a
href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/external_resources.html#implement-a-plugin-for-your-custom-resource-type">Custom
Plugin</a>
+for more details. If you just want to ensure that an external resource exists
in the TaskManager, then you only need to find the
+configuration key of that resource in the underlying resource management
system and configure the external resource framework accordingly.</p>
+
+<h1 id="conclusion">Conclusion</h1>
+
+<p>In the latest Flink release (Flink 1.11), an external resource
framework has been introduced to support requesting various types of
+resources from the underlying resource management systems, and supply all the
necessary information for using these resources to the
+operators. The first-party GPU plugin expands the application prospects of
Flink in the AI domain. Different resource types can be supported
+in a pluggable way. You can also implement your own plugins for custom
resource types.</p>
+
+<p>Future developments in this area include implementing operator level
resource isolation and fine-grained external resource scheduling.
+The community may kick this work off once <a
href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation">FLIP-56</a>
+is finished. If you have any suggestions or questions for the community, we
encourage you to sign up to the Apache Flink
+<a
href="https://flink.apache.org/community.html#mailing-lists">mailing
lists</a> and join the discussion there.</p>
+</description>
+<pubDate>Thu, 06 Aug 2020 10:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2020/08/06/external-resource.html</link>
+<guid isPermaLink="true">/news/2020/08/06/external-resource.html</guid>
+</item>
+
+<item>
<title>PyFlink: The integration of Pandas into PyFlink</title>
<description><p>Python has evolved into one of the most important
programming languages for many fields of data processing. So big has been
Python’s popularity, that it has pretty much become the default data processing
language for data scientists. On top of that, there is a plethora of
Python-based data processing tools such as NumPy, Pandas, and Scikit-learn that
have gained additional popularity due to their flexibility or powerful
functionalities.</p>
@@ -17057,229 +17192,5 @@ When using this backend, active state in streaming
programs can grow well beyond
<guid isPermaLink="true">/news/2016/02/11/release-0.10.2.html</guid>
</item>
-<item>
-<title>Flink 2015: A year in review, and a lookout to 2016</title>
-<description><p>With 2015 ending, we thought that this would be good
time to reflect
-on the amazing work done by the Flink community over this past year,
-and how much this community has grown.</p>
-
-<p>Overall, we have seen Flink grow in terms of functionality from an
-engine to one of the most complete open-source stream processing
-frameworks available. The community grew from a relatively small and
-geographically focused team, to a truly global, and one of the largest
-big data communities in the the Apache Software Foundation.</p>
-
-<p>We will also look at some interesting stats, including that the
-busiest days for Flink are Mondays (who would have thought :-).</p>
-
-<h1 id="community-growth">Community growth</h1>
-
-<p>Let us start with some simple statistics from <a
href="https://github.com/apache/flink">Flink’s
-github repository</a>. During 2015, the
-Flink community <strong>doubled</strong> in size, from about 75
contributors to
-over 150. Forks of the repository more than
<strong>tripled</strong> from 160 in
-February 2015 to 544 in December 2015, and the number of stars of the
-repository almost tripled from 289 to 813.</p>
-
-<center>
-<img src="/img/blog/community-growth.png"
style="height:400px;margin:15px" />
-</center>
-
-<p>Although Flink started out geographically in Berlin, Germany, the
-community is by now spread all around the globe, with many
-contributors from North America, Europe, and Asia. A simple search at
-meetup.com for groups that mention Flink as a focus area reveals <a
href="http://apache-flink.meetup.com/">16
-meetups around the globe</a>:</p>
-
-<center>
-<img src="/img/blog/meetup-map.png"
style="height:400px;margin:15px" />
-</center>
-
-<h1 id="flink-forward-2015">Flink Forward 2015</h1>
-
-<p>One of the highlights of the year for Flink was undoubtedly the <a
href="http://2015.flink-forward.org/">Flink
-Forward</a> conference, the first conference
-on Apache Flink that was held in October in Berlin. More than 250
-participants (roughly half based outside Germany where the conference
-was held) attended more than 33 technical talks from organizations
-including Google, MongoDB, Bouygues Telecom, NFLabs, Euranova, RedHat,
-IBM, Huawei, Intel, Ericsson, Capital One, Zalando, Amadeus, the Otto
-Group, and ResearchGate. If you have not yet watched their talks,
-check out the <a
href="http://2015.flink-forward.org/?post_type=day">slides</a>
and
-<a
href="https://www.youtube.com/playlist?list=PLDX4T_cnKjD31JeWR1aMOi9LXPRQ6nyHO">videos</a>
-from Flink Forward.</p>
-
-<center>
-<img src="/img/blog/ff-speakers.png"
style="height:400px;margin:15px" />
-</center>
-
-<h1 id="media-coverage">Media coverage</h1>
-
-<p>And of course, interest in Flink was picked up by the tech
-media. During 2015, articles about Flink appeared in
-<a
href="http://www.infoq.com/Apache-Flink/news/">InfoQ</a>,
-<a
href="http://www.zdnet.com/article/five-open-source-big-data-projects-to-watch/">ZDNet</a>,
-<a
href="http://www.datanami.com/tag/apache-flink/">Datanami</a>,
-<a
href="http://www.infoworld.com/article/2919602/hadoop/flink-hadoops-new-contender-for-mapreduce-spark.html">Infoworld</a>
-(including being one of the <a
href="http://www.infoworld.com/article/2982429/open-source-tools/bossie-awards-2015-the-best-open-source-big-data-tools.html">best
open source big data tools of
-2015</a>),
-the <a
href="http://blogs.gartner.com/nick-heudecker/apache-flink-offers-a-challenge-to-spark/">Gartner
-blog</a>,
-<a
href="http://dataconomy.com/tag/apache-flink/">Dataconomy</a>,
-<a
href="http://sdtimes.com/tag/apache-flink/">SDTimes</a>, the
<a
href="https://www.mapr.com/blog/apache-flink-new-way-handle-streaming-data">MapR
-blog</a>,
-<a
href="http://www.kdnuggets.com/2015/08/apache-flink-stream-processing.html">KDnuggets</a>,
-and
-<a
href="http://www.hadoopsphere.com/2015/02/distributed-data-processing-with-apache.html">HadoopSphere</a>.</p>
-
-<center>
-<img src="/img/blog/appeared-in.png"
style="height:400px;margin:15px" />
-</center>
-
-<p>It is interesting to see that Hadoop Summit EMEA 2016 had a whopping
-number of 17 (!) talks submitted that are mentioning Flink in their
-title and abstract:</p>
-
-<center>
-<img src="/img/blog/hadoop-summit.png"
style="height:400px;margin:15px" />
-</center>
-
-<h1 id="fun-with-stats-when-do-committers-commit">Fun with
stats: when do committers commit?</h1>
-
-<p>To get some deeper insight on what is happening in the Flink
-community, let us do some analytics on the git log of the project :-)
-The easiest thing we can do is count the number of commits at the
-repository in 2015. Running</p>
-
-<div class="highlight"><pre><code>git log
--pretty=oneline --after=1/1/2015 | wc -l
-</code></pre></div>
-
-<p>on the Flink repository yields a total of <strong>2203
commits</strong> in 2015.</p>
-
-<p>To dig deeper, we will use an open source tool called gitstats that
-will give us some interesting statistics on the committer
-behavior. You can create these also yourself and see many more by
-following four easy steps:</p>
-
-<ol>
- <li>Download gitstats from the <a
href="http://gitstats.sourceforge.net/">project
homepage</a>.. E.g., on OS X with homebrew, type</li>
-</ol>
-
-<div class="highlight"><pre><code>brew install
--HEAD homebrew/head-only/gitstats
-</code></pre></div>
-
-<ol>
- <li>Clone the Apache Flink git repository:</li>
-</ol>
-
-<div class="highlight"><pre><code>git clone
g...@github.com:apache/flink.git
-</code></pre></div>
-
-<ol>
- <li>Generate the statistics</li>
-</ol>
-
-<div class="highlight"><pre><code>gitstats flink/
flink-stats/
-</code></pre></div>
-
-<ol>
- <li>View all the statistics as an html page using your favorite
browser (e.g., chrome):</li>
-</ol>
-
-<div class="highlight"><pre><code>chrome
flink-stats/index.html
-</code></pre></div>
-
-<p>First, we can see a steady growth of lines of code in Flink since the
-initial Apache incubator project. During 2015, the codebase almost
-<strong>doubled</strong> from 500,000 LOC to 900,000 LOC.</p>
-
-<center>
-<img src="/img/blog/code-growth.png"
style="height:400px;margin:15px" />
-</center>
-
-<p>It is interesting to see when committers commit. For Flink, Monday
-afternoons are by far the most popular times to commit to the
-repository:</p>
-
-<center>
-<img src="/img/blog/commit-stats.png"
style="height:400px;margin:15px" />
-</center>
-
-<h1 id="feature-timeline">Feature timeline</h1>
-
-<p>So, what were the major features added to Flink and the Flink
-ecosystem during 2015? Here is a (non-exhaustive) chronological list:</p>
-
-<center>
-<img src="/img/blog/feature-timeline.png"
style="height:400px;margin:15px" />
-</center>
-
-<h1 id="roadmap-for-2016">Roadmap for 2016</h1>
-
-<p>With 2015 coming to a close, the Flink community has already started
-discussing Flink’s roadmap for the future. Some highlights
-are:</p>
-
-<ul>
- <li>
- <p><strong>Runtime scaling of streaming jobs:</strong>
streaming jobs are running
- forever, and need to react to a changing environment. Runtime
- scaling means dynamically increasing and decreasing the
- parallelism of a job to sustain certain SLAs, or react to changing
- input throughput.</p>
- </li>
- <li>
- <p><strong>SQL queries for static data sets and
streams:</strong> building on top of
- Flink’s Table API, users should be able to write SQL
- queries for static data sets, as well as SQL queries on data
- streams that continuously produce new results.</p>
- </li>
- <li>
- <p><strong>Streaming operators backed by managed
memory:</strong> currently,
- streaming operators like user-defined state and windows are backed
- by JVM heap objects. Moving those to Flink managed memory will add
- the ability to spill to disk, GC efficiency, as well as better
- control over memory utilization.</p>
- </li>
- <li>
- <p><strong>Library for detecting temporal event
patterns:</strong> a common use case
- for stream processing is detecting patterns in an event stream
- with timestamps. Flink makes this possible with its support for
- event time, so many of these operators can be surfaced in the form
- of a library.</p>
- </li>
- <li>
- <p><strong>Support for Apache Mesos, and resource-dynamic YARN
support:</strong>
- support for both Mesos and YARN, including dynamic allocation and
- release of resource for more resource elasticity (for both batch
- and stream processing).</p>
- </li>
- <li>
- <p><strong>Security:</strong> encrypt both the messages
exchanged between
- TaskManagers and JobManager, as well as the connections for data
- exchange between workers.</p>
- </li>
- <li>
- <p><strong>More streaming connectors, more runtime metrics,
and continuous
- DataStream API enhancements:</strong> add support for more sources and
- sinks (e.g., Amazon Kinesis, Cassandra, Flume, etc), expose more
- metrics to the user, and provide continuous improvements to the
- DataStream API.</p>
- </li>
-</ul>
-
-<p>If you are interested in these features, we highly encourage you to
-take a look at the <a
href="https://docs.google.com/document/d/1ExmtVpeVVT3TIhO1JoBpC5JKXm-778DAD7eqw5GANwE/edit">current
-draft</a>,
-and <a
href="https://mail-archives.apache.org/mod_mbox/flink-dev/201512.mbox/browser">join
the
-discussion</a>
-on the Flink mailing lists.</p>
-
-</description>
-<pubDate>Fri, 18 Dec 2015 11:00:00 +0100</pubDate>
-<link>https://flink.apache.org/news/2015/12/18/a-year-in-review.html</link>
-<guid isPermaLink="true">/news/2015/12/18/a-year-in-review.html</guid>
-</item>
-
</channel>
</rss>
diff --git a/content/blog/index.html b/content/blog/index.html
index 6b9e0c5..222f582 100644
--- a/content/blog/index.html
+++ b/content/blog/index.html
@@ -196,6 +196,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a
href="/news/2020/08/06/external-resource.html">Accelerating your workload with
GPU and other external resources</a></h2>
+
+ <p>06 Aug 2020
+ Yangze Guo </p>
+
+ <p>This post introduces the new External Resource Framework in Flink
1.11 and take GPU as an example to show how to accelerate your workload with
external resources.</p>
+
+ <p><a href="/news/2020/08/06/external-resource.html">Continue reading
»</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a
href="/2020/08/04/pyflink-pandas-udf-support-flink.html">PyFlink: The
integration of Pandas into PyFlink</a></h2>
<p>04 Aug 2020
@@ -326,23 +339,6 @@ perform streaming data visualization via Flink on Zeppelin
and how to use Apache
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2020/06/15/flink-on-zeppelin-part1.html">Flink on Zeppelin
Notebooks for Interactive Data Analysis - Part 1</a></h2>
-
- <p>15 Jun 2020
- Jeff Zhang (<a href="https://twitter.com/zjffdu";>@zjffdu</a>)</p>
-
- <p><p>The latest release of <a
href="https://zeppelin.apache.org/";>Apache Zeppelin</a> comes with a redesigned
interpreter for Apache Flink (version Flink 1.10+ is only supported moving
forward)
-that allows developers to use Flink directly on Zeppelin notebooks for
interactive data analysis. I wrote 2 posts about how to use Flink in Zeppelin.
This is part-1 where I explain how the Flink interpreter in Zeppelin works,
-and provide a tutorial for running Streaming ETL with Flink on Zeppelin.</p>
-
-</p>
-
- <p><a href="/news/2020/06/15/flink-on-zeppelin-part1.html">Continue
reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -375,6 +371,16 @@ and provide a tutorial for running Streaming ETL with
Flink on Zeppelin.</p>
<ul id="markdown-toc">
+ <li><a href="/news/2020/08/06/external-resource.html">Accelerating your
workload with GPU and other external resources</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2020/08/04/pyflink-pandas-udf-support-flink.html">PyFlink:
The integration of Pandas into PyFlink</a></li>
diff --git a/content/blog/page10/index.html b/content/blog/page10/index.html
index e5d166e..f5127c9 100644
--- a/content/blog/page10/index.html
+++ b/content/blog/page10/index.html
@@ -196,6 +196,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a
href="/news/2016/08/11/release-1.1.1.html">Flink 1.1.1 Released</a></h2>
+
+ <p>11 Aug 2016
+ </p>
+
+ <p><p>Today, the Flink community released Flink version 1.1.1.</p>
+
+</p>
+
+ <p><a href="/news/2016/08/11/release-1.1.1.html">Continue reading
»</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a
href="/news/2016/08/08/release-1.1.0.html">Announcing Apache Flink
1.1.0</a></h2>
<p>08 Aug 2016
@@ -327,19 +342,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2015/12/18/a-year-in-review.html">Flink 2015: A year in review, and
a lookout to 2016</a></h2>
-
- <p>18 Dec 2015 by Robert Metzger (<a
href="https://twitter.com/";>@rmetzger_</a>)
- </p>
-
- <p><p>With 2015 ending, we thought that this would be good time to
reflect on the amazing work done by the Flink community over this past year,
and how much this community has grown.</p></p>
-
- <p><a href="/news/2015/12/18/a-year-in-review.html">Continue reading
»</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -372,6 +374,16 @@
<ul id="markdown-toc">
+ <li><a href="/news/2020/08/06/external-resource.html">Accelerating your
workload with GPU and other external resources</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2020/08/04/pyflink-pandas-udf-support-flink.html">PyFlink:
The integration of Pandas into PyFlink</a></li>
diff --git a/content/blog/page11/index.html b/content/blog/page11/index.html
index cee4d02..70b8c8f 100644
--- a/content/blog/page11/index.html
+++ b/content/blog/page11/index.html
@@ -196,6 +196,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a
href="/news/2015/12/18/a-year-in-review.html">Flink 2015: A year in review, and
a lookout to 2016</a></h2>
+
+ <p>18 Dec 2015 by Robert Metzger (<a
href="https://twitter.com/";>@rmetzger_</a>)
+ </p>
+
+ <p><p>With 2015 ending, we thought that this would be good time to
reflect on the amazing work done by the Flink community over this past year,
and how much this community has grown.</p></p>
+
+ <p><a href="/news/2015/12/18/a-year-in-review.html">Continue reading
»</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a
href="/news/2015/12/11/storm-compatibility.html">Storm Compatibility in Apache
Flink: How to run existing Storm topologies on Flink</a></h2>
<p>11 Dec 2015 by Matthias J. Sax (<a
href="https://twitter.com/";>@MatthiasJSax</a>)
@@ -333,19 +346,6 @@ vertex-centric or gather-sum-apply to Flink dataflows.</p>
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2015/05/14/Community-update-April.html">April 2015 in the Flink
community</a></h2>
-
- <p>14 May 2015 by Kostas Tzoumas (<a
href="https://twitter.com/";>@kostas_tzoumas</a>)
- </p>
-
- <p><p>The monthly update from the Flink community. Including the
availability of a new preview release, lots of meetups and conference talks and
a great interview about Flink.</p></p>
-
- <p><a href="/news/2015/05/14/Community-update-April.html">Continue
reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -378,6 +378,16 @@ vertex-centric or gather-sum-apply to Flink dataflows.</p>
<ul id="markdown-toc">
+ <li><a href="/news/2020/08/06/external-resource.html">Accelerating your
workload with GPU and other external resources</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2020/08/04/pyflink-pandas-udf-support-flink.html">PyFlink:
The integration of Pandas into PyFlink</a></li>
diff --git a/content/blog/page12/index.html b/content/blog/page12/index.html
index 2863003..9c38ad9 100644
--- a/content/blog/page12/index.html
+++ b/content/blog/page12/index.html
@@ -196,6 +196,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a
href="/news/2015/05/14/Community-update-April.html">April 2015 in the Flink
community</a></h2>
+
+ <p>14 May 2015 by Kostas Tzoumas (<a
href="https://twitter.com/";>@kostas_tzoumas</a>)
+ </p>
+
+ <p><p>The monthly update from the Flink community. Including the
availability of a new preview release, lots of meetups and conference talks and
a great interview about Flink.</p></p>
+
+ <p><a href="/news/2015/05/14/Community-update-April.html">Continue
reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a
href="/news/2015/05/11/Juggling-with-Bits-and-Bytes.html">Juggling with Bits
and Bytes</a></h2>
<p>11 May 2015 by Fabian Hüske (<a
href="https://twitter.com/";>@fhueske</a>)
@@ -339,21 +352,6 @@ and offers a new API including definition of flexible
windows.</p>
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2014/11/18/hadoop-compatibility.html">Hadoop Compatibility in
Flink</a></h2>
-
- <p>18 Nov 2014 by Fabian Hüske (<a
href="https://twitter.com/";>@fhueske</a>)
- </p>
-
- <p><p><a href="http://hadoop.apache.org";>Apache Hadoop</a> is an
industry standard for scalable analytical data processing. Many data analysis
applications have been implemented as Hadoop MapReduce jobs and run in clusters
around the world. Apache Flink can be an alternative to MapReduce and improves
it in many dimensions. Among other features, Flink provides much better
performance and offers APIs in Java and Scala, which are very easy to use.
Similar to Hadoop, Flink’s APIs provi [...]
-
-</p>
-
- <p><a href="/news/2014/11/18/hadoop-compatibility.html">Continue reading
»</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -386,6 +384,16 @@ and offers a new API including definition of flexible
windows.</p>
<ul id="markdown-toc">
+ <li><a href="/news/2020/08/06/external-resource.html">Accelerating your
workload with GPU and other external resources</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2020/08/04/pyflink-pandas-udf-support-flink.html">PyFlink:
The integration of Pandas into PyFlink</a></li>
diff --git a/content/blog/page13/index.html b/content/blog/page13/index.html
index b495a42..1e6a993 100644
--- a/content/blog/page13/index.html
+++ b/content/blog/page13/index.html
@@ -196,6 +196,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a
href="/news/2014/11/18/hadoop-compatibility.html">Hadoop Compatibility in
Flink</a></h2>
+
+ <p>18 Nov 2014 by Fabian Hüske (<a
href="https://twitter.com/";>@fhueske</a>)
+ </p>
+
+ <p><p><a href="http://hadoop.apache.org";>Apache Hadoop</a> is an
industry standard for scalable analytical data processing. Many data analysis
applications have been implemented as Hadoop MapReduce jobs and run in clusters
around the world. Apache Flink can be an alternative to MapReduce and improves
it in many dimensions. Among other features, Flink provides much better
performance and offers APIs in Java and Scala, which are very easy to use.
Similar to Hadoop, Flink’s APIs provi [...]
+
+</p>
+
+ <p><a href="/news/2014/11/18/hadoop-compatibility.html">Continue reading
»</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a
href="/news/2014/11/04/release-0.7.0.html">Apache Flink 0.7.0 available</a></h2>
<p>04 Nov 2014
@@ -290,6 +305,16 @@ academic and open source project that Flink originates
from.</p>
<ul id="markdown-toc">
+ <li><a href="/news/2020/08/06/external-resource.html">Accelerating your
workload with GPU and other external resources</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2020/08/04/pyflink-pandas-udf-support-flink.html">PyFlink:
The integration of Pandas into PyFlink</a></li>
diff --git a/content/blog/page2/index.html b/content/blog/page2/index.html
index c2dc69a..09bbc9f 100644
--- a/content/blog/page2/index.html
+++ b/content/blog/page2/index.html
@@ -196,6 +196,23 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a
href="/news/2020/06/15/flink-on-zeppelin-part1.html">Flink on Zeppelin
Notebooks for Interactive Data Analysis - Part 1</a></h2>
+
+ <p>15 Jun 2020
+ Jeff Zhang (<a href="https://twitter.com/zjffdu";>@zjffdu</a>)</p>
+
+ <p><p>The latest release of <a
href="https://zeppelin.apache.org/";>Apache Zeppelin</a> comes with a redesigned
interpreter for Apache Flink (version Flink 1.10+ is only supported moving
forward)
+that allows developers to use Flink directly on Zeppelin notebooks for
interactive data analysis. I wrote 2 posts about how to use Flink in Zeppelin.
This is part-1 where I explain how the Flink interpreter in Zeppelin works,
+and provide a tutorial for running Streaming ETL with Flink on Zeppelin.</p>
+
+</p>
+
+ <p><a href="/news/2020/06/15/flink-on-zeppelin-part1.html">Continue
reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a
href="/news/2020/06/11/community-update.html">Flink Community Update -
June'20</a></h2>
<p>11 Jun 2020
@@ -318,22 +335,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2020/04/07/release-statefun-2.0.0.html">Stateful Functions 2.0 - An
Event-driven Database on Apache Flink</a></h2>
-
- <p>07 Apr 2020
- Stephan Ewen (<a
href="https://twitter.com/stephanewen";>@stephanewen</a>)</p>
-
- <p><p>Today, we are announcing the release of Stateful Functions
(StateFun) 2.0 — the first release of Stateful Functions as part of the Apache
Flink project.
-This release marks a big milestone: Stateful Functions 2.0 is not only an API
update, but the <strong>first version of an event-driven database</strong> that
is built on Apache Flink.</p>
-
-</p>
-
- <p><a href="/news/2020/04/07/release-statefun-2.0.0.html">Continue
reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -366,6 +367,16 @@ This release marks a big milestone: Stateful Functions 2.0
is not only an API up
<ul id="markdown-toc">
+ <li><a href="/news/2020/08/06/external-resource.html">Accelerating your
workload with GPU and other external resources</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2020/08/04/pyflink-pandas-udf-support-flink.html">PyFlink:
The integration of Pandas into PyFlink</a></li>
diff --git a/content/blog/page3/index.html b/content/blog/page3/index.html
index a1c4aed..57b70d9 100644
--- a/content/blog/page3/index.html
+++ b/content/blog/page3/index.html
@@ -196,6 +196,22 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a
href="/news/2020/04/07/release-statefun-2.0.0.html">Stateful Functions 2.0 - An
Event-driven Database on Apache Flink</a></h2>
+
+ <p>07 Apr 2020
+ Stephan Ewen (<a
href="https://twitter.com/stephanewen";>@stephanewen</a>)</p>
+
+ <p><p>Today, we are announcing the release of Stateful Functions
(StateFun) 2.0 — the first release of Stateful Functions as part of the Apache
Flink project.
+This release marks a big milestone: Stateful Functions 2.0 is not only an API
update, but the <strong>first version of an event-driven database</strong> that
is built on Apache Flink.</p>
+
+</p>
+
+ <p><a href="/news/2020/04/07/release-statefun-2.0.0.html">Continue
reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a
href="/news/2020/04/01/community-update.html">Flink Community Update -
April'20</a></h2>
<p>01 Apr 2020
@@ -318,19 +334,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2020/01/15/demo-fraud-detection.html">Advanced Flink Application
Patterns Vol.1: Case Study of a Fraud Detection System</a></h2>
-
- <p>15 Jan 2020
- Alexander Fedulov (<a
href="https://twitter.com/alex_fedulov";>@alex_fedulov</a>)</p>
-
- <p>In this series of blog posts you will learn about three powerful
Flink patterns for building streaming applications.</p>
-
- <p><a href="/news/2020/01/15/demo-fraud-detection.html">Continue reading
»</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -363,6 +366,16 @@
<ul id="markdown-toc">
+ <li><a href="/news/2020/08/06/external-resource.html">Accelerating your
workload with GPU and other external resources</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2020/08/04/pyflink-pandas-udf-support-flink.html">PyFlink:
The integration of Pandas into PyFlink</a></li>
diff --git a/content/blog/page4/index.html b/content/blog/page4/index.html
index dc3592f..a3fbf0e 100644
--- a/content/blog/page4/index.html
+++ b/content/blog/page4/index.html
@@ -196,6 +196,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a
href="/news/2020/01/15/demo-fraud-detection.html">Advanced Flink Application
Patterns Vol.1: Case Study of a Fraud Detection System</a></h2>
+
+ <p>15 Jan 2020
+ Alexander Fedulov (<a
href="https://twitter.com/alex_fedulov";>@alex_fedulov</a>)</p>
+
+ <p>In this series of blog posts you will learn about three powerful
Flink patterns for building streaming applications.</p>
+
+ <p><a href="/news/2020/01/15/demo-fraud-detection.html">Continue reading
»</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a
href="/news/2019/12/11/release-1.8.3.html">Apache Flink 1.8.3 Released</a></h2>
<p>11 Dec 2019
@@ -321,21 +334,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2019/07/02/release-1.8.1.html">Apache Flink 1.8.1 Released</a></h2>
-
- <p>02 Jul 2019
- Jincheng Sun (<a
href="https://twitter.com/sunjincheng121";>@sunjincheng121</a>)</p>
-
- <p><p>The Apache Flink community released the first bugfix version of
the Apache Flink 1.8 series.</p>
-
-</p>
-
- <p><a href="/news/2019/07/02/release-1.8.1.html">Continue reading
»</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -368,6 +366,16 @@
<ul id="markdown-toc">
+ <li><a href="/news/2020/08/06/external-resource.html">Accelerating your
workload with GPU and other external resources</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2020/08/04/pyflink-pandas-udf-support-flink.html">PyFlink:
The integration of Pandas into PyFlink</a></li>
diff --git a/content/blog/page5/index.html b/content/blog/page5/index.html
index 04ad03f..61cc1a9 100644
--- a/content/blog/page5/index.html
+++ b/content/blog/page5/index.html
@@ -196,6 +196,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a
href="/news/2019/07/02/release-1.8.1.html">Apache Flink 1.8.1 Released</a></h2>
+
+ <p>02 Jul 2019
+ Jincheng Sun (<a
href="https://twitter.com/sunjincheng121";>@sunjincheng121</a>)</p>
+
+ <p><p>The Apache Flink community released the first bugfix version of
the Apache Flink 1.8 series.</p>
+
+</p>
+
+ <p><a href="/news/2019/07/02/release-1.8.1.html">Continue reading
»</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/2019/06/26/broadcast-state.html">A
Practical Guide to Broadcast State in Apache Flink</a></h2>
<p>26 Jun 2019
@@ -320,19 +335,6 @@ for more details.</p>
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2019/02/25/monitoring-best-practices.html">Monitoring Apache Flink
Applications 101</a></h2>
-
- <p>25 Feb 2019
- Konstantin Knauf (<a
href="https://twitter.com/snntrable";>@snntrable</a>)</p>
-
- <p>The monitoring of business-critical applications is a crucial aspect
of a production deployment. It ensures that any degradation or downtime is
immediately identified and can be resolved as quickly as possible. In this
post, we discuss the most important metrics that indicate healthy Flink
applications.</p>
-
- <p><a href="/news/2019/02/25/monitoring-best-practices.html">Continue
reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -365,6 +367,16 @@ for more details.</p>
<ul id="markdown-toc">
+ <li><a href="/news/2020/08/06/external-resource.html">Accelerating your
workload with GPU and other external resources</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2020/08/04/pyflink-pandas-udf-support-flink.html">PyFlink:
The integration of Pandas into PyFlink</a></li>
diff --git a/content/blog/page6/index.html b/content/blog/page6/index.html
index 5b7a4a9..855fbcd 100644
--- a/content/blog/page6/index.html
+++ b/content/blog/page6/index.html
@@ -196,6 +196,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a
href="/news/2019/02/25/monitoring-best-practices.html">Monitoring Apache Flink
Applications 101</a></h2>
+
+ <p>25 Feb 2019
+ Konstantin Knauf (<a
href="https://twitter.com/snntrable";>@snntrable</a>)</p>
+
+ <p>The monitoring of business-critical applications is a crucial aspect
of a production deployment. It ensures that any degradation or downtime is
immediately identified and can be resolved as quickly as possible. In this
post, we discuss the most important metrics that indicate healthy Flink
applications.</p>
+
+ <p><a href="/news/2019/02/25/monitoring-best-practices.html">Continue
reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a
href="/news/2019/02/25/release-1.6.4.html">Apache Flink 1.6.4 Released</a></h2>
<p>25 Feb 2019
@@ -330,21 +343,6 @@ Please check the <a
href="https://issues.apache.org/jira/secure/ReleaseNote.jspa
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2018/09/20/release-1.6.1.html">Apache Flink 1.6.1 Released</a></h2>
-
- <p>20 Sep 2018
- </p>
-
- <p><p>The Apache Flink community released the first bugfix version of
the Apache Flink 1.6 series.</p>
-
-</p>
-
- <p><a href="/news/2018/09/20/release-1.6.1.html">Continue reading
»</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -377,6 +375,16 @@ Please check the <a
href="https://issues.apache.org/jira/secure/ReleaseNote.jspa
<ul id="markdown-toc">
+ <li><a href="/news/2020/08/06/external-resource.html">Accelerating your
workload with GPU and other external resources</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2020/08/04/pyflink-pandas-udf-support-flink.html">PyFlink:
The integration of Pandas into PyFlink</a></li>
diff --git a/content/blog/page7/index.html b/content/blog/page7/index.html
index f02c592..b89617d 100644
--- a/content/blog/page7/index.html
+++ b/content/blog/page7/index.html
@@ -196,6 +196,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a
href="/news/2018/09/20/release-1.6.1.html">Apache Flink 1.6.1 Released</a></h2>
+
+ <p>20 Sep 2018
+ </p>
+
+ <p><p>The Apache Flink community released the first bugfix version of
the Apache Flink 1.6 series.</p>
+
+</p>
+
+ <p><a href="/news/2018/09/20/release-1.6.1.html">Continue reading
»</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a
href="/news/2018/09/20/release-1.5.4.html">Apache Flink 1.5.4 Released</a></h2>
<p>20 Sep 2018
@@ -328,21 +343,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2018/02/15/release-1.4.1.html">Apache Flink 1.4.1 Released</a></h2>
-
- <p>15 Feb 2018
- </p>
-
- <p><p>The Apache Flink community released the first bugfix version of
the Apache Flink 1.4 series.</p>
-
-</p>
-
- <p><a href="/news/2018/02/15/release-1.4.1.html">Continue reading
»</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -375,6 +375,16 @@
<ul id="markdown-toc">
+ <li><a href="/news/2020/08/06/external-resource.html">Accelerating your
workload with GPU and other external resources</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2020/08/04/pyflink-pandas-udf-support-flink.html">PyFlink:
The integration of Pandas into PyFlink</a></li>
diff --git a/content/blog/page8/index.html b/content/blog/page8/index.html
index c204a8d..1b4a7be 100644
--- a/content/blog/page8/index.html
+++ b/content/blog/page8/index.html
@@ -196,6 +196,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a
href="/news/2018/02/15/release-1.4.1.html">Apache Flink 1.4.1 Released</a></h2>
+
+ <p>15 Feb 2018
+ </p>
+
+ <p><p>The Apache Flink community released the first bugfix version of
the Apache Flink 1.4 series.</p>
+
+</p>
+
+ <p><a href="/news/2018/02/15/release-1.4.1.html">Continue reading
»</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a
href="/features/2018/01/30/incremental-checkpointing.html">Managing Large State
in Apache Flink: An Intro to Incremental Checkpointing</a></h2>
<p>30 Jan 2018
@@ -329,21 +344,6 @@ what’s coming in Flink 1.4.0 as well as a preview of what
the Flink community
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2017/04/26/release-1.2.1.html">Apache Flink 1.2.1 Released</a></h2>
-
- <p>26 Apr 2017
- </p>
-
- <p><p>The Apache Flink community released the first bugfix version of
the Apache Flink 1.2 series.</p>
-
-</p>
-
- <p><a href="/news/2017/04/26/release-1.2.1.html">Continue reading
»</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -376,6 +376,16 @@ what’s coming in Flink 1.4.0 as well as a preview of what
the Flink community
<ul id="markdown-toc">
+ <li><a href="/news/2020/08/06/external-resource.html">Accelerating your
workload with GPU and other external resources</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2020/08/04/pyflink-pandas-udf-support-flink.html">PyFlink:
The integration of Pandas into PyFlink</a></li>
diff --git a/content/blog/page9/index.html b/content/blog/page9/index.html
index 1a42cc9..67691c2 100644
--- a/content/blog/page9/index.html
+++ b/content/blog/page9/index.html
@@ -196,6 +196,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a
href="/news/2017/04/26/release-1.2.1.html">Apache Flink 1.2.1 Released</a></h2>
+
+ <p>26 Apr 2017
+ </p>
+
+ <p><p>The Apache Flink community released the first bugfix version of
the Apache Flink 1.2 series.</p>
+
+</p>
+
+ <p><a href="/news/2017/04/26/release-1.2.1.html">Continue reading
»</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a
href="/news/2017/04/04/dynamic-tables.html">Continuous Queries on Dynamic
Tables</a></h2>
<p>04 Apr 2017 by Fabian Hueske, Shaoxuan Wang, and Xiaowei Jiang
@@ -323,21 +338,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a
href="/news/2016/08/11/release-1.1.1.html">Flink 1.1.1 Released</a></h2>
-
- <p>11 Aug 2016
- </p>
-
- <p><p>Today, the Flink community released Flink version 1.1.1.</p>
-
-</p>
-
- <p><a href="/news/2016/08/11/release-1.1.1.html">Continue reading
»</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -370,6 +370,16 @@
<ul id="markdown-toc">
+ <li><a href="/news/2020/08/06/external-resource.html">Accelerating your
workload with GPU and other external resources</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/2020/08/04/pyflink-pandas-udf-support-flink.html">PyFlink:
The integration of Pandas into PyFlink</a></li>
diff --git
a/content/img/blog/2020-08-06-accelerate-with-external-resources/ai-workflow.png
b/content/img/blog/2020-08-06-accelerate-with-external-resources/ai-workflow.png
new file mode 100644
index 0000000..8ef393a
Binary files /dev/null and
b/content/img/blog/2020-08-06-accelerate-with-external-resources/ai-workflow.png
differ
diff --git a/content/index.html b/content/index.html
index de5eb45..f6f58ae 100644
--- a/content/index.html
+++ b/content/index.html
@@ -568,6 +568,9 @@
<dl>
+ <dt> <a href="/news/2020/08/06/external-resource.html">Accelerating
your workload with GPU and other external resources</a></dt>
+ <dd>This post introduces the new External Resource Framework in Flink
1.11 and take GPU as an example to show how to accelerate your workload with
external resources.</dd>
+
<dt> <a
href="/2020/08/04/pyflink-pandas-udf-support-flink.html">PyFlink: The
integration of Pandas into PyFlink</a></dt>
<dd>The Apache Flink community put some great effort into integrating
Pandas with PyFlink in the latest Flink version 1.11. Some of the added
features include support for Pandas UDF and the conversion between Pandas
DataFrame and Table. In this article, we will introduce how these
functionalities work and how to use them with a step-by-step example.</dd>
@@ -579,11 +582,6 @@
<dt> <a href="/news/2020/07/27/community-update.html">Flink Community
Update - July'20</a></dt>
<dd>As July draws to an end, we look back at a monthful of activity in
the Flink community, including two releases (!) and some work around improving
the first-time contribution experience in the project. Also, events are
starting to pick up again, so we've put together a list of some great events
you can (virtually) attend in August!</dd>
-
- <dt> <a href="/2020/07/23/catalogs.html">Sharing is caring - Catalogs
in Flink SQL</a></dt>
- <dd><p>With an ever-growing number of people working with data, it’s a
common practice for companies to build self-service platforms with the goal of
democratizing their access across different teams and — especially — to enable
users from any background to be independent in their data needs. In such
environments, metadata management becomes a crucial aspect. Without it, users
often work blindly, spending too much time searching for datasets and their
location, figuring out data [...]
-
-</dd>
</dl>
diff --git a/content/news/2020/08/06/external-resource.html
b/content/news/2020/08/06/external-resource.html
new file mode 100644
index 0000000..5c39071
--- /dev/null
+++ b/content/news/2020/08/06/external-resource.html
@@ -0,0 +1,378 @@
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <!-- The above 3 meta tags *must* come first in the head; any other head
content must come *after* these tags -->
+ <title>Apache Flink: Accelerating your workload with GPU and other
external resources</title>
+ <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
+ <link rel="icon" href="/favicon.ico" type="image/x-icon">
+
+ <!-- Bootstrap -->
+ <link rel="stylesheet"
href="https://maxcdn.bootstrapcdn.com/bootstrap/3.4.1/css/bootstrap.min.css";>
+ <link rel="stylesheet" href="/css/flink.css">
+ <link rel="stylesheet" href="/css/syntax.css">
+
+ <!-- Blog RSS feed -->
+ <link href="/blog/feed.xml" rel="alternate" type="application/rss+xml"
title="Apache Flink Blog: RSS feed" />
+
+ <!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
+ <!-- We need to load Jquery in the header for custom google analytics
event tracking-->
+ <script src="/js/jquery.min.js"></script>
+
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media
queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script
src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js";></script>
+ <script
src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js";></script>
+ <![endif]-->
+ </head>
+ <body>
+
+
+ <!-- Main content. -->
+ <div class="container">
+ <div class="row">
+
+
+ <div id="sidebar" class="col-sm-3">
+
+
+<!-- Top navbar. -->
+ <nav class="navbar navbar-default">
+ <!-- The logo. -->
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed"
data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <div class="navbar-logo">
+ <a href="/">
+ <img alt="Apache Flink" src="/img/flink-header-logo.svg"
width="147px" height="73px">
+ </a>
+ </div>
+ </div><!-- /.navbar-header -->
+
+ <!-- The navigation links. -->
+ <div class="collapse navbar-collapse"
id="bs-example-navbar-collapse-1">
+ <ul class="nav navbar-nav navbar-main">
+
+ <!-- First menu section explains visitors what Flink is -->
+
+ <!-- What is Stream Processing? -->
+ <!--
+ <li><a href="/streamprocessing1.html">What is Stream
Processing?</a></li>
+ -->
+
+ <!-- What is Flink? -->
+ <li><a href="/flink-architecture.html">What is Apache
Flink?</a></li>
+
+
+
+ <!-- What is Stateful Functions? -->
+
+ <li><a href="/stateful-functions.html">What is Stateful
Functions?</a></li>
+
+ <!-- Use cases -->
+ <li><a href="/usecases.html">Use Cases</a></li>
+
+ <!-- Powered by -->
+ <li><a href="/poweredby.html">Powered By</a></li>
+
+
+
+ <!-- Second menu section aims to support Flink users -->
+
+ <!-- Downloads -->
+ <li><a href="/downloads.html">Downloads</a></li>
+
+ <!-- Getting Started -->
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown"
href="#">Getting Started<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a
href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/getting-started/index.html";
target="_blank">With Flink <small><span class="glyphicon
glyphicon-new-window"></span></small></a></li>
+ <li><a
href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/getting-started/project-setup.html";
target="_blank">With Flink Stateful Functions <small><span class="glyphicon
glyphicon-new-window"></span></small></a></li>
+ <li><a href="/training.html">Training Course</a></li>
+ </ul>
+ </li>
+
+ <!-- Documentation -->
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown"
href="#">Documentation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a
href="https://ci.apache.org/projects/flink/flink-docs-release-1.11";
target="_blank">Flink 1.11 (Latest stable release) <small><span
class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a
href="https://ci.apache.org/projects/flink/flink-docs-master";
target="_blank">Flink Master (Latest Snapshot) <small><span class="glyphicon
glyphicon-new-window"></span></small></a></li>
+ <li><a
href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1";
target="_blank">Flink Stateful Functions 2.1 (Latest stable release)
<small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a
href="https://ci.apache.org/projects/flink/flink-statefun-docs-master";
target="_blank">Flink Stateful Functions Master (Latest Snapshot) <small><span
class="glyphicon glyphicon-new-window"></span></small></a></li>
+ </ul>
+ </li>
+
+ <!-- getting help -->
+ <li><a href="/gettinghelp.html">Getting Help</a></li>
+
+ <!-- Blog -->
+ <li class="active"><a href="/blog/"><b>Flink Blog</b></a></li>
+
+
+ <!-- Flink-packages -->
+ <li>
+ <a href="https://flink-packages.org";
target="_blank">flink-packages.org <small><span class="glyphicon
glyphicon-new-window"></span></small></a>
+ </li>
+
+
+ <!-- Third menu section aim to support community and contributors
-->
+
+ <!-- Community -->
+ <li><a href="/community.html">Community & Project Info</a></li>
+
+ <!-- Roadmap -->
+ <li><a href="/roadmap.html">Roadmap</a></li>
+
+ <!-- Contribute -->
+ <li><a href="/contributing/how-to-contribute.html">How to
Contribute</a></li>
+
+
+ <!-- GitHub -->
+ <li>
+ <a href="https://github.com/apache/flink"; target="_blank">Flink
on GitHub <small><span class="glyphicon
glyphicon-new-window"></span></small></a>
+ </li>
+
+
+
+ <!-- Language Switcher -->
+ <li>
+
+
+ <!-- link to the Chinese home page when current is blog page
-->
+ <a href="/zh">中文版</a>
+
+
+ </li>
+
+ </ul>
+
+ <ul class="nav navbar-nav navbar-bottom">
+ <hr />
+
+ <!-- Twitter -->
+ <li><a href="https://twitter.com/apacheflink";
target="_blank">@ApacheFlink <small><span class="glyphicon
glyphicon-new-window"></span></small></a></li>
+
+ <!-- Visualizer -->
+ <li class=" hidden-md hidden-sm"><a href="/visualizer/"
target="_blank">Plan Visualizer <small><span class="glyphicon
glyphicon-new-window"></span></small></a></li>
+
+ <hr />
+
+ <li><a href="https://apache.org"; target="_blank">Apache Software
Foundation <small><span class="glyphicon
glyphicon-new-window"></span></small></a></li>
+
+ <li>
+ <style>
+ .smalllinks:link {
+ display: inline-block !important; background: none;
padding-top: 0px; padding-bottom: 0px; padding-right: 0px; min-width: 75px;
+ }
+ </style>
+
+ <a class="smalllinks" href="https://www.apache.org/licenses/";
target="_blank">License</a> <small><span class="glyphicon
glyphicon-new-window"></span></small>
+
+ <a class="smalllinks" href="https://www.apache.org/security/";
target="_blank">Security</a> <small><span class="glyphicon
glyphicon-new-window"></span></small>
+
+ <a class="smalllinks"
href="https://www.apache.org/foundation/sponsorship.html";
target="_blank">Donate</a> <small><span class="glyphicon
glyphicon-new-window"></span></small>
+
+ <a class="smalllinks"
href="https://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a>
<small><span class="glyphicon glyphicon-new-window"></span></small>
+ </li>
+
+ </ul>
+ </div><!-- /.navbar-collapse -->
+ </nav>
+
+ </div>
+ <div class="col-sm-9">
+ <div class="row-fluid">
+ <div class="col-sm-12">
+ <div class="row">
+ <h1>Accelerating your workload with GPU and other external resources</h1>
+ <p><i></i></p>
+
+ <article>
+ <p>06 Aug 2020 Yangze Guo </p>
+
+<p>Apache Flink 1.11 introduces a new <a
href="https://ci.apache.org/projects/flink/flink-docs-master/ops/external_resources.html";>External
Resource Framework</a>,
+which allows you to request external resources from the underlying resource
management systems (e.g., Kubernetes) and accelerate your workload with
+those resources. As Flink provides a first-party GPU plugin at the moment, we
will take GPU as an example and show how it affects Flink applications
+in the AI field. Other external resources (e.g. RDMA and SSD) can also be
supported <a
href="https://ci.apache.org/projects/flink/flink-docs-master/ops/external_resources.html#implement-a-plugin-for-your-custom-resource-type";>in
a pluggable manner</a>.</p>
+
+<h1 id="end-to-end-real-time-ai-with-gpu">End-to-end real-time AI with GPU</h1>
+
+<p>Recently, AI and Machine Learning have gained additional popularity and
have been widely used in various scenarios, such
+as personalized recommendation and image recognition. <a
href="https://flink.apache.org/";>Flink</a>, with the ability to support GPU
+allocation, can be used to build an end-to-end real-time AI workflow.</p>
+
+<h2 id="why-flink">Why Flink</h2>
+
+<p>Typical AI workloads fall into two categories: training and inference.</p>
+
+<center>
+<img
src="/img/blog/2020-08-06-accelerate-with-external-resources/ai-workflow.png"
width="800px" alt="Typical AI Workflow" />
+<br />
+<i><small>Typical AI Workflow</small></i>
+</center>
+<p><br /></p>
+
+<p>The training workload is usually a batch task, in which we train a model
from a bounded dataset. On the other hand, the inference
+workload tends to be a streaming job. It consumes an unbounded data stream,
which contains image data, for example, and uses a model
+to produce the output of predictions. Both workloads need to do data
preprocessing first. Flink, as a
+<a
href="https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html";>unified
batch and stream processing engine</a>, can be used to build an end-to-end AI
workflow naturally.</p>
+
+<p>In many cases, the training and inference workload can benefit a lot by
leveraging GPUs. <a
href="https://azure.microsoft.com/en-us/blog/gpus-vs-cpus-for-deployment-of-deep-learning-models/";>Research</a>
+shows that CPU cluster is outperformed by GPU cluster, which is of similar
cost, by about 400 percent. As training datasets
+are getting bigger and models more complex, supporting GPUs has become
mandatory for running AI workloads.</p>
+
+<p>With the <a
href="https://ci.apache.org/projects/flink/flink-docs-master/ops/external_resources.html";>External
Resource Framework</a>
+and its <a
href="https://ci.apache.org/projects/flink/flink-docs-master/ops/external_resources.html#plugin-for-gpu-resources";>GPU
plugin</a>, Flink
+can now request GPU resources from the external resource management system and
expose GPU information to operators. With this
+feature, users can now easily build end-to-end training and real-time
inference pipelines with GPU support on Flink.</p>
+
+<h2 id="example-mnist-inference-with-flink">Example: MNIST Inference with
Flink</h2>
+
+<p>We take the MNIST inference task as an example to show how to use the <a
href="https://ci.apache.org/projects/flink/flink-docs-master/ops/external_resources.html";>External
Resource Framework</a>
+and how to leverage GPUs in Flink. MNIST is a database of handwritten digits,
which is usually viewed as the HelloWorld of AI.
+The goal is to recognize a 28px*28px picture of a number from 0 to 9.</p>
+
+<p>First, you need to set configurations for the external resource framework
to enable GPU support:</p>
+
+<div class="highlight"><pre><code class="language-bash">external-resources: gpu
+<span class="c"># Define the driver factory class of gpu resource.</span>
+external-resource.gpu.driver-factory.class:
org.apache.flink.externalresource.gpu.GPUDriverFactory
+<span class="c"># Define the amount of gpu resource per TaskManager.</span>
+external-resource.gpu.amount: 1
+<span class="c"># Enable the coordination mode if you run it in standalone
mode</span>
+external-resource.gpu.param.discovery-script.args: --enable-coordination
+
+
+<span class="c"># If you run it on Yarn</span>
+external-resource.gpu.yarn.config-key: yarn.io/gpu
+<span class="c"># If you run it on Kubernetes</span>
+external-resource.gpu.kubernetes.config-key: nvidia.com/gpu</code></pre></div>
+
+<p>For more details of the configuration, please refer to the <a
href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/external_resources.html#configurations-1";>official
documentation</a>.</p>
+
+<p>In the MNIST inference task, we first need to read the images and do data
preprocessing. You can download <a
href="http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz";>training</a>
+or <a
href="http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz";>testing</a>
data from <a href="http://yann.lecun.com/exdb/mnist/";>this site</a>.
+We provide a simple <a
href="https://github.com/KarmaGYZ/flink-mnist/blob/master/src/main/java/org/apache/flink/MNISTReader.java";>MNISTReader</a>.
+It will read the image data located in the provided file path and transform
each image into a list of floating point numbers.</p>
+
+<p>Then, we need a classifier to recognize those images. A one-layer
pre-trained neural network, whose prediction accuracy is 92.14%,
+is used in our classify operator. To leverage GPUs in order to accelerate the
matrix-matrix multiplication, we use <a
href="https://github.com/jcuda/jcuda";>JCuda</a>
+to call the native Cuda API. The prediction logic of the <a
href="https://github.com/KarmaGYZ/flink-mnist/blob/master/src/main/java/org/apache/flink/MNISTClassifier.java";>MNISTClassifier</a>
is shown below.</p>
+
+<div class="highlight"><pre><code class="language-java"><span
class="kd">class</span> <span class="nc">MNISTClassifier</span> <span
class="kd">extends</span> <span class="n">RichMapFunction</span><span
class="o"><</span><span class="n">List</span><span
class="o"><</span><span class="n">Float</span><span class="o">>,</span>
<span class="n">Integer</span><span class="o">></span> <span
class="o">{</span>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span
class="nf">open</span><span class="o">(</span><span
class="n">Configuration</span> <span class="n">parameters</span><span
class="o">)</span> <span class="o">{</span>
+ <span class="c1">// Get the GPU information and select the first
GPU.</span>
+ <span class="kd">final</span> <span class="n">Set</span><span
class="o"><</span><span class="n">ExternalResourceInfo</span><span
class="o">></span> <span class="n">externalResourceInfos</span> <span
class="o">=</span> <span class="n">getRuntimeContext</span><span
class="o">().</span><span class="na">getExternalResourceInfos</span><span
class="o">(</span><span class="n">resourceName</span><span class="o">);</span>
+ <span class="kd">final</span> <span class="n">Optional</span><span
class="o"><</span><span class="n">String</span><span class="o">></span>
<span class="n">firstIndexOptional</span> <span class="o">=</span> <span
class="n">externalResourceInfos</span><span class="o">.</span><span
class="na">iterator</span><span class="o">().</span><span
class="na">next</span><span class="o">().</span><span
class="na">getProperty</span><span class="o">(</span><span
class="s">"index" [...]
+
+ <span class="c1">// Initialize JCublas with the selected GPU</span>
+ <span class="n">JCuda</span><span class="o">.</span><span
class="na">cudaSetDevice</span><span class="o">(</span><span
class="n">Integer</span><span class="o">.</span><span
class="na">parseInt</span><span class="o">(</span><span
class="n">firstIndexOptional</span><span class="o">.</span><span
class="na">get</span><span class="o">()));</span>
+ <span class="n">JCublas</span><span class="o">.</span><span
class="na">cublasInit</span><span class="o">();</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">Integer</span> <span
class="nf">map</span><span class="o">(</span><span class="n">List</span><span
class="o"><</span><span class="n">Float</span><span class="o">></span>
<span class="n">value</span><span class="o">)</span> <span class="o">{</span>
+ <span class="c1">// Performs multiplication using JCublas. The
matrixPointer points to our pre-trained model.</span>
+ <span class="n">JCublas</span><span class="o">.</span><span
class="na">cublasSgemv</span><span class="o">(</span><span
class="sc">'n'</span><span class="o">,</span> <span
class="n">DIMENSIONS</span><span class="o">.</span><span
class="na">f1</span><span class="o">,</span> <span
class="n">DIMENSIONS</span><span class="o">.</span><span
class="na">f0</span><span class="o">,</span> <span class="mf">1.0f</span><span
class="o">,</span>
+ <span class="n">matrixPointer</span><span class="o">,</span>
<span class="n">DIMENSIONS</span><span class="o">.</span><span
class="na">f1</span><span class="o">,</span> <span
class="n">inputPointer</span><span class="o">,</span> <span
class="mi">1</span><span class="o">,</span> <span class="mf">0.0f</span><span
class="o">,</span> <span class="n">outputPointer</span><span class="o">,</span>
<span class="mi">1</span><span class="o">);</span>
+
+ <span class="c1">// Read the result back from GPU.</span>
+ <span class="n">JCublas</span><span class="o">.</span><span
class="na">cublasGetVector</span><span class="o">(</span><span
class="n">DIMENSIONS</span><span class="o">.</span><span
class="na">f1</span><span class="o">,</span> <span class="n">Sizeof</span><span
class="o">.</span><span class="na">FLOAT</span><span class="o">,</span> <span
class="n">outputPointer</span><span class="o">,</span> <span
class="mi">1</span><span class="o">,</span> <span class="n">Pointer</span><span
class= [...]
+ <span class="kt">int</span> <span class="n">result</span> <span
class="o">=</span> <span class="mi">0</span><span class="o">;</span>
+ <span class="k">for</span> <span class="o">(</span><span
class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span
class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span
class="o"><</span> <span class="n">DIMENSIONS</span><span
class="o">.</span><span class="na">f1</span><span class="o">;</span> <span
class="o">++</span><span class="n">i</span><span class="o">)</span> <span
class="o">{</span>
+ <span class="n">result</span> <span class="o">=</span> <span
class="n">output</span><span class="o">[</span><span class="n">i</span><span
class="o">]</span> <span class="o">></span> <span
class="n">output</span><span class="o">[</span><span
class="n">result</span><span class="o">]</span> <span class="o">?</span> <span
class="n">i</span> <span class="o">:</span> <span class="n">result</span><span
class="o">;</span>
+ <span class="o">}</span>
+ <span class="k">return</span> <span class="n">result</span><span
class="o">;</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<p>The complete MNIST inference project can be found <a
href="https://github.com/KarmaGYZ/flink-mnist";>here</a>. In this project, we
simply
+print the inference result to <strong>STDOUT</strong>. In the actual
production environment, you could also write the result to Elasticsearch or
Kafka, for example.</p>
+
+<p>The MNIST inference task is just a simple case that shows you how the
external resource framework works and what Flink can
+do with GPU support. With Flink’s open source extension <a
href="https://github.com/alibaba/Alink";>Alink</a>, which contains a lot of
+pre-built algorithms based on Flink, and <a
href="https://github.com/alibaba/flink-ai-extended";>Tensorflow on Flink</a>,
some complex
+AI workloads, e.g. online learning, real-time inference service, could be
easily implemented as well.</p>
+
+<h1 id="other-external-resources">Other external resources</h1>
+
+<p>In addition to GPU support, there are many other external resources that
can be used to accelerate jobs in some specific scenarios.
+E.g. FPGA, for AI workloads, is supported by both Yarn and Kubernetes. Some
low-latency network devices, like RDMA and Solarflare, also
+provide their device plugin for Kubernetes. Currently, Yarn supports GPUs and
FPGAs, while the list of Kubernetes’ device plugins can be found <a
href="https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#examples";>here</a>.</p>
+
+<p>With the external resource framework, you only need to implement a plugin
that enables the operator to get the information
+for these external resources; see <a
href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/external_resources.html#implement-a-plugin-for-your-custom-resource-type";>Custom
Plugin</a>
+for more details. If you just want to ensure that an external resource exists
in the TaskManager, then you only need to find the
+configuration key of that resource in the underlying resource management
system and configure the external resource framework accordingly.</p>
+
+<h1 id="conclusion">Conclusion</h1>
+
+<p>In the latest Flink release (Flink 1.11), an external resource framework
has been introduced to support requesting various types of
+resources from the underlying resource management systems, and supply all the
necessary information for using these resources to the
+operators. The first-party GPU plugin expands the application prospects of
Flink in the AI domain. Different resource types can be supported
+in a pluggable way. You can also implement your own plugins for custom
resource types.</p>
+
+<p>Future developments in this area include implementing operator level
resource isolation and fine-grained external resource scheduling.
+The community may kick this work off once <a
href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation";>FLIP-56</a>
+is finished. If you have any suggestions or questions for the community, we
encourage you to sign up to the Apache Flink
+<a href="https://flink.apache.org/community.html#mailing-lists";>mailing
lists</a> and join the discussion there.</p>
+
+ </article>
+ </div>
+
+ <div class="row">
+ <div id="disqus_thread"></div>
+ <script type="text/javascript">
+ /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE
* * */
+ var disqus_shortname = 'stratosphere-eu'; // required: replace example
with your forum shortname
+
+ /* * * DON'T EDIT BELOW THIS LINE * * */
+ (function() {
+ var dsq = document.createElement('script'); dsq.type =
'text/javascript'; dsq.async = true;
+ dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
+ (document.getElementsByTagName('head')[0] ||
document.getElementsByTagName('body')[0]).appendChild(dsq);
+ })();
+ </script>
+ </div>
+ </div>
+</div>
+ </div>
+ </div>
+
+ <hr />
+
+ <div class="row">
+ <div class="footer text-center col-sm-12">
+ <p>Copyright © 2014-2019 <a href="http://apache.org";>The Apache
Software Foundation</a>. All Rights Reserved.</p>
+ <p>Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache
feather logo are either registered trademarks or trademarks of The Apache
Software Foundation.</p>
+ <p><a href="/privacy-policy.html">Privacy Policy</a> · <a
href="/blog/feed.xml">RSS feed</a></p>
+ </div>
+ </div>
+ </div><!-- /.container -->
+
+ <!-- Include all compiled plugins (below), or include individual files as
needed -->
+ <script
src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js";></script>
+ <script
src="https://cdnjs.cloudflare.com/ajax/libs/jquery.matchHeight/0.7.0/jquery.matchHeight-min.js";></script>
+ <script src="/js/codetabs.js"></script>
+ <script src="/js/stickysidebar.js"></script>
+
+ <!-- Google Analytics -->
+ <script>
+
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new
Date();a=s.createElement(o),
+
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-52545728-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+ </body>
+</html>
diff --git a/content/zh/index.html b/content/zh/index.html
index b70d5c2..6656e7c 100644
--- a/content/zh/index.html
+++ b/content/zh/index.html
@@ -565,6 +565,9 @@
<dl>
+ <dt> <a href="/news/2020/08/06/external-resource.html">Accelerating
your workload with GPU and other external resources</a></dt>
+ <dd>This post introduces the new External Resource Framework in Flink
1.11 and take GPU as an example to show how to accelerate your workload with
external resources.</dd>
+
<dt> <a
href="/2020/08/04/pyflink-pandas-udf-support-flink.html">PyFlink: The
integration of Pandas into PyFlink</a></dt>
<dd>The Apache Flink community put some great effort into integrating
Pandas with PyFlink in the latest Flink version 1.11. Some of the added
features include support for Pandas UDF and the conversion between Pandas
DataFrame and Table. In this article, we will introduce how these
functionalities work and how to use them with a step-by-step example.</dd>
@@ -576,11 +579,6 @@
<dt> <a href="/news/2020/07/27/community-update.html">Flink Community
Update - July'20</a></dt>
<dd>As July draws to an end, we look back at a monthful of activity in
the Flink community, including two releases (!) and some work around improving
the first-time contribution experience in the project. Also, events are
starting to pick up again, so we've put together a list of some great events
you can (virtually) attend in August!</dd>
-
- <dt> <a href="/2020/07/23/catalogs.html">Sharing is caring - Catalogs
in Flink SQL</a></dt>
- <dd><p>With an ever-growing number of people working with data, it’s a
common practice for companies to build self-service platforms with the goal of
democratizing their access across different teams and — especially — to enable
users from any background to be independent in their data needs. In such
environments, metadata management becomes a crucial aspect. Without it, users
often work blindly, spending too much time searching for datasets and their
location, figuring out data [...]
-
-</dd>
</dl>
