Modified: flink/site/blog/page2/index.html URL: http://svn.apache.org/viewvc/flink/site/blog/page2/index.html?rev=1661871&r1=1661870&r2=1661871&view=diff ============================================================================== --- flink/site/blog/page2/index.html (original) +++ flink/site/blog/page2/index.html Tue Feb 24 09:47:45 2015 @@ -140,6 +140,96 @@ <div class="col-md-8"> <article> + <h2><a href="/news/2014/05/31/release-0.5.html">Stratosphere version 0.5 available</a></h2> + <p class="meta">31 May 2014</p> + + <div><p>We are happy to announce a new major Stratosphere release, version 0.5. This release adds many new features and improves the interoperability, stability, and performance of the system. The major theme of the release is the completely new Java API that makes it easy to write powerful distributed programs.</p> + +<p>The release can be downloaded from the <a href="http://stratosphere.eu/downloads/";>Stratosphere website</a> and from <a href="https://github.com/stratosphere/stratosphere/releases/tag/release-0.5";>GitHub</a>. All components are available as Apache Maven dependencies, making it simple to include Stratosphere in other projects. The website provides <a href="http://stratosphere.eu/docs/0.5/";>extensive documentation</a> of the system and the new features.</p> + +<h2 id="shortlist-of-new-features">Shortlist of new Features</h2> + +<p>Below is a short list of the most important additions to the Stratosphere system.</p> + +<h4 id="new-java-api">New Java API</h4> + +<p>This release introduces a completely new <strong>data set-centric Java API</strong>. This programming model significantly eases the development of Stratosphere programs, supports flexible use of regular Java classes as data types, and adds many new built-in operators to simplify the writing of powerful programs. The result are programs that need less code, are more readable, interoperate better with existing code, and execute faster.</p> + +<p>Take a look at the <a href="http://stratosphere.eu/docs/0.5/programming_guides/examples_java.html";>examples</a> to get a feel for the API.</p> + +<h4 id="general-api-improvements">General API Improvements</h4> + +<p><strong>Broadcast Variables:</strong> Publish a data set to all instances of another operator. This is handy if the your operator depends on the result of a computation, e.g., filter all values smaller than the average.</p> + +<p><strong>Distributed Cache:</strong> Make (local and HDFS) files locally available on each machine processing a task.</p> + +<p><strong>Iteration Termination Improvements</strong> Iterative algorithms can now terminate based on intermediate data sets, not only through aggregated statistics.</p> + +<p><strong>Collection data sources and sinks:</strong> Speed-up the development and testing of Stratosphere programs by reading data from regular Java collections and inserting back into them.</p> + +<p><strong>JDBC data sources and sinks:</strong> Read data from and write data to relational databases using a JDBC driver.</p> + +<p><strong>Hadoop input format and output format support:</strong> Read and write data with any Hadoop input or output format.</p> + +<p><strong>Support for Avro encoded data:</strong> Read data that has been materialized using Avro.</p> + +<p><strong>Deflate Files:</strong> Stratosphere now transparently reads <code>.deflate</code> compressed files.</p> + +<h4 id="runtime-and-optimizer-improvements">Runtime and Optimizer Improvements</h4> + +<p><strong>DAG Runtime Streaming:</strong> Detection and resolution of streaming data flow deadlocks in the data flow optimizer.</p> + +<p><strong>Intermediate results across iteration boundaries:</strong> Intermediate results computed outside iterative parts can be used inside iterative parts of the program.</p> + +<p><strong>Stability fixes:</strong> Various stability fixes in both optimizer and runtime.</p> + +<h4 id="setup-&-tooling">Setup & Tooling</h4> + +<p><strong>Improved YARN support:</strong> Many improvements based on user-feedback: Packaging, Permissions, Error handling.</p> + +<p><strong>Java 8 compatibility</strong></p> + +<h2 id="contributors">Contributors</h2> + +<p>In total, 26 people have contributed to Stratosphere since the last release. Thank you for making this project possible!</p> + +<ul> +<li>Alexander Alexandrov</li> +<li>Jesus Camacho</li> +<li>Ufuk Celebi</li> +<li>Mikhail Erofeev</li> +<li>Stephan Ewen</li> +<li>Alexandr Ferodov</li> +<li>Filip Haase</li> +<li>Jonathan Hasenberg</li> +<li>Markus Holzemer</li> +<li>Fabian Hueske</li> +<li>Vasia Kalavri</li> +<li>Aljoscha Krettek</li> +<li>Rajika Kumarasiri</li> +<li>Sebastian Kunert</li> +<li>Aaron Lam</li> +<li>Robert Metzger</li> +<li>Faisal Moeen</li> +<li>Martin Neumann</li> +<li>Mingliang Qi</li> +<li>Till Rohrmann</li> +<li>Chesnay Schepler</li> +<li>Vyachislav Soludev</li> +<li>Tuan Trieu</li> +<li>Artem Tsikiridis</li> +<li>Timo Walther</li> +<li>Robert Waury</li> +</ul> + +<h2 id="stratosphere-is-going-apache">Stratosphere is going Apache</h2> + +<p>The Stratosphere project has been accepted to the Apache Incubator and will continue its work under the umbrella of the Apache Software Foundation. Due to a name conflict, we are switching the name of the project. We will make future releases of Stratosphere through the Apache foundation under a new name.</p> +</div> + <a href="/news/2014/05/31/release-0.5.html#disqus_thread">Stratosphere version 0.5 available</a> + </article> + + <article> <h2><a href="/news/2014/04/16/stratosphere-goes-apache-incubator.html">Stratosphere accepted as Apache Incubator Project</a></h2> <p class="meta">16 Apr 2014</p> @@ -685,54 +775,6 @@ For a complete overview of the renamings <a href="/news/2013/12/13/humboldt-innovation-award.html#disqus_thread">Stratosphere wins award at Humboldt Innovation Competition "Big Data: Research meets Startups"</a> </article> - <article> - <h2><a href="/news/2013/10/21/cikm2013-paper.html">Paper "âAll Roads Lead to Rome:â Optimistic Recovery for Distributed Iterative Data Processing" accepted at CIKM 2013</a></h2> - <p class="meta">21 Oct 2013</p> - - <div><p>Our paper "âAll Roads Lead to Rome:â Optimistic Recovery for Distributed -Iterative Data Processing" authored by Sebastian Schelter, Kostas -Tzoumas, Stephan Ewen and Volker Markl has been accepted accepted at the -ACM International Conference on Information and Knowledge Management -(CIKM 2013) in San Francisco.</p> - -<p><strong>Abstract</strong></p> - -<p>Executing data-parallel iterative algorithms on large datasets is -crucial for many advanced analytical applications in the fields of data -mining and machine learning. Current systems for executing iterative -tasks in large clusters typically achieve fault tolerance through -rollback recovery. The principle behind this pessimistic approach is to -periodically checkpoint the algorithm state. Upon failure, the system -restores a consistent state from a previously written checkpoint and -resumes execution from that point.</p> - -<p>We propose an optimistic recovery mechanism using algorithmic -compensations. Our method leverages the robust, self-correcting nature -of a large class of fixpoint algorithms used in data mining and machine -learning, which converge to the correct solution from various -intermediate consistent states. In the case of a failure, we apply a -user-defined compensate function that algorithmically creates such a -consistent state, instead of rolling back to a previous checkpointed -state. Our optimistic recovery does not checkpoint any state and hence -achieves optimal failure-free performance with respect to the overhead -necessary for guaranteeing fault tolerance. We illustrate the -applicability of this approach for three wide classes of problems. -Furthermore, we show how to implement the proposed optimistic recovery -mechanism in a data flow system. Similar to the Combine operator in -MapReduce, our proposed functionality is optional and can be applied to -increase performance without changing the semantics of programs. In an -experimental evaluation on large datasets, we show that our proposed -approach provides optimal failure-free performance. In the absence of -failures our optimistic scheme is able to outperform a pessimistic -approach by a factor of two to five. In presence of failures, our -approach provides fast recovery and outperforms pessimistic approaches -in the majority of cases.</p> - -<p><a href="/assets/papers/optimistic.pdf">Download the paper [PDF]</a></p> -</div> - <a href="/news/2013/10/21/cikm2013-paper.html#disqus_thread">Paper "âAll Roads Lead to Rome:â Optimistic Recovery for Distributed Iterative Data Processing" accepted at CIKM 2013</a> - </article> - </div> <div class="col-md-2"></div> </div>
Modified: flink/site/blog/page3/index.html URL: http://svn.apache.org/viewvc/flink/site/blog/page3/index.html?rev=1661871&r1=1661870&r2=1661871&view=diff ============================================================================== --- flink/site/blog/page3/index.html (original) +++ flink/site/blog/page3/index.html Tue Feb 24 09:47:45 2015 @@ -140,6 +140,54 @@ <div class="col-md-8"> <article> + <h2><a href="/news/2013/10/21/cikm2013-paper.html">Paper "âAll Roads Lead to Rome:â Optimistic Recovery for Distributed Iterative Data Processing" accepted at CIKM 2013</a></h2> + <p class="meta">21 Oct 2013</p> + + <div><p>Our paper "âAll Roads Lead to Rome:â Optimistic Recovery for Distributed +Iterative Data Processing" authored by Sebastian Schelter, Kostas +Tzoumas, Stephan Ewen and Volker Markl has been accepted accepted at the +ACM International Conference on Information and Knowledge Management +(CIKM 2013) in San Francisco.</p> + +<p><strong>Abstract</strong></p> + +<p>Executing data-parallel iterative algorithms on large datasets is +crucial for many advanced analytical applications in the fields of data +mining and machine learning. Current systems for executing iterative +tasks in large clusters typically achieve fault tolerance through +rollback recovery. The principle behind this pessimistic approach is to +periodically checkpoint the algorithm state. Upon failure, the system +restores a consistent state from a previously written checkpoint and +resumes execution from that point.</p> + +<p>We propose an optimistic recovery mechanism using algorithmic +compensations. Our method leverages the robust, self-correcting nature +of a large class of fixpoint algorithms used in data mining and machine +learning, which converge to the correct solution from various +intermediate consistent states. In the case of a failure, we apply a +user-defined compensate function that algorithmically creates such a +consistent state, instead of rolling back to a previous checkpointed +state. Our optimistic recovery does not checkpoint any state and hence +achieves optimal failure-free performance with respect to the overhead +necessary for guaranteeing fault tolerance. We illustrate the +applicability of this approach for three wide classes of problems. +Furthermore, we show how to implement the proposed optimistic recovery +mechanism in a data flow system. Similar to the Combine operator in +MapReduce, our proposed functionality is optional and can be applied to +increase performance without changing the semantics of programs. In an +experimental evaluation on large datasets, we show that our proposed +approach provides optimal failure-free performance. In the absence of +failures our optimistic scheme is able to outperform a pessimistic +approach by a factor of two to five. In presence of failures, our +approach provides fast recovery and outperforms pessimistic approaches +in the majority of cases.</p> + +<p><a href="/assets/papers/optimistic.pdf">Download the paper [PDF]</a></p> +</div> + <a href="/news/2013/10/21/cikm2013-paper.html#disqus_thread">Paper "âAll Roads Lead to Rome:â Optimistic Recovery for Distributed Iterative Data Processing" accepted at CIKM 2013</a> + </article> + + <article> <h2><a href="/news/2013/03/27/www-demo-paper.html">Demo Paper "Large-Scale Social-Media Analytics on Stratosphere" Accepted at WWW 2013</a></h2> <p class="meta">27 Mar 2013</p> Added: flink/site/q/quickstart-SNAPSHOT.sh URL: http://svn.apache.org/viewvc/flink/site/q/quickstart-SNAPSHOT.sh?rev=1661871&view=auto ============================================================================== --- flink/site/q/quickstart-SNAPSHOT.sh (added) +++ flink/site/q/quickstart-SNAPSHOT.sh Tue Feb 24 09:47:45 2015 @@ -0,0 +1,48 @@ +#!/usr/bin/env bash + +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + + +PACKAGE=quickstart + +mvn archetype:generate \ + -DarchetypeGroupId=org.apache.flink \ + -DarchetypeArtifactId=flink-quickstart-java \ + -DarchetypeVersion=0.9-SNAPSHOT \ + -DgroupId=org.apache.flink \ + -DartifactId=$PACKAGE \ + -Dversion=0.1 \ + -Dpackage=org.myorg.quickstart \ + -DinteractiveMode=false \ + -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/ + +# +# Give some guidance +# +echo -e "\\n\\n" +echo -e "\\tA sample quickstart Flink Job has been created." +echo -e "\\tSwitch into the directory using" +echo -e "\\t\\t cd $PACKAGE" +echo -e "\\tImport the project there using your favorite IDE (Import it as a maven project)" +echo -e "\\tBuild a jar inside the directory using" +echo -e "\\t\\t mvn clean package" +echo -e "\\tYou will find the runnable jar in $PACKAGE/target" +echo -e "\\tConsult our website if you have any troubles: http://flink.apache.org/community.html#mailing-lists"; +echo -e "\\n\\n" + Propchange: flink/site/q/quickstart-SNAPSHOT.sh ------------------------------------------------------------------------------ svn:executable = * Added: flink/site/q/quickstart-scala-SNAPSHOT.sh URL: http://svn.apache.org/viewvc/flink/site/q/quickstart-scala-SNAPSHOT.sh?rev=1661871&view=auto ============================================================================== --- flink/site/q/quickstart-scala-SNAPSHOT.sh (added) +++ flink/site/q/quickstart-scala-SNAPSHOT.sh Tue Feb 24 09:47:45 2015 @@ -0,0 +1,48 @@ +#!/usr/bin/env bash + +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + + +PACKAGE=quickstart + +mvn archetype:generate \ + -DarchetypeGroupId=org.apache.flink \ + -DarchetypeArtifactId=flink-quickstart-scala \ + -DarchetypeVersion=0.9-SNAPSHOT \ + -DgroupId=org.apache.flink \ + -DartifactId=$PACKAGE \ + -Dversion=0.1 \ + -Dpackage=org.myorg.quickstart \ + -DinteractiveMode=false \ + -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/ + +# +# Give some guidance +# +echo -e "\\n\\n" +echo -e "\\tA sample quickstart Flink Job has been created." +echo -e "\\tSwitch into the directory using" +echo -e "\\t\\t cd $PACKAGE" +echo -e "\\tImport the project there using your favorite IDE (Import it as a maven project)" +echo -e "\\tBuild a jar inside the directory using" +echo -e "\\t\\t mvn clean package" +echo -e "\\tYou will find the runnable jar in $PACKAGE/target" +echo -e "\\tConsult our website if you have any troubles: http://flink.apache.org/community.html#mailing-lists"; +echo -e "\\n\\n" + Propchange: flink/site/q/quickstart-scala-SNAPSHOT.sh ------------------------------------------------------------------------------ svn:executable = * Added: flink/site/q/quickstart-scala.sh URL: http://svn.apache.org/viewvc/flink/site/q/quickstart-scala.sh?rev=1661871&view=auto ============================================================================== --- flink/site/q/quickstart-scala.sh (added) +++ flink/site/q/quickstart-scala.sh Tue Feb 24 09:47:45 2015 @@ -0,0 +1,47 @@ +#!/usr/bin/env bash + +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + + +PACKAGE=quickstart + +mvn archetype:generate \ + -DarchetypeGroupId=org.apache.flink \ + -DarchetypeArtifactId=flink-quickstart-scala \ + -DarchetypeVersion=0.8.1 \ + -DgroupId=org.apache.flink \ + -DartifactId=$PACKAGE \ + -Dversion=0.1 \ + -Dpackage=org.myorg.quickstart \ + -DinteractiveMode=false + +# +# Give some guidance +# +echo -e "\\n\\n" +echo -e "\\tA sample quickstart Flink Job has been created." +echo -e "\\tSwitch into the directory using" +echo -e "\\t\\t cd $PACKAGE" +echo -e "\\tImport the project there using your favorite IDE (Import it as a maven project)" +echo -e "\\tBuild a jar inside the directory using" +echo -e "\\t\\t mvn clean package" +echo -e "\\tYou will find the runnable jar in $PACKAGE/target" +echo -e "\\tConsult our website if you have any troubles: http://flink.apache.org/community.html#mailing-lists"; +echo -e "\\n\\n" + Propchange: flink/site/q/quickstart-scala.sh ------------------------------------------------------------------------------ svn:executable = * Added: flink/site/q/quickstart.sh URL: http://svn.apache.org/viewvc/flink/site/q/quickstart.sh?rev=1661871&view=auto ============================================================================== --- flink/site/q/quickstart.sh (added) +++ flink/site/q/quickstart.sh Tue Feb 24 09:47:45 2015 @@ -0,0 +1,48 @@ +#!/usr/bin/env bash + +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + + +PACKAGE=quickstart + +mvn archetype:generate \ + -DarchetypeGroupId=org.apache.flink \ + -DarchetypeArtifactId=flink-quickstart-java \ + -DarchetypeVersion=0.8.1 \ + -DgroupId=org.apache.flink \ + -DartifactId=$PACKAGE \ + -Dversion=0.1 \ + -Dpackage=org.myorg.quickstart \ + -DinteractiveMode=false + +# +# Give some guidance +# +echo -e "\\n\\n" +echo -e "\\tA sample quickstart Flink Job has been created." +echo -e "\\tSwitch into the directory using" +echo -e "\\t\\t cd $PACKAGE" +echo -e "\\tImport the project there using your favorite IDE (Import it as a maven project)" +echo -e "\\tBuild a jar inside the directory using" +echo -e "\\t\\t mvn clean package" +echo -e "\\tYou will find the runnable jar in $PACKAGE/target" +echo -e "\\tConsult our website if you have any troubles: http://flink.apache.org/community.html#mailing-lists"; +echo -e "\\n\\n" + + Propchange: flink/site/q/quickstart.sh ------------------------------------------------------------------------------ svn:executable = *
