flink git commit: [FLINK-1406] [documentation] update compatibility notice with link to further documentation

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master 8f8efe2f5 - 96585f149


[FLINK-1406] [documentation] update compatibility notice with link to further 
documentation

This closes #314


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/96585f14
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/96585f14
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/96585f14

Branch: refs/heads/master
Commit: 96585f149c8d100f3d8fce2bb737743695f20c12
Parents: 8f8efe2
Author: Max m...@posteo.de
Authored: Fri Jan 16 11:02:47 2015 +0100
Committer: Fabian Hueske fhue...@apache.org
Committed: Fri Jan 23 22:20:37 2015 +0100

--
 docs/setup_quickstart.md | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/96585f14/docs/setup_quickstart.md
--
diff --git a/docs/setup_quickstart.md b/docs/setup_quickstart.md
index d987813..e45792d 100644
--- a/docs/setup_quickstart.md
+++ b/docs/setup_quickstart.md
@@ -26,7 +26,12 @@ under the License.
 Get Flink up and running in a few simple steps.
 
 ## Requirements
-Flink runs on all __UNIX-like__ environments: __Linux__, __Mac OS X__, 
__Cygwin__. The only requirement is to have a working __Java 6.x__ (or higher) 
installation.
+
+Flink runs on __Linux, Mac OS X, and Windows__. To be able to run Flink, the
+only requirement is to have a working __Java 6.x__ (or higher)
+installation. Windows users, please take a look at the
+[Flink on Windows](local_setup.html#flink-on-windows) guide which describes
+how to run Flink on Windows for local setups.
 
 ## Download
 Download the ready to run binary package. Choose the Flink distribution that 
__matches your Hadoop version__. If you are unsure which version to choose or 
you just want to run locally, pick the package for Hadoop 1.2.

[1/4] flink git commit: [FLINK-1318] [docs] Added documentation for CSV parsing options.

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master 5e1cc9e21 - a19b4a02b


[FLINK-1318] [docs] Added documentation for CSV parsing options.


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/27a479fe
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/27a479fe
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/27a479fe

Branch: refs/heads/master
Commit: 27a479fed3a58c2377220a292eda965b78241907
Parents: 2665cf4
Author: Fabian Hueske fhue...@apache.org
Authored: Mon Feb 2 17:39:05 2015 +0100
Committer: Fabian Hueske fhue...@apache.org
Committed: Thu Feb 5 11:17:38 2015 +0100

--
 docs/programming_guide.md | 56 ++
 1 file changed, 56 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/27a479fe/docs/programming_guide.md
--
diff --git a/docs/programming_guide.md b/docs/programming_guide.md
index 6ed6e04..3dcd770 100644
--- a/docs/programming_guide.md
+++ b/docs/programming_guide.md
@@ -1517,29 +1517,37 @@ shortcut methods on the *ExecutionEnvironment*.
 File-based:
 
 - `readTextFile(path)` / `TextInputFormat` - Reads files line wise and returns 
them as Strings.
+
 - `readTextFileWithValue(path)` / `TextValueInputFormat` - Reads files line 
wise and returns them as
   StringValues. StringValues are mutable strings.
+
 - `readCsvFile(path)` / `CsvInputFormat` - Parses files of comma (or another 
char) delimited fields.
   Returns a DataSet of tuples. Supports the basic java types and their Value 
counterparts as field
   types.
+
 - `readFileOfPrimitives(path, Class)` / `PrimitiveInputFormat` - Parses files 
of new-line (or another char sequence) delimited primitive data types such as 
`String` or `Integer`. 
 
 Collection-based:
 
 - `fromCollection(Collection)` - Creates a data set from the Java 
Java.util.Collection. All elements
   in the collection must be of the same type.
+
 - `fromCollection(Iterator, Class)` - Creates a data set from an iterator. The 
class specifies the
   data type of the elements returned by the iterator.
+
 - `fromElements(T ...)` - Creates a data set from the given sequence of 
objects. All objects must be
   of the same type.
+
 - `fromParallelCollection(SplittableIterator, Class)` - Creates a data set 
from an iterator, in
   parallel. The class specifies the data type of the elements returned by the 
iterator.
+
 - `generateSequence(from, to)` - Generates the squence of numbers in the given 
interval, in
   parallel.
 
 Generic:
 
 - `readFile(inputFormat, path)` / `FileInputFormat` - Accepts a file input 
format.
+
 - `createInput(inputFormat)` / `InputFormat` - Accepts a generic input format.
 
 **Examples**
@@ -1586,6 +1594,27 @@ DataSetTuple2String, Integer dbData =
 // manually provide the type information as shown in the examples above.
 {% endhighlight %}
 
+ Configuring CSV Parsing
+
+Flink offers a number of configuration options for CSV parsing:
+
+- `types(Class ... types)` specifies the types of the fields to parse. **It is 
mandatory to configure the types of the parsed fields.**
+
+- `lineDelimiter(String del)` specifies the delimiter of individual records. 
The default line delimiter is the new-line character `'\n'`.
+
+- `fieldDelimiter(String del)` specifies the delimiter that separates fields 
of a record. The default field delimiter is the comma character `','`.
+
+- `includeFields(boolean ... flag)`, `includeFields(String mask)`, or 
`includeFields(long bitMask)` defines which fields to read from the input file 
(and which to ignore). By default the first *n* fields (as defined by the 
number of types in the `types()` call) are parsed.
+
+- `parseQuotedStrings(char quoteChar)` enables quoted string parsing. Strings 
are parsed as quoted strings if the first character of the string field is the 
quote character (leading or tailing whitespaces are *not* trimmed). Field 
delimiters within quoted strings are ignored. Quoted string parsing fails if 
the last character of a quoted string field is not the quote character. If 
quoted string parsing is enabled and the first character of the field is *not* 
the quoting string, the string is parsed as unquoted string. By default, quoted 
string parsing is disabled.
+
+- `ignoreComments(String commentPrefix)` specifies a comment prefix. All lines 
that start with the specified comment prefix are not parsed and ignored. By 
default, no lines are ignored.
+
+- `ignoreInvalidLines()` enables lenient parsing, i.e., lines that cannot be 
correctly parsed are ignored. By default, lenient parsing is disabled and 
invalid lines raise an exception.
+
+- `ignoreFirstLine()` configures the InputFormat to ignore the first line of 
the input file. By default no line is ignored.
+
+
  

[2/4] flink git commit: [FLINK-1318] [api-breaking] Simplified quoted string parsing, made it optional, and use a configurable quote character

[fhueske]

[FLINK-1318] [api-breaking] Simplified quoted string parsing, made it optional, 
and use a configurable quote character

This closes #265


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/2665cf4e
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/2665cf4e
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/2665cf4e

Branch: refs/heads/master
Commit: 2665cf4e2a9e33e0e94ac7e0b7518a10445febbb
Parents: 5e1cc9e
Author: Fabian Hueske fhue...@apache.org
Authored: Mon Oct 20 15:18:20 2014 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Thu Feb 5 11:17:38 2015 +0100

--
 .../api/common/io/GenericCsvInputFormat.java|  91 +++--
 .../apache/flink/types/parser/FieldParser.java  |   4 +-
 .../apache/flink/types/parser/StringParser.java | 130 +++
 .../flink/types/parser/StringValueParser.java   | 102 +++
 .../types/parser/QuotedStringParserTest.java|  61 +
 .../parser/QuotedStringValueParserTest.java |  65 ++
 .../flink/types/parser/StringParserTest.java|  64 -
 .../types/parser/StringValueParserTest.java |  66 --
 .../types/parser/UnquotedStringParserTest.java  |  56 
 .../parser/UnquotedStringValueParserTest.java   |  56 
 .../types/parser/VarLengthStringParserTest.java |  88 ++---
 .../org/apache/flink/api/java/io/CsvReader.java |  25 +++-
 .../flink/api/java/io/CsvInputFormatTest.java   |  47 ++-
 .../flink/api/scala/ExecutionEnvironment.scala  |   6 +
 .../flink/api/scala/io/CsvInputFormatTest.scala |  41 ++
 15 files changed, 538 insertions(+), 364 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/2665cf4e/flink-core/src/main/java/org/apache/flink/api/common/io/GenericCsvInputFormat.java
--
diff --git 
a/flink-core/src/main/java/org/apache/flink/api/common/io/GenericCsvInputFormat.java
 
b/flink-core/src/main/java/org/apache/flink/api/common/io/GenericCsvInputFormat.java
index 151b1e2..662cf1b 100644
--- 
a/flink-core/src/main/java/org/apache/flink/api/common/io/GenericCsvInputFormat.java
+++ 
b/flink-core/src/main/java/org/apache/flink/api/common/io/GenericCsvInputFormat.java
@@ -25,6 +25,8 @@ import com.google.common.primitives.Ints;
 import org.apache.flink.core.fs.FileInputSplit;
 import org.apache.flink.core.fs.Path;
 import org.apache.flink.types.parser.FieldParser;
+import org.apache.flink.types.parser.StringParser;
+import org.apache.flink.types.parser.StringValueParser;
 import org.apache.flink.util.InstantiationUtil;
 
 import java.io.IOException;
@@ -41,15 +43,12 @@ public abstract class GenericCsvInputFormatOT extends 
DelimitedInputFormatOT

private static final byte[] DEFAULT_FIELD_DELIMITER = new byte[] {','};
 
-   private static final char QUOTE_CHARACTER = '';
-   
-   
// 

//  Variables for internal operation.
//  They are all transient, because we do not want them so be 
serialized 
// 

 
-   private transient FieldParserObject[] fieldParsers;
+   private transient FieldParser?[] fieldParsers;


// 

@@ -65,6 +64,10 @@ public abstract class GenericCsvInputFormatOT extends 
DelimitedInputFormatOT
private boolean lenient;

private boolean skipFirstLineAsHeader;
+
+   private boolean quotedStringParsing = false;
+
+   private byte quoteCharacter;


// 

@@ -124,6 +127,11 @@ public abstract class GenericCsvInputFormatOT extends 
DelimitedInputFormatOT
public void setSkipFirstLineAsHeader(boolean skipFirstLine) {
this.skipFirstLineAsHeader = skipFirstLine;
}
+
+   public void enableQuotedStringParsing(char quoteCharacter) {
+   quotedStringParsing = true;
+   this.quoteCharacter = (byte)quoteCharacter;
+   }

// 


@@ -254,7 +262,7 @@ public abstract class GenericCsvInputFormatOT extends 
DelimitedInputFormatOT

// instantiate the parsers
@SuppressWarnings(unchecked)
-   FieldParserObject[] parsers = new 
FieldParser[fieldTypes.length];
+   FieldParser?[] parsers = new FieldParser[fieldTypes.length];

[3/4] flink git commit: [FLINK-1443 [api-breaking] Extended split assigner interface by parallel task id.

[fhueske]

[FLINK-1443 [api-breaking] Extended split assigner interface by parallel task 
id.


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/7452802b
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/7452802b
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/7452802b

Branch: refs/heads/master
Commit: 7452802bc25c0915b7347d4faf2d60adcfc27644
Parents: 27a479f
Author: Fabian Hueske fhue...@apache.org
Authored: Tue Feb 3 15:15:52 2015 +0100
Committer: Fabian Hueske fhue...@apache.org
Committed: Thu Feb 5 11:18:03 2015 +0100

--
 .../common/io/DefaultInputSplitAssigner.java|  2 +-
 .../common/io/LocatableInputSplitAssigner.java  |  2 +-
 .../flink/core/io/InputSplitAssigner.java   |  6 ++-
 .../flink/core/io/DefaultSplitAssignerTest.java |  8 ++--
 .../core/io/LocatableSplitAssignerTest.java | 40 ++--
 .../flink/runtime/jobmanager/JobManager.scala   |  4 +-
 6 files changed, 33 insertions(+), 29 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/7452802b/flink-core/src/main/java/org/apache/flink/api/common/io/DefaultInputSplitAssigner.java
--
diff --git 
a/flink-core/src/main/java/org/apache/flink/api/common/io/DefaultInputSplitAssigner.java
 
b/flink-core/src/main/java/org/apache/flink/api/common/io/DefaultInputSplitAssigner.java
index 379fc4e..d69aa4f 100644
--- 
a/flink-core/src/main/java/org/apache/flink/api/common/io/DefaultInputSplitAssigner.java
+++ 
b/flink-core/src/main/java/org/apache/flink/api/common/io/DefaultInputSplitAssigner.java
@@ -51,7 +51,7 @@ public class DefaultInputSplitAssigner implements 
InputSplitAssigner {


@Override
-   public InputSplit getNextInputSplit(String host) {
+   public InputSplit getNextInputSplit(String host, int taskId) {
InputSplit next = null;

// keep the synchronized part short

http://git-wip-us.apache.org/repos/asf/flink/blob/7452802b/flink-core/src/main/java/org/apache/flink/api/common/io/LocatableInputSplitAssigner.java
--
diff --git 
a/flink-core/src/main/java/org/apache/flink/api/common/io/LocatableInputSplitAssigner.java
 
b/flink-core/src/main/java/org/apache/flink/api/common/io/LocatableInputSplitAssigner.java
index 92fbdca..c038da6 100644
--- 
a/flink-core/src/main/java/org/apache/flink/api/common/io/LocatableInputSplitAssigner.java
+++ 
b/flink-core/src/main/java/org/apache/flink/api/common/io/LocatableInputSplitAssigner.java
@@ -71,7 +71,7 @@ public final class LocatableInputSplitAssigner implements 
InputSplitAssigner {
// 

 
@Override
-   public LocatableInputSplit getNextInputSplit(String host) {
+   public LocatableInputSplit getNextInputSplit(String host, int taskId) {
 
// for a null host, we return a remote split
if (host == null) {

http://git-wip-us.apache.org/repos/asf/flink/blob/7452802b/flink-core/src/main/java/org/apache/flink/core/io/InputSplitAssigner.java
--
diff --git 
a/flink-core/src/main/java/org/apache/flink/core/io/InputSplitAssigner.java 
b/flink-core/src/main/java/org/apache/flink/core/io/InputSplitAssigner.java
index b4f72e9..3999c13 100644
--- a/flink-core/src/main/java/org/apache/flink/core/io/InputSplitAssigner.java
+++ b/flink-core/src/main/java/org/apache/flink/core/io/InputSplitAssigner.java
@@ -28,8 +28,10 @@ public interface InputSplitAssigner {
 * Returns the next input split that shall be consumed. The consumer's 
host is passed as a parameter
 * to allow localized assignments.
 * 
-* @param host The address of the host to assign the split to.
+* @param host The host address of split requesting task.
+* @param taskId The id of the split requesting task.
 * @return the next input split to be consumed, or codenull/code if 
no more splits remain.
 */
-   InputSplit getNextInputSplit(String host);
+   InputSplit getNextInputSplit(String host, int taskId);
+
 }

http://git-wip-us.apache.org/repos/asf/flink/blob/7452802b/flink-core/src/test/java/org/apache/flink/core/io/DefaultSplitAssignerTest.java
--
diff --git 
a/flink-core/src/test/java/org/apache/flink/core/io/DefaultSplitAssignerTest.java
 
b/flink-core/src/test/java/org/apache/flink/core/io/DefaultSplitAssignerTest.java
index d7d7cc8..f92b6af 100644
--- 
a/flink-core/src/test/java/org/apache/flink/core/io/DefaultSplitAssignerTest.java
+++ 

flink git commit: [FLINK-1490][fix][java-api] Fix incorrect local output sorting of nested types with field position keys.

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master a0e71b88a - 6187292bc


[FLINK-1490][fix][java-api] Fix incorrect local output sorting of nested types 
with field position keys.


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/6187292b
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/6187292b
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/6187292b

Branch: refs/heads/master
Commit: 6187292bca47e7a42fe73a3f8fead50d522be060
Parents: a0e71b8
Author: Fabian Hueske fhue...@apache.org
Authored: Fri Feb 6 15:11:25 2015 +0100
Committer: Fabian Hueske fhue...@apache.org
Committed: Fri Feb 6 16:00:50 2015 +0100

--
 .../flink/api/java/operators/DataSink.java  | 25 ---
 .../test/javaApiOperators/DataSinkITCase.java   | 44 
 .../util/CollectionDataSets.java| 21 ++
 3 files changed, 76 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/6187292b/flink-java/src/main/java/org/apache/flink/api/java/operators/DataSink.java
--
diff --git 
a/flink-java/src/main/java/org/apache/flink/api/java/operators/DataSink.java 
b/flink-java/src/main/java/org/apache/flink/api/java/operators/DataSink.java
index 9da5433..e646891 100644
--- a/flink-java/src/main/java/org/apache/flink/api/java/operators/DataSink.java
+++ b/flink-java/src/main/java/org/apache/flink/api/java/operators/DataSink.java
@@ -115,18 +115,33 @@ public class DataSinkT {
throw new InvalidProgramException(Order key out of 
tuple bounds.);
}
 
+   // get flat keys
+   Keys.ExpressionKeysT ek;
+   try {
+   ek = new Keys.ExpressionKeysT(new int[]{field}, 
this.type);
+   } catch(IllegalArgumentException iae) {
+   throw new InvalidProgramException(Invalid 
specification of field expression., iae);
+   }
+   int[] flatKeys = ek.computeLogicalKeyPositions();
+
if(this.sortKeyPositions == null) {
// set sorting info
-   this.sortKeyPositions = new int[] {field};
-   this.sortOrders = new Order[] {order};
+   this.sortKeyPositions = flatKeys;
+   this.sortOrders = new Order[flatKeys.length];
+   Arrays.fill(this.sortOrders, order);
} else {
// append sorting info to exising info
-   int newLength = this.sortKeyPositions.length + 1;
+   int oldLength = this.sortKeyPositions.length;
+   int newLength = oldLength + flatKeys.length;
this.sortKeyPositions = 
Arrays.copyOf(this.sortKeyPositions, newLength);
this.sortOrders = Arrays.copyOf(this.sortOrders, 
newLength);
-   this.sortKeyPositions[newLength-1] = field;
-   this.sortOrders[newLength-1] = order;
+
+   for(int i=0; iflatKeys.length; i++) {
+   this.sortKeyPositions[oldLength+i] = 
flatKeys[i];
+   this.sortOrders[oldLength+i] = order;
+   }
}
+
return this;
}
 

http://git-wip-us.apache.org/repos/asf/flink/blob/6187292b/flink-tests/src/test/java/org/apache/flink/test/javaApiOperators/DataSinkITCase.java
--
diff --git 
a/flink-tests/src/test/java/org/apache/flink/test/javaApiOperators/DataSinkITCase.java
 
b/flink-tests/src/test/java/org/apache/flink/test/javaApiOperators/DataSinkITCase.java
index d8663e8..6bd678f 100644
--- 
a/flink-tests/src/test/java/org/apache/flink/test/javaApiOperators/DataSinkITCase.java
+++ 
b/flink-tests/src/test/java/org/apache/flink/test/javaApiOperators/DataSinkITCase.java
@@ -208,8 +208,8 @@ public class DataSinkITCase extends 
MultipleProgramsTestBase {
 
final ExecutionEnvironment env = 
ExecutionEnvironment.getExecutionEnvironment();
 
-   DataSetTuple2Tuple2Integer, Integer, String ds =
-   
CollectionDataSets.getGroupSortedNestedTupleDataSet(env);
+   DataSetTuple3Tuple2Integer, Integer, String, Integer ds =
+   
CollectionDataSets.getGroupSortedNestedTupleDataSet2(env);
ds.writeAsText(resultPath)
.sortLocalOutput(f0.f1, Order.ASCENDING)
.sortLocalOutput(f1, Order.DESCENDING)
@@ -218,13 +218,39 @@ public class DataSinkITCase extends 
MultipleProgramsTestBase {
   

flink git commit: [FLINK-1485][documentation] Fix typo in Java join function example.

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master 6187292bc - 76dd725d7


[FLINK-1485][documentation] Fix typo in Java join function example.

This closes #369


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/76dd725d
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/76dd725d
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/76dd725d

Branch: refs/heads/master
Commit: 76dd725d7ce1757c778db58448fbac5036e626ef
Parents: 6187292
Author: jkirsch jkirschn...@gmail.com
Authored: Thu Feb 5 21:42:17 2015 +0100
Committer: Fabian Hueske fhue...@apache.org
Committed: Fri Feb 6 16:14:08 2015 +0100

--
 docs/dataset_transformations.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/76dd725d/docs/dataset_transformations.md
--
diff --git a/docs/dataset_transformations.md b/docs/dataset_transformations.md
index 2ef5123..223d24c 100644
--- a/docs/dataset_transformations.md
+++ b/docs/dataset_transformations.md
@@ -749,7 +749,7 @@ DataSetTuple2String, Double
 ratings.join(weights)
 
// key of the first input
-   .where(category)
+   .where(category)
 
// key of the second input
.equalTo(f0)

[1/2] flink git commit: [FLINK-1112] Additional checks for KeySelector group sorting and minor fixes

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master ba7a19c10 - 2b16c6042


[FLINK-1112] Additional checks for KeySelector group sorting and minor fixes


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/2b16c604
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/2b16c604
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/2b16c604

Branch: refs/heads/master
Commit: 2b16c6042227145241046d497f9be0e43242c9fb
Parents: f83db14
Author: Fabian Hueske fhue...@apache.org
Authored: Tue Dec 16 15:38:08 2014 +0100
Committer: Fabian Hueske fhue...@apache.org
Committed: Thu Jan 15 18:13:18 2015 +0100

--
 .../GroupingKeySelectorTranslationTest.java |  35 +---
 .../api/java/operators/SortedGrouping.java  |   2 +-
 .../api/java/operators/UnsortedGrouping.java|  17 +-
 .../apache/flink/api/scala/GroupedDataSet.scala |  19 ++-
 .../javaApiOperators/GroupReduceITCase.java |   2 -
 ...tomPartitioningGroupingKeySelectorTest.scala | 162 +++
 6 files changed, 134 insertions(+), 103 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/2b16c604/flink-compiler/src/test/java/org/apache/flink/compiler/custompartition/GroupingKeySelectorTranslationTest.java
--
diff --git 
a/flink-compiler/src/test/java/org/apache/flink/compiler/custompartition/GroupingKeySelectorTranslationTest.java
 
b/flink-compiler/src/test/java/org/apache/flink/compiler/custompartition/GroupingKeySelectorTranslationTest.java
index 8f446a7..a7b1167 100644
--- 
a/flink-compiler/src/test/java/org/apache/flink/compiler/custompartition/GroupingKeySelectorTranslationTest.java
+++ 
b/flink-compiler/src/test/java/org/apache/flink/compiler/custompartition/GroupingKeySelectorTranslationTest.java
@@ -115,7 +115,7 @@ public class GroupingKeySelectorTranslationTest extends 
CompilerTestBase {

data.groupBy(new 
TestKeySelectorTuple3Integer,Integer,Integer())
.withPartitioner(new TestPartitionerInt())
-   .sortGroup(1, Order.ASCENDING)
+   .sortGroup(new TestKeySelectorTuple3Integer, 
Integer, Integer(), Order.ASCENDING)
.reduceGroup(new 
IdentityGroupReducerTuple3Integer,Integer,Integer())
.print();

@@ -137,39 +137,6 @@ public class GroupingKeySelectorTranslationTest extends 
CompilerTestBase {
}

@Test
-   public void testCustomPartitioningKeySelectorGroupReduceSorted2() {
-   try {
-   ExecutionEnvironment env = 
ExecutionEnvironment.getExecutionEnvironment();
-   
-   DataSetTuple4Integer,Integer,Integer, Integer data 
= env.fromElements(new Tuple4Integer,Integer,Integer,Integer(0, 0, 0, 0))
-   .rebalance().setParallelism(4);
-   
-   data
-   .groupBy(new 
TestKeySelectorTuple4Integer,Integer,Integer,Integer())
-   .withPartitioner(new TestPartitionerInt())
-   .sortGroup(1, Order.ASCENDING)
-   .sortGroup(2, Order.DESCENDING)
-   .reduceGroup(new 
IdentityGroupReducerTuple4Integer,Integer,Integer,Integer())
-   .print();
-   
-   Plan p = env.createProgramPlan();
-   OptimizedPlan op = compileNoStats(p);
-   
-   SinkPlanNode sink = op.getDataSinks().iterator().next();
-   SingleInputPlanNode reducer = (SingleInputPlanNode) 
sink.getInput().getSource();
-   SingleInputPlanNode combiner = (SingleInputPlanNode) 
reducer.getInput().getSource();
-   
-   assertEquals(ShipStrategyType.FORWARD, 
sink.getInput().getShipStrategy());
-   assertEquals(ShipStrategyType.PARTITION_CUSTOM, 
reducer.getInput().getShipStrategy());
-   assertEquals(ShipStrategyType.FORWARD, 
combiner.getInput().getShipStrategy());
-   }
-   catch (Exception e) {
-   e.printStackTrace();
-   fail(e.getMessage());
-   }
-   }
-   
-   @Test
public void testCustomPartitioningKeySelectorInvalidType() {
try {
ExecutionEnvironment env = 
ExecutionEnvironment.getExecutionEnvironment();

flink git commit: [FLINK-1419] [runtime] Fix: distributed cache properly synchronized

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master 31641a68e - 563e54623


[FLINK-1419] [runtime] Fix: distributed cache properly synchronized

This closes #339


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/563e5462
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/563e5462
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/563e5462

Branch: refs/heads/master
Commit: 563e546236217dace58a8031d56d08a27e08160b
Parents: 31641a6
Author: zentol s.mo...@web.de
Authored: Mon Jan 26 11:07:53 2015 +0100
Committer: Fabian Hueske fhue...@apache.org
Committed: Mon Feb 2 00:27:39 2015 +0100

--
 .../flink/runtime/filecache/FileCache.java  | 56 
 1 file changed, 34 insertions(+), 22 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/563e5462/flink-runtime/src/main/java/org/apache/flink/runtime/filecache/FileCache.java
--
diff --git 
a/flink-runtime/src/main/java/org/apache/flink/runtime/filecache/FileCache.java 
b/flink-runtime/src/main/java/org/apache/flink/runtime/filecache/FileCache.java
index de8d59c..b838aa4 100644
--- 
a/flink-runtime/src/main/java/org/apache/flink/runtime/filecache/FileCache.java
+++ 
b/flink-runtime/src/main/java/org/apache/flink/runtime/filecache/FileCache.java
@@ -28,8 +28,6 @@ import java.util.concurrent.FutureTask;
 import java.util.concurrent.ScheduledExecutorService;
 import java.util.concurrent.TimeUnit;
 
-import org.apache.commons.lang3.tuple.ImmutablePair;
-import org.apache.commons.lang3.tuple.Pair;
 import org.apache.flink.api.common.cache.DistributedCache;
 import 
org.apache.flink.api.common.cache.DistributedCache.DistributedCacheEntry;
 import org.apache.flink.configuration.ConfigConstants;
@@ -60,7 +58,7 @@ public class FileCache {

private LocalFileSystem lfs = new LocalFileSystem();
 
-   private MapPairJobID, String, Integer count = new 
HashMapPairJobID,String, Integer();
+   private MapJobID, MapString, Integer jobCounts = new HashMapJobID, 
MapString, Integer();
 
private final ScheduledExecutorService executorService = 
Executors.newScheduledThreadPool(10, ExecutorThreadFactory.INSTANCE);
 
@@ -72,15 +70,18 @@ public class FileCache {
 * @return copy task
 */
public FutureTaskPath createTmpFile(String name, 
DistributedCacheEntry entry, JobID jobID) {
-   synchronized (count) {
-   PairJobID, String key = new ImmutablePairJobID, 
String(jobID, name);
-   if (count.containsKey(key)) {
-   count.put(key, count.get(key) + 1);
+   synchronized (lock) {
+   if (!jobCounts.containsKey(jobID)) {
+   jobCounts.put(jobID, new HashMapString, 
Integer());
+   }
+   MapString, Integer count = jobCounts.get(jobID);
+   if (count.containsKey(name)) {
+   count.put(name, count.get(name) + 1);
} else {
-   count.put(key, 1);
+   count.put(name, 1);
}
}
-   CopyProcess cp = new CopyProcess(name, entry, jobID);
+   CopyProcess cp = new CopyProcess(entry, jobID);
FutureTaskPath copyTask = new FutureTaskPath(cp);
executorService.submit(copyTask);
return copyTask;
@@ -93,7 +94,7 @@ public class FileCache {
 * @param jobID
 */
public void deleteTmpFile(String name, DistributedCacheEntry entry, 
JobID jobID) {
-   DeleteProcess dp = new DeleteProcess(name, entry, jobID, 
count.get(new ImmutablePairJobID, String(jobID,name)));
+   DeleteProcess dp = new DeleteProcess(name, entry, jobID);
executorService.schedule(dp, 5000L, TimeUnit.MILLISECONDS);
}
 
@@ -153,7 +154,7 @@ public class FileCache {
private String filePath;
private Boolean executable;
 
-   public CopyProcess(String name, DistributedCacheEntry e, JobID 
jobID) {
+   public CopyProcess(DistributedCacheEntry e, JobID jobID) {
this.filePath = e.filePath;
this.executable = e.isExecutable;
this.jobID = jobID;
@@ -179,24 +180,35 @@ public class FileCache {

private String name;
private JobID jobID;
-   private int oldCount;
+   private String filePath;
 
-   public DeleteProcess(String name, DistributedCacheEntry e, 
JobID jobID, int c) {
+   public 

[4/4] flink git commit: [FLINK-1443] Added support for replicated data sources. Introduced new PartitioningProperty for any distribution (random partitioning or full replication).

[fhueske]

[FLINK-1443] Added support for replicated data sources.
Introduced new PartitioningProperty for any distribution (random partitioning 
or full replication).


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/a19b4a02
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/a19b4a02
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/a19b4a02

Branch: refs/heads/master
Commit: a19b4a02bfa5237e0dcd2b264da36229546f23c0
Parents: 7452802
Author: Fabian Hueske fhue...@apache.org
Authored: Tue Feb 3 16:03:07 2015 +0100
Committer: Fabian Hueske fhue...@apache.org
Committed: Thu Feb 5 11:18:04 2015 +0100

--
 .../flink/compiler/dag/DataSourceNode.java  |  41 +-
 .../flink/compiler/dag/SingleInputNode.java |  24 +-
 .../apache/flink/compiler/dag/TwoInputNode.java |  33 ++
 .../dataproperties/GlobalProperties.java|  17 +-
 .../dataproperties/PartitioningProperty.java|  11 +-
 .../RequestedGlobalProperties.java  |  35 +-
 .../operators/AbstractJoinDescriptor.java   |   2 +-
 .../operators/AllGroupReduceProperties.java |   2 +-
 .../AllGroupWithPartialPreGroupProperties.java  |   2 +-
 .../operators/CartesianProductDescriptor.java   |   2 +-
 .../compiler/operators/CoGroupDescriptor.java   |   2 +-
 .../operators/CollectorMapDescriptor.java   |   6 +-
 .../compiler/operators/FilterDescriptor.java|   4 +-
 .../compiler/operators/FlatMapDescriptor.java   |   6 +-
 .../operators/GroupReduceProperties.java|   2 +-
 .../GroupReduceWithCombineProperties.java   |   2 +-
 .../flink/compiler/operators/MapDescriptor.java |   4 +-
 .../operators/MapPartitionDescriptor.java   |   4 +-
 .../operators/OperatorDescriptorSingle.java |   8 +-
 .../operators/PartialGroupProperties.java   |   2 +-
 .../compiler/operators/ReduceProperties.java|   2 +-
 .../apache/flink/compiler/DOPChangeTest.java|  10 +-
 .../compiler/ReplicatingDataSourceTest.java | 495 +++
 .../GlobalPropertiesFilteringTest.java  |  12 +-
 .../RequestedGlobalPropertiesFilteringTest.java |   2 +-
 .../api/common/io/ReplicatingInputFormat.java   | 115 +
 .../io/ReplicatingInputSplitAssigner.java   |  79 +++
 .../ReplicatingDataSourceITCase.java| 141 ++
 28 files changed, 1006 insertions(+), 59 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/a19b4a02/flink-compiler/src/main/java/org/apache/flink/compiler/dag/DataSourceNode.java
--
diff --git 
a/flink-compiler/src/main/java/org/apache/flink/compiler/dag/DataSourceNode.java
 
b/flink-compiler/src/main/java/org/apache/flink/compiler/dag/DataSourceNode.java
index 10c77ca..af2a92b 100644
--- 
a/flink-compiler/src/main/java/org/apache/flink/compiler/dag/DataSourceNode.java
+++ 
b/flink-compiler/src/main/java/org/apache/flink/compiler/dag/DataSourceNode.java
@@ -27,6 +27,7 @@ import java.util.Map;
 import org.apache.flink.api.common.io.FileInputFormat;
 import org.apache.flink.api.common.io.InputFormat;
 import org.apache.flink.api.common.io.NonParallelInput;
+import org.apache.flink.api.common.io.ReplicatingInputFormat;
 import org.apache.flink.api.common.io.statistics.BaseStatistics;
 import org.apache.flink.api.common.operators.GenericDataSourceBase;
 import org.apache.flink.api.common.operators.Operator;
@@ -48,6 +49,8 @@ public class DataSourceNode extends OptimizerNode {

private final boolean sequentialInput;
 
+   private final boolean replicatedInput;
+
/**
 * Creates a new DataSourceNode for the given contract.
 * 
@@ -67,6 +70,12 @@ public class DataSourceNode extends OptimizerNode {
} else {
this.sequentialInput = false;
}
+
+   if (pactContract.getUserCodeWrapper().getUserCodeObject() 
instanceof ReplicatingInputFormat) {
+   this.replicatedInput = true;
+   } else {
+   this.replicatedInput = false;
+   }
}
 
/**
@@ -174,17 +183,31 @@ public class DataSourceNode extends OptimizerNode {
if (this.cachedPlans != null) {
return this.cachedPlans;
}
-   
+
SourcePlanNode candidate = new SourcePlanNode(this, DataSource 
(+this.getPactContract().getName()+));
-   candidate.updatePropertiesWithUniqueSets(getUniqueFields());
-   
-   final Costs costs = new Costs();
-   if 
(FileInputFormat.class.isAssignableFrom(getPactContract().getFormatWrapper().getUserCodeClass())
 
-   this.estimatedOutputSize = 0)
-   {
-   

flink git commit: [FLINK-1654] [docs] Fix scala example in programming guide

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master 84e76f4d3 - fd9ca4def


[FLINK-1654] [docs] Fix scala example in programming guide

This closes #478


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/fd9ca4de
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/fd9ca4de
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/fd9ca4de

Branch: refs/heads/master
Commit: fd9ca4defd54fa150d33d042471e381e0a0a1164
Parents: 84e76f4
Author: Chiwan Park chiwanp...@icloud.com
Authored: Wed Mar 11 10:40:17 2015 +0900
Committer: Fabian Hueske fhue...@apache.org
Committed: Wed Mar 11 18:06:42 2015 +0100

--
 docs/programming_guide.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/fd9ca4de/docs/programming_guide.md
--
diff --git a/docs/programming_guide.md b/docs/programming_guide.md
index ee1138c..e1c79ee 100644
--- a/docs/programming_guide.md
+++ b/docs/programming_guide.md
@@ -1431,7 +1431,7 @@ public class WordWithCount {
 /div
 div data-lang=scala markdown=1
 {% highlight scala %}
-class WordWithCount(val word: String, val count: Int) {
+class WordWithCount(var word: String, var count: Int) {
 def this() {
   this(null, -1)
 }

svn commit: r1666595 - /flink/_posts/2015-03-13-peeking-into-Apache-Flinks-Engine-Room.md

[fhueske]

Author: fhueske
Date: Fri Mar 13 22:04:36 2015
New Revision: 1666595

URL: http://svn.apache.org/r1666595
Log:
Fixed typos in join blog post (2)

Modified:
flink/_posts/2015-03-13-peeking-into-Apache-Flinks-Engine-Room.md

Modified: flink/_posts/2015-03-13-peeking-into-Apache-Flinks-Engine-Room.md
URL: 
http://svn.apache.org/viewvc/flink/_posts/2015-03-13-peeking-into-Apache-Flinks-Engine-Room.md?rev=1666595r1=1666594r2=1666595view=diff
==
--- flink/_posts/2015-03-13-peeking-into-Apache-Flinks-Engine-Room.md (original)
+++ flink/_posts/2015-03-13-peeking-into-Apache-Flinks-Engine-Room.md Fri Mar 
13 22:04:36 2015
@@ -8,23 +8,23 @@ categories: news
 ##Peeking into Apache Flink's Engine Room
 Join Processing in Apache Flink
 
-Joins are prevalent operations in many data processing applications. Most data 
processing systems feature APIs that make joining datasets very easy. However, 
the internal algorithms for join processing are much more involved especially 
if large datasets need to be efficiently handled. Therefore, join processing 
serves as a good example to discuss the salient design points and 
implementation details of a data processing system.
+Joins are prevalent operations in many data processing applications. Most data 
processing systems feature APIs that make joining data sets very easy. However, 
the internal algorithms for join processing are much more involved especially 
if large data sets need to be efficiently handled. Therefore, join processing 
serves as a good example to discuss the salient design points and 
implementation details of a data processing system.
 
 In this blog post, we cut through Apache Flink’s layered architecture and 
take a look at its internals with a focus on how it handles joins. 
Specifically, I will
 
-* show how easy it is to join datasets using Flink’s fluent APIs, 
+* show how easy it is to join data sets using Flink’s fluent APIs, 
 * discuss basic distributed join strategies, Flink’s join implementations, 
and its memory management,
 * talk about Flink’s optimizer that automatically chooses join strategies,
-* show some performance numbers for joining datasets of different sizes, and 
finally
-* briefly discuss joining of co-located and pre-sorted datasets.
+* show some performance numbers for joining data sets of different sizes, and 
finally
+* briefly discuss joining of co-located and pre-sorted data sets.
 
 *Disclaimer*: This blog post is exclusively about equi-joins. Whenever I say 
“join” in the following, I actually mean “equi-join”.
 
 ###How do I join with Flink?
 
-Flink provides fluent APIs in Java and Scala to write data flow programs. 
Flink’s APIs are centered around parallel data collections which are called 
datasets. datasets are processed by applying Transformations that compute new 
datasets. Flink’s transformations include Map and Reduce as known from 
MapReduce [[1]](http://research.google.com/archive/mapreduce.html) but also 
operators for joining, co-grouping, and iterative processing. The documentation 
gives an overview of all available transformations 
[[2]](http://ci.apache.org/projects/flink/flink-docs-release-0.8/dataset_transformations.html).
 
+Flink provides fluent APIs in Java and Scala to write data flow programs. 
Flink’s APIs are centered around parallel data collections which are called 
data sets. data sets are processed by applying Transformations that compute new 
data sets. Flink’s transformations include Map and Reduce as known from 
MapReduce [[1]](http://research.google.com/archive/mapreduce.html) but also 
operators for joining, co-grouping, and iterative processing. The documentation 
gives an overview of all available transformations 
[[2]](http://ci.apache.org/projects/flink/flink-docs-release-0.8/dataset_transformations.html).
 
 
-Joining two Scala case class datasets is very easy as the following example 
shows:
+Joining two Scala case class data sets is very easy as the following example 
shows:
 
 ```scala
 // define your data types
@@ -37,7 +37,7 @@ val users: DataSet[User] = ...
 
 // filter the users data set
 val germanUsers = users.filter((u) = u.country.equals(de))
-// join datasets
+// join data sets
 val germanVisits: DataSet[(PageVisit, User)] =
   // equi-join condition (PageVisit.userId = User.id)
  visits.join(germanUsers).where(userId).equalTo(id)
@@ -55,14 +55,14 @@ See the documentation for more details o
 
 ###How does Flink join my data?
 
-Flink uses techniques which are well known from parallel database systems to 
efficiently execute parallel joins. A join operator must establish all pairs of 
elements from its input datasets for which the join condition evaluates to 
true. In a standalone system, the most straight-forward implementation of a 
join is the so-called nested-loop join which builds the full Cartesian product 
and evaluates the join condition for each

svn commit: r1666430 - in /flink: css/main/main.css site/css/main/main.css

[fhueske]

Author: fhueske
Date: Fri Mar 13 12:50:03 2015
New Revision: 1666430

URL: http://svn.apache.org/r1666430
Log:
Fixed bullet points for unordered lists

Modified:
flink/css/main/main.css
flink/site/css/main/main.css

Modified: flink/css/main/main.css
URL: 
http://svn.apache.org/viewvc/flink/css/main/main.css?rev=1666430r1=1666429r2=1666430view=diff
==
--- flink/css/main/main.css (original)
+++ flink/css/main/main.css Fri Mar 13 12:50:03 2015
@@ -15,7 +15,7 @@ body {
font-size: 100%;
 }
 
-ul {
+.af-main-nav ul {
list-style-type: none;
 }
 

Modified: flink/site/css/main/main.css
URL: 
http://svn.apache.org/viewvc/flink/site/css/main/main.css?rev=1666430r1=1666429r2=1666430view=diff
==
--- flink/site/css/main/main.css (original)
+++ flink/site/css/main/main.css Fri Mar 13 12:50:03 2015
@@ -15,7 +15,7 @@ body {
font-size: 100%;
 }
 
-ul {
+.af-main-nav ul {
list-style-type: none;
 }
 

flink git commit: [FLINK-1683] [jobmanager] Fix scheduling preference choice for non-unary execution tasks.

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master 81ebe980a - 07a9a56c1


[FLINK-1683] [jobmanager] Fix scheduling preference choice for non-unary 
execution tasks.

This closes #476


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/07a9a56c
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/07a9a56c
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/07a9a56c

Branch: refs/heads/master
Commit: 07a9a56c1726b4a7aeb3e682b887d77ab1b0e440
Parents: 81ebe98
Author: Fabian Hueske fhue...@apache.org
Authored: Tue Mar 10 19:19:27 2015 +0100
Committer: Fabian Hueske fhue...@apache.org
Committed: Fri Mar 13 14:20:08 2015 +0100

--
 .../runtime/executiongraph/ExecutionVertex.java | 28 
 1 file changed, 23 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/07a9a56c/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionVertex.java
--
diff --git 
a/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionVertex.java
 
b/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionVertex.java
index 41b78f8..794ca21 100644
--- 
a/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionVertex.java
+++ 
b/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionVertex.java
@@ -47,6 +47,7 @@ import java.util.ArrayList;
 import java.util.Collections;
 import java.util.HashSet;
 import java.util.List;
+import java.util.Set;
 import java.util.concurrent.CopyOnWriteArrayList;
 
 import static com.google.common.base.Preconditions.checkElementIndex;
@@ -348,22 +349,39 @@ public class ExecutionVertex implements Serializable {
return Collections.emptySet();
}
else {
-   HashSetInstance locations = new HashSetInstance();
-   
+
+   SetInstance locations = new HashSetInstance();
+   SetInstance inputLocations = new HashSetInstance();
+
+   // go over all inputs
for (int i = 0; i  inputEdges.length; i++) {
+   inputLocations.clear();
ExecutionEdge[] sources = inputEdges[i];
if (sources != null) {
+   // go over all input sources
for (int k = 0; k  sources.length; 
k++) {
+   // look-up assigned slot of 
input source
SimpleSlot sourceSlot = 
sources[k].getSource().getProducer().getCurrentAssignedResource();
if (sourceSlot != null) {
-   
locations.add(sourceSlot.getInstance());
-   if (locations.size()  
MAX_DISTINCT_LOCATIONS_TO_CONSIDER) {
-   return null;
+   // add input location
+   
inputLocations.add(sourceSlot.getInstance());
+   // inputs which have 
too many distinct sources are not considered
+   if 
(inputLocations.size()  MAX_DISTINCT_LOCATIONS_TO_CONSIDER) {
+   
inputLocations.clear();
+   break;
}
}
}
}
+   // keep the locations of the input with the 
least preferred locations
+   if(locations.isEmpty() || // nothing assigned 
yet
+   (!inputLocations.isEmpty()  
inputLocations.size()  locations.size())) {
+   // current input has fewer preferred 
locations
+   locations.clear();
+   locations.addAll(inputLocations);
+   }
}
+
return locations;
}
}

svn commit: r1666431 - in /flink: _posts/ img/blog/

[fhueske]

Author: fhueske
Date: Fri Mar 13 12:51:43 2015
New Revision: 1666431

URL: http://svn.apache.org/r1666431
Log:
Added ready-to-publish Join Blog Post

Added:
flink/_posts/2015-03-13-peeking-into-Apache-Flinks-Engine-Room.md
flink/img/blog/joins-broadcast.png   (with props)
flink/img/blog/joins-dist-perf.png   (with props)
flink/img/blog/joins-hhj.png   (with props)
flink/img/blog/joins-memmgmt.png   (with props)
flink/img/blog/joins-repartition.png   (with props)
flink/img/blog/joins-single-perf.png   (with props)
flink/img/blog/joins-smj.png   (with props)

Added: flink/_posts/2015-03-13-peeking-into-Apache-Flinks-Engine-Room.md
URL: 
http://svn.apache.org/viewvc/flink/_posts/2015-03-13-peeking-into-Apache-Flinks-Engine-Room.md?rev=1666431view=auto
==
--- flink/_posts/2015-03-13-peeking-into-Apache-Flinks-Engine-Room.md (added)
+++ flink/_posts/2015-03-13-peeking-into-Apache-Flinks-Engine-Room.md Fri Mar 
13 12:51:43 2015
@@ -0,0 +1,179 @@
+---
+layout: post
+title:  'Peeking into Apache Flinks Engine Room'
+date:   2015-03-13 10:00:00
+categories: news
+---
+
+##Peeking into Apache Flink's Engine Room
+Join Processing in Apache Flink
+
+Joins are prevalent operations in many data processing applications. Most data 
processing systems feature APIs that make joining datasets very easy. However, 
the internal algorithms for join processing are much more involved especially 
if large datasets need to be efficiently handled. Therefore, join processing 
serves as a good example to discuss the salient design points and 
implementation details of a data processing system.
+
+In this blog post, we cut through Apache Flink’s layered architecture and 
take a look at its internals with a focus on how it handles joins. 
Specifically, I will
+
+* show how easy it is to join datasets using Flink’s fluent APIs, 
+* discuss basic distributed join strategies, Flink’s join implementations, 
and its memory management,
+* talk about Flink’s optimizer that automatically chooses join strategies,
+* show some performance numbers for joining datasets of different sizes, and 
finally
+* briefly discuss joining of co-located and pre-sorted datasets.
+
+*Disclaimer*: This blog post is exclusively about equi-joins. Whenever I say 
“join” in the following, I actually mean “equi-join”.
+
+###How do I join with Flink?
+
+Flink provides fluent APIs in Java and Scala to write data flow programs. 
Flink’s APIs are centered around parallel data collections which are called 
datasets. datasets are processed by applying Transformations that compute new 
datasets. Flink’s transformations include Map and Reduce as known from 
MapReduce [[1]](http://research.google.com/archive/mapreduce.html) but also 
operators for joining, co-grouping, and iterative processing. The documentation 
gives an overview of all available transformations 
[[2]](http://ci.apache.org/projects/flink/flink-docs-release-0.8/dataset_transformations.html).
 
+
+Joining two Scala case class datasets is very easy as the following example 
shows:
+
+```scala
+// define your data types
+case class PageVisit(url: String, ip: String, userId: Long)
+case class User(id: Long, name: String, email: String, country: String)
+
+// get your data from somewhere
+val visits: DataSet[PageVisit] = ...
+val users: DataSet[User] = ...
+
+// filter the users data set
+val germanUsers = users.filter((u) = u.country.equals(de))
+// join datasets
+val germanVisits: DataSet[(PageVisit, User)] =
+  // equi-join condition (PageVisit.userId = User.id)
+ visits.join(germanUsers).where(userId).equalTo(id)
+
+```
+
+Flink’s APIs also allow to:
+
+* apply a user-defined join function to each pair of joined elements instead 
returning a `($Left, $Right)` tuple,
+* select fields of pairs of joined Tuple elements (projection), and
+* define composite join keys such as `.where(“orderDate”, 
“zipCode”).equalTo(“date”, “zip”)`.
+
+See the documentation for more details on Flink’s join features 
[[3]](http://ci.apache.org/projects/flink/flink-docs-release-0.8/dataset_transformations.html#join).
+
+
+###How does Flink join my data?
+
+Flink uses techniques which are well known from parallel database systems to 
efficiently execute parallel joins. A join operator must establish all pairs of 
elements from its input datasets for which the join condition evaluates to 
true. In a standalone system, the most straight-forward implementation of a 
join is the so-called nested-loop join which builds the full Cartesian product 
and evaluates the join condition for each pair of elements. This strategy has 
quadratic complexity and does obviously not scale to large inputs.
+
+In a distributed system joins are commonly processed in two steps:
+
+1. The data of both input is distributed across all parallel instances that 
participate in the join and
+1. each parallel

flink git commit: [FLINK-1683] [jobmanager] Fix scheduling preference choice for non-unary execution tasks.

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/release-0.8 9f18cbb3a - 095301e02


[FLINK-1683] [jobmanager] Fix scheduling preference choice for non-unary 
execution tasks.


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/095301e0
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/095301e0
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/095301e0

Branch: refs/heads/release-0.8
Commit: 095301e025a87608f3b8785d02580da7f87f1e76
Parents: 9f18cbb
Author: Fabian Hueske fhue...@apache.org
Authored: Tue Mar 10 19:19:27 2015 +0100
Committer: Fabian Hueske fhue...@apache.org
Committed: Wed Mar 11 17:49:31 2015 +0100

--
 .../runtime/executiongraph/ExecutionVertex.java | 31 
 1 file changed, 25 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/095301e0/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionVertex.java
--
diff --git 
a/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionVertex.java
 
b/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionVertex.java
index 57e441e..5240845 100644
--- 
a/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionVertex.java
+++ 
b/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionVertex.java
@@ -23,7 +23,6 @@ import static 
org.apache.flink.runtime.execution.ExecutionState.FAILED;
 import static org.apache.flink.runtime.execution.ExecutionState.FINISHED;
 
 import java.util.ArrayList;
-import java.util.HashSet;
 import java.util.List;
 import java.util.concurrent.CopyOnWriteArrayList;
 
@@ -44,6 +43,9 @@ import 
org.apache.flink.runtime.jobmanager.scheduler.CoLocationGroup;
 import org.apache.flink.runtime.jobmanager.scheduler.Scheduler;
 import 
org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException;
 
+import java.util.HashSet;
+import java.util.Set;
+
 /**
  * The ExecutionVertex is a parallel subtask of the execution. It may be 
executed once, or several times, each of
  * which time it spawns an {@link Execution}.
@@ -285,22 +287,39 @@ public class ExecutionVertex {
 * @return The preferred locations for this vertex execution, or null, 
if there is no preference.
 */
public IterableInstance getPreferredLocations() {
-   HashSetInstance locations = new HashSetInstance();
-   
+
+   SetInstance locations = new HashSetInstance();
+   SetInstance inputLocations = new HashSetInstance();
+
+   // go over all inputs
for (int i = 0; i  inputEdges.length; i++) {
+   inputLocations.clear();
ExecutionEdge[] sources = inputEdges[i];
if (sources != null) {
+   // go over all input sources
for (int k = 0; k  sources.length; k++) {
+   // look-up assigned slot of input source
SimpleSlot sourceSlot = 
sources[k].getSource().getProducer().getCurrentAssignedResource();
if (sourceSlot != null) {
-   
locations.add(sourceSlot.getInstance());
-   if (locations.size()  
MAX_DISTINCT_LOCATIONS_TO_CONSIDER) {
-   return null;
+   // add input location
+   
inputLocations.add(sourceSlot.getInstance());
+   // inputs which have too many 
distinct sources are not considered
+   if (inputLocations.size()  
MAX_DISTINCT_LOCATIONS_TO_CONSIDER) {
+   inputLocations.clear();
+   break;
}
}
}
}
+   // keep the locations of the input with the least 
preferred locations
+   if(locations.isEmpty() || // nothing assigned yet
+   (!inputLocations.isEmpty()  
inputLocations.size()  locations.size())) {
+   // current input has fewer preferred locations
+   locations.clear();
+   locations.addAll(inputLocations);
+   }
}
+
return 

flink git commit: [FLINK-1628] [optimizer] Fix partitioning properties for Joins and CoGroups.

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/release-0.8 d573926ab - 88c7ea256


[FLINK-1628] [optimizer] Fix partitioning properties for Joins and CoGroups.


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/88c7ea25
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/88c7ea25
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/88c7ea25

Branch: refs/heads/release-0.8
Commit: 88c7ea256e263dec52fbcbbd1e681ae088075edb
Parents: d573926
Author: Fabian Hueske fhue...@apache.org
Authored: Wed Mar 4 18:49:22 2015 +0100
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue Mar 10 08:52:47 2015 +0100

--
 .../dataproperties/GlobalProperties.java|   8 +
 .../RequestedGlobalProperties.java  |  48 +-
 .../operators/AbstractJoinDescriptor.java   |  49 +-
 .../compiler/operators/CoGroupDescriptor.java   |  72 +-
 .../operators/OperatorDescriptorDual.java   |  44 +-
 .../operators/SortMergeJoinDescriptor.java  |  13 +-
 .../compiler/FeedbackPropertiesMatchTest.java   |   8 +-
 .../flink/compiler/PartitioningReusageTest.java | 859 +++
 .../GlobalPropertiesMatchingTest.java   | 152 +++-
 .../api/common/operators/util/FieldList.java|  15 +-
 10 files changed, 1214 insertions(+), 54 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/88c7ea25/flink-compiler/src/main/java/org/apache/flink/compiler/dataproperties/GlobalProperties.java
--
diff --git 
a/flink-compiler/src/main/java/org/apache/flink/compiler/dataproperties/GlobalProperties.java
 
b/flink-compiler/src/main/java/org/apache/flink/compiler/dataproperties/GlobalProperties.java
index 7dedc53..001d847 100644
--- 
a/flink-compiler/src/main/java/org/apache/flink/compiler/dataproperties/GlobalProperties.java
+++ 
b/flink-compiler/src/main/java/org/apache/flink/compiler/dataproperties/GlobalProperties.java
@@ -180,6 +180,14 @@ public class GlobalProperties implements Cloneable {
return false;
}
}
+
+   public boolean isExactlyPartitionedOnFields(FieldList fields) {
+   if (this.partitioning.isPartitionedOnKey()  
fields.isExactMatch(this.partitioningFields)) {
+   return true;
+   } else {
+   return false;
+   }
+   }

public boolean matchesOrderedPartitioning(Ordering o) {
if (this.partitioning == 
PartitioningProperty.RANGE_PARTITIONED) {

http://git-wip-us.apache.org/repos/asf/flink/blob/88c7ea25/flink-compiler/src/main/java/org/apache/flink/compiler/dataproperties/RequestedGlobalProperties.java
--
diff --git 
a/flink-compiler/src/main/java/org/apache/flink/compiler/dataproperties/RequestedGlobalProperties.java
 
b/flink-compiler/src/main/java/org/apache/flink/compiler/dataproperties/RequestedGlobalProperties.java
index 4e9d60a..33c8476 100644
--- 
a/flink-compiler/src/main/java/org/apache/flink/compiler/dataproperties/RequestedGlobalProperties.java
+++ 
b/flink-compiler/src/main/java/org/apache/flink/compiler/dataproperties/RequestedGlobalProperties.java
@@ -21,6 +21,7 @@ package org.apache.flink.compiler.dataproperties;
 import org.apache.flink.api.common.distributions.DataDistribution;
 import org.apache.flink.api.common.functions.Partitioner;
 import org.apache.flink.api.common.operators.Ordering;
+import org.apache.flink.api.common.operators.util.FieldList;
 import org.apache.flink.api.common.operators.util.FieldSet;
 import org.apache.flink.compiler.CompilerException;
 import org.apache.flink.compiler.dag.OptimizerNode;
@@ -46,7 +47,7 @@ public final class RequestedGlobalProperties implements 
Cloneable {
private DataDistribution dataDistribution;  // optional data 
distribution, for a range partitioning

private Partitioner? customPartitioner;   // optional, 
partitioner for custom partitioning
-   
+
// 


/**
@@ -60,7 +61,9 @@ public final class RequestedGlobalProperties implements 
Cloneable {

/**
 * Sets the partitioning property for the global properties.
-* 
+* If the partitionFields are provided as {@link FieldSet} also subsets 
are valid,
+* if provided as {@link FieldList} partitioning fields must exactly 
match incl. order.
+*
 * @param partitionedFields
 */
public void setHashPartitioned(FieldSet partitionedFields) {
@@ -86,7 +89,14 @@ public final class RequestedGlobalProperties implements 
Cloneable {

[1/5] flink git commit: [FLINK-1512] [tests] Add integration tests for CsvReader

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master 033c69f94 - 1b42b6206


[FLINK-1512] [tests] Add integration tests for CsvReader

This closes #426


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/43ac967a
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/43ac967a
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/43ac967a

Branch: refs/heads/master
Commit: 43ac967acb589790f5b3befd6f932e325d4ba681
Parents: 7a6f296
Author: Chiwan Park chiwanp...@icloud.com
Authored: Wed Mar 25 15:22:43 2015 +0100
Committer: Fabian Hueske fhue...@apache.org
Committed: Wed Mar 25 20:38:59 2015 +0100

--
 .../flink/api/scala/io/CsvInputFormatTest.scala | 68 +---
 1 file changed, 32 insertions(+), 36 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/43ac967a/flink-tests/src/test/scala/org/apache/flink/api/scala/io/CsvInputFormatTest.scala
--
diff --git 
a/flink-tests/src/test/scala/org/apache/flink/api/scala/io/CsvInputFormatTest.scala
 
b/flink-tests/src/test/scala/org/apache/flink/api/scala/io/CsvInputFormatTest.scala
index 4bcd35a..0d74515 100644
--- 
a/flink-tests/src/test/scala/org/apache/flink/api/scala/io/CsvInputFormatTest.scala
+++ 
b/flink-tests/src/test/scala/org/apache/flink/api/scala/io/CsvInputFormatTest.scala
@@ -413,6 +413,31 @@ class CsvInputFormatTest {
 
   case class CaseClassItem(field1: Int, field2: String, field3: Double)
 
+  private def validatePOJOItem(format: ScalaCsvInputFormat[POJOItem]): Unit = {
+var result = new POJOItem()
+result = format.nextRecord(result)
+assertEquals(123, result.field1)
+assertEquals(HELLO, result.field2)
+assertEquals(3.123, result.field3, 0.001)
+
+result = format.nextRecord(result)
+assertEquals(456, result.field1)
+assertEquals(ABC, result.field2)
+assertEquals(1.234, result.field3, 0.001)
+  }
+
+  private def validateCaseClassItem(format: 
ScalaCsvInputFormat[CaseClassItem]): Unit = {
+var result = format.nextRecord(null)
+assertEquals(123, result.field1)
+assertEquals(HELLO, result.field2)
+assertEquals(3.123, result.field3, 0.001)
+
+result = format.nextRecord(null)
+assertEquals(456, result.field1)
+assertEquals(ABC, result.field2)
+assertEquals(1.234, result.field3, 0.001)
+  }
+
   @Test
   def testPOJOType(): Unit = {
 val fileContent = 123,HELLO,3.123\n + 456,ABC,1.234
@@ -425,16 +450,7 @@ class CsvInputFormatTest {
 format.configure(new Configuration)
 format.open(tempFile)
 
-var result = new POJOItem()
-result = format.nextRecord(result)
-assertEquals(123, result.field1)
-assertEquals(HELLO, result.field2)
-assertEquals(3.123, result.field3, 0.001)
-
-result = format.nextRecord(result)
-assertEquals(456, result.field1)
-assertEquals(ABC, result.field2)
-assertEquals(1.234, result.field3, 0.001)
+validatePOJOItem(format)
   }
 
   @Test
@@ -449,15 +465,7 @@ class CsvInputFormatTest {
 format.configure(new Configuration)
 format.open(tempFile)
 
-var result = format.nextRecord(null)
-assertEquals(123, result.field1)
-assertEquals(HELLO, result.field2)
-assertEquals(3.123, result.field3, 0.001)
-
-result = format.nextRecord(null)
-assertEquals(456, result.field1)
-assertEquals(ABC, result.field2)
-assertEquals(1.234, result.field3, 0.001)
+validateCaseClassItem(format)
   }
 
   @Test
@@ -474,36 +482,24 @@ class CsvInputFormatTest {
 format.configure(new Configuration)
 format.open(tempFile)
 
-var result = new POJOItem()
-result = format.nextRecord(result)
-assertEquals(123, result.field1)
-assertEquals(HELLO, result.field2)
-assertEquals(3.123, result.field3, 0.001)
-
-result = format.nextRecord(result)
-assertEquals(456, result.field1)
-assertEquals(ABC, result.field2)
-assertEquals(1.234, result.field3, 0.001)
+validatePOJOItem(format)
   }
   
   @Test
   def testPOJOTypeWithFieldSubsetAndDataSubset(): Unit = {
-val fileContent = 123,HELLO,3.123\n + 456,ABC,1.234
+val fileContent = HELLO,123,NODATA,3.123,NODATA\n + 
ABC,456,NODATA,1.234,NODATA
 val tempFile = createTempFile(fileContent)
 val typeInfo: TypeInformation[POJOItem] = createTypeInformation[POJOItem]
 val format = new ScalaCsvInputFormat[POJOItem](PATH, typeInfo)
 
 format.setDelimiter('\n')
 format.setFieldDelimiter(',')
-format.setFields(Array(false, true), Array(classOf[String]): 
Array[Class[_]])
+format.setFields(Array(true, true, false, true, false),
+  Array(classOf[String], classOf[Integer], classOf[java.lang.Double]): 
Array[Class[_]])
 format.setOrderOfPOJOFields(Array(field2, field1, field3))
 

[4/4] flink git commit: [FLINK-1466] Add support for complex types in Flink tuples for HCatInputFormats.

[fhueske]

[FLINK-1466] Add support for complex types in Flink tuples for HCatInputFormats.

This closes #411


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/bed3da4a
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/bed3da4a
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/bed3da4a

Branch: refs/heads/master
Commit: bed3da4a61a8637c0faa9632b8a05ccef8c5a6dc
Parents: a6acd2e
Author: Fabian Hueske fhue...@apache.org
Authored: Wed Feb 18 16:52:07 2015 +0100
Committer: Fabian Hueske fhue...@apache.org
Committed: Fri Feb 20 16:10:35 2015 +0100

--
 .../flink/hcatalog/HCatInputFormatBase.java | 29 +---
 .../flink/hcatalog/java/HCatInputFormat.java| 24 ++--
 .../flink/hcatalog/scala/HCatInputFormat.scala  | 27 +++---
 3 files changed, 58 insertions(+), 22 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/bed3da4a/flink-staging/flink-hcatalog/src/main/java/org/apache/flink/hcatalog/HCatInputFormatBase.java
--
diff --git 
a/flink-staging/flink-hcatalog/src/main/java/org/apache/flink/hcatalog/HCatInputFormatBase.java
 
b/flink-staging/flink-hcatalog/src/main/java/org/apache/flink/hcatalog/HCatInputFormatBase.java
index 59a6719..f23ac96 100644
--- 
a/flink-staging/flink-hcatalog/src/main/java/org/apache/flink/hcatalog/HCatInputFormatBase.java
+++ 
b/flink-staging/flink-hcatalog/src/main/java/org/apache/flink/hcatalog/HCatInputFormatBase.java
@@ -26,6 +26,7 @@ import 
org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
 import org.apache.flink.api.java.hadoop.mapreduce.utils.HadoopUtils;
 import org.apache.flink.api.java.hadoop.mapreduce.wrapper.HadoopInputSplit;
+import org.apache.flink.api.java.typeutils.GenericTypeInfo;
 import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
 import org.apache.flink.api.java.typeutils.TupleTypeInfo;
 import org.apache.flink.api.java.typeutils.WritableTypeInfo;
@@ -50,14 +51,15 @@ import java.io.ObjectInputStream;
 import java.io.ObjectOutputStream;
 import java.util.ArrayList;
 import java.util.List;
+import java.util.Map;
 
 /**
  * A InputFormat to read from HCatalog tables.
  * The InputFormat supports projection (selection and order of fields) and 
partition filters.
  *
- * Data can be returned as {@link org.apache.hive.hcatalog.data.HCatRecord} or 
Flink {@link org.apache.flink.api.java.tuple.Tuple}.
- * Flink Tuples are only supported for primitive type fields
- * (no STRUCT, ARRAY, or MAP data types) and have a size limitation.
+ * Data can be returned as {@link org.apache.hive.hcatalog.data.HCatRecord} or 
Flink-native tuple.
+ *
+ * Note: Flink tuples might only support a limited number of fields (depending 
on the API).
  *
  * @param T
  */
@@ -82,7 +84,7 @@ public abstract class HCatInputFormatBaseT implements 
InputFormatT, HadoopInp
/**
 * Creates a HCatInputFormat for the given database and table.
 * By default, the InputFormat returns {@link 
org.apache.hive.hcatalog.data.HCatRecord}.
-* The return type of the InputFormat can be changed to Flink {@link 
org.apache.flink.api.java.tuple.Tuple} by calling
+* The return type of the InputFormat can be changed to Flink-native 
tuples by calling
 * {@link HCatInputFormatBase#asFlinkTuples()}.
 *
 * @param database The name of the database to read from.
@@ -97,7 +99,7 @@ public abstract class HCatInputFormatBaseT implements 
InputFormatT, HadoopInp
 * Creates a HCatInputFormat for the given database, table, and
 * {@link org.apache.hadoop.conf.Configuration}.
 * By default, the InputFormat returns {@link 
org.apache.hive.hcatalog.data.HCatRecord}.
-* The return type of the InputFormat can be changed to Flink {@link 
org.apache.flink.api.java.tuple.Tuple} by calling
+* The return type of the InputFormat can be changed to Flink-native 
tuples by calling
 * {@link HCatInputFormatBase#asFlinkTuples()}.
 *
 * @param database The name of the database to read from.
@@ -159,15 +161,10 @@ public abstract class HCatInputFormatBaseT implements 
InputFormatT, HadoopInp
}
 
/**
-* Specifies that the InputFormat returns Flink {@link 
org.apache.flink.api.java.tuple.Tuple}
-* instead of {@link org.apache.hive.hcatalog.data.HCatRecord}.
-* At the moment, the following restrictions apply for returning Flink 
tuples:
+* Specifies that the InputFormat returns Flink tuples instead of
+* {@link org.apache.hive.hcatalog.data.HCatRecord}.
 *
-* ul
-* liOnly primitive type fields can be returned in Flink Tuples
-

flink git commit: Add missing import.

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master d11e09108 - 659ddc048


Add missing import.


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/659ddc04
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/659ddc04
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/659ddc04

Branch: refs/heads/master
Commit: 659ddc0488fc0d3578c994dbc246da4b75389b8d
Parents: d11e091
Author: Fabian Hueske fhue...@apache.org
Authored: Sat Mar 28 00:04:16 2015 +0100
Committer: Fabian Hueske fhue...@apache.org
Committed: Sat Mar 28 00:04:16 2015 +0100

--
 .../scala/org/apache/flink/runtime/taskmanager/TaskManager.scala   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/659ddc04/flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala
--
diff --git 
a/flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala
 
b/flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala
index a9bebca..f552717 100644
--- 
a/flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala
+++ 
b/flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala
@@ -20,7 +20,7 @@ package org.apache.flink.runtime.taskmanager
 import java.io.{File, IOException}
 import java.net.{InetAddress, InetSocketAddress}
 import java.util
-import java.util.concurrent.FutureTask
+import java.util.concurrent.{TimeUnit, FutureTask}
 import management.{GarbageCollectorMXBean, ManagementFactory, MemoryMXBean}
 
 import akka.actor._

[2/3] flink git commit: [FLINK-703] [java api] Use complete element as join key

[fhueske]

[FLINK-703] [java api] Use complete element as join key


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/30a74c76
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/30a74c76
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/30a74c76

Branch: refs/heads/master
Commit: 30a74c768b88043c173053f5d37bfb0fde85f149
Parents: e1618e2
Author: Chiwan Park chiwanp...@icloud.com
Authored: Mon Apr 6 03:18:23 2015 +0900
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue Apr 21 19:03:15 2015 +0200

--
 .../operators/base/CoGroupOperatorBase.java | 10 ++-
 .../base/GroupCombineOperatorBase.java  | 42 +
 .../operators/base/GroupReduceOperatorBase.java | 41 +
 .../apache/flink/api/java/operators/Keys.java   | 45 ++
 .../api/java/operator/CoGroupOperatorTest.java  | 94 +---
 .../flink/api/java/operator/GroupingTest.java   | 34 ++-
 .../api/java/operator/JoinOperatorTest.java | 72 +++
 .../optimizer/postpass/JavaApiPostPass.java |  5 --
 .../test/javaApiOperators/CoGroupITCase.java| 82 -
 .../javaApiOperators/GroupReduceITCase.java | 38 ++--
 .../flink/test/javaApiOperators/JoinITCase.java | 31 +++
 11 files changed, 406 insertions(+), 88 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/30a74c76/flink-core/src/main/java/org/apache/flink/api/common/operators/base/CoGroupOperatorBase.java
--
diff --git 
a/flink-core/src/main/java/org/apache/flink/api/common/operators/base/CoGroupOperatorBase.java
 
b/flink-core/src/main/java/org/apache/flink/api/common/operators/base/CoGroupOperatorBase.java
index dbebeb4..7be5650 100644
--- 
a/flink-core/src/main/java/org/apache/flink/api/common/operators/base/CoGroupOperatorBase.java
+++ 
b/flink-core/src/main/java/org/apache/flink/api/common/operators/base/CoGroupOperatorBase.java
@@ -32,6 +32,7 @@ import 
org.apache.flink.api.common.operators.util.ListKeyGroupedIterator;
 import org.apache.flink.api.common.operators.util.UserCodeClassWrapper;
 import org.apache.flink.api.common.operators.util.UserCodeObjectWrapper;
 import org.apache.flink.api.common.operators.util.UserCodeWrapper;
+import org.apache.flink.api.common.typeinfo.AtomicType;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
 import org.apache.flink.api.common.typeutils.CompositeType;
 import org.apache.flink.api.common.typeutils.GenericPairComparator;
@@ -273,12 +274,15 @@ public class CoGroupOperatorBaseIN1, IN2, OUT, FT 
extends CoGroupFunctionIN1,
return result;
}
 
+   @SuppressWarnings(unchecked)
private T TypeComparatorT getTypeComparator(ExecutionConfig 
executionConfig, TypeInformationT inputType, int[] inputKeys, boolean[] 
inputSortDirections) {
-   if (!(inputType instanceof CompositeType)) {
-   throw new InvalidProgramException(Input types of 
coGroup must be composite types.);
+   if (inputType instanceof CompositeType) {
+   return ((CompositeTypeT) 
inputType).createComparator(inputKeys, inputSortDirections, 0, executionConfig);
+   } else if (inputType instanceof AtomicType) {
+   return ((AtomicTypeT) 
inputType).createComparator(inputSortDirections[0], executionConfig);
}
 
-   return ((CompositeTypeT) 
inputType).createComparator(inputKeys, inputSortDirections, 0, executionConfig);
+   throw new InvalidProgramException(Input type of coGroup must 
be one of composite types or atomic types.);
}
 
private static class CoGroupSortListIteratorIN1, IN2 {

http://git-wip-us.apache.org/repos/asf/flink/blob/30a74c76/flink-core/src/main/java/org/apache/flink/api/common/operators/base/GroupCombineOperatorBase.java
--
diff --git 
a/flink-core/src/main/java/org/apache/flink/api/common/operators/base/GroupCombineOperatorBase.java
 
b/flink-core/src/main/java/org/apache/flink/api/common/operators/base/GroupCombineOperatorBase.java
index 27fbc1c..c7ba92b 100644
--- 
a/flink-core/src/main/java/org/apache/flink/api/common/operators/base/GroupCombineOperatorBase.java
+++ 
b/flink-core/src/main/java/org/apache/flink/api/common/operators/base/GroupCombineOperatorBase.java
@@ -31,6 +31,7 @@ import 
org.apache.flink.api.common.operators.SingleInputOperator;
 import org.apache.flink.api.common.operators.UnaryOperatorInformation;
 import org.apache.flink.api.common.operators.util.ListKeyGroupedIterator;
 import org.apache.flink.api.common.operators.util.UserCodeObjectWrapper;
+import org.apache.flink.api.common.typeinfo.AtomicType;
 import 

[1/3] flink git commit: [FLINK-703] [scala api] Use complete element as join key

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master e1618e287 - a0147c493


[FLINK-703] [scala api] Use complete element as join key

This closes #572


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/45e680c2
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/45e680c2
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/45e680c2

Branch: refs/heads/master
Commit: 45e680c2b6c9c2f64ce55423b755a13d402ff8ba
Parents: 30a74c7
Author: Chiwan Park chiwanp...@icloud.com
Authored: Mon Apr 6 05:07:11 2015 +0900
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue Apr 21 19:03:15 2015 +0200

--
 .../apache/flink/api/java/operators/Keys.java   |  4 +-
 .../org/apache/flink/api/scala/DataSet.scala|  1 -
 .../api/scala/unfinishedKeyPairOperation.scala  |  5 --
 .../api/scala/operators/CoGroupITCase.scala | 42 +++
 .../scala/operators/CoGroupOperatorTest.scala   | 57 +++
 .../api/scala/operators/GroupReduceITCase.scala |  9 +++
 .../api/scala/operators/GroupingTest.scala  | 38 +-
 .../flink/api/scala/operators/JoinITCase.scala  | 22 ++
 .../api/scala/operators/JoinOperatorTest.scala  | 74 ++--
 9 files changed, 236 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/45e680c2/flink-java/src/main/java/org/apache/flink/api/java/operators/Keys.java
--
diff --git 
a/flink-java/src/main/java/org/apache/flink/api/java/operators/Keys.java 
b/flink-java/src/main/java/org/apache/flink/api/java/operators/Keys.java
index ee233e8..69d306f 100644
--- a/flink-java/src/main/java/org/apache/flink/api/java/operators/Keys.java
+++ b/flink-java/src/main/java/org/apache/flink/api/java/operators/Keys.java
@@ -280,7 +280,7 @@ public abstract class KeysT {
if (!type.isKeyType()) {
throw new InvalidProgramException(This 
type ( + type + ) cannot be used as key.);
} else if (expressionsIn.length != 1 || 
!(Keys.ExpressionKeys.SELECT_ALL_CHAR.equals(expressionsIn[0]) || 
Keys.ExpressionKeys.SELECT_ALL_CHAR_SCALA.equals(expressionsIn[0]))) {
-   throw new 
IllegalArgumentException(Field expression for atomic type must be equal to '*' 
or '_'.);
+   throw new 
InvalidProgramException(Field expression for atomic type must be equal to '*' 
or '_'.);
}
 
keyFields = new 
ArrayListFlatFieldDescriptor(1);
@@ -297,7 +297,7 @@ public abstract class KeysT {
for (int i = 0; i  expressions.length; i++) {
ListFlatFieldDescriptor keys = 
cType.getFlatFields(expressions[i]); // use separate list to do a size check
if(keys.size() == 0) {
-   throw new 
IllegalArgumentException(Unable to extract key from expression 
'+expressions[i]+' on key +cType);
+   throw new 
InvalidProgramException(Unable to extract key from expression 
'+expressions[i]+' on key +cType);
}
keyFields.addAll(keys);
}

http://git-wip-us.apache.org/repos/asf/flink/blob/45e680c2/flink-scala/src/main/scala/org/apache/flink/api/scala/DataSet.scala
--
diff --git 
a/flink-scala/src/main/scala/org/apache/flink/api/scala/DataSet.scala 
b/flink-scala/src/main/scala/org/apache/flink/api/scala/DataSet.scala
index 3b80a23..5676229 100644
--- a/flink-scala/src/main/scala/org/apache/flink/api/scala/DataSet.scala
+++ b/flink-scala/src/main/scala/org/apache/flink/api/scala/DataSet.scala
@@ -807,7 +807,6 @@ class DataSet[T: ClassTag](set: JavaDataSet[T]) {
* This will not create a new DataSet, it will just attach the field names 
which will be
* used for grouping when executing a grouped operation.
*
-   * This only works on CaseClass DataSets.
*/
   def groupBy(firstField: String, otherFields: String*): GroupedDataSet[T] = {
 new GroupedDataSet[T](

http://git-wip-us.apache.org/repos/asf/flink/blob/45e680c2/flink-scala/src/main/scala/org/apache/flink/api/scala/unfinishedKeyPairOperation.scala
--
diff --git 
a/flink-scala/src/main/scala/org/apache/flink/api/scala/unfinishedKeyPairOperation.scala
 
b/flink-scala/src/main/scala/org/apache/flink/api/scala/unfinishedKeyPairOperation.scala
index de03687..08d0242 100644
--- 

[3/3] flink git commit: [FLINK-1891] Add check for empty storageDirectory path

[fhueske]

[FLINK-1891] Add check for empty storageDirectory path

This closes #601


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/a0147c49
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/a0147c49
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/a0147c49

Branch: refs/heads/master
Commit: a0147c493cf210a0914c35200ebfacd47515374d
Parents: 45e680c
Author: hongsibao hongsi...@huawei.com
Authored: Wed Apr 15 18:15:27 2015 +0800
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue Apr 21 19:03:16 2015 +0200

--
 .../java/org/apache/flink/runtime/blob/BlobUtils.java   | 12 
 1 file changed, 8 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/a0147c49/flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobUtils.java
--
diff --git 
a/flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobUtils.java 
b/flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobUtils.java
index 69687da..b7cd42e 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobUtils.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobUtils.java
@@ -64,15 +64,19 @@ public class BlobUtils {
 * @return the storage directory used by a BLOB service
 */
static File initStorageDirectory(String storageDirectory) {
-   File baseDir = storageDirectory != null ?
-   new File(storageDirectory) :
-   new File(System.getProperty(java.io.tmpdir));
+   File baseDir;
+   if (storageDirectory == null || 
storageDirectory.trim().isEmpty()) {
+   baseDir = new 
File(System.getProperty(java.io.tmpdir));
+   }
+   else {
+   baseDir = new File(storageDirectory);
+   }
 
File storageDir;
final int MAX_ATTEMPTS = 10;
int attempt;
 
-   for(attempt = 0; attempt  MAX_ATTEMPTS; attempt++){
+   for(attempt = 0; attempt  MAX_ATTEMPTS; attempt++) {
storageDir = new File(baseDir, String.format(
blobStore-%s, 
UUID.randomUUID().toString()));
 

flink git commit: [FLINK-1978] Fix POJO deserialization for reuse objects with NULL fields

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/release-0.8 ffc86f668 - d9afefb0e


[FLINK-1978] Fix POJO deserialization for reuse objects with NULL fields

Backported fix.


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/d9afefb0
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/d9afefb0
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/d9afefb0

Branch: refs/heads/release-0.8
Commit: d9afefb0e6b6e5947276837fc2d6d55a6a185956
Parents: ffc86f6
Author: Fabian Hueske fhue...@apache.org
Authored: Wed May 6 17:47:59 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Thu May 7 12:20:51 2015 +0200

--
 .../java/typeutils/runtime/PojoSerializer.java   | 19 +--
 .../typeutils/runtime/PojoSerializerTest.java| 15 ++-
 2 files changed, 27 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/d9afefb0/flink-java/src/main/java/org/apache/flink/api/java/typeutils/runtime/PojoSerializer.java
--
diff --git 
a/flink-java/src/main/java/org/apache/flink/api/java/typeutils/runtime/PojoSerializer.java
 
b/flink-java/src/main/java/org/apache/flink/api/java/typeutils/runtime/PojoSerializer.java
index 15e8537..e205a9d 100644
--- 
a/flink-java/src/main/java/org/apache/flink/api/java/typeutils/runtime/PojoSerializer.java
+++ 
b/flink-java/src/main/java/org/apache/flink/api/java/typeutils/runtime/PojoSerializer.java
@@ -167,7 +167,14 @@ public final class PojoSerializerT extends 
TypeSerializerT {
for (int i = 0; i  numFields; i++) {
Object value = fields[i].get(from);
if (value != null) {
-   Object copy = 
fieldSerializers[i].copy(fields[i].get(from), fields[i].get(reuse));
+   Object reuseValue = 
fields[i].get(reuse);
+   Object copy;
+   if(reuseValue != null) {
+   copy = 
fieldSerializers[i].copy(value, reuseValue);
+   }
+   else {
+   copy = 
fieldSerializers[i].copy(value);
+   }
fields[i].set(reuse, copy);
}
else {
@@ -256,7 +263,15 @@ public final class PojoSerializerT extends 
TypeSerializerT {
if(isNull) {
fields[i].set(reuse, null);
} else {
-   Object field = 
fieldSerializers[i].deserialize(fields[i].get(reuse), source);
+   Object field;
+   Object reuseField = 
fields[i].get(reuse);
+   if(reuseField != null) {
+   field = 
fieldSerializers[i].deserialize(reuseField, source);
+   }
+   else {
+   field = 
fieldSerializers[i].deserialize(source);
+   }
+
fields[i].set(reuse, field);
}
}

http://git-wip-us.apache.org/repos/asf/flink/blob/d9afefb0/flink-java/src/test/java/org/apache/flink/api/java/typeutils/runtime/PojoSerializerTest.java
--
diff --git 
a/flink-java/src/test/java/org/apache/flink/api/java/typeutils/runtime/PojoSerializerTest.java
 
b/flink-java/src/test/java/org/apache/flink/api/java/typeutils/runtime/PojoSerializerTest.java
index 641a902..a499241 100644
--- 
a/flink-java/src/test/java/org/apache/flink/api/java/typeutils/runtime/PojoSerializerTest.java
+++ 
b/flink-java/src/test/java/org/apache/flink/api/java/typeutils/runtime/PojoSerializerTest.java
@@ -19,6 +19,7 @@
 package org.apache.flink.api.java.typeutils.runtime;
 
 import java.util.ArrayList;
+import java.util.Date;
 import java.util.List;
 import java.util.Random;
 
@@ -67,11 +68,13 @@ public class PojoSerializerTest extends 
SerializerTestBasePojoSerializerTest.Te
Random rnd = new Random(874597969123412341L);
 
return new TestUserClass[]{
-   new TestUserClass(rnd.nextInt(), foo, 
rnd.nextDouble(), new int[]{1, 2, 3},
+   new TestUserClass(rnd.nextInt(), foo, 

[3/4] flink git commit: [FLINK-1682] Ported optimizer unit tests from Record API to Java API

[fhueske]

http://git-wip-us.apache.org/repos/asf/flink/blob/bd96ba8d/flink-optimizer/src/test/java/org/apache/flink/optimizer/PartitioningReusageTest.java
--
diff --git 
a/flink-optimizer/src/test/java/org/apache/flink/optimizer/PartitioningReusageTest.java
 
b/flink-optimizer/src/test/java/org/apache/flink/optimizer/PartitioningReusageTest.java
index f42eb02..41e0eb9 100644
--- 
a/flink-optimizer/src/test/java/org/apache/flink/optimizer/PartitioningReusageTest.java
+++ 
b/flink-optimizer/src/test/java/org/apache/flink/optimizer/PartitioningReusageTest.java
@@ -25,6 +25,7 @@ import 
org.apache.flink.api.common.operators.base.JoinOperatorBase;
 import org.apache.flink.api.common.operators.util.FieldList;
 import org.apache.flink.api.java.DataSet;
 import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.flink.api.java.io.DiscardingOutputFormat;
 import org.apache.flink.api.java.operators.translation.JavaPlan;
 import org.apache.flink.api.java.tuple.Tuple3;
 import org.apache.flink.optimizer.dataproperties.GlobalProperties;
@@ -52,7 +53,7 @@ public class PartitioningReusageTest extends CompilerTestBase 
{
.join(set2, 
JoinOperatorBase.JoinHint.REPARTITION_HASH_FIRST)
.where(0,1).equalTo(0,1).with(new 
MockJoin());
 
-   joined.print();
+   joined.output(new DiscardingOutputFormatTuple3Integer, 
Integer, Integer());
JavaPlan plan = env.createProgramPlan();
OptimizedPlan oPlan = compileWithStats(plan);
 
@@ -73,7 +74,7 @@ public class PartitioningReusageTest extends CompilerTestBase 
{
.join(set2, 
JoinOperatorBase.JoinHint.REPARTITION_HASH_FIRST)
.where(0,1).equalTo(2,1).with(new MockJoin());
 
-   joined.print();
+   joined.output(new DiscardingOutputFormatTuple3Integer, 
Integer, Integer());
JavaPlan plan = env.createProgramPlan();
OptimizedPlan oPlan = compileWithStats(plan);
 
@@ -96,7 +97,7 @@ public class PartitioningReusageTest extends CompilerTestBase 
{
.join(set2, 
JoinOperatorBase.JoinHint.REPARTITION_HASH_FIRST)
.where(0,1).equalTo(0,1).with(new MockJoin());
 
-   joined.print();
+   joined.output(new DiscardingOutputFormatTuple3Integer, 
Integer, Integer());
JavaPlan plan = env.createProgramPlan();
OptimizedPlan oPlan = compileWithStats(plan);
 
@@ -119,7 +120,7 @@ public class PartitioningReusageTest extends 
CompilerTestBase {
.join(set2, 
JoinOperatorBase.JoinHint.REPARTITION_HASH_FIRST)
.where(0,1).equalTo(2,1).with(new MockJoin());
 
-   joined.print();
+   joined.output(new DiscardingOutputFormatTuple3Integer, 
Integer, Integer());
JavaPlan plan = env.createProgramPlan();
OptimizedPlan oPlan = compileWithStats(plan);
 
@@ -142,7 +143,7 @@ public class PartitioningReusageTest extends 
CompilerTestBase {

JoinOperatorBase.JoinHint.REPARTITION_HASH_FIRST)
.where(0,1).equalTo(2,1).with(new MockJoin());
 
-   joined.print();
+   joined.output(new DiscardingOutputFormatTuple3Integer, 
Integer, Integer());
JavaPlan plan = env.createProgramPlan();
OptimizedPlan oPlan = compileWithStats(plan);
 
@@ -164,7 +165,7 @@ public class PartitioningReusageTest extends 
CompilerTestBase {
.join(set2, 
JoinOperatorBase.JoinHint.REPARTITION_HASH_FIRST)
.where(0,1).equalTo(2,1).with(new MockJoin());
 
-   joined.print();
+   joined.output(new DiscardingOutputFormatTuple3Integer, 
Integer, Integer());
JavaPlan plan = env.createProgramPlan();
OptimizedPlan oPlan = compileWithStats(plan);
 
@@ -187,7 +188,7 @@ public class PartitioningReusageTest extends 
CompilerTestBase {

JoinOperatorBase.JoinHint.REPARTITION_HASH_FIRST)
.where(0,1).equalTo(2,1).with(new MockJoin());
 
-   joined.print();
+   joined.output(new DiscardingOutputFormatTuple3Integer, 
Integer, Integer());
JavaPlan plan = env.createProgramPlan();
OptimizedPlan oPlan = compileWithStats(plan);
 
@@ -212,7 +213,7 @@ public class PartitioningReusageTest extends 
CompilerTestBase {

JoinOperatorBase.JoinHint.REPARTITION_HASH_FIRST)
.where(0,1).equalTo(0,1).with(new MockJoin());
 
-   joined.print();
+   

[2/4] flink git commit: [FLINK-1682] Ported optimizer unit tests from Record API to Java API

[fhueske]

http://git-wip-us.apache.org/repos/asf/flink/blob/bd96ba8d/flink-optimizer/src/test/java/org/apache/flink/optimizer/util/OperatorResolver.java
--
diff --git 
a/flink-optimizer/src/test/java/org/apache/flink/optimizer/util/OperatorResolver.java
 
b/flink-optimizer/src/test/java/org/apache/flink/optimizer/util/OperatorResolver.java
index 920b713..fe0a533 100644
--- 
a/flink-optimizer/src/test/java/org/apache/flink/optimizer/util/OperatorResolver.java
+++ 
b/flink-optimizer/src/test/java/org/apache/flink/optimizer/util/OperatorResolver.java
@@ -29,14 +29,13 @@ import java.util.Set;
 import org.apache.flink.api.common.Plan;
 import org.apache.flink.api.common.functions.RichFunction;
 import org.apache.flink.api.common.operators.Operator;
-import org.apache.flink.api.java.record.operators.BulkIteration;
-import org.apache.flink.api.java.record.operators.DeltaIteration;
+import org.apache.flink.api.common.operators.base.BulkIterationBase;
+import org.apache.flink.api.common.operators.base.DeltaIterationBase;
 import org.apache.flink.util.Visitor;
 
 /**
  * Utility to get operator instances from plans via name.
  */
-@SuppressWarnings(deprecation)
 public class OperatorResolver implements VisitorOperator? {

private final MapString, ListOperator? map;
@@ -109,11 +108,11 @@ public class OperatorResolver implements 
VisitorOperator? {
list.add(visitable);

// recurse into bulk iterations
-   if (visitable instanceof BulkIteration) {
-   ((BulkIteration) 
visitable).getNextPartialSolution().accept(this);
-   } else if (visitable instanceof DeltaIteration) {
-   ((DeltaIteration) 
visitable).getSolutionSetDelta().accept(this);
-   ((DeltaIteration) 
visitable).getNextWorkset().accept(this);
+   if (visitable instanceof BulkIterationBase) {
+   ((BulkIterationBase) 
visitable).getNextPartialSolution().accept(this);
+   } else if (visitable instanceof DeltaIterationBase) {
+   ((DeltaIterationBase) 
visitable).getSolutionSetDelta().accept(this);
+   ((DeltaIterationBase) 
visitable).getNextWorkset().accept(this);
}

return true;

[1/4] flink git commit: [FLINK-1951] Fix NullPointerException in delta iteration due to missing temp

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master 60ec68308 - bd96ba8d1


[FLINK-1951] Fix NullPointerException in delta iteration due to missing temp

This closes #641


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/adb321d6
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/adb321d6
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/adb321d6

Branch: refs/heads/master
Commit: adb321d61cc783b3a2a78f4e707104d75e1d63c0
Parents: 60ec683
Author: Fabian Hueske fhue...@apache.org
Authored: Thu Apr 30 17:34:02 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue May 5 22:56:36 2015 +0200

--
 .../plantranslate/JobGraphGenerator.java|  5 +-
 .../plantranslate/TempInIterationsTest.java | 81 
 2 files changed, 84 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/adb321d6/flink-optimizer/src/main/java/org/apache/flink/optimizer/plantranslate/JobGraphGenerator.java
--
diff --git 
a/flink-optimizer/src/main/java/org/apache/flink/optimizer/plantranslate/JobGraphGenerator.java
 
b/flink-optimizer/src/main/java/org/apache/flink/optimizer/plantranslate/JobGraphGenerator.java
index dc21c13..2630019 100644
--- 
a/flink-optimizer/src/main/java/org/apache/flink/optimizer/plantranslate/JobGraphGenerator.java
+++ 
b/flink-optimizer/src/main/java/org/apache/flink/optimizer/plantranslate/JobGraphGenerator.java
@@ -1163,8 +1163,9 @@ public class JobGraphGenerator implements 
VisitorPlanNode {
final TempMode tm = channel.getTempMode();
 
boolean needsMemory = false;
-   // Don't add a pipeline breaker if the data exchange is 
already blocking.
-   if (tm.breaksPipeline()  
channel.getDataExchangeMode() != DataExchangeMode.BATCH) {
+   // Don't add a pipeline breaker if the data exchange is 
already blocking, EXCEPT the channel is within an iteration.
+   if (tm.breaksPipeline() 
+   (channel.isOnDynamicPath() || 
channel.getDataExchangeMode() != DataExchangeMode.BATCH) ) {

config.setInputAsynchronouslyMaterialized(inputNum, true);
needsMemory = true;
}

http://git-wip-us.apache.org/repos/asf/flink/blob/adb321d6/flink-optimizer/src/test/java/org/apache/flink/optimizer/plantranslate/TempInIterationsTest.java
--
diff --git 
a/flink-optimizer/src/test/java/org/apache/flink/optimizer/plantranslate/TempInIterationsTest.java
 
b/flink-optimizer/src/test/java/org/apache/flink/optimizer/plantranslate/TempInIterationsTest.java
new file mode 100644
index 000..15cb03f
--- /dev/null
+++ 
b/flink-optimizer/src/test/java/org/apache/flink/optimizer/plantranslate/TempInIterationsTest.java
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.optimizer.plantranslate;
+
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.flink.api.java.io.DiscardingOutputFormat;
+import org.apache.flink.api.java.operators.DeltaIteration;
+import org.apache.flink.api.java.operators.translation.JavaPlan;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.optimizer.Optimizer;
+import org.apache.flink.optimizer.plan.OptimizedPlan;
+import org.apache.flink.optimizer.testfunctions.DummyFlatJoinFunction;
+import org.apache.flink.runtime.jobgraph.AbstractJobVertex;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.operators.util.TaskConfig;
+import org.junit.Test;
+
+import static org.junit.Assert.assertTrue;
+
+public class TempInIterationsTest {
+
+   /*
+* Tests whether temps barriers are correctly set 

[2/3] flink-web git commit: Added blog post about how Flink operates on binary data

[fhueske]

, lower garbage collection overhead, and also 
easier system configuration. With Flink’s Table API, the semantics of all 
operations such as aggregations and projections are known (in contrast to 
black-box user-defined functions). Hence we can generate code for Table API 
operations that directly operates on binary data. Further improvements include 
serialization layouts which are tailored towards the operations that are 
applied on the binary data and code generation for serializers and comparators. 
/p
+
+pThe groundwork (and a lot more) for operating on binary data is done but 
there is still some room for making Flink even better and faster. If you are 
crazy about performance and like to juggle with lot of bits and bytes, join the 
Flink community! /p
+
+h3 id=tl;dr;-give-me-three-things-to-remember!TL;DR; Give me three things 
to remember!/h3
+
+ul
+liFlink’s active memory management avoids nasty 
codeOutOfMemoryErrors/code that kill your JVMs and reduces garbage 
collection overhead./li
+liFlink features a highly efficient data de/serialization stack that 
facilitates operations on binary data and makes more data fit into memory./li
+liFlink’s DBMS-style operators operate natively on binary data yielding 
high performance in-memory and destage gracefully to disk if necessary./li
+/ul
+
+pbr
+smallWritten by Fabian Hueske (a 
href=https://twitter.com/fhueske;@fhueske/a)./small/p
+/div
+   a 
href=/news/2015/05/11/Juggling-with-Bits-and-Bytes.html#disqus_threadJuggling
 with Bits and Bytes/a
+   /article
+   
+   article
h2a 
href=/news/2015/04/13/release-0.9.0-milestone1.htmlAnnouncing Flink 
0.9.0-milestone1 preview release/a/h2
p class=meta13 Apr 2015/p
 
@@ -1663,78 +1853,6 @@ Flink serialization system improved a lot over time and 
by now surpasses the cap
a 
href=/news/2014/11/18/hadoop-compatibility.html#disqus_threadHadoop 
Compatibility in Flink/a
/article

-   article
-   h2a 
href=/news/2014/11/04/release-0.7.0.htmlApache Flink 0.7.0 available/a/h2
-   p class=meta04 Nov 2014/p
-
-   divpWe are pleased to announce the 
availability of Flink 0.7.0. This release includes new user-facing features as 
well as performance and bug fixes, brings the Scala and Java APIs in sync, and 
introduces Flink Streaming. A total of 34 people have contributed to this 
release, a big thanks to all of them!/p
-
-pDownload Flink 0.7.0 a 
href=http://flink.incubator.apache.org/downloads.html;here/a/p
-
-pSee the release changelog a 
href=https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522amp;version=12327648;here/a/p
-
-h2 id=overview-of-major-new-featuresOverview of major new features/h2
-
-pstrongFlink Streaming:/strong The gem of the 0.7.0 release is 
undoubtedly Flink Streaming. Available currently in alpha, Flink Streaming 
provides a Java API on top of Apache Flink that can consume streaming data 
sources (e.g., from Apache Kafka, Apache Flume, and others) and process them in 
real time. A dedicated blog post on Flink Streaming and its performance is 
coming up here soon. You can check out the Streaming programming guide a 
href=http://flink.incubator.apache.org/docs/0.7-incubating/streaming_guide.html;here/a./p
-
-pstrongNew Scala API:/strong The Scala API has been completely 
rewritten. The Java and Scala APIs have now the same syntax and transformations 
and will be kept from now on in sync in every future release. See the new Scala 
API a 
href=http://flink.incubator.apache.org/docs/0.7-incubating/programming_guide.html;here/a./p
-
-pstrongLogical key expressions:/strong You can now specify grouping and 
joining keys with logical names for member variables of POJO data types. For 
example, you can join two data sets as 
codepersons.join(cities).where(“zip”).equalTo(“zipcode”)/code. Read 
more a 
href=http://flink.incubator.apache.org/docs/0.7-incubating/programming_guide.html#specifying-keys;here/a./p
-
-pstrongHadoop MapReduce compatibility:/strong You can run unmodified 
Hadoop Mappers and Reducers (mapred API) in Flink, use all Hadoop data types, 
and read data with all Hadoop InputFormats./p
-
-pstrongCollection-based execution backend:/strong The collection-based 
execution backend enables you to execute a Flink job as a simple Java 
collections program, bypassing completely the Flink runtime and optimizer. This 
feature is extremely useful for prototyping, and embedding Flink jobs in 
projects in a very lightweight manner./p
-
-pstrongRecord API deprecated:/strong The (old) Stratosphere Record API 
has been marked as deprecated and is planned for removal in the 0.9.0 
release./p
-
-pstrongBLOB service:/strong This release contains

flink git commit: [FLINK-1975][gelly] Graph getUndirected improvement

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master 8f28f41dc - beb7f3122


[FLINK-1975][gelly] Graph getUndirected improvement

This closes #653


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/beb7f312
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/beb7f312
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/beb7f312

Branch: refs/heads/master
Commit: beb7f31227766cfa17691525ccc666d105d442da
Parents: 8f28f41
Author: andralungu lungu.an...@gmail.com
Authored: Fri May 8 19:01:53 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Fri May 15 15:36:38 2015 +0200

--
 .../src/main/java/org/apache/flink/graph/Graph.java | 12 +++-
 1 file changed, 11 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/beb7f312/flink-staging/flink-gelly/src/main/java/org/apache/flink/graph/Graph.java
--
diff --git 
a/flink-staging/flink-gelly/src/main/java/org/apache/flink/graph/Graph.java 
b/flink-staging/flink-gelly/src/main/java/org/apache/flink/graph/Graph.java
index 33d8e1c..0ddd2d4 100755
--- a/flink-staging/flink-gelly/src/main/java/org/apache/flink/graph/Graph.java
+++ b/flink-staging/flink-gelly/src/main/java/org/apache/flink/graph/Graph.java
@@ -714,7 +714,7 @@ public class GraphK, VV, EV {
 */
public GraphK, VV, EV getUndirected() {
 
-   DataSetEdgeK, EV undirectedEdges = 
edges.union(edges.map(new ReverseEdgesMapK, EV()));
+   DataSetEdgeK, EV undirectedEdges = edges.flatMap(new 
RegularAndReversedEdgesMapK, EV());
return new GraphK, VV, EV(vertices, undirectedEdges, 
this.context);
}
 
@@ -936,6 +936,16 @@ public class GraphK, VV, EV {
}
}
 
+   private static final class RegularAndReversedEdgesMapK, EV
+   implements FlatMapFunctionEdgeK, EV, EdgeK, EV {
+
+   @Override
+   public void flatMap(EdgeK, EV edge, CollectorEdgeK, EV 
out) throws Exception {
+   out.collect(new EdgeK, EV(edge.f0, edge.f1, edge.f2));
+   out.collect(new EdgeK, EV(edge.f1, edge.f0, edge.f2));
+   }
+   }
+
/**
 * Reverse the direction of the edges in the graph
 * 

[2/2] flink git commit: [FLINK-1820] Consistent behavior of numeric value parsers.

[fhueske]

[FLINK-1820] Consistent behavior of numeric value parsers.

This closes #566


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/39d526e6
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/39d526e6
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/39d526e6

Branch: refs/heads/master
Commit: 39d526e6f8b26ff35e1023c65293982285ffcc78
Parents: 6403dbd
Author: FelixNeutatz neut...@googlemail.com
Authored: Fri Apr 3 18:13:32 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue May 19 00:52:29 2015 +0200

--
 .../apache/flink/types/parser/ByteParser.java   |  68 +-
 .../flink/types/parser/ByteValueParser.java |   4 +
 .../apache/flink/types/parser/DoubleParser.java |  69 +-
 .../flink/types/parser/DoubleValueParser.java   |   8 +-
 .../apache/flink/types/parser/FieldParser.java  |   8 +-
 .../apache/flink/types/parser/FloatParser.java  |  69 +-
 .../flink/types/parser/FloatValueParser.java|   8 +-
 .../apache/flink/types/parser/IntParser.java|  69 +-
 .../flink/types/parser/IntValueParser.java  |   4 +
 .../apache/flink/types/parser/LongParser.java   |  51 
 .../flink/types/parser/LongValueParser.java |   4 +
 .../apache/flink/types/parser/ShortParser.java  |  47 +++
 .../flink/types/parser/ShortValueParser.java|   4 +
 .../flink/types/parser/ByteParserTest.java  |   7 +-
 .../flink/types/parser/ByteValueParserTest.java |   2 +-
 .../flink/types/parser/DoubleParserTest.java|   2 +-
 .../types/parser/DoubleValueParserTest.java |   2 +-
 .../flink/types/parser/FloatParserTest.java |   2 +-
 .../types/parser/FloatValueParserTest.java  |   2 +-
 .../flink/types/parser/IntParserTest.java   |   2 +-
 .../flink/types/parser/IntValueParserTest.java  |   3 +-
 .../flink/types/parser/LongParserTest.java  |   2 +-
 .../flink/types/parser/LongValueParserTest.java |   2 +-
 .../flink/types/parser/ParserTestBase.java  |  40 +-
 .../flink/types/parser/ShortParserTest.java |   2 +-
 .../types/parser/ShortValueParserTest.java  |   2 +-
 .../flink/api/java/io/CsvInputFormatTest.java   | 127 +--
 27 files changed, 405 insertions(+), 205 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/39d526e6/flink-core/src/main/java/org/apache/flink/types/parser/ByteParser.java
--
diff --git 
a/flink-core/src/main/java/org/apache/flink/types/parser/ByteParser.java 
b/flink-core/src/main/java/org/apache/flink/types/parser/ByteParser.java
index 5858da2..09e517a 100644
--- a/flink-core/src/main/java/org/apache/flink/types/parser/ByteParser.java
+++ b/flink-core/src/main/java/org/apache/flink/types/parser/ByteParser.java
@@ -21,22 +21,23 @@ package org.apache.flink.types.parser;
 
 
 public class ByteParser extends FieldParserByte {
-   
+
private byte result;
-   
+
@Override
public int parseField(byte[] bytes, int startPos, int limit, byte[] 
delimiter, Byte reusable) {
int val = 0;
boolean neg = false;
 
-   final int delimLimit = limit-delimiter.length+1;
-   
+   final int delimLimit = limit - delimiter.length + 1;
+
if (bytes[startPos] == '-') {
neg = true;
startPos++;
-   
+
// check for empty field with only the sign
-   if (startPos == limit || (startPos  delimLimit  
delimiterNext(bytes, startPos, delimiter))) {
+   if (startPos == limit || (startPos  delimLimit  
delimiterNext(bytes, startPos, 
+   delimiter))) {

setErrorState(ParseErrorState.NUMERIC_VALUE_ORPHAN_SIGN);
return -1;
}
@@ -44,6 +45,10 @@ public class ByteParser extends FieldParserByte {
 
for (int i = startPos; i  limit; i++) {
if (i  delimLimit  delimiterNext(bytes, i, 
delimiter)) {
+   if (i == startPos) {
+   
setErrorState(ParseErrorState.EMPTY_STRING);
+   return -1;
+   }
this.result = (byte) (neg ? -val : val);
return i + delimiter.length;
}
@@ -53,17 +58,17 @@ public class ByteParser extends FieldParserByte {
}
val *= 10;
val += bytes[i] - 48;
-   
+
if (val  Byte.MAX_VALUE  (!neg || val  

[2/3] flink git commit: Additional tests for Path.isAbsolute()

[fhueske]

Additional tests for Path.isAbsolute()


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/23fe0062
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/23fe0062
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/23fe0062

Branch: refs/heads/master
Commit: 23fe00629e4b36e5c4d3849c2711743148369631
Parents: 2f683af
Author: Fabian Hueske fhue...@apache.org
Authored: Sat Apr 4 21:35:38 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue Apr 7 14:46:15 2015 +0200

--
 .../java/org/apache/flink/core/fs/Path.java |  5 +--
 .../java/org/apache/flink/core/fs/PathTest.java | 35 +---
 2 files changed, 32 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/23fe0062/flink-core/src/main/java/org/apache/flink/core/fs/Path.java
--
diff --git a/flink-core/src/main/java/org/apache/flink/core/fs/Path.java 
b/flink-core/src/main/java/org/apache/flink/core/fs/Path.java
index 05bccd1..75155eb 100644
--- a/flink-core/src/main/java/org/apache/flink/core/fs/Path.java
+++ b/flink-core/src/main/java/org/apache/flink/core/fs/Path.java
@@ -312,10 +312,7 @@ public class Path implements IOReadableWritable, 
Serializable {
 */
public boolean isAbsolute() {
final int start = hasWindowsDrive(uri.getPath(), true) ? 3 : 0;
-   if (uri.getPath().length()  start) {
-   return uri.getPath().startsWith(SEPARATOR, start);
-   }
-   return true;
+   return uri.getPath().startsWith(SEPARATOR, start);
}
 
/**

http://git-wip-us.apache.org/repos/asf/flink/blob/23fe0062/flink-core/src/test/java/org/apache/flink/core/fs/PathTest.java
--
diff --git a/flink-core/src/test/java/org/apache/flink/core/fs/PathTest.java 
b/flink-core/src/test/java/org/apache/flink/core/fs/PathTest.java
index e4c4e88..8fa2cea 100644
--- a/flink-core/src/test/java/org/apache/flink/core/fs/PathTest.java
+++ b/flink-core/src/test/java/org/apache/flink/core/fs/PathTest.java
@@ -69,9 +69,6 @@ public class PathTest {
p = new Path(file:/C:/my/windows/path);
assertEquals(/C:/my/windows/path, p.toUri().getPath());
 
-   p = new Path(C:);
-   assertEquals(/C:, p.toUri().getPath());
-
try {
new Path((String)null);
fail();
@@ -98,21 +95,51 @@ public class PathTest {
@Test
public void testIsAbsolute() {
 
+   // UNIX
+
Path p = new Path(/my/abs/path);
assertTrue(p.isAbsolute());
 
+   p = new Path(/);
+   assertTrue(p.isAbsolute());
+
p = new Path(./my/rel/path);
assertFalse(p.isAbsolute());
 
+   p = new Path(my/rel/path);
+   assertFalse(p.isAbsolute());
+
+   // WINDOWS
+
p = new Path(C:/my/abs/windows/path);
assertTrue(p.isAbsolute());
 
-   p = new Path(file:/C:);
+   p = new Path(y:/my/abs/windows/path);
+   assertTrue(p.isAbsolute());
+
+   p = new Path(b:\\my\\abs\\windows\\path);
assertTrue(p.isAbsolute());
 
p = new Path(C:);
+   assertFalse(p.isAbsolute());
+
+   p = new Path(C:my\\relative\\path);
+   assertFalse(p.isAbsolute());
+
+   p = new Path(\\my\\dir);
+   assertTrue(p.isAbsolute());
+
+   p = new Path(\\);
assertTrue(p.isAbsolute());
 
+   p = new Path(.\\my\\relative\\path);
+   assertFalse(p.isAbsolute());
+
+   p = new Path(my\\relative\\path);
+   assertFalse(p.isAbsolute());
+
+   p = new Path(myServer\\myDir);
+   assertTrue(p.isAbsolute());
}
 
@Test

[3/3] flink git commit: [FLINK-1817] Fix ClassLoaderObjectInputStream to support primitive type classes

[fhueske]

[FLINK-1817] Fix ClassLoaderObjectInputStream to support primitive type classes

This closes #565


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/d33b4454
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/d33b4454
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/d33b4454

Branch: refs/heads/master
Commit: d33b44549d416c30f01ead9aacf5cf7ed30674ca
Parents: 23fe006
Author: Fabian Hueske fhue...@apache.org
Authored: Thu Apr 2 01:28:08 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue Apr 7 14:46:24 2015 +0200

--
 .../apache/flink/util/InstantiationUtil.java| 30 +++-
 1 file changed, 29 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/d33b4454/flink-core/src/main/java/org/apache/flink/util/InstantiationUtil.java
--
diff --git 
a/flink-core/src/main/java/org/apache/flink/util/InstantiationUtil.java 
b/flink-core/src/main/java/org/apache/flink/util/InstantiationUtil.java
index c7088f5..241e56a 100644
--- a/flink-core/src/main/java/org/apache/flink/util/InstantiationUtil.java
+++ b/flink-core/src/main/java/org/apache/flink/util/InstantiationUtil.java
@@ -34,6 +34,7 @@ import java.io.ObjectOutputStream;
 import java.io.ObjectStreamClass;
 import java.lang.reflect.Constructor;
 import java.lang.reflect.Modifier;
+import java.util.HashMap;
 
 /**
  * Utility class to create instances from class objects and checking failure 
reasons.
@@ -48,10 +49,37 @@ public class InstantiationUtil {
private static class ClassLoaderObjectInputStream extends 
ObjectInputStream {
private ClassLoader classLoader;
 
+   private static final HashMapString, Class? primitiveClasses
+   = new HashMapString, Class?(8, 1.0F);
+   static {
+   primitiveClasses.put(boolean, boolean.class);
+   primitiveClasses.put(byte, byte.class);
+   primitiveClasses.put(char, char.class);
+   primitiveClasses.put(short, short.class);
+   primitiveClasses.put(int, int.class);
+   primitiveClasses.put(long, long.class);
+   primitiveClasses.put(float, float.class);
+   primitiveClasses.put(double, double.class);
+   primitiveClasses.put(void, void.class);
+   }
+
@Override
public Class? resolveClass(ObjectStreamClass desc) throws 
IOException, ClassNotFoundException {
if (classLoader != null) {
-   return Class.forName(desc.getName(), false, 
classLoader);
+   String name = desc.getName();
+   try {
+   return Class.forName(name, false, 
classLoader);
+   } catch (ClassNotFoundException ex) {
+   // check if class is a primitive class
+   Class? cl = 
primitiveClasses.get(name);
+   if (cl != null) {
+   // return primitive class
+   return cl;
+   } else {
+   // throw ClassNotFoundException
+   throw ex;
+   }
+   }
}
 
return super.resolveClass(desc);

[5/5] flink git commit: [FLINK-1664] Adds check if a selected sort key is sortable

[fhueske]

[FLINK-1664] Adds check if a selected sort key is sortable

This closes #541


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/f36eb54e
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/f36eb54e
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/f36eb54e

Branch: refs/heads/master
Commit: f36eb54ee6d8cc130439def98559b6b0a70b6c7b
Parents: f39aec8
Author: Fabian Hueske fhue...@apache.org
Authored: Fri Mar 27 21:37:59 2015 +0100
Committer: Fabian Hueske fhue...@apache.org
Committed: Fri Apr 3 20:42:05 2015 +0200

--
 .../api/common/typeinfo/TypeInformation.java|  10 +-
 .../api/common/typeutils/CompositeType.java |  20 ++
 .../flink/api/java/SortPartitionOperator.java   |  30 +-
 .../flink/api/java/operators/DataSink.java  |  28 ++
 .../apache/flink/api/java/operators/Keys.java   |  14 +-
 .../api/java/operators/SortedGrouping.java  |  71 +++--
 .../flink/api/java/typeutils/PojoTypeInfo.java  |   7 +-
 .../api/java/typeutils/TupleTypeInfoBase.java   |  19 --
 .../flink/api/java/operator/DataSinkTest.java   |  46 +--
 .../flink/api/java/operator/GroupingTest.java   | 278 +--
 .../api/java/operator/SortPartitionTest.java| 204 ++
 11 files changed, 637 insertions(+), 90 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/f36eb54e/flink-core/src/main/java/org/apache/flink/api/common/typeinfo/TypeInformation.java
--
diff --git 
a/flink-core/src/main/java/org/apache/flink/api/common/typeinfo/TypeInformation.java
 
b/flink-core/src/main/java/org/apache/flink/api/common/typeinfo/TypeInformation.java
index 4fa02e3..bb50e32 100644
--- 
a/flink-core/src/main/java/org/apache/flink/api/common/typeinfo/TypeInformation.java
+++ 
b/flink-core/src/main/java/org/apache/flink/api/common/typeinfo/TypeInformation.java
@@ -132,7 +132,15 @@ public abstract class TypeInformationT implements 
Serializable {
 * @return True, if the type can be used as a key, false otherwise.
 */
public abstract boolean isKeyType();
-   
+
+   /**
+* Checks whether this type can be used as a key for sorting.
+* The order produced by sorting this type must be meaningful.
+*/
+   public boolean isSortKeyType() {
+   return isKeyType();
+   }
+
/**
 * Creates a serializer for the type. The serializer may use the 
ExecutionConfig
 * for parameterization.

http://git-wip-us.apache.org/repos/asf/flink/blob/f36eb54e/flink-core/src/main/java/org/apache/flink/api/common/typeutils/CompositeType.java
--
diff --git 
a/flink-core/src/main/java/org/apache/flink/api/common/typeutils/CompositeType.java
 
b/flink-core/src/main/java/org/apache/flink/api/common/typeutils/CompositeType.java
index 54a1e13..de39ec8 100644
--- 
a/flink-core/src/main/java/org/apache/flink/api/common/typeutils/CompositeType.java
+++ 
b/flink-core/src/main/java/org/apache/flink/api/common/typeutils/CompositeType.java
@@ -169,6 +169,26 @@ public abstract class CompositeTypeT extends 
TypeInformationT {
return getFieldIndex(fieldName) = 0;
}
 
+   @Override
+   public boolean isKeyType() {
+   for(int i=0;ithis.getArity();i++) {
+   if (!this.getTypeAt(i).isKeyType()) {
+   return false;
+   }
+   }
+   return true;
+   }
+
+   @Override
+   public boolean isSortKeyType() {
+   for(int i=0;ithis.getArity();i++) {
+   if (!this.getTypeAt(i).isSortKeyType()) {
+   return false;
+   }
+   }
+   return true;
+   }
+
/**
 * Returns the names of the composite fields of this type. The order of 
the returned array must
 * be consistent with the internal field index ordering.

http://git-wip-us.apache.org/repos/asf/flink/blob/f36eb54e/flink-java/src/main/java/org/apache/flink/api/java/SortPartitionOperator.java
--
diff --git 
a/flink-java/src/main/java/org/apache/flink/api/java/SortPartitionOperator.java 
b/flink-java/src/main/java/org/apache/flink/api/java/SortPartitionOperator.java
index c8f8bbc..988144b 100644
--- 
a/flink-java/src/main/java/org/apache/flink/api/java/SortPartitionOperator.java
+++ 
b/flink-java/src/main/java/org/apache/flink/api/java/SortPartitionOperator.java
@@ -24,9 +24,11 @@ import org.apache.flink.api.common.operators.Order;
 import org.apache.flink.api.common.operators.Ordering;
 import 

[4/5] flink git commit: [FLINK-1776] Add offsets to field indexes of semantic properties for operators with key selectors

[fhueske]

[FLINK-1776] Add offsets to field indexes of semantic properties for operators 
with key selectors

This closes #532


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/f39aec82
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/f39aec82
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/f39aec82

Branch: refs/heads/master
Commit: f39aec82d6cd1286f129b11366d101cb646716ee
Parents: dda8565
Author: Fabian Hueske fhue...@apache.org
Authored: Tue Mar 24 15:18:21 2015 +0100
Committer: Fabian Hueske fhue...@apache.org
Committed: Fri Apr 3 20:41:40 2015 +0200

--
 .../api/java/functions/SemanticPropUtil.java|   79 ++
 .../api/java/operators/CoGroupOperator.java |   25 +
 .../java/operators/GroupCombineOperator.java|   23 +
 .../api/java/operators/GroupReduceOperator.java |   22 +
 .../flink/api/java/operators/JoinOperator.java  |   23 +
 .../api/java/operators/ReduceOperator.java  |   22 +
 .../translation/TwoKeyExtractingMapper.java |3 +-
 .../java/functions/SemanticPropUtilTest.java| 1008 ++
 .../api/java/operator/CoGroupOperatorTest.java  |  132 +++
 .../java/operator/GroupCombineOperatorTest.java |  345 ++
 .../java/operator/GroupReduceOperatorTest.java  |  345 ++
 .../api/java/operator/JoinOperatorTest.java |  183 +++-
 .../api/java/operator/ReduceOperatorTest.java   |  238 +
 .../javaApiOperators/GroupReduceITCase.java |   58 +
 14 files changed, 2054 insertions(+), 452 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/f39aec82/flink-java/src/main/java/org/apache/flink/api/java/functions/SemanticPropUtil.java
--
diff --git 
a/flink-java/src/main/java/org/apache/flink/api/java/functions/SemanticPropUtil.java
 
b/flink-java/src/main/java/org/apache/flink/api/java/functions/SemanticPropUtil.java
index 3343671..4569be3 100644
--- 
a/flink-java/src/main/java/org/apache/flink/api/java/functions/SemanticPropUtil.java
+++ 
b/flink-java/src/main/java/org/apache/flink/api/java/functions/SemanticPropUtil.java
@@ -153,6 +153,85 @@ public class SemanticPropUtil {
return dsp;
}
 
+   /**
+* Creates SemanticProperties by adding an offset to each input field 
index of the given SemanticProperties.
+*
+* @param props The SemanticProperties to which the offset is added.
+* @param numInputFields The original number of fields of the input.
+* @param offset The offset that is added to each input field index.
+* @return New SemanticProperties with added offset.
+*/
+   public static SingleInputSemanticProperties 
addSourceFieldOffset(SingleInputSemanticProperties props,
+   
int 
numInputFields, int offset) {
+
+   SingleInputSemanticProperties offsetProps = new 
SingleInputSemanticProperties();
+   if (props.getReadFields(0) != null) {
+   FieldSet offsetReadFields = new FieldSet();
+   for (int r : props.getReadFields(0)) {
+   offsetReadFields = offsetReadFields.addField(r 
+ offset);
+   }
+   offsetProps.addReadFields(offsetReadFields);
+   }
+   for (int s = 0; s  numInputFields; s++) {
+   FieldSet targetFields = 
props.getForwardingTargetFields(0, s);
+   for (int t : targetFields) {
+   offsetProps.addForwardedField(s + offset, t);
+   }
+   }
+   return offsetProps;
+   }
+
+   /**
+* Creates SemanticProperties by adding offsets to each input field 
index of the given SemanticProperties.
+*
+* @param props The SemanticProperties to which the offset is added.
+* @param numInputFields1 The original number of fields of the first 
input.
+* @param numInputFields2 The original number of fields of the second 
input.
+* @param offset1 The offset that is added to each input field index of 
the first input.
+* @param offset2 The offset that is added to each input field index of 
the second input.
+* @return New SemanticProperties with added offsets.
+*/
+   public static DualInputSemanticProperties 
addSourceFieldOffsets(DualInputSemanticProperties props,
+   
int numInputFields1, 
int numInputFields2,
+  

[3/5] flink git commit: [FLINK-1776] Add offsets to field indexes of semantic properties for operators with key selectors

[fhueske]

http://git-wip-us.apache.org/repos/asf/flink/blob/f39aec82/flink-java/src/test/java/org/apache/flink/api/java/functions/SemanticPropUtilTest.java
--
diff --git 
a/flink-java/src/test/java/org/apache/flink/api/java/functions/SemanticPropUtilTest.java
 
b/flink-java/src/test/java/org/apache/flink/api/java/functions/SemanticPropUtilTest.java
index cf81550..372c0f7 100644
--- 
a/flink-java/src/test/java/org/apache/flink/api/java/functions/SemanticPropUtilTest.java
+++ 
b/flink-java/src/test/java/org/apache/flink/api/java/functions/SemanticPropUtilTest.java
@@ -19,6 +19,7 @@
 
 package org.apache.flink.api.java.functions;
 
+import org.apache.flink.api.common.operators.SemanticProperties;
 import 
org.apache.flink.api.common.operators.SemanticProperties.InvalidSemanticAnnotationException;
 import org.apache.flink.api.common.typeutils.CompositeType;
 import org.apache.flink.api.java.typeutils.TypeExtractor;
@@ -34,6 +35,8 @@ import org.apache.flink.api.java.tuple.Tuple5;
 import org.apache.flink.api.java.typeutils.TupleTypeInfo;
 import org.junit.Test;
 
+import static org.junit.Assert.assertTrue;
+
 public class SemanticPropUtilTest {

private final TypeInformation? threeIntTupleType = new 
TupleTypeInfoTuple3Integer, Integer, Integer(BasicTypeInfo.INT_TYPE_INFO,
@@ -74,43 +77,43 @@ public class SemanticPropUtilTest {
 
int[] pMap = new int[] {3,0,4};
SingleInputSemanticProperties sp = 
SemanticPropUtil.createProjectionPropertiesSingle(pMap, (CompositeType?) 
fiveIntTupleType);
-   Assert.assertTrue(sp.getForwardingTargetFields(0, 
0).contains(1));
-   Assert.assertTrue(sp.getForwardingTargetFields(0, 
3).contains(0));
-   Assert.assertTrue(sp.getForwardingTargetFields(0, 
4).contains(2));
+   assertTrue(sp.getForwardingTargetFields(0, 0).contains(1));
+   assertTrue(sp.getForwardingTargetFields(0, 3).contains(0));
+   assertTrue(sp.getForwardingTargetFields(0, 4).contains(2));
 
pMap = new int[] {2,2,1,1};
sp = SemanticPropUtil.createProjectionPropertiesSingle(pMap, 
(CompositeType?) fiveIntTupleType);
-   Assert.assertTrue(sp.getForwardingTargetFields(0, 1).size() == 
2);
-   Assert.assertTrue(sp.getForwardingTargetFields(0, 
1).contains(2));
-   Assert.assertTrue(sp.getForwardingTargetFields(0, 
1).contains(3));
-   Assert.assertTrue(sp.getForwardingTargetFields(0, 2).size() == 
2);
-   Assert.assertTrue(sp.getForwardingTargetFields(0, 
2).contains(0));
-   Assert.assertTrue(sp.getForwardingTargetFields(0, 
2).contains(1));
+   assertTrue(sp.getForwardingTargetFields(0, 1).size() == 2);
+   assertTrue(sp.getForwardingTargetFields(0, 1).contains(2));
+   assertTrue(sp.getForwardingTargetFields(0, 1).contains(3));
+   assertTrue(sp.getForwardingTargetFields(0, 2).size() == 2);
+   assertTrue(sp.getForwardingTargetFields(0, 2).contains(0));
+   assertTrue(sp.getForwardingTargetFields(0, 2).contains(1));
 
pMap = new int[] {2,0};
sp = SemanticPropUtil.createProjectionPropertiesSingle(pMap, 
(CompositeType?) nestedTupleType);
-   Assert.assertTrue(sp.getForwardingTargetFields(0, 
4).contains(0));
-   Assert.assertTrue(sp.getForwardingTargetFields(0, 
0).contains(1));
-   Assert.assertTrue(sp.getForwardingTargetFields(0, 
1).contains(2));
-   Assert.assertTrue(sp.getForwardingTargetFields(0, 
2).contains(3));
+   assertTrue(sp.getForwardingTargetFields(0, 4).contains(0));
+   assertTrue(sp.getForwardingTargetFields(0, 0).contains(1));
+   assertTrue(sp.getForwardingTargetFields(0, 1).contains(2));
+   assertTrue(sp.getForwardingTargetFields(0, 2).contains(3));
 
pMap = new int[] {2,0,1};
sp = SemanticPropUtil.createProjectionPropertiesSingle(pMap, 
(CompositeType?) deepNestedTupleType);
-   Assert.assertTrue(sp.getForwardingTargetFields(0, 
6).contains(0));
-   Assert.assertTrue(sp.getForwardingTargetFields(0, 
0).contains(1));
-   Assert.assertTrue(sp.getForwardingTargetFields(0, 
1).contains(2));
-   Assert.assertTrue(sp.getForwardingTargetFields(0, 
2).contains(3));
-   Assert.assertTrue(sp.getForwardingTargetFields(0, 
3).contains(4));
-   Assert.assertTrue(sp.getForwardingTargetFields(0, 
4).contains(5));
-   Assert.assertTrue(sp.getForwardingTargetFields(0, 
5).contains(6));
+   assertTrue(sp.getForwardingTargetFields(0, 6).contains(0));
+   assertTrue(sp.getForwardingTargetFields(0, 0).contains(1));
+   assertTrue(sp.getForwardingTargetFields(0, 1).contains(2));
+

[1/3] flink git commit: [FLINK-1832] [scripts] fix conversion to Windows path with spaces

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master 07aff921b - 0afed4dcf


[FLINK-1832] [scripts] fix conversion to Windows path with spaces


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/ae6ff29d
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/ae6ff29d
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/ae6ff29d

Branch: refs/heads/master
Commit: ae6ff29dc6e19f8af431857817cd44b6796699c2
Parents: 07aff92
Author: Maximilian Michels m...@apache.org
Authored: Tue Apr 7 12:56:12 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Wed Apr 8 09:49:25 2015 +0200

--
 flink-dist/src/main/flink-bin/bin/config.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/ae6ff29d/flink-dist/src/main/flink-bin/bin/config.sh
--
diff --git a/flink-dist/src/main/flink-bin/bin/config.sh 
b/flink-dist/src/main/flink-bin/bin/config.sh
index 232df3e..66e0b80 100755
--- a/flink-dist/src/main/flink-bin/bin/config.sh
+++ b/flink-dist/src/main/flink-bin/bin/config.sh
@@ -25,7 +25,7 @@
 manglePath() {
 UNAME=$(uname -s)
 if [ ${UNAME:0:6} == CYGWIN ]; then
-echo `cygpath -w $1`
+echo `cygpath -w $1`
 else
 echo $1
 fi
@@ -35,7 +35,7 @@ manglePathList() {
 UNAME=$(uname -s)
 # a path list, for example a java classpath
 if [ ${UNAME:0:6} == CYGWIN ]; then
-echo `cygpath -wp $1`
+echo `cygpath -wp $1`
 else
 echo $1
 fi

[3/3] flink git commit: [FLINK-1840] Fix BLOB manager on Windows

[fhueske]

[FLINK-1840] Fix BLOB manager on Windows

This closes #578


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/0afed4dc
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/0afed4dc
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/0afed4dc

Branch: refs/heads/master
Commit: 0afed4dcf9f9639fc90a8f18933be0516b9d7cd3
Parents: cc7eda1
Author: Fabian Hueske fhue...@apache.org
Authored: Tue Apr 7 23:04:59 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Wed Apr 8 09:52:15 2015 +0200

--
 .../flink/runtime/blob/BlobServerConnection.java   | 13 -
 1 file changed, 4 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/0afed4dc/flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobServerConnection.java
--
diff --git 
a/flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobServerConnection.java
 
b/flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobServerConnection.java
index 50b1f24..4f242a2 100644
--- 
a/flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobServerConnection.java
+++ 
b/flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobServerConnection.java
@@ -30,6 +30,7 @@ import java.net.SocketException;
 import java.security.MessageDigest;
 
 import org.apache.flink.api.common.JobID;
+import org.apache.flink.shaded.com.google.common.io.Files;
 import org.apache.flink.util.InstantiationUtil;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -307,24 +308,18 @@ class BlobServerConnection extends Thread {
md.update(buf, 0, bytesExpected);
}
}
-
+   fos.close();
 
if (contentAddressable == NAME_ADDRESSABLE) {
File storageFile = 
this.blobServer.getStorageLocation(jobID, key);
-   if (!incomingFile.renameTo(storageFile)) {
-   throw new 
IOException(String.format(Cannot move staging file %s to BLOB file %s,
-   
incomingFile.getAbsolutePath(), storageFile.getAbsolutePath()));
-   }
+   Files.move(incomingFile, storageFile);
incomingFile = null;
outputStream.write(RETURN_OKAY);
}
else {
BlobKey blobKey = new BlobKey(md.digest());
File storageFile = 
blobServer.getStorageLocation(blobKey);
-   if (!incomingFile.renameTo(storageFile)) {
-   throw new 
IOException(String.format(Cannot move staging file %s to BLOB file %s,
-   
incomingFile.getAbsolutePath(), storageFile.getAbsolutePath()));
-   }
+   Files.move(incomingFile, storageFile);
incomingFile = null;
 
// Return computed key to client for validation

[3/3] flink git commit: [FLINK-1840] Fix BLOB manager on Windows

[fhueske]

[FLINK-1840] Fix BLOB manager on Windows


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/8f126a31
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/8f126a31
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/8f126a31

Branch: refs/heads/release-0.9.0-milestone-1
Commit: 8f126a314e660109ca5d9fbb5106fb92d0ce78d9
Parents: 8b9ba52
Author: Fabian Hueske fhue...@apache.org
Authored: Tue Apr 7 23:04:59 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Wed Apr 8 09:55:36 2015 +0200

--
 .../flink/runtime/blob/BlobServerConnection.java   | 13 -
 1 file changed, 4 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/8f126a31/flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobServerConnection.java
--
diff --git 
a/flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobServerConnection.java
 
b/flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobServerConnection.java
index 50b1f24..4f242a2 100644
--- 
a/flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobServerConnection.java
+++ 
b/flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobServerConnection.java
@@ -30,6 +30,7 @@ import java.net.SocketException;
 import java.security.MessageDigest;
 
 import org.apache.flink.api.common.JobID;
+import org.apache.flink.shaded.com.google.common.io.Files;
 import org.apache.flink.util.InstantiationUtil;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -307,24 +308,18 @@ class BlobServerConnection extends Thread {
md.update(buf, 0, bytesExpected);
}
}
-
+   fos.close();
 
if (contentAddressable == NAME_ADDRESSABLE) {
File storageFile = 
this.blobServer.getStorageLocation(jobID, key);
-   if (!incomingFile.renameTo(storageFile)) {
-   throw new 
IOException(String.format(Cannot move staging file %s to BLOB file %s,
-   
incomingFile.getAbsolutePath(), storageFile.getAbsolutePath()));
-   }
+   Files.move(incomingFile, storageFile);
incomingFile = null;
outputStream.write(RETURN_OKAY);
}
else {
BlobKey blobKey = new BlobKey(md.digest());
File storageFile = 
blobServer.getStorageLocation(blobKey);
-   if (!incomingFile.renameTo(storageFile)) {
-   throw new 
IOException(String.format(Cannot move staging file %s to BLOB file %s,
-   
incomingFile.getAbsolutePath(), storageFile.getAbsolutePath()));
-   }
+   Files.move(incomingFile, storageFile);
incomingFile = null;
 
// Return computed key to client for validation

[1/3] flink git commit: [FLINK-1832] [scripts] fix conversion to Windows path with spaces

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/release-0.9.0-milestone-1 287b4e9a3 - 8f126a314


[FLINK-1832] [scripts] fix conversion to Windows path with spaces


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/3ec0c190
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/3ec0c190
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/3ec0c190

Branch: refs/heads/release-0.9.0-milestone-1
Commit: 3ec0c190c5b833439a130bbbe89c3c73d75d11ec
Parents: 287b4e9
Author: Maximilian Michels m...@apache.org
Authored: Tue Apr 7 12:56:12 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Wed Apr 8 09:54:45 2015 +0200

--
 flink-dist/src/main/flink-bin/bin/config.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/3ec0c190/flink-dist/src/main/flink-bin/bin/config.sh
--
diff --git a/flink-dist/src/main/flink-bin/bin/config.sh 
b/flink-dist/src/main/flink-bin/bin/config.sh
index 232df3e..66e0b80 100755
--- a/flink-dist/src/main/flink-bin/bin/config.sh
+++ b/flink-dist/src/main/flink-bin/bin/config.sh
@@ -25,7 +25,7 @@
 manglePath() {
 UNAME=$(uname -s)
 if [ ${UNAME:0:6} == CYGWIN ]; then
-echo `cygpath -w $1`
+echo `cygpath -w $1`
 else
 echo $1
 fi
@@ -35,7 +35,7 @@ manglePathList() {
 UNAME=$(uname -s)
 # a path list, for example a java classpath
 if [ ${UNAME:0:6} == CYGWIN ]; then
-echo `cygpath -wp $1`
+echo `cygpath -wp $1`
 else
 echo $1
 fi

[2/3] flink git commit: [FLINK-1832] [scripts] fix start scripts for Cygwin and paths with spaces.

[fhueske]

[FLINK-1832] [scripts] fix start scripts for Cygwin and paths with spaces.


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/8b9ba524
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/8b9ba524
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/8b9ba524

Branch: refs/heads/release-0.9.0-milestone-1
Commit: 8b9ba52471b08b5aaa0f4af5ce8fa167c121543f
Parents: 3ec0c19
Author: Fabian Hueske fhue...@apache.org
Authored: Tue Apr 7 20:59:44 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Wed Apr 8 09:55:31 2015 +0200

--
 flink-dist/src/main/flink-bin/bin/config.sh | 15 ---
 flink-dist/src/main/flink-bin/bin/flink | 20 ++--
 flink-dist/src/main/flink-bin/bin/jobmanager.sh | 19 ++-
 .../src/main/flink-bin/bin/taskmanager.sh   | 19 ++-
 flink-dist/src/main/flink-bin/bin/webclient.sh  | 18 ++
 5 files changed, 20 insertions(+), 71 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/8b9ba524/flink-dist/src/main/flink-bin/bin/config.sh
--
diff --git a/flink-dist/src/main/flink-bin/bin/config.sh 
b/flink-dist/src/main/flink-bin/bin/config.sh
index 66e0b80..def4f09 100755
--- a/flink-dist/src/main/flink-bin/bin/config.sh
+++ b/flink-dist/src/main/flink-bin/bin/config.sh
@@ -17,6 +17,18 @@
 # limitations under the License.
 

 
+constructFlinkClassPath() {
+
+for jarfile in $FLINK_LIB_DIR/*.jar ; do
+if [[ $FLINK_CLASSPATH =  ]]; then
+FLINK_CLASSPATH=$jarfile;
+else
+FLINK_CLASSPATH=$FLINK_CLASSPATH:$jarfile
+fi
+done
+
+echo $FLINK_CLASSPATH
+}
 
 # These are used to mangle paths that are passed to java when using 
 # cygwin. Cygwin paths are like linux paths, i.e. /path/to/somewhere
@@ -189,9 +201,6 @@ fi
 # KEY_JOBM_HEAP_MB and KEY_TASKM_HEAP_MB for that!
 JVM_ARGS=
 
-# Default classpath 
-CLASSPATH=`manglePathList $( echo $FLINK_LIB_DIR/*.jar . | sed 's/ /:/g' )`
-
 # Check if deprecated HADOOP_HOME is set.
 if [ -n $HADOOP_HOME ]; then
 # HADOOP_HOME is set. Check if its a Hadoop 1.x or 2.x HADOOP_HOME path

http://git-wip-us.apache.org/repos/asf/flink/blob/8b9ba524/flink-dist/src/main/flink-bin/bin/flink
--
diff --git a/flink-dist/src/main/flink-bin/bin/flink 
b/flink-dist/src/main/flink-bin/bin/flink
old mode 100755
new mode 100644
index 7247260..094ddf4
--- a/flink-dist/src/main/flink-bin/bin/flink
+++ b/flink-dist/src/main/flink-bin/bin/flink
@@ -27,23 +27,7 @@ if [ $FLINK_IDENT_STRING =  ]; then
 FLINK_IDENT_STRING=$USER
 fi
 
-
-# auxiliary function to construct a lightweight classpath for the
-# Flink CLI client
-constructCLIClientClassPath() {
-
-   for jarfile in $FLINK_LIB_DIR/*.jar ; do
-   if [[ $CC_CLASSPATH =  ]]; then
-   CC_CLASSPATH=$jarfile;
-   else
-   CC_CLASSPATH=$CC_CLASSPATH:$jarfile
-   fi
-   done
-
-   echo $CC_CLASSPATH
-}
-
-CC_CLASSPATH=`manglePathList $(constructCLIClientClassPath)`
+CC_CLASSPATH=`constructFlinkClassPath`
 
 log=$FLINK_LOG_DIR/flink-$FLINK_IDENT_STRING-flink-client-$HOSTNAME.log
 log_setting=-Dlog.file=$log 
-Dlog4j.configuration=file:$FLINK_CONF_DIR/log4j-cli.properties 
-Dlogback.configurationFile=file:$FLINK_CONF_DIR/logback.xml
@@ -51,4 +35,4 @@ log_setting=-Dlog.file=$log 
-Dlog4j.configuration=file:$FLINK_CONF_DIR/log4
 export FLINK_CONF_DIR
 
 # Add HADOOP_CLASSPATH to allow the usage of Hadoop file systems
-$JAVA_RUN $JVM_ARGS $log_setting -classpath 
$CC_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS org.apache.flink.client.CliFrontend $*
+$JAVA_RUN $JVM_ARGS $log_setting -classpath `manglePathList 
$CC_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS` 
org.apache.flink.client.CliFrontend $*

http://git-wip-us.apache.org/repos/asf/flink/blob/8b9ba524/flink-dist/src/main/flink-bin/bin/jobmanager.sh
--
diff --git a/flink-dist/src/main/flink-bin/bin/jobmanager.sh 
b/flink-dist/src/main/flink-bin/bin/jobmanager.sh
index 0298f01..93e5ee3 100755
--- a/flink-dist/src/main/flink-bin/bin/jobmanager.sh
+++ b/flink-dist/src/main/flink-bin/bin/jobmanager.sh
@@ -32,26 +32,11 @@ if [ $JAVA_VERSION -lt 18 ]; then
 JVM_ARGS=$JVM_ARGS -XX:MaxPermSize=256m
 fi
 
-
-
 if [ $FLINK_IDENT_STRING =  ]; then
 FLINK_IDENT_STRING=$USER
 fi
 
-# auxilliary function to construct a the classpath for the jobManager
-constructJobManagerClassPath() {
-for jarfile in $FLINK_LIB_DIR/*.jar ; do
-if [[ $FLINK_JM_CLASSPATH 

[2/2] flink git commit: [FLINK-1832] [scripts] fix start scripts for Cygwin and paths with spaces.

[fhueske]

[FLINK-1832] [scripts] fix start scripts for Cygwin and paths with spaces.


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/f2ee7838
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/f2ee7838
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/f2ee7838

Branch: refs/heads/release-0.8
Commit: f2ee7838be18d5d41849310edc1160eeb10118be
Parents: 1cc6ee4
Author: Fabian Hueske fhue...@apache.org
Authored: Tue Apr 7 20:59:44 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Wed Apr 8 10:15:59 2015 +0200

--
 flink-dist/src/main/flink-bin/bin/config.sh | 15 ---
 flink-dist/src/main/flink-bin/bin/flink | 20 ++--
 flink-dist/src/main/flink-bin/bin/jobmanager.sh | 18 +++---
 .../src/main/flink-bin/bin/taskmanager.sh   | 19 ++-
 flink-dist/src/main/flink-bin/bin/webclient.sh  | 18 ++
 5 files changed, 21 insertions(+), 69 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/f2ee7838/flink-dist/src/main/flink-bin/bin/config.sh
--
diff --git a/flink-dist/src/main/flink-bin/bin/config.sh 
b/flink-dist/src/main/flink-bin/bin/config.sh
index 35861b8..26edda4 100755
--- a/flink-dist/src/main/flink-bin/bin/config.sh
+++ b/flink-dist/src/main/flink-bin/bin/config.sh
@@ -17,6 +17,18 @@
 # limitations under the License.
 

 
+constructFlinkClassPath() {
+
+for jarfile in $FLINK_LIB_DIR/*.jar ; do
+if [[ $FLINK_CLASSPATH =  ]]; then
+FLINK_CLASSPATH=$jarfile;
+else
+FLINK_CLASSPATH=$FLINK_CLASSPATH:$jarfile
+fi
+done
+
+echo $FLINK_CLASSPATH
+}
 
 # These are used to mangle paths that are passed to java when using 
 # cygwin. Cygwin paths are like linux paths, i.e. /path/to/somewhere
@@ -187,9 +199,6 @@ fi
 # KEY_JOBM_HEAP_MB and KEY_TASKM_HEAP_MB for that!
 JVM_ARGS=
 
-# Default classpath 
-CLASSPATH=`manglePathList $( echo $FLINK_LIB_DIR/*.jar . | sed 's/ /:/g' )`
-
 # Auxilliary function which extracts the name of host from a line which
 # also potentialy includes topology information and the instance type
 extractHostName() {

http://git-wip-us.apache.org/repos/asf/flink/blob/f2ee7838/flink-dist/src/main/flink-bin/bin/flink
--
diff --git a/flink-dist/src/main/flink-bin/bin/flink 
b/flink-dist/src/main/flink-bin/bin/flink
old mode 100755
new mode 100644
index e5dd3c6..79c4a6f
--- a/flink-dist/src/main/flink-bin/bin/flink
+++ b/flink-dist/src/main/flink-bin/bin/flink
@@ -27,27 +27,11 @@ if [ $FLINK_IDENT_STRING =  ]; then
 FLINK_IDENT_STRING=$USER
 fi
 
-
-# auxiliary function to construct a lightweight classpath for the
-# Flink CLI client
-constructCLIClientClassPath() {
-
-   for jarfile in $FLINK_LIB_DIR/*.jar ; do
-   if [[ $CC_CLASSPATH =  ]]; then
-   CC_CLASSPATH=$jarfile;
-   else
-   CC_CLASSPATH=$CC_CLASSPATH:$jarfile
-   fi
-   done
-
-   echo $CC_CLASSPATH
-}
-
-CC_CLASSPATH=`manglePathList $(constructCLIClientClassPath)`
+CC_CLASSPATH=`constructFlinkClassPath`
 
 log=$FLINK_LOG_DIR/flink-$FLINK_IDENT_STRING-flink-client-$HOSTNAME.log
 log_setting=-Dlog.file=$log 
-Dlog4j.configuration=file:$FLINK_CONF_DIR/log4j.properties 
-Dlogback.configurationFile=file:$FLINK_CONF_DIR/logback.xml
 
 export FLINK_CONF_DIR
 
-$JAVA_RUN $JVM_ARGS $log_setting -classpath $CC_CLASSPATH 
org.apache.flink.client.CliFrontend $*
+$JAVA_RUN $JVM_ARGS $log_setting -classpath `manglePathList 
$CC_CLASSPATH` org.apache.flink.client.CliFrontend $*

http://git-wip-us.apache.org/repos/asf/flink/blob/f2ee7838/flink-dist/src/main/flink-bin/bin/jobmanager.sh
--
diff --git a/flink-dist/src/main/flink-bin/bin/jobmanager.sh 
b/flink-dist/src/main/flink-bin/bin/jobmanager.sh
index 0a9f9f7..74ff22a 100755
--- a/flink-dist/src/main/flink-bin/bin/jobmanager.sh
+++ b/flink-dist/src/main/flink-bin/bin/jobmanager.sh
@@ -44,20 +44,7 @@ if [ $FLINK_IDENT_STRING =  ]; then
 FLINK_IDENT_STRING=$USER
 fi
 
-# auxilliary function to construct a the classpath for the jobmanager
-constructJobManagerClassPath() {
-for jarfile in $FLINK_LIB_DIR/*.jar ; do
-if [[ $FLINK_JM_CLASSPATH =  ]]; then
-FLINK_JM_CLASSPATH=$jarfile;
-else
-FLINK_JM_CLASSPATH=$FLINK_JM_CLASSPATH:$jarfile
-fi
-done
-
-echo $FLINK_JM_CLASSPATH
-}
-
-FLINK_JM_CLASSPATH=`manglePathList $(constructJobManagerClassPath)`
+FLINK_JM_CLASSPATH=`constructFlinkClassPath`
 
 

flink-web git commit: Fixed link to list of starter tasks

[fhueske]

Repository: flink-web
Updated Branches:
  refs/heads/asf-site 5b6e5b6ca - fee4a5dda


Fixed link to list of starter tasks


Project: http://git-wip-us.apache.org/repos/asf/flink-web/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink-web/commit/fee4a5dd
Tree: http://git-wip-us.apache.org/repos/asf/flink-web/tree/fee4a5dd
Diff: http://git-wip-us.apache.org/repos/asf/flink-web/diff/fee4a5dd

Branch: refs/heads/asf-site
Commit: fee4a5dda56f36d38f3add5a16c2dcaf2ebd4b97
Parents: 5b6e5b6
Author: Fabian Hueske fhue...@gmail.com
Authored: Fri Jun 5 12:09:20 2015 +0200
Committer: Fabian Hueske fhue...@gmail.com
Committed: Fri Jun 5 12:09:20 2015 +0200

--
 content/coding-guidelines.html | 1 +
 content/how-to-contribute.html | 2 +-
 how-to-contribute.md   | 2 +-
 3 files changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink-web/blob/fee4a5dd/content/coding-guidelines.html
--
diff --git a/content/coding-guidelines.html b/content/coding-guidelines.html
index e1d7fc4..b59a77a 100644
--- a/content/coding-guidelines.html
+++ b/content/coding-guidelines.html
@@ -183,6 +183,7 @@
   /li
   listrongNo wildcard imports/strong. Do not use wildcard imports in the 
core files. They can cause problems when adding to the code and in some cases 
even during refactoring. Exceptions are the Tuple classes, Tuple-related 
utilities, and Flink user programs, when importing operators/functions. Tests 
are a special case of the user programs./li
   listrongNo unused imports/strong. Remove all unused imports./li
+  listrongUse Guava Checks/strong. To increase homogeneity, consistently 
use Guava methods checkNotNull and checkArgument rather than Apache Commons 
Validate./li
   listrongNo raw generic types/strong. Do not use raw generic types, 
unless strictly necessary (sometime necessary for signature matches, 
arrays)./li
   listrongSupress warnings/strong. Add annotations to suppress warnings, 
if they cannot be avoided (such as “unchecked”, or “serial”)./li
   li

http://git-wip-us.apache.org/repos/asf/flink-web/blob/fee4a5dd/content/how-to-contribute.html
--
diff --git a/content/how-to-contribute.html b/content/how-to-contribute.html
index 3f18084..441330d 100644
--- a/content/how-to-contribute.html
+++ b/content/how-to-contribute.html
@@ -171,7 +171,7 @@
 
 pWe maintain all known issues and feature drafts in the a 
href=https://issues.apache.org/jira/issues/?jql=project+%3D+FLINK;Flink 
project JIRA/a./p
 
-pWe also try to maintain a a 
href=https://issues.apache.org/jira/browse/FLINK-992?jql=project%20%3D%20FLINK%20AND%20labels%20%3D%20starter%20AND%20status%20in%20(Open%2C%20Reopened)list
 of simple “starter issues”/a that we believe are good tasks for new 
contributors. Those tasks are meant to allow people to get into the project and 
become familiar with the process of contributing. Feel free to ask questions 
about issues that you would be interested in working on./p
+pWe also try to maintain a a 
href=https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20%3D%20starter%20ORDER%20BY%20priority%20DESC;list
 of simple “starter issues”/a that we believe are good tasks for new 
contributors. Those tasks are meant to allow people to get into the project and 
become familiar with the process of contributing. Feel free to ask questions 
about issues that you would be interested in working on./p
 
 pIn addition, you can find a list of ideas for projects and improvements in 
the a 
href=https://cwiki.apache.org/confluence/display/FLINK/Project+Ideas;projects 
Wiki page/a./p
 

http://git-wip-us.apache.org/repos/asf/flink-web/blob/fee4a5dd/how-to-contribute.md
--
diff --git a/how-to-contribute.md b/how-to-contribute.md
index ae85918..f0b5b50 100644
--- a/how-to-contribute.md
+++ b/how-to-contribute.md
@@ -10,7 +10,7 @@ The Flink project welcomes all sorts contributions in the 
form of code (improvem
 
 We maintain all known issues and feature drafts in the [Flink project 
JIRA](https://issues.apache.org/jira/issues/?jql=project+%3D+FLINK).
 
-We also try to maintain a a 
href=https://issues.apache.org/jira/browse/FLINK-992?jql=project%20%3D%20FLINK%20AND%20labels%20%3D%20starter%20AND%20status%20in%20(Open%2C%20Reopened)list
 of simple starter issues/a that we believe are good tasks for new 
contributors. Those tasks are meant to allow people to get into the project and 
become familiar with the process of contributing. Feel free to ask questions 
about issues that you would be interested in working on.
+We also try to maintain a a 

flink git commit: [FLINK-2196] [javaAPI] Moved misplaced SortPartitionOperator class

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master b5b71e1c4 - a8e085078


[FLINK-2196] [javaAPI] Moved misplaced SortPartitionOperator class

This closes #817


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/a8e08507
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/a8e08507
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/a8e08507

Branch: refs/heads/master
Commit: a8e085078499319113a213575c9f057144395f98
Parents: b5b71e1
Author: Fabian Hueske fhue...@apache.org
Authored: Wed Jun 10 11:58:08 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Wed Jun 10 14:49:15 2015 +0200

--
 .../java/org/apache/flink/api/java/DataSet.java |   1 +
 .../flink/api/java/SortPartitionOperator.java   | 208 ---
 .../java/operators/SortPartitionOperator.java   | 207 ++
 .../org/apache/flink/api/scala/DataSet.scala|   2 +-
 4 files changed, 209 insertions(+), 209 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/a8e08507/flink-java/src/main/java/org/apache/flink/api/java/DataSet.java
--
diff --git a/flink-java/src/main/java/org/apache/flink/api/java/DataSet.java 
b/flink-java/src/main/java/org/apache/flink/api/java/DataSet.java
index f1eed3e..e217e53 100644
--- a/flink-java/src/main/java/org/apache/flink/api/java/DataSet.java
+++ b/flink-java/src/main/java/org/apache/flink/api/java/DataSet.java
@@ -71,6 +71,7 @@ import org.apache.flink.api.java.operators.PartitionOperator;
 import org.apache.flink.api.java.operators.ProjectOperator;
 import org.apache.flink.api.java.operators.ProjectOperator.Projection;
 import org.apache.flink.api.java.operators.ReduceOperator;
+import org.apache.flink.api.java.operators.SortPartitionOperator;
 import org.apache.flink.api.java.operators.SortedGrouping;
 import org.apache.flink.api.java.operators.UnionOperator;
 import org.apache.flink.api.java.operators.UnsortedGrouping;

http://git-wip-us.apache.org/repos/asf/flink/blob/a8e08507/flink-java/src/main/java/org/apache/flink/api/java/SortPartitionOperator.java
--
diff --git 
a/flink-java/src/main/java/org/apache/flink/api/java/SortPartitionOperator.java 
b/flink-java/src/main/java/org/apache/flink/api/java/SortPartitionOperator.java
deleted file mode 100644
index 988144b..000
--- 
a/flink-java/src/main/java/org/apache/flink/api/java/SortPartitionOperator.java
+++ /dev/null
@@ -1,208 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.api.java;
-
-import org.apache.flink.api.common.InvalidProgramException;
-import org.apache.flink.api.common.operators.Operator;
-import org.apache.flink.api.common.operators.Order;
-import org.apache.flink.api.common.operators.Ordering;
-import org.apache.flink.api.common.operators.UnaryOperatorInformation;
-import org.apache.flink.api.common.operators.base.SortPartitionOperatorBase;
-import org.apache.flink.api.common.typeinfo.TypeInformation;
-import org.apache.flink.api.common.typeutils.CompositeType;
-import org.apache.flink.api.java.operators.Keys;
-import org.apache.flink.api.java.operators.SingleInputOperator;
-import org.apache.flink.api.java.typeutils.TupleTypeInfoBase;
-
-import java.util.Arrays;
-
-/**
- * This operator represents a DataSet with locally sorted partitions.
- *
- * @param T The type of the DataSet with locally sorted partitions.
- */
-public class SortPartitionOperatorT extends SingleInputOperatorT, T, 
SortPartitionOperatorT {
-
-   private int[] sortKeyPositions;
-
-   private Order[] sortOrders;
-
-   private final String sortLocationName;
-
-
-   public SortPartitionOperator(DataSetT dataSet, int sortField, Order 
sortOrder, String sortLocationName) {
-   super(dataSet, dataSet.getType());
-   this.sortLocationName = sortLocationName;
-
-   int[] flatOrderKeys = getFlatFields(sortField);
-   

flink git commit: [FLINK-2199] Escape UTF characters in Scala Shell welcome squirrel.

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master 63b8c3647 - 2ea1e7db2


[FLINK-2199] Escape UTF characters in Scala Shell welcome squirrel.

This closes #821


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/2ea1e7db
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/2ea1e7db
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/2ea1e7db

Branch: refs/heads/master
Commit: 2ea1e7db2e9474fd7e5f46f40dbadba2af9427ce
Parents: 63b8c36
Author: Fabian Hueske fhue...@apache.org
Authored: Wed Jun 10 17:43:05 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Thu Jun 11 12:03:51 2015 +0200

--
 .../org.apache.flink/api/scala/FlinkILoop.scala | 71 +++-
 1 file changed, 38 insertions(+), 33 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/2ea1e7db/flink-staging/flink-scala-shell/src/main/scala/org.apache.flink/api/scala/FlinkILoop.scala
--
diff --git 
a/flink-staging/flink-scala-shell/src/main/scala/org.apache.flink/api/scala/FlinkILoop.scala
 
b/flink-staging/flink-scala-shell/src/main/scala/org.apache.flink/api/scala/FlinkILoop.scala
index 83d0c72..10b7ad9 100644
--- 
a/flink-staging/flink-scala-shell/src/main/scala/org.apache.flink/api/scala/FlinkILoop.scala
+++ 
b/flink-staging/flink-scala-shell/src/main/scala/org.apache.flink/api/scala/FlinkILoop.scala
@@ -144,39 +144,40 @@ class FlinkILoop(
*/
   override def printWelcome() {
 echo(
+  // scalastyle:off
   
- ▒▓██▓██▒
- ▓████▒▒█▓▒▓███▓▒
-  ▓███▓░░▒▒▒▓██▒  ▒
-░██▒   ▒▒▓▓█▓▓▒░  ▒████
-██▒ ░▒▓███▒▒█▒█▒
-  ░▓████   ▓░▒██
-▓█   ▒▒▒▒▒▓██▓░▒░▓▓█
-  █░ █   ▒▒░   ███▓▓█ 
▒█▒▒▒
-  ████░   ▒▓█▓  ██▒▒▒ 
▓███▒
-   ░▒█▓▓██   ▓█▒▓█▒▓██▓ 
░█░
- ▓░▒▓████▒ ██ ▒█
█▓░▒█▒░▒█▒
-███▓░██▓  ▓█   █   █▓ 
▒▓█▓▓█▒
-  ░██▓  ░█░█  █▒ ▒█████▓▒ 
██▓░▒
- ███░ ░ █░  ▓ ░█ █████▒░░
░█░▓  ▓░
-██▓█ ▒▒▓▒  ▓███████▓░   
▒█▒ ▒▓ ▓██▓
- ▒██▓ ▓█ █▓█   ░▒█████▓▓▒░ 
██▒▒  █ ▒  ▓█▒
- ▓█▓  ▓█ ██▓ ░▓▓▓▓▓▓▓▒  
▒██▓   ░█▒
- ▓██ ▓███▓▒░  ░▓▓▓███▓ 
 ░▒░ ▓█
- ██▓██▒
░▒▓▓███▓▓▓▓▓██████▓▒
▓███  █
-▓███▒ ███   ░▓▓▒░░   ░▓████▓░  
░▒▓▒  █▓
-█▓▒▒▓▓██  ░▒▒░░░▒▒▒▒▓██▓░
█▓
-██ ▓░▒█   ▓▓▓▓▒░░  ▒█▓   
▒▓▓██▓▓▒  ▒▒▓
-▓█▓ ▓▒█  █▓░  ░▒▓▓██▒░▓█▒  
 ▒▒▒░▒▒▓█████▒
- ██░ ▓█▒█▒  ▒▓▓▒  ▓██░  
░░░░   ░█▒
- ▓█   ▒█▓   ░ █░▒█  █▓
-  █▓   ██ █░ ▓▓
▒█▓▓▓▒█░
-   █▓ ░▓██░   ▓▒  
▓█▓▒░░░▒▓█░▒█
-██   ▓█▓░  ▒░▒█▒██▒
  ▓▓
- ▓█▒   ▒█▓▒░ ▒▒ 
█▒█▓▒▒░░▒██
-  ░██▒▒▓▓▒ ▓██▓▒█▒ 
░▓▓▓▓▒█▓
-░▓██▒  ▓░  ▒█▓█  
░░▒▒▒
-
▒▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒░░▓▓
  ▓░▒█░
+ \u2592\u2593\u2588\u2588\u2593\u2588\u2588\u2592
+ 
\u2593\u2588\u2588\u2588\u2588\u2592\u2592\u2588\u2593\u2592\u2593\u2588\u2588\u2588\u2593\u2592
+  

flink git commit: [hotfix] Removed execute() that followed print() in quickstart wordcount jobs

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master b59c81bc4 - d38154129


[hotfix] Removed execute() that followed print() in quickstart wordcount jobs


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/d3815412
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/d3815412
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/d3815412

Branch: refs/heads/master
Commit: d38154129b8298d998b955e6450a05b79f7cd995
Parents: b59c81b
Author: Fabian Hueske fhue...@apache.org
Authored: Tue Jun 16 22:11:21 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue Jun 16 22:11:21 2015 +0200

--
 .../resources/archetype-resources/src/main/java/WordCount.java   | 4 +---
 .../resources/archetype-resources/src/main/scala/WordCount.scala | 4 +---
 2 files changed, 2 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/d3815412/flink-quickstart/flink-quickstart-java/src/main/resources/archetype-resources/src/main/java/WordCount.java
--
diff --git 
a/flink-quickstart/flink-quickstart-java/src/main/resources/archetype-resources/src/main/java/WordCount.java
 
b/flink-quickstart/flink-quickstart-java/src/main/resources/archetype-resources/src/main/java/WordCount.java
index f250996..6202238 100644
--- 
a/flink-quickstart/flink-quickstart-java/src/main/resources/archetype-resources/src/main/java/WordCount.java
+++ 
b/flink-quickstart/flink-quickstart-java/src/main/resources/archetype-resources/src/main/java/WordCount.java
@@ -63,11 +63,9 @@ public class WordCount {
.groupBy(0)
.sum(1);
 
-   // emit result
+   // execute and print result
counts.print();
 
-   // execute program
-   env.execute(WordCount Example);
}
 
//

http://git-wip-us.apache.org/repos/asf/flink/blob/d3815412/flink-quickstart/flink-quickstart-scala/src/main/resources/archetype-resources/src/main/scala/WordCount.scala
--
diff --git 
a/flink-quickstart/flink-quickstart-scala/src/main/resources/archetype-resources/src/main/scala/WordCount.scala
 
b/flink-quickstart/flink-quickstart-scala/src/main/resources/archetype-resources/src/main/scala/WordCount.scala
index 150830f..b88dcc6 100644
--- 
a/flink-quickstart/flink-quickstart-scala/src/main/resources/archetype-resources/src/main/scala/WordCount.scala
+++ 
b/flink-quickstart/flink-quickstart-scala/src/main/resources/archetype-resources/src/main/scala/WordCount.scala
@@ -46,10 +46,8 @@ object WordCount {
   .groupBy(0)
   .sum(1)
 
-// emit result
+// execute and print result
 counts.print()
 
-// execute program
-env.execute(WordCount Example)
   }
 }

flink git commit: [hotfix] Removed execute() that followed print() in quickstart wordcount jobs

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/release-0.9 2359b49f5 - 0a8142a99


[hotfix] Removed execute() that followed print() in quickstart wordcount jobs


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/0a8142a9
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/0a8142a9
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/0a8142a9

Branch: refs/heads/release-0.9
Commit: 0a8142a9916c6fbe6db4d2a1f0199acd5737d836
Parents: 2359b49
Author: Fabian Hueske fhue...@apache.org
Authored: Tue Jun 16 22:11:21 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue Jun 16 22:13:12 2015 +0200

--
 .../resources/archetype-resources/src/main/java/WordCount.java   | 4 +---
 .../resources/archetype-resources/src/main/scala/WordCount.scala | 4 +---
 2 files changed, 2 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/0a8142a9/flink-quickstart/flink-quickstart-java/src/main/resources/archetype-resources/src/main/java/WordCount.java
--
diff --git 
a/flink-quickstart/flink-quickstart-java/src/main/resources/archetype-resources/src/main/java/WordCount.java
 
b/flink-quickstart/flink-quickstart-java/src/main/resources/archetype-resources/src/main/java/WordCount.java
index f250996..6202238 100644
--- 
a/flink-quickstart/flink-quickstart-java/src/main/resources/archetype-resources/src/main/java/WordCount.java
+++ 
b/flink-quickstart/flink-quickstart-java/src/main/resources/archetype-resources/src/main/java/WordCount.java
@@ -63,11 +63,9 @@ public class WordCount {
.groupBy(0)
.sum(1);
 
-   // emit result
+   // execute and print result
counts.print();
 
-   // execute program
-   env.execute(WordCount Example);
}
 
//

http://git-wip-us.apache.org/repos/asf/flink/blob/0a8142a9/flink-quickstart/flink-quickstart-scala/src/main/resources/archetype-resources/src/main/scala/WordCount.scala
--
diff --git 
a/flink-quickstart/flink-quickstart-scala/src/main/resources/archetype-resources/src/main/scala/WordCount.scala
 
b/flink-quickstart/flink-quickstart-scala/src/main/resources/archetype-resources/src/main/scala/WordCount.scala
index 150830f..b88dcc6 100644
--- 
a/flink-quickstart/flink-quickstart-scala/src/main/resources/archetype-resources/src/main/scala/WordCount.scala
+++ 
b/flink-quickstart/flink-quickstart-scala/src/main/resources/archetype-resources/src/main/scala/WordCount.scala
@@ -46,10 +46,8 @@ object WordCount {
   .groupBy(0)
   .sum(1)
 
-// emit result
+// execute and print result
 counts.print()
 
-// execute program
-env.execute(WordCount Example)
   }
 }

flink git commit: [FLINK-2207] Fix TableAPI conversion documenation and further renamings for consistency.

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master e45c5dc56 - e4b569505


[FLINK-2207] Fix TableAPI conversion documenation and further renamings for 
consistency.

This closes #829


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/e4b56950
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/e4b56950
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/e4b56950

Branch: refs/heads/master
Commit: e4b569505ba1c4169d39e24c9dc4ff0ca03bf992
Parents: e45c5dc
Author: Fabian Hueske fhue...@apache.org
Authored: Fri Jun 12 11:36:03 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Fri Jun 12 14:20:46 2015 +0200

--
 docs/libs/table.md|  8 
 .../flink/api/scala/table/TableConversions.scala  |  4 ++--
 .../main/scala/org/apache/flink/api/table/Table.scala |  2 +-
 .../apache/flink/examples/scala/PageRankTable.scala   |  2 +-
 .../flink/examples/scala/StreamingTableFilter.scala   |  2 +-
 .../flink/api/scala/table/test/FilterITCase.scala |  6 +++---
 .../flink/api/scala/table/test/JoinITCase.scala   | 14 +++---
 7 files changed, 19 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/e4b56950/docs/libs/table.md
--
diff --git a/docs/libs/table.md b/docs/libs/table.md
index bcd2cb1..829c9cf 100644
--- a/docs/libs/table.md
+++ b/docs/libs/table.md
@@ -52,7 +52,7 @@ import org.apache.flink.api.scala.table._
 case class WC(word: String, count: Int)
 val input = env.fromElements(WC(hello, 1), WC(hello, 1), WC(ciao, 1))
 val expr = input.toTable
-val result = expr.groupBy('word).select('word, 'count.sum as 'count).toSet[WC]
+val result = expr.groupBy('word).select('word, 'count.sum as 
'count).toDataSet[WC]
 {% endhighlight %}
 
 The expression DSL uses Scala symbols to refer to field names and we use code 
generation to
@@ -69,7 +69,7 @@ case class MyResult(a: String, d: Int)
 
 val input1 = env.fromElements(...).toTable('a, 'b)
 val input2 = env.fromElements(...).toTable('c, 'd)
-val joined = input1.join(input2).where(b = a  d  42).select(a, 
d).toSet[MyResult]
+val joined = input1.join(input2).where(b = a  d  42).select(a, 
d).toDataSet[MyResult]
 {% endhighlight %}
 
 Notice, how a DataSet can be converted to a Table by using `as` and specifying 
new
@@ -108,14 +108,14 @@ DataSetWC input = env.fromElements(
 new WC(Ciao, 1),
 new WC(Hello, 1));
 
-Table table = tableEnv.toTable(input);
+Table table = tableEnv.fromDataSet(input);
 
 Table filtered = table
 .groupBy(word)
 .select(word.count as count, word)
 .filter(count = 2);
 
-DataSetWC result = tableEnv.toSet(filtered, WC.class);
+DataSetWC result = tableEnv.toDataSet(filtered, WC.class);
 {% endhighlight %}
 
 When using Java, the embedded DSL for specifying expressions cannot be used. 
Only String expressions

http://git-wip-us.apache.org/repos/asf/flink/blob/e4b56950/flink-staging/flink-table/src/main/scala/org/apache/flink/api/scala/table/TableConversions.scala
--
diff --git 
a/flink-staging/flink-table/src/main/scala/org/apache/flink/api/scala/table/TableConversions.scala
 
b/flink-staging/flink-table/src/main/scala/org/apache/flink/api/scala/table/TableConversions.scala
index b9c0a5e..4f2172e 100644
--- 
a/flink-staging/flink-table/src/main/scala/org/apache/flink/api/scala/table/TableConversions.scala
+++ 
b/flink-staging/flink-table/src/main/scala/org/apache/flink/api/scala/table/TableConversions.scala
@@ -33,14 +33,14 @@ class TableConversions(table: Table) {
   /**
* Converts the [[Table]] to a [[DataSet]].
*/
-  def toSet[T: TypeInformation]: DataSet[T] = {
+  def toDataSet[T: TypeInformation]: DataSet[T] = {
  new ScalaBatchTranslator().translate[T](table.operation)
   }
 
   /**
* Converts the [[Table]] to a [[DataStream]].
*/
-  def toStream[T: TypeInformation]: DataStream[T] = {
+  def toDataStream[T: TypeInformation]: DataStream[T] = {
 new ScalaStreamingTranslator().translate[T](table.operation)
   }
 }

http://git-wip-us.apache.org/repos/asf/flink/blob/e4b56950/flink-staging/flink-table/src/main/scala/org/apache/flink/api/table/Table.scala
--
diff --git 
a/flink-staging/flink-table/src/main/scala/org/apache/flink/api/table/Table.scala
 
b/flink-staging/flink-table/src/main/scala/org/apache/flink/api/table/Table.scala
index 83d5239..fdb125b 100644
--- 
a/flink-staging/flink-table/src/main/scala/org/apache/flink/api/table/Table.scala
+++ 
b/flink-staging/flink-table/src/main/scala/org/apache/flink/api/table/Table.scala
@@ -39,7 +39,7 @@ import 

[1/4] flink git commit: [FLINK-2206] Fix incorrect counts of finished, canceled, and failed jobs in webinterface

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/release-0.9 ecfde6dd9 - f5f0709c9


[FLINK-2206] Fix incorrect counts of finished, canceled, and failed jobs in 
webinterface

This closes #826


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/e513be72
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/e513be72
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/e513be72

Branch: refs/heads/release-0.9
Commit: e513be72a486b4f2e13c617eb6d4d08c03503ae7
Parents: ecfde6d
Author: Fabian Hueske fhue...@apache.org
Authored: Fri Jun 12 01:45:03 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Fri Jun 12 14:26:47 2015 +0200

--
 .../jobmanager/web/JobManagerInfoServlet.java   | 31 +
 .../js/jobmanagerFrontend.js| 36 +---
 .../runtime/jobmanager/MemoryArchivist.scala| 17 +
 .../runtime/messages/ArchiveMessages.scala  | 11 +-
 4 files changed, 75 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/e513be72/flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/web/JobManagerInfoServlet.java
--
diff --git 
a/flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/web/JobManagerInfoServlet.java
 
b/flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/web/JobManagerInfoServlet.java
index 6d58306..3fc3c82 100644
--- 
a/flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/web/JobManagerInfoServlet.java
+++ 
b/flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/web/JobManagerInfoServlet.java
@@ -66,6 +66,7 @@ import org.apache.flink.util.ExceptionUtils;
 import org.apache.flink.util.StringUtils;
 import org.eclipse.jetty.io.EofException;
 
+import scala.Tuple3;
 import scala.concurrent.Await;
 import scala.concurrent.Future;
 import scala.concurrent.duration.FiniteDuration;
@@ -117,6 +118,20 @@ public class JobManagerInfoServlet extends HttpServlet {
writeJsonForArchive(resp.getWriter(), 
archivedJobs);
}
}
+   else if(jobcounts.equals(req.getParameter(get))) {
+   response = Patterns.ask(archive, 
ArchiveMessages.getRequestJobCounts(),
+   new Timeout(timeout));
+
+   result = Await.result(response, timeout);
+
+   if(!(result instanceof Tuple3)) {
+   throw new 
RuntimeException(RequestJobCounts requires a response of type  +
+   Tuple3. Instead the 
response is of type  + result.getClass() +
+   .);
+   } else {
+   writeJsonForJobCounts(resp.getWriter(), 
(Tuple3)result);
+   }
+   }
else if(job.equals(req.getParameter(get))) {
String jobId = req.getParameter(job);
 
@@ -341,6 +356,22 @@ public class JobManagerInfoServlet extends HttpServlet {
}
 
/**
+* Writes Json with the job counts
+*
+* @param wrt
+* @param counts
+*/
+   private void writeJsonForJobCounts(PrintWriter wrt, Tuple3Integer, 
Integer, Integer jobCounts) {
+
+   wrt.write({);
+   wrt.write(\finished\:  + jobCounts._1() + ,);
+   wrt.write(\canceled\:  + jobCounts._2() + ,);
+   wrt.write(\failed\:+ jobCounts._3());
+   wrt.write(});
+
+   }
+
+   /**
 * Writes infos about archived job in Json format, including 
groupvertices and groupverticetimes
 *
 * @param wrt

http://git-wip-us.apache.org/repos/asf/flink/blob/e513be72/flink-runtime/src/main/resources/web-docs-infoserver/js/jobmanagerFrontend.js
--
diff --git 
a/flink-runtime/src/main/resources/web-docs-infoserver/js/jobmanagerFrontend.js 
b/flink-runtime/src/main/resources/web-docs-infoserver/js/jobmanagerFrontend.js
index 92f6979..63d287c 100644
--- 
a/flink-runtime/src/main/resources/web-docs-infoserver/js/jobmanagerFrontend.js
+++ 
b/flink-runtime/src/main/resources/web-docs-infoserver/js/jobmanagerFrontend.js
@@ -81,6 +81,22 @@ function poll(jobId) {
 })();
 
 /*
+ * Polls the job execution counts on page load and every 2 seconds
+ */
+(function pollJobCounts() {
+   $.ajax({ url : jobsInfo?get=jobcounts, cache: false, type : GET,
+   success : function(json) {
+
+ 

[3/4] flink git commit: [FLINK-2072] [ml] Adds quickstart guide

[fhueske]

[FLINK-2072] [ml] Adds quickstart guide

This closes #792.


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/40e2df5a
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/40e2df5a
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/40e2df5a

Branch: refs/heads/release-0.9
Commit: 40e2df5acf9385cc3c6e3a947b4bf6cd2bd375b3
Parents: af0fee5
Author: Theodore Vasiloudis t...@sics.se
Authored: Fri Jun 5 11:09:11 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Fri Jun 12 14:27:29 2015 +0200

--
 docs/libs/ml/contribution_guide.md |  10 +-
 docs/libs/ml/index.md  |  27 ++--
 docs/libs/ml/quickstart.md | 216 +++-
 3 files changed, 235 insertions(+), 18 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/40e2df5a/docs/libs/ml/contribution_guide.md
--
diff --git a/docs/libs/ml/contribution_guide.md 
b/docs/libs/ml/contribution_guide.md
index 89f05c0..f0754cb 100644
--- a/docs/libs/ml/contribution_guide.md
+++ b/docs/libs/ml/contribution_guide.md
@@ -36,7 +36,7 @@ Everything from this guide also applies to FlinkML.
 
 ## Pick a Topic
 
-If you are looking for some new ideas, then you should check out the list of 
[unresolved issues on 
JIRA](https://issues.apache.org/jira/issues/?jql=component%20%3D%20%22Machine%20Learning%20Library%22%20AND%20project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20priority%20DESC).
+If you are looking for some new ideas you should first look into our 
[roadmap](vision_roadmap.html#Roadmap), then you should check out the list of 
[unresolved issues on 
JIRA](https://issues.apache.org/jira/issues/?jql=component%20%3D%20%22Machine%20Learning%20Library%22%20AND%20project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20priority%20DESC).
 Once you decide to contribute to one of these issues, you should take 
ownership of it and track your progress with this issue.
 That way, the other contributors know the state of the different issues and 
redundant work is avoided.
 
@@ -61,7 +61,7 @@ Thus, an integration test could look the following:
 {% highlight scala %}
 class ExampleITSuite extends FlatSpec with FlinkTestBase {
   behavior of An example algorithm
-  
+
   it should do something in {
 ...
   }
@@ -81,12 +81,12 @@ Every new algorithm is described by a single markdown file.
 This file should contain at least the following points:
 
 1. What does the algorithm do
-2. How does the algorithm work (or reference to description) 
+2. How does the algorithm work (or reference to description)
 3. Parameter description with default values
 4. Code snippet showing how the algorithm is used
 
 In order to use latex syntax in the markdown file, you have to include 
`mathjax: include` in the YAML front matter.
- 
+
 {% highlight java %}
 ---
 mathjax: include
@@ -103,4 +103,4 @@ See `docs/_include/latex_commands.html` for the complete 
list of predefined late
 ## Contributing
 
 Once you have implemented the algorithm with adequate test coverage and added 
documentation, you are ready to open a pull request.
-Details of how to open a pull request can be found 
[here](http://flink.apache.org/how-to-contribute.html#contributing-code--documentation).
 
+Details of how to open a pull request can be found 
[here](http://flink.apache.org/how-to-contribute.html#contributing-code--documentation).

http://git-wip-us.apache.org/repos/asf/flink/blob/40e2df5a/docs/libs/ml/index.md
--
diff --git a/docs/libs/ml/index.md b/docs/libs/ml/index.md
index de9137d..9ff7a4b 100644
--- a/docs/libs/ml/index.md
+++ b/docs/libs/ml/index.md
@@ -21,9 +21,9 @@ under the License.
 --
 
 FlinkML is the Machine Learning (ML) library for Flink. It is a new effort in 
the Flink community,
-with a growing list of algorithms and contributors. With FlinkML we aim to 
provide 
-scalable ML algorithms, an intuitive API, and tools that help minimize glue 
code in end-to-end ML 
-systems. You can see more details about our goals and where the library is 
headed in our [vision 
+with a growing list of algorithms and contributors. With FlinkML we aim to 
provide
+scalable ML algorithms, an intuitive API, and tools that help minimize glue 
code in end-to-end ML
+systems. You can see more details about our goals and where the library is 
headed in our [vision
 and roadmap here](vision_roadmap.html).
 
 * This will be replaced by the TOC
@@ -55,10 +55,13 @@ FlinkML currently supports the following algorithms:
 
 ## Getting Started
 
-First, you have to [set up a Flink 

[4/4] flink git commit: [FLINK-2194] [type extractor] Excludes Writable type from WritableTypeInformation to be treated as an interface

[fhueske]

[FLINK-2194] [type extractor] Excludes Writable type from 
WritableTypeInformation to be treated as an interface

This closes #814.


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/f5f0709c
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/f5f0709c
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/f5f0709c

Branch: refs/heads/release-0.9
Commit: f5f0709c9ba090cd1321dec761a27db556082630
Parents: 40e2df5
Author: Till Rohrmann trohrm...@apache.org
Authored: Tue Jun 9 16:11:05 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Fri Jun 12 14:28:52 2015 +0200

--
 .../java/org/apache/flink/api/java/typeutils/TypeExtractor.java| 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/f5f0709c/flink-java/src/main/java/org/apache/flink/api/java/typeutils/TypeExtractor.java
--
diff --git 
a/flink-java/src/main/java/org/apache/flink/api/java/typeutils/TypeExtractor.java
 
b/flink-java/src/main/java/org/apache/flink/api/java/typeutils/TypeExtractor.java
index 71d0cee..41644f9 100644
--- 
a/flink-java/src/main/java/org/apache/flink/api/java/typeutils/TypeExtractor.java
+++ 
b/flink-java/src/main/java/org/apache/flink/api/java/typeutils/TypeExtractor.java
@@ -1152,7 +1152,7 @@ public class TypeExtractor {
}

// check for writable types
-   if(Writable.class.isAssignableFrom(clazz)) {
+   if(Writable.class.isAssignableFrom(clazz)  
!Writable.class.equals(clazz)) {
return (TypeInformationOUT) 
WritableTypeInfo.getWritableTypeInfo((Class? extends Writable) clazz);
}

[2/4] flink git commit: [FLINK-2207] Fix TableAPI conversion documenation and further renamings for consistency.

[fhueske]

[FLINK-2207] Fix TableAPI conversion documenation and further renamings for 
consistency.

This closes #829


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/af0fee51
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/af0fee51
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/af0fee51

Branch: refs/heads/release-0.9
Commit: af0fee512bde4a5dc5c08a3cc17da788a06cd113
Parents: e513be7
Author: Fabian Hueske fhue...@apache.org
Authored: Fri Jun 12 11:36:03 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Fri Jun 12 14:27:03 2015 +0200

--
 docs/libs/table.md|  8 
 .../flink/api/scala/table/TableConversions.scala  |  4 ++--
 .../main/scala/org/apache/flink/api/table/Table.scala |  2 +-
 .../apache/flink/examples/scala/PageRankTable.scala   |  2 +-
 .../flink/examples/scala/StreamingTableFilter.scala   |  2 +-
 .../flink/api/scala/table/test/FilterITCase.scala |  6 +++---
 .../flink/api/scala/table/test/JoinITCase.scala   | 14 +++---
 7 files changed, 19 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/af0fee51/docs/libs/table.md
--
diff --git a/docs/libs/table.md b/docs/libs/table.md
index bcd2cb1..829c9cf 100644
--- a/docs/libs/table.md
+++ b/docs/libs/table.md
@@ -52,7 +52,7 @@ import org.apache.flink.api.scala.table._
 case class WC(word: String, count: Int)
 val input = env.fromElements(WC(hello, 1), WC(hello, 1), WC(ciao, 1))
 val expr = input.toTable
-val result = expr.groupBy('word).select('word, 'count.sum as 'count).toSet[WC]
+val result = expr.groupBy('word).select('word, 'count.sum as 
'count).toDataSet[WC]
 {% endhighlight %}
 
 The expression DSL uses Scala symbols to refer to field names and we use code 
generation to
@@ -69,7 +69,7 @@ case class MyResult(a: String, d: Int)
 
 val input1 = env.fromElements(...).toTable('a, 'b)
 val input2 = env.fromElements(...).toTable('c, 'd)
-val joined = input1.join(input2).where(b = a  d  42).select(a, 
d).toSet[MyResult]
+val joined = input1.join(input2).where(b = a  d  42).select(a, 
d).toDataSet[MyResult]
 {% endhighlight %}
 
 Notice, how a DataSet can be converted to a Table by using `as` and specifying 
new
@@ -108,14 +108,14 @@ DataSetWC input = env.fromElements(
 new WC(Ciao, 1),
 new WC(Hello, 1));
 
-Table table = tableEnv.toTable(input);
+Table table = tableEnv.fromDataSet(input);
 
 Table filtered = table
 .groupBy(word)
 .select(word.count as count, word)
 .filter(count = 2);
 
-DataSetWC result = tableEnv.toSet(filtered, WC.class);
+DataSetWC result = tableEnv.toDataSet(filtered, WC.class);
 {% endhighlight %}
 
 When using Java, the embedded DSL for specifying expressions cannot be used. 
Only String expressions

http://git-wip-us.apache.org/repos/asf/flink/blob/af0fee51/flink-staging/flink-table/src/main/scala/org/apache/flink/api/scala/table/TableConversions.scala
--
diff --git 
a/flink-staging/flink-table/src/main/scala/org/apache/flink/api/scala/table/TableConversions.scala
 
b/flink-staging/flink-table/src/main/scala/org/apache/flink/api/scala/table/TableConversions.scala
index b9c0a5e..4f2172e 100644
--- 
a/flink-staging/flink-table/src/main/scala/org/apache/flink/api/scala/table/TableConversions.scala
+++ 
b/flink-staging/flink-table/src/main/scala/org/apache/flink/api/scala/table/TableConversions.scala
@@ -33,14 +33,14 @@ class TableConversions(table: Table) {
   /**
* Converts the [[Table]] to a [[DataSet]].
*/
-  def toSet[T: TypeInformation]: DataSet[T] = {
+  def toDataSet[T: TypeInformation]: DataSet[T] = {
  new ScalaBatchTranslator().translate[T](table.operation)
   }
 
   /**
* Converts the [[Table]] to a [[DataStream]].
*/
-  def toStream[T: TypeInformation]: DataStream[T] = {
+  def toDataStream[T: TypeInformation]: DataStream[T] = {
 new ScalaStreamingTranslator().translate[T](table.operation)
   }
 }

http://git-wip-us.apache.org/repos/asf/flink/blob/af0fee51/flink-staging/flink-table/src/main/scala/org/apache/flink/api/table/Table.scala
--
diff --git 
a/flink-staging/flink-table/src/main/scala/org/apache/flink/api/table/Table.scala
 
b/flink-staging/flink-table/src/main/scala/org/apache/flink/api/table/Table.scala
index 83d5239..fdb125b 100644
--- 
a/flink-staging/flink-table/src/main/scala/org/apache/flink/api/table/Table.scala
+++ 
b/flink-staging/flink-table/src/main/scala/org/apache/flink/api/table/Table.scala
@@ -39,7 +39,7 @@ import org.apache.flink.api.table.plan._
  *   val table = set.toTable('a, 'b)
  *   ...
  *   val 

flink git commit: [FLINK-2061] [java api] Fix GenericCsvInputFormat skipping fields error with quoted string

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master 7cb35ea91 - 12cd15622


[FLINK-2061] [java api] Fix GenericCsvInputFormat skipping fields error with 
quoted string

This closes #734


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/12cd1562
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/12cd1562
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/12cd1562

Branch: refs/heads/master
Commit: 12cd15622afbbcbb40c9dd80151f4ca459a26922
Parents: 7cb35ea
Author: Chiwan Park chiwanp...@icloud.com
Authored: Wed May 27 22:24:59 2015 +0900
Committer: Fabian Hueske fhue...@apache.org
Committed: Fri May 29 21:34:25 2015 +0200

--
 .../api/common/io/GenericCsvInputFormat.java|  1 +
 .../flink/api/java/io/CsvInputFormatTest.java   | 32 
 2 files changed, 33 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/12cd1562/flink-core/src/main/java/org/apache/flink/api/common/io/GenericCsvInputFormat.java
--
diff --git 
a/flink-core/src/main/java/org/apache/flink/api/common/io/GenericCsvInputFormat.java
 
b/flink-core/src/main/java/org/apache/flink/api/common/io/GenericCsvInputFormat.java
index 1803a2b..b132ca2 100644
--- 
a/flink-core/src/main/java/org/apache/flink/api/common/io/GenericCsvInputFormat.java
+++ 
b/flink-core/src/main/java/org/apache/flink/api/common/io/GenericCsvInputFormat.java
@@ -444,6 +444,7 @@ public abstract class GenericCsvInputFormatOT extends 
DelimitedInputFormatOT
 
// quoted string parsing enabled and field is quoted
// search for ending quote character
+   i++;
while(i  limit  bytes[i] != quoteCharacter) {
i++;
}

http://git-wip-us.apache.org/repos/asf/flink/blob/12cd1562/flink-java/src/test/java/org/apache/flink/api/java/io/CsvInputFormatTest.java
--
diff --git 
a/flink-java/src/test/java/org/apache/flink/api/java/io/CsvInputFormatTest.java 
b/flink-java/src/test/java/org/apache/flink/api/java/io/CsvInputFormatTest.java
index 3d87984..11cd6da 100644
--- 
a/flink-java/src/test/java/org/apache/flink/api/java/io/CsvInputFormatTest.java
+++ 
b/flink-java/src/test/java/org/apache/flink/api/java/io/CsvInputFormatTest.java
@@ -981,6 +981,38 @@ public class CsvInputFormatTest {
}
}
 
+   @Test
+   public void testQuotedStringParsingWithIncludeFields() throws Exception 
{
+   final String fileContent = \20:41:52-1-3-2015\|\Re: 
Taskmanager memory error in Eclipse\| +
+   \Blahblah 
b...@blahblah.org\|\bla\|\blubb\;
+
+   final File tempFile = 
File.createTempFile(CsvReaderQuotedString, tmp);
+   tempFile.deleteOnExit();
+   tempFile.setWritable(true);
+
+   OutputStreamWriter writer = new OutputStreamWriter(new 
FileOutputStream(tempFile));
+   writer.write(fileContent);
+   writer.close();
+
+   TypeInformationTuple2String, String typeInfo = 
TupleTypeInfo.getBasicTupleTypeInfo(String.class, String.class);
+   CsvInputFormatTuple2String, String inputFormat = new 
CsvInputFormatTuple2String, String(new Path(tempFile.toURI().toString()), 
typeInfo);
+
+   inputFormat.enableQuotedStringParsing('');
+   inputFormat.setFieldDelimiter('|');
+   inputFormat.setDelimiter('\n');
+   inputFormat.setFields(new boolean[]{true, false, true}, new 
Class[]{String.class, String.class});
+
+   inputFormat.configure(new Configuration());
+   FileInputSplit[] splits = inputFormat.createInputSplits(1);
+
+   inputFormat.open(splits[0]);
+
+   Tuple2String, String record = inputFormat.nextRecord(new 
Tuple2String, String());
+
+   assertEquals(20:41:52-1-3-2015, record.f0);
+   assertEquals(Blahblah b...@blahblah.org, record.f1);
+   }
+
// 

// Custom types for testing
// 

flink git commit: [quickfix] Made recursive enumeration parameter flag public in FileInputFormat

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master eba6f9a2f - baeedba5d


[quickfix] Made recursive enumeration parameter flag public in FileInputFormat


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/baeedba5
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/baeedba5
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/baeedba5

Branch: refs/heads/master
Commit: baeedba5d1dbf43180dcec77f50dcf988edb105d
Parents: eba6f9a
Author: Fabian Hueske fhue...@apache.org
Authored: Tue May 26 17:07:11 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue May 26 17:07:34 2015 +0200

--
 .../main/java/org/apache/flink/api/common/io/FileInputFormat.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/baeedba5/flink-core/src/main/java/org/apache/flink/api/common/io/FileInputFormat.java
--
diff --git 
a/flink-core/src/main/java/org/apache/flink/api/common/io/FileInputFormat.java 
b/flink-core/src/main/java/org/apache/flink/api/common/io/FileInputFormat.java
index 1d86ece..6183778 100644
--- 
a/flink-core/src/main/java/org/apache/flink/api/common/io/FileInputFormat.java
+++ 
b/flink-core/src/main/java/org/apache/flink/api/common/io/FileInputFormat.java
@@ -886,7 +886,7 @@ public abstract class FileInputFormatOT implements 
InputFormatOT, FileInputSp
/**
 * The config parameter which defines whether input directories are 
recursively traversed.
 */
-   private static final String ENUMERATE_NESTED_FILES_FLAG = 
recursive.file.enumeration;
+   public static final String ENUMERATE_NESTED_FILES_FLAG = 
recursive.file.enumeration;


// --- Config Builder 
-

[1/2] flink git commit: [FLINK-2043] Change the KMeansDataGenerator to allow passing a custom path

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master 21207fd52 - 3586ced35


[FLINK-2043] Change the KMeansDataGenerator to allow passing a custom path

This closes #721


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/3586ced3
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/3586ced3
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/3586ced3

Branch: refs/heads/master
Commit: 3586ced3550ac036638a8dff011c01de99f9ed5e
Parents: 7164b2b
Author: Pietro Pinoli pietro.pin...@gmail.com
Authored: Sun May 24 13:35:35 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Wed May 27 00:39:08 2015 +0200

--
 .../clustering/util/KMeansDataGenerator.java| 23 ++--
 1 file changed, 12 insertions(+), 11 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/3586ced3/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/clustering/util/KMeansDataGenerator.java
--
diff --git 
a/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/clustering/util/KMeansDataGenerator.java
 
b/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/clustering/util/KMeansDataGenerator.java
index 897e0ca..2cb7457 100644
--- 
a/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/clustering/util/KMeansDataGenerator.java
+++ 
b/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/clustering/util/KMeansDataGenerator.java
@@ -52,14 +52,15 @@ public class KMeansDataGenerator {
 * p
 * The generator creates to files:
 * ul
-* licode{tmp.dir}/points/code for the data points
-* licode{tmp.dir}/centers/code for the cluster centers
+* licodelt; output-path gt;/points/code for the data points
+* licodelt; output-path gt;/centers/code for the cluster 
centers
 * /ul 
 * 
 * @param args 
 * ol
 * liInt: Number of data points
 * liInt: Number of cluster centers
+* libOptional/b String: Output path, default value is {tmp.dir}
 * libOptional/b Double: Standard deviation of data points
 * libOptional/b Double: Value range of cluster centers
 * libOptional/b Long: Random seed
@@ -69,20 +70,20 @@ public class KMeansDataGenerator {
 
// check parameter count
if (args.length  2) {
-   System.out.println(KMeansDataGenerator 
numberOfDataPoints numberOfClusterCenters [relative stddev] [centroid 
range] [seed]);
+   System.out.println(KMeansDataGenerator 
numberOfDataPoints numberOfClusterCenters [output-path] [relative 
stddev] [centroid range] [seed]);
System.exit(1);
}
 
// parse parameters
final int numDataPoints = Integer.parseInt(args[0]);
final int k = Integer.parseInt(args[1]);
-   final double stddev = args.length  2 ? 
Double.parseDouble(args[2]) : RELATIVE_STDDEV;
-   final double range = args.length  3 ? 
Double.parseDouble(args[4]) : DEFAULT_VALUE_RANGE;
-   final long firstSeed = args.length  4 ? 
Long.parseLong(args[4]) : DEFAULT_SEED;
+   final String outDir = args.length  2 ? args[2] : 
System.getProperty(java.io.tmpdir);
+   final double stddev = args.length  3 ? 
Double.parseDouble(args[3]) : RELATIVE_STDDEV;
+   final double range = args.length  4 ? 
Double.parseDouble(args[4]) : DEFAULT_VALUE_RANGE;
+   final long firstSeed = args.length  5 ? 
Long.parseLong(args[5]) : DEFAULT_SEED;

final double absoluteStdDev = stddev * range;
final Random random = new Random(firstSeed);
-   final String tmpDir = System.getProperty(java.io.tmpdir);

// the means around which data points are distributed
final double[][] means = uniformRandomCenters(random, k, 
DIMENSIONALITY, range);
@@ -90,7 +91,7 @@ public class KMeansDataGenerator {
// write the points out
BufferedWriter pointsOut = null;
try {
-   pointsOut = new BufferedWriter(new FileWriter(new 
File(tmpDir+/+POINTS_FILE)));
+   pointsOut = new BufferedWriter(new FileWriter(new 
File(outDir+/+POINTS_FILE)));
StringBuilder buffer = new StringBuilder();

double[] point = new double[DIMENSIONALITY];
@@ -115,7 +116,7 @@ public class KMeansDataGenerator {
// write the uniformly distributed centers to a 

[2/2] flink git commit: [FLINK-1848] Fix for file paths with Windows drive letters

[fhueske]

[FLINK-1848] Fix for file paths with Windows drive letters

This closes #712


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/7164b2b6
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/7164b2b6
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/7164b2b6

Branch: refs/heads/master
Commit: 7164b2b643985b99c6688b62174de42a71deb71b
Parents: 21207fd
Author: Fabian Hueske fhue...@apache.org
Authored: Thu May 21 22:24:12 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Wed May 27 00:39:08 2015 +0200

--
 .../src/main/java/org/apache/flink/core/fs/Path.java|  6 +-
 .../org/apache/flink/core/fs/local/LocalFileSystem.java |  7 ++-
 .../test/java/org/apache/flink/core/fs/PathTest.java| 12 
 3 files changed, 23 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/7164b2b6/flink-core/src/main/java/org/apache/flink/core/fs/Path.java
--
diff --git a/flink-core/src/main/java/org/apache/flink/core/fs/Path.java 
b/flink-core/src/main/java/org/apache/flink/core/fs/Path.java
index 75155eb..c47bc0d 100644
--- a/flink-core/src/main/java/org/apache/flink/core/fs/Path.java
+++ b/flink-core/src/main/java/org/apache/flink/core/fs/Path.java
@@ -260,7 +260,11 @@ public class Path implements IOReadableWritable, 
Serializable {
path = path.replaceAll(/+, /);
 
// remove tailing separator
-   if(!path.equals(SEPARATOR)  path.endsWith(SEPARATOR)) {
+   if(!path.equals(SEPARATOR)// UNIX root 
path
+   !path.matches(/\\p{Alpha}+:/)   // Windows 
root path
+   path.endsWith(SEPARATOR))
+   {
+   // remove tailing slash
path = path.substring(0, path.length() - 
SEPARATOR.length());
}
 

http://git-wip-us.apache.org/repos/asf/flink/blob/7164b2b6/flink-core/src/main/java/org/apache/flink/core/fs/local/LocalFileSystem.java
--
diff --git 
a/flink-core/src/main/java/org/apache/flink/core/fs/local/LocalFileSystem.java 
b/flink-core/src/main/java/org/apache/flink/core/fs/local/LocalFileSystem.java
index 2313a41..9dd9e30 100644
--- 
a/flink-core/src/main/java/org/apache/flink/core/fs/local/LocalFileSystem.java
+++ 
b/flink-core/src/main/java/org/apache/flink/core/fs/local/LocalFileSystem.java
@@ -228,8 +228,13 @@ public class LocalFileSystem extends FileSystem {
 */
public boolean mkdirs(final Path f) throws IOException {
 
-   final Path parent = f.getParent();
final File p2f = pathToFile(f);
+
+   if(p2f.isDirectory()) {
+   return true;
+   }
+
+   final Path parent = f.getParent();
return (parent == null || mkdirs(parent))  (p2f.mkdir() || 
p2f.isDirectory());
}
 

http://git-wip-us.apache.org/repos/asf/flink/blob/7164b2b6/flink-core/src/test/java/org/apache/flink/core/fs/PathTest.java
--
diff --git a/flink-core/src/test/java/org/apache/flink/core/fs/PathTest.java 
b/flink-core/src/test/java/org/apache/flink/core/fs/PathTest.java
index 8fa2cea..66816ad 100644
--- a/flink-core/src/test/java/org/apache/flink/core/fs/PathTest.java
+++ b/flink-core/src/test/java/org/apache/flink/core/fs/PathTest.java
@@ -117,12 +117,24 @@ public class PathTest {
p = new Path(y:/my/abs/windows/path);
assertTrue(p.isAbsolute());
 
+   p = new Path(/y:/my/abs/windows/path);
+   assertTrue(p.isAbsolute());
+
p = new Path(b:\\my\\abs\\windows\\path);
assertTrue(p.isAbsolute());
 
+   p = new Path(/c:/my/dir);
+   assertTrue(p.isAbsolute());
+
+   p = new Path(/C:/);
+   assertTrue(p.isAbsolute());
+
p = new Path(C:);
assertFalse(p.isAbsolute());
 
+   p = new Path(C:/);
+   assertTrue(p.isAbsolute());
+
p = new Path(C:my\\relative\\path);
assertFalse(p.isAbsolute());
 

flink git commit: [FLINK-2095][docs] Added missing screenshots to webclient documentation

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master 949ff8dae - 2859f59f8


[FLINK-2095][docs] Added missing screenshots to webclient documentation

This closes #976


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/2859f59f
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/2859f59f
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/2859f59f

Branch: refs/heads/master
Commit: 2859f59f84aae46791b51493f8777679de496a86
Parents: 949ff8d
Author: mjsax mj...@informatik.hu-berlin.de
Authored: Mon Aug 3 22:04:35 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue Aug 4 00:26:36 2015 +0200

--
 docs/apis/web_client.md | 55 +---
 1 file changed, 37 insertions(+), 18 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/2859f59f/docs/apis/web_client.md
--
diff --git a/docs/apis/web_client.md b/docs/apis/web_client.md
index 8e3c84a..4749d90 100644
--- a/docs/apis/web_client.md
+++ b/docs/apis/web_client.md
@@ -46,16 +46,27 @@ The web interface provides two views:
 
 ### Job View
 
-The interface starts serving the job view. 
-
-You can **upload** a Flink program as a jar file. To **execute** an uploaded 
program:
-
-* select it from the job list on the left, 
-* enter (optional) execution options in the *Flink Options* field (bottom 
left),
-* enter (optional) program arguments in the *Program Arguments* field 
(bottom left), and
-* click on the *Run Job* button (bottom right).
-
-If the *“Show optimizer plan”* option is enabled (default), the *plan 
view* is display next, otherwise the job is directly submitted to the 
JobManager for execution.
+div class=row
+   div class=col-md-4
+   The interface starts serving the job view. br
+   br
+   You can strongupload/strong a Flink program as a jar file. 
br
+br
+   To strongexecute/strong an uploaded program:
+   ul
+   liselect it from the job list on the left,/li
+   lienter (optional) execution options in the 
emFlink Options/em field (bottom left),/li
+   lienter (optional) program arguments in the 
emProgram Arguments/em field (bottom left), and/li
+   liclick on the emRun Job/em button (bottom 
right)./li
+   /ul
+   If the em“Show optimizer plan”/em option is enabled 
(default), the emplan view/em is display next, otherwise the job is 
directly submitted to the JobManager for execution.
+   /div
+   div class=col-md-8
+   a data-lightbox=compiler href={{ site.baseurl 
}}/page/img/webclient_job_view.png data-lightbox=example-1img 
class=img-responsive src={{ site.baseurl }}/page/img/webclient_job_view.png 
//a
+   /div
+/div
+
+br
 
 The web interface can also handle multiple Flink jobs within a single jar 
file. To use this feature, package all required class files of all jobs into a 
single jar and specify the entry classes for each job as comma-separated-values 
in *program-class* argument within the jar's manifest file. The job view 
displays each entry class and you can pick any of them to preview the plan 
and/or submit the job to the JobManager. In case the jar's manifest file does 
not specify any entry class, you can specify it in the options field as:
 
@@ -73,11 +84,19 @@ Furthermore, for each entry class implementing 
```ProgramDescription``` interfac
 
 ### Plan View
 
-The plan view shows the optimized execution plan of the submitted program in 
the upper half of the page. The bottom part of the page displays detailed 
information about the currently selected plan operator including:
-
-* the chosen shipping strategies (local forward, hash partition, range 
partition, broadcast, ...),
-* the chosen local strategy (sort, hash join, merge join, ...),
-* inferred data properties (partitioning, grouping, sorting), and 
-* used optimizer estimates (data size, I/O and network costs, ...).
-
-To submit the job for execution, click again on the *Run Job* button in the 
bottom right.
+div class=row
+   div class=col-md-4
+   The plan view shows the optimized execution plan of the 
submitted program in the upper half of the page. The bottom part of the page 
displays detailed information about the currently selected plan operator 
including:
+   ul
+   lithe chosen shipping strategies (local forward, hash 
partition, range partition, broadcast, ...),/li
+   lithe chosen local strategy (sort, hash join, merge 
join, ...),/li
+   liinferred data properties (partitioning, grouping, 
sorting), and /li
+ 

flink git commit: [FLINK-2422] [web client] Added explicit link in case browser is not redirecting properly

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master 2859f59f8 - bb5e61d24


[FLINK-2422] [web client] Added explicit link in case browser is not 
redirecting properly

This closes #946


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/bb5e61d2
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/bb5e61d2
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/bb5e61d2

Branch: refs/heads/master
Commit: bb5e61d242ad272a16b487a0cf461774b73e648b
Parents: 2859f59
Author: buzdin buz...@gmail.com
Authored: Tue Jul 28 16:45:25 2015 +0100
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue Aug 4 00:44:03 2015 +0200

--
 flink-clients/src/main/resources/web-docs/index.html | 2 ++
 1 file changed, 2 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/bb5e61d2/flink-clients/src/main/resources/web-docs/index.html
--
diff --git a/flink-clients/src/main/resources/web-docs/index.html 
b/flink-clients/src/main/resources/web-docs/index.html
index bfba666..e4275d8 100644
--- a/flink-clients/src/main/resources/web-docs/index.html
+++ b/flink-clients/src/main/resources/web-docs/index.html
@@ -22,5 +22,7 @@ under the License.
 meta HTTP-EQUIV=REFRESH content=0; url=launch.html
 /head
 body
+pYou are being redirected.../p
+pClick a href=launch.htmlhere/a if nothing happens./p
 /body
 /html

flink git commit: [FLINK-2422] [web client] Added explicit link in case browser is not redirecting properly

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/release-0.9 0186a979f - acb6ea81a


[FLINK-2422] [web client] Added explicit link in case browser is not 
redirecting properly


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/acb6ea81
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/acb6ea81
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/acb6ea81

Branch: refs/heads/release-0.9
Commit: acb6ea81abb7588549f9a1bd3dae5e555886a7c9
Parents: 0186a97
Author: buzdin buz...@gmail.com
Authored: Tue Jul 28 16:45:25 2015 +0100
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue Aug 4 00:47:30 2015 +0200

--
 flink-clients/src/main/resources/web-docs/index.html | 2 ++
 1 file changed, 2 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/acb6ea81/flink-clients/src/main/resources/web-docs/index.html
--
diff --git a/flink-clients/src/main/resources/web-docs/index.html 
b/flink-clients/src/main/resources/web-docs/index.html
index bfba666..e4275d8 100644
--- a/flink-clients/src/main/resources/web-docs/index.html
+++ b/flink-clients/src/main/resources/web-docs/index.html
@@ -22,5 +22,7 @@ under the License.
 meta HTTP-EQUIV=REFRESH content=0; url=launch.html
 /head
 body
+pYou are being redirected.../p
+pClick a href=launch.htmlhere/a if nothing happens./p
 /body
 /html

[2/4] flink git commit: [FLINK-1882] Removed RemotedCollector classes

[fhueske]

[FLINK-1882] Removed RemotedCollector classes

This closes #985


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/100e8c5f
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/100e8c5f
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/100e8c5f

Branch: refs/heads/master
Commit: 100e8c5ff9f6d25b3d5db326a5f31b9c4432e334
Parents: 5546a1e
Author: zentol s.mo...@web.de
Authored: Sat Jul 25 15:17:55 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Wed Aug 5 14:46:35 2015 +0200

--
 .../RemoteCollectorOutputFormatExample.java | 114 --
 .../flink/api/java/io/RemoteCollector.java  |  46 
 .../api/java/io/RemoteCollectorConsumer.java|  26 ---
 .../flink/api/java/io/RemoteCollectorImpl.java  | 228 ---
 .../java/io/RemoteCollectorOutputFormat.java| 175 --
 5 files changed, 589 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/100e8c5f/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/misc/RemoteCollectorOutputFormatExample.java
--
diff --git 
a/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/misc/RemoteCollectorOutputFormatExample.java
 
b/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/misc/RemoteCollectorOutputFormatExample.java
deleted file mode 100644
index f524718..000
--- 
a/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/misc/RemoteCollectorOutputFormatExample.java
+++ /dev/null
@@ -1,114 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.examples.java.misc;
-
-import java.util.HashSet;
-import java.util.Set;
-
-import org.apache.flink.api.common.functions.FlatMapFunction;
-import org.apache.flink.api.java.DataSet;
-import org.apache.flink.api.java.ExecutionEnvironment;
-import org.apache.flink.api.java.aggregation.Aggregations;
-import org.apache.flink.api.java.io.RemoteCollectorConsumer;
-import org.apache.flink.api.java.io.RemoteCollectorImpl;
-import org.apache.flink.api.java.tuple.Tuple2;
-import org.apache.flink.util.Collector;
-
-/**
- * Implements the WordCount program that computes a simple word occurrence
- * histogram over some sample data and collects the results with an
- * implementation of a {@link RemoteCollectorConsumer}.
- */
-@SuppressWarnings(serial)
-public class RemoteCollectorOutputFormatExample {
-
-   public static void main(String[] args) throws Exception {
-
-   /**
-* We create a remote {@link ExecutionEnvironment} here, 
because this
-* OutputFormat is designed for use in a distributed setting. 
For local
-* use you should consider using the {@link 
LocalCollectionOutputFormat
-* T}.
-*/
-   final ExecutionEnvironment env = 
ExecutionEnvironment.createRemoteEnvironment(remote, 6124,
-   /path/to/your/file.jar);
-
-   // get input data
-   DataSetString text = env.fromElements(
-   To be, or not to be,--that is the question:--,
-   Whether 'tis nobler in the mind to suffer,
-   The slings and arrows of outrageous fortune,
-   Or to take arms against a sea of troubles,);
-
-   DataSetTuple2String, Integer counts =
-   // split up the lines in pairs (2-tuples) containing: (word,1)
-   text.flatMap(new LineSplitter())
-   // group by the tuple field 0 and sum up tuple field 1
-   .groupBy(0).aggregate(Aggregations.SUM, 1);
-
-   // emit result
-   RemoteCollectorImpl.collectLocal(counts,
-   new RemoteCollectorConsumerTuple2String, 
Integer() {
-   // user defined 

[3/4] flink git commit: [FLINK-1680] Remove Tachyon test and rename Maven module to flink-fs-tests

[fhueske]

[FLINK-1680] Remove Tachyon test and rename Maven module to flink-fs-tests

This closes #987


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/fb7e6342
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/fb7e6342
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/fb7e6342

Branch: refs/heads/master
Commit: fb7e6342211d116a2db13933241d3546bbf8d4e8
Parents: 0aa6f0c
Author: Robert Metzger rmetz...@apache.org
Authored: Tue Aug 4 13:35:12 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Wed Aug 5 14:46:35 2015 +0200

--
 flink-staging/flink-fs-tests/pom.xml|  78 +
 .../flink/tachyon/FileStateHandleTest.java  | 126 ++
 .../java/org/apache/flink/tachyon/HDFSTest.java | 174 +++
 .../src/test/resources/log4j.properties |  31 
 flink-staging/flink-tachyon/pom.xml | 113 
 .../flink/tachyon/FileStateHandleTest.java  | 126 --
 .../java/org/apache/flink/tachyon/HDFSTest.java | 157 -
 .../tachyon/TachyonFileSystemWrapperTest.java   | 167 --
 .../src/test/resources/log4j.properties |  31 
 .../src/test/resources/tachyonHadoopConf.xml|  28 ---
 flink-staging/pom.xml   |   4 +-
 11 files changed, 411 insertions(+), 624 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/fb7e6342/flink-staging/flink-fs-tests/pom.xml
--
diff --git a/flink-staging/flink-fs-tests/pom.xml 
b/flink-staging/flink-fs-tests/pom.xml
new file mode 100644
index 000..fe1abb3
--- /dev/null
+++ b/flink-staging/flink-fs-tests/pom.xml
@@ -0,0 +1,78 @@
+?xml version=1.0 encoding=UTF-8?
+!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+License); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+  http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+--
+project xmlns=http://maven.apache.org/POM/4.0.0; 
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance;
+   xsi:schemaLocation=http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/maven-v4_0_0.xsd;
+
+   modelVersion4.0.0/modelVersion
+
+   parent
+   groupIdorg.apache.flink/groupId
+   artifactIdflink-staging/artifactId
+   version0.10-SNAPSHOT/version
+   relativePath../relativePath
+   /parent
+
+   artifactIdflink-fs-tests/artifactId
+   nameflink-fs-tests/name
+
+   packagingjar/packaging
+
+   !--
+   This is a Hadoop2 only flink module.
+   --
+   dependencies
+   dependency
+   groupIdorg.apache.flink/groupId
+   artifactId${shading-artifact.name}/artifactId
+   version${project.version}/version
+   scopetest/scope
+   /dependency
+   dependency
+   groupIdorg.apache.flink/groupId
+   artifactIdflink-core/artifactId
+   version${project.version}/version
+   scopetest/scope
+   /dependency
+   dependency
+   groupIdorg.apache.flink/groupId
+   artifactIdflink-java-examples/artifactId
+   version${project.version}/version
+   scopetest/scope
+   /dependency
+   dependency
+   groupIdorg.apache.flink/groupId
+   artifactIdflink-avro/artifactId
+   version${project.version}/version
+   scopetest/scope
+   /dependency
+   dependency
+   groupIdorg.apache.hadoop/groupId
+   artifactIdhadoop-hdfs/artifactId
+   scopetest/scope
+   typetest-jar/type
+   
version${hadoop.version}/version!--$NO-MVN-MAN-VER$--
+   /dependency
+   dependency
+   groupIdorg.apache.hadoop/groupId
+   artifactIdhadoop-common/artifactId
+   

[4/4] flink git commit: [FLINK-2467] Example WordCountStorm.jar is not packaged correctly - fixed assembly xml file

[fhueske]

[FLINK-2467] Example WordCountStorm.jar is not packaged correctly
 - fixed assembly xml file

This closes #974


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/0aa6f0cb
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/0aa6f0cb
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/0aa6f0cb

Branch: refs/heads/master
Commit: 0aa6f0cb7b3510e0e9aa938411b6db77f7f7505e
Parents: 100e8c5
Author: mjsax mj...@informatik.hu-berlin.de
Authored: Mon Aug 3 15:51:46 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Wed Aug 5 14:46:35 2015 +0200

--
 .../src/assembly/word-count-storm.xml| 8 
 1 file changed, 4 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/0aa6f0cb/flink-contrib/flink-storm-compatibility/flink-storm-compatibility-examples/src/assembly/word-count-storm.xml
--
diff --git 
a/flink-contrib/flink-storm-compatibility/flink-storm-compatibility-examples/src/assembly/word-count-storm.xml
 
b/flink-contrib/flink-storm-compatibility/flink-storm-compatibility-examples/src/assembly/word-count-storm.xml
index 9721115..96ac429 100644
--- 
a/flink-contrib/flink-storm-compatibility/flink-storm-compatibility-examples/src/assembly/word-count-storm.xml
+++ 
b/flink-contrib/flink-storm-compatibility/flink-storm-compatibility-examples/src/assembly/word-count-storm.xml
@@ -34,9 +34,9 @@ under the License.
 outputDirectory//outputDirectory
 unpacktrue/unpack
 includes
-!-- need to be added explicitly to get 'defaults.yaml' --
 includeorg.apache.storm:storm-core:jar/include
-includeorg.apache.flink:flink-storm-examples:jar/include
+
includeorg.apache.flink:flink-storm-compatibility-core:jar/include
+
includeorg.apache.flink:flink-storm-compatibility-examples:jar/include
 /includes
 unpackOptions
 includes
@@ -60,10 +60,10 @@ under the License.
 
includeorg/apache/flink/stormcompatibility/api/*.class/include
 
includeorg/apache/flink/stormcompatibility/wrappers/*.class/include
 !-- Word Count --
-
includeorg/apache/flink/stormcompatibility/wordcount/StormWordCountRemoteBySubmitter.class
-/include
+
includeorg/apache/flink/stormcompatibility/wordcount/StormWordCountRemoteBySubmitter.class/include
 
includeorg/apache/flink/stormcompatibility/wordcount/WordCountTopology.class/include
 
includeorg/apache/flink/stormcompatibility/wordcount/stormoperators/*.class/include
+
includeorg/apache/flink/stormcompatibility/util/*.class/include
 
includeorg/apache/flink/examples/java/wordcount/util/WordCountData.class/include
 /includes
 /unpackOptions

[1/4] flink git commit: [FLINK-2447] [java api] TypeExtractor returns wrong type info when a Tuple has two fields of the same POJO type

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master 941ac6dfd - fb7e63422


[FLINK-2447] [java api] TypeExtractor returns wrong type info when a Tuple has 
two fields of the same POJO type

This closes #986


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/5546a1ef
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/5546a1ef
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/5546a1ef

Branch: refs/heads/master
Commit: 5546a1efabcc9c5f500abdb4e38a5cc05d35980a
Parents: 941ac6d
Author: twalthr twal...@apache.org
Authored: Tue Aug 4 15:30:28 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Wed Aug 5 12:00:56 2015 +0200

--
 .../flink/api/java/typeutils/TypeExtractor.java | 53 
 .../type/extractor/PojoTypeExtractionTest.java  | 63 
 2 files changed, 104 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/5546a1ef/flink-java/src/main/java/org/apache/flink/api/java/typeutils/TypeExtractor.java
--
diff --git 
a/flink-java/src/main/java/org/apache/flink/api/java/typeutils/TypeExtractor.java
 
b/flink-java/src/main/java/org/apache/flink/api/java/typeutils/TypeExtractor.java
index 41644f9..1ae8d3d 100644
--- 
a/flink-java/src/main/java/org/apache/flink/api/java/typeutils/TypeExtractor.java
+++ 
b/flink-java/src/main/java/org/apache/flink/api/java/typeutils/TypeExtractor.java
@@ -26,9 +26,7 @@ import java.lang.reflect.ParameterizedType;
 import java.lang.reflect.Type;
 import java.lang.reflect.TypeVariable;
 import java.util.ArrayList;
-import java.util.HashSet;
 import java.util.List;
-import java.util.Set;
 
 import org.apache.avro.specific.SpecificRecordBase;
 import org.apache.flink.api.common.functions.CoGroupFunction;
@@ -66,15 +64,33 @@ import com.google.common.base.Preconditions;
  * functions.
  */
 public class TypeExtractor {
+
+   /*
+* NOTE: Most methods of the TypeExtractor work with a so-called 
typeHierarchy.
+* The type hierarchy describes all types (Classes, ParameterizedTypes, 
TypeVariables etc. ) and intermediate
+* types from a given type of a function or type (e.g. MyMapper, 
Tuple2) until a current type
+* (depends on the method, e.g. MyPojoFieldType).
+*
+* Thus, it fully qualifies types until tuple/POJO field level.
+*
+* A typical typeHierarchy could look like:
+*
+* UDF: MyMapFunction.class
+* top-level UDF: MyMapFunctionBase.class
+* RichMapFunction: RichMapFunction.class
+* MapFunction: MapFunction.class
+* Function's OUT: Tuple1MyPojo
+* user-defined POJO: MyPojo.class
+* user-defined top-level POJO: MyPojoBase.class
+* POJO field: Tuple1String
+* Field type: String.class
+*
+*/

private static final Logger LOG = 
LoggerFactory.getLogger(TypeExtractor.class);
 
-   // We need this to detect recursive types and not get caught
-   // in an endless recursion
-   private SetClass? alreadySeen;
-
protected TypeExtractor() {
-   alreadySeen = new HashSetClass?();
+   // only create instances for special use cases
}
 
// 

@@ -416,10 +432,12 @@ public class TypeExtractor {

TypeInformation?[] tupleSubTypes = new 
TypeInformation?[subtypes.length];
for (int i = 0; i  subtypes.length; i++) {
+   ArrayListType subTypeHierarchy = new 
ArrayListType(typeHierarchy);
+   subTypeHierarchy.add(subtypes[i]);
// sub type could not be determined with 
materializing
// try to derive the type info of the 
TypeVariable from the immediate base child input as a last attempt
if (subtypes[i] instanceof TypeVariable?) {
-   tupleSubTypes[i] = 
createTypeInfoFromInputs((TypeVariable?) subtypes[i], typeHierarchy, in1Type, 
in2Type);
+   tupleSubTypes[i] = 
createTypeInfoFromInputs((TypeVariable?) subtypes[i], subTypeHierarchy, 
in1Type, in2Type);

// variable could not be determined
if (tupleSubTypes[i] == null) {
@@ -430,7 +448,7 @@ public class TypeExtractor {
+ all 
variables in the return type can be deduced from the input type(s).);
  

flink git commit: [FLINK-2447] [java api] TypeExtractor returns wrong type info when a Tuple has two fields of the same POJO type

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/release-0.9 80d3478c0 - ec3b98327


[FLINK-2447] [java api] TypeExtractor returns wrong type info when a Tuple has 
two fields of the same POJO type


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/ec3b9832
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/ec3b9832
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/ec3b9832

Branch: refs/heads/release-0.9
Commit: ec3b983276e85e0baed98ffa8acf6709f20032de
Parents: 80d3478
Author: twalthr twal...@apache.org
Authored: Tue Aug 4 15:30:28 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Wed Aug 5 16:25:32 2015 +0200

--
 .../flink/api/java/typeutils/TypeExtractor.java | 53 
 .../type/extractor/PojoTypeExtractionTest.java  | 63 
 2 files changed, 104 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/ec3b9832/flink-java/src/main/java/org/apache/flink/api/java/typeutils/TypeExtractor.java
--
diff --git 
a/flink-java/src/main/java/org/apache/flink/api/java/typeutils/TypeExtractor.java
 
b/flink-java/src/main/java/org/apache/flink/api/java/typeutils/TypeExtractor.java
index 41644f9..1ae8d3d 100644
--- 
a/flink-java/src/main/java/org/apache/flink/api/java/typeutils/TypeExtractor.java
+++ 
b/flink-java/src/main/java/org/apache/flink/api/java/typeutils/TypeExtractor.java
@@ -26,9 +26,7 @@ import java.lang.reflect.ParameterizedType;
 import java.lang.reflect.Type;
 import java.lang.reflect.TypeVariable;
 import java.util.ArrayList;
-import java.util.HashSet;
 import java.util.List;
-import java.util.Set;
 
 import org.apache.avro.specific.SpecificRecordBase;
 import org.apache.flink.api.common.functions.CoGroupFunction;
@@ -66,15 +64,33 @@ import com.google.common.base.Preconditions;
  * functions.
  */
 public class TypeExtractor {
+
+   /*
+* NOTE: Most methods of the TypeExtractor work with a so-called 
typeHierarchy.
+* The type hierarchy describes all types (Classes, ParameterizedTypes, 
TypeVariables etc. ) and intermediate
+* types from a given type of a function or type (e.g. MyMapper, 
Tuple2) until a current type
+* (depends on the method, e.g. MyPojoFieldType).
+*
+* Thus, it fully qualifies types until tuple/POJO field level.
+*
+* A typical typeHierarchy could look like:
+*
+* UDF: MyMapFunction.class
+* top-level UDF: MyMapFunctionBase.class
+* RichMapFunction: RichMapFunction.class
+* MapFunction: MapFunction.class
+* Function's OUT: Tuple1MyPojo
+* user-defined POJO: MyPojo.class
+* user-defined top-level POJO: MyPojoBase.class
+* POJO field: Tuple1String
+* Field type: String.class
+*
+*/

private static final Logger LOG = 
LoggerFactory.getLogger(TypeExtractor.class);
 
-   // We need this to detect recursive types and not get caught
-   // in an endless recursion
-   private SetClass? alreadySeen;
-
protected TypeExtractor() {
-   alreadySeen = new HashSetClass?();
+   // only create instances for special use cases
}
 
// 

@@ -416,10 +432,12 @@ public class TypeExtractor {

TypeInformation?[] tupleSubTypes = new 
TypeInformation?[subtypes.length];
for (int i = 0; i  subtypes.length; i++) {
+   ArrayListType subTypeHierarchy = new 
ArrayListType(typeHierarchy);
+   subTypeHierarchy.add(subtypes[i]);
// sub type could not be determined with 
materializing
// try to derive the type info of the 
TypeVariable from the immediate base child input as a last attempt
if (subtypes[i] instanceof TypeVariable?) {
-   tupleSubTypes[i] = 
createTypeInfoFromInputs((TypeVariable?) subtypes[i], typeHierarchy, in1Type, 
in2Type);
+   tupleSubTypes[i] = 
createTypeInfoFromInputs((TypeVariable?) subtypes[i], subTypeHierarchy, 
in1Type, in2Type);

// variable could not be determined
if (tupleSubTypes[i] == null) {
@@ -430,7 +448,7 @@ public class TypeExtractor {
+ all 
variables in the return type can be deduced from the input type(s).);
  

flink git commit: Revert API-breaking change: [FLINK-2238][api] Generalized fromCollection(Seq) to fromCollection(Iterable)

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/release-0.9 012dd0503 - 0186a979f


Revert API-breaking change: [FLINK-2238][api] Generalized fromCollection(Seq) 
to fromCollection(Iterable)

This reverts commit 3e31ea647c76c85cd2371427b5735cb2d9a84266.


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/0186a979
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/0186a979
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/0186a979

Branch: refs/heads/release-0.9
Commit: 0186a979fbb01a5e17cfaa9b1f128cae2015889d
Parents: 012dd05
Author: Fabian Hueske fhue...@apache.org
Authored: Mon Aug 3 12:04:06 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Mon Aug 3 12:05:06 2015 +0200

--
 .../scala/org/apache/flink/api/scala/ExecutionEnvironment.scala  | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/0186a979/flink-scala/src/main/scala/org/apache/flink/api/scala/ExecutionEnvironment.scala
--
diff --git 
a/flink-scala/src/main/scala/org/apache/flink/api/scala/ExecutionEnvironment.scala
 
b/flink-scala/src/main/scala/org/apache/flink/api/scala/ExecutionEnvironment.scala
index d53c54c..28e8458 100644
--- 
a/flink-scala/src/main/scala/org/apache/flink/api/scala/ExecutionEnvironment.scala
+++ 
b/flink-scala/src/main/scala/org/apache/flink/api/scala/ExecutionEnvironment.scala
@@ -455,13 +455,13 @@ class ExecutionEnvironment(javaEnv: JavaEnv) {
   }
 
   /**
-   * Creates a DataSet from the given non-empty [[Iterable]].
+   * Creates a DataSet from the given non-empty [[Seq]].
*
* Note that this operation will result in a non-parallel data source, i.e. 
a data source with
* a parallelism of one.
*/
   def fromCollection[T: ClassTag : TypeInformation](
-  data: Iterable[T]): DataSet[T] = {
+  data: Seq[T]): DataSet[T] = {
 require(data != null, Data must not be null.)
 
 val typeInfo = implicitly[TypeInformation[T]]

[4/5] flink git commit: [FLINK-2105] [tests] Move duplicate utility classes to testutil package

[fhueske]

[FLINK-2105] [tests] Move duplicate utility classes to testutil package


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/df9f4819
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/df9f4819
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/df9f4819

Branch: refs/heads/master
Commit: df9f4819b9368600c7531dbf4d4ec42c1cddea8f
Parents: db0b008
Author: r-pogalz r.pog...@campus.tu-berlin.de
Authored: Mon Aug 3 12:59:01 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue Aug 4 21:35:27 2015 +0200

--
 .../flink/runtime/operators/MatchDriver.java|   8 +-
 .../sort/AbstractMergeInnerJoinIterator.java| 108 ++
 .../sort/AbstractMergeMatchIterator.java| 107 --
 .../sort/NonReusingMergeInnerJoinIterator.java  |  59 +++
 .../sort/NonReusingMergeMatchIterator.java  |  59 ---
 .../sort/ReusingMergeInnerJoinIterator.java |  64 
 .../sort/ReusingMergeMatchIterator.java |  64 
 ...ReusingSortMergeInnerJoinIteratorITCase.java | 318 
 .../NonReusingSortMergeMatchIteratorITCase.java | 371 ---
 ...ReusingSortMergeInnerJoinIteratorITCase.java | 318 
 .../ReusingSortMergeMatchIteratorITCase.java| 371 ---
 .../operators/testutils/CollectionIterator.java |  61 +++
 .../runtime/operators/testutils/Match.java  |  63 
 .../testutils/MatchRemovingMatcher.java |  58 +++
 .../testutils/SimpleTupleJoinFunction.java  |  41 ++
 .../operators/util/HashVsSortMiniBenchmark.java |   6 +-
 16 files changed, 1097 insertions(+), 979 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/df9f4819/flink-runtime/src/main/java/org/apache/flink/runtime/operators/MatchDriver.java
--
diff --git 
a/flink-runtime/src/main/java/org/apache/flink/runtime/operators/MatchDriver.java
 
b/flink-runtime/src/main/java/org/apache/flink/runtime/operators/MatchDriver.java
index 0381aab..e54fca5 100644
--- 
a/flink-runtime/src/main/java/org/apache/flink/runtime/operators/MatchDriver.java
+++ 
b/flink-runtime/src/main/java/org/apache/flink/runtime/operators/MatchDriver.java
@@ -22,7 +22,7 @@ package org.apache.flink.runtime.operators;
 import org.apache.flink.api.common.ExecutionConfig;
 import 
org.apache.flink.runtime.operators.hash.NonReusingBuildFirstHashMatchIterator;
 import 
org.apache.flink.runtime.operators.hash.NonReusingBuildSecondHashMatchIterator;
-import org.apache.flink.runtime.operators.sort.NonReusingMergeMatchIterator;
+import 
org.apache.flink.runtime.operators.sort.NonReusingMergeInnerJoinIterator;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 import org.apache.flink.api.common.functions.FlatJoinFunction;
@@ -33,7 +33,7 @@ import org.apache.flink.runtime.io.disk.iomanager.IOManager;
 import org.apache.flink.runtime.memorymanager.MemoryManager;
 import 
org.apache.flink.runtime.operators.hash.ReusingBuildFirstHashMatchIterator;
 import 
org.apache.flink.runtime.operators.hash.ReusingBuildSecondHashMatchIterator;
-import org.apache.flink.runtime.operators.sort.ReusingMergeMatchIterator;
+import org.apache.flink.runtime.operators.sort.ReusingMergeInnerJoinIterator;
 import org.apache.flink.runtime.operators.util.JoinTaskIterator;
 import org.apache.flink.runtime.operators.util.TaskConfig;
 import org.apache.flink.util.Collector;
@@ -126,7 +126,7 @@ public class MatchDriverIT1, IT2, OT implements 
PactDriverFlatJoinFunctionIT
if (this.objectReuseEnabled) {
switch (ls) {
case MERGE:
-   this.matchIterator = new 
ReusingMergeMatchIteratorIT1, IT2, OT(in1, in2, serializer1, comparator1, 
serializer2, comparator2, pairComparatorFactory.createComparator12(comparator1, 
comparator2), memoryManager, ioManager, numPages, 
this.taskContext.getOwningNepheleTask());
+   this.matchIterator = new 
ReusingMergeInnerJoinIteratorIT1, IT2, OT(in1, in2, serializer1, comparator1, 
serializer2, comparator2, pairComparatorFactory.createComparator12(comparator1, 
comparator2), memoryManager, ioManager, numPages, 
this.taskContext.getOwningNepheleTask());
 
break;
case HYBRIDHASH_BUILD_FIRST:
@@ -141,7 +141,7 @@ public class MatchDriverIT1, IT2, OT implements 
PactDriverFlatJoinFunctionIT
} else {
switch (ls) {
case MERGE:
-   this.matchIterator = new 
NonReusingMergeMatchIteratorIT1, IT2, OT(in1, in2, serializer1, comparator1, 
serializer2, comparator2, 

[5/5] flink git commit: [FLINK-2105] Implement Sort-Merge Outer Join algorithm

[fhueske]

[FLINK-2105] Implement Sort-Merge Outer Join algorithm

This closes #907


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/941ac6df
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/941ac6df
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/941ac6df

Branch: refs/heads/master
Commit: 941ac6dfd446d8e97e2fe2f589164978602adf94
Parents: df9f481
Author: r-pogalz r.pog...@campus.tu-berlin.de
Authored: Mon Aug 3 12:59:48 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue Aug 4 23:51:27 2015 +0200

--
 .../operators/sort/AbstractMergeIterator.java   |  58 +--
 .../sort/AbstractMergeOuterJoinIterator.java| 189 
 .../sort/NonReusingMergeOuterJoinIterator.java  |  60 +++
 .../sort/ReusingMergeOuterJoinIterator.java |  63 +++
 ...bstractSortMergeOuterJoinIteratorITCase.java | 462 +++
 ...ReusingSortMergeInnerJoinIteratorITCase.java |   4 +-
 ...ReusingSortMergeOuterJoinIteratorITCase.java |  82 
 ...ReusingSortMergeInnerJoinIteratorITCase.java |   4 +-
 ...ReusingSortMergeOuterJoinIteratorITCase.java |  82 
 .../runtime/operators/testutils/Match.java  |   2 +-
 .../testutils/MatchRemovingJoiner.java  |  58 +++
 .../testutils/MatchRemovingMatcher.java |  58 ---
 12 files changed, 1030 insertions(+), 92 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/941ac6df/flink-runtime/src/main/java/org/apache/flink/runtime/operators/sort/AbstractMergeIterator.java
--
diff --git 
a/flink-runtime/src/main/java/org/apache/flink/runtime/operators/sort/AbstractMergeIterator.java
 
b/flink-runtime/src/main/java/org/apache/flink/runtime/operators/sort/AbstractMergeIterator.java
index 9a61c14..c01afc7 100644
--- 
a/flink-runtime/src/main/java/org/apache/flink/runtime/operators/sort/AbstractMergeIterator.java
+++ 
b/flink-runtime/src/main/java/org/apache/flink/runtime/operators/sort/AbstractMergeIterator.java
@@ -115,20 +115,20 @@ public abstract class AbstractMergeIteratorT1, T2, O 
implements JoinTaskIterat
}
 
/**
-* Calls the codeJoinFunction#match()/code method for all two 
key-value pairs that share the same key and come
-* from different inputs. The output of the codematch()/code method 
is forwarded.
+* Calls the codeJoinFunction#join()/code method for all two 
key-value pairs that share the same key and come
+* from different inputs. The output of the codejoin()/code method 
is forwarded.
 * p
 * This method first zig-zags between the two sorted inputs in order to 
find a common
-* key, and then calls the match stub with the cross product of the 
values.
+* key, and then calls the join stub with the cross product of the 
values.
 *
 * @throws Exception Forwards all exceptions from the user code and the 
I/O system.
 * @see 
org.apache.flink.runtime.operators.util.JoinTaskIterator#callWithNextKey(org.apache.flink.api.common.functions.FlatJoinFunction,
 org.apache.flink.util.Collector)
 */
@Override
-   public abstract boolean callWithNextKey(final FlatJoinFunctionT1, T2, 
O matchFunction, final CollectorO collector)
+   public abstract boolean callWithNextKey(final FlatJoinFunctionT1, T2, 
O joinFunction, final CollectorO collector)
throws Exception;
 
-   protected void crossMatchingGroup(IteratorT1 values1, IteratorT2 
values2, FlatJoinFunctionT1, T2, O matchFunction, CollectorO collector) 
throws Exception {
+   protected void crossMatchingGroup(IteratorT1 values1, IteratorT2 
values2, FlatJoinFunctionT1, T2, O joinFunction, CollectorO collector) 
throws Exception {
final T1 firstV1 = values1.next();
final T2 firstV2 = values2.next();
 
@@ -143,23 +143,23 @@ public abstract class AbstractMergeIteratorT1, T2, O 
implements JoinTaskIterat
if (v2HasNext) {
// both sides contain more than one value
// TODO: Decide which side to spill and which 
to block!
-   crossMwithNValues(firstV1, values1, firstV2, 
values2, matchFunction, collector);
+   crossMwithNValues(firstV1, values1, firstV2, 
values2, joinFunction, collector);
} else {
-   crossSecond1withNValues(firstV2, firstV1, 
values1, matchFunction, collector);
+   crossSecond1withNValues(firstV2, firstV1, 
values1, joinFunction, collector);
}
} else {
if (v2HasNext) {
-   crossFirst1withNValues(firstV1, 

[3/5] flink git commit: [FLINK-2105] [tests] Move duplicate utility classes to testutil package

[fhueske]

http://git-wip-us.apache.org/repos/asf/flink/blob/df9f4819/flink-runtime/src/test/java/org/apache/flink/runtime/operators/testutils/MatchRemovingMatcher.java
--
diff --git 
a/flink-runtime/src/test/java/org/apache/flink/runtime/operators/testutils/MatchRemovingMatcher.java
 
b/flink-runtime/src/test/java/org/apache/flink/runtime/operators/testutils/MatchRemovingMatcher.java
new file mode 100644
index 000..f69b4d7
--- /dev/null
+++ 
b/flink-runtime/src/test/java/org/apache/flink/runtime/operators/testutils/MatchRemovingMatcher.java
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.operators.testutils;
+
+import org.apache.flink.api.common.functions.FlatJoinFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.util.Collector;
+import org.junit.Assert;
+
+import java.util.Collection;
+import java.util.Map;
+
+
+public final class MatchRemovingMatcher implements 
FlatJoinFunctionTuple2Integer,String,Tuple2Integer,String,Tuple2Integer,String
 {
+   private static final long serialVersionUID = 1L;
+
+   private final MapInteger, CollectionMatch toRemoveFrom;
+
+   public MatchRemovingMatcher(MapInteger, CollectionMatch map) {
+   this.toRemoveFrom = map;
+   }
+
+   @Override
+   public void join(Tuple2Integer, String rec1, Tuple2Integer, String 
rec2, CollectorTuple2Integer, String out) throws Exception {
+   final Integer key = rec1 != null ? (Integer) rec1.getField(0) : 
(Integer) rec2.getField(0);
+   final String value1 = rec1 != null ? (String) rec1.getField(1) 
: null;
+   final String value2 = rec2 != null ? (String) rec2.getField(1) 
: null;
+
+   CollectionMatch matches = this.toRemoveFrom.get(key);
+   if (matches == null) {
+   Assert.fail(Match  + key +  -  + value1 + : + 
value2 +  is unexpected.);
+   }
+
+   boolean contained = matches.remove(new Match(value1, value2));
+   if (!contained) {
+   Assert.fail(Produced match was not contained:  + key 
+  -  + value1 + : + value2);
+   }
+   if (matches.isEmpty()) {
+   this.toRemoveFrom.remove(key);
+   }
+   }
+}

http://git-wip-us.apache.org/repos/asf/flink/blob/df9f4819/flink-runtime/src/test/java/org/apache/flink/runtime/operators/testutils/SimpleTupleJoinFunction.java
--
diff --git 
a/flink-runtime/src/test/java/org/apache/flink/runtime/operators/testutils/SimpleTupleJoinFunction.java
 
b/flink-runtime/src/test/java/org/apache/flink/runtime/operators/testutils/SimpleTupleJoinFunction.java
new file mode 100644
index 000..06a62e5
--- /dev/null
+++ 
b/flink-runtime/src/test/java/org/apache/flink/runtime/operators/testutils/SimpleTupleJoinFunction.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.operators.testutils;
+
+import org.apache.flink.api.common.functions.FlatJoinFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.util.Collector;
+
+/**
+ * Simple flat join function that joins two binary tuples and considers null 
cases.
+ */
+public class 

[2/3] flink git commit: [FLINK-2205] Fix confusing entries in JobManager WebUI JobConfig section.

[fhueske]

[FLINK-2205] Fix confusing entries in JobManager WebUI JobConfig section.

Default display for 'Number of execution retries' is now 'deactivated'
and for 'Job parallelism' is 'auto', as suggested in JIRA.

This closes #927


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/b2d8c40a
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/b2d8c40a
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/b2d8c40a

Branch: refs/heads/master
Commit: b2d8c40a06e0a36e90913d316ff2b003b701fee1
Parents: 06b37bf
Author: Enrique Bautista ebautista...@gmail.com
Authored: Tue Jul 21 18:52:05 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue Aug 4 18:14:52 2015 +0200

--
 .../flink/runtime/jobmanager/web/JobManagerInfoServlet.java  | 2 +-
 .../src/main/resources/web-docs-infoserver/js/analyzer.js| 4 
 2 files changed, 5 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/b2d8c40a/flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/web/JobManagerInfoServlet.java
--
diff --git 
a/flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/web/JobManagerInfoServlet.java
 
b/flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/web/JobManagerInfoServlet.java
index ce57714..0ecc941 100644
--- 
a/flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/web/JobManagerInfoServlet.java
+++ 
b/flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/web/JobManagerInfoServlet.java
@@ -436,7 +436,7 @@ public class JobManagerInfoServlet extends HttpServlet {
if(ec != null) {
wrt.write(\executionConfig\: {);
wrt.write(\Execution Mode\: 
\+ec.getExecutionMode()+\,);
-   wrt.write(\Number of execution retries\: 
\+ec.getNumberOfExecutionRetries()+\,);
+   wrt.write(\Max. number of execution 
retries\: \+ec.getNumberOfExecutionRetries()+\,);
wrt.write(\Job parallelism\: 
\+ec.getParallelism()+\,);
wrt.write(\Object reuse mode\: 
\+ec.isObjectReuseEnabled()+\);
ExecutionConfig.GlobalJobParameters uc = 
ec.getGlobalJobParameters();

http://git-wip-us.apache.org/repos/asf/flink/blob/b2d8c40a/flink-runtime/src/main/resources/web-docs-infoserver/js/analyzer.js
--
diff --git 
a/flink-runtime/src/main/resources/web-docs-infoserver/js/analyzer.js 
b/flink-runtime/src/main/resources/web-docs-infoserver/js/analyzer.js
index 3934017..4030f80 100644
--- a/flink-runtime/src/main/resources/web-docs-infoserver/js/analyzer.js
+++ b/flink-runtime/src/main/resources/web-docs-infoserver/js/analyzer.js
@@ -107,6 +107,10 @@ function analyzeTime(json, stacked) {
$.each(job.executionConfig, function(key, value) {
if(key == userConfig) {
return;
+   } else if(key == Max. number of execution retries  
value == -1) {
+   value = deactivated;
+   } else if(key == Job parallelism  value == -1) {
+   value = auto;
}
configTable += 
trtd+key+/tdtd+value+/td/tr;
});

[2/2] flink git commit: [FLINK-2442] [fix] FieldPositionKeys support Pojo fields

[fhueske]

[FLINK-2442] [fix] FieldPositionKeys support Pojo fields


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/80d3478c
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/80d3478c
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/80d3478c

Branch: refs/heads/release-0.9
Commit: 80d3478c0be6c346c0d33d966385c3db4eb06769
Parents: 8797890
Author: Fabian Hueske fhue...@apache.org
Authored: Thu Jul 30 21:44:06 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue Aug 4 21:10:19 2015 +0200

--
 .../apache/flink/api/java/operators/Keys.java   | 50 ++--
 .../api/java/typeutils/TupleTypeInfoBase.java   | 20 
 .../flink/api/java/operators/KeysTest.java  | 27 +++
 3 files changed, 52 insertions(+), 45 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/80d3478c/flink-java/src/main/java/org/apache/flink/api/java/operators/Keys.java
--
diff --git 
a/flink-java/src/main/java/org/apache/flink/api/java/operators/Keys.java 
b/flink-java/src/main/java/org/apache/flink/api/java/operators/Keys.java
index 69d306f..09874e5 100644
--- a/flink-java/src/main/java/org/apache/flink/api/java/operators/Keys.java
+++ b/flink-java/src/main/java/org/apache/flink/api/java/operators/Keys.java
@@ -223,43 +223,43 @@ public abstract class KeysT {
} else {
groupingFields = 
rangeCheckFields(groupingFields, type.getArity() -1);
}
-   CompositeType? compositeType = (CompositeType?) 
type;
Preconditions.checkArgument(groupingFields.length  0, 
Grouping fields can not be empty at this point);

keyFields = new 
ArrayListFlatFieldDescriptor(type.getTotalFields());
// for each key, find the field:
for(int j = 0; j  groupingFields.length; j++) {
+   int keyPos = groupingFields[j];
+
+   int offset = 0;
for(int i = 0; i  type.getArity(); i++) {
-   TypeInformation? fieldType = 
compositeType.getTypeAt(i);
-   
-   if(groupingFields[j] == i) { // check 
if user set the key
-   int keyId = 
countNestedElementsBefore(compositeType, i) + i;
-   if(fieldType instanceof 
TupleTypeInfoBase) {
-   TupleTypeInfoBase? 
tupleFieldType = (TupleTypeInfoBase?) fieldType;
-   
tupleFieldType.addAllFields(keyId, keyFields);
-   } else {
-   
Preconditions.checkArgument(fieldType instanceof AtomicType, Wrong field 
type);
-   keyFields.add(new 
FlatFieldDescriptor(keyId, fieldType));
+
+   TypeInformation fieldType = 
((CompositeType?) type).getTypeAt(i);
+   if(i  keyPos) {
+   // not yet there, increment key 
offset
+   offset += 
fieldType.getTotalFields();
+   }
+   else {
+   // arrived at key position
+   if(fieldType instanceof 
CompositeType) {
+   // add all nested 
fields of composite type
+   ((CompositeType) 
fieldType).getFlatFields(*, offset, keyFields);
}
-   
+   else if(fieldType instanceof 
AtomicType) {
+   // add atomic type field
+   keyFields.add(new 
FlatFieldDescriptor(offset, fieldType));
+   }
+   else {
+   // type should either 
be composite or atomic
+   throw new 
InvalidProgramException(Field type is neither CompositeType nor AtomicType: 
+fieldType);
+   }
+  

[1/3] flink git commit: [FLINK-2005] Remove Record API from jdbc module

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master d570d078a - 30761572b


[FLINK-2005] Remove Record API from jdbc module

This closes #982


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/06b37bf5
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/06b37bf5
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/06b37bf5

Branch: refs/heads/master
Commit: 06b37bf550315bd1d5be7dc3ed6638fd21768e1a
Parents: d570d07
Author: zentol s.mo...@web.de
Authored: Tue Aug 4 12:45:22 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue Aug 4 18:13:30 2015 +0200

--
 flink-staging/flink-jdbc/pom.xml|  12 -
 .../java/record/io/jdbc/JDBCInputFormat.java| 389 ---
 .../java/record/io/jdbc/JDBCOutputFormat.java   | 359 -
 .../record/io/jdbc/example/JDBCExample.java | 136 ---
 .../java/record/io/jdbc/DevNullLogStream.java   |  30 --
 .../record/io/jdbc/JDBCInputFormatTest.java | 214 --
 .../record/io/jdbc/JDBCOutputFormatTest.java| 225 ---
 7 files changed, 1365 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/06b37bf5/flink-staging/flink-jdbc/pom.xml
--
diff --git a/flink-staging/flink-jdbc/pom.xml b/flink-staging/flink-jdbc/pom.xml
index 7b499a7..a3976c1 100644
--- a/flink-staging/flink-jdbc/pom.xml
+++ b/flink-staging/flink-jdbc/pom.xml
@@ -41,18 +41,6 @@ under the License.
artifactIdflink-java/artifactId
version${project.version}/version
/dependency
-   
-   dependency
-   groupIdorg.apache.flink/groupId
-   artifactIdflink-core/artifactId
-   version${project.version}/version
-   /dependency
-   
-   dependency
-   groupIdorg.apache.flink/groupId
-   artifactIdflink-clients/artifactId
-   version${project.version}/version
-   /dependency
 
dependency
groupIdorg.apache.derby/groupId

http://git-wip-us.apache.org/repos/asf/flink/blob/06b37bf5/flink-staging/flink-jdbc/src/main/java/org/apache/flink/api/java/record/io/jdbc/JDBCInputFormat.java
--
diff --git 
a/flink-staging/flink-jdbc/src/main/java/org/apache/flink/api/java/record/io/jdbc/JDBCInputFormat.java
 
b/flink-staging/flink-jdbc/src/main/java/org/apache/flink/api/java/record/io/jdbc/JDBCInputFormat.java
deleted file mode 100644
index 3cd295b..000
--- 
a/flink-staging/flink-jdbc/src/main/java/org/apache/flink/api/java/record/io/jdbc/JDBCInputFormat.java
+++ /dev/null
@@ -1,389 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.api.java.record.io.jdbc;
-
-import java.sql.Connection;
-import java.sql.DriverManager;
-import java.sql.ResultSet;
-import java.sql.ResultSetMetaData;
-import java.sql.SQLException;
-import java.sql.Statement;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import org.apache.flink.api.common.io.NonParallelInput;
-import org.apache.flink.api.java.record.io.GenericInputFormat;
-import org.apache.flink.configuration.Configuration;
-import org.apache.flink.types.BooleanValue;
-import org.apache.flink.types.DoubleValue;
-import org.apache.flink.types.FloatValue;
-import org.apache.flink.types.IntValue;
-import org.apache.flink.types.LongValue;
-import org.apache.flink.types.NullValue;
-import org.apache.flink.types.Record;
-import org.apache.flink.types.ShortValue;
-import org.apache.flink.types.StringValue;
-
-/**
- * InputFormat to read data from a database and generate PactReords.
- * The InputFormat has to be configured with the query, and either all
- * connection parameters or a complete database URL.{@link Configuration} The 
position of a value inside a Record is

[3/3] flink git commit: [FLINK-2442] [fix] FieldPositionKeys support Pojo fields

[fhueske]

[FLINK-2442] [fix] FieldPositionKeys support Pojo fields

This closes #963


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/30761572
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/30761572
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/30761572

Branch: refs/heads/master
Commit: 30761572b5040669b07d261ec9b109797debc549
Parents: b2d8c40
Author: Fabian Hueske fhue...@apache.org
Authored: Thu Jul 30 21:44:06 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue Aug 4 18:16:30 2015 +0200

--
 .../apache/flink/api/java/operators/Keys.java   | 50 ++--
 .../api/java/typeutils/TupleTypeInfoBase.java   | 20 
 .../flink/api/java/operators/KeysTest.java  | 27 +++
 3 files changed, 52 insertions(+), 45 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/30761572/flink-java/src/main/java/org/apache/flink/api/java/operators/Keys.java
--
diff --git 
a/flink-java/src/main/java/org/apache/flink/api/java/operators/Keys.java 
b/flink-java/src/main/java/org/apache/flink/api/java/operators/Keys.java
index 69d306f..09874e5 100644
--- a/flink-java/src/main/java/org/apache/flink/api/java/operators/Keys.java
+++ b/flink-java/src/main/java/org/apache/flink/api/java/operators/Keys.java
@@ -223,43 +223,43 @@ public abstract class KeysT {
} else {
groupingFields = 
rangeCheckFields(groupingFields, type.getArity() -1);
}
-   CompositeType? compositeType = (CompositeType?) 
type;
Preconditions.checkArgument(groupingFields.length  0, 
Grouping fields can not be empty at this point);

keyFields = new 
ArrayListFlatFieldDescriptor(type.getTotalFields());
// for each key, find the field:
for(int j = 0; j  groupingFields.length; j++) {
+   int keyPos = groupingFields[j];
+
+   int offset = 0;
for(int i = 0; i  type.getArity(); i++) {
-   TypeInformation? fieldType = 
compositeType.getTypeAt(i);
-   
-   if(groupingFields[j] == i) { // check 
if user set the key
-   int keyId = 
countNestedElementsBefore(compositeType, i) + i;
-   if(fieldType instanceof 
TupleTypeInfoBase) {
-   TupleTypeInfoBase? 
tupleFieldType = (TupleTypeInfoBase?) fieldType;
-   
tupleFieldType.addAllFields(keyId, keyFields);
-   } else {
-   
Preconditions.checkArgument(fieldType instanceof AtomicType, Wrong field 
type);
-   keyFields.add(new 
FlatFieldDescriptor(keyId, fieldType));
+
+   TypeInformation fieldType = 
((CompositeType?) type).getTypeAt(i);
+   if(i  keyPos) {
+   // not yet there, increment key 
offset
+   offset += 
fieldType.getTotalFields();
+   }
+   else {
+   // arrived at key position
+   if(fieldType instanceof 
CompositeType) {
+   // add all nested 
fields of composite type
+   ((CompositeType) 
fieldType).getFlatFields(*, offset, keyFields);
}
-   
+   else if(fieldType instanceof 
AtomicType) {
+   // add atomic type field
+   keyFields.add(new 
FlatFieldDescriptor(offset, fieldType));
+   }
+   else {
+   // type should either 
be composite or atomic
+   throw new 
InvalidProgramException(Field type is neither CompositeType nor AtomicType: 
+fieldType);
+   }
+ 

[1/2] flink git commit: [FLINK-2205] Fix confusing entries in JobManager WebUI JobConfig section.

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/release-0.9 acb6ea81a - 80d3478c0


[FLINK-2205] Fix confusing entries in JobManager WebUI JobConfig section.

Default display for 'Number of execution retries' is now 'deactivated'
and for 'Job parallelism' is 'auto', as suggested in JIRA.


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/8797890d
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/8797890d
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/8797890d

Branch: refs/heads/release-0.9
Commit: 8797890d2804541f38ebe1c8661603848f3e5a39
Parents: acb6ea8
Author: Enrique Bautista ebautista...@gmail.com
Authored: Tue Jul 21 18:52:05 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue Aug 4 21:09:54 2015 +0200

--
 .../flink/runtime/jobmanager/web/JobManagerInfoServlet.java  | 2 +-
 .../src/main/resources/web-docs-infoserver/js/analyzer.js| 4 
 2 files changed, 5 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/8797890d/flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/web/JobManagerInfoServlet.java
--
diff --git 
a/flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/web/JobManagerInfoServlet.java
 
b/flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/web/JobManagerInfoServlet.java
index 82ab63e..a164d5a 100644
--- 
a/flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/web/JobManagerInfoServlet.java
+++ 
b/flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/web/JobManagerInfoServlet.java
@@ -441,7 +441,7 @@ public class JobManagerInfoServlet extends HttpServlet {
if(ec != null) {
wrt.write(\executionConfig\: {);
wrt.write(\Execution Mode\: 
\+ec.getExecutionMode()+\,);
-   wrt.write(\Number of execution retries\: 
\+ec.getNumberOfExecutionRetries()+\,);
+   wrt.write(\Max. number of execution 
retries\: \+ec.getNumberOfExecutionRetries()+\,);
wrt.write(\Job parallelism\: 
\+ec.getParallelism()+\,);
wrt.write(\Object reuse mode\: 
\+ec.isObjectReuseEnabled()+\);
ExecutionConfig.GlobalJobParameters uc = 
ec.getGlobalJobParameters();

http://git-wip-us.apache.org/repos/asf/flink/blob/8797890d/flink-runtime/src/main/resources/web-docs-infoserver/js/analyzer.js
--
diff --git 
a/flink-runtime/src/main/resources/web-docs-infoserver/js/analyzer.js 
b/flink-runtime/src/main/resources/web-docs-infoserver/js/analyzer.js
index 3934017..4030f80 100644
--- a/flink-runtime/src/main/resources/web-docs-infoserver/js/analyzer.js
+++ b/flink-runtime/src/main/resources/web-docs-infoserver/js/analyzer.js
@@ -107,6 +107,10 @@ function analyzeTime(json, stacked) {
$.each(job.executionConfig, function(key, value) {
if(key == userConfig) {
return;
+   } else if(key == Max. number of execution retries  
value == -1) {
+   value = deactivated;
+   } else if(key == Job parallelism  value == -1) {
+   value = auto;
}
configTable += 
trtd+key+/tdtd+value+/td/tr;
});

[1/5] flink git commit: [FLINK-2105] Extract abstract superclass, interface from MergeMatchIterators, KeyGroupedIterators

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master 30761572b - 941ac6dfd


[FLINK-2105] Extract abstract superclass, interface from MergeMatchIterators, 
KeyGroupedIterators


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/0dc6849a
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/0dc6849a
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/0dc6849a

Branch: refs/heads/master
Commit: 0dc6849a594b61a6cad8ee582ca1758f0349a72b
Parents: 3076157
Author: Johann Kovacs m...@jkovacs.de
Authored: Fri Jul 10 17:21:58 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue Aug 4 21:35:26 2015 +0200

--
 .../operators/sort/AbstractMergeIterator.java   | 356 +
 .../sort/AbstractMergeMatchIterator.java| 107 +
 .../sort/NonReusingMergeMatchIterator.java  | 382 +-
 .../sort/ReusingMergeMatchIterator.java | 389 +--
 .../flink/runtime/util/KeyGroupedIterator.java  |  31 ++
 .../util/NonReusingKeyGroupedIterator.java  |   2 +-
 .../runtime/util/ReusingKeyGroupedIterator.java |   5 +-
 7 files changed, 517 insertions(+), 755 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/0dc6849a/flink-runtime/src/main/java/org/apache/flink/runtime/operators/sort/AbstractMergeIterator.java
--
diff --git 
a/flink-runtime/src/main/java/org/apache/flink/runtime/operators/sort/AbstractMergeIterator.java
 
b/flink-runtime/src/main/java/org/apache/flink/runtime/operators/sort/AbstractMergeIterator.java
new file mode 100644
index 000..9a61c14
--- /dev/null
+++ 
b/flink-runtime/src/main/java/org/apache/flink/runtime/operators/sort/AbstractMergeIterator.java
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.operators.sort;
+
+import org.apache.flink.api.common.functions.FlatJoinFunction;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypePairComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.io.disk.iomanager.IOManager;
+import org.apache.flink.runtime.jobgraph.tasks.AbstractInvokable;
+import org.apache.flink.runtime.memorymanager.MemoryAllocationException;
+import org.apache.flink.runtime.memorymanager.MemoryManager;
+import 
org.apache.flink.runtime.operators.resettable.NonReusingBlockResettableIterator;
+import 
org.apache.flink.runtime.operators.resettable.SpillingResettableIterator;
+import org.apache.flink.runtime.operators.util.JoinTaskIterator;
+import org.apache.flink.runtime.util.KeyGroupedIterator;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.MutableObjectIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+
+public abstract class AbstractMergeIteratorT1, T2, O implements 
JoinTaskIteratorT1, T2, O {
+
+   private final Logger LOG = LoggerFactory.getLogger(getClass());
+
+   protected TypePairComparatorT1, T2 pairComparator;
+
+   protected KeyGroupedIteratorT1 iterator1;
+   protected KeyGroupedIteratorT2 iterator2;
+
+   protected final TypeSerializerT1 serializer1;
+   protected final TypeSerializerT2 serializer2;
+
+   private final NonReusingBlockResettableIteratorT2 blockIt;// for 
N:M cross products with same key
+
+   private final IOManager ioManager;
+   private final MemoryManager memoryManager;
+   private final ListMemorySegment memoryForSpillingIterator;
+
+   // instances for object reuse
+   protected T1 copy1;
+   protected T1 spillHeadCopy;
+   protected T2 copy2;
+   protected T2 blockHeadCopy;
+
+   public AbstractMergeIterator(MutableObjectIteratorT1 input1, 
MutableObjectIteratorT2 input2,
+   

[2/5] flink git commit: [FLINK-2105] Add support for sorted but sparse test data generation

[fhueske]

[FLINK-2105] Add support for sorted but sparse test data generation


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/db0b0087
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/db0b0087
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/db0b0087

Branch: refs/heads/master
Commit: db0b0087b02985f55bcc6e65571b11ca33b0886f
Parents: 0dc6849
Author: Johann Kovacs m...@jkovacs.de
Authored: Fri Jul 10 17:26:05 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue Aug 4 21:35:27 2015 +0200

--
 .../runtime/operators/testutils/TestData.java   | 207 +++
 1 file changed, 207 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/db0b0087/flink-runtime/src/test/java/org/apache/flink/runtime/operators/testutils/TestData.java
--
diff --git 
a/flink-runtime/src/test/java/org/apache/flink/runtime/operators/testutils/TestData.java
 
b/flink-runtime/src/test/java/org/apache/flink/runtime/operators/testutils/TestData.java
index fd34a3b..8688d4e 100644
--- 
a/flink-runtime/src/test/java/org/apache/flink/runtime/operators/testutils/TestData.java
+++ 
b/flink-runtime/src/test/java/org/apache/flink/runtime/operators/testutils/TestData.java
@@ -21,6 +21,8 @@ package org.apache.flink.runtime.operators.testutils;
 import java.util.Comparator;
 import java.util.Random;
 
+import org.apache.flink.api.java.tuple.Tuple;
+import org.apache.flink.api.java.tuple.Tuple2;
 import org.apache.flink.types.IntValue;
 import org.apache.flink.types.Record;
 import org.apache.flink.types.StringValue;
@@ -279,6 +281,169 @@ public final class TestData {
this.counter = 0;
}
}
+
+   /**
+* Tuple2Integer, String generator.
+*/
+   public static class TupleGenerator implements 
MutableObjectIteratorTuple2Integer, String {
+
+   public enum KeyMode {
+   SORTED, RANDOM, SORTED_SPARSE
+   };
+
+   public enum ValueMode {
+   FIX_LENGTH, RANDOM_LENGTH, CONSTANT
+   };
+
+   private static char[] alpha = { 'a', 'b', 'c', 'd', 'e', 'f', 
'g', 'h', 'i', 'j', 'k', 'l', 'm', 'a', 'b', 'c',
+   'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 
'm' };
+
+   private final long seed;
+
+   private final int keyMax;
+
+   private final float keyDensity;
+
+   private final int valueLength;
+
+   private final KeyMode keyMode;
+
+   private final ValueMode valueMode;
+
+   private Random random;
+
+   private int counter;
+
+   private int key;
+   private String value;
+
+   public TupleGenerator(long seed, int keyMax, int valueLength) {
+   this(seed, keyMax, valueLength, KeyMode.RANDOM, 
ValueMode.FIX_LENGTH);
+   }
+
+   public TupleGenerator(long seed, int keyMax, int valueLength, 
KeyMode keyMode, ValueMode valueMode) {
+   this(seed, keyMax, valueLength, keyMode, valueMode, 
null);
+   }
+
+   public TupleGenerator(long seed, int keyMax, int valueLength, 
KeyMode keyMode, ValueMode valueMode, String constant) {
+   this(seed, keyMax, 1.0f, valueLength, keyMode, 
valueMode, constant);
+   }
+
+   public TupleGenerator(long seed, int keyMax, float keyDensity, 
int valueLength, KeyMode keyMode, ValueMode valueMode, String constant) {
+   this.seed = seed;
+   this.keyMax = keyMax;
+   this.keyDensity = keyDensity;
+   this.valueLength = valueLength;
+   this.keyMode = keyMode;
+   this.valueMode = valueMode;
+
+   this.random = new Random(seed);
+   this.counter = 0;
+
+   this.value = constant == null ? null : constant;
+   }
+
+   public Tuple2Integer, String next(Tuple2Integer, String 
reuse) {
+   this.key = nextKey();
+   if (this.valueMode != ValueMode.CONSTANT) {
+   this.value = randomString();
+   }
+   reuse.setFields(this.key, this.value);
+   return reuse;
+   }
+
+   public Tuple2Integer, String next() {
+   return next(new Tuple2Integer, String());
+   }
+
+   public boolean next(org.apache.flink.types.Value[] target) {
+   this.key = nextKey();
+  

flink git commit: [FLINK-2359] [java api] Add factory methods to the Java TupleX types

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master 08ca9ffa9 - 2d191ab05


[FLINK-2359] [java api] Add factory methods to the Java TupleX types

This closes #911


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/2d191ab0
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/2d191ab0
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/2d191ab0

Branch: refs/heads/master
Commit: 2d191ab05fff3392697fbce8dd909003c86d34e1
Parents: 08ca9ff
Author: Gabor Gevay gga...@gmail.com
Authored: Tue Jul 14 18:34:05 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Fri Jul 17 14:50:36 2015 +0200

--
 .../org/apache/flink/api/java/io/CsvReader.java |  2 +-
 .../flink/api/java/operators/JoinOperator.java  |  2 +-
 .../org/apache/flink/api/java/tuple/Tuple.java  |  2 -
 .../org/apache/flink/api/java/tuple/Tuple1.java | 14 -
 .../apache/flink/api/java/tuple/Tuple10.java| 14 -
 .../apache/flink/api/java/tuple/Tuple11.java| 14 -
 .../apache/flink/api/java/tuple/Tuple12.java| 14 -
 .../apache/flink/api/java/tuple/Tuple13.java| 14 -
 .../apache/flink/api/java/tuple/Tuple14.java| 14 -
 .../apache/flink/api/java/tuple/Tuple15.java| 14 -
 .../apache/flink/api/java/tuple/Tuple16.java| 14 -
 .../apache/flink/api/java/tuple/Tuple17.java| 14 -
 .../apache/flink/api/java/tuple/Tuple18.java| 14 -
 .../apache/flink/api/java/tuple/Tuple19.java| 14 -
 .../org/apache/flink/api/java/tuple/Tuple2.java | 14 -
 .../apache/flink/api/java/tuple/Tuple20.java| 14 -
 .../apache/flink/api/java/tuple/Tuple21.java| 14 -
 .../apache/flink/api/java/tuple/Tuple22.java| 14 -
 .../apache/flink/api/java/tuple/Tuple23.java| 14 -
 .../apache/flink/api/java/tuple/Tuple24.java| 14 -
 .../apache/flink/api/java/tuple/Tuple25.java| 14 -
 .../org/apache/flink/api/java/tuple/Tuple3.java | 14 -
 .../org/apache/flink/api/java/tuple/Tuple4.java | 14 -
 .../org/apache/flink/api/java/tuple/Tuple5.java | 14 -
 .../org/apache/flink/api/java/tuple/Tuple6.java | 14 -
 .../org/apache/flink/api/java/tuple/Tuple7.java | 14 -
 .../org/apache/flink/api/java/tuple/Tuple8.java | 14 -
 .../org/apache/flink/api/java/tuple/Tuple9.java | 14 -
 .../flink/api/java/tuple/TupleGenerator.java| 55 +---
 .../base/CoGroupOperatorCollectionTest.java |  2 +-
 .../SemanticPropertiesPrecedenceTest.java   |  8 +--
 31 files changed, 369 insertions(+), 52 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/2d191ab0/flink-java/src/main/java/org/apache/flink/api/java/io/CsvReader.java
--
diff --git 
a/flink-java/src/main/java/org/apache/flink/api/java/io/CsvReader.java 
b/flink-java/src/main/java/org/apache/flink/api/java/io/CsvReader.java
index da5ee49..c5d6cb5 100644
--- a/flink-java/src/main/java/org/apache/flink/api/java/io/CsvReader.java
+++ b/flink-java/src/main/java/org/apache/flink/api/java/io/CsvReader.java
@@ -1064,4 +1064,4 @@ public class CsvReader {
}
 
// END_OF_TUPLE_DEPENDENT_CODE
-}
\ No newline at end of file
+}

http://git-wip-us.apache.org/repos/asf/flink/blob/2d191ab0/flink-java/src/main/java/org/apache/flink/api/java/operators/JoinOperator.java
--
diff --git 
a/flink-java/src/main/java/org/apache/flink/api/java/operators/JoinOperator.java
 
b/flink-java/src/main/java/org/apache/flink/api/java/operators/JoinOperator.java
index 1e5baab..e76e3c9 100644
--- 
a/flink-java/src/main/java/org/apache/flink/api/java/operators/JoinOperator.java
+++ 
b/flink-java/src/main/java/org/apache/flink/api/java/operators/JoinOperator.java
@@ -1323,7 +1323,7 @@ public abstract class JoinOperatorI1, I2, OUT extends 
TwoInputUdfOperatorI1,
 
/**
 * Chooses a projectTupleX according to the length of
-* {@link 
org.apache.flink.api.java.operators.JoinOperator.JoinProjection#fieldIndexes} 
+* {@link 
org.apache.flink.api.java.operators.JoinOperator.JoinProjection#fieldIndexes}
 * 
 * @return The projected DataSet.
 * 

http://git-wip-us.apache.org/repos/asf/flink/blob/2d191ab0/flink-java/src/main/java/org/apache/flink/api/java/tuple/Tuple.java
--
diff --git 
a/flink-java/src/main/java/org/apache/flink/api/java/tuple/Tuple.java 
b/flink-java/src/main/java/org/apache/flink/api/java/tuple/Tuple.java
index 9da50c7..145d215 100644
--- a/flink-java/src/main/java/org/apache/flink/api/java/tuple/Tuple.java
+++ 

flink git commit: [FLINK-2353] Respect JobConfigurable interface in Hadoop mapred wrappers

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master 84b39dcb5 - 91675296e


[FLINK-2353] Respect JobConfigurable interface in Hadoop mapred wrappers

This closes #908


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/91675296
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/91675296
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/91675296

Branch: refs/heads/master
Commit: 91675296e1d0fcc503ab3af9c5da7fdb83b78fc5
Parents: 84b39dc
Author: Fabian Hueske fhue...@apache.org
Authored: Tue Jul 14 00:59:20 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue Jul 14 01:00:24 2015 +0200

--
 .../flink/api/java/hadoop/mapred/HadoopInputFormatBase.java | 9 -
 .../api/java/hadoop/mapred/HadoopOutputFormatBase.java  | 7 ++-
 .../api/java/hadoop/mapred/wrapper/HadoopInputSplit.java| 4 
 3 files changed, 18 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/91675296/flink-java/src/main/java/org/apache/flink/api/java/hadoop/mapred/HadoopInputFormatBase.java
--
diff --git 
a/flink-java/src/main/java/org/apache/flink/api/java/hadoop/mapred/HadoopInputFormatBase.java
 
b/flink-java/src/main/java/org/apache/flink/api/java/hadoop/mapred/HadoopInputFormatBase.java
index 40f6631..d5dbf38 100644
--- 
a/flink-java/src/main/java/org/apache/flink/api/java/hadoop/mapred/HadoopInputFormatBase.java
+++ 
b/flink-java/src/main/java/org/apache/flink/api/java/hadoop/mapred/HadoopInputFormatBase.java
@@ -34,6 +34,7 @@ import org.apache.flink.core.io.InputSplitAssigner;
 import org.apache.hadoop.conf.Configurable;
 import org.apache.hadoop.mapred.FileInputFormat;
 import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.JobConfigurable;
 import org.apache.hadoop.mapred.RecordReader;
 import org.apache.hadoop.util.ReflectionUtils;
 import org.slf4j.Logger;
@@ -82,7 +83,13 @@ public abstract class HadoopInputFormatBaseK, V, T 
implements InputFormatT, H

@Override
public void configure(Configuration parameters) {
-   // nothing to do
+   // configure MR InputFormat if necessary
+   if(this.mapredInputFormat instanceof Configurable) {
+   
((Configurable)this.mapredInputFormat).setConf(this.jobConf);
+   }
+   else if(this.mapredInputFormat instanceof JobConfigurable) {
+   
((JobConfigurable)this.mapredInputFormat).configure(this.jobConf);
+   }
}

@Override

http://git-wip-us.apache.org/repos/asf/flink/blob/91675296/flink-java/src/main/java/org/apache/flink/api/java/hadoop/mapred/HadoopOutputFormatBase.java
--
diff --git 
a/flink-java/src/main/java/org/apache/flink/api/java/hadoop/mapred/HadoopOutputFormatBase.java
 
b/flink-java/src/main/java/org/apache/flink/api/java/hadoop/mapred/HadoopOutputFormatBase.java
index a6a318c..d6dfc2e 100644
--- 
a/flink-java/src/main/java/org/apache/flink/api/java/hadoop/mapred/HadoopOutputFormatBase.java
+++ 
b/flink-java/src/main/java/org/apache/flink/api/java/hadoop/mapred/HadoopOutputFormatBase.java
@@ -28,6 +28,7 @@ import org.apache.flink.configuration.Configuration;
 import org.apache.hadoop.conf.Configurable;
 import org.apache.hadoop.mapred.FileOutputCommitter;
 import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.JobConfigurable;
 import org.apache.hadoop.mapred.JobContext;
 import org.apache.hadoop.mapred.JobID;
 import org.apache.hadoop.mapred.RecordWriter;
@@ -67,9 +68,13 @@ public abstract class HadoopOutputFormatBaseK, V, T 
implements OutputFormatT
 
@Override
public void configure(Configuration parameters) {
-   if(this.mapredOutputFormat instanceof Configurable){
+   // configure MR OutputFormat if necessary
+   if(this.mapredOutputFormat instanceof Configurable) {

((Configurable)this.mapredOutputFormat).setConf(this.jobConf);
}
+   else if(this.mapredOutputFormat instanceof JobConfigurable) {
+   
((JobConfigurable)this.mapredOutputFormat).configure(this.jobConf);
+   }
}
 
/**

http://git-wip-us.apache.org/repos/asf/flink/blob/91675296/flink-java/src/main/java/org/apache/flink/api/java/hadoop/mapred/wrapper/HadoopInputSplit.java
--
diff --git 
a/flink-java/src/main/java/org/apache/flink/api/java/hadoop/mapred/wrapper/HadoopInputSplit.java
 
b/flink-java/src/main/java/org/apache/flink/api/java/hadoop/mapred/wrapper/HadoopInputSplit.java
index 

flink git commit: [FLINK-2280] GenericTypeComparator.compare() respects ascending flag

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/release-0.9 acd43176e - 451eb8245


[FLINK-2280] GenericTypeComparator.compare() respects ascending flag

This closes #894


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/451eb824
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/451eb824
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/451eb824

Branch: refs/heads/release-0.9
Commit: 451eb82459f6ba49d0503b1ed2e4248b691574ff
Parents: acd4317
Author: Fabian Hueske fhue...@apache.org
Authored: Tue Jun 30 15:11:11 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Tue Jul 14 11:17:47 2015 +0200

--
 .../flink/api/java/typeutils/runtime/GenericTypeComparator.java   | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/451eb824/flink-java/src/main/java/org/apache/flink/api/java/typeutils/runtime/GenericTypeComparator.java
--
diff --git 
a/flink-java/src/main/java/org/apache/flink/api/java/typeutils/runtime/GenericTypeComparator.java
 
b/flink-java/src/main/java/org/apache/flink/api/java/typeutils/runtime/GenericTypeComparator.java
index aad3c41..28fea6a 100644
--- 
a/flink-java/src/main/java/org/apache/flink/api/java/typeutils/runtime/GenericTypeComparator.java
+++ 
b/flink-java/src/main/java/org/apache/flink/api/java/typeutils/runtime/GenericTypeComparator.java
@@ -88,7 +88,8 @@ public class GenericTypeComparatorT extends ComparableT 
extends TypeComparat
 
@Override
public int compare(T first, T second) {
-   return first.compareTo(second);
+   int cmp = first.compareTo(second);
+   return this.ascending ? cmp : -cmp;
}
 
@Override

flink git commit: [FLINK-1963] Improve distinct() transformation

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master c06213706 - 08ca9ffa9


[FLINK-1963] Improve distinct() transformation

This closes #905


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/08ca9ffa
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/08ca9ffa
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/08ca9ffa

Branch: refs/heads/master
Commit: 08ca9ffa9a95610c073145a09e731311e728c4fd
Parents: c062137
Author: pietro pinoli pie...@pietros-mbp.lan
Authored: Mon Jul 13 13:32:20 2015 +0200
Committer: Fabian Hueske fhue...@apache.org
Committed: Fri Jul 17 01:39:25 2015 +0200

--
 .../java/org/apache/flink/api/java/DataSet.java | 15 ++---
 .../api/java/operators/DistinctOperator.java| 26 +++-
 .../api/java/operator/DistinctOperatorTest.java | 65 +++-
 .../org/apache/flink/api/scala/DataSet.scala| 49 ++-
 .../test/javaApiOperators/DistinctITCase.java   | 44 +
 .../api/scala/operators/DistinctITCase.scala| 39 
 .../scala/operators/DistinctOperatorTest.scala  | 26 
 7 files changed, 223 insertions(+), 41 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/08ca9ffa/flink-java/src/main/java/org/apache/flink/api/java/DataSet.java
--
diff --git a/flink-java/src/main/java/org/apache/flink/api/java/DataSet.java 
b/flink-java/src/main/java/org/apache/flink/api/java/DataSet.java
index c628b04..81ba279 100644
--- a/flink-java/src/main/java/org/apache/flink/api/java/DataSet.java
+++ b/flink-java/src/main/java/org/apache/flink/api/java/DataSet.java
@@ -606,13 +606,13 @@ public abstract class DataSetT {
}

/**
-* Returns a distinct set of a {@link Tuple} {@link DataSet} using 
expression keys.
-* p
-* The field position keys specify the fields of Tuples or Pojos on 
which the decision is made if two elements are distinct or
-* not.
+* Returns a distinct set of a {@link DataSet} using expression keys.
 * p
+* The field expression keys specify the fields of a {@link 
org.apache.flink.api.common.typeutils.CompositeType}
+* (e.g., Tuple or Pojo type) on which the decision is made if two 
elements are distinct or not.
+* In case of a {@link 
org.apache.flink.api.common.typeinfo.AtomicType}, only the wildcard expression 
(*) is valid.
 *
-* @param fields One or more field positions on which the distinction 
of the DataSet is decided. 
+* @param fields One or more field expressions on which the distinction 
of the DataSet is decided.
 * @return A DistinctOperator that represents the distinct DataSet.
 */
public DistinctOperatorT distinct(String... fields) {
@@ -620,9 +620,10 @@ public abstract class DataSetT {
}

/**
-* Returns a distinct set of a {@link Tuple} {@link DataSet} using all 
fields of the tuple.
+* Returns a distinct set of a {@link DataSet}.
 * p
-* Note: This operator can only be applied to Tuple DataSets.
+* If the input is a {@link 
org.apache.flink.api.common.typeutils.CompositeType} (Tuple or Pojo type),
+* distinct is performed on all fields and each field must be a key type
 * 
 * @return A DistinctOperator that represents the distinct DataSet.
 */

http://git-wip-us.apache.org/repos/asf/flink/blob/08ca9ffa/flink-java/src/main/java/org/apache/flink/api/java/operators/DistinctOperator.java
--
diff --git 
a/flink-java/src/main/java/org/apache/flink/api/java/operators/DistinctOperator.java
 
b/flink-java/src/main/java/org/apache/flink/api/java/operators/DistinctOperator.java
index 686823c..a6eb43e 100644
--- 
a/flink-java/src/main/java/org/apache/flink/api/java/operators/DistinctOperator.java
+++ 
b/flink-java/src/main/java/org/apache/flink/api/java/operators/DistinctOperator.java
@@ -26,6 +26,7 @@ import 
org.apache.flink.api.common.operators.SingleInputSemanticProperties;
 import org.apache.flink.api.common.operators.UnaryOperatorInformation;
 import org.apache.flink.api.common.operators.base.GroupReduceOperatorBase;
 import org.apache.flink.api.common.operators.base.MapOperatorBase;
+import org.apache.flink.api.common.typeinfo.AtomicType;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
 import org.apache.flink.api.common.typeutils.CompositeType;
 import org.apache.flink.api.common.functions.RichGroupReduceFunction;
@@ -47,28 +48,21 @@ public class DistinctOperatorT extends 
SingleInputOperatorT, T, DistinctOpera
private final KeysT keys;

private final String distinctLocationName;
-   
+

[1/2] flink git commit: [FLINK-2874] Fix Avro getter/setter recognition

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/release-0.9 96099951e -> ab694a3b2


[FLINK-2874] Fix Avro getter/setter recognition

This closes #1252


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/4392cf25
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/4392cf25
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/4392cf25

Branch: refs/heads/release-0.9
Commit: 4392cf25388cdf61ab5c6c12d5ea8c95b62f1db5
Parents: 9609995
Author: Ulf Karlsson 
Authored: Sun Oct 11 00:04:57 2015 +0200
Committer: Fabian Hueske 
Committed: Fri Oct 23 10:27:51 2015 +0200

--
 .../flink/api/java/typeutils/TypeExtractor.java | 17 ++--
 .../api/java/typeutils/PojoTypeInfoTest.java| 95 +++-
 2 files changed, 104 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/4392cf25/flink-java/src/main/java/org/apache/flink/api/java/typeutils/TypeExtractor.java
--
diff --git 
a/flink-java/src/main/java/org/apache/flink/api/java/typeutils/TypeExtractor.java
 
b/flink-java/src/main/java/org/apache/flink/api/java/typeutils/TypeExtractor.java
index 2e45107..e15650f 100644
--- 
a/flink-java/src/main/java/org/apache/flink/api/java/typeutils/TypeExtractor.java
+++ 
b/flink-java/src/main/java/org/apache/flink/api/java/typeutils/TypeExtractor.java
@@ -30,6 +30,7 @@ import java.util.ArrayList;
 import java.util.List;
 
 import org.apache.avro.specific.SpecificRecordBase;
+import org.apache.commons.lang3.ClassUtils;
 import org.apache.flink.api.common.functions.CoGroupFunction;
 import org.apache.flink.api.common.functions.CrossFunction;
 import org.apache.flink.api.common.functions.FlatJoinFunction;
@@ -1260,22 +1261,26 @@ public class TypeExtractor {
return true;
} else {
boolean hasGetter = false, hasSetter = false;
-   final String fieldNameLow = f.getName().toLowerCase();
-   
+   final String fieldNameLow = 
f.getName().toLowerCase().replaceAll("_", "");
+
Type fieldType = f.getGenericType();
+   Class fieldTypeWrapper = 
ClassUtils.primitiveToWrapper(f.getType());
+
TypeVariable fieldTypeGeneric = null;
if(fieldType instanceof TypeVariable) {
fieldTypeGeneric = (TypeVariable) fieldType;
fieldType = 
materializeTypeVariable(typeHierarchy, (TypeVariable)fieldType);
}
for(Method m : clazz.getMethods()) {
+   final String methodNameLow = 
m.getName().toLowerCase().replaceAll("_", "");
+
// check for getter
if( // The name should be "get" 
or "" (for scala) or "is" for boolean fields.
-   
(m.getName().toLowerCase().equals("get"+fieldNameLow) || 
m.getName().toLowerCase().equals("is"+fieldNameLow) || 
m.getName().toLowerCase().equals(fieldNameLow)) &&
+   
(methodNameLow.equals("get"+fieldNameLow) || 
methodNameLow.equals("is"+fieldNameLow) || methodNameLow.equals(fieldNameLow)) 
&&
// no arguments for the getter
m.getParameterTypes().length == 0 &&
// return type is same as field type 
(or the generic variant of it)
-   (m.getGenericReturnType().equals( 
fieldType ) || (fieldTypeGeneric != null && 
m.getGenericReturnType().equals(fieldTypeGeneric)) )
+   (m.getGenericReturnType().equals( 
fieldType ) || (fieldTypeWrapper != null && m.getReturnType().equals( 
fieldTypeWrapper )) || (fieldTypeGeneric != null && 
m.getGenericReturnType().equals(fieldTypeGeneric)) )
) {
if(hasGetter) {
throw new 
IllegalStateException("Detected more than one getter");
@@ -1283,9 +1288,9 @@ public class TypeExtractor {
hasGetter = true;
}
// check for setters (_$eq for scala)
-   
if((m.getName().toLowerCase().equals("set"+fieldNameLow) || 
m.getName().toLowerCase().equals(fieldNameLow+"_$eq")) &&
+   if((methodNameLow.equals("set"+fieldNameLow) || 
methodNameLow.equals(fieldNameLow+"_$eq")) &&

[2/4] flink git commit: [FLINK-2893] [runtime] Consistent naming of recovery config parameters

[fhueske]

[FLINK-2893] [runtime] Consistent naming of recovery config parameters

Rename config key prefix from 'ha.zookeeper' to 'recovery.zookeeper'
Rename config key from 'state.backend.fs.dir.recovery' => 
'state.backend.fs.recoverydir'
Move ZooKeeper file system state backend configuration keys

This closes #1286


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/8ec828c9
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/8ec828c9
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/8ec828c9

Branch: refs/heads/release-0.10
Commit: 8ec828c9444711ab5956904bcffc149a5724e1e6
Parents: c0d7073
Author: Ufuk Celebi 
Authored: Thu Oct 22 12:58:26 2015 +0200
Committer: Fabian Hueske 
Committed: Fri Oct 23 16:05:26 2015 +0200

--
 docs/setup/config.md| 18 +--
 docs/setup/jobmanager_high_availability.md  |  6 ++--
 .../flink/configuration/ConfigConstants.java| 34 ++--
 flink-dist/src/main/resources/flink-conf.yaml   |  2 +-
 .../apache/flink/runtime/blob/BlobServer.java   |  2 +-
 .../flink/runtime/blob/FileSystemBlobStore.java |  4 +--
 .../flink/runtime/util/ZooKeeperUtils.java  |  4 +--
 .../flink/runtime/blob/BlobRecoveryITCase.java  |  2 +-
 .../BlobLibraryCacheRecoveryITCase.java |  2 +-
 .../runtime/testutils/JobManagerProcess.java|  2 +-
 .../runtime/testutils/TaskManagerProcess.java   |  2 +-
 .../runtime/testutils/ZooKeeperTestUtils.java   |  2 +-
 .../flink/test/recovery/ChaosMonkeyITCase.java  |  2 +-
 .../ZooKeeperLeaderElectionITCase.java  |  4 +--
 .../flink/yarn/YARNHighAvailabilityITCase.java  |  4 +--
 15 files changed, 45 insertions(+), 45 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/8ec828c9/docs/setup/config.md
--
diff --git a/docs/setup/config.md b/docs/setup/config.md
index 0e7b2ee..447a329 100644
--- a/docs/setup/config.md
+++ b/docs/setup/config.md
@@ -390,23 +390,23 @@ Flink supports the 'standalone' mode where only a single 
JobManager runs and no
 The high availability mode 'zookeeper' supports the execution of multiple 
JobManagers and JobManager state checkpointing.
 Among the group of JobManagers, ZooKeeper elects one of them as the leader 
which is responsible for the cluster execution.
 In case of a JobManager failure, a standby JobManager will be elected as the 
new leader and is given the last checkpointed JobManager state.
-In order to use the 'zookeeper' mode, it is mandatory to also define the 
`ha.zookeeper.quorum` configuration value.
+In order to use the 'zookeeper' mode, it is mandatory to also define the 
`recovery.zookeeper.quorum` configuration value.
 
-- `ha.zookeeper.quorum`: Defines the ZooKeeper quorum URL which is used to 
connet to the ZooKeeper cluster when the 'zookeeper' recovery mode is selected
+- `recovery.zookeeper.quorum`: Defines the ZooKeeper quorum URL which is used 
to connet to the ZooKeeper cluster when the 'zookeeper' recovery mode is 
selected
 
-- `ha.zookeeper.dir`: (Default '/flink') Defines the root dir under which the 
ZooKeeper recovery mode will create znodes. 
+- `recovery.zookeeper.path.root`: (Default '/flink') Defines the root dir 
under which the ZooKeeper recovery mode will create znodes. 
 
-- `ha.zookeeper.dir.latch`: (Default '/leaderlatch') Defines the znode of the 
leader latch which is used to elect the leader.
+- `recovery.zookeeper.path.latch`: (Default '/leaderlatch') Defines the znode 
of the leader latch which is used to elect the leader.
 
-- `ha.zookeeper.dir.leader`: (Default '/leader') Defines the znode of the 
leader which contains the URL to the leader and the current leader session ID
+- `recovery.zookeeper.path.leader`: (Default '/leader') Defines the znode of 
the leader which contains the URL to the leader and the current leader session 
ID
 
-- `ha.zookeeper.client.session-timeout`: (Default '6') Defines the session 
timeout for the ZooKeeper session in ms.
+- `recovery.zookeeper.client.session-timeout`: (Default '6') Defines the 
session timeout for the ZooKeeper session in ms.
 
-- `ha.zookeeper.client.connection-timeout`: (Default '15000') Defines the 
connection timeout for ZooKeeper in ms.
+- `recovery.zookeeper.client.connection-timeout`: (Default '15000') Defines 
the connection timeout for ZooKeeper in ms.
 
-- `ha.zookeeper.client.retry-wait`: (Default '5000') Defines the pause between 
consecutive retries in ms.
+- `recovery.zookeeper.client.retry-wait`: (Default '5000') Defines the pause 
between consecutive retries in ms.
 
-- `ha.zookeeper.client.max-retry-attempts`: (Default '3') Defines the number 
of connection retries before the client gives up.
+- 

[3/4] flink git commit: [FLINK-1982] [record-api] Remove dependencies on Record API from flink-runtime tests

[fhueske]

http://git-wip-us.apache.org/repos/asf/flink/blob/c0d7073a/flink-runtime/src/test/java/org/apache/flink/runtime/operators/MatchTaskTest.java
--
diff --git 
a/flink-runtime/src/test/java/org/apache/flink/runtime/operators/MatchTaskTest.java
 
b/flink-runtime/src/test/java/org/apache/flink/runtime/operators/MatchTaskTest.java
deleted file mode 100644
index 6c4659d..000
--- 
a/flink-runtime/src/test/java/org/apache/flink/runtime/operators/MatchTaskTest.java
+++ /dev/null
@@ -1,1019 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.runtime.operators;
-
-import java.util.ArrayList;
-import java.util.List;
-import java.util.concurrent.atomic.AtomicBoolean;
-import java.util.concurrent.atomic.AtomicReference;
-
-import org.apache.flink.api.common.ExecutionConfig;
-import org.apache.flink.api.common.functions.FlatJoinFunction;
-import org.apache.flink.api.common.typeutils.record.RecordComparator;
-import 
org.apache.flink.api.common.typeutils.record.RecordPairComparatorFactory;
-import org.apache.flink.api.java.record.functions.JoinFunction;
-import 
org.apache.flink.runtime.operators.testutils.DelayingInfinitiveInputIterator;
-import org.apache.flink.runtime.operators.testutils.DriverTestBase;
-import org.apache.flink.runtime.operators.testutils.ExpectedTestException;
-import org.apache.flink.runtime.operators.testutils.NirvanaOutputList;
-import org.apache.flink.runtime.operators.testutils.TaskCancelThread;
-import org.apache.flink.runtime.operators.testutils.UniformRecordGenerator;
-import org.apache.flink.types.IntValue;
-import org.apache.flink.types.Key;
-import org.apache.flink.types.Record;
-import org.apache.flink.util.Collector;
-
-import org.junit.Assert;
-import org.junit.Test;
-
-import static org.junit.Assert.*;
-
-@SuppressWarnings("deprecation")
-public class MatchTaskTest extends DriverTestBase> {
-   
-   private static final long HASH_MEM = 6*1024*1024;
-   
-   private static final long SORT_MEM = 3*1024*1024;
-
-   private static final int NUM_SORTER = 2;
-   
-   private static final long BNLJN_MEM = 10 * PAGE_SIZE;
-
-   private final double bnljn_frac;
-
-   private final double hash_frac;
-   
-   @SuppressWarnings("unchecked")
-   private final RecordComparator comparator1 = new RecordComparator(
-   new int[]{0}, (Class>[])new Class[]{ 
IntValue.class });
-   
-   @SuppressWarnings("unchecked")
-   private final RecordComparator comparator2 = new RecordComparator(
-   new int[]{0}, (Class>[])new Class[]{ 
IntValue.class });
-   
-   private final List outList = new ArrayList();
-   
-   
-   public MatchTaskTest(ExecutionConfig config) {
-   super(config, HASH_MEM, NUM_SORTER, SORT_MEM);
-   bnljn_frac = 
(double)BNLJN_MEM/this.getMemoryManager().getMemorySize();
-   hash_frac = 
(double)HASH_MEM/this.getMemoryManager().getMemorySize();
-   }
-   
-   
-   @Test
-   public void testSortBoth1MatchTask() {
-   final int keyCnt1 = 20;
-   final int valCnt1 = 1;
-   
-   final int keyCnt2 = 10;
-   final int valCnt2 = 2;
-   
-   setOutput(this.outList);
-   addDriverComparator(this.comparator1);
-   addDriverComparator(this.comparator2);
-   
getTaskConfig().setDriverPairComparator(RecordPairComparatorFactory.get());
-   getTaskConfig().setDriverStrategy(DriverStrategy.INNER_MERGE);
-   getTaskConfig().setRelativeMemoryDriver(bnljn_frac);
-   setNumFileHandlesForSort(4);
-   
-   final JoinDriver testTask = new 
JoinDriver();
-   
-   try {
-   addInputSorted(new UniformRecordGenerator(keyCnt1, 
valCnt1, false), this.comparator1.duplicate());
-   addInputSorted(new UniformRecordGenerator(keyCnt2, 
valCnt2, false), 

[2/5] flink git commit: [FLINK-1982] [record-api] Remove dependencies on Record API from flink-runtime tests

[fhueske]

[FLINK-1982] [record-api] Remove dependencies on Record API from flink-runtime 
tests

Rename Match*Test to Join*Test and MapTaskTest to FlatMapTaskTest

This closes #1294


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/3c8a6588
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/3c8a6588
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/3c8a6588

Branch: refs/heads/master
Commit: 3c8a6588ac9bd35984d6e3b1eef916461a262fe4
Parents: 7ff071f
Author: Fabian Hueske 
Authored: Thu Oct 22 21:10:41 2015 +0200
Committer: Fabian Hueske 
Committed: Fri Oct 23 13:01:37 2015 +0200

--
 .../operators/CombineTaskExternalITCase.java|   16 +-
 .../flink/runtime/operators/CrossTaskTest.java  |   41 +-
 .../runtime/operators/DataSinkTaskTest.java |   52 +-
 .../runtime/operators/DataSourceTaskTest.java   |   16 +-
 .../runtime/operators/FlatMapTaskTest.java  |  149 +++
 .../operators/JoinTaskExternalITCase.java   |  165 +++
 .../flink/runtime/operators/JoinTaskTest.java   | 1017 +
 .../flink/runtime/operators/MapTaskTest.java|  151 ---
 .../operators/MatchTaskExternalITCase.java  |  167 ---
 .../flink/runtime/operators/MatchTaskTest.java  | 1019 --
 .../operators/ReduceTaskExternalITCase.java |   16 +-
 .../flink/runtime/operators/ReduceTaskTest.java |   19 +-
 .../operators/chaining/ChainTaskTest.java   |   25 +-
 .../operators/testutils/TaskTestBase.java   |   18 +-
 14 files changed, 1428 insertions(+), 1443 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/3c8a6588/flink-runtime/src/test/java/org/apache/flink/runtime/operators/CombineTaskExternalITCase.java
--
diff --git 
a/flink-runtime/src/test/java/org/apache/flink/runtime/operators/CombineTaskExternalITCase.java
 
b/flink-runtime/src/test/java/org/apache/flink/runtime/operators/CombineTaskExternalITCase.java
index 4905e57..800bca7 100644
--- 
a/flink-runtime/src/test/java/org/apache/flink/runtime/operators/CombineTaskExternalITCase.java
+++ 
b/flink-runtime/src/test/java/org/apache/flink/runtime/operators/CombineTaskExternalITCase.java
@@ -22,11 +22,11 @@ import java.util.ArrayList;
 import java.util.HashMap;
 
 import org.apache.flink.api.common.ExecutionConfig;
-import org.apache.flink.api.java.record.operators.ReduceOperator;
 import org.apache.flink.runtime.operators.testutils.ExpectedTestException;
 import org.apache.flink.util.Collector;
 import org.junit.Assert;
 import org.apache.flink.api.common.functions.RichGroupReduceFunction;
+import 
org.apache.flink.api.common.functions.RichGroupReduceFunction.Combinable;
 import org.apache.flink.api.common.typeutils.record.RecordComparator;
 import org.apache.flink.runtime.operators.testutils.DriverTestBase;
 import org.apache.flink.runtime.operators.testutils.UniformRecordGenerator;
@@ -42,7 +42,7 @@ public class CombineTaskExternalITCase extends 
DriverTestBase outList = new ArrayList();
+   private final ArrayList outList = new ArrayList<>();

@SuppressWarnings("unchecked")
private final RecordComparator comparator = new RecordComparator(
@@ -69,7 +69,7 @@ public class CombineTaskExternalITCase extends 
DriverTestBase testTask = new 
GroupReduceCombineDriver();
+   final GroupReduceCombineDriver testTask = new 
GroupReduceCombineDriver<>();

try {
testDriver(testTask, MockCombiningReduceStub.class);
@@ -85,7 +85,7 @@ public class CombineTaskExternalITCase extends 
DriverTestBase aggMap = new 
HashMap();
+   final HashMap aggMap = new HashMap<>();
for (Record record : this.outList) {
IntValue key = new IntValue();
IntValue value = new IntValue();
@@ -123,7 +123,7 @@ public class CombineTaskExternalITCase extends 
DriverTestBase testTask = new 
GroupReduceCombineDriver();
+   final GroupReduceCombineDriver testTask = new 
GroupReduceCombineDriver<>();
 
try {
testDriver(testTask, MockCombiningReduceStub.class);
@@ -139,7 +139,7 @@ public class CombineTaskExternalITCase extends 
DriverTestBase aggMap = new 
HashMap();
+   final HashMap aggMap = new HashMap<>();
for (Record record : this.outList) {
IntValue key = new IntValue();
IntValue value = new IntValue();
@@ -166,7 +166,7 @@ public class CombineTaskExternalITCase extends 
DriverTestBase {
private static 

[4/4] flink git commit: [FLINK-1982] [record-api] Remove dependencies on Record API from flink-runtime tests

[fhueske]

[FLINK-1982] [record-api] Remove dependencies on Record API from flink-runtime 
tests

Rename Match*Test to Join*Test and MapTaskTest to FlatMapTaskTest

This closes #1294


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/c0d7073a
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/c0d7073a
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/c0d7073a

Branch: refs/heads/release-0.10
Commit: c0d7073a5166fdcd6cfbea741ae0bfa132e2bf04
Parents: 712c868
Author: Fabian Hueske 
Authored: Thu Oct 22 21:10:41 2015 +0200
Committer: Fabian Hueske 
Committed: Fri Oct 23 16:05:26 2015 +0200

--
 .../operators/CombineTaskExternalITCase.java|   16 +-
 .../flink/runtime/operators/CrossTaskTest.java  |   41 +-
 .../runtime/operators/DataSinkTaskTest.java |   52 +-
 .../runtime/operators/DataSourceTaskTest.java   |   16 +-
 .../runtime/operators/FlatMapTaskTest.java  |  149 +++
 .../operators/JoinTaskExternalITCase.java   |  165 +++
 .../flink/runtime/operators/JoinTaskTest.java   | 1017 +
 .../flink/runtime/operators/MapTaskTest.java|  151 ---
 .../operators/MatchTaskExternalITCase.java  |  167 ---
 .../flink/runtime/operators/MatchTaskTest.java  | 1019 --
 .../operators/ReduceTaskExternalITCase.java |   16 +-
 .../flink/runtime/operators/ReduceTaskTest.java |   19 +-
 .../operators/chaining/ChainTaskTest.java   |   25 +-
 .../operators/testutils/TaskTestBase.java   |   18 +-
 14 files changed, 1428 insertions(+), 1443 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/c0d7073a/flink-runtime/src/test/java/org/apache/flink/runtime/operators/CombineTaskExternalITCase.java
--
diff --git 
a/flink-runtime/src/test/java/org/apache/flink/runtime/operators/CombineTaskExternalITCase.java
 
b/flink-runtime/src/test/java/org/apache/flink/runtime/operators/CombineTaskExternalITCase.java
index 4905e57..800bca7 100644
--- 
a/flink-runtime/src/test/java/org/apache/flink/runtime/operators/CombineTaskExternalITCase.java
+++ 
b/flink-runtime/src/test/java/org/apache/flink/runtime/operators/CombineTaskExternalITCase.java
@@ -22,11 +22,11 @@ import java.util.ArrayList;
 import java.util.HashMap;
 
 import org.apache.flink.api.common.ExecutionConfig;
-import org.apache.flink.api.java.record.operators.ReduceOperator;
 import org.apache.flink.runtime.operators.testutils.ExpectedTestException;
 import org.apache.flink.util.Collector;
 import org.junit.Assert;
 import org.apache.flink.api.common.functions.RichGroupReduceFunction;
+import 
org.apache.flink.api.common.functions.RichGroupReduceFunction.Combinable;
 import org.apache.flink.api.common.typeutils.record.RecordComparator;
 import org.apache.flink.runtime.operators.testutils.DriverTestBase;
 import org.apache.flink.runtime.operators.testutils.UniformRecordGenerator;
@@ -42,7 +42,7 @@ public class CombineTaskExternalITCase extends 
DriverTestBase outList = new ArrayList();
+   private final ArrayList outList = new ArrayList<>();

@SuppressWarnings("unchecked")
private final RecordComparator comparator = new RecordComparator(
@@ -69,7 +69,7 @@ public class CombineTaskExternalITCase extends 
DriverTestBase testTask = new 
GroupReduceCombineDriver();
+   final GroupReduceCombineDriver testTask = new 
GroupReduceCombineDriver<>();

try {
testDriver(testTask, MockCombiningReduceStub.class);
@@ -85,7 +85,7 @@ public class CombineTaskExternalITCase extends 
DriverTestBase aggMap = new 
HashMap();
+   final HashMap aggMap = new HashMap<>();
for (Record record : this.outList) {
IntValue key = new IntValue();
IntValue value = new IntValue();
@@ -123,7 +123,7 @@ public class CombineTaskExternalITCase extends 
DriverTestBase testTask = new 
GroupReduceCombineDriver();
+   final GroupReduceCombineDriver testTask = new 
GroupReduceCombineDriver<>();
 
try {
testDriver(testTask, MockCombiningReduceStub.class);
@@ -139,7 +139,7 @@ public class CombineTaskExternalITCase extends 
DriverTestBase aggMap = new 
HashMap();
+   final HashMap aggMap = new HashMap<>();
for (Record record : this.outList) {
IntValue key = new IntValue();
IntValue value = new IntValue();
@@ -166,7 +166,7 @@ public class CombineTaskExternalITCase extends 
DriverTestBase {
private 

[1/4] flink git commit: [FLINK-2895] Duplicate immutable object creation

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/release-0.10 712c868eb -> 15d3f10c1


[FLINK-2895] Duplicate immutable object creation

Operators defer object creation when object reuse is disabled.

This closes #1288


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/15d3f10c
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/15d3f10c
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/15d3f10c

Branch: refs/heads/release-0.10
Commit: 15d3f10c1a52b2b1cc8fff793648da4a56e72dcc
Parents: 8ec828c
Author: Greg Hogan 
Authored: Thu Oct 22 09:31:09 2015 -0400
Committer: Fabian Hueske 
Committed: Fri Oct 23 16:05:26 2015 +0200

--
 .../org/apache/flink/runtime/operators/AllReduceDriver.java  | 8 +++-
 .../java/org/apache/flink/runtime/operators/NoOpDriver.java  | 4 +---
 .../org/apache/flink/runtime/operators/ReduceDriver.java | 4 ++--
 3 files changed, 6 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/15d3f10c/flink-runtime/src/main/java/org/apache/flink/runtime/operators/AllReduceDriver.java
--
diff --git 
a/flink-runtime/src/main/java/org/apache/flink/runtime/operators/AllReduceDriver.java
 
b/flink-runtime/src/main/java/org/apache/flink/runtime/operators/AllReduceDriver.java
index 06f22c5..1d35fdb 100644
--- 
a/flink-runtime/src/main/java/org/apache/flink/runtime/operators/AllReduceDriver.java
+++ 
b/flink-runtime/src/main/java/org/apache/flink/runtime/operators/AllReduceDriver.java
@@ -108,7 +108,6 @@ public class AllReduceDriver implements 
Driver {
final MutableObjectIterator input = this.input;
final TypeSerializer serializer = this.serializer;
 
-
if (objectReuseEnabled) {
T val1 = serializer.createInstance();
 
@@ -123,14 +122,13 @@ public class AllReduceDriver implements 
Driver {
 
this.taskContext.getOutputCollector().collect(val1);
} else {
-   T val1 = serializer.createInstance();
-
-   if ((val1 = input.next(val1)) == null) {
+   T val1;
+   if ((val1 = input.next()) == null) {
return;
}
 
T val2;
-   while (running && (val2 = 
input.next(serializer.createInstance())) != null) {
+   while (running && (val2 = input.next()) != null) {
val1 = stub.reduce(val1, val2);
}
 

http://git-wip-us.apache.org/repos/asf/flink/blob/15d3f10c/flink-runtime/src/main/java/org/apache/flink/runtime/operators/NoOpDriver.java
--
diff --git 
a/flink-runtime/src/main/java/org/apache/flink/runtime/operators/NoOpDriver.java
 
b/flink-runtime/src/main/java/org/apache/flink/runtime/operators/NoOpDriver.java
index fcd2716..428cfe4 100644
--- 
a/flink-runtime/src/main/java/org/apache/flink/runtime/operators/NoOpDriver.java
+++ 
b/flink-runtime/src/main/java/org/apache/flink/runtime/operators/NoOpDriver.java
@@ -21,7 +21,6 @@ package org.apache.flink.runtime.operators;
 
 import org.apache.flink.api.common.ExecutionConfig;
 import org.apache.flink.api.common.functions.AbstractRichFunction;
-import org.apache.flink.api.common.typeutils.TypeSerializer;
 import org.apache.flink.util.Collector;
 import org.apache.flink.util.MutableObjectIterator;
 import org.slf4j.Logger;
@@ -87,8 +86,7 @@ public class NoOpDriver implements 
Driver {
}
} else {
T record;
-   TypeSerializer serializer = 
this.taskContext.getInputSerializer(0).getSerializer();
-   while (this.running && ((record = 
input.next(serializer.createInstance())) != null)) {
+   while (this.running && ((record = input.next()) != 
null)) {
output.collect(record);
}
 

http://git-wip-us.apache.org/repos/asf/flink/blob/15d3f10c/flink-runtime/src/main/java/org/apache/flink/runtime/operators/ReduceDriver.java
--
diff --git 
a/flink-runtime/src/main/java/org/apache/flink/runtime/operators/ReduceDriver.java
 
b/flink-runtime/src/main/java/org/apache/flink/runtime/operators/ReduceDriver.java
index 970441e..6a7c42c 100644
--- 
a/flink-runtime/src/main/java/org/apache/flink/runtime/operators/ReduceDriver.java
+++ 

[1/5] flink git commit: [FLINK-1982] [record-api] Remove dependencies on Record API from flink-runtime tests

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master 7ff071f66 -> bbb75c599


http://git-wip-us.apache.org/repos/asf/flink/blob/3c8a6588/flink-runtime/src/test/java/org/apache/flink/runtime/operators/MatchTaskTest.java
--
diff --git 
a/flink-runtime/src/test/java/org/apache/flink/runtime/operators/MatchTaskTest.java
 
b/flink-runtime/src/test/java/org/apache/flink/runtime/operators/MatchTaskTest.java
deleted file mode 100644
index 6c4659d..000
--- 
a/flink-runtime/src/test/java/org/apache/flink/runtime/operators/MatchTaskTest.java
+++ /dev/null
@@ -1,1019 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.runtime.operators;
-
-import java.util.ArrayList;
-import java.util.List;
-import java.util.concurrent.atomic.AtomicBoolean;
-import java.util.concurrent.atomic.AtomicReference;
-
-import org.apache.flink.api.common.ExecutionConfig;
-import org.apache.flink.api.common.functions.FlatJoinFunction;
-import org.apache.flink.api.common.typeutils.record.RecordComparator;
-import 
org.apache.flink.api.common.typeutils.record.RecordPairComparatorFactory;
-import org.apache.flink.api.java.record.functions.JoinFunction;
-import 
org.apache.flink.runtime.operators.testutils.DelayingInfinitiveInputIterator;
-import org.apache.flink.runtime.operators.testutils.DriverTestBase;
-import org.apache.flink.runtime.operators.testutils.ExpectedTestException;
-import org.apache.flink.runtime.operators.testutils.NirvanaOutputList;
-import org.apache.flink.runtime.operators.testutils.TaskCancelThread;
-import org.apache.flink.runtime.operators.testutils.UniformRecordGenerator;
-import org.apache.flink.types.IntValue;
-import org.apache.flink.types.Key;
-import org.apache.flink.types.Record;
-import org.apache.flink.util.Collector;
-
-import org.junit.Assert;
-import org.junit.Test;
-
-import static org.junit.Assert.*;
-
-@SuppressWarnings("deprecation")
-public class MatchTaskTest extends DriverTestBase> {
-   
-   private static final long HASH_MEM = 6*1024*1024;
-   
-   private static final long SORT_MEM = 3*1024*1024;
-
-   private static final int NUM_SORTER = 2;
-   
-   private static final long BNLJN_MEM = 10 * PAGE_SIZE;
-
-   private final double bnljn_frac;
-
-   private final double hash_frac;
-   
-   @SuppressWarnings("unchecked")
-   private final RecordComparator comparator1 = new RecordComparator(
-   new int[]{0}, (Class>[])new Class[]{ 
IntValue.class });
-   
-   @SuppressWarnings("unchecked")
-   private final RecordComparator comparator2 = new RecordComparator(
-   new int[]{0}, (Class>[])new Class[]{ 
IntValue.class });
-   
-   private final List outList = new ArrayList();
-   
-   
-   public MatchTaskTest(ExecutionConfig config) {
-   super(config, HASH_MEM, NUM_SORTER, SORT_MEM);
-   bnljn_frac = 
(double)BNLJN_MEM/this.getMemoryManager().getMemorySize();
-   hash_frac = 
(double)HASH_MEM/this.getMemoryManager().getMemorySize();
-   }
-   
-   
-   @Test
-   public void testSortBoth1MatchTask() {
-   final int keyCnt1 = 20;
-   final int valCnt1 = 1;
-   
-   final int keyCnt2 = 10;
-   final int valCnt2 = 2;
-   
-   setOutput(this.outList);
-   addDriverComparator(this.comparator1);
-   addDriverComparator(this.comparator2);
-   
getTaskConfig().setDriverPairComparator(RecordPairComparatorFactory.get());
-   getTaskConfig().setDriverStrategy(DriverStrategy.INNER_MERGE);
-   getTaskConfig().setRelativeMemoryDriver(bnljn_frac);
-   setNumFileHandlesForSort(4);
-   
-   final JoinDriver testTask = new 
JoinDriver();
-   
-   try {
-   addInputSorted(new UniformRecordGenerator(keyCnt1, 
valCnt1, false), this.comparator1.duplicate());
-   

[3/5] flink git commit: [FLINK-2893] [runtime] Consistent naming of recovery config parameters

[fhueske]

[FLINK-2893] [runtime] Consistent naming of recovery config parameters

Rename config key prefix from 'ha.zookeeper' to 'recovery.zookeeper'
Rename config key from 'state.backend.fs.dir.recovery' => 
'state.backend.fs.recoverydir'
Move ZooKeeper file system state backend configuration keys

This closes #1286


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/ab2895fa
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/ab2895fa
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/ab2895fa

Branch: refs/heads/master
Commit: ab2895fa988018300395557d110de23a3a2166c9
Parents: 3c8a658
Author: Ufuk Celebi 
Authored: Thu Oct 22 12:58:26 2015 +0200
Committer: Fabian Hueske 
Committed: Fri Oct 23 13:03:02 2015 +0200

--
 docs/setup/config.md| 18 +--
 docs/setup/jobmanager_high_availability.md  |  6 ++--
 .../flink/configuration/ConfigConstants.java| 34 ++--
 flink-dist/src/main/resources/flink-conf.yaml   |  2 +-
 .../apache/flink/runtime/blob/BlobServer.java   |  2 +-
 .../flink/runtime/blob/FileSystemBlobStore.java |  4 +--
 .../flink/runtime/util/ZooKeeperUtils.java  |  4 +--
 .../flink/runtime/blob/BlobRecoveryITCase.java  |  2 +-
 .../BlobLibraryCacheRecoveryITCase.java |  2 +-
 .../runtime/testutils/JobManagerProcess.java|  2 +-
 .../runtime/testutils/TaskManagerProcess.java   |  2 +-
 .../runtime/testutils/ZooKeeperTestUtils.java   |  2 +-
 .../flink/test/recovery/ChaosMonkeyITCase.java  |  2 +-
 .../ZooKeeperLeaderElectionITCase.java  |  4 +--
 .../flink/yarn/YARNHighAvailabilityITCase.java  |  4 +--
 15 files changed, 45 insertions(+), 45 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/ab2895fa/docs/setup/config.md
--
diff --git a/docs/setup/config.md b/docs/setup/config.md
index 0e7b2ee..447a329 100644
--- a/docs/setup/config.md
+++ b/docs/setup/config.md
@@ -390,23 +390,23 @@ Flink supports the 'standalone' mode where only a single 
JobManager runs and no
 The high availability mode 'zookeeper' supports the execution of multiple 
JobManagers and JobManager state checkpointing.
 Among the group of JobManagers, ZooKeeper elects one of them as the leader 
which is responsible for the cluster execution.
 In case of a JobManager failure, a standby JobManager will be elected as the 
new leader and is given the last checkpointed JobManager state.
-In order to use the 'zookeeper' mode, it is mandatory to also define the 
`ha.zookeeper.quorum` configuration value.
+In order to use the 'zookeeper' mode, it is mandatory to also define the 
`recovery.zookeeper.quorum` configuration value.
 
-- `ha.zookeeper.quorum`: Defines the ZooKeeper quorum URL which is used to 
connet to the ZooKeeper cluster when the 'zookeeper' recovery mode is selected
+- `recovery.zookeeper.quorum`: Defines the ZooKeeper quorum URL which is used 
to connet to the ZooKeeper cluster when the 'zookeeper' recovery mode is 
selected
 
-- `ha.zookeeper.dir`: (Default '/flink') Defines the root dir under which the 
ZooKeeper recovery mode will create znodes. 
+- `recovery.zookeeper.path.root`: (Default '/flink') Defines the root dir 
under which the ZooKeeper recovery mode will create znodes. 
 
-- `ha.zookeeper.dir.latch`: (Default '/leaderlatch') Defines the znode of the 
leader latch which is used to elect the leader.
+- `recovery.zookeeper.path.latch`: (Default '/leaderlatch') Defines the znode 
of the leader latch which is used to elect the leader.
 
-- `ha.zookeeper.dir.leader`: (Default '/leader') Defines the znode of the 
leader which contains the URL to the leader and the current leader session ID
+- `recovery.zookeeper.path.leader`: (Default '/leader') Defines the znode of 
the leader which contains the URL to the leader and the current leader session 
ID
 
-- `ha.zookeeper.client.session-timeout`: (Default '6') Defines the session 
timeout for the ZooKeeper session in ms.
+- `recovery.zookeeper.client.session-timeout`: (Default '6') Defines the 
session timeout for the ZooKeeper session in ms.
 
-- `ha.zookeeper.client.connection-timeout`: (Default '15000') Defines the 
connection timeout for ZooKeeper in ms.
+- `recovery.zookeeper.client.connection-timeout`: (Default '15000') Defines 
the connection timeout for ZooKeeper in ms.
 
-- `ha.zookeeper.client.retry-wait`: (Default '5000') Defines the pause between 
consecutive retries in ms.
+- `recovery.zookeeper.client.retry-wait`: (Default '5000') Defines the pause 
between consecutive retries in ms.
 
-- `ha.zookeeper.client.max-retry-attempts`: (Default '3') Defines the number 
of connection retries before the client gives up.
+- 

[4/5] flink git commit: [FLINK-2898] [build] Invert Travis CI build order

[fhueske]

[FLINK-2898] [build] Invert Travis CI build order

This closes #1290


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/d2e4a27d
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/d2e4a27d
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/d2e4a27d

Branch: refs/heads/master
Commit: d2e4a27d66faf756c8da5236588c4043ba559ecd
Parents: ab2895f
Author: Greg Hogan 
Authored: Thu Oct 22 11:12:43 2015 -0400
Committer: Fabian Hueske 
Committed: Fri Oct 23 13:05:36 2015 +0200

--
 .travis.yml | 16 
 1 file changed, 8 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/d2e4a27d/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index a626519..7213a0c 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -14,16 +14,16 @@ language: java
 #See https://issues.apache.org/jira/browse/FLINK-1072
 matrix:
   include:
-- jdk: "openjdk7" # this uploads the Hadoop 1 build to Maven and S3
-  env: PROFILE="-Dhadoop.profile=1"
-- jdk: "oraclejdk7" # this uploads the Hadoop 2 build to Maven and S3
-  env: PROFILE="-Dhadoop.version=2.3.0"
-- jdk: "openjdk7"
-  env: PROFILE="-Dhadoop.version=2.4.0 -Dscala-2.11 -Pinclude-yarn-tests"
-- jdk: "oraclejdk8"
-  env: PROFILE="-Dhadoop.version=2.5.0 -Dmaven.javadoc.skip=true 
-Pinclude-yarn-tests"
 - jdk: "oraclejdk8"
   env: PROFILE="-Dhadoop.version=2.6.0 -Dscala-2.11 -Pinclude-tez 
-Pinclude-yarn-tests -Dmaven.javadoc.skip=true"
+- jdk: "oraclejdk8"
+  env: PROFILE="-Dhadoop.version=2.5.0 -Dmaven.javadoc.skip=true 
-Pinclude-yarn-tests"
+- jdk: "openjdk7"
+  env: PROFILE="-Dhadoop.version=2.4.0 -Dscala-2.11 -Pinclude-yarn-tests"
+- jdk: "oraclejdk7" # this uploads the Hadoop 2 build to Maven and S3
+  env: PROFILE="-Dhadoop.version=2.3.0"
+- jdk: "openjdk7" # this uploads the Hadoop 1 build to Maven and S3
+  env: PROFILE="-Dhadoop.profile=1"
 
 
 git:

[1/8] flink git commit: [FLINK-2890] Port StringSerializationSpeedBenchmark to JMH. [FLINK-2889] Port LongSerializationSpeedBenchmark to JMH.

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/master daa357aca -> 6a8e90b36


http://git-wip-us.apache.org/repos/asf/flink/blob/75a52574/flink-core/src/test/java/org/apache/flink/core/memory/benchmarks/PureOffHeapMemorySegmentOutView.java
--
diff --git 
a/flink-core/src/test/java/org/apache/flink/core/memory/benchmarks/PureOffHeapMemorySegmentOutView.java
 
b/flink-core/src/test/java/org/apache/flink/core/memory/benchmarks/PureOffHeapMemorySegmentOutView.java
deleted file mode 100644
index d898dee..000
--- 
a/flink-core/src/test/java/org/apache/flink/core/memory/benchmarks/PureOffHeapMemorySegmentOutView.java
+++ /dev/null
@@ -1,359 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.core.memory.benchmarks;
-
-import org.apache.flink.core.memory.DataInputView;
-import org.apache.flink.core.memory.DataOutputView;
-
-import java.io.EOFException;
-import java.io.IOException;
-import java.io.UTFDataFormatException;
-import java.util.List;
-
-public final class PureOffHeapMemorySegmentOutView implements DataOutputView {
-
-   private PureOffHeapMemorySegment currentSegment;// the current 
memory segment to write to
-
-   private int positionInSegment;  // the 
offset in the current segment
-   
-   private final int segmentSize;  // the size of 
the memory segments
-
-   private final  List memorySource;
-   
-   private final List fullSegments;
-   
-
-   private byte[] utfBuffer;   // the reusable array for UTF 
encodings
-
-
-   public PureOffHeapMemorySegmentOutView(List 
emptySegments,
-   
   List fullSegmentTarget, int segmentSize) {
-   this.segmentSize = segmentSize;
-   this.currentSegment = emptySegments.remove(emptySegments.size() 
- 1);
-
-   this.memorySource = emptySegments;
-   this.fullSegments = fullSegmentTarget;
-   this.fullSegments.add(getCurrentSegment());
-   }
-
-
-   public void reset() {
-   if (this.fullSegments.size() != 0) {
-   throw new IllegalStateException("The target list still 
contains memory segments.");
-   }
-
-   clear();
-   try {
-   advance();
-   }
-   catch (IOException ioex) {
-   throw new RuntimeException("Error getting first segment 
for record collector.", ioex);
-   }
-   }
-   
-   // 

-   //  Page Management
-   // 

-
-   public PureOffHeapMemorySegment nextSegment(PureOffHeapMemorySegment 
current, int positionInCurrent) throws EOFException {
-   int size = this.memorySource.size();
-   if (size > 0) {
-   final PureOffHeapMemorySegment next = 
this.memorySource.remove(size - 1);
-   this.fullSegments.add(next);
-   return next;
-   } else {
-   throw new EOFException();
-   }
-   }
-   
-   public PureOffHeapMemorySegment getCurrentSegment() {
-   return this.currentSegment;
-   }
-
-   public int getCurrentPositionInSegment() {
-   return this.positionInSegment;
-   }
-   
-   public int getSegmentSize() {
-   return this.segmentSize;
-   }
-   
-   protected void advance() throws IOException {
-   this.currentSegment = nextSegment(this.currentSegment, 
this.positionInSegment);
-   this.positionInSegment = 0;
-   }
-   
-   protected void seekOutput(PureOffHeapMemorySegment seg, int position) {
-   this.currentSegment = seg;
-   this.positionInSegment = position;
-   }
-
-   protected 

[5/8] flink git commit: [FLINK-2890] Port StringSerializationSpeedBenchmark to JMH. [FLINK-2889] Port LongSerializationSpeedBenchmark to JMH.

[fhueske]

[FLINK-2890] Port StringSerializationSpeedBenchmark to JMH.
[FLINK-2889] Port LongSerializationSpeedBenchmark to JMH.

This closes #1284
This closes #1283


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/75a52574
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/75a52574
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/75a52574

Branch: refs/heads/master
Commit: 75a5257412606ac70113850439457cce7da3b2e6
Parents: daa357a
Author: gallenvara 
Authored: Thu Oct 22 14:35:11 2015 +0800
Committer: Fabian Hueske 
Committed: Mon Oct 26 20:04:38 2015 +0100

--
 flink-benchmark/pom.xml |9 +-
 .../memory/LongSerializationSpeedBenchmark.java |  225 +++
 .../StringSerializationSpeedBenchmark.java  |  209 +++
 .../segments/CoreMemorySegmentOutView.java  |  360 
 .../segments/MemorySegmentSpeedBenchmark.java   | 1633 ++
 .../memory/segments/PureHeapMemorySegment.java  |  466 +
 .../segments/PureHeapMemorySegmentOutView.java  |  359 
 .../segments/PureHybridMemorySegment.java   |  887 ++
 .../PureHybridMemorySegmentOutView.java |  359 
 .../segments/PureOffHeapMemorySegment.java  |  790 +
 .../PureOffHeapMemorySegmentOutView.java|  359 
 .../benchmarks/CoreMemorySegmentOutView.java|  360 
 .../LongSerializationSpeedBenchmark.java|  232 ---
 .../benchmarks/MemorySegmentSpeedBenchmark.java | 1633 --
 .../benchmarks/PureHeapMemorySegment.java   |  466 -
 .../PureHeapMemorySegmentOutView.java   |  359 
 .../benchmarks/PureHybridMemorySegment.java |  887 --
 .../PureHybridMemorySegmentOutView.java |  359 
 .../benchmarks/PureOffHeapMemorySegment.java|  790 -
 .../PureOffHeapMemorySegmentOutView.java|  359 
 .../StringSerializationSpeedBenchmark.java  |  207 ---
 21 files changed, 5655 insertions(+), 5653 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/75a52574/flink-benchmark/pom.xml
--
diff --git a/flink-benchmark/pom.xml b/flink-benchmark/pom.xml
index 281273f..d6ba1d7 100644
--- a/flink-benchmark/pom.xml
+++ b/flink-benchmark/pom.xml
@@ -36,7 +36,7 @@ under the License.
 
   
 UTF-8
-1.4.1
+1.11
 benchmarks
   
 
@@ -57,6 +57,13 @@ under the License.
   ${jmh.version}
   provided
 
+
+  org.apache.flink
+  flink-core
+  ${project.version}
+  test-jar
+  test
+  
   
 
   

http://git-wip-us.apache.org/repos/asf/flink/blob/75a52574/flink-benchmark/src/test/java/org/apache/flink/benchmark/core/memory/LongSerializationSpeedBenchmark.java
--
diff --git 
a/flink-benchmark/src/test/java/org/apache/flink/benchmark/core/memory/LongSerializationSpeedBenchmark.java
 
b/flink-benchmark/src/test/java/org/apache/flink/benchmark/core/memory/LongSerializationSpeedBenchmark.java
new file mode 100644
index 000..3e2605a
--- /dev/null
+++ 
b/flink-benchmark/src/test/java/org/apache/flink/benchmark/core/memory/LongSerializationSpeedBenchmark.java
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.benchmark.core.memory;
+
+import org.apache.flink.api.common.typeutils.base.LongSerializer;
+import 
org.apache.flink.benchmark.core.memory.segments.CoreMemorySegmentOutView;
+import org.apache.flink.benchmark.core.memory.segments.PureHeapMemorySegment;
+import 
org.apache.flink.benchmark.core.memory.segments.PureHeapMemorySegmentOutView;
+import org.apache.flink.benchmark.core.memory.segments.PureHybridMemorySegment;
+import 
org.apache.flink.benchmark.core.memory.segments.PureHybridMemorySegmentOutView;
+import org.apache.flink.core.memory.HeapMemorySegment;
+import org.apache.flink.core.memory.HybridMemorySegment;
+import org.apache.flink.core.memory.MemorySegment;

[7/8] flink git commit: [FLINK-2827] Close FileInputStream through try-with-resources to avoid unused open stream.

[fhueske]

[FLINK-2827] Close FileInputStream through try-with-resources to avoid unused 
open stream.

This closes #1276


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/6a8e90b3
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/6a8e90b3
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/6a8e90b3

Branch: refs/heads/master
Commit: 6a8e90b3621bb96304b325a4ae8f7f5575ec909a
Parents: 7265d81
Author: Saumitra Shahapure 
Authored: Tue Oct 20 19:28:31 2015 +0200
Committer: Fabian Hueske 
Committed: Mon Oct 26 20:05:03 2015 +0100

--
 .../apache/flink/streaming/connectors/twitter/TwitterSource.java | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/6a8e90b3/flink-streaming-connectors/flink-connector-twitter/src/main/java/org/apache/flink/streaming/connectors/twitter/TwitterSource.java
--
diff --git 
a/flink-streaming-connectors/flink-connector-twitter/src/main/java/org/apache/flink/streaming/connectors/twitter/TwitterSource.java
 
b/flink-streaming-connectors/flink-connector-twitter/src/main/java/org/apache/flink/streaming/connectors/twitter/TwitterSource.java
index bad0f8c..d290f98 100644
--- 
a/flink-streaming-connectors/flink-connector-twitter/src/main/java/org/apache/flink/streaming/connectors/twitter/TwitterSource.java
+++ 
b/flink-streaming-connectors/flink-connector-twitter/src/main/java/org/apache/flink/streaming/connectors/twitter/TwitterSource.java
@@ -133,10 +133,8 @@ public class TwitterSource extends 
RichSourceFunction {
private Properties loadAuthenticationProperties() {

Properties properties = new Properties();
-   try {
-   InputStream input = new FileInputStream(authPath);
+   try (InputStream input = new FileInputStream(authPath)) {
properties.load(input);
-   input.close();
} catch (Exception e) {
throw new RuntimeException("Cannot open .properties 
file: " + authPath, e);
}

[3/8] flink git commit: [FLINK-2890] Port StringSerializationSpeedBenchmark to JMH. [FLINK-2889] Port LongSerializationSpeedBenchmark to JMH.

[fhueske]

http://git-wip-us.apache.org/repos/asf/flink/blob/75a52574/flink-benchmark/src/test/java/org/apache/flink/benchmark/core/memory/segments/PureOffHeapMemorySegmentOutView.java
--
diff --git 
a/flink-benchmark/src/test/java/org/apache/flink/benchmark/core/memory/segments/PureOffHeapMemorySegmentOutView.java
 
b/flink-benchmark/src/test/java/org/apache/flink/benchmark/core/memory/segments/PureOffHeapMemorySegmentOutView.java
new file mode 100644
index 000..424a190
--- /dev/null
+++ 
b/flink-benchmark/src/test/java/org/apache/flink/benchmark/core/memory/segments/PureOffHeapMemorySegmentOutView.java
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.benchmark.core.memory.segments;
+
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.UTFDataFormatException;
+import java.util.List;
+
+public final class PureOffHeapMemorySegmentOutView implements DataOutputView {
+
+   private PureOffHeapMemorySegment currentSegment;// the current 
memory segment to write to
+
+   private int positionInSegment;  // the 
offset in the current segment
+   
+   private final int segmentSize;  // the size of 
the memory segments
+
+   private final  List memorySource;
+   
+   private final List fullSegments;
+   
+
+   private byte[] utfBuffer;   // the reusable array for UTF 
encodings
+
+
+   public PureOffHeapMemorySegmentOutView(List 
emptySegments,
+   
   List fullSegmentTarget, int segmentSize) {
+   this.segmentSize = segmentSize;
+   this.currentSegment = emptySegments.remove(emptySegments.size() 
- 1);
+
+   this.memorySource = emptySegments;
+   this.fullSegments = fullSegmentTarget;
+   this.fullSegments.add(getCurrentSegment());
+   }
+
+
+   public void reset() {
+   if (this.fullSegments.size() != 0) {
+   throw new IllegalStateException("The target list still 
contains memory segments.");
+   }
+
+   clear();
+   try {
+   advance();
+   }
+   catch (IOException ioex) {
+   throw new RuntimeException("Error getting first segment 
for record collector.", ioex);
+   }
+   }
+   
+   // 

+   //  Page Management
+   // 

+
+   public PureOffHeapMemorySegment nextSegment(PureOffHeapMemorySegment 
current, int positionInCurrent) throws EOFException {
+   int size = this.memorySource.size();
+   if (size > 0) {
+   final PureOffHeapMemorySegment next = 
this.memorySource.remove(size - 1);
+   this.fullSegments.add(next);
+   return next;
+   } else {
+   throw new EOFException();
+   }
+   }
+   
+   public PureOffHeapMemorySegment getCurrentSegment() {
+   return this.currentSegment;
+   }
+
+   public int getCurrentPositionInSegment() {
+   return this.positionInSegment;
+   }
+   
+   public int getSegmentSize() {
+   return this.segmentSize;
+   }
+   
+   protected void advance() throws IOException {
+   this.currentSegment = nextSegment(this.currentSegment, 
this.positionInSegment);
+   this.positionInSegment = 0;
+   }
+   
+   protected void seekOutput(PureOffHeapMemorySegment seg, int position) {
+   this.currentSegment = seg;
+   this.positionInSegment = position;
+   }
+
+   protected void clear() {
+  

[2/8] flink git commit: [FLINK-2890] Port StringSerializationSpeedBenchmark to JMH. [FLINK-2889] Port LongSerializationSpeedBenchmark to JMH.

[fhueske]

http://git-wip-us.apache.org/repos/asf/flink/blob/75a52574/flink-core/src/test/java/org/apache/flink/core/memory/benchmarks/PureHeapMemorySegment.java
--
diff --git 
a/flink-core/src/test/java/org/apache/flink/core/memory/benchmarks/PureHeapMemorySegment.java
 
b/flink-core/src/test/java/org/apache/flink/core/memory/benchmarks/PureHeapMemorySegment.java
deleted file mode 100644
index e247eed..000
--- 
a/flink-core/src/test/java/org/apache/flink/core/memory/benchmarks/PureHeapMemorySegment.java
+++ /dev/null
@@ -1,466 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.core.memory.benchmarks;
-
-import org.apache.flink.core.memory.MemoryUtils;
-
-import java.io.DataInput;
-import java.io.DataOutput;
-import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.nio.ByteOrder;
-
-public final class PureHeapMemorySegment {
-
-   /** Constant that flags the byte order. Because this is a boolean 
constant,
-* the JIT compiler can use this well to aggressively eliminate the 
non-applicable code paths */
-   private static final boolean LITTLE_ENDIAN = (ByteOrder.nativeOrder() 
== ByteOrder.LITTLE_ENDIAN);
-
-   /** The array in which the data is stored. */
-   private byte[] memory;
-
-   /** Wrapper for I/O requests. */
-   private ByteBuffer wrapper;
-
-   /** The size, stored extra, because we may clear the reference to the 
byte array */
-   private final int size;
-
-   // 
-
-   // Constructors
-   // 
-
-
-   /**
-* Creates a new memory segment that represents the data in the given 
byte array.
-*
-* @param memory The byte array that holds the data.
-*/
-   public PureHeapMemorySegment(byte[] memory) {
-   this.memory = memory;
-   this.size = memory.length;
-   }
-
-   // 
-
-   //  Direct Memory Segment Specifics
-   // 
-
-
-   /**
-* Gets the byte array that backs this memory segment.
-*
-* @return The byte array that backs this memory segment.
-*/
-   public byte[] getArray() {
-   return this.memory;
-   }
-
-   // 
-
-   //MemorySegment Accessors
-   // 
-
-   
-   public final boolean isFreed() {
-   return this.memory == null;
-   }
-   
-   public final void free() {
-   this.wrapper = null;
-   this.memory = null;
-   }
-   
-   public final int size() {
-   return this.size;
-   }
-   
-   public final ByteBuffer wrap(int offset, int length) {
-   if (offset > this.memory.length || offset > this.memory.length 
- length) {
-   throw new IndexOutOfBoundsException();
-   }
-
-   if (this.wrapper == null) {
-   this.wrapper = ByteBuffer.wrap(this.memory, offset, 
length);
-   }
-   else {
-   this.wrapper.limit(offset + length);
-   this.wrapper.position(offset);
-   }
-
-   return this.wrapper;
-   }
-   
-   // 

-   //Random Access get() and put() methods
-   // 

-   
-   public final byte get(int index) {
-   return this.memory[index];
-   }
-   
-   public final void put(int index, byte b) {
-   this.memory[index] = b;
-   }
-   
-   public 

[6/8] flink git commit: [FLINK-2853] Port MutableHashTablePerformanceBenchmark to JMH.

[fhueske]

[FLINK-2853] Port MutableHashTablePerformanceBenchmark to JMH.

This closes #1267


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/e5ee55bd
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/e5ee55bd
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/e5ee55bd

Branch: refs/heads/master
Commit: e5ee55bdaf0dcd6e3dc38b52964a8570b22dd671
Parents: 75a5257
Author: gallenvara 
Authored: Fri Oct 16 16:48:30 2015 +0800
Committer: Fabian Hueske 
Committed: Mon Oct 26 20:05:03 2015 +0100

--
 flink-benchmark/pom.xml |  11 +-
 .../MutableHashTablePerformanceBenchmark.java   | 359 +++
 .../MutableHashTablePerformanceBenchmark.java   | 262 --
 3 files changed, 368 insertions(+), 264 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/e5ee55bd/flink-benchmark/pom.xml
--
diff --git a/flink-benchmark/pom.xml b/flink-benchmark/pom.xml
index d6ba1d7..b22e0d1 100644
--- a/flink-benchmark/pom.xml
+++ b/flink-benchmark/pom.xml
@@ -56,14 +56,21 @@ under the License.
   jmh-generator-annprocess
   ${jmh.version}
   provided
+  
+
+  org.apache.flink
+  flink-runtime
+  0.10-SNAPSHOT
+  test-jar
+  test
 
 
   org.apache.flink
-  flink-core
+  flink-java
   ${project.version}
   test-jar
   test
-  
+
   
 
   

http://git-wip-us.apache.org/repos/asf/flink/blob/e5ee55bd/flink-benchmark/src/test/java/org/apache/flink/benchmark/runtime/operators/hash/MutableHashTablePerformanceBenchmark.java
--
diff --git 
a/flink-benchmark/src/test/java/org/apache/flink/benchmark/runtime/operators/hash/MutableHashTablePerformanceBenchmark.java
 
b/flink-benchmark/src/test/java/org/apache/flink/benchmark/runtime/operators/hash/MutableHashTablePerformanceBenchmark.java
new file mode 100644
index 000..186c595
--- /dev/null
+++ 
b/flink-benchmark/src/test/java/org/apache/flink/benchmark/runtime/operators/hash/MutableHashTablePerformanceBenchmark.java
@@ -0,0 +1,359 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.benchmark.runtime.operators.hash;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypePairComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.io.disk.iomanager.IOManager;
+import org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync;
+import org.apache.flink.runtime.jobgraph.tasks.AbstractInvokable;
+import org.apache.flink.runtime.memory.MemoryAllocationException;
+import org.apache.flink.runtime.memory.MemoryManager;
+import org.apache.flink.runtime.operators.hash.MutableHashTable;
+import org.apache.flink.runtime.operators.testutils.DummyInvokable;
+import org.apache.flink.runtime.operators.testutils.types.StringPair;
+import org.apache.flink.runtime.operators.testutils.types.StringPairComparator;
+import 
org.apache.flink.runtime.operators.testutils.types.StringPairPairComparator;
+import org.apache.flink.runtime.operators.testutils.types.StringPairSerializer;
+import org.apache.flink.util.MutableObjectIterator;
+
+import org.junit.Assert;
+import org.openjdk.jmh.annotations.*;
+import org.openjdk.jmh.runner.Runner;
+import org.openjdk.jmh.runner.options.Options;
+import org.openjdk.jmh.runner.options.OptionsBuilder;
+
+import static org.junit.Assert.fail;
+
+@State(Scope.Thread)
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MILLISECONDS)
+public class MutableHashTablePerformanceBenchmark {
+   
+   private static final AbstractInvokable MEM_OWNER = new DummyInvokable();
+   
+   private MemoryManager memManager;
+   

[8/8] flink git commit: [FLINK-2919] Port FieldAccessMinibenchmark to JMH.

[fhueske]

[FLINK-2919] Port FieldAccessMinibenchmark to JMH.

This closes #1300


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/7265d81f
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/7265d81f
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/7265d81f

Branch: refs/heads/master
Commit: 7265d81ff95aff4ddfbcbd4ef25869ea8f159769
Parents: e5ee55b
Author: gallenvara 
Authored: Mon Oct 26 15:15:14 2015 +0800
Committer: Fabian Hueske 
Committed: Mon Oct 26 20:05:03 2015 +0100

--
 .../runtime/FieldAccessMinibenchmark.java   | 170 +++
 .../runtime/FieldAccessMinibenchmark.java   | 150 
 2 files changed, 170 insertions(+), 150 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/7265d81f/flink-benchmark/src/test/java/org/apache/flink/benchmark/api/java/typeutils/runtime/FieldAccessMinibenchmark.java
--
diff --git 
a/flink-benchmark/src/test/java/org/apache/flink/benchmark/api/java/typeutils/runtime/FieldAccessMinibenchmark.java
 
b/flink-benchmark/src/test/java/org/apache/flink/benchmark/api/java/typeutils/runtime/FieldAccessMinibenchmark.java
new file mode 100644
index 000..0a434e9
--- /dev/null
+++ 
b/flink-benchmark/src/test/java/org/apache/flink/benchmark/api/java/typeutils/runtime/FieldAccessMinibenchmark.java
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.benchmark.api.java.typeutils.runtime;
+
+import org.openjdk.jmh.annotations.*;
+import org.openjdk.jmh.results.RunResult;
+import org.openjdk.jmh.runner.Runner;
+import org.openjdk.jmh.runner.RunnerException;
+import org.openjdk.jmh.runner.options.Options;
+import org.openjdk.jmh.runner.options.OptionsBuilder;
+
+import java.lang.management.ManagementFactory;
+import java.lang.management.RuntimeMXBean;
+import java.lang.reflect.Field;
+import java.util.Collection;
+import java.util.concurrent.TimeUnit;
+
+@State(Scope.Thread)
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MILLISECONDS)
+public class FieldAccessMinibenchmark {
+
+   private final long RUNS = 10L;
+   static Field wordDescField;
+   static Field wordField;
+   static {
+   try {
+   wordDescField = WC.class.getField("wordDesc");
+   wordField = 
ComplexWordDescriptor.class.getField("word");
+   } catch (Exception e) {
+   e.printStackTrace();
+   }
+   }
+
+   @Setup
+   public void warmUp() throws NoSuchFieldException,IllegalAccessException{
+   WC word0 = new WC(14, "Hallo");
+   WC word1 = new WC(3, "Hola");
+   for (long i = 0; i < 1; i++) {
+   compareCodeGenPublicFields(word0, word1);
+   compareCodeGenMethods(word0, word1);
+   compareReflective(word0, word1);
+   }
+   }
+
+   public static class ComplexWordDescriptor {
+   public String word;
+
+   public String getWord() {
+   return word;
+   }
+   }
+
+   public static class WC {
+   public int count;
+   public ComplexWordDescriptor wordDesc;
+
+   public WC(int c, String s) throws NoSuchFieldException,
+   SecurityException {
+   this.count = c;
+   this.wordDesc = new ComplexWordDescriptor();
+   this.wordDesc.word = s;
+   }
+
+   public ComplexWordDescriptor getWordDesc() {
+   return wordDesc;
+   }
+
+   }
+
+   public static int compareCodeGenPublicFields(WC w1, WC w2) {
+   return w1.wordDesc.word.compareTo(w2.wordDesc.word);
+   }
+
+   public static int compareCodeGenMethods(WC w1, WC w2) {
+   

[01/10] flink git commit: [FLINK-2800] [kryo] Fix Kryo serialization to clear buffered data

[fhueske]

Repository: flink
Updated Branches:
  refs/heads/release-0.10 51ac46e80 -> e1f30b046


[FLINK-2800] [kryo] Fix Kryo serialization to clear buffered data

The Kryo serializer uses Kryo's Output class to buffer individual write 
operations before
it is written to the underlying output stream. This Output class is flushed by 
Flink's
KryoSerializer upon finishing its serialize call. However, in case of an 
exception when
flushing the Output, the buffered data is kept in the buffer. Since Flink uses 
EOFExceptions
to mark that an underlying buffer is full and has to be spilled, for example, 
it can happen
that the record triggering the spilling is written twice after it is rewritten. 
The reason
is that Kryo's Output buffer still contains the serialization data of the 
failed attempt which
is also flushed to the emptied output stream.

This duplication of records can lead to corrupted data which eventually let's 
the Flink program
crash. The problem is solved by clearing Kryo's Output when the flush operation 
was not successful.

This closes #1308


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/44b03f20
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/44b03f20
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/44b03f20

Branch: refs/heads/release-0.10
Commit: 44b03f20a3cc96a447051e9ab0f2cf04f9829966
Parents: cdc0dfd
Author: Till Rohrmann 
Authored: Wed Oct 28 18:40:41 2015 +0100
Committer: Fabian Hueske 
Committed: Thu Oct 29 13:25:07 2015 +0100

--
 .../typeutils/runtime/kryo/KryoSerializer.java  |  17 +-
 .../runtime/kryo/KryoClearedBufferTest.java | 283 +++
 2 files changed, 297 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/flink/blob/44b03f20/flink-java/src/main/java/org/apache/flink/api/java/typeutils/runtime/kryo/KryoSerializer.java
--
diff --git 
a/flink-java/src/main/java/org/apache/flink/api/java/typeutils/runtime/kryo/KryoSerializer.java
 
b/flink-java/src/main/java/org/apache/flink/api/java/typeutils/runtime/kryo/KryoSerializer.java
index f825fc6..8549e26 100644
--- 
a/flink-java/src/main/java/org/apache/flink/api/java/typeutils/runtime/kryo/KryoSerializer.java
+++ 
b/flink-java/src/main/java/org/apache/flink/api/java/typeutils/runtime/kryo/KryoSerializer.java
@@ -58,7 +58,7 @@ import java.util.Objects;
  * @param  The type to be serialized.
  */
 public class KryoSerializer extends TypeSerializer {
-   
+
private static final long serialVersionUID = 3L;
 
// 

@@ -82,7 +82,7 @@ public class KryoSerializer extends TypeSerializer {

private transient Input input;
private transient Output output;
-   
+
// 

 
public KryoSerializer(Class type, ExecutionConfig executionConfig){
@@ -182,11 +182,22 @@ public class KryoSerializer extends TypeSerializer {
previousOut = target;
}
 
+   // Sanity check: Make sure that the output is cleared/has been 
flushed by the last call
+   // otherwise data might be written multiple times in case of a 
previous EOFException
+   if (output.position() != 0) {
+   throw new IllegalStateException("The Kryo Output still 
contains data from a previous " +
+   "serialize call. It has to be flushed or 
cleared at the end of the serialize call.");
+   }
+
try {
kryo.writeClassAndObject(output, record);
output.flush();
}
catch (KryoException ke) {
+   // make sure that the Kryo output buffer is cleared in 
case that we can recover from
+   // the exception (e.g. EOFException which denotes 
buffer full)
+   output.clear();
+
Throwable cause = ke.getCause();
if (cause instanceof EOFException) {
throw (EOFException) cause;
@@ -212,7 +223,7 @@ public class KryoSerializer extends TypeSerializer {
} catch (KryoException ke) {
Throwable cause = ke.getCause();
 
-   if(cause instanceof EOFException) {
+   if (cause instanceof EOFException) {
throw (EOFException) cause;
} else {
throw ke;

[10/10] flink git commit: [FLINK-2559] Clean up JavaDocs

[fhueske]

[FLINK-2559] Clean up JavaDocs

- Remove broken HTML tags like , , ...
- close unclosed HTML tags
- replaces special chars by HTML escaping, e.g., '<' by 
- wrap code examples by {@code}
- fix incorrect @see and @link references
- fix incorrect @throws
- fix typos

This closes #1298


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/e1f30b04
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/e1f30b04
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/e1f30b04

Branch: refs/heads/release-0.10
Commit: e1f30b046166330fd42d486c09a937d30d048ba0
Parents: 44b03f2
Author: Hubert Czerpak 
Authored: Thu Oct 22 22:51:25 2015 +0100
Committer: Fabian Hueske 
Committed: Thu Oct 29 13:25:27 2015 +0100

--
 .../org/apache/flink/client/CliFrontend.java|   3 +-
 .../storm/excamation/ExclamationLocal.java  |  13 +-
 .../storm/excamation/ExclamationTopology.java   |  13 +-
 .../storm/excamation/ExclamationWithBolt.java   |  13 +-
 .../storm/excamation/ExclamationWithSpout.java  |  13 +-
 .../flink/storm/split/SpoutSplitExample.java|   4 +-
 .../storm/wordcount/BoltTokenizerWordCount.java |   9 +-
 .../wordcount/BoltTokenizerWordCountPojo.java   |   9 +-
 .../BoltTokenizerWordCountWithNames.java|   9 +-
 .../storm/wordcount/SpoutSourceWordCount.java   |  11 +-
 .../flink/storm/wordcount/WordCountLocal.java   |  11 +-
 .../storm/wordcount/WordCountLocalByName.java   |  11 +-
 .../wordcount/WordCountRemoteByClient.java  |  11 +-
 .../wordcount/WordCountRemoteBySubmitter.java   |  11 +-
 .../storm/wordcount/WordCountTopology.java  |   9 +-
 .../org/apache/flink/storm/api/FlinkClient.java |   2 +-
 .../flink/storm/api/FlinkTopologyBuilder.java   |   4 +-
 .../flink/storm/util/SplitStreamType.java   |   3 +-
 .../flink/storm/wrappers/BoltWrapper.java   |   4 +-
 .../flink/storm/wrappers/SpoutWrapper.java  |  10 +-
 .../model/tweet/entities/Entities.java  |   2 +-
 .../model/tweet/entities/HashTags.java  |   2 +-
 .../model/tweet/entities/Media.java |   2 +-
 .../model/tweet/entities/Symbol.java|   2 +-
 .../model/tweet/entities/URL.java   |   2 +-
 .../model/tweet/entities/UserMention.java   |   2 +-
 .../flink/api/common/ExecutionConfig.java   |   2 +-
 .../apache/flink/api/common/ExecutionMode.java  |   8 +-
 .../api/common/accumulators/Histogram.java  |   2 +-
 .../api/common/aggregators/Aggregator.java  |   2 +-
 .../common/distributions/DataDistribution.java  |   2 +-
 .../api/common/functions/CoGroupFunction.java   |   4 +-
 .../api/common/functions/CrossFunction.java |   4 +-
 .../api/common/functions/FilterFunction.java|   4 +-
 .../api/common/functions/FlatJoinFunction.java  |   4 +-
 .../api/common/functions/FlatMapFunction.java   |   4 +-
 .../api/common/functions/FoldFunction.java  |   4 +-
 .../common/functions/GroupReduceFunction.java   |   4 +-
 .../api/common/functions/JoinFunction.java  |   4 +-
 .../flink/api/common/functions/MapFunction.java |   4 +-
 .../common/functions/MapPartitionFunction.java  |   4 +-
 .../api/common/functions/ReduceFunction.java|   4 +-
 .../functions/RichGroupReduceFunction.java  |   2 +-
 .../flink/api/common/io/FileOutputFormat.java   |   2 +-
 .../common/operators/AbstractUdfOperator.java   |   2 +-
 .../api/common/typeutils/TypeComparator.java|   8 +-
 .../flink/configuration/ConfigConstants.java|   2 +-
 .../org/apache/flink/core/fs/FileSystem.java|  18 ++-
 .../apache/flink/core/memory/MemorySegment.java |   4 +-
 .../java/org/apache/flink/types/Record.java |   2 +-
 .../org/apache/flink/types/StringValue.java |   4 +-
 .../java/org/apache/flink/util/Visitable.java   |   4 +-
 .../flink/examples/java/clustering/KMeans.java  |   4 +-
 .../clustering/util/KMeansDataGenerator.java|   2 +
 .../flink/examples/java/distcp/DistCp.java  |   2 +-
 .../examples/java/graph/PageRankBasic.java  |   2 +-
 .../examples/java/ml/LinearRegression.java  |   4 +-
 .../examples/java/relational/TPCHQuery10.java   |   4 +-
 .../examples/java/relational/TPCHQuery3.java|   4 +-
 .../java/relational/WebLogAnalysis.java |   8 +-
 .../examples/java/wordcount/PojoExample.java|   2 +-
 .../examples/java/wordcount/WordCount.java  |   2 +-
 .../examples/java/wordcount/WordCountMeta.java  |   4 +-
 .../java/org/apache/flink/api/java/DataSet.java | 124 +--
 .../java/org/apache/flink/api/java/Utils.java   |   1 -
 .../api/java/functions/FunctionAnnotation.java  |  30 ++---
 .../java/hadoop/mapred/HadoopInputFormat.java   |   2 +-
 .../java/hadoop/mapred/HadoopOutputFormat.java  |   2 +-
 .../flink/api/java/io/SplitDataProperties.java  |  16 +--
 .../api/java/operators/CoGroupOperator.java |  42 +++
 

[04/10] flink git commit: [FLINK-2934] Remove placeholder pages for job.statistics, taskmanager.log and taskmanager.stdout

[fhueske]

http://git-wip-us.apache.org/repos/asf/flink/blob/6a13b9ff/flink-runtime-web/web-dashboard/web/js/index.js
--
diff --git a/flink-runtime-web/web-dashboard/web/js/index.js 
b/flink-runtime-web/web-dashboard/web/js/index.js
index d370765..2554a3b 100644
--- a/flink-runtime-web/web-dashboard/web/js/index.js
+++ b/flink-runtime-web/web-dashboard/web/js/index.js
@@ -155,13 +155,6 @@ angular.module('flinkApp', ['ui.router', 
'angularMoment']).run(["$rootScope", fu
 controller: 'JobTimelineVertexController'
   }
 }
-  }).state("single-job.statistics", {
-url: "/statistics",
-views: {
-  details: {
-templateUrl: "partials/jobs/job.statistics.html"
-  }
-}
   }).state("single-job.exceptions", {
 url: "/exceptions",
 views: {
@@ -208,20 +201,6 @@ angular.module('flinkApp', ['ui.router', 
'angularMoment']).run(["$rootScope", fu
 templateUrl: "partials/taskmanager/taskmanager.metrics.html"
   }
 }
-  }).state("single-manager.log", {
-url: "/logfile",
-views: {
-  details: {
-templateUrl: "partials/taskmanager/taskmanager.logfile.html"
-  }
-}
-  }).state("single-manager.stdout", {
-url: "/stdout",
-views: {
-  details: {
-templateUrl: "partials/taskmanager/taskmanager.stdout.html"
-  }
-}
   }).state("jobmanager", {
 url: "/jobmanager",
 views: {
@@ -1417,4 +1396,4 @@ angular.module('flinkApp').service('TaskManagersService', 
["$http", "flinkConfig
   return this;
 }]);
 
-//# 
sourceMappingURL=data:application/json;base64,eyJ2ZXJzaW9uIjozLCJzb3VyY2VzIjpbImluZGV4LmNvZmZlZSIsImluZGV4LmpzIiwiY29tbW9uL2RpcmVjdGl2ZXMuY29mZmVlIiwiY29tbW9uL2RpcmVjdGl2ZXMuanMiLCJjb21tb24vZmlsdGVycy5jb2ZmZWUiLCJjb21tb24vZmlsdGVycy5qcyIsImNvbW1vbi9zZXJ2aWNlcy5jb2ZmZWUiLCJjb21tb24vc2VydmljZXMuanMiLCJtb2R1bGVzL2pvYm1hbmFnZXIvam9ibWFuYWdlci5jdHJsLmNvZmZlZSIsIm1vZHVsZXMvam9ibWFuYWdlci9qb2JtYW5hZ2VyLmN0cmwuanMiLCJtb2R1bGVzL2pvYm1hbmFnZXIvam9ibWFuYWdlci5zdmMuY29mZmVlIiwibW9kdWxlcy9qb2JtYW5hZ2VyL2pvYm1hbmFnZXIuc3ZjLmpzIiwibW9kdWxlcy9qb2JzL2pvYnMuY3RybC5jb2ZmZWUiLCJtb2R1bGVzL2pvYnMvam9icy5jdHJsLmpzIiwibW9kdWxlcy9qb2JzL2pvYnMuZGlyLmNvZmZlZSIsIm1vZHVsZXMvam9icy9qb2JzLmRpci5qcyIsIm1vZHVsZXMvam9icy9qb2JzLnN2Yy5jb2ZmZWUiLCJtb2R1bGVzL2pvYnMvam9icy5zdmMuanMiLCJtb2R1bGVzL292ZXJ2aWV3L292ZXJ2aWV3LmN0cmwuY29mZmVlIiwibW9kdWxlcy9vdmVydmlldy9vdmVydmlldy5jdHJsLmpzIiwibW9kdWxlcy9vdmVydmlldy9vdmVydmlldy5zdmMuY29mZmVlIiwibW9kdWxlcy9vdmVydmlldy9vdmVydmlldy5zdmMuanMiLCJtb2R1bGVzL3Rhc2ttYW5hZ2VyL3Rhc2ttYW5
 
hZ2VyLmN0cmwuY29mZmVlIiwibW9kdWxlcy90YXNrbWFuYWdlci90YXNrbWFuYWdlci5jdHJsLmpzIiwibW9kdWxlcy90YXNrbWFuYWdlci90YXNrbWFuYWdlci5kaXIuY29mZmVlIiwibW9kdWxlcy90YXNrbWFuYWdlci90YXNrbWFuYWdlci5kaXIuanMiLCJtb2R1bGVzL3Rhc2ttYW5hZ2VyL3Rhc2ttYW5hZ2VyLnN2Yy5jb2ZmZWUiLCJtb2R1bGVzL3Rhc2ttYW5hZ2VyL3Rhc2ttYW5hZ2VyLnN2Yy5qcyJdLCJuYW1lcyI6W10sIm1hcHBpbmdzIjoiQUFrQkEsUUFBUSxPQUFPLFlBQVksQ0FBQyxhQUFhLGtCQUl4QyxtQkFBSSxTQUFDLFlBQUQ7RUFDSCxXQUFXLGlCQUFpQjtFQ3JCNUIsT0RzQkEsV0FBVyxjQUFjLFdBQUE7SUFDdkIsV0FBVyxpQkFBaUIsQ0FBQyxXQUFXO0lDckJ4QyxPRHNCQSxXQUFXLGVBQWU7O0lBSTdCLE1BQU0sZUFBZTtFQUNwQixvQkFBb0I7R0FLckIsK0RBQUksU0FBQyxhQUFhLGFBQWEsYUFBYSxXQUF4QztFQUNILFlBQVksYUFBYSxLQUFLLFNBQUMsUUFBRDtJQUM1QixRQUFRLE9BQU8sYUFBYTtJQUU1QixZQUFZO0lDNUJaLE9EOEJBLFVBQVUsV0FBQTtNQzdCUixPRDhCQSxZQUFZO09BQ1osWUFBWTs7RUFFaEIsV0FBVyxXQUFXO0lBQ3BCLFFBQVE7TUFDTixRQUFROzs7RUFZWixXQUFXLGNBQWMsUUFBUTtJQUNoQyxNQUFNO0lBQ04sS0FBSztJQUNMLE1BQU07S0FDSixNQUFNLFNBQVMscUJBQXFCLFFBQVE7RUFFL0MsV0FBVyxRQUFRO0lBQ2xCLFFBQVEsQ0FBQyxXQUFXLFdBQVcsV0FB
 
VyxXQUFXLFdBQVcsV0FBVyxXQUMxRSxXQUFXLFdBQVcsV0FBVztJQUNsQyxPQUFPO01BQ04saUJBQWlCO01BQ2pCLE9BQU87UUFDTixZQUFZOzs7SUFHZCxPQUFPO01BQ04sT0FBTztRQUNOLFVBQVU7UUFDVixZQUFZO1FBQ1osZUFBZTs7O0lBR2pCLFNBQVM7TUFDUixhQUFhO01BQ2IsaUJBQWlCO01BQ2pCLFFBQVE7O0lBRVQsUUFBUTtNQUNQLFdBQVc7UUFDVixZQUFZO1FBQ1osVUFBVTs7O0lBR1osT0FBTztNQUNOLGVBQWU7TUFDZixRQUFRO1FBQ1AsT0FBTztVQUNOLFVBQVU7Ozs7SUFJYixPQUFPO01BQ04sbUJBQW1CO01BQ25CLE9BQU87UUFDTixPQUFPO1VBQ04sZUFBZTs7O01BR2pCLFFBQVE7UUFDUCxPQUFPO1VBQ04sVUFBVTs7OztJQUliLGFBQWE7TUFDWixhQUFhO1FBQ1osV0FBVzs7O0lBSWIsYUFBYTs7RUN4Q2QsT0Q0Q0EsV0FBVyxXQUFXLFdBQVc7SUFLbEMsaUNBQU8sU0FBQyx1QkFBRDtFQy9DTixPRGdEQSxzQkFBc0I7SUFJdkIsZ0RBQU8sU0FBQyxnQkFBZ0Isb0JBQWpCO0VBQ04sZUFBZSxNQUFNLFlBQ25CO0lBQUEsS0FBSztJQUNMLE9BQ0U7TUFBQSxNQUNFO1FBQUEsYUFBYTtRQUNiLFlBQVk7OztLQUVqQixNQUFNLGdCQUNMO0lBQUEsS0FBSztJQUNMLE9BQ0U7TUFBQSxNQUNFO1FBQUEsYUFBYTtRQUNiLFlBQVk7OztLQUVqQixNQUFNLGtCQUNMO0lBQUEsS0FBSztJQUNMLE9BQ0U7TUFBQSxNQUNFO1FBQUEsYUFBYTtRQUNiLFlBQVk7OztLQUVqQixNQUFNLGNBQ0w7SUFBQSxLQUFLO0lBQ
 

[09/10] flink git commit: [FLINK-2559] Clean up JavaDocs

[fhueske]

http://git-wip-us.apache.org/repos/asf/flink/blob/e1f30b04/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/relational/WebLogAnalysis.java
--
diff --git 
a/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/relational/WebLogAnalysis.java
 
b/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/relational/WebLogAnalysis.java
index 9425291..79944ae 100644
--- 
a/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/relational/WebLogAnalysis.java
+++ 
b/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/relational/WebLogAnalysis.java
@@ -35,7 +35,7 @@ import 
org.apache.flink.examples.java.relational.util.WebLogDataGenerator;
  * This program processes web logs and relational data. 
  * It implements the following relational query:
  *
- * 
+ * {@code
  * SELECT 
  *   r.pageURL, 
  *   r.pageRank, 
@@ -50,13 +50,13 @@ import 
org.apache.flink.examples.java.relational.util.WebLogDataGenerator;
  *  WHERE v.destUrl = d.url 
  *AND v.visitDate < [date]
  *   );
- * 
+ * }
  *
  * 
  * Input files are plain text CSV files using the pipe character ('|') as 
field separator.
  * The tables referenced in the query can be generated using the {@link 
WebLogDataGenerator} and 
  * have the following schemas
- * 
+ * {@code
  * CREATE TABLE Documents (
  *url VARCHAR(100) PRIMARY KEY,
  *contents TEXT );
@@ -76,7 +76,7 @@ import 
org.apache.flink.examples.java.relational.util.WebLogDataGenerator;
  *languageCode VARCHAR(6),
  *searchWord VARCHAR(32),
  *duration INT );
- * 
+ * }
  * 
  * 
  * Usage: WebLogAnalysis documents path ranks path 
visits path result path

http://git-wip-us.apache.org/repos/asf/flink/blob/e1f30b04/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/wordcount/PojoExample.java
--
diff --git 
a/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/wordcount/PojoExample.java
 
b/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/wordcount/PojoExample.java
index b001d12..0a8595f 100644
--- 
a/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/wordcount/PojoExample.java
+++ 
b/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/wordcount/PojoExample.java
@@ -112,7 +112,7 @@ public class PojoExample {
/**
 * Implements the string tokenizer that splits sentences into words as 
a user-defined
 * FlatMapFunction. The function takes a line (String) and splits it 
into 
-* multiple pairs in the form of "(word,1)" (Tuple2).
+* multiple pairs in the form of "(word,1)" ({@code Tuple2}).
 */
public static final class Tokenizer implements FlatMapFunction {
private static final long serialVersionUID = 1L;

http://git-wip-us.apache.org/repos/asf/flink/blob/e1f30b04/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/wordcount/WordCount.java
--
diff --git 
a/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/wordcount/WordCount.java
 
b/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/wordcount/WordCount.java
index e6b8418..c677160 100644
--- 
a/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/wordcount/WordCount.java
+++ 
b/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/wordcount/WordCount.java
@@ -90,7 +90,7 @@ public class WordCount {
/**
 * Implements the string tokenizer that splits sentences into words as 
a user-defined
 * FlatMapFunction. The function takes a line (String) and splits it 
into 
-* multiple pairs in the form of "(word,1)" (Tuple2).
+* multiple pairs in the form of "(word,1)" ({@code Tuple2}).
 */
public static final class Tokenizer implements FlatMapFunction> {
 

http://git-wip-us.apache.org/repos/asf/flink/blob/e1f30b04/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/wordcount/WordCountMeta.java
--
diff --git 
a/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/wordcount/WordCountMeta.java
 
b/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/wordcount/WordCountMeta.java
index b8bd7c1..200fd0f 100644
--- 

[03/10] flink git commit: [FLINK-2934] Remove placeholder pages for job.statistics, taskmanager.log and taskmanager.stdout

[fhueske]

http://git-wip-us.apache.org/repos/asf/flink/blob/6a13b9ff/flink-runtime-web/web-dashboard/web/partials/jobs/job.html
--
diff --git a/flink-runtime-web/web-dashboard/web/partials/jobs/job.html 
b/flink-runtime-web/web-dashboard/web/partials/jobs/job.html
index 9d3e171..4134274 100644
--- a/flink-runtime-web/web-dashboard/web/partials/jobs/job.html
+++ b/flink-runtime-web/web-dashboard/web/partials/jobs/job.html
@@ -36,7 +36,6 @@ limitations under the License.
 
   
 Plan
-Job Accumulators / 
Statistics
 Timeline
 Exceptions
 Properties

http://git-wip-us.apache.org/repos/asf/flink/blob/6a13b9ff/flink-runtime-web/web-dashboard/web/partials/taskmanager/taskmanager.html
--
diff --git 
a/flink-runtime-web/web-dashboard/web/partials/taskmanager/taskmanager.html 
b/flink-runtime-web/web-dashboard/web/partials/taskmanager/taskmanager.html
index 17cb834..c89970e 100644
--- a/flink-runtime-web/web-dashboard/web/partials/taskmanager/taskmanager.html
+++ b/flink-runtime-web/web-dashboard/web/partials/taskmanager/taskmanager.html
@@ -26,8 +26,6 @@ limitations under the License.
 
   
 Metrics
-Logs
-Stdout
   
 
 

[07/10] flink git commit: [FLINK-2559] Clean up JavaDocs

[fhueske]

http://git-wip-us.apache.org/repos/asf/flink/blob/e1f30b04/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/SingleOutputStreamOperator.java
--
diff --git 
a/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/SingleOutputStreamOperator.java
 
b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/SingleOutputStreamOperator.java
index 33d5a3c..3722908 100644
--- 
a/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/SingleOutputStreamOperator.java
+++ 
b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/SingleOutputStreamOperator.java
@@ -133,7 +133,8 @@ public class SingleOutputStreamOperator

[09/13] flink git commit: [FLINK-2559] Clean up JavaDocs

[fhueske]

[FLINK-2559] Clean up JavaDocs

- Remove broken HTML tags like , , ...
- close unclosed HTML tags
- replaces special chars by HTML escaping, e.g., '<' by 
- wrap code examples by {@code}
- fix incorrect @see and @link references
- fix incorrect @throws
- fix typos

This closes #1298


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/680b5a90
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/680b5a90
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/680b5a90

Branch: refs/heads/master
Commit: 680b5a90de2428b4b6b986f7b40409c2d1d181a7
Parents: ec7bf50
Author: Hubert Czerpak 
Authored: Thu Oct 22 22:51:25 2015 +0100
Committer: Fabian Hueske 
Committed: Thu Oct 29 10:52:22 2015 +0100

--
 .../org/apache/flink/client/CliFrontend.java|   3 +-
 .../storm/excamation/ExclamationLocal.java  |  13 +-
 .../storm/excamation/ExclamationTopology.java   |  13 +-
 .../storm/excamation/ExclamationWithBolt.java   |  13 +-
 .../storm/excamation/ExclamationWithSpout.java  |  13 +-
 .../flink/storm/split/SpoutSplitExample.java|   4 +-
 .../storm/wordcount/BoltTokenizerWordCount.java |   9 +-
 .../wordcount/BoltTokenizerWordCountPojo.java   |   9 +-
 .../BoltTokenizerWordCountWithNames.java|   9 +-
 .../storm/wordcount/SpoutSourceWordCount.java   |  11 +-
 .../flink/storm/wordcount/WordCountLocal.java   |  11 +-
 .../storm/wordcount/WordCountLocalByName.java   |  11 +-
 .../wordcount/WordCountRemoteByClient.java  |  11 +-
 .../wordcount/WordCountRemoteBySubmitter.java   |  11 +-
 .../storm/wordcount/WordCountTopology.java  |   9 +-
 .../org/apache/flink/storm/api/FlinkClient.java |   2 +-
 .../flink/storm/api/FlinkTopologyBuilder.java   |   4 +-
 .../flink/storm/util/SplitStreamType.java   |   3 +-
 .../flink/storm/wrappers/BoltWrapper.java   |   4 +-
 .../flink/storm/wrappers/SpoutWrapper.java  |  10 +-
 .../model/tweet/entities/Entities.java  |   2 +-
 .../model/tweet/entities/HashTags.java  |   2 +-
 .../model/tweet/entities/Media.java |   2 +-
 .../model/tweet/entities/Symbol.java|   2 +-
 .../model/tweet/entities/URL.java   |   2 +-
 .../model/tweet/entities/UserMention.java   |   2 +-
 .../flink/api/common/ExecutionConfig.java   |   2 +-
 .../apache/flink/api/common/ExecutionMode.java  |   8 +-
 .../api/common/accumulators/Histogram.java  |   2 +-
 .../api/common/aggregators/Aggregator.java  |   2 +-
 .../common/distributions/DataDistribution.java  |   2 +-
 .../api/common/functions/CoGroupFunction.java   |   4 +-
 .../api/common/functions/CrossFunction.java |   4 +-
 .../api/common/functions/FilterFunction.java|   4 +-
 .../api/common/functions/FlatJoinFunction.java  |   4 +-
 .../api/common/functions/FlatMapFunction.java   |   4 +-
 .../api/common/functions/FoldFunction.java  |   4 +-
 .../common/functions/GroupReduceFunction.java   |   4 +-
 .../api/common/functions/JoinFunction.java  |   4 +-
 .../flink/api/common/functions/MapFunction.java |   4 +-
 .../common/functions/MapPartitionFunction.java  |   4 +-
 .../api/common/functions/ReduceFunction.java|   4 +-
 .../functions/RichGroupReduceFunction.java  |   2 +-
 .../flink/api/common/io/FileOutputFormat.java   |   2 +-
 .../common/operators/AbstractUdfOperator.java   |   2 +-
 .../api/common/typeutils/TypeComparator.java|   8 +-
 .../flink/configuration/ConfigConstants.java|   2 +-
 .../org/apache/flink/core/fs/FileSystem.java|  18 ++-
 .../apache/flink/core/memory/MemorySegment.java |   4 +-
 .../java/org/apache/flink/types/Record.java |   2 +-
 .../org/apache/flink/types/StringValue.java |   4 +-
 .../java/org/apache/flink/util/Visitable.java   |   4 +-
 .../flink/examples/java/clustering/KMeans.java  |   4 +-
 .../clustering/util/KMeansDataGenerator.java|   2 +
 .../flink/examples/java/distcp/DistCp.java  |   2 +-
 .../examples/java/graph/PageRankBasic.java  |   2 +-
 .../examples/java/ml/LinearRegression.java  |   4 +-
 .../examples/java/relational/TPCHQuery10.java   |   4 +-
 .../examples/java/relational/TPCHQuery3.java|   4 +-
 .../java/relational/WebLogAnalysis.java |   8 +-
 .../examples/java/wordcount/PojoExample.java|   2 +-
 .../examples/java/wordcount/WordCount.java  |   2 +-
 .../examples/java/wordcount/WordCountMeta.java  |   4 +-
 .../java/org/apache/flink/api/java/DataSet.java | 124 +--
 .../java/org/apache/flink/api/java/Utils.java   |   1 -
 .../api/java/functions/FunctionAnnotation.java  |  30 ++---
 .../java/hadoop/mapred/HadoopInputFormat.java   |   2 +-
 .../java/hadoop/mapred/HadoopOutputFormat.java  |   2 +-
 .../flink/api/java/io/SplitDataProperties.java  |  16 +--
 .../api/java/operators/CoGroupOperator.java |  42 +++