date:20150217

[jira] [Commented] (FLINK-1421) Implement a SAMOA Adapter for Flink Streaming

2015-02-17 Thread Fay Beligianni (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325088#comment-14325088
 ] 

Fay Beligianni commented on FLINK-1421:
---

Hello,

I am trying to run a SAMOA  task on a Flink cluster but a RuntimeException it 
is thrown: "org.apache.commons.lang.SerializationException: 
java.lang.ClassNotFoundException" (The whole exception message is quoted at the 
end of my comment).

Specifically,  the deployed JAR is submitted to the cluster through Flink's 
command line interface. 
We inspected the deployed JAR we verified that the class, for which  the 
"ClassNotFoundException" is thrown,  IS in the JAR...   
 
I have to mention that the same program works like a charm when we are 
executing it through the IDE.
Moreover, we managed to execute it locally, through the command line interface, 
when we added manually the deployable JAR in the Flink library folder, but that 
was just for testing purposes.

Any help would be appreciated!

-Exception Message-

org.apache.flink.client.program.ProgramInvocationException: The program 
execution failed: java.lang.RuntimeException: 
org.apache.commons.lang.SerializationException: 
java.lang.ClassNotFoundException: 
com.yahoo.labs.samoa.examples.HelloWorldContentEvent at 
org.apache.flink.streaming.api.streamvertex.OutputHandler.invokeUserFunction(OutputHandler.java:232)
 at 
org.apache.flink.streaming.api.streamvertex.StreamVertex.invoke(StreamVertex.java:121)
 at 
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
 at java.lang.Thread.run(Thread.java:745) Caused by: 
org.apache.commons.lang.SerializationException: 
java.lang.ClassNotFoundException: 
com.yahoo.labs.samoa.examples.HelloWorldContentEvent at 
org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:165)
 at 
org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:192)
 at 
com.yahoo.labs.flink.SamoaTypeSerializer.deserialize(SamoaTypeSerializer.java:84)
 at 
com.yahoo.labs.flink.SamoaTypeSerializer.deserialize(SamoaTypeSerializer.java:33)
 at 
org.apache.flink.streaming.api.streamrecord.StreamRecordSerializer.deserialize(StreamRecordSerializer.java:107)
 at 
org.apache.flink.streaming.api.streamrecord.StreamRecordSerializer.deserialize(StreamRecordSerializer.java:29)
 at 
org.apache.flink.runtime.plugable.ReusingDeserializationDelegate.read(ReusingDeserializationDelegate.java:57)
 at 
org.apache.flink.runtime.io.network.serialization.SpillingAdaptiveSpanningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRecordDeserializer.java:111)
 at 
org.apache.flink.runtime.io.network.api.reader.AbstractRecordReader.getNextRecord(AbstractRecordReader.java:66)
 at 
org.apache.flink.runtime.io.network.api.reader.MutableRecordReader.next(MutableRecordReader.java:33)
 at 
org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:59)
 at 
org.apache.flink.streaming.api.invokable.StreamInvokable.readNext(StreamInvokable.java:102)
 at 
com.yahoo.labs.flink.topology.impl.FlinkProcessingItem.invoke(FlinkProcessingItem.java:143)
 at 
org.apache.flink.streaming.api.streamvertex.StreamVertex.invokeUserFunction(StreamVertex.java:85)
 at 
org.apache.flink.streaming.api.streamvertex.OutputHandler.invokeUserFunction(OutputHandler.java:229)
 ... 3 more Caused by: java.lang.ClassNotFoundException: 
com.yahoo.labs.samoa.examples.HelloWorldContentEvent at 
java.net.URLClassLoader$1.run(URLClassLoader.java:372) at 
java.net.URLClassLoader$1.run(URLClassLoader.java:361) at 
java.security.AccessController.doPrivileged(Native Method) at 
java.net.URLClassLoader.findClass(URLClassLoader.java:360) at 
java.lang.ClassLoader.loadClass(ClassLoader.java:424) at 
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at 
java.lang.ClassLoader.loadClass(ClassLoader.java:357) at 
java.lang.Class.forName0(Native Method) at 
java.lang.Class.forName(Class.java:340) at 
java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:626) at 
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613) at 
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518) at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774) at 
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) at 
java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) at 
org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:162)
 ... 17 more at org.apache.flink.client.program.Client.run(Client.java:345) at 
org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:68)
 at 
org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:49)
 at com.yahoo.labs.flink.FlinkDoTask.main(FlinkDoTask.java:88) at 
sun.reflect.NativeMeth

[jira] [Commented] (FLINK-947) Add support for "Named Datasets"

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325060#comment-14325060
 ] 

ASF GitHub Bot commented on FLINK-947:
--

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/405#issuecomment-74773124
  
Very very nice work! I've played around a bit with it and the first 
impression is very good.

+1 to merge the pull request. The change is very big, but stable enough to 
be merged to master. Not merging it soon would probably cause a lot of work on 
@aljoscha side.



> Add support for "Named Datasets"
> 
>
> Key: FLINK-947
> URL: https://issues.apache.org/jira/browse/FLINK-947
> Project: Flink
>  Issue Type: New Feature
>  Components: Java API
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Minor
>
> This would create an API that is a mix between SQL like declarativity and the 
> power of user defined functions. Example user code could look like this:
> {code:Java}
> NamedDataSet one = ...
> NamedDataSet two = ...
> NamedDataSet result = one.join(two).where("key").equalTo("otherKey")
>   .project("a", "b", "c")
>   .map( (UserTypeIn in) -> return new UserTypeOut(...) )
>   .print();
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-947] Add a declarative expression API

2015-02-17 Thread rmetzger

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/405#issuecomment-74773124
  
Very very nice work! I've played around a bit with it and the first 
impression is very good.

+1 to merge the pull request. The change is very big, but stable enough to 
be merged to master. Not merging it soon would probably cause a lot of work on 
@aljoscha side.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-947) Add support for "Named Datasets"

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325036#comment-14325036
 ] 

ASF GitHub Bot commented on FLINK-947:
--

Github user rmetzger commented on a diff in the pull request:

https://github.com/apache/flink/pull/405#discussion_r24862039
  
--- Diff: 
flink-staging/flink-linq/src/main/scala/org/apache/flink/api/scala/expressions/package.scala
 ---
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.api.scala
+
+import com.google.common.base.Preconditions
+import org.apache.flink.api.expressions.{Row, ExpressionOperation}
+import org.apache.flink.api.common.typeutils.CompositeType
+import org.apache.flink.streaming.api.scala.DataStream
+
+import scala.language.implicitConversions
+
+/**
+ * == Language Integrated Queries (aka Expression Operations) ==
+ *
+ * Importing this package with:
+ *
+ * {{{
+ *   import org.apache.flink.api.scala.expressions._
+ * }}}
+ *
+ * imports implicit conversions for converting a [[DataSet]] or 
[[DataStream]] to an
+ * [[ExpressionOperation]]. This can be used to perform SQL-like queries 
on data. Please have
+ * a look at [[ExpressionOperation]] to see which operations are supported 
and
+ * [[org.apache.flink.api.scala.expressions.ImplicitExpressionOperations]] 
to see how an
+ * expression can be specified.
+ *
+ * Inside an expression operation you can use Scala Symbols to refer to 
field names. One would
+ * refer to field `a` by writing `'a`. Sometimes it is necessary to 
manually confert a
+ * Scala literal to an Expression Literal, in those cases use `Literal`, 
as in `Literal(3)`.
+ *
+ * Example:
+ *
+ * {{{
+ *   import org.apache.flink.api.scala._
+ *   import org.apache.flink.api.scala.expressions._
+ *
+ *   val env = ExecutionEnvironment.getExecutionEnvironment
+ *   val input = env.fromElements(("Hello", 2), ("Hello", 5), ("Ciao", 3))
+ *   val result = input.as('word, 'count).groupBy('word).select('word, 
'count.avg)
+ *   result.print()
+ *
+ *   env.execute()
+ * }}}
+ *
+ * The result of an [[ExpressionOperation]] can be converted back to the 
underlying API
+ * representation using `as`:
+ *
+ * {{{
+ *   case class Word(word: String, count: Int)
+ *
+ *   val result = in.select(...).as('word, 'count)
+ *   val set = result.as[Word]
+ * }}}
+ */
+package object expressions extends ImplicitExpressionConversions {
+
+  implicit def dataSet2DataSetConversions[T](set: DataSet[T]): 
DataSetConversions[T] = {
+Preconditions.checkArgument(set.getType.isInstanceOf[CompositeType[T]])
--- End diff --

I think we need a good error message here, telling the user why the type 
cannot be used with the expr lang.


> Add support for "Named Datasets"
> 
>
> Key: FLINK-947
> URL: https://issues.apache.org/jira/browse/FLINK-947
> Project: Flink
>  Issue Type: New Feature
>  Components: Java API
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Minor
>
> This would create an API that is a mix between SQL like declarativity and the 
> power of user defined functions. Example user code could look like this:
> {code:Java}
> NamedDataSet one = ...
> NamedDataSet two = ...
> NamedDataSet result = one.join(two).where("key").equalTo("otherKey")
>   .project("a", "b", "c")
>   .map( (UserTypeIn in) -> return new UserTypeOut(...) )
>   .print();
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-947] Add a declarative expression API

2015-02-17 Thread rmetzger

Github user rmetzger commented on a diff in the pull request:

https://github.com/apache/flink/pull/405#discussion_r24862039
  
--- Diff: 
flink-staging/flink-linq/src/main/scala/org/apache/flink/api/scala/expressions/package.scala
 ---
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.api.scala
+
+import com.google.common.base.Preconditions
+import org.apache.flink.api.expressions.{Row, ExpressionOperation}
+import org.apache.flink.api.common.typeutils.CompositeType
+import org.apache.flink.streaming.api.scala.DataStream
+
+import scala.language.implicitConversions
+
+/**
+ * == Language Integrated Queries (aka Expression Operations) ==
+ *
+ * Importing this package with:
+ *
+ * {{{
+ *   import org.apache.flink.api.scala.expressions._
+ * }}}
+ *
+ * imports implicit conversions for converting a [[DataSet]] or 
[[DataStream]] to an
+ * [[ExpressionOperation]]. This can be used to perform SQL-like queries 
on data. Please have
+ * a look at [[ExpressionOperation]] to see which operations are supported 
and
+ * [[org.apache.flink.api.scala.expressions.ImplicitExpressionOperations]] 
to see how an
+ * expression can be specified.
+ *
+ * Inside an expression operation you can use Scala Symbols to refer to 
field names. One would
+ * refer to field `a` by writing `'a`. Sometimes it is necessary to 
manually confert a
+ * Scala literal to an Expression Literal, in those cases use `Literal`, 
as in `Literal(3)`.
+ *
+ * Example:
+ *
+ * {{{
+ *   import org.apache.flink.api.scala._
+ *   import org.apache.flink.api.scala.expressions._
+ *
+ *   val env = ExecutionEnvironment.getExecutionEnvironment
+ *   val input = env.fromElements(("Hello", 2), ("Hello", 5), ("Ciao", 3))
+ *   val result = input.as('word, 'count).groupBy('word).select('word, 
'count.avg)
+ *   result.print()
+ *
+ *   env.execute()
+ * }}}
+ *
+ * The result of an [[ExpressionOperation]] can be converted back to the 
underlying API
+ * representation using `as`:
+ *
+ * {{{
+ *   case class Word(word: String, count: Int)
+ *
+ *   val result = in.select(...).as('word, 'count)
+ *   val set = result.as[Word]
+ * }}}
+ */
+package object expressions extends ImplicitExpressionConversions {
+
+  implicit def dataSet2DataSetConversions[T](set: DataSet[T]): 
DataSetConversions[T] = {
+Preconditions.checkArgument(set.getType.isInstanceOf[CompositeType[T]])
--- End diff --

I think we need a good error message here, telling the user why the type 
cannot be used with the expr lang.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (FLINK-1574) Flink fails due to non-initialized RuntimeContext in CombiningUnilateralSortMerger

2015-02-17 Thread Fabian Hueske (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-1574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fabian Hueske resolved FLINK-1574.
--
Resolution: Fixed

Fixed with 30e3893bdb80ba148abdcb981fc92d176d05e15a

> Flink fails due to non-initialized RuntimeContext in 
> CombiningUnilateralSortMerger
> --
>
> Key: FLINK-1574
> URL: https://issues.apache.org/jira/browse/FLINK-1574
> Project: Flink
>  Issue Type: Bug
>  Components: Local Runtime
>Affects Versions: 0.9
>Reporter: Fabian Hueske
>Priority: Critical
> Fix For: 0.9
>
>
> The {{AggregatingUdf}} combiner cannot be opened in a 
> {{CombiningUnilateralSortMerger }} because its {{RuntimeContext}} is {{null}}.
> The problem is that the runtime context of the {{RegularPactTask}} is set 
> after the local strategies are initialized.
> I have a fix for that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1388) POJO support for writeAsCsv

2015-02-17 Thread Adnan Khan (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325020#comment-14325020
 ] 

Adnan Khan commented on FLINK-1388:
---

Hey thanks. 

* What exactly do you mean by integration with {{DataSet.internalWriteAsCsv}}?
* When you say "TypeExtraction on every tuple" - Do you mean the specific calls 
to {{TypeExtractor.createTypeInfo()}} on every tuple?
* That makes sense. I'm going to go with Timo's suggestion and try to modify 
{{org.apache.flink.api.java.io.CsvOutputFormat}} which extends 
{{FileOutputFormat}}

> POJO support for writeAsCsv
> ---
>
> Key: FLINK-1388
> URL: https://issues.apache.org/jira/browse/FLINK-1388
> Project: Flink
>  Issue Type: New Feature
>  Components: Java API
>Reporter: Timo Walther
>Assignee: Adnan Khan
>Priority: Minor
>
> It would be great if one could simply write out POJOs in CSV format.
> {code}
> public class MyPojo {
>String a;
>int b;
> }
> {code}
> to:
> {code}
> # CSV file of org.apache.flink.MyPojo: String a, int b
> "Hello World", 42
> "Hello World 2", 47
> ...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-1388) POJO support for writeAsCsv

2015-02-17 Thread Adnan Khan (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324987#comment-14324987
 ] 

Adnan Khan edited comment on FLINK-1388 at 2/17/15 10:11 PM:
-

Oh that makes sense, I'll look at that. I don't see any tests for 
{{org.apache.flink.api.java.io.CsvOutputFormat}}. Should I add some similar to 
{{org.apache.flink.api.java.io.record.CsvOutputFormatTest}}? But for tuples and 
POJOs of course.


was (Author: khnd):
Oh that makes sense, I'll look at that. I don't see any tests for 
{{org.apache.flink.api.java.io.CsvOutputFormat}}. Should I add some similar to 
{org.apache.flink.api.java.io.record.CsvOutputFormatTest}}? But for tuples and 
POJOs of course.

> POJO support for writeAsCsv
> ---
>
> Key: FLINK-1388
> URL: https://issues.apache.org/jira/browse/FLINK-1388
> Project: Flink
>  Issue Type: New Feature
>  Components: Java API
>Reporter: Timo Walther
>Assignee: Adnan Khan
>Priority: Minor
>
> It would be great if one could simply write out POJOs in CSV format.
> {code}
> public class MyPojo {
>String a;
>int b;
> }
> {code}
> to:
> {code}
> # CSV file of org.apache.flink.MyPojo: String a, int b
> "Hello World", 42
> "Hello World 2", 47
> ...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1388) POJO support for writeAsCsv

2015-02-17 Thread Adnan Khan (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324987#comment-14324987
 ] 

Adnan Khan commented on FLINK-1388:
---

Oh that makes sense, I'll look at that. I don't see any tests for 
{{org.apache.flink.api.java.io.CsvOutputFormat}}. Should I add some similar to 
{org.apache.flink.api.java.io.record.CsvOutputFormatTest}}? But for tuples and 
POJOs of course.

> POJO support for writeAsCsv
> ---
>
> Key: FLINK-1388
> URL: https://issues.apache.org/jira/browse/FLINK-1388
> Project: Flink
>  Issue Type: New Feature
>  Components: Java API
>Reporter: Timo Walther
>Assignee: Adnan Khan
>Priority: Minor
>
> It would be great if one could simply write out POJOs in CSV format.
> {code}
> public class MyPojo {
>String a;
>int b;
> }
> {code}
> to:
> {code}
> # CSV file of org.apache.flink.MyPojo: String a, int b
> "Hello World", 42
> "Hello World 2", 47
> ...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (FLINK-1388) POJO support for writeAsCsv

2015-02-17 Thread Adnan Khan (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adnan Khan updated FLINK-1388:
--
Comment: was deleted

(was: Oh that makes sense, I'll look at that. I don't see any tests for 
{{org.apache.flink.api.java.io.CsvOutputFormat}}. Should I add some similar to 
{org.apache.flink.api.java.io.record.CsvOutputFormatTest}}? But for tuples and 
POJOs of course.)

> POJO support for writeAsCsv
> ---
>
> Key: FLINK-1388
> URL: https://issues.apache.org/jira/browse/FLINK-1388
> Project: Flink
>  Issue Type: New Feature
>  Components: Java API
>Reporter: Timo Walther
>Assignee: Adnan Khan
>Priority: Minor
>
> It would be great if one could simply write out POJOs in CSV format.
> {code}
> public class MyPojo {
>String a;
>int b;
> }
> {code}
> to:
> {code}
> # CSV file of org.apache.flink.MyPojo: String a, int b
> "Hello World", 42
> "Hello World 2", 47
> ...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1388) POJO support for writeAsCsv

2015-02-17 Thread Adnan Khan (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324989#comment-14324989
 ] 

Adnan Khan commented on FLINK-1388:
---

Oh that makes sense, I'll look at that. I don't see any tests for 
{{org.apache.flink.api.java.io.CsvOutputFormat}}. Should I add some similar to 
{org.apache.flink.api.java.io.record.CsvOutputFormatTest}}? But for tuples and 
POJOs of course.

> POJO support for writeAsCsv
> ---
>
> Key: FLINK-1388
> URL: https://issues.apache.org/jira/browse/FLINK-1388
> Project: Flink
>  Issue Type: New Feature
>  Components: Java API
>Reporter: Timo Walther
>Assignee: Adnan Khan
>Priority: Minor
>
> It would be great if one could simply write out POJOs in CSV format.
> {code}
> public class MyPojo {
>String a;
>int b;
> }
> {code}
> to:
> {code}
> # CSV file of org.apache.flink.MyPojo: String a, int b
> "Hello World", 42
> "Hello World 2", 47
> ...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-947] Add a declarative expression API

2015-02-17 Thread rmetzger

Github user rmetzger commented on a diff in the pull request:

https://github.com/apache/flink/pull/405#discussion_r24858527
  
--- Diff: docs/linq.md ---
@@ -0,0 +1,65 @@
+---
+title: "Language-Integrated Queries"
+---
+
+
+* This will be replaced by the TOC
+{:toc}
+
+**Language-Integrated Queries are an experimental feature and can 
currently only be used with
--- End diff --

Good to see some documentation as well!

Which types are supported by the expression API? Only scala case-classes? 
POJOs ? Even more? 
Would be could if you could add that to the documentation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-947) Add support for "Named Datasets"

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324968#comment-14324968
 ] 

ASF GitHub Bot commented on FLINK-947:
--

Github user rmetzger commented on a diff in the pull request:

https://github.com/apache/flink/pull/405#discussion_r24858527
  
--- Diff: docs/linq.md ---
@@ -0,0 +1,65 @@
+---
+title: "Language-Integrated Queries"
+---
+
+
+* This will be replaced by the TOC
+{:toc}
+
+**Language-Integrated Queries are an experimental feature and can 
currently only be used with
--- End diff --

Good to see some documentation as well!

Which types are supported by the expression API? Only scala case-classes? 
POJOs ? Even more? 
Would be could if you could add that to the documentation.


> Add support for "Named Datasets"
> 
>
> Key: FLINK-947
> URL: https://issues.apache.org/jira/browse/FLINK-947
> Project: Flink
>  Issue Type: New Feature
>  Components: Java API
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Minor
>
> This would create an API that is a mix between SQL like declarativity and the 
> power of user defined functions. Example user code could look like this:
> {code:Java}
> NamedDataSet one = ...
> NamedDataSet two = ...
> NamedDataSet result = one.join(two).where("key").equalTo("otherKey")
>   .project("a", "b", "c")
>   .map( (UserTypeIn in) -> return new UserTypeOut(...) )
>   .print();
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-947) Add support for "Named Datasets"

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324948#comment-14324948
 ] 

ASF GitHub Bot commented on FLINK-947:
--

Github user rmetzger commented on a diff in the pull request:

https://github.com/apache/flink/pull/405#discussion_r24857326
  
--- Diff: flink-staging/flink-linq/pom.xml ---
@@ -0,0 +1,227 @@
+
+
+http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/maven-v4_0_0.xsd";>
+
+   4.0.0
+
+   
+   org.apache.flink
+   flink-staging
+   0.9-SNAPSHOT
+   ..
+   
+
+   flink-linq
+   flink-linq
+
+   jar
+
+   
+
+   
+   org.apache.flink
+   flink-scala
+   ${project.version}
+   
+
+   
+   org.apache.flink
+   flink-streaming-scala
+   ${project.version}
+   
+
+   
+   org.apache.flink
+   flink-scala-examples
+   ${project.version}
+   
+
+   
+   org.scala-lang
+   scala-reflect
+   
+
+   
+   org.scala-lang
+   scala-library
+   
+
+   
+   org.scala-lang
+   scala-compiler
+   
+
--- End diff --

I think its really not an issue to directly add your dependencies to the 
pom.
Imagine we change something in the `flink-scala` module.
I was actually thinking about adding a check to maven that every dependency 
has to be added directly. I'm pretty sure there are cases in the project where 
we use stuff Apache Commons libraries which come from external dependencies.


> Add support for "Named Datasets"
> 
>
> Key: FLINK-947
> URL: https://issues.apache.org/jira/browse/FLINK-947
> Project: Flink
>  Issue Type: New Feature
>  Components: Java API
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Minor
>
> This would create an API that is a mix between SQL like declarativity and the 
> power of user defined functions. Example user code could look like this:
> {code:Java}
> NamedDataSet one = ...
> NamedDataSet two = ...
> NamedDataSet result = one.join(two).where("key").equalTo("otherKey")
>   .project("a", "b", "c")
>   .map( (UserTypeIn in) -> return new UserTypeOut(...) )
>   .print();
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-947] Add a declarative expression API

2015-02-17 Thread rmetzger

Github user rmetzger commented on a diff in the pull request:

https://github.com/apache/flink/pull/405#discussion_r24857326
  
--- Diff: flink-staging/flink-linq/pom.xml ---
@@ -0,0 +1,227 @@
+
+
+http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/maven-v4_0_0.xsd";>
+
+   4.0.0
+
+   
+   org.apache.flink
+   flink-staging
+   0.9-SNAPSHOT
+   ..
+   
+
+   flink-linq
+   flink-linq
+
+   jar
+
+   
+
+   
+   org.apache.flink
+   flink-scala
+   ${project.version}
+   
+
+   
+   org.apache.flink
+   flink-streaming-scala
+   ${project.version}
+   
+
+   
+   org.apache.flink
+   flink-scala-examples
+   ${project.version}
+   
+
+   
+   org.scala-lang
+   scala-reflect
+   
+
+   
+   org.scala-lang
+   scala-library
+   
+
+   
+   org.scala-lang
+   scala-compiler
+   
+
--- End diff --

I think its really not an issue to directly add your dependencies to the 
pom.
Imagine we change something in the `flink-scala` module.
I was actually thinking about adding a check to maven that every dependency 
has to be added directly. I'm pretty sure there are cases in the project where 
we use stuff Apache Commons libraries which come from external dependencies.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1417] Automatically register types with...

2015-02-17 Thread rmetzger

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/393#issuecomment-74754391
  
No, its basically CoGrouping two "big" files (~50 and ~75 GB) 

https://github.com/project-flink/flink-perf/blob/master/flink-jobs/src/main/java/com/github/projectflink/avro/AvroPerf.java


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1417) Automatically register nested types at Kryo

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324893#comment-14324893
 ] 

ASF GitHub Bot commented on FLINK-1417:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/393#issuecomment-74754391
  
No, its basically CoGrouping two "big" files (~50 and ~75 GB) 

https://github.com/project-flink/flink-perf/blob/master/flink-jobs/src/main/java/com/github/projectflink/avro/AvroPerf.java


> Automatically register nested types at Kryo
> ---
>
> Key: FLINK-1417
> URL: https://issues.apache.org/jira/browse/FLINK-1417
> Project: Flink
>  Issue Type: Improvement
>  Components: Java API, Scala API
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
> Fix For: 0.9
>
>
> Currently, the {{GenericTypeInfo}} registers the class of the type at Kryo. 
> In order to get the best performance, it should recursively walk the classes 
> and make sure that it registered all contained subtypes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1417) Automatically register nested types at Kryo

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324810#comment-14324810
 ] 

ASF GitHub Bot commented on FLINK-1417:
---

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/393#issuecomment-74744132
  
What exactly are you running? TPC-H Query 3? Maybe we should test how fast 
Kryo would be with the PojoComparator.


> Automatically register nested types at Kryo
> ---
>
> Key: FLINK-1417
> URL: https://issues.apache.org/jira/browse/FLINK-1417
> Project: Flink
>  Issue Type: Improvement
>  Components: Java API, Scala API
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
> Fix For: 0.9
>
>
> Currently, the {{GenericTypeInfo}} registers the class of the type at Kryo. 
> In order to get the best performance, it should recursively walk the classes 
> and make sure that it registered all contained subtypes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1417] Automatically register types with...

2015-02-17 Thread aljoscha

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/393#issuecomment-74744132
  
What exactly are you running? TPC-H Query 3? Maybe we should test how fast 
Kryo would be with the PojoComparator.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-1576) Change the Gelly examples to be consistent with the other Flink examples

2015-02-17 Thread Andra Lungu (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andra Lungu updated FLINK-1576:
---
Summary: Change the Gelly examples to be consistent with the other Flink 
examples  (was: Change the examples to be consistent with the other Flink 
examples)

> Change the Gelly examples to be consistent with the other Flink examples
> 
>
> Key: FLINK-1576
> URL: https://issues.apache.org/jira/browse/FLINK-1576
> Project: Flink
>  Issue Type: Improvement
>  Components: Gelly
>Affects Versions: 0.8
>Reporter: Andra Lungu
>  Labels: easyfix
>
> The current Gelly examples just work on default input data. 
> If we look at the other Flink examples, e.g. Connected Components, they also 
> allow input data to be read from a text file passed as a parameter to the 
> main method. 
>  
> It would be nice to follow the same approach in our examples. A first step in 
> that direction is the SSSP example. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-1576) Change the examples to be consistent with the other Flink examples

2015-02-17 Thread Andra Lungu (JIRA)

Andra Lungu created FLINK-1576:
--

 Summary: Change the examples to be consistent with the other Flink 
examples
 Key: FLINK-1576
 URL: https://issues.apache.org/jira/browse/FLINK-1576
 Project: Flink
  Issue Type: Improvement
  Components: Gelly
Affects Versions: 0.8
Reporter: Andra Lungu


The current Gelly examples just work on default input data. 

If we look at the other Flink examples, e.g. Connected Components, they also 
allow input data to be read from a text file passed as a parameter to the main 
method. 
 
It would be nice to follow the same approach in our examples. A first step in 
that direction is the SSSP example. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1522) Add tests for the library methods and examples

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324596#comment-14324596
 ] 

ASF GitHub Bot commented on FLINK-1522:
---

Github user andralungu commented on the pull request:

https://github.com/apache/flink/pull/414#issuecomment-74719038
  
@rmetzger , thanks for the heads up!
Now, instead of returining null, I have a default vertex data set and a 
default edge data set. Tell me what you think :). 


> Add tests for the library methods and examples
> --
>
> Key: FLINK-1522
> URL: https://issues.apache.org/jira/browse/FLINK-1522
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Reporter: Vasia Kalavri
>Assignee: Daniel Bali
>  Labels: easyfix, test
>
> The current tests in gelly test one method at a time. We should have some 
> tests for complete applications. As a start, we could add one test case per 
> example and this way also make sure that our graph library methods actually 
> give correct results.
> I'm assigning this to [~andralungu] because she has already implemented the 
> test for SSSP, but I will help as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1522][gelly] Added test for SSSP Exampl...

2015-02-17 Thread andralungu

Github user andralungu commented on the pull request:

https://github.com/apache/flink/pull/414#issuecomment-74719038
  
@rmetzger , thanks for the heads up!
Now, instead of returining null, I have a default vertex data set and a 
default edge data set. Tell me what you think :). 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-1575) JobManagerConnectionTest.testResolveUnreachableActorRemoteHost times out on travis

2015-02-17 Thread Robert Metzger (JIRA)

Robert Metzger created FLINK-1575:
-

 Summary: 
JobManagerConnectionTest.testResolveUnreachableActorRemoteHost times out on 
travis
 Key: FLINK-1575
 URL: https://issues.apache.org/jira/browse/FLINK-1575
 Project: Flink
  Issue Type: Bug
Reporter: Robert Metzger


This might be related to FLINK-1529.

I saw this issue now at least twice on travis:
https://travis-ci.org/rmetzger/flink/jobs/51108554

{code}
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 200.266 sec <<< 
FAILURE! - in org.apache.flink.runtime.jobmanager.JobManagerConnectionTest
testResolveUnreachableActorRemoteHost(org.apache.flink.runtime.jobmanager.JobManagerConnectionTest)
  Time elapsed: 100.215 sec  <<< ERROR!
java.util.concurrent.TimeoutException: Futures timed out after [10 
milliseconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at 
scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
at 
scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:107)
at akka.remote.Remoting.start(Remoting.scala:173)
at 
akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:184)
at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:579)
at akka.actor.ActorSystemImpl._start(ActorSystem.scala:577)
at akka.actor.ActorSystemImpl.start(ActorSystem.scala:588)
at akka.actor.ActorSystem$.apply(ActorSystem.scala:111)
at akka.actor.ActorSystem$.apply(ActorSystem.scala:104)
at akka.actor.ActorSystem$.create(ActorSystem.scala:66)
at 
org.apache.flink.runtime.akka.AkkaUtils$.createActorSystem(AkkaUtils.scala:71)
at 
org.apache.flink.runtime.akka.AkkaUtils$.createActorSystem(AkkaUtils.scala:61)
at 
org.apache.flink.runtime.jobmanager.JobManagerConnectionTest.testResolveUnreachableActorRemoteHost(JobManagerConnectionTest.scala:88)

testResolveUnreachableActorLocalHost(org.apache.flink.runtime.jobmanager.JobManagerConnectionTest)
  Time elapsed: 100.031 sec  <<< ERROR!
java.util.concurrent.TimeoutException: Futures timed out after [10 
milliseconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at 
scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
at 
scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:107)
at akka.remote.Remoting.start(Remoting.scala:173)
at 
akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:184)
at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:579)
at akka.actor.ActorSystemImpl._start(ActorSystem.scala:577)
at akka.actor.ActorSystemImpl.start(ActorSystem.scala:588)
at akka.actor.ActorSystem$.apply(ActorSystem.scala:111)
at akka.actor.ActorSystem$.apply(ActorSystem.scala:104)
at akka.actor.ActorSystem$.create(ActorSystem.scala:66)
at 
org.apache.flink.runtime.akka.AkkaUtils$.createActorSystem(AkkaUtils.scala:71)
at 
org.apache.flink.runtime.akka.AkkaUtils$.createActorSystem(AkkaUtils.scala:61)
at 
org.apache.flink.runtime.jobmanager.JobManagerConnectionTest.testResolveUnreachableActorLocalHost(JobManagerConnectionTest.scala:45)

Running org.apache.flink.runtime.operators.hash.MemoryHashTableTest
[ERROR] [02/17/2015 17:38:04.250] [main] [Remoting] Remoting error: [Startup 
timed out] [
akka.remote.RemoteTransportException: Startup timed out
at 
akka.remote.Remoting.akka$remote$Remoting$$notifyError(Remoting.scala:129)
at akka.remote.Remoting.start(Remoting.scala:191)
at 
akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:184)
at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:579)
at akka.actor.ActorSystemImpl._start(ActorSystem.scala:577)
at akka.actor.ActorSystemImpl.start(ActorSystem.scala:588)
at akka.actor.ActorSystem$.apply(ActorSystem.scala:111)
at akka.actor.ActorSystem$.apply(ActorSystem.scala:104)
at akka.actor.ActorSystem$.create(ActorSystem.scala:66)
at 
org.apache.flink.runtime.akka.AkkaUtils$.createActorSystem(AkkaUtils.scala:71)
at 
org.apache.flink.runtime.akka.AkkaUtils$.createActorSystem(AkkaUtils.scala:61)
at 
org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:605)
at 
org.apache.flink.runtime.jobmanager.JobManager.runJobManager(JobManager.scala)
at 
org.apache.flink.runtime.jobmanager.JobManagerStartupTest.testJobManagerStartupFails(Job

[jira] [Created] (FLINK-1574) Flink fails due to non-initialized RuntimeContext in CombiningUnilateralSortMerger

2015-02-17 Thread Fabian Hueske (JIRA)

Fabian Hueske created FLINK-1574:


 Summary: Flink fails due to non-initialized RuntimeContext in 
CombiningUnilateralSortMerger
 Key: FLINK-1574
 URL: https://issues.apache.org/jira/browse/FLINK-1574
 Project: Flink
  Issue Type: Bug
  Components: Local Runtime
Affects Versions: 0.9
Reporter: Fabian Hueske
Priority: Critical
 Fix For: 0.9


The {{AggregatingUdf}} combiner cannot be opened in a 
{{CombiningUnilateralSortMerger }} because its {{RuntimeContext}} is {{null}}.
The problem is that the runtime context of the {{RegularPactTask}} is set after 
the local strategies are initialized.

I have a fix for that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1417) Automatically register nested types at Kryo

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324527#comment-14324527
 ] 

ASF GitHub Bot commented on FLINK-1417:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/393#issuecomment-74712141
  
The pull request now also contains changes to fix two classloading issues 
with the newly added `ExecutionConfig`.
Also, it fixes the Web interface for YARN.

I would like to merge this change soon. I've tested the code on extensively 
on a cluster and it seems to be stable.
I'm still working on the performance numbers.
The results I've got so far are the following:

Scale Factor 1 (lineitem only)
Pojo: 20
Kryo (with key selectors):  13
Avro (with key selectors): 10, 15, 13

Scale factor 100:
Pojo: 1634,

I'll update the comment to complete it.


> Automatically register nested types at Kryo
> ---
>
> Key: FLINK-1417
> URL: https://issues.apache.org/jira/browse/FLINK-1417
> Project: Flink
>  Issue Type: Improvement
>  Components: Java API, Scala API
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
> Fix For: 0.9
>
>
> Currently, the {{GenericTypeInfo}} registers the class of the type at Kryo. 
> In order to get the best performance, it should recursively walk the classes 
> and make sure that it registered all contained subtypes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1417] Automatically register types with...

2015-02-17 Thread rmetzger

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/393#issuecomment-74712141
  
The pull request now also contains changes to fix two classloading issues 
with the newly added `ExecutionConfig`.
Also, it fixes the Web interface for YARN.

I would like to merge this change soon. I've tested the code on extensively 
on a cluster and it seems to be stable.
I'm still working on the performance numbers.
The results I've got so far are the following:

Scale Factor 1 (lineitem only)
Pojo: 20
Kryo (with key selectors):  13
Avro (with key selectors): 10, 15, 13

Scale factor 100:
Pojo: 1634,

I'll update the comment to complete it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1466] Add HCatInputFormats to read from...

2015-02-17 Thread rmetzger

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/411#issuecomment-74710618
  
I wonder how much work it would be to add support for the last three 
missing types.
With Kryo we should be able to serialize all of them easily.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1466) Add InputFormat to read HCatalog tables

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324509#comment-14324509
 ] 

ASF GitHub Bot commented on FLINK-1466:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/411#issuecomment-74710618
  
I wonder how much work it would be to add support for the last three 
missing types.
With Kryo we should be able to serialize all of them easily.


> Add InputFormat to read HCatalog tables
> ---
>
> Key: FLINK-1466
> URL: https://issues.apache.org/jira/browse/FLINK-1466
> Project: Flink
>  Issue Type: New Feature
>  Components: Java API, Scala API
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
>Priority: Minor
>
> HCatalog is a metadata repository and InputFormat to make Hive tables 
> accessible to other frameworks such as Pig.
> Adding support for HCatalog would give access to Hive managed data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1466) Add InputFormat to read HCatalog tables

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324499#comment-14324499
 ] 

ASF GitHub Bot commented on FLINK-1466:
---

Github user rmetzger commented on a diff in the pull request:

https://github.com/apache/flink/pull/411#discussion_r24833839
  
--- Diff: 
flink-staging/flink-hcatalog/src/main/java/org/apache/flink/hcatalog/HCatInputFormatBase.java
 ---
@@ -0,0 +1,413 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.hcatalog;
+
+import org.apache.flink.api.common.io.InputFormat;
+import org.apache.flink.api.common.io.LocatableInputSplitAssigner;
+import org.apache.flink.api.common.io.statistics.BaseStatistics;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.hadoop.mapreduce.utils.HadoopUtils;
+import org.apache.flink.api.java.hadoop.mapreduce.wrapper.HadoopInputSplit;
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.api.java.typeutils.WritableTypeInfo;
+import org.apache.flink.core.io.InputSplitAssigner;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.WritableComparable;
+import org.apache.hadoop.mapreduce.InputSplit;
+import org.apache.hadoop.mapreduce.JobContext;
+import org.apache.hadoop.mapreduce.JobID;
+import org.apache.hadoop.mapreduce.RecordReader;
+import org.apache.hadoop.mapreduce.TaskAttemptContext;
+import org.apache.hadoop.mapreduce.TaskAttemptID;
+import org.apache.hive.hcatalog.common.HCatException;
+import org.apache.hive.hcatalog.common.HCatUtil;
+import org.apache.hive.hcatalog.data.DefaultHCatRecord;
+import org.apache.hive.hcatalog.data.HCatRecord;
+import org.apache.hive.hcatalog.data.schema.HCatFieldSchema;
+import org.apache.hive.hcatalog.data.schema.HCatSchema;
+
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * A InputFormat to read from HCatalog tables.
+ * The InputFormat supports projection (selection and order of fields) and 
partition filters.
+ *
+ * Data can be returned as {@link 
org.apache.hive.hcatalog.data.HCatRecord} or Flink {@link 
org.apache.flink.api.java.tuple.Tuple}.
+ * Flink Tuples are only supported for primitive type fields
+ * (no STRUCT, ARRAY, or MAP data types) and have a size limitation.
+ *
+ * @param 
+ */
+public abstract class HCatInputFormatBase implements InputFormat, ResultTypeQueryable {
+
+   private static final long serialVersionUID = 1L;
+
+   private Configuration configuration;
+
+   private org.apache.hive.hcatalog.mapreduce.HCatInputFormat 
hCatInputFormat;
+   private RecordReader recordReader;
+   private boolean fetched = false;
+   private boolean hasNext;
+
+   protected String[] fieldNames = new String[0];
+   protected HCatSchema outputSchema;
+
+   private TypeInformation resultType;
+
+   public HCatInputFormatBase() { }
+
+   /**
+* Creates a HCatInputFormat for the given database and table.
+* By default, the InputFormat returns {@link 
org.apache.hive.hcatalog.data.HCatRecord}.
+* The return type of the InputFormat can be changed to Flink {@link 
org.apache.flink.api.java.tuple.Tuple} by calling
+* {@link HCatInputFormatBase#asFlinkTuples()}.
+*
+* @param database The name of the database to read from.
+* @param table The name of the table to read.
+* @throws java.io.IOException
+*/
+   public HCatInputFormatBase(String database, String table) throws 
IOException {
+   thi

[GitHub] flink pull request: [FLINK-1466] Add HCatInputFormats to read from...

2015-02-17 Thread rmetzger

Github user rmetzger commented on a diff in the pull request:

https://github.com/apache/flink/pull/411#discussion_r24833839
  
--- Diff: 
flink-staging/flink-hcatalog/src/main/java/org/apache/flink/hcatalog/HCatInputFormatBase.java
 ---
@@ -0,0 +1,413 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.hcatalog;
+
+import org.apache.flink.api.common.io.InputFormat;
+import org.apache.flink.api.common.io.LocatableInputSplitAssigner;
+import org.apache.flink.api.common.io.statistics.BaseStatistics;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.hadoop.mapreduce.utils.HadoopUtils;
+import org.apache.flink.api.java.hadoop.mapreduce.wrapper.HadoopInputSplit;
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.api.java.typeutils.WritableTypeInfo;
+import org.apache.flink.core.io.InputSplitAssigner;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.WritableComparable;
+import org.apache.hadoop.mapreduce.InputSplit;
+import org.apache.hadoop.mapreduce.JobContext;
+import org.apache.hadoop.mapreduce.JobID;
+import org.apache.hadoop.mapreduce.RecordReader;
+import org.apache.hadoop.mapreduce.TaskAttemptContext;
+import org.apache.hadoop.mapreduce.TaskAttemptID;
+import org.apache.hive.hcatalog.common.HCatException;
+import org.apache.hive.hcatalog.common.HCatUtil;
+import org.apache.hive.hcatalog.data.DefaultHCatRecord;
+import org.apache.hive.hcatalog.data.HCatRecord;
+import org.apache.hive.hcatalog.data.schema.HCatFieldSchema;
+import org.apache.hive.hcatalog.data.schema.HCatSchema;
+
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * A InputFormat to read from HCatalog tables.
+ * The InputFormat supports projection (selection and order of fields) and 
partition filters.
+ *
+ * Data can be returned as {@link 
org.apache.hive.hcatalog.data.HCatRecord} or Flink {@link 
org.apache.flink.api.java.tuple.Tuple}.
+ * Flink Tuples are only supported for primitive type fields
+ * (no STRUCT, ARRAY, or MAP data types) and have a size limitation.
+ *
+ * @param 
+ */
+public abstract class HCatInputFormatBase implements InputFormat, ResultTypeQueryable {
+
+   private static final long serialVersionUID = 1L;
+
+   private Configuration configuration;
+
+   private org.apache.hive.hcatalog.mapreduce.HCatInputFormat 
hCatInputFormat;
+   private RecordReader recordReader;
+   private boolean fetched = false;
+   private boolean hasNext;
+
+   protected String[] fieldNames = new String[0];
+   protected HCatSchema outputSchema;
+
+   private TypeInformation resultType;
+
+   public HCatInputFormatBase() { }
+
+   /**
+* Creates a HCatInputFormat for the given database and table.
+* By default, the InputFormat returns {@link 
org.apache.hive.hcatalog.data.HCatRecord}.
+* The return type of the InputFormat can be changed to Flink {@link 
org.apache.flink.api.java.tuple.Tuple} by calling
+* {@link HCatInputFormatBase#asFlinkTuples()}.
+*
+* @param database The name of the database to read from.
+* @param table The name of the table to read.
+* @throws java.io.IOException
+*/
+   public HCatInputFormatBase(String database, String table) throws 
IOException {
+   this(database, table, new Configuration());
+   }
+
+   /**
+* Creates a HCatInputFormat for the given database, table, and
+* {@link org.apache.hadoop.conf.Configuration}.
+* By default, the InputFormat returns {@link 
org

[jira] [Created] (FLINK-1573) Add per-job metrics to flink.

2015-02-17 Thread Robert Metzger (JIRA)

Robert Metzger created FLINK-1573:
-

 Summary: Add per-job metrics to flink.
 Key: FLINK-1573
 URL: https://issues.apache.org/jira/browse/FLINK-1573
 Project: Flink
  Issue Type: Sub-task
Affects Versions: 0.9
Reporter: Robert Metzger
Assignee: Robert Metzger


With FLINK-1501, we have JVM specific metrics (mainly monitoring the TMs).

With this task, I would like to add metrics which are job-specific.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1522][gelly] Added test for SSSP Exampl...

2015-02-17 Thread rmetzger

Github user rmetzger commented on a diff in the pull request:

https://github.com/apache/flink/pull/414#discussion_r24829996
  
--- Diff: 
flink-staging/flink-gelly/src/main/java/org/apache/flink/graph/example/SingleSourceShortestPathsExample.java
 ---
@@ -56,4 +63,77 @@ public static void main(String[] args) throws Exception {
public String getDescription() {
return "Single Source Shortest Paths";
}
+
+   // 
**
+   // UTIL METHODS
+   // 
**
+
+   private static boolean fileOutput = false;
+
+   private static Long srcVertexId = null;
+
+   private static String verticesInputPath = null;
+
+   private static String edgesInputPath = null;
+
+   private static String outputPath = null;
+
+   private static int maxIterations = 5;
+
+   private static boolean parseParameters(String[] args) {
+
+   if (args.length > 0) {
+   if (args.length == 5) {
+   fileOutput = true;
+   srcVertexId = Long.parseLong(args[0]);
+   verticesInputPath = args[1];
+   edgesInputPath = args[2];
+   outputPath = args[3];
+   maxIterations = Integer.parseInt(args[4]);
+   } else {
+   System.err.println("Usage: 
SingleSourceShortestPaths " +
+   "");
+   return false;
+   }
+   }
+   return true;
+   }
+
+   private static DataSet> 
getVerticesDataSet(ExecutionEnvironment env) {
+   if (fileOutput) {
+   return env.readCsvFile(verticesInputPath)
+   .lineDelimiter("\n")
+   .types(Long.class, Double.class)
+   .map(new MapFunction, Vertex>() {
+
+   @Override
+   public Vertex 
map(Tuple2 tuple2) throws Exception {
+   return new Vertex(tuple2.f0, tuple2.f1);
+   }
+   });
+   } else {
+   System.err.println("Usage: SingleSourceShortestPaths 
" +
+   "");
+   return null;
--- End diff --

I suspect the code will fail with a null pointer exception?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1561) Improve build server robustness by not reusing JVMs in integration tests

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324424#comment-14324424
 ] 

ASF GitHub Bot commented on FLINK-1561:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/412#issuecomment-74700668
  
+1


> Improve build server robustness by not reusing JVMs in integration tests
> 
>
> Key: FLINK-1561
> URL: https://issues.apache.org/jira/browse/FLINK-1561
> Project: Flink
>  Issue Type: Improvement
>  Components: Build System
>Affects Versions: 0.9
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 0.9
>
>
> Right now, JVM forks are reused across unit and integration tests. While it 
> makes a lot of sense for the very quick unit tests, the integration tests 
> benefit less.
> We also see spurious JVM failures in the integration tests that are most 
> likely related to JVM issues. An example is the trace below:
> {code}
> java.io.IOException: Cannot allocate memory
>   at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>   at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
>   at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>   at sun.nio.ch.IOUtil.write(IOUtil.java:65)
>   at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
>   at 
> org.apache.flink.runtime.io.disk.iomanager.SegmentWriteRequest.write(AsynchronousFileIOChannel.java:270)
>   at 
> org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync$WriterThread.run(IOManagerAsync.java:440)
> {code}
> I think we can improve reliability of Travis tests by not reusing forks for 
> integration tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1561] [build system] Use a fresh fork f...

2015-02-17 Thread rmetzger

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/412#issuecomment-74700668
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1522) Add tests for the library methods and examples

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324429#comment-14324429
 ] 

ASF GitHub Bot commented on FLINK-1522:
---

Github user rmetzger commented on a diff in the pull request:

https://github.com/apache/flink/pull/414#discussion_r24829996
  
--- Diff: 
flink-staging/flink-gelly/src/main/java/org/apache/flink/graph/example/SingleSourceShortestPathsExample.java
 ---
@@ -56,4 +63,77 @@ public static void main(String[] args) throws Exception {
public String getDescription() {
return "Single Source Shortest Paths";
}
+
+   // 
**
+   // UTIL METHODS
+   // 
**
+
+   private static boolean fileOutput = false;
+
+   private static Long srcVertexId = null;
+
+   private static String verticesInputPath = null;
+
+   private static String edgesInputPath = null;
+
+   private static String outputPath = null;
+
+   private static int maxIterations = 5;
+
+   private static boolean parseParameters(String[] args) {
+
+   if (args.length > 0) {
+   if (args.length == 5) {
+   fileOutput = true;
+   srcVertexId = Long.parseLong(args[0]);
+   verticesInputPath = args[1];
+   edgesInputPath = args[2];
+   outputPath = args[3];
+   maxIterations = Integer.parseInt(args[4]);
+   } else {
+   System.err.println("Usage: 
SingleSourceShortestPaths " +
+   "");
+   return false;
+   }
+   }
+   return true;
+   }
+
+   private static DataSet> 
getVerticesDataSet(ExecutionEnvironment env) {
+   if (fileOutput) {
+   return env.readCsvFile(verticesInputPath)
+   .lineDelimiter("\n")
+   .types(Long.class, Double.class)
+   .map(new MapFunction, Vertex>() {
+
+   @Override
+   public Vertex 
map(Tuple2 tuple2) throws Exception {
+   return new Vertex(tuple2.f0, tuple2.f1);
+   }
+   });
+   } else {
+   System.err.println("Usage: SingleSourceShortestPaths 
" +
+   "");
+   return null;
--- End diff --

I suspect the code will fail with a null pointer exception?


> Add tests for the library methods and examples
> --
>
> Key: FLINK-1522
> URL: https://issues.apache.org/jira/browse/FLINK-1522
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Reporter: Vasia Kalavri
>Assignee: Daniel Bali
>  Labels: easyfix, test
>
> The current tests in gelly test one method at a time. We should have some 
> tests for complete applications. As a start, we could add one test case per 
> example and this way also make sure that our graph library methods actually 
> give correct results.
> I'm assigning this to [~andralungu] because she has already implemented the 
> test for SSSP, but I will help as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-1572) Output directories are created before input paths are checked

2015-02-17 Thread Robert Metzger (JIRA)

Robert Metzger created FLINK-1572:
-

 Summary: Output directories are created before input paths are 
checked
 Key: FLINK-1572
 URL: https://issues.apache.org/jira/browse/FLINK-1572
 Project: Flink
  Issue Type: Improvement
  Components: JobManager
Affects Versions: 0.9
Reporter: Robert Metzger
Priority: Minor


Flink is first creating the output directories for a job before creating the 
input splits.
If a job's input directories are wrong, the system will have created output 
directories for a failed job.

It would be much better if the system is creating the output directories "on 
demand" before data is actually written.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-1556) JobClient does not wait until a job failed completely if submission exception

2015-02-17 Thread Robert Metzger (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324364#comment-14324364
 ] 

Robert Metzger edited comment on FLINK-1556 at 2/17/15 4:15 PM:


Also, it seems that these "fail-fast" jobs are not properly removed from the 
jobmanager?

http://imgur.com/PyuQEfm



was (Author: rmetzger):
Also, it seems that these "failearily" jobs are not properly removed from the 
jobmanager?

http://imgur.com/PyuQEfm


> JobClient does not wait until a job failed completely if submission exception
> -
>
> Key: FLINK-1556
> URL: https://issues.apache.org/jira/browse/FLINK-1556
> Project: Flink
>  Issue Type: Bug
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>
> If an exception occurs during job submission the {{JobClient}} received a 
> {{SubmissionFailure}}. Upon receiving this message, the {{JobClient}} 
> terminates itself and returns the error to the {{Client}}. This indicates to 
> the user that the job has been completely failed which is not necessarily 
> true. 
> If the user directly after such a failure submits another job, then it might 
> be the case that not all slots of the formerly failed job are returned. This 
> can lead to a {{NoRessourceAvailableException}}.
> We can solve this problem by waiting for the completion of the job failure in 
> the {{JobClient}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1556) JobClient does not wait until a job failed completely if submission exception

2015-02-17 Thread Robert Metzger (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324364#comment-14324364
 ] 

Robert Metzger commented on FLINK-1556:
---

Also, it seems that these "failearily" jobs are not properly removed from the 
jobmanager?

http://imgur.com/PyuQEfm


> JobClient does not wait until a job failed completely if submission exception
> -
>
> Key: FLINK-1556
> URL: https://issues.apache.org/jira/browse/FLINK-1556
> Project: Flink
>  Issue Type: Bug
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>
> If an exception occurs during job submission the {{JobClient}} received a 
> {{SubmissionFailure}}. Upon receiving this message, the {{JobClient}} 
> terminates itself and returns the error to the {{Client}}. This indicates to 
> the user that the job has been completely failed which is not necessarily 
> true. 
> If the user directly after such a failure submits another job, then it might 
> be the case that not all slots of the formerly failed job are returned. This 
> can lead to a {{NoRessourceAvailableException}}.
> We can solve this problem by waiting for the completion of the job failure in 
> the {{JobClient}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (FLINK-1556) JobClient does not wait until a job failed completely if submission exception

2015-02-17 Thread Robert Metzger (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger reopened FLINK-1556:
---
  Assignee: Till Rohrmann

The issue hasn't been implemented/tested properly.

I'm reopening it.

I was submitting a job with a wrong directory configured. I got the following 
output:
{code}
./flink run -v -p 152 -c com.github.projectflink.avro.GenerateLineitems 
../../testjob/flink-jobs/target/flink-jobs-0.1-SNAPSHOT.jar -p 144 -o 
hdfs:///user/robert/datasets/tpch100/
16:55:04,748 WARN  org.apache.hadoop.util.NativeCodeLoader  
 - Unable to load native-hadoop library for your platform... using builtin-java 
classes where applicable
Found a yarn properties file (.yarn-properties) file, using 
"cloud-31.dima.tu-berlin.de:33806" to connect to the JobManager
Submission of job with ID 11bc00bb2a27105221a4137048e3a763 was unsuccessful, 
because File or directory already exists. Existing files and directories are 
not overwritten in NO_OVERWRITE mode. Use OVERWRITE mode to overwrite existing 
files and directories..
{code}
But the CliFrontend didn't stop. (I waited for more than a minute).

jstack:

{code}
$ jstack 25475
2015-02-17 16:56:43
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.76-b04 mixed mode):

"Attach Listener" daemon prio=10 tid=0x013c9000 nid=0x63dc waiting on 
condition [0x]
   java.lang.Thread.State: RUNNABLE

"Hashed wheel timer #1" daemon prio=10 tid=0x01375000 nid=0x63a9 
waiting on condition [0x7f3582a96000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at 
org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:483)
at 
org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:392)
at 
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at java.lang.Thread.run(Thread.java:745)

"New I/O server boss #6" daemon prio=10 tid=0x7f3584123000 nid=0x63a8 
runnable [0x7f3582b97000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00070ac52878> (a sun.nio.ch.Util$2)
- locked <0x00070ac52868> (a java.util.Collections$UnmodifiableSet)
- locked <0x00070ac52750> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at 
org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
at 
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at 
org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at 
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at 
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #5" daemon prio=10 tid=0x7f3584124800 nid=0x63a7 runnable 
[0x7f3582c98000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00070ac35e20> (a sun.nio.ch.Util$2)
- locked <0x00070ac35e10> (a java.util.Collections$UnmodifiableSet)
- locked <0x00070ac35cf8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at 
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at 
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at 
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at 
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at 
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at 
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concur

[jira] [Commented] (FLINK-1556) JobClient does not wait until a job failed completely if submission exception

2015-02-17 Thread Robert Metzger (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324359#comment-14324359
 ] 

Robert Metzger commented on FLINK-1556:
---

I think showing the whole stacktrace of the exception is helpful to understand 
the deployment issue better.

> JobClient does not wait until a job failed completely if submission exception
> -
>
> Key: FLINK-1556
> URL: https://issues.apache.org/jira/browse/FLINK-1556
> Project: Flink
>  Issue Type: Bug
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>
> If an exception occurs during job submission the {{JobClient}} received a 
> {{SubmissionFailure}}. Upon receiving this message, the {{JobClient}} 
> terminates itself and returns the error to the {{Client}}. This indicates to 
> the user that the job has been completely failed which is not necessarily 
> true. 
> If the user directly after such a failure submits another job, then it might 
> be the case that not all slots of the formerly failed job are returned. This 
> can lead to a {{NoRessourceAvailableException}}.
> We can solve this problem by waiting for the completion of the job failure in 
> the {{JobClient}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1566) WindowIntegrationTest fails

2015-02-17 Thread Gyula Fora (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324356#comment-14324356
 ] 

Gyula Fora commented on FLINK-1566:
---

it is a concurrency issue while adding to an ArrayList from multiple threads. 
Turns out its not thread-safe :P

https://github.com/mbalassi/flink/commit/c34f0cb160e91742fe18de87b99d22c039f3f8ea

I will push it if travis passes

> WindowIntegrationTest fails
> ---
>
> Key: FLINK-1566
> URL: https://issues.apache.org/jira/browse/FLINK-1566
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Reporter: Till Rohrmann
>
> I'm seeing spurious failures of the {{WindowIntegrationTest}}.
> The error is the following:
> {code}
> java.lang.AssertionError: 
> Expected :[[3], [4], [5], [11], [10], [16]]
> Actual   :[[3], null, [5], [11], [10], [16]]
>  
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:144)
>   at 
> org.apache.flink.streaming.api.invokable.operator.windowing.WindowIntegrationTest.validateOutput(WindowIntegrationTest.java:151)
>   at 
> org.apache.flink.streaming.api.invokable.operator.windowing.WindowIntegrationTest.test(WindowIntegrationTest.java:134)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-1522) Add tests for the library methods and examples

2015-02-17 Thread Daniel Bali (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Bali reassigned FLINK-1522:
--

Assignee: Daniel Bali  (was: Andra Lungu)

> Add tests for the library methods and examples
> --
>
> Key: FLINK-1522
> URL: https://issues.apache.org/jira/browse/FLINK-1522
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Reporter: Vasia Kalavri
>Assignee: Daniel Bali
>  Labels: easyfix, test
>
> The current tests in gelly test one method at a time. We should have some 
> tests for complete applications. As a start, we could add one test case per 
> example and this way also make sure that our graph library methods actually 
> give correct results.
> I'm assigning this to [~andralungu] because she has already implemented the 
> test for SSSP, but I will help as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1515) [Gelly] Enable access to aggregators and broadcast sets in vertex-centric iteration

2015-02-17 Thread Vasia Kalavri (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324250#comment-14324250
 ] 

Vasia Kalavri commented on FLINK-1515:
--

Sounds good to me, thanks :)

> [Gelly] Enable access to aggregators and broadcast sets in vertex-centric 
> iteration
> ---
>
> Key: FLINK-1515
> URL: https://issues.apache.org/jira/browse/FLINK-1515
> Project: Flink
>  Issue Type: Improvement
>  Components: Gelly
>Reporter: Vasia Kalavri
>Assignee: Martin Kiefer
>
> Currently, aggregators and broadcast sets cannot be accessed through Gelly's  
> {{runVertexCentricIteration}} method. The functionality is already present in 
> the {{VertexCentricIteration}} and we just need to expose it.
> This could be done like this: We create a method 
> {{createVertexCentricIteration}}, which will return a 
> {{VertexCentricIteration}} object and we change {{runVertexCentricIteration}} 
> to accept this as a parameter (and return the graph after running this 
> iteration).
> The user can configure the {{VertexCentricIteration}} by directly calling the 
> public methods {{registerAggregator}}, {{setName}}, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1522) Add tests for the library methods and examples

2015-02-17 Thread Daniel Bali (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324239#comment-14324239
 ] 

Daniel Bali commented on FLINK-1522:


Hi Andra!

Vasia said you have other issues to work on, so I'd happily continue the work 
on this one.

> Add tests for the library methods and examples
> --
>
> Key: FLINK-1522
> URL: https://issues.apache.org/jira/browse/FLINK-1522
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Reporter: Vasia Kalavri
>Assignee: Andra Lungu
>  Labels: easyfix, test
>
> The current tests in gelly test one method at a time. We should have some 
> tests for complete applications. As a start, we could add one test case per 
> example and this way also make sure that our graph library methods actually 
> give correct results.
> I'm assigning this to [~andralungu] because she has already implemented the 
> test for SSSP, but I will help as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (FLINK-1522) Add tests for the library methods and examples

2015-02-17 Thread Daniel Bali (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Bali updated FLINK-1522:
---
Comment: was deleted

(was: Hi Andra!

Vasia said you have other issues to work on, so I'd happily continue the work 
on this one.)

> Add tests for the library methods and examples
> --
>
> Key: FLINK-1522
> URL: https://issues.apache.org/jira/browse/FLINK-1522
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Reporter: Vasia Kalavri
>Assignee: Andra Lungu
>  Labels: easyfix, test
>
> The current tests in gelly test one method at a time. We should have some 
> tests for complete applications. As a start, we could add one test case per 
> example and this way also make sure that our graph library methods actually 
> give correct results.
> I'm assigning this to [~andralungu] because she has already implemented the 
> test for SSSP, but I will help as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1522) Add tests for the library methods and examples

2015-02-17 Thread Daniel Bali (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324238#comment-14324238
 ] 

Daniel Bali commented on FLINK-1522:


Hi Andra!

Vasia said you have other issues to work on, so I'd happily continue the work 
on this one.

> Add tests for the library methods and examples
> --
>
> Key: FLINK-1522
> URL: https://issues.apache.org/jira/browse/FLINK-1522
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Reporter: Vasia Kalavri
>Assignee: Andra Lungu
>  Labels: easyfix, test
>
> The current tests in gelly test one method at a time. We should have some 
> tests for complete applications. As a start, we could add one test case per 
> example and this way also make sure that our graph library methods actually 
> give correct results.
> I'm assigning this to [~andralungu] because she has already implemented the 
> test for SSSP, but I will help as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1522) Add tests for the library methods and examples

2015-02-17 Thread Daniel Bali (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324237#comment-14324237
 ] 

Daniel Bali commented on FLINK-1522:


Hi Andra!

Vasia said you have other issues to work on, so I'd happily continue the work 
on this one.

> Add tests for the library methods and examples
> --
>
> Key: FLINK-1522
> URL: https://issues.apache.org/jira/browse/FLINK-1522
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Reporter: Vasia Kalavri
>Assignee: Andra Lungu
>  Labels: easyfix, test
>
> The current tests in gelly test one method at a time. We should have some 
> tests for complete applications. As a start, we could add one test case per 
> example and this way also make sure that our graph library methods actually 
> give correct results.
> I'm assigning this to [~andralungu] because she has already implemented the 
> test for SSSP, but I will help as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (FLINK-1522) Add tests for the library methods and examples

2015-02-17 Thread Daniel Bali (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Bali updated FLINK-1522:
---
Comment: was deleted

(was: Hi Andra!

Vasia said you have other issues to work on, so I'd happily continue the work 
on this one.)

> Add tests for the library methods and examples
> --
>
> Key: FLINK-1522
> URL: https://issues.apache.org/jira/browse/FLINK-1522
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Reporter: Vasia Kalavri
>Assignee: Andra Lungu
>  Labels: easyfix, test
>
> The current tests in gelly test one method at a time. We should have some 
> tests for complete applications. As a start, we could add one test case per 
> example and this way also make sure that our graph library methods actually 
> give correct results.
> I'm assigning this to [~andralungu] because she has already implemented the 
> test for SSSP, but I will help as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1552) Allow secondary sorts in Vertex Centric Iteration

2015-02-17 Thread Vasia Kalavri (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324184#comment-14324184
 ] 

Vasia Kalavri commented on FLINK-1552:
--

No problem, let's see if someone picks it up until then :)
What was your IMPRO3 algorithm btw? We could maybe use it as an example. Thanks!

> Allow secondary sorts in Vertex Centric Iteration
> -
>
> Key: FLINK-1552
> URL: https://issues.apache.org/jira/browse/FLINK-1552
> Project: Flink
>  Issue Type: Wish
>  Components: Gelly
>Reporter: Martin Kiefer
>Priority: Minor
>
> The `VertexCentricIteration` class holds the logic to transform a 
> `VertexUpdateFunction` and a `MessagingFunction` into an iteration with two 
> CoGroup operators working on the set of messages and edges. Graph algorithms 
> can profit from implying an order on the edges or messages based on their 
> value and/or the vertex ID. This can be implemented easily making use of 
> secondary sorts. I would suggest extending the `VertexCentricIteration` to 
> allow to specify these kind of orderings optionally.
> For example, this comes handy when it is necessary to find the edges with the 
> minimum or maximum value or the algorithm requires to pick the edge with 
> lower vertex ID for edges with equal value. Similar use cases might be found 
> for orders on the messages. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1522) Add tests for the library methods and examples

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324166#comment-14324166
 ] 

ASF GitHub Bot commented on FLINK-1522:
---

GitHub user andralungu opened a pull request:

https://github.com/apache/flink/pull/414

[FLINK-1522][gelly] Added test for SSSP Example



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andralungu/flink test_sssp

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/414.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #414


commit 9d82699db37f590c157dc2b21b253732bedbc47a
Author: andralungu 
Date:   2015-02-17T13:17:49Z

[FLINK-1522][gelly] Added test for SSSP Example




> Add tests for the library methods and examples
> --
>
> Key: FLINK-1522
> URL: https://issues.apache.org/jira/browse/FLINK-1522
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Reporter: Vasia Kalavri
>Assignee: Andra Lungu
>  Labels: easyfix, test
>
> The current tests in gelly test one method at a time. We should have some 
> tests for complete applications. As a start, we could add one test case per 
> example and this way also make sure that our graph library methods actually 
> give correct results.
> I'm assigning this to [~andralungu] because she has already implemented the 
> test for SSSP, but I will help as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1522][gelly] Added test for SSSP Exampl...

2015-02-17 Thread andralungu

GitHub user andralungu opened a pull request:

https://github.com/apache/flink/pull/414

[FLINK-1522][gelly] Added test for SSSP Example



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andralungu/flink test_sssp

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/414.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #414


commit 9d82699db37f590c157dc2b21b253732bedbc47a
Author: andralungu 
Date:   2015-02-17T13:17:49Z

[FLINK-1522][gelly] Added test for SSSP Example




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1567) Add option to switch between Avro and Kryo serialization for GenericTypes

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324164#comment-14324164
 ] 

ASF GitHub Bot commented on FLINK-1567:
---

GitHub user rmetzger opened a pull request:

https://github.com/apache/flink/pull/413

[FLINK-1567] Add switch to use the AvroSerializer for GenericTypeInfo

Note this is a fix for 0.8 only.
The fix for master is contained in #393

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmetzger/flink avro-switch-0.8

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/413.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #413


commit 41685068df11d18149c329b1773ae7ab40d08e73
Author: Robert Metzger 
Date:   2015-02-17T11:05:05Z

[FLINK-1567] Add switch to use the AvroSerializer for GenericTypeInfo




> Add option to switch between Avro and Kryo serialization for GenericTypes
> -
>
> Key: FLINK-1567
> URL: https://issues.apache.org/jira/browse/FLINK-1567
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 0.8, 0.9
>Reporter: Robert Metzger
>
> Allow users to switch the underlying serializer for GenericTypes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1567] Add switch to use the AvroSeriali...

2015-02-17 Thread rmetzger

GitHub user rmetzger opened a pull request:

https://github.com/apache/flink/pull/413

[FLINK-1567] Add switch to use the AvroSerializer for GenericTypeInfo

Note this is a fix for 0.8 only.
The fix for master is contained in #393

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmetzger/flink avro-switch-0.8

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/413.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #413


commit 41685068df11d18149c329b1773ae7ab40d08e73
Author: Robert Metzger 
Date:   2015-02-17T11:05:05Z

[FLINK-1567] Add switch to use the AvroSerializer for GenericTypeInfo




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1417) Automatically register nested types at Kryo

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324162#comment-14324162
 ] 

ASF GitHub Bot commented on FLINK-1417:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/393#issuecomment-74665100
  
Thank you for reviewing the pull request.

I've addressed your remark, added a fix for another issue (FLINK-1567) and 
now I'll run the performance tests.


> Automatically register nested types at Kryo
> ---
>
> Key: FLINK-1417
> URL: https://issues.apache.org/jira/browse/FLINK-1417
> Project: Flink
>  Issue Type: Improvement
>  Components: Java API, Scala API
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
> Fix For: 0.9
>
>
> Currently, the {{GenericTypeInfo}} registers the class of the type at Kryo. 
> In order to get the best performance, it should recursively walk the classes 
> and make sure that it registered all contained subtypes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1417] Automatically register types with...

2015-02-17 Thread rmetzger

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/393#issuecomment-74665100
  
Thank you for reviewing the pull request.

I've addressed your remark, added a fix for another issue (FLINK-1567) and 
now I'll run the performance tests.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-1351) Inconclusive error when TaskManager cannot connect to JobManager

2015-02-17 Thread Till Rohrmann (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-1351.

Resolution: Cannot Reproduce

Once we can reproduce this error, we'll open the issue again.

> Inconclusive error when TaskManager cannot connect to JobManager
> 
>
> Key: FLINK-1351
> URL: https://issues.apache.org/jira/browse/FLINK-1351
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager, TaskManager
>Affects Versions: 0.9
>Reporter: Stephan Ewen
>Assignee: Till Rohrmann
>
> The taskmanager currently registers at the jobmanager by resolving the akka 
> URL
> {code}
> val jobManager = context.actorSelection(jobManagerAkkaURL)
> {code}
> When the actor lookup fails (actor systems cannot connect), it gives an 
> unspecific timeout message. This is the case when the TaskManager cannot 
> connect to the JobManager.
> This should be fixed to give a conclusive error message.
> I suggest to add a test where TaskManager is started without JobManager actor 
> system being available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-1571) Add failure-case tests for blocking data exchange

2015-02-17 Thread Ufuk Celebi (JIRA)

Ufuk Celebi created FLINK-1571:
--

 Summary: Add failure-case tests for blocking data exchange
 Key: FLINK-1571
 URL: https://issues.apache.org/jira/browse/FLINK-1571
 Project: Flink
  Issue Type: Sub-task
  Components: Distributed Runtime
Reporter: Ufuk Celebi






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-1570) Add failure-case tests for pipelined data exchange

2015-02-17 Thread Ufuk Celebi (JIRA)

Ufuk Celebi created FLINK-1570:
--

 Summary: Add failure-case tests for pipelined data exchange
 Key: FLINK-1570
 URL: https://issues.apache.org/jira/browse/FLINK-1570
 Project: Flink
  Issue Type: Sub-task
  Components: Distributed Runtime
Affects Versions: master
Reporter: Ufuk Celebi






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-1569) Object reuse mode is not working with KeySelector functions.

2015-02-17 Thread Robert Metzger (JIRA)

Robert Metzger created FLINK-1569:
-

 Summary: Object reuse mode is not working with KeySelector 
functions.
 Key: FLINK-1569
 URL: https://issues.apache.org/jira/browse/FLINK-1569
 Project: Flink
  Issue Type: Bug
  Components: Java API
Affects Versions: 0.9
Reporter: Robert Metzger


The following code works correctly when object reuse is switched off.
When switching it on, the results are wrong.
Using a string-based key selection (putting "name") works for both cases.
{code}
@Test
public void testWithAvroGenericSer() throws Exception {
final ExecutionEnvironment env = 
ExecutionEnvironment.getExecutionEnvironment();
//  env.getConfig().enableObjectReuse();
Path in = new Path(inFile.getAbsoluteFile().toURI());

AvroInputFormat users = new AvroInputFormat(in, 
User.class);
DataSet usersDS = env.createInput(users);

DataSet> res = usersDS.groupBy(new 
KeySelector() {
@Override
public String getKey(User value) throws Exception {
return String.valueOf(value.getName());
}
}).reduceGroup(new GroupReduceFunction>() {
@Override
public void reduce(Iterable values, 
Collector> out) throws Exception {
for(User u : values) {
out.collect(new Tuple2(u.getName().toString(), 1));
}
}
});

res.writeAsText(resultPath);
res.print();
env.execute("Avro Key selection");


expected = "(Charlie,1)\n(Alyssa,1)\n";
}
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1552) Allow secondary sorts in Vertex Centric Iteration

2015-02-17 Thread Martin Kiefer (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324142#comment-14324142
 ] 

Martin Kiefer commented on FLINK-1552:
--

The feature would have been great for the algorithm I implemented for Alexander 
Alexandrov's IMPRO3 lecture but now I'm pretty much out of capacities for the 
next two months. If nobody else will takes care of the issue until then, I 
probably will.

> Allow secondary sorts in Vertex Centric Iteration
> -
>
> Key: FLINK-1552
> URL: https://issues.apache.org/jira/browse/FLINK-1552
> Project: Flink
>  Issue Type: Wish
>  Components: Gelly
>Reporter: Martin Kiefer
>Priority: Minor
>
> The `VertexCentricIteration` class holds the logic to transform a 
> `VertexUpdateFunction` and a `MessagingFunction` into an iteration with two 
> CoGroup operators working on the set of messages and edges. Graph algorithms 
> can profit from implying an order on the edges or messages based on their 
> value and/or the vertex ID. This can be implemented easily making use of 
> secondary sorts. I would suggest extending the `VertexCentricIteration` to 
> allow to specify these kind of orderings optionally.
> For example, this comes handy when it is necessary to find the edges with the 
> minimum or maximum value or the algorithm requires to pick the edge with 
> lower vertex ID for edges with equal value. Similar use cases might be found 
> for orders on the messages. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-952) TypeExtractor requires the argument types of the UDF to be identical to the parameter types

2015-02-17 Thread Till Rohrmann (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann reassigned FLINK-952:
---

Assignee: (was: Till Rohrmann)

> TypeExtractor requires the argument types of the UDF to be identical to the 
> parameter types
> ---
>
> Key: FLINK-952
> URL: https://issues.apache.org/jira/browse/FLINK-952
> Project: Flink
>  Issue Type: Bug
>  Components: Local Runtime
>Reporter: Till Rohrmann
>
> The TypeExtractor checks for each operation whether the DataSet element types 
> are valid arguments for the UDF. However, it checks for strict equality 
> instead of a subtype relationship. Thus the following computation would not 
> work even though it should semantically be correct.
> DataSet[B].map(new MapFunction[A,A](){ A map(A x)}) with B being a sub class 
> of A.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: Add auto-parallelism to Jobs (0.8 branch)

2015-02-17 Thread tillrohrmann

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/410#issuecomment-74661555

But currently the system does not support multi-user/multi-job scenarios so
well either. If I'm not mistaken, then the scheduler schedules the tasks
eagerly which means that two jobs could take required slots away from each
other. As a consequence, both will fail if not properly configured.

On Tue, Feb 17, 2015 at 11:01 AM, Fabian Hueske 
wrote:

> Using max parallelism basically prohibits to run more than one program at
> a time. I don't think that would be a good default mode.
>
> â
> Reply to this email directly or view it on GitHub
> .
>

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1566) WindowIntegrationTest fails

2015-02-17 Thread JIRA


[ 
https://issues.apache.org/jira/browse/FLINK-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324135#comment-14324135
 ] 

Márton Balassi commented on FLINK-1566:
---

Most of the time the test passes, sometimes a value is either missing or null.

> WindowIntegrationTest fails
> ---
>
> Key: FLINK-1566
> URL: https://issues.apache.org/jira/browse/FLINK-1566
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Reporter: Till Rohrmann
>
> I'm seeing spurious failures of the {{WindowIntegrationTest}}.
> The error is the following:
> {code}
> java.lang.AssertionError: 
> Expected :[[3], [4], [5], [11], [10], [16]]
> Actual   :[[3], null, [5], [11], [10], [16]]
>  
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:144)
>   at 
> org.apache.flink.streaming.api.invokable.operator.windowing.WindowIntegrationTest.validateOutput(WindowIntegrationTest.java:151)
>   at 
> org.apache.flink.streaming.api.invokable.operator.windowing.WindowIntegrationTest.test(WindowIntegrationTest.java:134)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1561] [build system] Use a fresh fork f...

2015-02-17 Thread uce

Github user uce commented on the pull request:

https://github.com/apache/flink/pull/412#issuecomment-74660789
  
Looks good to me. +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1561) Improve build server robustness by not reusing JVMs in integration tests

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324134#comment-14324134
 ] 

ASF GitHub Bot commented on FLINK-1561:
---

Github user uce commented on the pull request:

https://github.com/apache/flink/pull/412#issuecomment-74660789
  
Looks good to me. +1


> Improve build server robustness by not reusing JVMs in integration tests
> 
>
> Key: FLINK-1561
> URL: https://issues.apache.org/jira/browse/FLINK-1561
> Project: Flink
>  Issue Type: Improvement
>  Components: Build System
>Affects Versions: 0.9
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 0.9
>
>
> Right now, JVM forks are reused across unit and integration tests. While it 
> makes a lot of sense for the very quick unit tests, the integration tests 
> benefit less.
> We also see spurious JVM failures in the integration tests that are most 
> likely related to JVM issues. An example is the trace below:
> {code}
> java.io.IOException: Cannot allocate memory
>   at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>   at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
>   at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>   at sun.nio.ch.IOUtil.write(IOUtil.java:65)
>   at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
>   at 
> org.apache.flink.runtime.io.disk.iomanager.SegmentWriteRequest.write(AsynchronousFileIOChannel.java:270)
>   at 
> org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync$WriterThread.run(IOManagerAsync.java:440)
> {code}
> I think we can improve reliability of Travis tests by not reusing forks for 
> integration tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-1568) Add failure-case tests for data exchange

2015-02-17 Thread Ufuk Celebi (JIRA)

Ufuk Celebi created FLINK-1568:
--

 Summary: Add failure-case tests for data exchange
 Key: FLINK-1568
 URL: https://issues.apache.org/jira/browse/FLINK-1568
 Project: Flink
  Issue Type: Improvement
  Components: Distributed Runtime
Affects Versions: master
Reporter: Ufuk Celebi


There are currently no tests of the distributed runtime data exchange in 
failure scenarios.

The tests need to verify how errors are propagated to the tasks and how they 
affect the state of the exchanged intermediate result.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1515) [Gelly] Enable access to aggregators and broadcast sets in vertex-centric iteration

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324128#comment-14324128
 ] 

ASF GitHub Bot commented on FLINK-1515:
---

Github user martinkiefer commented on the pull request:

https://github.com/apache/flink/pull/402#issuecomment-74660459
  
I could change `PageRank` to read the number of vertices by broadcasting 
the result of `numberofVertices()` instead of providing it in the constructor 
of the algorithm. I guess, this is the use case that comes most naturally.


> [Gelly] Enable access to aggregators and broadcast sets in vertex-centric 
> iteration
> ---
>
> Key: FLINK-1515
> URL: https://issues.apache.org/jira/browse/FLINK-1515
> Project: Flink
>  Issue Type: Improvement
>  Components: Gelly
>Reporter: Vasia Kalavri
>Assignee: Martin Kiefer
>
> Currently, aggregators and broadcast sets cannot be accessed through Gelly's  
> {{runVertexCentricIteration}} method. The functionality is already present in 
> the {{VertexCentricIteration}} and we just need to expose it.
> This could be done like this: We create a method 
> {{createVertexCentricIteration}}, which will return a 
> {{VertexCentricIteration}} object and we change {{runVertexCentricIteration}} 
> to accept this as a parameter (and return the graph after running this 
> iteration).
> The user can configure the {{VertexCentricIteration}} by directly calling the 
> public methods {{registerAggregator}}, {{setName}}, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1515]Splitted runVertexCentricIteration...

2015-02-17 Thread martinkiefer

Github user martinkiefer commented on the pull request:

https://github.com/apache/flink/pull/402#issuecomment-74660459
  
I could change `PageRank` to read the number of vertices by broadcasting 
the result of `numberofVertices()` instead of providing it in the constructor 
of the algorithm. I guess, this is the use case that comes most naturally.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-1554) LocalFlinkMiniCluster cannot start multiple local TaskManagers

2015-02-17 Thread Till Rohrmann (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-1554.

Resolution: Fixed

Fixed via 649f158391aebcefcaabcdaa02d5c8b95be45777

> LocalFlinkMiniCluster cannot start multiple local TaskManagers
> --
>
> Key: FLINK-1554
> URL: https://issues.apache.org/jira/browse/FLINK-1554
> Project: Flink
>  Issue Type: Bug
>Reporter: Till Rohrmann
>
> The {{LocalFlinkMiniCluster}} cannot currently start multiple 
> {{TaskManager}}s in the same {{ActorSystem}}, because it assigns them the 
> same actor name.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-1563) NullPointer during state update call

2015-02-17 Thread Till Rohrmann (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-1563.

Resolution: Fixed

Fixed via ac7bf427900e60abef3ff4344a5873349e6d108e.

> NullPointer during state update call
> 
>
> Key: FLINK-1563
> URL: https://issues.apache.org/jira/browse/FLINK-1563
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 0.9
>Reporter: Stephan Ewen
>Assignee: Till Rohrmann
> Fix For: 0.9
>
>
> The stack trace is below
> {code}
> java.lang.NullPointerException
>   at 
> org.apache.flink.runtime.executiongraph.Execution.sendUpdateTaskRpcCall(Execution.java:779)
>   at 
> org.apache.flink.runtime.executiongraph.Execution.sendPartitionInfos(Execution.java:620)
>   at 
> org.apache.flink.runtime.executiongraph.Execution.switchToRunning(Execution.java:690)
>   at 
> org.apache.flink.runtime.executiongraph.Execution.access$100(Execution.java:91)
>   at 
> org.apache.flink.runtime.executiongraph.Execution$2.onComplete(Execution.java:339)
>   at akka.dispatch.OnComplete.internal(Future.scala:247)
>   at akka.dispatch.OnComplete.internal(Future.scala:244)
>   at akka.dispatch.japi$CallbackBridge.apply(Future.scala:174)
>   at akka.dispatch.japi$CallbackBridge.apply(Future.scala:171)
>   at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
>   at 
> scala.concurrent.impl.ExecutionContextImpl$$anon$3.exec(ExecutionContextImpl.scala:107)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-1549) Add proper exception handling for YarnJobManager

2015-02-17 Thread Till Rohrmann (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-1549.

Resolution: Fixed

Fixed via 447ce0aa60358212a9716f6b4319f7e9566d6eaa

> Add proper exception handling for YarnJobManager
> 
>
> Key: FLINK-1549
> URL: https://issues.apache.org/jira/browse/FLINK-1549
> Project: Flink
>  Issue Type: Bug
>Reporter: Till Rohrmann
>
> The YarnJobManager actor thread calls methods which can throw an exception. 
> These exceptions should be caught and properly handled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-1556) JobClient does not wait until a job failed completely if submission exception

2015-02-17 Thread Till Rohrmann (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-1556.

Resolution: Fixed

Fixed via ac7bf427900e60abef3ff4344a5873349e6d108e

> JobClient does not wait until a job failed completely if submission exception
> -
>
> Key: FLINK-1556
> URL: https://issues.apache.org/jira/browse/FLINK-1556
> Project: Flink
>  Issue Type: Bug
>Reporter: Till Rohrmann
>
> If an exception occurs during job submission the {{JobClient}} received a 
> {{SubmissionFailure}}. Upon receiving this message, the {{JobClient}} 
> terminates itself and returns the error to the {{Client}}. This indicates to 
> the user that the job has been completely failed which is not necessarily 
> true. 
> If the user directly after such a failure submits another job, then it might 
> be the case that not all slots of the formerly failed job are returned. This 
> can lead to a {{NoRessourceAvailableException}}.
> We can solve this problem by waiting for the completion of the job failure in 
> the {{JobClient}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1556) JobClient does not wait until a job failed completely if submission exception

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324122#comment-14324122
 ] 

ASF GitHub Bot commented on FLINK-1556:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/406


> JobClient does not wait until a job failed completely if submission exception
> -
>
> Key: FLINK-1556
> URL: https://issues.apache.org/jira/browse/FLINK-1556
> Project: Flink
>  Issue Type: Bug
>Reporter: Till Rohrmann
>
> If an exception occurs during job submission the {{JobClient}} received a 
> {{SubmissionFailure}}. Upon receiving this message, the {{JobClient}} 
> terminates itself and returns the error to the {{Client}}. This indicates to 
> the user that the job has been completely failed which is not necessarily 
> true. 
> If the user directly after such a failure submits another job, then it might 
> be the case that not all slots of the formerly failed job are returned. This 
> can lead to a {{NoRessourceAvailableException}}.
> We can solve this problem by waiting for the completion of the job failure in 
> the {{JobClient}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1549) Add proper exception handling for YarnJobManager

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324120#comment-14324120
 ] 

ASF GitHub Bot commented on FLINK-1549:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/397


> Add proper exception handling for YarnJobManager
> 
>
> Key: FLINK-1549
> URL: https://issues.apache.org/jira/browse/FLINK-1549
> Project: Flink
>  Issue Type: Bug
>Reporter: Till Rohrmann
>
> The YarnJobManager actor thread calls methods which can throw an exception. 
> These exceptions should be caught and properly handled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1554) LocalFlinkMiniCluster cannot start multiple local TaskManagers

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324121#comment-14324121
 ] 

ASF GitHub Bot commented on FLINK-1554:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/403


> LocalFlinkMiniCluster cannot start multiple local TaskManagers
> --
>
> Key: FLINK-1554
> URL: https://issues.apache.org/jira/browse/FLINK-1554
> Project: Flink
>  Issue Type: Bug
>Reporter: Till Rohrmann
>
> The {{LocalFlinkMiniCluster}} cannot currently start multiple 
> {{TaskManager}}s in the same {{ActorSystem}}, because it assigns them the 
> same actor name.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1554] Allows the LocalFlinkMiniCluster ...

2015-02-17 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/403


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1549] Adds proper exception handling to...

2015-02-17 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/397


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1556] Corrects faulty JobClient behavio...

2015-02-17 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/406


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-1567) Add option to switch between Avro and Kryo serialization for GenericTypes

2015-02-17 Thread Robert Metzger (JIRA)

Robert Metzger created FLINK-1567:
-

 Summary: Add option to switch between Avro and Kryo serialization 
for GenericTypes
 Key: FLINK-1567
 URL: https://issues.apache.org/jira/browse/FLINK-1567
 Project: Flink
  Issue Type: Improvement
Affects Versions: 0.8, 0.9
Reporter: Robert Metzger


Allow users to switch the underlying serializer for GenericTypes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: Add auto-parallelism to Jobs (0.8 branch)

2015-02-17 Thread uce

Github user uce commented on the pull request:

https://github.com/apache/flink/pull/410#issuecomment-74651790
  
+1

I think setup via `ExecutionConfig` is the way to go.

I agree with @rmetzger that we should merge it to master as well. The 
important thing is to document it though. :-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-1566) WindowIntegrationTest fails

2015-02-17 Thread Till Rohrmann (JIRA)

Till Rohrmann created FLINK-1566:


 Summary: WindowIntegrationTest fails
 Key: FLINK-1566
 URL: https://issues.apache.org/jira/browse/FLINK-1566
 Project: Flink
  Issue Type: Bug
  Components: Streaming
Reporter: Till Rohrmann


I'm seeing spurious failures of the {{WindowIntegrationTest}}.

The error is the following:

{code}
java.lang.AssertionError: 
Expected :[[3], [4], [5], [11], [10], [16]]
Actual   :[[3], null, [5], [11], [10], [16]]
 
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.flink.streaming.api.invokable.operator.windowing.WindowIntegrationTest.validateOutput(WindowIntegrationTest.java:151)
at 
org.apache.flink.streaming.api.invokable.operator.windowing.WindowIntegrationTest.test(WindowIntegrationTest.java:134)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1563) NullPointer during state update call

2015-02-17 Thread Till Rohrmann (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324057#comment-14324057
 ] 

Till Rohrmann commented on FLINK-1563:
--

Will be fixed as part of FLINK-1556.

> NullPointer during state update call
> 
>
> Key: FLINK-1563
> URL: https://issues.apache.org/jira/browse/FLINK-1563
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 0.9
>Reporter: Stephan Ewen
>Assignee: Till Rohrmann
> Fix For: 0.9
>
>
> The stack trace is below
> {code}
> java.lang.NullPointerException
>   at 
> org.apache.flink.runtime.executiongraph.Execution.sendUpdateTaskRpcCall(Execution.java:779)
>   at 
> org.apache.flink.runtime.executiongraph.Execution.sendPartitionInfos(Execution.java:620)
>   at 
> org.apache.flink.runtime.executiongraph.Execution.switchToRunning(Execution.java:690)
>   at 
> org.apache.flink.runtime.executiongraph.Execution.access$100(Execution.java:91)
>   at 
> org.apache.flink.runtime.executiongraph.Execution$2.onComplete(Execution.java:339)
>   at akka.dispatch.OnComplete.internal(Future.scala:247)
>   at akka.dispatch.OnComplete.internal(Future.scala:244)
>   at akka.dispatch.japi$CallbackBridge.apply(Future.scala:174)
>   at akka.dispatch.japi$CallbackBridge.apply(Future.scala:171)
>   at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
>   at 
> scala.concurrent.impl.ExecutionContextImpl$$anon$3.exec(ExecutionContextImpl.scala:107)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1521] Chained operators respect reuse

2015-02-17 Thread fhueske

Github user fhueske commented on the pull request:

https://github.com/apache/flink/pull/392#issuecomment-74650731
  
+1
I opened FLINK-1565 for documenting object reuse behavior.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-1565) Document object reuse behavior

2015-02-17 Thread Fabian Hueske (JIRA)

Fabian Hueske created FLINK-1565:


 Summary: Document object reuse behavior
 Key: FLINK-1565
 URL: https://issues.apache.org/jira/browse/FLINK-1565
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Fabian Hueske
 Fix For: 0.9


The documentation needs to be extended and describe the object reuse behavior 
of Flink and its implications for how to implement functions.

The documentation must at least cover the default reuse mode:
* new objects through iterators and in reduce functions
* chaining behavior (objects are passed on to the next function which might 
modify it)

Optionally, the documentation could describe the object reuse switch introduced 
by FLINK-1137.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1521) Some Chained Drivers do not respect object-reuse/non-reuse flag

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324052#comment-14324052
 ] 

ASF GitHub Bot commented on FLINK-1521:
---

Github user fhueske commented on the pull request:

https://github.com/apache/flink/pull/392#issuecomment-74650731
  
+1
I opened FLINK-1565 for documenting object reuse behavior.


> Some Chained Drivers do not respect object-reuse/non-reuse flag
> ---
>
> Key: FLINK-1521
> URL: https://issues.apache.org/jira/browse/FLINK-1521
> Project: Flink
>  Issue Type: Bug
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1564) Make sure BLOB client downloads files only once

2015-02-17 Thread Stephan Ewen (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324045#comment-14324045
 ] 

Stephan Ewen commented on FLINK-1564:
-

This is currently not happening, since the LibraryCache guards its requests 
with an exclusive lock.

> Make sure BLOB client downloads files only once
> ---
>
> Key: FLINK-1564
> URL: https://issues.apache.org/jira/browse/FLINK-1564
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 0.8, 0.9
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Minor
> Fix For: 0.8, 0.9
>
>
> Currently, the BLOB manager may download files multiple times, if concurrent 
> requests are issued. We should prevent that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-1564) Make sure BLOB client downloads files only once

2015-02-17 Thread Stephan Ewen (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen updated FLINK-1564:

Priority: Minor  (was: Major)

> Make sure BLOB client downloads files only once
> ---
>
> Key: FLINK-1564
> URL: https://issues.apache.org/jira/browse/FLINK-1564
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 0.8, 0.9
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Minor
> Fix For: 0.8, 0.9
>
>
> Currently, the BLOB manager may download files multiple times, if concurrent 
> requests are issued. We should prevent that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-1564) Make sure BLOB client downloads files only once

2015-02-17 Thread Stephan Ewen (JIRA)

Stephan Ewen created FLINK-1564:
---

 Summary: Make sure BLOB client downloads files only once
 Key: FLINK-1564
 URL: https://issues.apache.org/jira/browse/FLINK-1564
 Project: Flink
  Issue Type: Bug
  Components: Distributed Runtime
Affects Versions: 0.8, 0.9
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 0.9, 0.8


Currently, the BLOB manager may download files multiple times, if concurrent 
requests are issued. We should prevent that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-1563) NullPointer during state update call

2015-02-17 Thread Stephan Ewen (JIRA)

Stephan Ewen created FLINK-1563:
---

 Summary: NullPointer during state update call
 Key: FLINK-1563
 URL: https://issues.apache.org/jira/browse/FLINK-1563
 Project: Flink
  Issue Type: Bug
  Components: JobManager
Affects Versions: 0.9
Reporter: Stephan Ewen
Assignee: Till Rohrmann
 Fix For: 0.9


The stack trace is below
{code}
java.lang.NullPointerException
at 
org.apache.flink.runtime.executiongraph.Execution.sendUpdateTaskRpcCall(Execution.java:779)
at 
org.apache.flink.runtime.executiongraph.Execution.sendPartitionInfos(Execution.java:620)
at 
org.apache.flink.runtime.executiongraph.Execution.switchToRunning(Execution.java:690)
at 
org.apache.flink.runtime.executiongraph.Execution.access$100(Execution.java:91)
at 
org.apache.flink.runtime.executiongraph.Execution$2.onComplete(Execution.java:339)
at akka.dispatch.OnComplete.internal(Future.scala:247)
at akka.dispatch.OnComplete.internal(Future.scala:244)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:174)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:171)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at 
scala.concurrent.impl.ExecutionContextImpl$$anon$3.exec(ExecutionContextImpl.scala:107)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1561] [build system] Use a fresh fork f...

2015-02-17 Thread StephanEwen

GitHub user StephanEwen opened a pull request:

https://github.com/apache/flink/pull/412

[FLINK-1561] [build system] Use a fresh fork for each integration test run

Helps the build stability on Travis for integration tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StephanEwen/incubator-flink flink1561

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/412.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #412


commit a8c99d5a8fc6a700a670da5443d9a808ac5eaaa0
Author: Stephan Ewen 
Date:   2015-02-17T10:10:48Z

[FLINK-1561] [build system] Use a fresh fork for each integration test run




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1561) Improve build server robustness by not reusing JVMs in integration tests

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324027#comment-14324027
 ] 

ASF GitHub Bot commented on FLINK-1561:
---

GitHub user StephanEwen opened a pull request:

https://github.com/apache/flink/pull/412

[FLINK-1561] [build system] Use a fresh fork for each integration test run

Helps the build stability on Travis for integration tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StephanEwen/incubator-flink flink1561

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/412.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #412


commit a8c99d5a8fc6a700a670da5443d9a808ac5eaaa0
Author: Stephan Ewen 
Date:   2015-02-17T10:10:48Z

[FLINK-1561] [build system] Use a fresh fork for each integration test run




> Improve build server robustness by not reusing JVMs in integration tests
> 
>
> Key: FLINK-1561
> URL: https://issues.apache.org/jira/browse/FLINK-1561
> Project: Flink
>  Issue Type: Improvement
>  Components: Build System
>Affects Versions: 0.9
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 0.9
>
>
> Right now, JVM forks are reused across unit and integration tests. While it 
> makes a lot of sense for the very quick unit tests, the integration tests 
> benefit less.
> We also see spurious JVM failures in the integration tests that are most 
> likely related to JVM issues. An example is the trace below:
> {code}
> java.io.IOException: Cannot allocate memory
>   at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>   at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
>   at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>   at sun.nio.ch.IOUtil.write(IOUtil.java:65)
>   at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
>   at 
> org.apache.flink.runtime.io.disk.iomanager.SegmentWriteRequest.write(AsynchronousFileIOChannel.java:270)
>   at 
> org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync$WriterThread.run(IOManagerAsync.java:440)
> {code}
> I think we can improve reliability of Travis tests by not reusing forks for 
> integration tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-1562) Introduce retries for fetching data from the BLOB manager

2015-02-17 Thread Stephan Ewen (JIRA)

Stephan Ewen created FLINK-1562:
---

 Summary: Introduce retries for fetching data from the BLOB manager
 Key: FLINK-1562
 URL: https://issues.apache.org/jira/browse/FLINK-1562
 Project: Flink
  Issue Type: Bug
  Components: Distributed Runtime
Affects Versions: 0.8, 0.9
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 0.9, 0.8


Fetch requests from the blob manager sometimes fail with connection resets. The 
blob client should retry on such a failure to compensate for spurious fetch 
failures.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-1561) Improve build server robustness by not reusing JVMs in integration tests

2015-02-17 Thread Stephan Ewen (JIRA)

Stephan Ewen created FLINK-1561:
---

 Summary: Improve build server robustness by not reusing JVMs in 
integration tests
 Key: FLINK-1561
 URL: https://issues.apache.org/jira/browse/FLINK-1561
 Project: Flink
  Issue Type: Improvement
  Components: Build System
Affects Versions: 0.9
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 0.9


Right now, JVM forks are reused across unit and integration tests. While it 
makes a lot of sense for the very quick unit tests, the integration tests 
benefit less.

We also see spurious JVM failures in the integration tests that are most likely 
related to JVM issues. An example is the trace below:
{code}
java.io.IOException: Cannot allocate memory
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
at 
org.apache.flink.runtime.io.disk.iomanager.SegmentWriteRequest.write(AsynchronousFileIOChannel.java:270)
at 
org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync$WriterThread.run(IOManagerAsync.java:440)
{code}

I think we can improve reliability of Travis tests by not reusing forks for 
integration tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: Add auto-parallelism to Jobs (0.8 branch)

2015-02-17 Thread StephanEwen

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/410#issuecomment-74643785
  
I agree with Fabian that it is not a good default behavior to grab 
everything that is possible.
It should be an explicit request by the user. For YARN single job sessions, 
we can make this the default, otherwise it is not very friendly.

`getNumberOfAvailableSlots()` changes very fast during multi user 
operation. Dusing single user operation between jobs (where I see the auto 
parallelism useful), it is the same as `getTotalNumberOfSlots()`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Add auto-parallelism to Jobs (0.8 branch)

2015-02-17 Thread fhueske

Github user fhueske commented on the pull request:

https://github.com/apache/flink/pull/410#issuecomment-74643360
  
Using max parallelism basically prohibits to run more than one program at a 
time. I don't think that would be a good default mode. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Add auto-parallelism to Jobs (0.8 branch)

2015-02-17 Thread mxm

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/410#discussion_r24804242
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/JobManager.java 
---
@@ -374,6 +375,8 @@ public JobSubmissionResult submitJob(JobGraph job) 
throws IOException {
LOG.debug(String.format("Running master 
initialization of job %s (%s)", job.getJobID(), job.getName()));
}
 
+   final int numSlots = scheduler.getTotalNumberOfSlots();
--- End diff --

Shouldn't this be set to `getNumberOfAvailableSlots()` for the 
PARALLELISM_AUTO_MAX case?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Add auto-parallelism to Jobs (0.8 branch)

2015-02-17 Thread mxm

Github user mxm commented on the pull request:

https://github.com/apache/flink/pull/410#issuecomment-74642763
  
Right now, the user has to set the parallelism to 
`ExecutionConfig.PARALLELISM_AUTO_MAX`. Why not use all available task slots by 
default? I understand, that we shouldn't simply grab all resources but the auto 
parallelism will only grab the resources which were already granted to Flink.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (FLINK-1557) Move WebFrontend out of JobManager Actor

2015-02-17 Thread Stephan Ewen (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-1557.
-
Resolution: Fixed

Fixed via c1e326707a6c5b3cc550945f010eb4cabea24ad3

> Move WebFrontend out of JobManager Actor
> 
>
> Key: FLINK-1557
> URL: https://issues.apache.org/jira/browse/FLINK-1557
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager
>Affects Versions: 0.9
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 0.9
>
>
> Currently, the Webfrontent is started inside the JobManager actor.
> I propose to move it out of the actor, since it is a heavy component that is 
> independent of the actor and whose failure cannot be controlled by the job 
> manager.
> The change is fairly straightforward, since the web frontend only 
> communicates with the jobmanager through actor messages.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1557) Move WebFrontend out of JobManager Actor

2015-02-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324000#comment-14324000
 ] 

ASF GitHub Bot commented on FLINK-1557:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/409


> Move WebFrontend out of JobManager Actor
> 
>
> Key: FLINK-1557
> URL: https://issues.apache.org/jira/browse/FLINK-1557
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager
>Affects Versions: 0.9
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 0.9
>
>
> Currently, the Webfrontent is started inside the JobManager actor.
> I propose to move it out of the actor, since it is a heavy component that is 
> independent of the actor and whose failure cannot be controlled by the job 
> manager.
> The change is fairly straightforward, since the web frontend only 
> communicates with the jobmanager through actor messages.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

1 2 >

1 - 100 of 110 matches

Mail list logo