[jira] [Updated] (FLINK-1697) Add alternating least squares algorithm for matrix factorization to ML library
[ https://issues.apache.org/jira/browse/FLINK-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-1697: - Labels: ML (was: ) Add alternating least squares algorithm for matrix factorization to ML library -- Key: FLINK-1697 URL: https://issues.apache.org/jira/browse/FLINK-1697 Project: Flink Issue Type: Improvement Reporter: Till Rohrmann Assignee: Till Rohrmann Labels: ML Add alternating least squares algorithm for matrix factorization to ML library -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1679) Document how degree of parallelism / parallelism / slots are connected to each other
[ https://issues.apache.org/jira/browse/FLINK-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364785#comment-14364785 ] ASF GitHub Bot commented on FLINK-1679: --- Github user uce commented on the pull request: https://github.com/apache/flink/pull/488#issuecomment-82212002 Very nice and exhaustive change. One question though: do we want to just rename configuration parameters and methods like that or should we keep the old config keys (parallelization.degree.default) and the old config methods (setDegreeOfParallelism) as well and deprecate them first? I think I am in favour of option 2. Document how degree of parallelism / parallelism / slots are connected to each other --- Key: FLINK-1679 URL: https://issues.apache.org/jira/browse/FLINK-1679 Project: Flink Issue Type: Task Components: Documentation Affects Versions: 0.9 Reporter: Robert Metzger Assignee: Maximilian Michels I see too many users being confused about properly setting up Flink with respect to parallelism. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1679] rename degree of parallelism to p...
Github user uce commented on the pull request: https://github.com/apache/flink/pull/488#issuecomment-82212002 Very nice and exhaustive change. One question though: do we want to just rename configuration parameters and methods like that or should we keep the old config keys (parallelization.degree.default) and the old config methods (setDegreeOfParallelism) as well and deprecate them first? I think I am in favour of option 2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1350] [runtime] Add blocking result par...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/471#issuecomment-82554365 Look good, nice test coverage and fits very well with the recent execution mode changes. Only downside: This pull request does contains many cases where only whitespaces where changed, or the order of imports. This is not really helping the reviews and clutters the change history of files and hinders git blame. I know that some IDEs to that automatically, but it would be nice to deactivate that in the future. Why the hell some IDEs think that adding 100 changed lines to a file (where only one typo was fixed) would help is beyond me... Minor comment: The JoinDeadlock test creates a dedicated input format that does not do anything different then `ExecutionEnvironment#generateSequence` would do, if the source parallelism was set to one. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1350) Add blocking intermediate result partitions
[ https://issues.apache.org/jira/browse/FLINK-1350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365854#comment-14365854 ] ASF GitHub Bot commented on FLINK-1350: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/471#issuecomment-82554365 Look good, nice test coverage and fits very well with the recent execution mode changes. Only downside: This pull request does contains many cases where only whitespaces where changed, or the order of imports. This is not really helping the reviews and clutters the change history of files and hinders git blame. I know that some IDEs to that automatically, but it would be nice to deactivate that in the future. Why the hell some IDEs think that adding 100 changed lines to a file (where only one typo was fixed) would help is beyond me... Minor comment: The JoinDeadlock test creates a dedicated input format that does not do anything different then `ExecutionEnvironment#generateSequence` would do, if the source parallelism was set to one. Add blocking intermediate result partitions --- Key: FLINK-1350 URL: https://issues.apache.org/jira/browse/FLINK-1350 Project: Flink Issue Type: Improvement Components: Distributed Runtime Reporter: Ufuk Celebi Assignee: Ufuk Celebi The current state of runtime support for intermediate results (see https://github.com/apache/incubator-flink/pull/254 and FLINK-986) only supports pipelined intermediate results (with back pressure), which are consumed as they are being produced. The next variant we need to support are blocking intermediate results (without back pressure), which are fully produced before being consumed. This is for example desirable in situations, where we currently may run into deadlocks when running pipelined. I will start working on this on top of my pending pull request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1714]Fix the bug of logger class loader
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/490 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1650) Suppress Akka's Netty Shutdown Errors through the log config
[ https://issues.apache.org/jira/browse/FLINK-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365953#comment-14365953 ] Robert Metzger commented on FLINK-1650: --- I've asked on the Akka userlist regarding this issue. Suppress Akka's Netty Shutdown Errors through the log config Key: FLINK-1650 URL: https://issues.apache.org/jira/browse/FLINK-1650 Project: Flink Issue Type: Bug Components: other Affects Versions: 0.9 Reporter: Stephan Ewen Assignee: Stephan Ewen Fix For: 0.9 I suggest to set the logging for `org.jboss.netty.channel.DefaultChannelPipeline` to error, in order to get rid of the misleading stack trace caused by an akka/netty hickup on shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-441] Rename Pact* and Nephele* classes ...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/492#issuecomment-82577264 We can do that, I am okay with this. So far, all renaming was strictly internal. The renaming of the maven project may affect some people who have added it as a (redundant) dependency. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: Fix issue where Windows paths were not recogni...
Github user balidani commented on the pull request: https://github.com/apache/flink/pull/491#issuecomment-82586976 Yeah, that makes sense, I added two more assertions. Sorry, I forgot to check checkstyle before committing. Should I squash these commits in the end? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (FLINK-1467) Job deployment fails with NPE on JobManager, if TMs did not start properly
[
https://issues.apache.org/jira/browse/FLINK-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen resolved FLINK-1467.
-
Resolution: Duplicate
Subsumed by [FLINK-1580]
Job deployment fails with NPE on JobManager, if TMs did not start properly
--
Key: FLINK-1467
URL: https://issues.apache.org/jira/browse/FLINK-1467
Project: Flink
Issue Type: Bug
Components: JobManager
Reporter: Robert Metzger
I have a Flink cluster started where all TaskManagers died
(misconfiguration). The JobManager needs more than 200 seconds to realize
that (on the TaskManagers overview, you see timeouts 200). When submitting
a job, you'll get the following exception:
{code}
org.apache.flink.client.program.ProgramInvocationException: The program
execution failed: java.lang.Exception: Failed to deploy the task CHAIN
DataSource (Generator: class io.airlift.tpch.NationGenerator) - Map (Map at
writeAsFormattedText(DataSet.java:1132)) (1/1) - execution #0 to slot SubSlot
0 (f8d11026ec5a11f0b273184c74ec4f29 (0) - ALLOCATED/ALIVE):
java.lang.NullPointerException
at
org.apache.flink.runtime.taskmanager.TaskManager.org$apache$flink$runtime$taskmanager$TaskManager$$submitTask(TaskManager.scala:346)
at
org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$receiveWithLogMessages$1.applyOrElse(TaskManager.scala:248)
at
scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at
org.apache.flink.yarn.YarnTaskManager$$anonfun$receiveYarnMessages$1.applyOrElse(YarnTaskManager.scala:32)
at scala.PartialFunction$OrElse.apply(PartialFunction.scala:162)
at
org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMessages.scala:41)
at
org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMessages.scala:27)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
at
org.apache.flink.runtime.ActorLogMessages$$anon$1.applyOrElse(ActorLogMessages.scala:27)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at
org.apache.flink.runtime.taskmanager.TaskManager.aroundReceive(TaskManager.scala:78)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
at akka.dispatch.Mailbox.run(Mailbox.scala:221)
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
at
org.apache.flink.runtime.executiongraph.Execution$2.onComplete(Execution.java:311)
at akka.dispatch.OnComplete.internal(Future.scala:247)
at akka.dispatch.OnComplete.internal(Future.scala:244)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:174)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:171)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at
scala.concurrent.impl.ExecutionContextImpl$$anon$3.exec(ExecutionContextImpl.scala:107)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
at org.apache.flink.client.program.Client.run(Client.java:345)
at org.apache.flink.client.program.Client.run(Client.java:304)
at org.apache.flink.client.program.Client.run(Client.java:298)
at
org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:55)
at flink.generators.programs.TPCHGenerator.main(TPCHGenerator.java:80)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
at
[GitHub] flink pull request: [FLINK-1514][Gelly] Add a Gather-Sum-Apply ite...
Github user vasia commented on the pull request: https://github.com/apache/flink/pull/408#issuecomment-82588861 Hi @balidani, thanks a lot for the changes! I'll try to run some tests on a cluster soon and test the new version. I have a question about the graph coloring example. As far as I understand, in this algorithm, colors are represented by numbers (vertex values) and the goal is to color the graph with the minimum number of colors, so that no 2 neighboring vertices have the same color. So, in each superstep, each vertex gathers the colors of its neighbors in a set and then assigns itself the minimum color that isn't in this set. However, it seems to me that in your implementation, you're only propagating the minimum neighbor value, similar to what you would do in connected components. Is there anything I'm missing here? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1514) [Gelly] Add a Gather-Sum-Apply iteration method
[ https://issues.apache.org/jira/browse/FLINK-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366041#comment-14366041 ] ASF GitHub Bot commented on FLINK-1514: --- Github user vasia commented on the pull request: https://github.com/apache/flink/pull/408#issuecomment-82588861 Hi @balidani, thanks a lot for the changes! I'll try to run some tests on a cluster soon and test the new version. I have a question about the graph coloring example. As far as I understand, in this algorithm, colors are represented by numbers (vertex values) and the goal is to color the graph with the minimum number of colors, so that no 2 neighboring vertices have the same color. So, in each superstep, each vertex gathers the colors of its neighbors in a set and then assigns itself the minimum color that isn't in this set. However, it seems to me that in your implementation, you're only propagating the minimum neighbor value, similar to what you would do in connected components. Is there anything I'm missing here? Thanks! [Gelly] Add a Gather-Sum-Apply iteration method --- Key: FLINK-1514 URL: https://issues.apache.org/jira/browse/FLINK-1514 Project: Flink Issue Type: New Feature Components: Gelly Affects Versions: 0.9 Reporter: Vasia Kalavri Assignee: Daniel Bali This will be a method that implements the GAS computation model, but without the scatter step. The phases can be mapped into the following steps inside a delta iteration: gather: a map on each srcVertex, edge, trgVertex that produces a partial value sum: a reduce that combines the partial values apply: join with vertex set to update the vertex values using the results of sum and the previous state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: Fix issue where Windows paths were not recogni...
Github user balidani commented on a diff in the pull request:
https://github.com/apache/flink/pull/491#discussion_r26613854
--- Diff: flink-core/src/test/java/org/apache/flink/core/fs/PathTest.java
---
@@ -63,6 +63,15 @@ public void testPathFromString() {
assertEquals(/my/path, p.toUri().getPath());
assertEquals(file, p.toUri().getScheme());
+ p = new Path(C:/my/windows/path);
+ assertEquals(/C:/my/windows/path, p.toUri().getPath());
+
+ p = new Path(file:/C:/my/windows/path);
+ assertEquals(/C:/my/windows/path, p.toUri().getPath());
+
+ p = new Path(C:);
+ assertEquals(/C:, p.toUri().getPath());
--- End diff --
Well the test passes, not sure if it's semantically correct though. If it's
not correct, then `Path` should be changed I guess?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-1713) Add support for blocking data exchange in closed loop iterations
[ https://issues.apache.org/jira/browse/FLINK-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366140#comment-14366140 ] Henry Saputra commented on FLINK-1713: -- HI [~uce], could you elaborate more on what do you mean by the closed loops? Add support for blocking data exchange in closed loop iterations Key: FLINK-1713 URL: https://issues.apache.org/jira/browse/FLINK-1713 Project: Flink Issue Type: Improvement Components: Distributed Runtime Affects Versions: master Reporter: Ufuk Celebi Priority: Minor The way that blocking intermediate results are currently managed prohibits them from being used inside of closed loops. A blocking result has to be fully produced before its receivers are deployed and there is no notion of single iterations etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1695] Kick off of Flink's machine learn...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/479#issuecomment-82539185 Looks nice, +1 to add this. Also, +1 to add the JIRA issue number to the commits and squash some basic cleanup commits. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: Remove -j and -a parameters which seemed no lo...
Github user hsaputra commented on the pull request: https://github.com/apache/flink/pull/482#issuecomment-82605275 This is merged by @StephanEwen with http://git-wip-us.apache.org/repos/asf/flink/commit/72ca014e. Thanks Stephan. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: Remove -j and -a parameters which seemed no lo...
Github user hsaputra closed the pull request at: https://github.com/apache/flink/pull/482 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1679] rename degree of parallelism to p...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/488#issuecomment-82219329 I think this is very API breaking - in my opinion, we should keep the old methods where it breaks the API, or at least keep them for one more version (deprecated). That does not prevent us from merging the fixes to comments and docs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1679) Document how degree of parallelism / parallelism / slots are connected to each other
[ https://issues.apache.org/jira/browse/FLINK-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364796#comment-14364796 ] ASF GitHub Bot commented on FLINK-1679: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/488#issuecomment-82219329 I think this is very API breaking - in my opinion, we should keep the old methods where it breaks the API, or at least keep them for one more version (deprecated). That does not prevent us from merging the fixes to comments and docs. Document how degree of parallelism / parallelism / slots are connected to each other --- Key: FLINK-1679 URL: https://issues.apache.org/jira/browse/FLINK-1679 Project: Flink Issue Type: Task Components: Documentation Affects Versions: 0.9 Reporter: Robert Metzger Assignee: Maximilian Michels I see too many users being confused about properly setting up Flink with respect to parallelism. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-1711) Replace all usages off commons.Validate with guava.check
[
https://issues.apache.org/jira/browse/FLINK-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen updated FLINK-1711:
Labels: easyfix starter (was: )
Replace all usages off commons.Validate with guava.check
Key: FLINK-1711
URL: https://issues.apache.org/jira/browse/FLINK-1711
Project: Flink
Issue Type: Improvement
Affects Versions: 0.9
Reporter: Stephan Ewen
Priority: Minor
Labels: easyfix, starter
Fix For: 0.9
Per discussion on the mailing list, we decided to increase homogeneity. One
part is to consistently use the Guava methods {{checkNotNull}} and
{{checkArgument}}, rather than Apache Commons Lang3 {{Validate}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1701) Serialization problems with TestStreamEnvironment
[ https://issues.apache.org/jira/browse/FLINK-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364833#comment-14364833 ] Péter Szabó commented on FLINK-1701: It solved the problem, thank you very much, Stephan! The anonymous inner class was the bad guy here. Peter Serialization problems with TestStreamEnvironment - Key: FLINK-1701 URL: https://issues.apache.org/jira/browse/FLINK-1701 Project: Flink Issue Type: Bug Components: Streaming Reporter: Péter Szabó When mapWindowing a stream of tuples of a custom class with TestStreamEnvironment, serialization of StreamDiscretizer fails in StreamConfig. [Example here|https://github.com/mbalassi/flink/blob/6682762bca7b08cb87c68ee4b2ee592a5da3479e/flink-staging/flink-streaming/flink-streaming-core/src/test/java/org/apache/flink/streaming/api/DeltaWithCustomClassTest.java] The same example runs with StreamExecutionEnvironment without any problem. I'm wondering what is the difference. StackTrace: java.lang.RuntimeException: Cannot serialize invokable object class org.apache.flink.streaming.api.invokable.operator.windowing.StreamDiscretizer at org.apache.flink.streaming.api.StreamConfig.setUserInvokable(StreamConfig.java:173) at org.apache.flink.streaming.api.StreamingJobGraphGenerator.setVertexConfig(StreamingJobGraphGenerator.java:213) at org.apache.flink.streaming.api.StreamingJobGraphGenerator.createChain(StreamingJobGraphGenerator.java:129) at org.apache.flink.streaming.api.StreamingJobGraphGenerator.createChain(StreamingJobGraphGenerator.java:121) at org.apache.flink.streaming.api.StreamingJobGraphGenerator.createChain(StreamingJobGraphGenerator.java:116) at org.apache.flink.streaming.api.StreamingJobGraphGenerator.createChain(StreamingJobGraphGenerator.java:121) at org.apache.flink.streaming.api.StreamingJobGraphGenerator.setChaining(StreamingJobGraphGenerator.java:95) at org.apache.flink.streaming.api.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:86) at org.apache.flink.streaming.api.StreamGraph.getJobGraph(StreamGraph.java:510) at org.apache.flink.streaming.util.TestStreamEnvironment.execute(TestStreamEnvironment.java:59) at org.apache.flink.streaming.util.TestStreamEnvironment.execute(TestStreamEnvironment.java:54) at org.apache.flink.streaming.api.ComplexIntegrationTest.complexIntegrationTest7(ComplexIntegrationTest.java:433) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runner.JUnitCore.run(JUnitCore.java:160) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134) Caused by: org.apache.commons.lang3.SerializationException: java.io.NotSerializableException: org.apache.flink.streaming.api.ComplexIntegrationTest at org.apache.commons.lang3.SerializationUtils.serialize(SerializationUtils.java:157) at org.apache.commons.lang3.SerializationUtils.serialize(SerializationUtils.java:179) at
[jira] [Created] (FLINK-1712) Restructure Maven Projects
Stephan Ewen created FLINK-1712: --- Summary: Restructure Maven Projects Key: FLINK-1712 URL: https://issues.apache.org/jira/browse/FLINK-1712 Project: Flink Issue Type: Improvement Affects Versions: 0.9 Reporter: Stephan Ewen Priority: Minor Project consolidation - flink-hadoop (shaded fat jar) - Core (Core and Java) - (core-scala) - Streaming (core + java) - (streaming-scala) - Runtime - Optimizer (may also be merged with Client) - Client (or Client + Optimizer) - Examples (Java + Scala + Streaming Java + Streaming Scala) - Tests (test-utils (compile) and tests (test)) - Quickstarts - Quickstart Java - Quickstart Scala - connectors / Input/Output Formats - Avro - HBase - HadoopCompartibility - HCatalogue - JDBC - kafka - rabbit - ... - staging - Gelly - ML - spargel (deprecated) - expression API - contrib - yarn - dist - yarn tests - java 8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1615] [java api] SimpleTweetInputFormat
Github user Elbehery commented on the pull request: https://github.com/apache/flink/pull/442#issuecomment-82254957 I have revised the commit .. All the files has the license header, except a resource file which contains 4 tweets for testing purpose .. Could this be the problem ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: modify the getlogger of the class
GitHub user matadorhong opened a pull request: https://github.com/apache/flink/pull/490 modify the getlogger of the class You can merge this pull request into a Git repository by running: $ git pull https://github.com/matadorhong/flink logclass_317 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/490.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #490 commit c23fa92f32758d57863a4d19542f4d20546267c5 Author: Hong Sibao hongsi...@huawei.com Date: 2015-03-17T10:46:22Z modify the getlogger of the class --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (FLINK-1712) Restructure Maven Projects
[ https://issues.apache.org/jira/browse/FLINK-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-1712: -- Component/s: Build System Restructure Maven Projects -- Key: FLINK-1712 URL: https://issues.apache.org/jira/browse/FLINK-1712 Project: Flink Issue Type: Improvement Components: Build System Affects Versions: 0.9 Reporter: Stephan Ewen Priority: Minor Project consolidation - flink-hadoop (shaded fat jar) - Core (Core and Java) - (core-scala) - Streaming (core + java) - (streaming-scala) - Runtime - Optimizer (may also be merged with Client) - Client (or Client + Optimizer) - Examples (Java + Scala + Streaming Java + Streaming Scala) - Tests (test-utils (compile) and tests (test)) - Quickstarts - Quickstart Java - Quickstart Scala - connectors / Input/Output Formats - Avro - HBase - HadoopCompartibility - HCatalogue - JDBC - kafka - rabbit - ... - staging - Gelly - ML - spargel (deprecated) - expression API - contrib - yarn - dist - yarn tests - java 8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1615] [java api] SimpleTweetInputFormat
Github user Elbehery commented on the pull request: https://github.com/apache/flink/pull/442#issuecomment-82261434 DONE --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1615) Introduces a new InputFormat for Tweets
[ https://issues.apache.org/jira/browse/FLINK-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364881#comment-14364881 ] ASF GitHub Bot commented on FLINK-1615: --- Github user Elbehery commented on the pull request: https://github.com/apache/flink/pull/442#issuecomment-82261434 DONE Introduces a new InputFormat for Tweets --- Key: FLINK-1615 URL: https://issues.apache.org/jira/browse/FLINK-1615 Project: Flink Issue Type: New Feature Components: flink-contrib Affects Versions: 0.8.1 Reporter: mustafa elbehery Priority: Minor An event-driven parser for Tweets into Java Pojos. It parses all the important part of the tweet into Java objects. Tested on cluster and the performance in pretty well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-1711) Replace all usages off commons.Validate with guava.check
Stephan Ewen created FLINK-1711:
---
Summary: Replace all usages off commons.Validate with guava.check
Key: FLINK-1711
URL: https://issues.apache.org/jira/browse/FLINK-1711
Project: Flink
Issue Type: Improvement
Affects Versions: 0.9
Reporter: Stephan Ewen
Priority: Minor
Fix For: 0.9
Per discussion on the mailing list, we decided to increase homogeneity. One
part is to consistently use the Guava methods {{checkNotNull}} and
{{checkArgument}}, rather than Apache Commons Lang3 {{Validate}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLINK-1671) Add execution modes for programs
[
https://issues.apache.org/jira/browse/FLINK-1671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen updated FLINK-1671:
Labels: starter (was: )
Add execution modes for programs
Key: FLINK-1671
URL: https://issues.apache.org/jira/browse/FLINK-1671
Project: Flink
Issue Type: Bug
Affects Versions: 0.9
Reporter: Stephan Ewen
Assignee: Stephan Ewen
Fix For: 0.9
Currently, there is a single way that programs get executed: Pipelined. With
the new code for batch shuffles (https://github.com/apache/flink/pull/471),
we have much more flexibility and I would like to expose that.
I suggest to add more execution modes that can be chosen on the
`ExecutionEnvironment`:
- {{BATCH}} A mode where every shuffle is executed in a batch way, meaning
preceding operators must be done before successors start. Only for the batch
programs (d'oh).
- {{PIPELINED}} This is the mode corresponding to the current execution
mode. It pipelines where possible and batches, where deadlocks would
otherwise happen. Initially, I would make this the default (be close to the
current behavior). Only available for batch programs.
- {{PIPELINED_WITH_BATCH_FALLBACK}} This would start out with pipelining
shuffles and fall back to batch shuffles upon failure and recovery, or once
it sees that not enough slots are available to bring up all operators at once
(requirement for pipelining).
- {{STREAMING}} This is the default and only way for streaming programs.
All communication is pipelined, and the special streaming checkpointing code
is activated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLINK-1671) Add execution modes for programs
[
https://issues.apache.org/jira/browse/FLINK-1671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen updated FLINK-1671:
Labels: (was: easyfix starter)
Add execution modes for programs
Key: FLINK-1671
URL: https://issues.apache.org/jira/browse/FLINK-1671
Project: Flink
Issue Type: Bug
Affects Versions: 0.9
Reporter: Stephan Ewen
Assignee: Stephan Ewen
Fix For: 0.9
Currently, there is a single way that programs get executed: Pipelined. With
the new code for batch shuffles (https://github.com/apache/flink/pull/471),
we have much more flexibility and I would like to expose that.
I suggest to add more execution modes that can be chosen on the
`ExecutionEnvironment`:
- {{BATCH}} A mode where every shuffle is executed in a batch way, meaning
preceding operators must be done before successors start. Only for the batch
programs (d'oh).
- {{PIPELINED}} This is the mode corresponding to the current execution
mode. It pipelines where possible and batches, where deadlocks would
otherwise happen. Initially, I would make this the default (be close to the
current behavior). Only available for batch programs.
- {{PIPELINED_WITH_BATCH_FALLBACK}} This would start out with pipelining
shuffles and fall back to batch shuffles upon failure and recovery, or once
it sees that not enough slots are available to bring up all operators at once
(requirement for pipelining).
- {{STREAMING}} This is the default and only way for streaming programs.
All communication is pipelined, and the special streaming checkpointing code
is activated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1615) Introduces a new InputFormat for Tweets
[ https://issues.apache.org/jira/browse/FLINK-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364863#comment-14364863 ] ASF GitHub Bot commented on FLINK-1615: --- Github user Elbehery commented on the pull request: https://github.com/apache/flink/pull/442#issuecomment-82254957 I have revised the commit .. All the files has the license header, except a resource file which contains 4 tweets for testing purpose .. Could this be the problem ? Introduces a new InputFormat for Tweets --- Key: FLINK-1615 URL: https://issues.apache.org/jira/browse/FLINK-1615 Project: Flink Issue Type: New Feature Components: flink-contrib Affects Versions: 0.8.1 Reporter: mustafa elbehery Priority: Minor An event-driven parser for Tweets into Java Pojos. It parses all the important part of the tweet into Java objects. Tested on cluster and the performance in pretty well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLINK-1671) Add execution modes for programs
[
https://issues.apache.org/jira/browse/FLINK-1671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen resolved FLINK-1671.
-
Resolution: Fixed
Fixed in 1c50d87c190f4845161bb3e281a5904efb32a605
Add execution modes for programs
Key: FLINK-1671
URL: https://issues.apache.org/jira/browse/FLINK-1671
Project: Flink
Issue Type: Bug
Affects Versions: 0.9
Reporter: Stephan Ewen
Assignee: Stephan Ewen
Fix For: 0.9
Currently, there is a single way that programs get executed: Pipelined. With
the new code for batch shuffles (https://github.com/apache/flink/pull/471),
we have much more flexibility and I would like to expose that.
I suggest to add more execution modes that can be chosen on the
`ExecutionEnvironment`:
- {{BATCH}} A mode where every shuffle is executed in a batch way, meaning
preceding operators must be done before successors start. Only for the batch
programs (d'oh).
- {{PIPELINED}} This is the mode corresponding to the current execution
mode. It pipelines where possible and batches, where deadlocks would
otherwise happen. Initially, I would make this the default (be close to the
current behavior). Only available for batch programs.
- {{PIPELINED_WITH_BATCH_FALLBACK}} This would start out with pipelining
shuffles and fall back to batch shuffles upon failure and recovery, or once
it sees that not enough slots are available to bring up all operators at once
(requirement for pipelining).
- {{STREAMING}} This is the default and only way for streaming programs.
All communication is pipelined, and the special streaming checkpointing code
is activated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-1713) Add support for blocking data exchange in closed loop iterations
Ufuk Celebi created FLINK-1713: -- Summary: Add support for blocking data exchange in closed loop iterations Key: FLINK-1713 URL: https://issues.apache.org/jira/browse/FLINK-1713 Project: Flink Issue Type: Improvement Components: Distributed Runtime Affects Versions: master Reporter: Ufuk Celebi Priority: Minor The way that blocking intermediate results are currently managed prohibits them from being used inside of closed loops. A blocking result has to be fully produced before its receivers are deployed and there is no notion of single iterations etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: modify the getlogger of the class
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/490#issuecomment-82269385 Thanks for the pull request. Looks good to merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1671] Add different execution modes to ...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/487 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1679) Document how degree of parallelism / parallelism / slots are connected to each other
[ https://issues.apache.org/jira/browse/FLINK-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364840#comment-14364840 ] ASF GitHub Bot commented on FLINK-1679: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/488#issuecomment-82251340 @uce @StephanEwen I agree with you two that it's better to keep the old API methods and mark them as deprecated while introducing the new ones. When the API has been stabilized and 1.0 comes up, we can remove the old methods. Document how degree of parallelism / parallelism / slots are connected to each other --- Key: FLINK-1679 URL: https://issues.apache.org/jira/browse/FLINK-1679 Project: Flink Issue Type: Task Components: Documentation Affects Versions: 0.9 Reporter: Robert Metzger Assignee: Maximilian Michels I see too many users being confused about properly setting up Flink with respect to parallelism. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1615] [java api] SimpleTweetInputFormat
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/442#issuecomment-82260288 Yes, this can be the problem. Can you add a licence header (with comments) to this file? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1615) Introduces a new InputFormat for Tweets
[ https://issues.apache.org/jira/browse/FLINK-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364875#comment-14364875 ] ASF GitHub Bot commented on FLINK-1615: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/442#issuecomment-82260288 Yes, this can be the problem. Can you add a licence header (with comments) to this file? Introduces a new InputFormat for Tweets --- Key: FLINK-1615 URL: https://issues.apache.org/jira/browse/FLINK-1615 Project: Flink Issue Type: New Feature Components: flink-contrib Affects Versions: 0.8.1 Reporter: mustafa elbehery Priority: Minor An event-driven parser for Tweets into Java Pojos. It parses all the important part of the tweet into Java objects. Tested on cluster and the performance in pretty well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1671) Add execution modes for programs
[
https://issues.apache.org/jira/browse/FLINK-1671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364804#comment-14364804
]
ASF GitHub Bot commented on FLINK-1671:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/487
Add execution modes for programs
Key: FLINK-1671
URL: https://issues.apache.org/jira/browse/FLINK-1671
Project: Flink
Issue Type: Bug
Affects Versions: 0.9
Reporter: Stephan Ewen
Assignee: Stephan Ewen
Fix For: 0.9
Currently, there is a single way that programs get executed: Pipelined. With
the new code for batch shuffles (https://github.com/apache/flink/pull/471),
we have much more flexibility and I would like to expose that.
I suggest to add more execution modes that can be chosen on the
`ExecutionEnvironment`:
- {{BATCH}} A mode where every shuffle is executed in a batch way, meaning
preceding operators must be done before successors start. Only for the batch
programs (d'oh).
- {{PIPELINED}} This is the mode corresponding to the current execution
mode. It pipelines where possible and batches, where deadlocks would
otherwise happen. Initially, I would make this the default (be close to the
current behavior). Only available for batch programs.
- {{PIPELINED_WITH_BATCH_FALLBACK}} This would start out with pipelining
shuffles and fall back to batch shuffles upon failure and recovery, or once
it sees that not enough slots are available to bring up all operators at once
(requirement for pipelining).
- {{STREAMING}} This is the default and only way for streaming programs.
All communication is pipelined, and the special streaming checkpointing code
is activated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1679] rename degree of parallelism to p...
Github user mxm commented on the pull request: https://github.com/apache/flink/pull/488#issuecomment-82251340 @uce @StephanEwen I agree with you two that it's better to keep the old API methods and mark them as deprecated while introducing the new ones. When the API has been stabilized and 1.0 comes up, we can remove the old methods. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: Add support for building Flink with Scala 2.11
Github user aalexandrov commented on the pull request: https://github.com/apache/flink/pull/477#issuecomment-82266786 If we go with the suffix, we basically have two options: 1. Add the suffix only to modules that use Scala 1. Add the suffix to all maven modules, regardless whether they use Scala or not Downside of option (1) is that we might break split names incrementally if we add Scala in the future. Downside of option (2) is the more LOC that need to be adapted in the POMs. My two cents are for (2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-441) Renaming in pact-compiler
[ https://issues.apache.org/jira/browse/FLINK-441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365791#comment-14365791 ] ASF GitHub Bot commented on FLINK-441: -- GitHub user StephanEwen opened a pull request: https://github.com/apache/flink/pull/492 [FLINK-441] Rename Pact* and Nephele* classes in the optimizer Renames the package `org.apache.flink.compiler` to `org.apache.fink.optimizer` Renames `PactCompiler` to `Optimizer` Renames `NepheleJobGraphGenerator` to `JobGraphGenerator` Renames `PactConnection` to `DagConnection` (since the enclosing data structure is the Optimizer DAG). Renames instances of internal variables and methods that referred to `PactContract` to refer to `Operator` instead. Removes the unnecessary context checker. Was only ever valuable in the deprecated record API. Makes the various graph traversals (previously inner classes of the Optimizer) standalone classes, which makes the Optimizer class more graspable. Migrates some changes from the deprecated Record API to the Java API. You can merge this pull request into a Git repository by running: $ git pull https://github.com/StephanEwen/incubator-flink optimizer_rename Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/492.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #492 commit c3868feb6e9618aeb8bdcd500f5f0380abc587f8 Author: Stephan Ewen se...@apache.org Date: 2015-03-17T09:53:35Z [FLINK-441] [optimizer] Rename o.a.flink.compiler to o.a.flink.optimizer commit 86f15955a025da5dbf52b243c7340b2d94341c36 Author: Stephan Ewen se...@apache.org Date: 2015-03-17T09:57:08Z [FLINK-441] [optimizer] Removed obsolete plan validator commit 5c44709642d1442d7bf9a59746a217f37167fec6 Author: Stephan Ewen se...@apache.org Date: 2015-03-17T10:02:33Z [FLINK-441] [optimizer] Remove obsolete and unused utility classes commit 40ef1b93aa5261b05bfee7d9ff8cf67554f80122 Author: Stephan Ewen se...@apache.org Date: 2015-03-17T10:24:35Z [FLINK-441] [optimizer] Rename Pact* and Nephele* classes Also clean up and improve various comments and method names. commit 211f2742d40240f3820c43aef80108849b4808ff Author: Stephan Ewen se...@apache.org Date: 2015-03-17T15:14:50Z [optimizer] Migrate first set of tests (branching plans) to new API commit 3f67a33d84d011942ea4e21f266914a3e53dda72 Author: Stephan Ewen se...@apache.org Date: 2015-03-17T15:43:28Z [optimizer] Moved optimizer graph traversals to dedicated classes to simplify Optimizer class. Renaming in pact-compiler - Key: FLINK-441 URL: https://issues.apache.org/jira/browse/FLINK-441 Project: Flink Issue Type: Sub-task Reporter: GitHub Import Priority: Minor Labels: github-import Fix For: pre-apache I would like to do a cleanup and renaming in the pact-compiler. Most of the work is in line with the recent global renaming, but I also want to clear and organize the various representation structures for the optimized plan. I open this issue to keep track and discuss the suggested renaming. We'll have to coordinate the merging of this issue because some renamings (e.g. PactCompiler - Compiler) seem to affect a lot of other packages. ### Global Scope (Wide Dependencies) The following names are part of the public API of stratosphere-compiler. Their renaming will probably affect a lot of other modules. In ```eu.stratosphere.compiler```: * ```PactCompiler``` ⇒ ```Compiler``` ### Module Scope (Narrow Dependencies) The following names are part of the internal API of stratosphere-compiler. Their renaming will probably affect only stratosphere-compiler and stratosphere-tests. In ```eu.stratosphere.compiler```: * ```DataStatistics``` ⇒ ```StatsStore``` This should be developed as an API for data stats over *expressions* instead of just over *data sources*. * ```NonCachingDataStatistics``` ⇒ *delete*. This class does not seem to be used. Imported from GitHub Url: https://github.com/stratosphere/stratosphere/issues/441 Created by: [aalexandrov|https://github.com/aalexandrov] Labels: Created at: Mon Jan 27 12:33:50 CET 2014 State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-441) Renaming in pact-compiler
[ https://issues.apache.org/jira/browse/FLINK-441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-441: - Assignee: Stephan Ewen Renaming in pact-compiler - Key: FLINK-441 URL: https://issues.apache.org/jira/browse/FLINK-441 Project: Flink Issue Type: Sub-task Reporter: GitHub Import Assignee: Stephan Ewen Priority: Minor Labels: github-import Fix For: pre-apache I would like to do a cleanup and renaming in the pact-compiler. Most of the work is in line with the recent global renaming, but I also want to clear and organize the various representation structures for the optimized plan. I open this issue to keep track and discuss the suggested renaming. We'll have to coordinate the merging of this issue because some renamings (e.g. PactCompiler - Compiler) seem to affect a lot of other packages. ### Global Scope (Wide Dependencies) The following names are part of the public API of stratosphere-compiler. Their renaming will probably affect a lot of other modules. In ```eu.stratosphere.compiler```: * ```PactCompiler``` ⇒ ```Compiler``` ### Module Scope (Narrow Dependencies) The following names are part of the internal API of stratosphere-compiler. Their renaming will probably affect only stratosphere-compiler and stratosphere-tests. In ```eu.stratosphere.compiler```: * ```DataStatistics``` ⇒ ```StatsStore``` This should be developed as an API for data stats over *expressions* instead of just over *data sources*. * ```NonCachingDataStatistics``` ⇒ *delete*. This class does not seem to be used. Imported from GitHub Url: https://github.com/stratosphere/stratosphere/issues/441 Created by: [aalexandrov|https://github.com/aalexandrov] Labels: Created at: Mon Jan 27 12:33:50 CET 2014 State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1706] Spilling BarrierBuffer added + ba...
GitHub user gyfora opened a pull request: https://github.com/apache/flink/pull/493 [FLINK-1706] Spilling BarrierBuffer added + basic tests This commit introduces disk spilling for the BarrierBuffers based on the availability of the buffers in the buffer pools of the input gates. This is a first version, with proper resource cleanup still missing. I am putting it up here for review as this is a critical part of the ft logic. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mbalassi/flink FLINK-1706 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/493.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #493 commit 7b0cc330a9717110f53fe24535a59b830f420eb3 Author: Gyula Fora gyf...@apache.org Date: 2015-03-17T16:39:27Z [FLINK-1706] Spilling BarrierBuffer added + basic tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1695) Create machine learning library
[ https://issues.apache.org/jira/browse/FLINK-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365811#comment-14365811 ] ASF GitHub Bot commented on FLINK-1695: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/479#issuecomment-82539185 Looks nice, +1 to add this. Also, +1 to add the JIRA issue number to the commits and squash some basic cleanup commits. Create machine learning library --- Key: FLINK-1695 URL: https://issues.apache.org/jira/browse/FLINK-1695 Project: Flink Issue Type: Improvement Reporter: Till Rohrmann Assignee: Till Rohrmann Labels: ML Create the infrastructure for Flink's machine learning library. This includes the creation of the module structure and the implementation of basic types such as vectors and matrices. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: Remove -j and -a parameters which seemed no lo...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/482#issuecomment-82530221 I think this should go in. The simple the parameter style the better. And without flags (just with arguments) reads easier, as far as I am concerned. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: Merge pull request #1 from apache/master
Github user matadorhong closed the pull request at: https://github.com/apache/flink/pull/495 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: Merge pull request #1 from apache/master
GitHub user matadorhong opened a pull request: https://github.com/apache/flink/pull/495 Merge pull request #1 from apache/master update You can merge this pull request into a Git repository by running: $ git pull https://github.com/matadorhong/flink master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/495.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #495 commit 68de654200b796c45e2f250c0d359c56a3d474bc Author: Sibao Hong hongsi...@huawei.com Date: 2015-03-13T07:03:51Z Merge pull request #1 from apache/master update --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1713) Add support for blocking data exchange in closed loop iterations
[ https://issues.apache.org/jira/browse/FLINK-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366205#comment-14366205 ] Alexander Alexandrov commented on FLINK-1713: - I think he means Flink's native iteration support, i.e. execution plans which are wrapped in an IterationHead and IterationTail operators. Add support for blocking data exchange in closed loop iterations Key: FLINK-1713 URL: https://issues.apache.org/jira/browse/FLINK-1713 Project: Flink Issue Type: Improvement Components: Distributed Runtime Affects Versions: master Reporter: Ufuk Celebi Priority: Minor The way that blocking intermediate results are currently managed prohibits them from being used inside of closed loops. A blocking result has to be fully produced before its receivers are deployed and there is no notion of single iterations etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1500) exampleScalaPrograms.EnumTriangleOptITCase does not finish on Travis
[ https://issues.apache.org/jira/browse/FLINK-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366223#comment-14366223 ] Till Rohrmann commented on FLINK-1500: -- I haven't seen it for a long time. exampleScalaPrograms.EnumTriangleOptITCase does not finish on Travis Key: FLINK-1500 URL: https://issues.apache.org/jira/browse/FLINK-1500 Project: Flink Issue Type: Bug Reporter: Till Rohrmann The test case org.apache.flink.test.exampleScalaPrograms.EnumTriangleOptITCase does not finish on Travis. This problem is non-deterministic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1695) Create machine learning library
[ https://issues.apache.org/jira/browse/FLINK-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366228#comment-14366228 ] ASF GitHub Bot commented on FLINK-1695: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/479 Create machine learning library --- Key: FLINK-1695 URL: https://issues.apache.org/jira/browse/FLINK-1695 Project: Flink Issue Type: Improvement Reporter: Till Rohrmann Assignee: Till Rohrmann Labels: ML Create the infrastructure for Flink's machine learning library. This includes the creation of the module structure and the implementation of basic types such as vectors and matrices. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1695] Kick off of Flink's machine learn...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/479 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Closed] (FLINK-1695) Create machine learning library
[ https://issues.apache.org/jira/browse/FLINK-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann closed FLINK-1695. Resolution: Fixed Added in cf492be965fbbfeade8d200cd2c495fd10fd3a1b Create machine learning library --- Key: FLINK-1695 URL: https://issues.apache.org/jira/browse/FLINK-1695 Project: Flink Issue Type: Improvement Reporter: Till Rohrmann Assignee: Till Rohrmann Labels: ML Create the infrastructure for Flink's machine learning library. This includes the creation of the module structure and the implementation of basic types such as vectors and matrices. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-1698) Add polynomial base feature mapper to ML library
[
https://issues.apache.org/jira/browse/FLINK-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-1698.
Resolution: Fixed
Added in effea93d72710dc9fa8184abc2d97ee33794b84f
Add polynomial base feature mapper to ML library
Key: FLINK-1698
URL: https://issues.apache.org/jira/browse/FLINK-1698
Project: Flink
Issue Type: Improvement
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Labels: ML
Add feature mapper which maps a vector into the polynomial feature space.
This can be used as a preprocessing step prior to applying a {{Learner}} of
Flink's ML library.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (FLINK-1696) Add multiple linear regression to ML library
[ https://issues.apache.org/jira/browse/FLINK-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann closed FLINK-1696. Resolution: Fixed Added in ff83c8c2141486b1719a8c451a485da606784d1f Add multiple linear regression to ML library Key: FLINK-1696 URL: https://issues.apache.org/jira/browse/FLINK-1696 Project: Flink Issue Type: Improvement Reporter: Till Rohrmann Assignee: Till Rohrmann Labels: ML Add multiple linear regression to ML library. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: Cleanup of low level Kafka consumer (Persisten...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/474#issuecomment-82621443 First of all: Great work @gaborhermann! The whole Kafka connector code is well written and contains logic for a lot of cases. I think it was already quite some work to get it to the state it currently is. I took this pull request as a base to add an integration test which is starting a local Zookeeper and Kafka instance. I also fixed a critical issue in the streaming code, regarding exception forwarding. My current WIP is here: https://github.com/rmetzger/flink/tree/kafka-gabor The code needs some more care until its in a good state. The following is missing: - More integration tests: Testing of the `PersistentKafkaSource` with taskmanager failure and job restart, also tests for other stuff (partitioner as class as instance) - Add means to `KafkaSource` and `KafkaSink` for users to pass their own configuration properties to Kafka. - There are some TODOs in the code - Testing of the code in a distributed setup. I'll work on the cluster testing tomorrow. In the course of that, I'm probably going to add more code to my wip branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Closed] (FLINK-1697) Add alternating least squares algorithm for matrix factorization to ML library
[ https://issues.apache.org/jira/browse/FLINK-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann closed FLINK-1697. Resolution: Fixed Added in 21e2d96f893e4460a8d85c501e31dc09ed2f0043 Add alternating least squares algorithm for matrix factorization to ML library -- Key: FLINK-1697 URL: https://issues.apache.org/jira/browse/FLINK-1697 Project: Flink Issue Type: Improvement Reporter: Till Rohrmann Assignee: Till Rohrmann Labels: ML Add alternating least squares algorithm for matrix factorization to ML library -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: Add final modifier to PollingThread#lock objec...
GitHub user hsaputra opened a pull request: https://github.com/apache/flink/pull/494 Add final modifier to PollingThread#lock object Add final modifier to PollingThread#lock object to make sure immutability as lock object. You can merge this pull request into a Git repository by running: $ git pull https://github.com/hsaputra/flink add_final_lock_pollingthread Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/494.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #494 commit 44d550312b029f755129dee74bdbea528256bb23 Author: Henry Saputra henry.sapu...@gmail.com Date: 2015-03-17T23:14:44Z Add final modifier to PollingThread#lock object to make sure immutability. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: Fix issue where Windows paths were not recogni...
Github user balidani commented on the pull request: https://github.com/apache/flink/pull/491#issuecomment-82625011 Hi! I squashed them, I hope it's done correctly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: Fix issue where Windows paths were not recogni...
Github user balidani commented on the pull request: https://github.com/apache/flink/pull/491#issuecomment-82567110 Hi! I added some test cases to `PathTest`. Is it okay like this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1714) Fix the bug of logger class loader
[ https://issues.apache.org/jira/browse/FLINK-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365931#comment-14365931 ] ASF GitHub Bot commented on FLINK-1714: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/490 Fix the bug of logger class loader -- Key: FLINK-1714 URL: https://issues.apache.org/jira/browse/FLINK-1714 Project: Flink Issue Type: Improvement Components: JobManager Affects Versions: master Reporter: Sibao Hong Priority: Minor For log in class SecurityUtils, the LoggerFactory.getLogger() function should be for SecurityUtils not MapPartitionDriver. It is a bug I think -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1342) Quickstart's assembly can possibly filter out user's code
[ https://issues.apache.org/jira/browse/FLINK-1342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365933#comment-14365933 ] Stephan Ewen commented on FLINK-1342: - Merged into 0.9 in 04a738836f30d8967c427208ab04b1aef34f370a Quickstart's assembly can possibly filter out user's code - Key: FLINK-1342 URL: https://issues.apache.org/jira/browse/FLINK-1342 Project: Flink Issue Type: Bug Affects Versions: 0.9 Reporter: Márton Balassi Assignee: Robert Metzger Priority: Critical Fix For: 0.9, 0.8.2 I've added a quick solution for [1] for the time being. The assembly still filters out everything from the org.apache.flink namespace, so any user code placed there will be missing from the fat jar. If we do not use filtering at all the size of the jar goes up to almost 100 MB. [1] https://issues.apache.org/jira/browse/FLINK-1225 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1682) Port Record-API based optimizer tests to new Java API
[ https://issues.apache.org/jira/browse/FLINK-1682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365944#comment-14365944 ] Stephan Ewen commented on FLINK-1682: - I did some small porting as part of this pull request https://github.com/apache/flink/pull/492 Port Record-API based optimizer tests to new Java API - Key: FLINK-1682 URL: https://issues.apache.org/jira/browse/FLINK-1682 Project: Flink Issue Type: Sub-task Components: Optimizer Reporter: Fabian Hueske Assignee: Fabian Hueske Priority: Minor Fix For: 0.9 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLINK-1714) Fix the bug of logger class loader
[ https://issues.apache.org/jira/browse/FLINK-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen resolved FLINK-1714. - Resolution: Fixed Fix Version/s: 0.9 Fixed via 05e06995d7099cfb33bab71aea798dcfdcddc599 Thank you for the patch! Fix the bug of logger class loader -- Key: FLINK-1714 URL: https://issues.apache.org/jira/browse/FLINK-1714 Project: Flink Issue Type: Improvement Components: JobManager Affects Versions: master Reporter: Sibao Hong Priority: Minor Fix For: 0.9 For log in class SecurityUtils, the LoggerFactory.getLogger() function should be for SecurityUtils not MapPartitionDriver. It is a bug I think -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLINK-1414) Remove quickstart-*.sh from git source and put them to the website's svn
[ https://issues.apache.org/jira/browse/FLINK-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen resolved FLINK-1414. - Resolution: Fixed Fix Version/s: 0.9 Fixed via 9b4f483ff456dfa42a76978dbb3adb095a3f499b Remove quickstart-*.sh from git source and put them to the website's svn Key: FLINK-1414 URL: https://issues.apache.org/jira/browse/FLINK-1414 Project: Flink Issue Type: Task Components: Project Website Reporter: Robert Metzger Assignee: Robert Metzger Priority: Minor Fix For: 0.9 The quickstart.sh script is currently (due to historic reasons) located in the main source repo. It probably better fits into the homepage because it is independent of the versions in the pom.xml files. This also makes the release maintenance easier. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLINK-1540) JobManagerFailsITCase Startup timed out on travis
[
https://issues.apache.org/jira/browse/FLINK-1540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen resolved FLINK-1540.
-
Resolution: Fixed
Fix Version/s: 0.9
Fixed by dropping the older Akka version after shading conflicting dependencies.
Fixed as of 84e76f4d3274e07176f7377b7b739b6f180c6296
JobManagerFailsITCase Startup timed out on travis
-
Key: FLINK-1540
URL: https://issues.apache.org/jira/browse/FLINK-1540
Project: Flink
Issue Type: Bug
Components: JobManager
Affects Versions: 0.9
Reporter: Robert Metzger
Fix For: 0.9
{code}
[ERROR] [02/13/2015 15:47:47.429]
[main-ScalaTest-running-JobManagerFailsITCase] [Remoting] Remoting error:
[Startup timed out] [
akka.remote.RemoteTransportException: Startup timed out
at
akka.remote.Remoting.akka$remote$Remoting$$notifyError(Remoting.scala:129)
at akka.remote.Remoting.start(Remoting.scala:191)
at
akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:184)
at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:579)
at akka.actor.ActorSystemImpl._start(ActorSystem.scala:577)
at akka.actor.ActorSystemImpl.start(ActorSystem.scala:588)
at akka.actor.ActorSystem$.apply(ActorSystem.scala:111)
at akka.actor.ActorSystem$.apply(ActorSystem.scala:104)
at akka.actor.ActorSystem$.create(ActorSystem.scala:66)
at
org.apache.flink.runtime.akka.AkkaUtils$.createActorSystem(AkkaUtils.scala:68)
at
org.apache.flink.runtime.minicluster.FlinkMiniCluster.startJobManagerActorSystem(FlinkMiniCluster.scala:91)
at
org.apache.flink.runtime.minicluster.FlinkMiniCluster.init(FlinkMiniCluster.scala:51)
at
org.apache.flink.runtime.testingUtils.TestingCluster.init(TestingCluster.scala:36)
at
org.apache.flink.runtime.testingUtils.TestingUtils$.startTestingClusterDeathWatch(TestingUtils.scala:106)
at
org.apache.flink.runtime.jobmanager.JobManagerFailsITCase$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(JobManagerFailsITCase.scala:45)
at
org.apache.flink.runtime.jobmanager.JobManagerFailsITCase$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(JobManagerFailsITCase.scala:42)
at
org.apache.flink.runtime.jobmanager.JobManagerFailsITCase$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(JobManagerFailsITCase.scala:42)
at
org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.WordSpecLike$$anon$1.apply(WordSpecLike.scala:953)
at org.scalatest.Suite$class.withFixture(Suite.scala:1122)
at
org.apache.flink.runtime.jobmanager.JobManagerFailsITCase.withFixture(JobManagerFailsITCase.scala:32)
at
org.scalatest.WordSpecLike$class.invokeWithFixture$1(WordSpecLike.scala:950)
at
org.scalatest.WordSpecLike$$anonfun$runTest$1.apply(WordSpecLike.scala:962)
at
org.scalatest.WordSpecLike$$anonfun$runTest$1.apply(WordSpecLike.scala:962)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
at org.scalatest.WordSpecLike$class.runTest(WordSpecLike.scala:962)
at
org.apache.flink.runtime.jobmanager.JobManagerFailsITCase.runTest(JobManagerFailsITCase.scala:32)
at
org.scalatest.WordSpecLike$$anonfun$runTests$1.apply(WordSpecLike.scala:1021)
at
org.scalatest.WordSpecLike$$anonfun$runTests$1.apply(WordSpecLike.scala:1021)
at
org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
at
org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
at
org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:390)
at
org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:427)
at
org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
at
org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
at org.scalatest.WordSpecLike$class.runTests(WordSpecLike.scala:1021)
at
org.apache.flink.runtime.jobmanager.JobManagerFailsITCase.runTests(JobManagerFailsITCase.scala:32)
at
[jira] [Resolved] (FLINK-1546) Failed job causes JobManager to shutdown due to uncatched WebFrontend exception
[
https://issues.apache.org/jira/browse/FLINK-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen resolved FLINK-1546.
-
Resolution: Fixed
Fix Version/s: 0.9
Job Archiving was fixed in 8ae0dc2d768aecfa3129df553f43d827792b65d7
Failed job causes JobManager to shutdown due to uncatched WebFrontend
exception
---
Key: FLINK-1546
URL: https://issues.apache.org/jira/browse/FLINK-1546
Project: Flink
Issue Type: Bug
Components: JobManager
Affects Versions: 0.9
Reporter: Robert Metzger
Fix For: 0.9
{code}
16:59:26,588 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$2$$anon$1
- Status of job ef19b2b201d4b81f031334cb76eadc78 (Basic Page Rank Example)
changed to FAILEDCleanup job ef19b2b201d4b81f031334cb76eadc78..
16:59:26,591 ERROR akka.actor.OneForOneStrategy
- Can only archive the job from a terminal state
java.lang.IllegalStateException: Can only archive the job from a terminal
state
at
org.apache.flink.runtime.executiongraph.ExecutionGraph.prepareForArchiving(ExecutionGraph.java:648)
at
org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$removeJob(JobManager.scala:508)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$receiveWithLogMessages$1.applyOrElse(JobManager.scala:271)
at
scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at
org.apache.flink.yarn.YarnJobManager$$anonfun$receiveYarnMessages$1.applyOrElse(YarnJobManager.scala:70)
at scala.PartialFunction$OrElse.apply(PartialFunction.scala:162)
at
org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMessages.scala:37)
at
org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMessages.scala:30)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
at
org.apache.flink.runtime.ActorLogMessages$$anon$1.applyOrElse(ActorLogMessages.scala:30)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at
org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:86)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
at akka.dispatch.Mailbox.run(Mailbox.scala:221)
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
16:59:26,595 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$2$$anon$1
- Stopping webserver.
16:59:26,654 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$2$$anon$1
- Stopped webserver.
16:59:26,656 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$2$$anon$1
- Stopping job manager akka://flink/user/jobmanager.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1714) Fix the bug of logger class loader
[ https://issues.apache.org/jira/browse/FLINK-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365863#comment-14365863 ] ASF GitHub Bot commented on FLINK-1714: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/490#issuecomment-82555937 Thanks for the fix, will merge this... Fix the bug of logger class loader -- Key: FLINK-1714 URL: https://issues.apache.org/jira/browse/FLINK-1714 Project: Flink Issue Type: Improvement Components: JobManager Affects Versions: master Reporter: Sibao Hong Priority: Minor For log in class SecurityUtils, the LoggerFactory.getLogger() function should be for SecurityUtils not MapPartitionDriver. It is a bug I think -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: Fix issue where Windows paths were not recogni...
Github user balidani commented on the pull request:
https://github.com/apache/flink/pull/491#issuecomment-82577438
I just realized that this fails in Travis, because in `hasWindowsDrive`
(Path.java, line 282) there is `if (!OperatingSystem.isWindows()) {`. What can
we do in this situation?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Fix issue where Windows paths were not recogni...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/491#issuecomment-82578379 I think that we should identify windows drives independently of whether we are on windows. The client may run on Linux, the cluster windows (or vice versa), that is totally valid. I would change the method to not depend on the operating system. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-441) Renaming in pact-compiler
[ https://issues.apache.org/jira/browse/FLINK-441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365975#comment-14365975 ] ASF GitHub Bot commented on FLINK-441: -- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/492#issuecomment-82579347 I think it is fine if we mention in the release notes that the name has changed. All the quickstart pom users won't be affected. Renaming in pact-compiler - Key: FLINK-441 URL: https://issues.apache.org/jira/browse/FLINK-441 Project: Flink Issue Type: Sub-task Reporter: GitHub Import Assignee: Stephan Ewen Priority: Minor Labels: github-import Fix For: pre-apache I would like to do a cleanup and renaming in the pact-compiler. Most of the work is in line with the recent global renaming, but I also want to clear and organize the various representation structures for the optimized plan. I open this issue to keep track and discuss the suggested renaming. We'll have to coordinate the merging of this issue because some renamings (e.g. PactCompiler - Compiler) seem to affect a lot of other packages. ### Global Scope (Wide Dependencies) The following names are part of the public API of stratosphere-compiler. Their renaming will probably affect a lot of other modules. In ```eu.stratosphere.compiler```: * ```PactCompiler``` ⇒ ```Compiler``` ### Module Scope (Narrow Dependencies) The following names are part of the internal API of stratosphere-compiler. Their renaming will probably affect only stratosphere-compiler and stratosphere-tests. In ```eu.stratosphere.compiler```: * ```DataStatistics``` ⇒ ```StatsStore``` This should be developed as an API for data stats over *expressions* instead of just over *data sources*. * ```NonCachingDataStatistics``` ⇒ *delete*. This class does not seem to be used. Imported from GitHub Url: https://github.com/stratosphere/stratosphere/issues/441 Created by: [aalexandrov|https://github.com/aalexandrov] Labels: Created at: Mon Jan 27 12:33:50 CET 2014 State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1635) Remove Apache Thrift dependency from Flink
[ https://issues.apache.org/jira/browse/FLINK-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366023#comment-14366023 ] Stephan Ewen commented on FLINK-1635: - [~rmetzger] This is removed an fixed, if I understand correctly? Remove Apache Thrift dependency from Flink -- Key: FLINK-1635 URL: https://issues.apache.org/jira/browse/FLINK-1635 Project: Flink Issue Type: Bug Components: Java API Affects Versions: 0.9 Reporter: Robert Metzger I've added Thrift and Protobuf to Flink to support it out of the box with Kryo. However, after trying to access a HCatalog/Hive table yesterday using Flink I found that there is a dependency conflict between Flink and Hive (on thrift). Maybe it makes more sense to properly document our serialization framework and provide a copypaste solution on how to get thrift/protobuf et al to work with Flink. Please chime in if you are against removing the out of the box support for protobuf and kryo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1350) Add blocking intermediate result partitions
[ https://issues.apache.org/jira/browse/FLINK-1350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365862#comment-14365862 ] ASF GitHub Bot commented on FLINK-1350: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/471#issuecomment-82555635 Once this is in, we can start removing the deadlock detection in the optimizer and the pipeline breaking caches. Add blocking intermediate result partitions --- Key: FLINK-1350 URL: https://issues.apache.org/jira/browse/FLINK-1350 Project: Flink Issue Type: Improvement Components: Distributed Runtime Reporter: Ufuk Celebi Assignee: Ufuk Celebi The current state of runtime support for intermediate results (see https://github.com/apache/incubator-flink/pull/254 and FLINK-986) only supports pipelined intermediate results (with back pressure), which are consumed as they are being produced. The next variant we need to support are blocking intermediate results (without back pressure), which are fully produced before being consumed. This is for example desirable in situations, where we currently may run into deadlocks when running pipelined. I will start working on this on top of my pending pull request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1714]Fix the bug of logger class loader
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/490#issuecomment-82555937 Thanks for the fix, will merge this... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-441) Renaming in pact-compiler
[ https://issues.apache.org/jira/browse/FLINK-441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365960#comment-14365960 ] ASF GitHub Bot commented on FLINK-441: -- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/492#issuecomment-82577264 We can do that, I am okay with this. So far, all renaming was strictly internal. The renaming of the maven project may affect some people who have added it as a (redundant) dependency. Renaming in pact-compiler - Key: FLINK-441 URL: https://issues.apache.org/jira/browse/FLINK-441 Project: Flink Issue Type: Sub-task Reporter: GitHub Import Assignee: Stephan Ewen Priority: Minor Labels: github-import Fix For: pre-apache I would like to do a cleanup and renaming in the pact-compiler. Most of the work is in line with the recent global renaming, but I also want to clear and organize the various representation structures for the optimized plan. I open this issue to keep track and discuss the suggested renaming. We'll have to coordinate the merging of this issue because some renamings (e.g. PactCompiler - Compiler) seem to affect a lot of other packages. ### Global Scope (Wide Dependencies) The following names are part of the public API of stratosphere-compiler. Their renaming will probably affect a lot of other modules. In ```eu.stratosphere.compiler```: * ```PactCompiler``` ⇒ ```Compiler``` ### Module Scope (Narrow Dependencies) The following names are part of the internal API of stratosphere-compiler. Their renaming will probably affect only stratosphere-compiler and stratosphere-tests. In ```eu.stratosphere.compiler```: * ```DataStatistics``` ⇒ ```StatsStore``` This should be developed as an API for data stats over *expressions* instead of just over *data sources*. * ```NonCachingDataStatistics``` ⇒ *delete*. This class does not seem to be used. Imported from GitHub Url: https://github.com/stratosphere/stratosphere/issues/441 Created by: [aalexandrov|https://github.com/aalexandrov] Labels: Created at: Mon Jan 27 12:33:50 CET 2014 State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLINK-1584) Spurious failure of TaskManagerFailsITCase
[
https://issues.apache.org/jira/browse/FLINK-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen resolved FLINK-1584.
-
Resolution: Fixed
Fixed with the switch to the newer akka version (enabled by shading away
conflicting dependencies)
84e76f4d3274e07176f7377b7b739b6f180c6296
Spurious failure of TaskManagerFailsITCase
--
Key: FLINK-1584
URL: https://issues.apache.org/jira/browse/FLINK-1584
Project: Flink
Issue Type: Bug
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Fix For: 0.9
The {{TaskManagerFailsITCase}} fails spuriously on Travis. The reason might
be that different test cases try to access the same {{JobManager}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1500) exampleScalaPrograms.EnumTriangleOptITCase does not finish on Travis
[ https://issues.apache.org/jira/browse/FLINK-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366001#comment-14366001 ] Stephan Ewen commented on FLINK-1500: - Have we seen this again, or was this an artifact of one of the bugs we fixed in the last weeks (like intermediate result partition lookup) ? exampleScalaPrograms.EnumTriangleOptITCase does not finish on Travis Key: FLINK-1500 URL: https://issues.apache.org/jira/browse/FLINK-1500 Project: Flink Issue Type: Bug Reporter: Till Rohrmann The test case org.apache.flink.test.exampleScalaPrograms.EnumTriangleOptITCase does not finish on Travis. This problem is non-deterministic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1350] [runtime] Add blocking result par...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/471#issuecomment-82555635 Once this is in, we can start removing the deadlock detection in the optimizer and the pipeline breaking caches. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1342) Quickstart's assembly can possibly filter out user's code
[ https://issues.apache.org/jira/browse/FLINK-1342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365932#comment-14365932 ] ASF GitHub Bot commented on FLINK-1342: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/486 Quickstart's assembly can possibly filter out user's code - Key: FLINK-1342 URL: https://issues.apache.org/jira/browse/FLINK-1342 Project: Flink Issue Type: Bug Affects Versions: 0.9 Reporter: Márton Balassi Assignee: Robert Metzger Priority: Critical Fix For: 0.9, 0.8.2 I've added a quick solution for [1] for the time being. The assembly still filters out everything from the org.apache.flink namespace, so any user code placed there will be missing from the fat jar. If we do not use filtering at all the size of the jar goes up to almost 100 MB. [1] https://issues.apache.org/jira/browse/FLINK-1225 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1342] Fix filtering of usercode from qu...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/486 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-441) Renaming in pact-compiler
[ https://issues.apache.org/jira/browse/FLINK-441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365952#comment-14365952 ] ASF GitHub Bot commented on FLINK-441: -- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/492#issuecomment-82576423 Why aren't we renaming the `flink-compiler` module into `flink-optimizer`? Its an internal module only, so users won't see what we reorganized our code internally. Renaming in pact-compiler - Key: FLINK-441 URL: https://issues.apache.org/jira/browse/FLINK-441 Project: Flink Issue Type: Sub-task Reporter: GitHub Import Assignee: Stephan Ewen Priority: Minor Labels: github-import Fix For: pre-apache I would like to do a cleanup and renaming in the pact-compiler. Most of the work is in line with the recent global renaming, but I also want to clear and organize the various representation structures for the optimized plan. I open this issue to keep track and discuss the suggested renaming. We'll have to coordinate the merging of this issue because some renamings (e.g. PactCompiler - Compiler) seem to affect a lot of other packages. ### Global Scope (Wide Dependencies) The following names are part of the public API of stratosphere-compiler. Their renaming will probably affect a lot of other modules. In ```eu.stratosphere.compiler```: * ```PactCompiler``` ⇒ ```Compiler``` ### Module Scope (Narrow Dependencies) The following names are part of the internal API of stratosphere-compiler. Their renaming will probably affect only stratosphere-compiler and stratosphere-tests. In ```eu.stratosphere.compiler```: * ```DataStatistics``` ⇒ ```StatsStore``` This should be developed as an API for data stats over *expressions* instead of just over *data sources*. * ```NonCachingDataStatistics``` ⇒ *delete*. This class does not seem to be used. Imported from GitHub Url: https://github.com/stratosphere/stratosphere/issues/441 Created by: [aalexandrov|https://github.com/aalexandrov] Labels: Created at: Mon Jan 27 12:33:50 CET 2014 State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-441] Rename Pact* and Nephele* classes ...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/492#issuecomment-82576423 Why aren't we renaming the `flink-compiler` module into `flink-optimizer`? Its an internal module only, so users won't see what we reorganized our code internally. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (FLINK-1535) Use usercode class loader to serialize/deserialize accumulators
[ https://issues.apache.org/jira/browse/FLINK-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen updated FLINK-1535: Priority: Blocker (was: Major) Use usercode class loader to serialize/deserialize accumulators --- Key: FLINK-1535 URL: https://issues.apache.org/jira/browse/FLINK-1535 Project: Flink Issue Type: Bug Components: JobManager Affects Versions: 0.9 Reporter: Stephan Ewen Assignee: Stephan Ewen Priority: Blocker Fix For: 0.9 Currently, accumulators are transferred via simple Akka Messages. Since the accumulators may be user defined types, we should use the user code class loader for code loading when deserializing them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-441] Rename Pact* and Nephele* classes ...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/492#issuecomment-82579347 I think it is fine if we mention in the release notes that the name has changed. All the quickstart pom users won't be affected. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: Fix issue where Windows paths were not recogni...
Github user balidani commented on the pull request: https://github.com/apache/flink/pull/491#issuecomment-82580762 I removed that part, I hope you meant it like this :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (FLINK-1459) Collect DataSet to client
[ https://issues.apache.org/jira/browse/FLINK-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen resolved FLINK-1459. - Resolution: Fixed Fix Version/s: 0.9 Assignee: Stephan Ewen This has been implemented a while back, actually. Sorry for the late update. Implemented in 3dc2fe1dc300146e5209023274c0b0d04277f9ee Collect DataSet to client - Key: FLINK-1459 URL: https://issues.apache.org/jira/browse/FLINK-1459 Project: Flink Issue Type: Improvement Reporter: John Sandiford Assignee: Stephan Ewen Fix For: 0.9 Hi, I may well have missed something obvious here but I cannot find an easy way to extract the values in a DataSet to the client. Spark has collect, collectAsMap etc... (I need to pass the values from a small aggregated DataSet back to a machine learning library which is controlling the iterations.) The only way I could find to do this was to implement my own in memory OutputFormat. This is not ideal, but does work. Many thanks, John val env = ExecutionEnvironment.getExecutionEnvironment val data: DataSet[Double] = env.fromElements(1.0, 2.0, 3.0, 4.0) val result = data.reduce((a, b) = a) val valuesOnClient = result.??? env.execute(Simple example) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: Fix issue where Windows paths were not recogni...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/491#issuecomment-82584339 Yeah, that is a good start. Does it make sense to also check for the pattern driverletter-colon (c:) ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1467) Job deployment fails with NPE on JobManager, if TMs did not start properly
[
https://issues.apache.org/jira/browse/FLINK-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366026#comment-14366026
]
Stephan Ewen commented on FLINK-1467:
-
The null-pointer exception has fixed in one of the TaskManager / Akka exception
reworks.
The fix for the root cause (TaskManagers fail fast when memory initialization
fails) is part of [FLINK-1580].
I am closing this as a duplicate.
Job deployment fails with NPE on JobManager, if TMs did not start properly
--
Key: FLINK-1467
URL: https://issues.apache.org/jira/browse/FLINK-1467
Project: Flink
Issue Type: Bug
Components: JobManager
Reporter: Robert Metzger
I have a Flink cluster started where all TaskManagers died
(misconfiguration). The JobManager needs more than 200 seconds to realize
that (on the TaskManagers overview, you see timeouts 200). When submitting
a job, you'll get the following exception:
{code}
org.apache.flink.client.program.ProgramInvocationException: The program
execution failed: java.lang.Exception: Failed to deploy the task CHAIN
DataSource (Generator: class io.airlift.tpch.NationGenerator) - Map (Map at
writeAsFormattedText(DataSet.java:1132)) (1/1) - execution #0 to slot SubSlot
0 (f8d11026ec5a11f0b273184c74ec4f29 (0) - ALLOCATED/ALIVE):
java.lang.NullPointerException
at
org.apache.flink.runtime.taskmanager.TaskManager.org$apache$flink$runtime$taskmanager$TaskManager$$submitTask(TaskManager.scala:346)
at
org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$receiveWithLogMessages$1.applyOrElse(TaskManager.scala:248)
at
scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at
org.apache.flink.yarn.YarnTaskManager$$anonfun$receiveYarnMessages$1.applyOrElse(YarnTaskManager.scala:32)
at scala.PartialFunction$OrElse.apply(PartialFunction.scala:162)
at
org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMessages.scala:41)
at
org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMessages.scala:27)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
at
org.apache.flink.runtime.ActorLogMessages$$anon$1.applyOrElse(ActorLogMessages.scala:27)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at
org.apache.flink.runtime.taskmanager.TaskManager.aroundReceive(TaskManager.scala:78)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
at akka.dispatch.Mailbox.run(Mailbox.scala:221)
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
at
org.apache.flink.runtime.executiongraph.Execution$2.onComplete(Execution.java:311)
at akka.dispatch.OnComplete.internal(Future.scala:247)
at akka.dispatch.OnComplete.internal(Future.scala:244)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:174)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:171)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at
scala.concurrent.impl.ExecutionContextImpl$$anon$3.exec(ExecutionContextImpl.scala:107)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
at org.apache.flink.client.program.Client.run(Client.java:345)
at org.apache.flink.client.program.Client.run(Client.java:304)
at org.apache.flink.client.program.Client.run(Client.java:298)
at
org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:55)
at flink.generators.programs.TPCHGenerator.main(TPCHGenerator.java:80)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
[GitHub] flink pull request: Fix issue where Windows paths were not recogni...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/491#issuecomment-82588549 Yeah, squashing would be good... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: Add support for building Flink with Scala 2.11
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/477#issuecomment-82285264 +1 for option (2) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-1714) Fix the bug of logger class loader
Sibao Hong created FLINK-1714: - Summary: Fix the bug of logger class loader Key: FLINK-1714 URL: https://issues.apache.org/jira/browse/FLINK-1714 Project: Flink Issue Type: Improvement Components: JobManager Affects Versions: master Reporter: Sibao Hong Priority: Minor For log in class SecurityUtils, the LoggerFactory.getLogger() function should be for SecurityUtils not MapPartitionDriver. It is a bug I think -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1576] [gelly] improvements to the gelly...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/470 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1695) Create machine learning library
[ https://issues.apache.org/jira/browse/FLINK-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365249#comment-14365249 ] ASF GitHub Bot commented on FLINK-1695: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/479#issuecomment-82392282 merge it. To get some bonus points, include the JIRA id with the commits Create machine learning library --- Key: FLINK-1695 URL: https://issues.apache.org/jira/browse/FLINK-1695 Project: Flink Issue Type: Improvement Reporter: Till Rohrmann Assignee: Till Rohrmann Create the infrastructure for Flink's machine learning library. This includes the creation of the module structure and the implementation of basic types such as vectors and matrices. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1695) Create machine learning library
[ https://issues.apache.org/jira/browse/FLINK-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365245#comment-14365245 ] ASF GitHub Bot commented on FLINK-1695: --- Github user tillrohrmann commented on the pull request: https://github.com/apache/flink/pull/479#issuecomment-82391642 I added the missing Scala docs. If there are no objections, then I would merge the PR once Travis has passed. Create machine learning library --- Key: FLINK-1695 URL: https://issues.apache.org/jira/browse/FLINK-1695 Project: Flink Issue Type: Improvement Reporter: Till Rohrmann Assignee: Till Rohrmann Create the infrastructure for Flink's machine learning library. This includes the creation of the module structure and the implementation of basic types such as vectors and matrices. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1695] Kick off of Flink's machine learn...
Github user tillrohrmann commented on the pull request: https://github.com/apache/flink/pull/479#issuecomment-82391642 I added the missing Scala docs. If there are no objections, then I would merge the PR once Travis has passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1652] fixes superstep increment in Coll...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/464 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Closed] (FLINK-1576) Change the Gelly examples to be consistent with the other Flink examples
[ https://issues.apache.org/jira/browse/FLINK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vasia Kalavri closed FLINK-1576. Resolution: Implemented Change the Gelly examples to be consistent with the other Flink examples Key: FLINK-1576 URL: https://issues.apache.org/jira/browse/FLINK-1576 Project: Flink Issue Type: Improvement Components: Gelly Affects Versions: 0.8.0 Reporter: Andra Lungu Assignee: Vasia Kalavri Labels: easyfix The current Gelly examples just work on default input data. If we look at the other Flink examples, e.g. Connected Components, they also allow input data to be read from a text file passed as a parameter to the main method. It would be nice to follow the same approach in our examples. A first step in that direction is the SSSP example. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-1652) Wrong superstep number in VertexCentricIteration in Collection mode
[ https://issues.apache.org/jira/browse/FLINK-1652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vasia Kalavri closed FLINK-1652. Resolution: Fixed Wrong superstep number in VertexCentricIteration in Collection mode --- Key: FLINK-1652 URL: https://issues.apache.org/jira/browse/FLINK-1652 Project: Flink Issue Type: Bug Components: Gelly, Iterations, Spargel Reporter: Vasia Kalavri Assignee: Vasia Kalavri When in collection execution mode, the superstep number is not correctly updated for Spargel's and Gelly's VertexCentricIteration. There seems to be to problem with DeltaIteration. See also relevant [discussion in dev@ | https://mail-archives.apache.org/mod_mbox/flink-dev/201503.mbox/%3CCAK5ODX4YiNqqSXAYrK0PAwvEDYm%2Bjakvvu8%3Dvup62H4Vwc_uMQ%40mail.gmail.com%3E]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1593) Improve exception handling int Streaming runtime
[
https://issues.apache.org/jira/browse/FLINK-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365219#comment-14365219
]
Robert Metzger commented on FLINK-1593:
---
Hey,
I'm currently testing Flink streaming and I found the
{{StreamInvokable.callUserFunctionAndLogException()}} method.
The method name indicates very much that the system is still logging exceptions
without failing the job.
Am I overseeing something or can I safely remove the method and propagate the
exception through the system?
Improve exception handling int Streaming runtime
Key: FLINK-1593
URL: https://issues.apache.org/jira/browse/FLINK-1593
Project: Flink
Issue Type: Improvement
Components: Streaming
Reporter: Gyula Fora
Assignee: Gyula Fora
Fix For: 0.9
The current exception handling logic is not very helpful when trying to debug
an application. In many cases with serialization/user code or other
exceptions, the error is just logged and not propagated properly. This should
mainly be fixed in the StreamInvokables.
Some improvements that could be made:
-Serialization/Deserialiaztion and other system errors should be propagated
instead of just silently logged
-User code exceptions should be better handled, for instance I think it would
be helpful to log them to INFO so that users can instantly see it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1695] Kick off of Flink's machine learn...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/479#issuecomment-82392282 merge it. To get some bonus points, include the JIRA id with the commits --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
