[jira] [Commented] (FLINK-1434) Web interface cannot be used to run streaming programs
[ https://issues.apache.org/jira/browse/FLINK-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14294410#comment-14294410 ] ASF GitHub Bot commented on FLINK-1434: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/334 > Web interface cannot be used to run streaming programs > -- > > Key: FLINK-1434 > URL: https://issues.apache.org/jira/browse/FLINK-1434 > Project: Flink > Issue Type: Bug > Components: Streaming, Webfrontend >Affects Versions: 0.9 >Reporter: Gyula Fora >Assignee: Gyula Fora > > Flink streaming programs currently cannot be submitted through the web > client. When you try run the jar you get a ProgramInvocationException. > The reason for this might be that streaming programs completely bypass the > use of Plans for job execution and the streaming execution environment > directly submits the jobgraph to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1434) Web interface cannot be used to run streaming programs
[ https://issues.apache.org/jira/browse/FLINK-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14291102#comment-14291102 ] ASF GitHub Bot commented on FLINK-1434: --- Github user mbalassi commented on the pull request: https://github.com/apache/flink/pull/334#issuecomment-71374393 Now it is +1 for merge on my side. > Web interface cannot be used to run streaming programs > -- > > Key: FLINK-1434 > URL: https://issues.apache.org/jira/browse/FLINK-1434 > Project: Flink > Issue Type: Bug > Components: Streaming, Webfrontend >Affects Versions: 0.9 >Reporter: Gyula Fora >Assignee: Gyula Fora > > Flink streaming programs currently cannot be submitted through the web > client. When you try run the jar you get a ProgramInvocationException. > The reason for this might be that streaming programs completely bypass the > use of Plans for job execution and the streaming execution environment > directly submits the jobgraph to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1434) Web interface cannot be used to run streaming programs
[
https://issues.apache.org/jira/browse/FLINK-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14291048#comment-14291048
]
ASF GitHub Bot commented on FLINK-1434:
---
Github user gyfora commented on a diff in the pull request:
https://github.com/apache/flink/pull/334#discussion_r23503616
--- Diff: flink-addons/flink-streaming/flink-streaming-core/pom.xml ---
@@ -48,6 +48,12 @@ under the License.
${project.version}
test
+
+
+org.apache.sling
+org.apache.sling.commons.json
+2.0.6
+
--- End diff --
This is already a dependency of the flink-connectors.
It might be good at some point to refactor the batch JSON plan generator to
use some actual JSON parser to make it look less messy
> Web interface cannot be used to run streaming programs
> --
>
> Key: FLINK-1434
> URL: https://issues.apache.org/jira/browse/FLINK-1434
> Project: Flink
> Issue Type: Bug
> Components: Streaming, Webfrontend
>Affects Versions: 0.9
>Reporter: Gyula Fora
>Assignee: Gyula Fora
>
> Flink streaming programs currently cannot be submitted through the web
> client. When you try run the jar you get a ProgramInvocationException.
> The reason for this might be that streaming programs completely bypass the
> use of Plans for job execution and the streaming execution environment
> directly submits the jobgraph to the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1434) Web interface cannot be used to run streaming programs
[
https://issues.apache.org/jira/browse/FLINK-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14291047#comment-14291047
]
ASF GitHub Bot commented on FLINK-1434:
---
Github user gyfora commented on a diff in the pull request:
https://github.com/apache/flink/pull/334#discussion_r23503611
--- Diff:
flink-addons/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/StreamGraph.java
---
@@ -536,4 +549,79 @@ public long getIterationTimeout(String vertexName) {
return iterationTimeouts.get(vertexName);
}
+ public String getOperatorName(String vertexName) {
+ return operatorNames.get(vertexName);
+ }
+
+ @Override
+ public String getStreamingPlanAsJSON() {
+
+ try {
+ JSONObject json = new JSONObject();
+ JSONArray nodes = new JSONArray();
+
+ json.put("nodes", nodes);
+
+ for (String id : operatorNames.keySet()) {
+ JSONObject node = new JSONObject();
+ nodes.put(node);
+
+ node.put("id", Integer.valueOf(id));
+ node.put("type", getOperatorName(id));
+
+ if (sources.contains(id)) {
+ node.put("pact", "Data Source");
+ } else {
+ node.put("pact", "Data Stream");
+ }
+
+ node.put("contents", getOperatorName(id) + " at
"
+ +
getInvokable(id).getUserFunction().getClass().getSimpleName());
+ node.put("parallelism", getParallelism(id));
+
+ int numIn = getInEdges(id).size();
+ if (numIn > 0) {
+
+ JSONArray inputs = new JSONArray();
+ node.put("predecessors", inputs);
+
+ for (int i = 0; i < numIn; i++) {
+
+ String inID =
getInEdges(id).get(i);
+
+ JSONObject input = new
JSONObject();
+ inputs.put(input);
+
+ input.put("id",
Integer.valueOf(inID));
+ input.put("ship_strategy",
getOutPartitioner(inID, id).getStrategy());
+ if (i == 0) {
+ input.put("side",
"first");
+ } else if (i == 1) {
+ input.put("side",
"second");
+ }
+ }
+ }
+
+ }
+ return json.toString();
+ } catch (Exception e) {
--- End diff --
Fixed
> Web interface cannot be used to run streaming programs
> --
>
> Key: FLINK-1434
> URL: https://issues.apache.org/jira/browse/FLINK-1434
> Project: Flink
> Issue Type: Bug
> Components: Streaming, Webfrontend
>Affects Versions: 0.9
>Reporter: Gyula Fora
>Assignee: Gyula Fora
>
> Flink streaming programs currently cannot be submitted through the web
> client. When you try run the jar you get a ProgramInvocationException.
> The reason for this might be that streaming programs completely bypass the
> use of Plans for job execution and the streaming execution environment
> directly submits the jobgraph to the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1434) Web interface cannot be used to run streaming programs
[
https://issues.apache.org/jira/browse/FLINK-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14291042#comment-14291042
]
ASF GitHub Bot commented on FLINK-1434:
---
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/334#discussion_r23503540
--- Diff:
flink-addons/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/StreamGraph.java
---
@@ -536,4 +549,79 @@ public long getIterationTimeout(String vertexName) {
return iterationTimeouts.get(vertexName);
}
+ public String getOperatorName(String vertexName) {
+ return operatorNames.get(vertexName);
+ }
+
+ @Override
+ public String getStreamingPlanAsJSON() {
+
+ try {
+ JSONObject json = new JSONObject();
+ JSONArray nodes = new JSONArray();
+
+ json.put("nodes", nodes);
+
+ for (String id : operatorNames.keySet()) {
+ JSONObject node = new JSONObject();
+ nodes.put(node);
+
+ node.put("id", Integer.valueOf(id));
+ node.put("type", getOperatorName(id));
+
+ if (sources.contains(id)) {
+ node.put("pact", "Data Source");
+ } else {
+ node.put("pact", "Data Stream");
+ }
+
+ node.put("contents", getOperatorName(id) + " at
"
+ +
getInvokable(id).getUserFunction().getClass().getSimpleName());
+ node.put("parallelism", getParallelism(id));
+
+ int numIn = getInEdges(id).size();
+ if (numIn > 0) {
+
+ JSONArray inputs = new JSONArray();
+ node.put("predecessors", inputs);
+
+ for (int i = 0; i < numIn; i++) {
+
+ String inID =
getInEdges(id).get(i);
+
+ JSONObject input = new
JSONObject();
+ inputs.put(input);
+
+ input.put("id",
Integer.valueOf(inID));
+ input.put("ship_strategy",
getOutPartitioner(inID, id).getStrategy());
+ if (i == 0) {
+ input.put("side",
"first");
+ } else if (i == 1) {
+ input.put("side",
"second");
+ }
+ }
+ }
+
+ }
+ return json.toString();
+ } catch (Exception e) {
--- End diff --
maybe we should at least LOG.debug() the exception?
> Web interface cannot be used to run streaming programs
> --
>
> Key: FLINK-1434
> URL: https://issues.apache.org/jira/browse/FLINK-1434
> Project: Flink
> Issue Type: Bug
> Components: Streaming, Webfrontend
>Affects Versions: 0.9
>Reporter: Gyula Fora
>Assignee: Gyula Fora
>
> Flink streaming programs currently cannot be submitted through the web
> client. When you try run the jar you get a ProgramInvocationException.
> The reason for this might be that streaming programs completely bypass the
> use of Plans for job execution and the streaming execution environment
> directly submits the jobgraph to the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1434) Web interface cannot be used to run streaming programs
[
https://issues.apache.org/jira/browse/FLINK-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14291041#comment-14291041
]
ASF GitHub Bot commented on FLINK-1434:
---
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/334#discussion_r23503538
--- Diff: flink-addons/flink-streaming/flink-streaming-core/pom.xml ---
@@ -48,6 +48,12 @@ under the License.
${project.version}
test
+
+
+org.apache.sling
+org.apache.sling.commons.json
+2.0.6
+
--- End diff --
how many new / and which transitive dependencies are added by this
dependency?
> Web interface cannot be used to run streaming programs
> --
>
> Key: FLINK-1434
> URL: https://issues.apache.org/jira/browse/FLINK-1434
> Project: Flink
> Issue Type: Bug
> Components: Streaming, Webfrontend
>Affects Versions: 0.9
>Reporter: Gyula Fora
>Assignee: Gyula Fora
>
> Flink streaming programs currently cannot be submitted through the web
> client. When you try run the jar you get a ProgramInvocationException.
> The reason for this might be that streaming programs completely bypass the
> use of Plans for job execution and the streaming execution environment
> directly submits the jobgraph to the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1434) Web interface cannot be used to run streaming programs
[ https://issues.apache.org/jira/browse/FLINK-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290875#comment-14290875 ] ASF GitHub Bot commented on FLINK-1434: --- Github user gyfora commented on the pull request: https://github.com/apache/flink/pull/334#issuecomment-71342807 Travis passes, just some time-out error :/ > Web interface cannot be used to run streaming programs > -- > > Key: FLINK-1434 > URL: https://issues.apache.org/jira/browse/FLINK-1434 > Project: Flink > Issue Type: Bug > Components: Streaming, Webfrontend >Affects Versions: 0.9 >Reporter: Gyula Fora >Assignee: Gyula Fora > > Flink streaming programs currently cannot be submitted through the web > client. When you try run the jar you get a ProgramInvocationException. > The reason for this might be that streaming programs completely bypass the > use of Plans for job execution and the streaming execution environment > directly submits the jobgraph to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1434) Web interface cannot be used to run streaming programs
[ https://issues.apache.org/jira/browse/FLINK-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290830#comment-14290830 ] ASF GitHub Bot commented on FLINK-1434: --- GitHub user gyfora opened a pull request: https://github.com/apache/flink/pull/334 [FLINK-1434] [FLINK-1401] Streaming support added for webclient This pull request enables the Flink webclient to execute streaming programs, and also adds support for plan visualisation for streaming programs. The main modification was to add a common interface for both batch and streaming optimized plans, called FlinkPlan. Both OptimizedPlan and StreamingPlan implements this common interface. The methods in the Client, PackagedProgram and webclient have been modified to detect whether it is a Streaming or a Batch plan and take actions accordingly. These should be refined later when we bring the streaming package out of the flink-addons, hopefully for the 0.9 release. I also added a simple JSON plan generator for streaming programs. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mbalassi/flink webclient Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/334.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #334 commit ac29dfea400ac438b9acb44b4b4cd7431c62b0cd Author: Gyula Fora Date: 2015-01-24T21:12:03Z [FLINK-1434] [FLINK-1401] Streaming support added for webclient > Web interface cannot be used to run streaming programs > -- > > Key: FLINK-1434 > URL: https://issues.apache.org/jira/browse/FLINK-1434 > Project: Flink > Issue Type: Bug > Components: Streaming, Webfrontend >Affects Versions: 0.9 >Reporter: Gyula Fora >Assignee: Gyula Fora > > Flink streaming programs currently cannot be submitted through the web > client. When you try run the jar you get a ProgramInvocationException. > The reason for this might be that streaming programs completely bypass the > use of Plans for job execution and the streaming execution environment > directly submits the jobgraph to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1434) Web interface cannot be used to run streaming programs
[ https://issues.apache.org/jira/browse/FLINK-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287514#comment-14287514 ] Gyula Fora commented on FLINK-1434: --- It seems like I was unaware of the OptimizerPlanEnvironment when I implemented the StreamExecutionEnvironment so it is probably trying to start a minicluster when invoked from the webclient. > Web interface cannot be used to run streaming programs > -- > > Key: FLINK-1434 > URL: https://issues.apache.org/jira/browse/FLINK-1434 > Project: Flink > Issue Type: Bug > Components: Streaming, Webfrontend >Affects Versions: 0.9 >Reporter: Gyula Fora > > Flink streaming programs currently cannot be submitted through the web > client. When you try run the jar you get a ProgramInvocationException. > The reason for this might be that streaming programs completely bypass the > use of Plans for job execution and the streaming execution environment > directly submits the jobgraph to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1434) Web interface cannot be used to run streaming programs
[ https://issues.apache.org/jira/browse/FLINK-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287352#comment-14287352 ] Gyula Fora commented on FLINK-1434: --- This actually seems to be a ClassLoader issue, but I wonder why don't have the same issue when running from the command line client. [~uce] or [~StephanEwen] any ideas here? > Web interface cannot be used to run streaming programs > -- > > Key: FLINK-1434 > URL: https://issues.apache.org/jira/browse/FLINK-1434 > Project: Flink > Issue Type: Bug > Components: Streaming, Webfrontend >Affects Versions: 0.9 >Reporter: Gyula Fora > > Flink streaming programs currently cannot be submitted through the web > client. When you try run the jar you get a ProgramInvocationException. > The reason for this might be that streaming programs completely bypass the > use of Plans for job execution and the streaming execution environment > directly submits the jobgraph to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1434) Web interface cannot be used to run streaming programs
[ https://issues.apache.org/jira/browse/FLINK-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287289#comment-14287289 ] Gyula Fora commented on FLINK-1434: --- This issue is also linked with https://issues.apache.org/jira/browse/FLINK-1401 > Web interface cannot be used to run streaming programs > -- > > Key: FLINK-1434 > URL: https://issues.apache.org/jira/browse/FLINK-1434 > Project: Flink > Issue Type: Bug > Components: Streaming, Webfrontend >Affects Versions: 0.9 >Reporter: Gyula Fora > > Flink streaming programs currently cannot be submitted through the web > client. When you try run the jar you get a ProgramInvocationException. > The reason for this might be that streaming programs completely bypass the > use of Plans for job execution and the streaming execution environment > directly submits the jobgraph to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
