[jira] [Commented] (BEAM-1107) Display user names for steps in the Flink Web UI
[ https://issues.apache.org/jira/browse/BEAM-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730375#comment-15730375 ] Aljoscha Krettek commented on BEAM-1107: Yep, you're right but even in the black text the operation names (MapPartition, GroupCombine and so on) are hardcoded in Flink right now so we cannot change that coming from Beam-on-Flink. Changing that would require changes to Flink (which I'm not opposed to). > Display user names for steps in the Flink Web UI > > > Key: BEAM-1107 > URL: https://issues.apache.org/jira/browse/BEAM-1107 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Daniel Halperin >Assignee: Aljoscha Krettek > Attachments: screenshot-1.png > > > [copying in-person / email discussion at Strata Singapore to JIRA] > The FlinkBatchTransformTranslators use transform.getName() [1] -- this is the > "SDK name" for the transform. > The "user name" for the transform is not available here, it is in fact on the > TransformHierarchy.Node as node.getFullName() [2]. > getFullName() is used some in Flink, but not when setting step names. > I drafted a quick commit that sort of propagates the user names to the web UI > (but only for DataSource, and still too verbose: > https://github.com/dhalperi/incubator-beam/commit/a2f1fb06b22a85ec738e4f2a604c9a129891916c) > Before this change, the "ReadLines" step showed up as: "DataSource (at > Read(CompressedSource) > (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat))" > With this change, it shows up as "DataSource (at ReadLines/Read > (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat))" > which I think is closer. [I'd still like it to JUST be "ReadLines/Read" e.g.]. > Thoughts? > [1] > https://github.com/apache/incubator-beam/blob/master/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java#L129 > [2] > https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/runners/TransformHierarchy.java#L252 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-1107) Display user names for steps in the Flink Web UI
[ https://issues.apache.org/jira/browse/BEAM-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730313#comment-15730313 ] Daniel Halperin commented on BEAM-1107: --- Ack -- I guess I have this intuition there's opportunity for more cleanup, but I may be wrong (or it may be a Flink-general, not Beam-on-Flink issue). E.g., look at the attached screenshot: * The name (grey) at the top is MapPartition -> Map -> GroupCombine -> Map * The name of the steps (black) includes the identical as the grey, with additionally (step name) * The Operation: text (small, grey) at the bottom includes the same (almost - logical vs physical?) information, although there appears to be some HTML error with inserting a break tag inside another break tag. > Display user names for steps in the Flink Web UI > > > Key: BEAM-1107 > URL: https://issues.apache.org/jira/browse/BEAM-1107 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Daniel Halperin >Assignee: Aljoscha Krettek > Attachments: screenshot-1.png > > > [copying in-person / email discussion at Strata Singapore to JIRA] > The FlinkBatchTransformTranslators use transform.getName() [1] -- this is the > "SDK name" for the transform. > The "user name" for the transform is not available here, it is in fact on the > TransformHierarchy.Node as node.getFullName() [2]. > getFullName() is used some in Flink, but not when setting step names. > I drafted a quick commit that sort of propagates the user names to the web UI > (but only for DataSource, and still too verbose: > https://github.com/dhalperi/incubator-beam/commit/a2f1fb06b22a85ec738e4f2a604c9a129891916c) > Before this change, the "ReadLines" step showed up as: "DataSource (at > Read(CompressedSource) > (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat))" > With this change, it shows up as "DataSource (at ReadLines/Read > (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat))" > which I think is closer. [I'd still like it to JUST be "ReadLines/Read" e.g.]. > Thoughts? > [1] > https://github.com/apache/incubator-beam/blob/master/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java#L129 > [2] > https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/runners/TransformHierarchy.java#L252 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-1107) Display user names for steps in the Flink Web UI
[
https://issues.apache.org/jira/browse/BEAM-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730212#comment-15730212
]
Daniel Halperin commented on BEAM-1107:
---
Also copying [~aljoscha]'s response :)
{quote}
I think we can get it down to "Data Source (ReadLines/Read)" (and similarly for
other operators). The problem is that the String parameter is not the correct
way to set the name of the operator but some other (admittedly weird) thing
called "location name". To set the name we have to call .name(String) on the
created operator after creating it.
{quote}
> Display user names for steps in the Flink Web UI
>
>
> Key: BEAM-1107
> URL: https://issues.apache.org/jira/browse/BEAM-1107
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Daniel Halperin
>Assignee: Aljoscha Krettek
>
> [copying in-person / email discussion at Strata Singapore to JIRA]
> The FlinkBatchTransformTranslators use transform.getName() [1] -- this is the
> "SDK name" for the transform.
> The "user name" for the transform is not available here, it is in fact on the
> TransformHierarchy.Node as node.getFullName() [2].
> getFullName() is used some in Flink, but not when setting step names.
> I drafted a quick commit that sort of propagates the user names to the web UI
> (but only for DataSource, and still too verbose:
> https://github.com/dhalperi/incubator-beam/commit/a2f1fb06b22a85ec738e4f2a604c9a129891916c)
> Before this change, the "ReadLines" step showed up as: "DataSource (at
> Read(CompressedSource)
> (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat))"
> With this change, it shows up as "DataSource (at ReadLines/Read
> (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat))"
> which I think is closer. [I'd still like it to JUST be "ReadLines/Read" e.g.].
> Thoughts?
> [1]
> https://github.com/apache/incubator-beam/blob/master/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java#L129
> [2]
> https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/runners/TransformHierarchy.java#L252
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
