[jira] [Commented] (FLINK-17775) Cannot set batch job name when using collect
[
https://issues.apache.org/jira/browse/FLINK-17775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17355626#comment-17355626
]
Nikola commented on FLINK-17775:
Hi [~paul8263] that would be great!
> Cannot set batch job name when using collect
>
>
> Key: FLINK-17775
> URL: https://issues.apache.org/jira/browse/FLINK-17775
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.8.3, 1.9.3, 1.10.1
>Reporter: Nikola
>Priority: Major
> Labels: auto-deprioritized-critical, stale-major
>
> We have a batch job in the likes of this:
>
> {code:java}
> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
> DataSet dataSet = getDataSet();
> dataSet
> .sortPartition(MyRow::getCount, Order.DESCENDING)
> .setParallelism(1)
> .flatMap(new MyFlatMap())
> .collect();
> env.execute("Job at " + Instant.now().toString());
> {code}
> However, the job name in the flink UI is not "Job at " but the default
> as if I didn't put anything.
>
> Is there way to have my own flink job name?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17775) Cannot set batch job name when using collect
[
https://issues.apache.org/jira/browse/FLINK-17775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17355606#comment-17355606
]
Yao Zhang commented on FLINK-17775:
---
Hi [~nikobearrr],
I investigated the corresponding part in DataStream.java:
{code:java}
public CloseableIterator executeAndCollect() throws Exception {
return executeAndCollect("DataStream Collect");
}
public CloseableIterator executeAndCollect(String jobExecutionName)
throws Exception {
return executeAndCollectWithClient(jobExecutionName).iterator;
}
{code}
It provides two implementations, with one of them can specify the job execution
name. But in DataSet.java we cannot assign that name.
I think we can add a collect(String jobExecutionName) method for DataSet.
Could you please assign this issue to me?
> Cannot set batch job name when using collect
>
>
> Key: FLINK-17775
> URL: https://issues.apache.org/jira/browse/FLINK-17775
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.8.3, 1.9.3, 1.10.1
>Reporter: Nikola
>Priority: Major
> Labels: auto-deprioritized-critical, stale-major
>
> We have a batch job in the likes of this:
>
> {code:java}
> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
> DataSet dataSet = getDataSet();
> dataSet
> .sortPartition(MyRow::getCount, Order.DESCENDING)
> .setParallelism(1)
> .flatMap(new MyFlatMap())
> .collect();
> env.execute("Job at " + Instant.now().toString());
> {code}
> However, the job name in the flink UI is not "Job at " but the default
> as if I didn't put anything.
>
> Is there way to have my own flink job name?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17775) Cannot set batch job name when using collect
[
https://issues.apache.org/jira/browse/FLINK-17775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17336964#comment-17336964
]
Flink Jira Bot commented on FLINK-17775:
This issue was labeled "stale-critical" 7 ago and has not received any updates
so it is being deprioritized. If this ticket is actually Critical, please raise
the priority and ask a committer to assign you the issue or revive the public
discussion.
> Cannot set batch job name when using collect
>
>
> Key: FLINK-17775
> URL: https://issues.apache.org/jira/browse/FLINK-17775
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.8.3, 1.9.3, 1.10.1
>Reporter: Nikola
>Priority: Critical
> Labels: stale-critical
>
> We have a batch job in the likes of this:
>
> {code:java}
> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
> DataSet dataSet = getDataSet();
> dataSet
> .sortPartition(MyRow::getCount, Order.DESCENDING)
> .setParallelism(1)
> .flatMap(new MyFlatMap())
> .collect();
> env.execute("Job at " + Instant.now().toString());
> {code}
> However, the job name in the flink UI is not "Job at " but the default
> as if I didn't put anything.
>
> Is there way to have my own flink job name?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17775) Cannot set batch job name when using collect
[
https://issues.apache.org/jira/browse/FLINK-17775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17329038#comment-17329038
]
Flink Jira Bot commented on FLINK-17775:
This critical issue is unassigned and itself and all of its Sub-Tasks have not
been updated for 7 days. So, it has been labeled "stale-critical". If this
ticket is indeed critical, please either assign yourself or give an update.
Afterwards, please remove the label. In 7 days the issue will be deprioritized.
> Cannot set batch job name when using collect
>
>
> Key: FLINK-17775
> URL: https://issues.apache.org/jira/browse/FLINK-17775
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.8.3, 1.9.3, 1.10.1
>Reporter: Nikola
>Priority: Critical
> Labels: stale-critical
>
> We have a batch job in the likes of this:
>
> {code:java}
> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
> DataSet dataSet = getDataSet();
> dataSet
> .sortPartition(MyRow::getCount, Order.DESCENDING)
> .setParallelism(1)
> .flatMap(new MyFlatMap())
> .collect();
> env.execute("Job at " + Instant.now().toString());
> {code}
> However, the job name in the flink UI is not "Job at " but the default
> as if I didn't put anything.
>
> Is there way to have my own flink job name?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17775) Cannot set batch job name when using collect
[
https://issues.apache.org/jira/browse/FLINK-17775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17226842#comment-17226842
]
Nikola commented on FLINK-17775:
Hi [~zhuzh], this is still an issue with flink 1.11.2
If we run a job like that:
dataSet
.sortPartition(MyRow::getCount, Order.DESCENDING)
.setParallelism(1)
.flatMap(new MyFlatMap())
.collect();
There is no possibility to set the flink job name. The jobname becomes
something like this: "Flink Java Job at Thu Nov 05 16:56:54 UTC 2020"
> Cannot set batch job name when using collect
>
>
> Key: FLINK-17775
> URL: https://issues.apache.org/jira/browse/FLINK-17775
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.8.3, 1.9.3, 1.10.1
>Reporter: Nikola
>Priority: Critical
>
> We have a batch job in the likes of this:
>
> {code:java}
> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
> DataSet dataSet = getDataSet();
> dataSet
> .sortPartition(MyRow::getCount, Order.DESCENDING)
> .setParallelism(1)
> .flatMap(new MyFlatMap())
> .collect();
> env.execute("Job at " + Instant.now().toString());
> {code}
> However, the job name in the flink UI is not "Job at " but the default
> as if I didn't put anything.
>
> Is there way to have my own flink job name?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17775) Cannot set batch job name when using collect
[
https://issues.apache.org/jira/browse/FLINK-17775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17178480#comment-17178480
]
Zhu Zhu commented on FLINK-17775:
-
[~aljoscha] just to confirm, is this no longer a problem since 1.11 and should
we close this issue as won't fix?
> Cannot set batch job name when using collect
>
>
> Key: FLINK-17775
> URL: https://issues.apache.org/jira/browse/FLINK-17775
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.8.3, 1.9.3, 1.10.1
>Reporter: Nikola
>Priority: Critical
>
> We have a batch job in the likes of this:
>
> {code:java}
> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
> DataSet dataSet = getDataSet();
> dataSet
> .sortPartition(MyRow::getCount, Order.DESCENDING)
> .setParallelism(1)
> .flatMap(new MyFlatMap())
> .collect();
> env.execute("Job at " + Instant.now().toString());
> {code}
> However, the job name in the flink UI is not "Job at " but the default
> as if I didn't put anything.
>
> Is there way to have my own flink job name?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17775) Cannot set batch job name when using collect
[
https://issues.apache.org/jira/browse/FLINK-17775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17113996#comment-17113996
]
Aljoscha Krettek commented on FLINK-17775:
--
Indeed, there is currently no way of setting a Job name when using
{{collect()}}.
You job is not executed twice because of how both Flink Web UI submission and
the Docker entrypoints have worked for quite a while now: Inside the
{{execute()}} call we throw an exception and catch it in the surrounding code
to get the job out from a program. You can see this here, for example:
https://github.com/apache/flink/blob/release-1.10/flink-clients/src/main/java/org/apache/flink/client/program/OptimizerPlanEnvironment.java.
This is changed with the implementation of FLINK-16654.
> Cannot set batch job name when using collect
>
>
> Key: FLINK-17775
> URL: https://issues.apache.org/jira/browse/FLINK-17775
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.8.3, 1.9.3, 1.10.1
>Reporter: Nikola
>Priority: Critical
>
> We have a batch job in the likes of this:
>
> {code:java}
> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
> DataSet dataSet = getDataSet();
> dataSet
> .sortPartition(MyRow::getCount, Order.DESCENDING)
> .setParallelism(1)
> .flatMap(new MyFlatMap())
> .collect();
> env.execute("Job at " + Instant.now().toString());
> {code}
> However, the job name in the flink UI is not "Job at " but the default
> as if I didn't put anything.
>
> Is there way to have my own flink job name?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17775) Cannot set batch job name when using collect
[
https://issues.apache.org/jira/browse/FLINK-17775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112176#comment-17112176
]
Nikola commented on FLINK-17775:
Hi [~aljoscha], that seems to be my bad. I can indeed remove the last
{{env.execute()}} and my job will work just fine. I am using flink in docker
which we start through {{bin/taskmanager.sh}} and {{bin/jobmanager.sh}}
However, regarding the issue it seems there is no way around it at the moment
as the code you point to does not take anyhow a job name in consideration.
On the other hand, When I am using both .collect() and env.execute() you said
my job will run twice. However, I cannot see my job running twice (or 2 jobs
running). I can see only one.
> Cannot set batch job name when using collect
>
>
> Key: FLINK-17775
> URL: https://issues.apache.org/jira/browse/FLINK-17775
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.8.3, 1.9.3, 1.10.1
>Reporter: Nikola
>Priority: Critical
>
> We have a batch job in the likes of this:
>
> {code:java}
> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
> DataSet dataSet = getDataSet();
> dataSet
> .sortPartition(MyRow::getCount, Order.DESCENDING)
> .setParallelism(1)
> .flatMap(new MyFlatMap())
> .collect();
> env.execute("Job at " + Instant.now().toString());
> {code}
> However, the job name in the flink UI is not "Job at " but the default
> as if I didn't put anything.
>
> Is there way to have my own flink job name?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17775) Cannot set batch job name when using collect
[
https://issues.apache.org/jira/browse/FLINK-17775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110243#comment-17110243
]
Aljoscha Krettek commented on FLINK-17775:
--
How are you executing the job? In the IDE, or with {{bin/flink}}?
> Cannot set batch job name when using collect
>
>
> Key: FLINK-17775
> URL: https://issues.apache.org/jira/browse/FLINK-17775
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.8.3, 1.9.3, 1.10.1
>Reporter: Nikola
>Priority: Critical
>
> We have a batch job in the likes of this:
>
> {code:java}
> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
> DataSet dataSet = getDataSet();
> dataSet
> .sortPartition(MyRow::getCount, Order.DESCENDING)
> .setParallelism(1)
> .flatMap(new MyFlatMap())
> .collect();
> env.execute("Job at " + Instant.now().toString());
> {code}
> However, the job name in the flink UI is not "Job at " but the default
> as if I didn't put anything.
>
> Is there way to have my own flink job name?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17775) Cannot set batch job name when using collect
[
https://issues.apache.org/jira/browse/FLINK-17775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110176#comment-17110176
]
Nikola commented on FLINK-17775:
Hi [~aljoscha], I have tried running without the env.execute() but then I
cannot see my job starting? What would the correct way be?
> Cannot set batch job name when using collect
>
>
> Key: FLINK-17775
> URL: https://issues.apache.org/jira/browse/FLINK-17775
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.8.3, 1.9.3, 1.10.1
>Reporter: Nikola
>Priority: Critical
>
> We have a batch job in the likes of this:
>
> {code:java}
> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
> DataSet dataSet = getDataSet();
> dataSet
> .sortPartition(MyRow::getCount, Order.DESCENDING)
> .setParallelism(1)
> .flatMap(new MyFlatMap())
> .collect();
> env.execute("Job at " + Instant.now().toString());
> {code}
> However, the job name in the flink UI is not "Job at " but the default
> as if I didn't put anything.
>
> Is there way to have my own flink job name?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17775) Cannot set batch job name when using collect
[
https://issues.apache.org/jira/browse/FLINK-17775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110173#comment-17110173
]
Aljoscha Krettek commented on FLINK-17775:
--
This is a different issue, {{collect()}} will internally call {{execute()}} on
the environment without giving a name:
https://github.com/apache/flink/blob/fd91220008d29411d8fcc1ffa95eeddc3e6d756a/flink-java/src/main/java/org/apache/flink/api/java/DataSet.java#L413
In your example, your job would execute twice, but the second time without any
operators.
> Cannot set batch job name when using collect
>
>
> Key: FLINK-17775
> URL: https://issues.apache.org/jira/browse/FLINK-17775
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.8.3, 1.9.3, 1.10.1
>Reporter: Nikola
>Priority: Critical
>
> We have a batch job in the likes of this:
>
> {code:java}
> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
> DataSet dataSet = getDataSet();
> dataSet
> .sortPartition(MyRow::getCount, Order.DESCENDING)
> .setParallelism(1)
> .flatMap(new MyFlatMap())
> .collect();
> env.execute("Job at " + Instant.now().toString());
> {code}
> However, the job name in the flink UI is not "Job at " but the default
> as if I didn't put anything.
>
> Is there way to have my own flink job name?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17775) Cannot set batch job name when using collect
[
https://issues.apache.org/jira/browse/FLINK-17775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109521#comment-17109521
]
Benchao Li commented on FLINK-17775:
You can try the patch from FLINK-17615. If it works for your case, then could
you close this issue?
> Cannot set batch job name when using collect
>
>
> Key: FLINK-17775
> URL: https://issues.apache.org/jira/browse/FLINK-17775
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.8.3, 1.9.3, 1.10.1
>Reporter: Nikola
>Priority: Critical
>
> We have a batch job in the likes of this:
>
> {code:java}
> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
> DataSet dataSet = getDataSet();
> dataSet
> .sortPartition(MyRow::getCount, Order.DESCENDING)
> .setParallelism(1)
> .flatMap(new MyFlatMap())
> .collect();
> env.execute("Job at " + Instant.now().toString());
> {code}
> However, the job name in the flink UI is not "Job at " but the default
> as if I didn't put anything.
>
> Is there way to have my own flink job name?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17775) Cannot set batch job name when using collect
[
https://issues.apache.org/jira/browse/FLINK-17775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109508#comment-17109508
]
Nikola commented on FLINK-17775:
Hi [~libenchao] it looks like it is. I am not sure how the user there submits
their job, but it's the same end result.
> Cannot set batch job name when using collect
>
>
> Key: FLINK-17775
> URL: https://issues.apache.org/jira/browse/FLINK-17775
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.8.3, 1.9.3, 1.10.1
>Reporter: Nikola
>Priority: Critical
>
> We have a batch job in the likes of this:
>
> {code:java}
> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
> DataSet dataSet = getDataSet();
> dataSet
> .sortPartition(MyRow::getCount, Order.DESCENDING)
> .setParallelism(1)
> .flatMap(new MyFlatMap())
> .collect();
> env.execute("Job at " + Instant.now().toString());
> {code}
> However, the job name in the flink UI is not "Job at " but the default
> as if I didn't put anything.
>
> Is there way to have my own flink job name?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17775) Cannot set batch job name when using collect
[
https://issues.apache.org/jira/browse/FLINK-17775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109505#comment-17109505
]
Benchao Li commented on FLINK-17775:
[~nikobearrr] Does this is a duplicate of
https://issues.apache.org/jira/browse/FLINK-17615 ?
> Cannot set batch job name when using collect
>
>
> Key: FLINK-17775
> URL: https://issues.apache.org/jira/browse/FLINK-17775
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.8.3, 1.9.3, 1.10.1
>Reporter: Nikola
>Priority: Critical
>
> We have a batch job in the likes of this:
>
> {code:java}
> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
> DataSet dataSet = getDataSet();
> dataSet
> .sortPartition(MyRow::getCount, Order.DESCENDING)
> .setParallelism(1)
> .flatMap(new MyFlatMap())
> .collect();
> env.execute("Job at " + Instant.now().toString());
> {code}
> However, the job name in the flink UI is not "Job at " but the default
> as if I didn't put anything.
>
> Is there way to have my own flink job name?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
