[jira] [Reopened] (FLINK-5486) Lack of synchronization in BucketingSink#handleRestoredBucketState()
[
https://issues.apache.org/jira/browse/FLINK-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao reopened FLINK-5486:
-
> Lack of synchronization in BucketingSink#handleRestoredBucketState()
>
>
> Key: FLINK-5486
> URL: https://issues.apache.org/jira/browse/FLINK-5486
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
>Affects Versions: 1.5.0, 1.4.2, 1.6.0
>Reporter: Ted Yu
>Assignee: zhangminglei
>Priority: Major
>
> Here is related code:
> {code}
>
> handlePendingFilesForPreviousCheckpoints(bucketState.pendingFilesPerCheckpoint);
> synchronized (bucketState.pendingFilesPerCheckpoint) {
> bucketState.pendingFilesPerCheckpoint.clear();
> }
> {code}
> The handlePendingFilesForPreviousCheckpoints() call should be enclosed inside
> the synchronization block. Otherwise during the processing of
> handlePendingFilesForPreviousCheckpoints(), some entries of the map may be
> cleared.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-5486) Lack of synchronization in BucketingSink#handleRestoredBucketState()
[
https://issues.apache.org/jira/browse/FLINK-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao closed FLINK-5486.
---
Resolution: Not A Problem
Closed for the same reason as explained
[here|https://issues.apache.org/jira/browse/FLINK-4534?focusedCommentId=16550597&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16550597].
> Lack of synchronization in BucketingSink#handleRestoredBucketState()
>
>
> Key: FLINK-5486
> URL: https://issues.apache.org/jira/browse/FLINK-5486
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
>Affects Versions: 1.5.0, 1.4.2, 1.6.0
>Reporter: Ted Yu
>Assignee: zhangminglei
>Priority: Major
>
> Here is related code:
> {code}
>
> handlePendingFilesForPreviousCheckpoints(bucketState.pendingFilesPerCheckpoint);
> synchronized (bucketState.pendingFilesPerCheckpoint) {
> bucketState.pendingFilesPerCheckpoint.clear();
> }
> {code}
> The handlePendingFilesForPreviousCheckpoints() call should be enclosed inside
> the synchronization block. Otherwise during the processing of
> handlePendingFilesForPreviousCheckpoints(), some entries of the map may be
> cleared.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-5486) Lack of synchronization in BucketingSink#handleRestoredBucketState()
[
https://issues.apache.org/jira/browse/FLINK-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649304#comment-16649304
]
Gary Yao edited comment on FLINK-5486 at 10/14/18 9:02 AM:
---
Closed for the same reason as explained
[here|https://issues.apache.org/jira/browse/FLINK-4534?focusedCommentId=16550597&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16550597].
Please reopen if you think otherwise.
was (Author: gjy):
Closed for the same reason as explained
[here|https://issues.apache.org/jira/browse/FLINK-4534?focusedCommentId=16550597&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16550597].
> Lack of synchronization in BucketingSink#handleRestoredBucketState()
>
>
> Key: FLINK-5486
> URL: https://issues.apache.org/jira/browse/FLINK-5486
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
>Affects Versions: 1.5.0, 1.4.2, 1.6.0
>Reporter: Ted Yu
>Assignee: zhangminglei
>Priority: Major
>
> Here is related code:
> {code}
>
> handlePendingFilesForPreviousCheckpoints(bucketState.pendingFilesPerCheckpoint);
> synchronized (bucketState.pendingFilesPerCheckpoint) {
> bucketState.pendingFilesPerCheckpoint.clear();
> }
> {code}
> The handlePendingFilesForPreviousCheckpoints() call should be enclosed inside
> the synchronization block. Otherwise during the processing of
> handlePendingFilesForPreviousCheckpoints(), some entries of the map may be
> cleared.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-5486) Lack of synchronization in BucketingSink#handleRestoredBucketState()
[
https://issues.apache.org/jira/browse/FLINK-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao closed FLINK-5486.
---
Resolution: Not A Problem
> Lack of synchronization in BucketingSink#handleRestoredBucketState()
>
>
> Key: FLINK-5486
> URL: https://issues.apache.org/jira/browse/FLINK-5486
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
>Affects Versions: 1.5.0, 1.4.2, 1.6.0
>Reporter: Ted Yu
>Assignee: zhangminglei
>Priority: Major
>
> Here is related code:
> {code}
>
> handlePendingFilesForPreviousCheckpoints(bucketState.pendingFilesPerCheckpoint);
> synchronized (bucketState.pendingFilesPerCheckpoint) {
> bucketState.pendingFilesPerCheckpoint.clear();
> }
> {code}
> The handlePendingFilesForPreviousCheckpoints() call should be enclosed inside
> the synchronization block. Otherwise during the processing of
> handlePendingFilesForPreviousCheckpoints(), some entries of the map may be
> cleared.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] GJL commented on issue #4356: [FLINK-5486] Fix lacking of synchronization in BucketingSink#handleRe…
GJL commented on issue #4356: [FLINK-5486] Fix lacking of synchronization in BucketingSink#handleRe… URL: https://github.com/apache/flink/pull/4356#issuecomment-429608448 I am closing this PR because [FLINK-5486](https://issues.apache.org/jira/browse/FLINK-5486?focusedCommentId=16649304&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16649304) got closed. Please reopen the ticket/PR if you disagree. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-5486) Lack of synchronization in BucketingSink#handleRestoredBucketState()
[
https://issues.apache.org/jira/browse/FLINK-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649306#comment-16649306
]
ASF GitHub Bot commented on FLINK-5486:
---
GJL closed pull request #4356: [FLINK-5486] Fix lacking of synchronization in
BucketingSink#handleRe…
URL: https://github.com/apache/flink/pull/4356
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java
b/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java
index 55400c6ce19..bd5058a4fdf 100644
---
a/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java
+++
b/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java
@@ -719,8 +719,9 @@ private void handleRestoredBucketState(State
restoredState) {
bucketState.isWriterOpen = false;
handlePendingFilesForPreviousCheckpoints(bucketState.pendingFilesPerCheckpoint);
-
- bucketState.pendingFilesPerCheckpoint.clear();
+ synchronized (bucketState.pendingFilesPerCheckpoint) {
+ bucketState.pendingFilesPerCheckpoint.clear();
+ }
}
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Lack of synchronization in BucketingSink#handleRestoredBucketState()
>
>
> Key: FLINK-5486
> URL: https://issues.apache.org/jira/browse/FLINK-5486
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
>Affects Versions: 1.5.0, 1.4.2, 1.6.0
>Reporter: Ted Yu
>Assignee: zhangminglei
>Priority: Major
>
> Here is related code:
> {code}
>
> handlePendingFilesForPreviousCheckpoints(bucketState.pendingFilesPerCheckpoint);
> synchronized (bucketState.pendingFilesPerCheckpoint) {
> bucketState.pendingFilesPerCheckpoint.clear();
> }
> {code}
> The handlePendingFilesForPreviousCheckpoints() call should be enclosed inside
> the synchronization block. Otherwise during the processing of
> handlePendingFilesForPreviousCheckpoints(), some entries of the map may be
> cleared.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] GJL closed pull request #4356: [FLINK-5486] Fix lacking of synchronization in BucketingSink#handleRe…
GJL closed pull request #4356: [FLINK-5486] Fix lacking of synchronization in
BucketingSink#handleRe…
URL: https://github.com/apache/flink/pull/4356
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java
b/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java
index 55400c6ce19..bd5058a4fdf 100644
---
a/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java
+++
b/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java
@@ -719,8 +719,9 @@ private void handleRestoredBucketState(State
restoredState) {
bucketState.isWriterOpen = false;
handlePendingFilesForPreviousCheckpoints(bucketState.pendingFilesPerCheckpoint);
-
- bucketState.pendingFilesPerCheckpoint.clear();
+ synchronized (bucketState.pendingFilesPerCheckpoint) {
+ bucketState.pendingFilesPerCheckpoint.clear();
+ }
}
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-5486) Lack of synchronization in BucketingSink#handleRestoredBucketState()
[
https://issues.apache.org/jira/browse/FLINK-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649305#comment-16649305
]
ASF GitHub Bot commented on FLINK-5486:
---
GJL commented on issue #4356: [FLINK-5486] Fix lacking of synchronization in
BucketingSink#handleRe…
URL: https://github.com/apache/flink/pull/4356#issuecomment-429608448
I am closing this PR because
[FLINK-5486](https://issues.apache.org/jira/browse/FLINK-5486?focusedCommentId=16649304&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16649304)
got closed. Please reopen the ticket/PR if you disagree.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Lack of synchronization in BucketingSink#handleRestoredBucketState()
>
>
> Key: FLINK-5486
> URL: https://issues.apache.org/jira/browse/FLINK-5486
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
>Affects Versions: 1.5.0, 1.4.2, 1.6.0
>Reporter: Ted Yu
>Assignee: zhangminglei
>Priority: Major
>
> Here is related code:
> {code}
>
> handlePendingFilesForPreviousCheckpoints(bucketState.pendingFilesPerCheckpoint);
> synchronized (bucketState.pendingFilesPerCheckpoint) {
> bucketState.pendingFilesPerCheckpoint.clear();
> }
> {code}
> The handlePendingFilesForPreviousCheckpoints() call should be enclosed inside
> the synchronization block. Otherwise during the processing of
> handlePendingFilesForPreviousCheckpoints(), some entries of the map may be
> cleared.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10543) Leverage efficient timer deletion in relational operators
Fabian Hueske created FLINK-10543: - Summary: Leverage efficient timer deletion in relational operators Key: FLINK-10543 URL: https://issues.apache.org/jira/browse/FLINK-10543 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: Fabian Hueske FLINK-9423 added support for efficient timer deletions. This feature is available since Flink 1.6 and should be used by the relational operator of SQL and Table API. Currently, we use a few workarounds to handle situations when deleting timers would be the better solution. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] zentol commented on issue #6542: [FLINK-6437][History Server] Move history server configuration to a separate file
zentol commented on issue #6542: [FLINK-6437][History Server] Move history server configuration to a separate file URL: https://github.com/apache/flink/pull/6542#issuecomment-429609834 My opinion hasn't changed, I still don't think that it is worth it. Separating the HistoryServer from your cluster environment is already trivial: just copy flink-dist to separate locations. The downside here is you end up with a plethora of unused jars in the HistoryServer copy, but I'd argue that the HistoryServer shouldn't be bundled with flink-dist in such a messy way and instead be distributed as a standalone application. However this shouldn't be discussed in this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-6437) Move history server configuration to a separate file
[
https://issues.apache.org/jira/browse/FLINK-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649311#comment-16649311
]
ASF GitHub Bot commented on FLINK-6437:
---
zentol commented on issue #6542: [FLINK-6437][History Server] Move history
server configuration to a separate file
URL: https://github.com/apache/flink/pull/6542#issuecomment-429609834
My opinion hasn't changed, I still don't think that it is worth it.
Separating the HistoryServer from your cluster environment is already
trivial: just copy flink-dist to separate locations.
The downside here is you end up with a plethora of unused jars in the
HistoryServer copy, but I'd argue that the HistoryServer shouldn't be bundled
with flink-dist in such a messy way and instead be distributed as a standalone
application. However this shouldn't be discussed in this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move history server configuration to a separate file
>
>
> Key: FLINK-6437
> URL: https://issues.apache.org/jira/browse/FLINK-6437
> Project: Flink
> Issue Type: Improvement
> Components: History Server
>Affects Versions: 1.3.0
>Reporter: Stephan Ewen
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> I suggest to keep the {{flink-conf.yaml}} leaner by moving configuration of
> the History Server to a different file.
> In general, I would propose to move configurations of separate, independent
> and optional components to individual config files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] Guibo-Pan commented on issue #6773: [FLINK-10152] [table] Add reverse function in Table API and SQL
Guibo-Pan commented on issue #6773: [FLINK-10152] [table] Add reverse function in Table API and SQL URL: https://github.com/apache/flink/pull/6773#issuecomment-429611382 @pnowojski Could you help review this PR? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10152) Add reverse supported in TableAPI and SQL
[
https://issues.apache.org/jira/browse/FLINK-10152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649319#comment-16649319
]
ASF GitHub Bot commented on FLINK-10152:
Guibo-Pan commented on issue #6773: [FLINK-10152] [table] Add reverse function
in Table API and SQL
URL: https://github.com/apache/flink/pull/6773#issuecomment-429611382
@pnowojski Could you help review this PR? Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add reverse supported in TableAPI and SQL
> -
>
> Key: FLINK-10152
> URL: https://issues.apache.org/jira/browse/FLINK-10152
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Reporter: Guibo Pan
>Assignee: Guibo Pan
>Priority: Major
> Labels: pull-request-available
>
> reverse is useful function for procession String.
> for example:
> {code:java}
> select reverse("abcd") // return "dcba"
> {code}
> It is supported as a UDF in Hive, more details please see[1] [1]:
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-StringFunctions
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10534) Add idle timeout for a flink session cluster
[
https://issues.apache.org/jira/browse/FLINK-10534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649327#comment-16649327
]
ouyangzhe commented on FLINK-10534:
---
[~gjy] , If I want to get job result back or run multi jobs in main method
like shown below, detach mode is not suitable,
{code:java}
xxx = resultDataSet.collect();
// use xxx to do something then sumbit another job
env.execute("test");{code}
now, A session cluster will be started for this case, in CliFrontend.java:
{code:java}
final ClusterSpecification clusterSpecification =
customCommandLine.getClusterSpecification(commandLine);
client = clusterDescriptor.deploySessionCluster(clusterSpecification);
{code}
the session cluster may not be shut down if user's main process killed, and
will always be running in the yarn cluster.
> Add idle timeout for a flink session cluster
>
>
> Key: FLINK-10534
> URL: https://issues.apache.org/jira/browse/FLINK-10534
> Project: Flink
> Issue Type: New Feature
> Components: Cluster Management
>Affects Versions: 1.7.0
>Reporter: ouyangzhe
>Assignee: ouyangzhe
>Priority: Major
> Attachments: 屏幕快照 2018-10-12 上午10.24.08.png
>
>
> The flink session cluster on yarn will aways be running while has no jobs
> running at all, it will occupy the yarn resources for no use.
> Taskmanagers will be released after an idle timeout, but jobmanager will keep
> running.
> I propose to add a configuration to limit the idle timeout for jobmanager
> too, if no job running after a specified timeout, the flink cluster auto
> finish itself.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-10534) Add idle timeout for a flink session cluster
[
https://issues.apache.org/jira/browse/FLINK-10534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649327#comment-16649327
]
ouyangzhe edited comment on FLINK-10534 at 10/14/18 10:13 AM:
--
[~gjy] , If I want to get job result back or run multi jobs in main method
like shown below, detach mode is not suitable,
{code:java}
xxx = resultDataSet.collect();
// use xxx to do something then submit another job
env.execute("test");{code}
now, A session cluster will be started for this case, in CliFrontend.java:
{code:java}
final ClusterSpecification clusterSpecification =
customCommandLine.getClusterSpecification(commandLine);
client = clusterDescriptor.deploySessionCluster(clusterSpecification);
{code}
the session cluster may not be shut down if user's main process killed, and
will always be running in the yarn cluster.
was (Author: yeshan):
[~gjy] , If I want to get job result back or run multi jobs in main method
like shown below, detach mode is not suitable,
{code:java}
xxx = resultDataSet.collect();
// use xxx to do something then sumbit another job
env.execute("test");{code}
now, A session cluster will be started for this case, in CliFrontend.java:
{code:java}
final ClusterSpecification clusterSpecification =
customCommandLine.getClusterSpecification(commandLine);
client = clusterDescriptor.deploySessionCluster(clusterSpecification);
{code}
the session cluster may not be shut down if user's main process killed, and
will always be running in the yarn cluster.
> Add idle timeout for a flink session cluster
>
>
> Key: FLINK-10534
> URL: https://issues.apache.org/jira/browse/FLINK-10534
> Project: Flink
> Issue Type: New Feature
> Components: Cluster Management
>Affects Versions: 1.7.0
>Reporter: ouyangzhe
>Assignee: ouyangzhe
>Priority: Major
> Attachments: 屏幕快照 2018-10-12 上午10.24.08.png
>
>
> The flink session cluster on yarn will aways be running while has no jobs
> running at all, it will occupy the yarn resources for no use.
> Taskmanagers will be released after an idle timeout, but jobmanager will keep
> running.
> I propose to add a configuration to limit the idle timeout for jobmanager
> too, if no job running after a specified timeout, the flink cluster auto
> finish itself.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9607) Support ParquetTableSink
[ https://issues.apache.org/jira/browse/FLINK-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649354#comment-16649354 ] Guibo Pan commented on FLINK-9607: -- [~mingleizhang] I wonder what the status of this issue is, could I assign it to myself and work on it ? > Support ParquetTableSink > > > Key: FLINK-9607 > URL: https://issues.apache.org/jira/browse/FLINK-9607 > Project: Flink > Issue Type: New Feature >Reporter: zhangminglei >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-10518) Inefficient design in ContinuousFileMonitoringFunction
[
https://issues.apache.org/jira/browse/FLINK-10518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guibo Pan reassigned FLINK-10518:
-
Assignee: Guibo Pan
> Inefficient design in ContinuousFileMonitoringFunction
> --
>
> Key: FLINK-10518
> URL: https://issues.apache.org/jira/browse/FLINK-10518
> Project: Flink
> Issue Type: Improvement
> Components: filesystem-connector
>Affects Versions: 1.5.2
>Reporter: Huyen Levan
>Assignee: Guibo Pan
>Priority: Major
> Labels: Source:FileSystem
>
> The ContinuousFileMonitoringFunction class keeps track of the latest file
> modification time to rule out all files it has processed in the previous
> cycles. For a long-running job, the list of eligible files will be much
> smaller than the list of all files in the folder being monitored.
> In the current implementation of the getInputSplitsSortedByModTime method, a
> (big) list of all available splits are created first, and then every single
> split is checked with the list of eligible files.
> {quote}for (FileInputSplit split:
> format.createInputSplits(readerParallelism)) {
> FileStatus fileStatus = eligibleFiles.get(split.getPath());
> if (fileStatus != null) {
> {quote}
> The improvement can be done as:
> * Listing of all files should be done once in
> _ContinuousFileMonitoringFunction.listEligibleFiles()_ (as of now it is done
> the 2nd time in _FileInputFormat.createInputSplits()_ )
> * The list of file-splits should then be created from the list of paths in
> eligibleFiles.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] yanghua opened a new pull request #6839: [FLINK-10253] Run MetricQueryService with lower priority
yanghua opened a new pull request #6839: [FLINK-10253] Run MetricQueryService with lower priority URL: https://github.com/apache/flink/pull/6839 ## What is the purpose of the change *This pull request makes MetricQueryService run with lower priority* ## Brief change log - *Run MetricQueryService with lower priority* ## Verifying this change This change is already covered by existing tests, such as *AkkaUtilsTest*. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10253) Run MetricQueryService with lower priority
[
https://issues.apache.org/jira/browse/FLINK-10253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649364#comment-16649364
]
ASF GitHub Bot commented on FLINK-10253:
yanghua opened a new pull request #6839: [FLINK-10253] Run MetricQueryService
with lower priority
URL: https://github.com/apache/flink/pull/6839
## What is the purpose of the change
*This pull request makes MetricQueryService run with lower priority*
## Brief change log
- *Run MetricQueryService with lower priority*
## Verifying this change
This change is already covered by existing tests, such as *AkkaUtilsTest*.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no**
/ don't know)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs
/ **not documented**)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Run MetricQueryService with lower priority
> --
>
> Key: FLINK-10253
> URL: https://issues.apache.org/jira/browse/FLINK-10253
> Project: Flink
> Issue Type: Sub-task
> Components: Metrics
>Affects Versions: 1.5.3, 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.7.0, 1.6.2, 1.5.5
>
>
> We should run the {{MetricQueryService}} with a lower priority than the main
> Flink components. An idea would be to start the underlying threads with a
> lower priority.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10253) Run MetricQueryService with lower priority
[
https://issues.apache.org/jira/browse/FLINK-10253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-10253:
---
Labels: pull-request-available (was: )
> Run MetricQueryService with lower priority
> --
>
> Key: FLINK-10253
> URL: https://issues.apache.org/jira/browse/FLINK-10253
> Project: Flink
> Issue Type: Sub-task
> Components: Metrics
>Affects Versions: 1.5.3, 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.7.0, 1.6.2, 1.5.5
>
>
> We should run the {{MetricQueryService}} with a lower priority than the main
> Flink components. An idea would be to start the underlying threads with a
> lower priority.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] alexeyt820 commented on issue #6703: [FLINK-9697] Provide connector for Kafka 2.0.0
alexeyt820 commented on issue #6703: [FLINK-9697] Provide connector for Kafka 2.0.0 URL: https://github.com/apache/flink/pull/6703#issuecomment-429635177 Can deserializer access ConsumerRecord to get headers and timestamp? Or it will be PR on top this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-9697) Provide connector for Kafka 2.0.0
[ https://issues.apache.org/jira/browse/FLINK-9697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649430#comment-16649430 ] ASF GitHub Bot commented on FLINK-9697: --- alexeyt820 commented on issue #6703: [FLINK-9697] Provide connector for Kafka 2.0.0 URL: https://github.com/apache/flink/pull/6703#issuecomment-429635177 Can deserializer access ConsumerRecord to get headers and timestamp? Or it will be PR on top this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide connector for Kafka 2.0.0 > - > > Key: FLINK-9697 > URL: https://issues.apache.org/jira/browse/FLINK-9697 > Project: Flink > Issue Type: Improvement >Reporter: Ted Yu >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > > Kafka 2.0.0 would be released soon. > Here is vote thread: > [http://search-hadoop.com/m/Kafka/uyzND1vxnEd23QLxb?subj=+VOTE+2+0+0+RC1] > We should provide connector for Kafka 2.0.0 once it is released. > Upgrade to 2.0 documentation : > http://kafka.apache.org/20/documentation.html#upgrade_2_0_0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10544) Remove custom settings.xml for snapshot deployments
Chesnay Schepler created FLINK-10544: Summary: Remove custom settings.xml for snapshot deployments Key: FLINK-10544 URL: https://issues.apache.org/jira/browse/FLINK-10544 Project: Flink Issue Type: Bug Components: Build System Affects Versions: 1.5.4, 1.6.1, 1.4.2, 1.7.0 Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.7.0 Our snapshot builds are broken since we use custom authentication stuff. According to infra (see FLINK-17138) we can just use the standard maven settings available on jenkins. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10544) Remove custom settings.xml for snapshot deployments
[ https://issues.apache.org/jira/browse/FLINK-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649435#comment-16649435 ] Chesnay Schepler commented on FLINK-10544: -- master: bab4ec161b2119dbaa6188910ab7b05c3015ff40 I'm verifying that the changes work correctly for master. If successful I will update the remaining snapshot branches. > Remove custom settings.xml for snapshot deployments > --- > > Key: FLINK-10544 > URL: https://issues.apache.org/jira/browse/FLINK-10544 > Project: Flink > Issue Type: Bug > Components: Build System >Affects Versions: 1.4.2, 1.6.1, 1.7.0, 1.5.4 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.7.0 > > > Our snapshot builds are broken since we use custom authentication stuff. > According to infra (see FLINK-17138) we can just use the standard maven > settings available on jenkins. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] aljoscha commented on issue #6703: [FLINK-9697] Provide connector for Kafka 2.0.0
aljoscha commented on issue #6703: [FLINK-9697] Provide connector for Kafka 2.0.0 URL: https://github.com/apache/flink/pull/6703#issuecomment-429640076 @yanghua Which one needs discussion with me? @alexeyt820 I would suggest to do that change on top of this PR. But we have to move quickly on that one after this PR is merged. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-9697) Provide connector for Kafka 2.0.0
[ https://issues.apache.org/jira/browse/FLINK-9697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649454#comment-16649454 ] ASF GitHub Bot commented on FLINK-9697: --- aljoscha commented on issue #6703: [FLINK-9697] Provide connector for Kafka 2.0.0 URL: https://github.com/apache/flink/pull/6703#issuecomment-429640076 @yanghua Which one needs discussion with me? @alexeyt820 I would suggest to do that change on top of this PR. But we have to move quickly on that one after this PR is merged. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide connector for Kafka 2.0.0 > - > > Key: FLINK-9697 > URL: https://issues.apache.org/jira/browse/FLINK-9697 > Project: Flink > Issue Type: Improvement >Reporter: Ted Yu >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > > Kafka 2.0.0 would be released soon. > Here is vote thread: > [http://search-hadoop.com/m/Kafka/uyzND1vxnEd23QLxb?subj=+VOTE+2+0+0+RC1] > We should provide connector for Kafka 2.0.0 once it is released. > Upgrade to 2.0 documentation : > http://kafka.apache.org/20/documentation.html#upgrade_2_0_0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (FLINK-9990) Add regexp_extract supported in TableAPI and SQL
[
https://issues.apache.org/jira/browse/FLINK-9990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xingcan Cui resolved FLINK-9990.
Resolution: Implemented
Implemented in 1.7.0 5dc360984143005f73b8f70f97ed6b1c2afd7dc3
> Add regexp_extract supported in TableAPI and SQL
>
>
> Key: FLINK-9990
> URL: https://issues.apache.org/jira/browse/FLINK-9990
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Minor
> Labels: pull-request-available
>
> regex_extract is a very useful function, it returns a string based on a regex
> pattern and a index.
> For example :
> {code:java}
> regexp_extract('foothebar', 'foo(.*?)(bar)', 2) // returns 'bar.'
> {code}
> It is provided as a UDF in Hive, more details please see[1].
> [1]: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-10544) Remove custom settings.xml for snapshot deployments
[ https://issues.apache.org/jira/browse/FLINK-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-10544. Resolution: Fixed Fix Version/s: 1.5.5 1.6.2 1.4.3 1.6: 004b582167e6e0eb38fdee1675b4376fe3d9bf4e 1.5: 10d88af5b056e3197cff098ff6e8e56c48be295d 1.4: bab4ec161b2119dbaa6188910ab7b05c3015ff40 > Remove custom settings.xml for snapshot deployments > --- > > Key: FLINK-10544 > URL: https://issues.apache.org/jira/browse/FLINK-10544 > Project: Flink > Issue Type: Bug > Components: Build System >Affects Versions: 1.4.2, 1.6.1, 1.7.0, 1.5.4 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.4.3, 1.7.0, 1.6.2, 1.5.5 > > > Our snapshot builds are broken since we use custom authentication stuff. > According to infra (see FLINK-17138) we can just use the standard maven > settings available on jenkins. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] eliaslevy commented on a change in pull request #6703: [FLINK-9697] Provide connector for Kafka 2.0.0
eliaslevy commented on a change in pull request #6703: [FLINK-9697] Provide connector for Kafka 2.0.0 URL: https://github.com/apache/flink/pull/6703#discussion_r225008944 ## File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer.java ## @@ -0,0 +1,321 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.streaming.connectors.kafka; + +import org.apache.flink.annotation.PublicEvolving; +import org.apache.flink.api.common.serialization.DeserializationSchema; +import org.apache.flink.metrics.MetricGroup; +import org.apache.flink.streaming.api.functions.AssignerWithPeriodicWatermarks; +import org.apache.flink.streaming.api.functions.AssignerWithPunctuatedWatermarks; +import org.apache.flink.streaming.api.operators.StreamingRuntimeContext; +import org.apache.flink.streaming.connectors.kafka.config.OffsetCommitMode; +import org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher; +import org.apache.flink.streaming.connectors.kafka.internal.KafkaPartitionDiscoverer; +import org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher; +import org.apache.flink.streaming.connectors.kafka.internals.AbstractPartitionDiscoverer; +import org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition; +import org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicsDescriptor; +import org.apache.flink.streaming.util.serialization.KeyedDeserializationSchema; +import org.apache.flink.streaming.util.serialization.KeyedDeserializationSchemaWrapper; +import org.apache.flink.util.PropertiesUtil; +import org.apache.flink.util.SerializedValue; + +import org.apache.kafka.clients.consumer.ConsumerConfig; +import org.apache.kafka.clients.consumer.KafkaConsumer; +import org.apache.kafka.clients.consumer.OffsetAndTimestamp; +import org.apache.kafka.common.TopicPartition; +import org.apache.kafka.common.serialization.ByteArrayDeserializer; + +import java.util.Collection; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Properties; +import java.util.regex.Pattern; + +import static org.apache.flink.util.Preconditions.checkNotNull; +import static org.apache.flink.util.PropertiesUtil.getBoolean; +import static org.apache.flink.util.PropertiesUtil.getLong; + +/** + * The Flink Kafka Consumer is a streaming data source that pulls a parallel data stream from + * Apache Kafka 2.x. The consumer can run in multiple parallel instances, each of which will pull + * data from one or more Kafka partitions. + * + * The Flink Kafka Consumer participates in checkpointing and guarantees that no data is lost + * during a failure, and that the computation processes elements "exactly once". + * (Note: These guarantees naturally assume that Kafka itself does not loose any data.) + * + * Please note that Flink snapshots the offsets internally as part of its distributed checkpoints. The offsets + * committed to Kafka / ZooKeeper are only to bring the outside view of progress in sync with Flink's view + * of the progress. That way, monitoring and other jobs can get a view of how far the Flink Kafka consumer + * has consumed a topic. + * + * Please refer to Kafka's documentation for the available configuration properties: + * http://kafka.apache.org/documentation.html#newconsumerconfigs + */ Review comment: Apologies. I misread it. It is correct as it is. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-9697) Provide connector for Kafka 2.0.0
[ https://issues.apache.org/jira/browse/FLINK-9697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649532#comment-16649532 ] ASF GitHub Bot commented on FLINK-9697: --- eliaslevy commented on a change in pull request #6703: [FLINK-9697] Provide connector for Kafka 2.0.0 URL: https://github.com/apache/flink/pull/6703#discussion_r225008944 ## File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer.java ## @@ -0,0 +1,321 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.streaming.connectors.kafka; + +import org.apache.flink.annotation.PublicEvolving; +import org.apache.flink.api.common.serialization.DeserializationSchema; +import org.apache.flink.metrics.MetricGroup; +import org.apache.flink.streaming.api.functions.AssignerWithPeriodicWatermarks; +import org.apache.flink.streaming.api.functions.AssignerWithPunctuatedWatermarks; +import org.apache.flink.streaming.api.operators.StreamingRuntimeContext; +import org.apache.flink.streaming.connectors.kafka.config.OffsetCommitMode; +import org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher; +import org.apache.flink.streaming.connectors.kafka.internal.KafkaPartitionDiscoverer; +import org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher; +import org.apache.flink.streaming.connectors.kafka.internals.AbstractPartitionDiscoverer; +import org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition; +import org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicsDescriptor; +import org.apache.flink.streaming.util.serialization.KeyedDeserializationSchema; +import org.apache.flink.streaming.util.serialization.KeyedDeserializationSchemaWrapper; +import org.apache.flink.util.PropertiesUtil; +import org.apache.flink.util.SerializedValue; + +import org.apache.kafka.clients.consumer.ConsumerConfig; +import org.apache.kafka.clients.consumer.KafkaConsumer; +import org.apache.kafka.clients.consumer.OffsetAndTimestamp; +import org.apache.kafka.common.TopicPartition; +import org.apache.kafka.common.serialization.ByteArrayDeserializer; + +import java.util.Collection; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Properties; +import java.util.regex.Pattern; + +import static org.apache.flink.util.Preconditions.checkNotNull; +import static org.apache.flink.util.PropertiesUtil.getBoolean; +import static org.apache.flink.util.PropertiesUtil.getLong; + +/** + * The Flink Kafka Consumer is a streaming data source that pulls a parallel data stream from + * Apache Kafka 2.x. The consumer can run in multiple parallel instances, each of which will pull + * data from one or more Kafka partitions. + * + * The Flink Kafka Consumer participates in checkpointing and guarantees that no data is lost + * during a failure, and that the computation processes elements "exactly once". + * (Note: These guarantees naturally assume that Kafka itself does not loose any data.) + * + * Please note that Flink snapshots the offsets internally as part of its distributed checkpoints. The offsets + * committed to Kafka / ZooKeeper are only to bring the outside view of progress in sync with Flink's view + * of the progress. That way, monitoring and other jobs can get a view of how far the Flink Kafka consumer + * has consumed a topic. + * + * Please refer to Kafka's documentation for the available configuration properties: + * http://kafka.apache.org/documentation.html#newconsumerconfigs + */ Review comment: Apologies. I misread it. It is correct as it is. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide connector for Kafka 2.0.0 > - > > Key: FLINK-9697 > URL: htt
[GitHub] yanghua commented on issue #6542: [FLINK-6437][History Server] Move history server configuration to a separate file
yanghua commented on issue #6542: [FLINK-6437][History Server] Move history server configuration to a separate file URL: https://github.com/apache/flink/pull/6542#issuecomment-429680654 @zentol OK, so what do you think is the correct way to handle it? How to separate History Server? If your solution is more reasonable, then I am happy to accept and implement it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-6437) Move history server configuration to a separate file
[
https://issues.apache.org/jira/browse/FLINK-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649613#comment-16649613
]
ASF GitHub Bot commented on FLINK-6437:
---
yanghua commented on issue #6542: [FLINK-6437][History Server] Move history
server configuration to a separate file
URL: https://github.com/apache/flink/pull/6542#issuecomment-429680654
@zentol OK, so what do you think is the correct way to handle it? How to
separate History Server? If your solution is more reasonable, then I am happy
to accept and implement it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move history server configuration to a separate file
>
>
> Key: FLINK-6437
> URL: https://issues.apache.org/jira/browse/FLINK-6437
> Project: Flink
> Issue Type: Improvement
> Components: History Server
>Affects Versions: 1.3.0
>Reporter: Stephan Ewen
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> I suggest to keep the {{flink-conf.yaml}} leaner by moving configuration of
> the History Server to a different file.
> In general, I would propose to move configurations of separate, independent
> and optional components to individual config files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] yanghua commented on a change in pull request #6703: [FLINK-9697] Provide connector for Kafka 2.0.0
yanghua commented on a change in pull request #6703: [FLINK-9697] Provide connector for Kafka 2.0.0 URL: https://github.com/apache/flink/pull/6703#discussion_r225022890 ## File path: flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/table/descriptors/KafkaValidator.java ## @@ -36,6 +36,7 @@ public static final String CONNECTOR_VERSION_VALUE_09 = "0.9"; public static final String CONNECTOR_VERSION_VALUE_010 = "0.10"; public static final String CONNECTOR_VERSION_VALUE_011 = "0.11"; + public static final String CONNECTOR_VERSION_VALUE_20 = "2.0"; Review comment: @aljoscha It's here. The kafka version will be verified in KafkaValidator#validateVersion. I recommend removing the validation method and refactoring this constant as: ```java public static final String CONNECTOR_UNTRACKED_VERSION_VALUE = "untracked" ``` What do you think? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-9697) Provide connector for Kafka 2.0.0
[ https://issues.apache.org/jira/browse/FLINK-9697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649640#comment-16649640 ] ASF GitHub Bot commented on FLINK-9697: --- yanghua commented on a change in pull request #6703: [FLINK-9697] Provide connector for Kafka 2.0.0 URL: https://github.com/apache/flink/pull/6703#discussion_r225022890 ## File path: flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/table/descriptors/KafkaValidator.java ## @@ -36,6 +36,7 @@ public static final String CONNECTOR_VERSION_VALUE_09 = "0.9"; public static final String CONNECTOR_VERSION_VALUE_010 = "0.10"; public static final String CONNECTOR_VERSION_VALUE_011 = "0.11"; + public static final String CONNECTOR_VERSION_VALUE_20 = "2.0"; Review comment: @aljoscha It's here. The kafka version will be verified in KafkaValidator#validateVersion. I recommend removing the validation method and refactoring this constant as: ```java public static final String CONNECTOR_UNTRACKED_VERSION_VALUE = "untracked" ``` What do you think? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide connector for Kafka 2.0.0 > - > > Key: FLINK-9697 > URL: https://issues.apache.org/jira/browse/FLINK-9697 > Project: Flink > Issue Type: Improvement >Reporter: Ted Yu >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > > Kafka 2.0.0 would be released soon. > Here is vote thread: > [http://search-hadoop.com/m/Kafka/uyzND1vxnEd23QLxb?subj=+VOTE+2+0+0+RC1] > We should provide connector for Kafka 2.0.0 once it is released. > Upgrade to 2.0 documentation : > http://kafka.apache.org/20/documentation.html#upgrade_2_0_0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] yanghua commented on issue #6703: [FLINK-9697] Provide connector for Kafka 2.0.0
yanghua commented on issue #6703: [FLINK-9697] Provide connector for Kafka 2.0.0 URL: https://github.com/apache/flink/pull/6703#issuecomment-429687222 > Can deserializer access ConsumerRecord to get headers and timestamp? Or it will be PR on top this PR? @alexeyt820 I agree with @aljoscha that we should not add more changes to this PR so that its review progress is further delayed (because it is already large enough). We are trying to get it to catch up with the release point of Flink 1.7. Of course, the issue you mentioned is also valuable. I feel that once this PR is merged, we can immediately put the issue on the agenda. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-9697) Provide connector for Kafka 2.0.0
[ https://issues.apache.org/jira/browse/FLINK-9697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649645#comment-16649645 ] ASF GitHub Bot commented on FLINK-9697: --- yanghua commented on issue #6703: [FLINK-9697] Provide connector for Kafka 2.0.0 URL: https://github.com/apache/flink/pull/6703#issuecomment-429687222 > Can deserializer access ConsumerRecord to get headers and timestamp? Or it will be PR on top this PR? @alexeyt820 I agree with @aljoscha that we should not add more changes to this PR so that its review progress is further delayed (because it is already large enough). We are trying to get it to catch up with the release point of Flink 1.7. Of course, the issue you mentioned is also valuable. I feel that once this PR is merged, we can immediately put the issue on the agenda. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide connector for Kafka 2.0.0 > - > > Key: FLINK-9697 > URL: https://issues.apache.org/jira/browse/FLINK-9697 > Project: Flink > Issue Type: Improvement >Reporter: Ted Yu >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > > Kafka 2.0.0 would be released soon. > Here is vote thread: > [http://search-hadoop.com/m/Kafka/uyzND1vxnEd23QLxb?subj=+VOTE+2+0+0+RC1] > We should provide connector for Kafka 2.0.0 once it is released. > Upgrade to 2.0 documentation : > http://kafka.apache.org/20/documentation.html#upgrade_2_0_0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9607) Support ParquetTableSink
[ https://issues.apache.org/jira/browse/FLINK-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649648#comment-16649648 ] vinoyang commented on FLINK-9607: - hi [~guibopan] , [~mingleizhang] has released this issue. Please feel free to assign it to yourself. > Support ParquetTableSink > > > Key: FLINK-9607 > URL: https://issues.apache.org/jira/browse/FLINK-9607 > Project: Flink > Issue Type: New Feature >Reporter: zhangminglei >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-10251) Handle oversized response messages in AkkaRpcActor
[
https://issues.apache.org/jira/browse/FLINK-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
vinoyang reassigned FLINK-10251:
Assignee: vinoyang
> Handle oversized response messages in AkkaRpcActor
> --
>
> Key: FLINK-10251
> URL: https://issues.apache.org/jira/browse/FLINK-10251
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Coordination
>Affects Versions: 1.5.3, 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Fix For: 1.7.0, 1.6.2, 1.5.5
>
>
> The {{AkkaRpcActor}} should check whether an RPC response which is sent to a
> remote sender does not exceed the maximum framesize of the underlying
> {{ActorSystem}}. If this is the case we should fail fast instead. We can
> achieve this by serializing the response and sending the serialized byte
> array.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] yanghua commented on issue #6839: [FLINK-10253] Run MetricQueryService with lower priority
yanghua commented on issue #6839: [FLINK-10253] Run MetricQueryService with lower priority URL: https://github.com/apache/flink/pull/6839#issuecomment-429688635 The `DefaultDispatcherPrerequisites` is a final class now. So we can not extend it. I implemented the `PriorityThreadsDispatcherPrerequisites ` with a wrapper pattern. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10253) Run MetricQueryService with lower priority
[
https://issues.apache.org/jira/browse/FLINK-10253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649650#comment-16649650
]
ASF GitHub Bot commented on FLINK-10253:
yanghua commented on issue #6839: [FLINK-10253] Run MetricQueryService with
lower priority
URL: https://github.com/apache/flink/pull/6839#issuecomment-429688635
The `DefaultDispatcherPrerequisites` is a final class now. So we can not
extend it. I implemented the `PriorityThreadsDispatcherPrerequisites ` with a
wrapper pattern.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Run MetricQueryService with lower priority
> --
>
> Key: FLINK-10253
> URL: https://issues.apache.org/jira/browse/FLINK-10253
> Project: Flink
> Issue Type: Sub-task
> Components: Metrics
>Affects Versions: 1.5.3, 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.7.0, 1.6.2, 1.5.5
>
>
> We should run the {{MetricQueryService}} with a lower priority than the main
> Flink components. An idea would be to start the underlying threads with a
> lower priority.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10095) Change the serialisation order in TTL value wrapper
[ https://issues.apache.org/jira/browse/FLINK-10095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649655#comment-16649655 ] aitozi commented on FLINK-10095: Hi, [~azagrebin] I have a little question here: If we use the order uservalue first, we can also work directly with serialized value by skipping looking into the rest of 8 bytes at the end？So I think do not need to change the order of serialization, do you think so? And can also optimize when expire the liststate. > Change the serialisation order in TTL value wrapper > --- > > Key: FLINK-10095 > URL: https://issues.apache.org/jira/browse/FLINK-10095 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing >Reporter: Andrey Zagrebin >Assignee: Andrey Zagrebin >Priority: Major > Labels: pull-request-available > Fix For: 1.6.0 > > > The first implementation of TTL value wrapper has the following serialisation > order: first user value, then last modification timestamp. > This was planned for potential optimisation where we could peek at the last > bytes of last element of list to check whether the whole list expired or not. > After careful consideration, it is not the frequent case and the list is > handled per entry mostly. > Having the fixed part (the timestamp) at the beginning looks more promising > in future as we can skip looking into the rest of bytes if needed when > working directly with serialised value. > This issue suggests to change the serialisation order. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-5726) Add the RocketMQ plugin for the Apache Flink
[ https://issues.apache.org/jira/browse/FLINK-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yuemeng reassigned FLINK-5726: -- Assignee: yuemeng > Add the RocketMQ plugin for the Apache Flink > > > Key: FLINK-5726 > URL: https://issues.apache.org/jira/browse/FLINK-5726 > Project: Flink > Issue Type: Task > Components: Streaming Connectors >Reporter: Longda Feng >Assignee: yuemeng >Priority: Minor > > Apache RocketMQ® is an open source distributed messaging and streaming data > platform. It has been used in a lot of companies. Please refer to > http://rocketmq.incubator.apache.org/ for more details. > Since the Apache RocketMq 4.0 will be released in the next few days, we can > start the job of adding the RocketMq plugin for the Apache Flink. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] Clarkkkkk commented on issue #6839: [FLINK-10253] Run MetricQueryService with lower priority
Clark commented on issue #6839: [FLINK-10253] Run MetricQueryService with lower priority URL: https://github.com/apache/flink/pull/6839#issuecomment-429700148 Thanks for your contribution @yanghua . Please refer to the README in flink-docs to regenerate the doc after your modify the option related classes in order to pass the test. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10253) Run MetricQueryService with lower priority
[
https://issues.apache.org/jira/browse/FLINK-10253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649692#comment-16649692
]
ASF GitHub Bot commented on FLINK-10253:
Clark commented on issue #6839: [FLINK-10253] Run MetricQueryService with
lower priority
URL: https://github.com/apache/flink/pull/6839#issuecomment-429700148
Thanks for your contribution @yanghua . Please refer to the README in
flink-docs to regenerate the doc after your modify the option related classes
in order to pass the test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Run MetricQueryService with lower priority
> --
>
> Key: FLINK-10253
> URL: https://issues.apache.org/jira/browse/FLINK-10253
> Project: Flink
> Issue Type: Sub-task
> Components: Metrics
>Affects Versions: 1.5.3, 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.7.0, 1.6.2, 1.5.5
>
>
> We should run the {{MetricQueryService}} with a lower priority than the main
> Flink components. An idea would be to start the underlying threads with a
> lower priority.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] Clarkkkkk commented on issue #6839: [FLINK-10253] Run MetricQueryService with lower priority
Clark commented on issue #6839: [FLINK-10253] Run MetricQueryService with lower priority URL: https://github.com/apache/flink/pull/6839#issuecomment-429700906 One more question, should we expose the threads priority configuration to the user? The purpose of this pr is to lower the priority of metric query service, but the user can configure the threads priority to 10. Maybe we should at least restrict the maximum of threads priority. Before diving into the code, I am waiting for inputs from @tillrohrmann . This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10253) Run MetricQueryService with lower priority
[
https://issues.apache.org/jira/browse/FLINK-10253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649693#comment-16649693
]
ASF GitHub Bot commented on FLINK-10253:
Clark commented on issue #6839: [FLINK-10253] Run MetricQueryService with
lower priority
URL: https://github.com/apache/flink/pull/6839#issuecomment-429700906
One more question, should we expose the threads priority configuration to
the user? The purpose of this pr is to lower the priority of metric query
service, but the user can configure the threads priority to 10. Maybe we should
at least restrict the maximum of threads priority. Before diving into the code,
I am waiting for inputs from @tillrohrmann .
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Run MetricQueryService with lower priority
> --
>
> Key: FLINK-10253
> URL: https://issues.apache.org/jira/browse/FLINK-10253
> Project: Flink
> Issue Type: Sub-task
> Components: Metrics
>Affects Versions: 1.5.3, 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.7.0, 1.6.2, 1.5.5
>
>
> We should run the {{MetricQueryService}} with a lower priority than the main
> Flink components. An idea would be to start the underlying threads with a
> lower priority.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10545) Remove JobManagerLeaderSessionIDITCase
TisonKun created FLINK-10545:
Summary: Remove JobManagerLeaderSessionIDITCase
Key: FLINK-10545
URL: https://issues.apache.org/jira/browse/FLINK-10545
Project: Flink
Issue Type: Sub-task
Components: Tests
Affects Versions: 1.7.0
Reporter: TisonKun
Assignee: TisonKun
Fix For: 1.7.0
{{JobManagerLeaderSessionIDITCase.scala}} is based on legacy mode and I think
we now have {{FencingToken}} and need not to maintain or port this test. Just
simply remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] TisonKun opened a new pull request #6840: [FLINK-10545] [tests] Remove JobManagerLeaderSessionIDITCase
TisonKun opened a new pull request #6840: [FLINK-10545] [tests] Remove
JobManagerLeaderSessionIDITCase
URL: https://github.com/apache/flink/pull/6840
## What is the purpose of the change
{{JobManagerLeaderSessionIDITCase.scala}} is based on legacy mode and I
think we now have {{FencingToken}} and need not to maintain or port this test.
Just simply remove it.
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (**no**)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (**no**)
- The serializers: (**no**)
- The runtime per-record code paths (performance sensitive): (**no**)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (**no**)
- The S3 file system connector: (**no**)
## Documentation
- Does this pull request introduce a new feature? (**no**)
- If yes, how is the feature documented? (**not applicable**)
cc @tillrohrmann @StefanRRichter
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (FLINK-10545) Remove JobManagerLeaderSessionIDITCase
[
https://issues.apache.org/jira/browse/FLINK-10545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-10545:
---
Labels: pull-request-available (was: )
> Remove JobManagerLeaderSessionIDITCase
> --
>
> Key: FLINK-10545
> URL: https://issues.apache.org/jira/browse/FLINK-10545
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 1.7.0
>Reporter: TisonKun
>Assignee: TisonKun
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> {{JobManagerLeaderSessionIDITCase.scala}} is based on legacy mode and I think
> we now have {{FencingToken}} and need not to maintain or port this test. Just
> simply remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10545) Remove JobManagerLeaderSessionIDITCase
[
https://issues.apache.org/jira/browse/FLINK-10545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649702#comment-16649702
]
ASF GitHub Bot commented on FLINK-10545:
TisonKun opened a new pull request #6840: [FLINK-10545] [tests] Remove
JobManagerLeaderSessionIDITCase
URL: https://github.com/apache/flink/pull/6840
## What is the purpose of the change
{{JobManagerLeaderSessionIDITCase.scala}} is based on legacy mode and I
think we now have {{FencingToken}} and need not to maintain or port this test.
Just simply remove it.
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (**no**)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (**no**)
- The serializers: (**no**)
- The runtime per-record code paths (performance sensitive): (**no**)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (**no**)
- The S3 file system connector: (**no**)
## Documentation
- Does this pull request introduce a new feature? (**no**)
- If yes, how is the feature documented? (**not applicable**)
cc @tillrohrmann @StefanRRichter
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Remove JobManagerLeaderSessionIDITCase
> --
>
> Key: FLINK-10545
> URL: https://issues.apache.org/jira/browse/FLINK-10545
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 1.7.0
>Reporter: TisonKun
>Assignee: TisonKun
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> {{JobManagerLeaderSessionIDITCase.scala}} is based on legacy mode and I think
> we now have {{FencingToken}} and need not to maintain or port this test. Just
> simply remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] TisonKun opened a new pull request #6841: [FLINK-10405] [tests] Port JobManagerFailsITCase to new code base
TisonKun opened a new pull request #6841: [FLINK-10405] [tests] Port JobManagerFailsITCase to new code base URL: https://github.com/apache/flink/pull/6841 ## What is the purpose of the change Port `JobManagerFailsITCase` to new code base. Basically I think that taskmanager can connect to the new `Dispatcher` and the whole job goes into a clean state after Dispatcher(old jobmanager somehow) fails is covered by `JobManagerHAProcessFailureBatchRecoveryITCase`. In fact, it is not only for batch but covers streaming. So just use it and add a bit verification. ## Brief change log - Remove `JobManagerFailsITCase.scala` - Verify number of taskmanager to cover that taskmanager would connect to the new `Dispatcher`. ## Verifying this change This change is itself a test. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (**no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**no**) - The serializers: (**no**) - The runtime per-record code paths (performance sensitive): (**no**) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (**no**) - The S3 file system connector: (**no**) ## Documentation - Does this pull request introduce a new feature? (**no**) - If yes, how is the feature documented? (**not applicable**) cc @tillrohrmann @GJL This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10405) Port JobManagerFailsITCase to new code base
[
https://issues.apache.org/jira/browse/FLINK-10405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649708#comment-16649708
]
ASF GitHub Bot commented on FLINK-10405:
TisonKun opened a new pull request #6841: [FLINK-10405] [tests] Port
JobManagerFailsITCase to new code base
URL: https://github.com/apache/flink/pull/6841
## What is the purpose of the change
Port `JobManagerFailsITCase` to new code base. Basically I think that
taskmanager can connect to the new `Dispatcher` and the whole job goes into a
clean state after Dispatcher(old jobmanager somehow) fails is covered by
`JobManagerHAProcessFailureBatchRecoveryITCase`. In fact, it is not only for
batch but covers streaming. So just use it and add a bit verification.
## Brief change log
- Remove `JobManagerFailsITCase.scala`
- Verify number of taskmanager to cover that taskmanager would connect to
the new `Dispatcher`.
## Verifying this change
This change is itself a test.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (**no**)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (**no**)
- The serializers: (**no**)
- The runtime per-record code paths (performance sensitive): (**no**)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (**no**)
- The S3 file system connector: (**no**)
## Documentation
- Does this pull request introduce a new feature? (**no**)
- If yes, how is the feature documented? (**not applicable**)
cc @tillrohrmann @GJL
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Port JobManagerFailsITCase to new code base
> ---
>
> Key: FLINK-10405
> URL: https://issues.apache.org/jira/browse/FLINK-10405
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 1.7.0
>Reporter: TisonKun
>Assignee: TisonKun
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> Port {{JobManagerFailsITCase}} to new code base.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10405) Port JobManagerFailsITCase to new code base
[
https://issues.apache.org/jira/browse/FLINK-10405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-10405:
---
Labels: pull-request-available (was: )
> Port JobManagerFailsITCase to new code base
> ---
>
> Key: FLINK-10405
> URL: https://issues.apache.org/jira/browse/FLINK-10405
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 1.7.0
>Reporter: TisonKun
>Assignee: TisonKun
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> Port {{JobManagerFailsITCase}} to new code base.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] yanghua commented on issue #6839: [FLINK-10253] Run MetricQueryService with lower priority
yanghua commented on issue #6839: [FLINK-10253] Run MetricQueryService with lower priority URL: https://github.com/apache/flink/pull/6839#issuecomment-429709386 > Thanks for your contribution @yanghua . Please refer to the README in flink-docs to regenerate the doc after your modify the option related classes in order to pass the test. Thanks for reminding me, I have updated the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10253) Run MetricQueryService with lower priority
[
https://issues.apache.org/jira/browse/FLINK-10253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649728#comment-16649728
]
ASF GitHub Bot commented on FLINK-10253:
yanghua commented on issue #6839: [FLINK-10253] Run MetricQueryService with
lower priority
URL: https://github.com/apache/flink/pull/6839#issuecomment-429709386
> Thanks for your contribution @yanghua . Please refer to the README in
flink-docs to regenerate the doc after your modify the option related classes
in order to pass the test.
Thanks for reminding me, I have updated the PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Run MetricQueryService with lower priority
> --
>
> Key: FLINK-10253
> URL: https://issues.apache.org/jira/browse/FLINK-10253
> Project: Flink
> Issue Type: Sub-task
> Components: Metrics
>Affects Versions: 1.5.3, 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.7.0, 1.6.2, 1.5.5
>
>
> We should run the {{MetricQueryService}} with a lower priority than the main
> Flink components. An idea would be to start the underlying threads with a
> lower priority.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] yanghua commented on issue #6839: [FLINK-10253] Run MetricQueryService with lower priority
yanghua commented on issue #6839: [FLINK-10253] Run MetricQueryService with lower priority URL: https://github.com/apache/flink/pull/6839#issuecomment-429710107 > One more question, should we expose the threads priority configuration to the user? The purpose of this pr is to lower the priority of metric query service, but the user can configure the threads priority to 10. Maybe we should at least restrict the maximum of threads priority. Before diving into the code, I am waiting for inputs from @tillrohrmann . I think the configuration provides flexibility, which allows us to make adjustments without changing the code. But the configuration brings the possibility of unreasonable settings, which is bound to exist. Users always have to pay for his actions if they ignore warnings. Of course, as a protective measure, limiting the maximum value he can set, sounds like a good idea. If @tillrohrmann thinks this plan is worth adopting, then I will increase this limit. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10253) Run MetricQueryService with lower priority
[
https://issues.apache.org/jira/browse/FLINK-10253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649729#comment-16649729
]
ASF GitHub Bot commented on FLINK-10253:
yanghua commented on issue #6839: [FLINK-10253] Run MetricQueryService with
lower priority
URL: https://github.com/apache/flink/pull/6839#issuecomment-429710107
> One more question, should we expose the threads priority configuration to
the user? The purpose of this pr is to lower the priority of metric query
service, but the user can configure the threads priority to 10. Maybe we should
at least restrict the maximum of threads priority. Before diving into the code,
I am waiting for inputs from @tillrohrmann .
I think the configuration provides flexibility, which allows us to make
adjustments without changing the code. But the configuration brings the
possibility of unreasonable settings, which is bound to exist. Users always
have to pay for his actions if they ignore warnings. Of course, as a protective
measure, limiting the maximum value he can set, sounds like a good idea. If
@tillrohrmann thinks this plan is worth adopting, then I will increase this
limit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Run MetricQueryService with lower priority
> --
>
> Key: FLINK-10253
> URL: https://issues.apache.org/jira/browse/FLINK-10253
> Project: Flink
> Issue Type: Sub-task
> Components: Metrics
>Affects Versions: 1.5.3, 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.7.0, 1.6.2, 1.5.5
>
>
> We should run the {{MetricQueryService}} with a lower priority than the main
> Flink components. An idea would be to start the underlying threads with a
> lower priority.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10546) Remove StandaloneMiniCluster
TisonKun created FLINK-10546:
Summary: Remove StandaloneMiniCluster
Key: FLINK-10546
URL: https://issues.apache.org/jira/browse/FLINK-10546
Project: Flink
Issue Type: Task
Components: JobManager
Affects Versions: 1.7.0
Reporter: TisonKun
Assignee: TisonKun
Fix For: 1.7.0
Before doing as title, I want to check that is the {{StandaloneMiniCluster}}
still in use?
IIRC there once be a deprecation of `start-local.sh` but I don’t know if it is
relevant.
Further, this class seems unused in all other place. Since it depends on
legacy mode, I wonder whether we can *JUST* remove it.
cc [~till.rohrmann] and [~Zentol]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10546) Remove StandaloneMiniCluster
[
https://issues.apache.org/jira/browse/FLINK-10546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
TisonKun updated FLINK-10546:
-
Description:
Before doing as title, I want to check that is the {{StandaloneMiniCluster}}
still in use?
IIRC there once be a deprecation of {{start-local.sh}} but I don’t know if it
is relevant.
Further, this class seems unused in all other place. Since it depends on
legacy mode, I wonder whether we can *JUST* remove it.
cc [~till.rohrmann] and [~Zentol]
was:
Before doing as title, I want to check that is the {{StandaloneMiniCluster}}
still in use?
IIRC there once be a deprecation of `start-local.sh` but I don’t know if it is
relevant.
Further, this class seems unused in all other place. Since it depends on
legacy mode, I wonder whether we can *JUST* remove it.
cc [~till.rohrmann] and [~Zentol]
> Remove StandaloneMiniCluster
>
>
> Key: FLINK-10546
> URL: https://issues.apache.org/jira/browse/FLINK-10546
> Project: Flink
> Issue Type: Task
> Components: JobManager
>Affects Versions: 1.7.0
>Reporter: TisonKun
>Assignee: TisonKun
>Priority: Major
> Fix For: 1.7.0
>
>
> Before doing as title, I want to check that is the {{StandaloneMiniCluster}}
> still in use?
> IIRC there once be a deprecation of {{start-local.sh}} but I don’t know if
> it is relevant.
> Further, this class seems unused in all other place. Since it depends on
> legacy mode, I wonder whether we can *JUST* remove it.
>
> cc [~till.rohrmann] and [~Zentol]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10540) Remove legacy FlinkMiniCluster
[
https://issues.apache.org/jira/browse/FLINK-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
TisonKun updated FLINK-10540:
-
Summary: Remove legacy FlinkMiniCluster (was: Remove legacy
LocalFlinkMiniCluster)
> Remove legacy FlinkMiniCluster
> --
>
> Key: FLINK-10540
> URL: https://issues.apache.org/jira/browse/FLINK-10540
> Project: Flink
> Issue Type: Sub-task
>Affects Versions: 1.7.0
>Reporter: TisonKun
>Priority: Major
> Fix For: 1.7.0
>
>
> {{LocalFlinkMiniCluster}} is based on legacy cluster mode and should be no
> longer used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10540) Remove legacy FlinkMiniCluster
[
https://issues.apache.org/jira/browse/FLINK-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
TisonKun updated FLINK-10540:
-
Description: {{FlinkMiniCluster}} is based on legacy cluster mode and
should be no longer used. (was: {{LocalFlinkMiniCluster}} is based on legacy
cluster mode and should be no longer used.)
> Remove legacy FlinkMiniCluster
> --
>
> Key: FLINK-10540
> URL: https://issues.apache.org/jira/browse/FLINK-10540
> Project: Flink
> Issue Type: Sub-task
>Affects Versions: 1.7.0
>Reporter: TisonKun
>Priority: Major
> Fix For: 1.7.0
>
>
> {{FlinkMiniCluster}} is based on legacy cluster mode and should be no longer
> used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10135) Certain cluster-level metrics are no longer exposed
[
https://issues.apache.org/jira/browse/FLINK-10135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-10135:
-
Summary: Certain cluster-level metrics are no longer exposed (was: The
JobManager doesn't report the cluster-level metrics)
> Certain cluster-level metrics are no longer exposed
> ---
>
> Key: FLINK-10135
> URL: https://issues.apache.org/jira/browse/FLINK-10135
> Project: Flink
> Issue Type: Bug
> Components: JobManager, Metrics
>Affects Versions: 1.5.0, 1.6.0, 1.7.0
>Reporter: Joey Echeverria
>Assignee: vinoyang
>Priority: Critical
> Labels: pull-request-available
>
> In [the documentation for
> metrics|https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/metrics.html#cluster]
> in the Flink 1.5.0 release, it says that the following metrics are reported
> by the JobManager:
> {noformat}
> numRegisteredTaskManagers
> numRunningJobs
> taskSlotsAvailable
> taskSlotsTotal
> {noformat}
> In the job manager REST endpoint
> ({{http://:8081/jobmanager/metrics}}), those metrics don't
> appear.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] zentol closed pull request #6702: [FLINK-10135] The JobManager does not report the cluster-level metrics
zentol closed pull request #6702: [FLINK-10135] The JobManager does not report
the cluster-level metrics
URL: https://github.com/apache/flink/pull/6702
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosResourceManager.java
b/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosResourceManager.java
index 89d0cfd3452..ae51a2027e8 100644
---
a/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosResourceManager.java
+++
b/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosResourceManager.java
@@ -50,6 +50,7 @@
import org.apache.flink.runtime.heartbeat.HeartbeatServices;
import org.apache.flink.runtime.highavailability.HighAvailabilityServices;
import org.apache.flink.runtime.metrics.MetricRegistry;
+import org.apache.flink.runtime.metrics.groups.JobManagerMetricGroup;
import org.apache.flink.runtime.resourcemanager.JobLeaderIdService;
import org.apache.flink.runtime.resourcemanager.ResourceManager;
import
org.apache.flink.runtime.resourcemanager.exceptions.ResourceManagerException;
@@ -165,7 +166,8 @@ public MesosResourceManager(
MesosConfiguration mesosConfig,
MesosTaskManagerParameters taskManagerParameters,
ContainerSpecification taskManagerContainerSpec,
- @Nullable String webUiUrl) {
+ @Nullable String webUiUrl,
+ JobManagerMetricGroup jobManagerMetricGroup) {
super(
rpcService,
resourceManagerEndpointId,
@@ -176,7 +178,8 @@ public MesosResourceManager(
metricRegistry,
jobLeaderIdService,
clusterInformation,
- fatalErrorHandler);
+ fatalErrorHandler,
+ jobManagerMetricGroup);
this.mesosServices = Preconditions.checkNotNull(mesosServices);
this.actorSystem =
Preconditions.checkNotNull(mesosServices.getLocalActorSystem());
diff --git
a/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosResourceManagerFactory.java
b/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosResourceManagerFactory.java
index 9582e9f2e23..f53ffb301a8 100644
---
a/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosResourceManagerFactory.java
+++
b/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosResourceManagerFactory.java
@@ -27,6 +27,7 @@
import org.apache.flink.runtime.heartbeat.HeartbeatServices;
import org.apache.flink.runtime.highavailability.HighAvailabilityServices;
import org.apache.flink.runtime.metrics.MetricRegistry;
+import org.apache.flink.runtime.metrics.groups.JobManagerMetricGroup;
import org.apache.flink.runtime.resourcemanager.ResourceManager;
import org.apache.flink.runtime.resourcemanager.ResourceManagerFactory;
import org.apache.flink.runtime.resourcemanager.ResourceManagerRuntimeServices;
@@ -62,7 +63,17 @@ public MesosResourceManagerFactory(@Nonnull MesosServices
mesosServices, @Nonnul
}
@Override
- public ResourceManager
createResourceManager(Configuration configuration, ResourceID resourceId,
RpcService rpcService, HighAvailabilityServices highAvailabilityServices,
HeartbeatServices heartbeatServices, MetricRegistry metricRegistry,
FatalErrorHandler fatalErrorHandler, ClusterInformation clusterInformation,
@Nullable String webInterfaceUrl) throws Exception {
+ public ResourceManager createResourceManager(
+ Configuration configuration,
+ ResourceID resourceId,
+ RpcService rpcService,
+ HighAvailabilityServices highAvailabilityServices,
+ HeartbeatServices heartbeatServices,
+ MetricRegistry metricRegistry,
+ FatalErrorHandler fatalErrorHandler,
+ ClusterInformation clusterInformation,
+ @Nullable String webInterfaceUrl,
+ JobManagerMetricGroup jobManagerMetricGroup) throws
Exception {
final ResourceManagerRuntimeServicesConfiguration
rmServicesConfiguration =
ResourceManagerRuntimeServicesConfiguration.fromConfiguration(configuration);
final ResourceManagerRuntimeServices rmRuntimeServices =
ResourceManagerRuntimeServices.fromConfiguration(
rmServicesConfiguration,
@@ -85,6 +96,7 @@ public MesosResourceM
[jira] [Commented] (FLINK-10135) The JobManager doesn't report the cluster-level metrics
[
https://issues.apache.org/jira/browse/FLINK-10135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649752#comment-16649752
]
ASF GitHub Bot commented on FLINK-10135:
zentol closed pull request #6702: [FLINK-10135] The JobManager does not report
the cluster-level metrics
URL: https://github.com/apache/flink/pull/6702
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosResourceManager.java
b/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosResourceManager.java
index 89d0cfd3452..ae51a2027e8 100644
---
a/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosResourceManager.java
+++
b/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosResourceManager.java
@@ -50,6 +50,7 @@
import org.apache.flink.runtime.heartbeat.HeartbeatServices;
import org.apache.flink.runtime.highavailability.HighAvailabilityServices;
import org.apache.flink.runtime.metrics.MetricRegistry;
+import org.apache.flink.runtime.metrics.groups.JobManagerMetricGroup;
import org.apache.flink.runtime.resourcemanager.JobLeaderIdService;
import org.apache.flink.runtime.resourcemanager.ResourceManager;
import
org.apache.flink.runtime.resourcemanager.exceptions.ResourceManagerException;
@@ -165,7 +166,8 @@ public MesosResourceManager(
MesosConfiguration mesosConfig,
MesosTaskManagerParameters taskManagerParameters,
ContainerSpecification taskManagerContainerSpec,
- @Nullable String webUiUrl) {
+ @Nullable String webUiUrl,
+ JobManagerMetricGroup jobManagerMetricGroup) {
super(
rpcService,
resourceManagerEndpointId,
@@ -176,7 +178,8 @@ public MesosResourceManager(
metricRegistry,
jobLeaderIdService,
clusterInformation,
- fatalErrorHandler);
+ fatalErrorHandler,
+ jobManagerMetricGroup);
this.mesosServices = Preconditions.checkNotNull(mesosServices);
this.actorSystem =
Preconditions.checkNotNull(mesosServices.getLocalActorSystem());
diff --git
a/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosResourceManagerFactory.java
b/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosResourceManagerFactory.java
index 9582e9f2e23..f53ffb301a8 100644
---
a/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosResourceManagerFactory.java
+++
b/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosResourceManagerFactory.java
@@ -27,6 +27,7 @@
import org.apache.flink.runtime.heartbeat.HeartbeatServices;
import org.apache.flink.runtime.highavailability.HighAvailabilityServices;
import org.apache.flink.runtime.metrics.MetricRegistry;
+import org.apache.flink.runtime.metrics.groups.JobManagerMetricGroup;
import org.apache.flink.runtime.resourcemanager.ResourceManager;
import org.apache.flink.runtime.resourcemanager.ResourceManagerFactory;
import org.apache.flink.runtime.resourcemanager.ResourceManagerRuntimeServices;
@@ -62,7 +63,17 @@ public MesosResourceManagerFactory(@Nonnull MesosServices
mesosServices, @Nonnul
}
@Override
- public ResourceManager
createResourceManager(Configuration configuration, ResourceID resourceId,
RpcService rpcService, HighAvailabilityServices highAvailabilityServices,
HeartbeatServices heartbeatServices, MetricRegistry metricRegistry,
FatalErrorHandler fatalErrorHandler, ClusterInformation clusterInformation,
@Nullable String webInterfaceUrl) throws Exception {
+ public ResourceManager createResourceManager(
+ Configuration configuration,
+ ResourceID resourceId,
+ RpcService rpcService,
+ HighAvailabilityServices highAvailabilityServices,
+ HeartbeatServices heartbeatServices,
+ MetricRegistry metricRegistry,
+ FatalErrorHandler fatalErrorHandler,
+ ClusterInformation clusterInformation,
+ @Nullable String webInterfaceUrl,
+ JobManagerMetricGroup jobManagerMetricGroup) throws
Exception {
final ResourceManagerRuntimeServicesConfiguration
rmServicesConfiguration =
ResourceManagerRuntimeServices
[jira] [Updated] (FLINK-10135) Certain cluster-level metrics are no longer exposed
[
https://issues.apache.org/jira/browse/FLINK-10135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-10135:
-
Fix Version/s: 1.5.5
1.6.2
1.7.0
> Certain cluster-level metrics are no longer exposed
> ---
>
> Key: FLINK-10135
> URL: https://issues.apache.org/jira/browse/FLINK-10135
> Project: Flink
> Issue Type: Bug
> Components: JobManager, Metrics
>Affects Versions: 1.5.0, 1.6.0, 1.7.0
>Reporter: Joey Echeverria
>Assignee: vinoyang
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.7.0, 1.6.2, 1.5.5
>
>
> In [the documentation for
> metrics|https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/metrics.html#cluster]
> in the Flink 1.5.0 release, it says that the following metrics are reported
> by the JobManager:
> {noformat}
> numRegisteredTaskManagers
> numRunningJobs
> taskSlotsAvailable
> taskSlotsTotal
> {noformat}
> In the job manager REST endpoint
> ({{http://:8081/jobmanager/metrics}}), those metrics don't
> appear.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10135) Certain cluster-level metrics are no longer exposed
[
https://issues.apache.org/jira/browse/FLINK-10135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649753#comment-16649753
]
Chesnay Schepler commented on FLINK-10135:
--
master: 4ff6483f367591b894726ed1874c813c786b588a
> Certain cluster-level metrics are no longer exposed
> ---
>
> Key: FLINK-10135
> URL: https://issues.apache.org/jira/browse/FLINK-10135
> Project: Flink
> Issue Type: Bug
> Components: JobManager, Metrics
>Affects Versions: 1.5.0, 1.6.0, 1.7.0
>Reporter: Joey Echeverria
>Assignee: vinoyang
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.7.0, 1.6.2, 1.5.5
>
>
> In [the documentation for
> metrics|https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/metrics.html#cluster]
> in the Flink 1.5.0 release, it says that the following metrics are reported
> by the JobManager:
> {noformat}
> numRegisteredTaskManagers
> numRunningJobs
> taskSlotsAvailable
> taskSlotsTotal
> {noformat}
> In the job manager REST endpoint
> ({{http://:8081/jobmanager/metrics}}), those metrics don't
> appear.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10547) Remove LegacyCLI
vinoyang created FLINK-10547: Summary: Remove LegacyCLI Key: FLINK-10547 URL: https://issues.apache.org/jira/browse/FLINK-10547 Project: Flink Issue Type: Sub-task Reporter: vinoyang Assignee: vinoyang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10548) Remove LegacyStandaloneClusterDescriptor
vinoyang created FLINK-10548: Summary: Remove LegacyStandaloneClusterDescriptor Key: FLINK-10548 URL: https://issues.apache.org/jira/browse/FLINK-10548 Project: Flink Issue Type: Sub-task Reporter: vinoyang Assignee: vinoyang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10549) Remove LegacyJobRetrievalITCase
TisonKun created FLINK-10549:
Summary: Remove LegacyJobRetrievalITCase
Key: FLINK-10549
URL: https://issues.apache.org/jira/browse/FLINK-10549
Project: Flink
Issue Type: Sub-task
Components: Tests
Affects Versions: 1.7.0
Reporter: TisonKun
Assignee: TisonKun
Fix For: 1.7.0
We already have {{JobRetrievalITCase}} and all test cases of
{{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10549) Remove Legacy* Tests
[
https://issues.apache.org/jira/browse/FLINK-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
TisonKun updated FLINK-10549:
-
Summary: Remove Legacy* Tests (was: Remove LegacyJobRetrievalITCase)
> Remove Legacy* Tests
>
>
> Key: FLINK-10549
> URL: https://issues.apache.org/jira/browse/FLINK-10549
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 1.7.0
>Reporter: TisonKun
>Assignee: TisonKun
>Priority: Major
> Fix For: 1.7.0
>
>
> We already have {{JobRetrievalITCase}} and all test cases of
> {{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10549) Remove Legacy* Tests
[
https://issues.apache.org/jira/browse/FLINK-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
TisonKun updated FLINK-10549:
-
Description:
This Jira tracks the removal of tests that start with "LegacyXXX" and covered
by a test named "XXX" and covering the same topic.
1. We already have {{JobRetrievalITCase}} and all test cases of
{{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
was:We already have {{JobRetrievalITCase}} and all test cases of
{{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
> Remove Legacy* Tests
>
>
> Key: FLINK-10549
> URL: https://issues.apache.org/jira/browse/FLINK-10549
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 1.7.0
>Reporter: TisonKun
>Assignee: TisonKun
>Priority: Major
> Fix For: 1.7.0
>
>
> This Jira tracks the removal of tests that start with "LegacyXXX" and covered
> by a test named "XXX" and covering the same topic.
> 1. We already have {{JobRetrievalITCase}} and all test cases of
> {{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10549) Remove Legacy* Tests
[
https://issues.apache.org/jira/browse/FLINK-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
TisonKun updated FLINK-10549:
-
Description:
This Jira tracks the removal of tests that start with "LegacyXXX" and covered
by a test named "XXX" and covering the same topic.
# We already have {{JobRetrievalITCase}} and all test cases of
{{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
# We already have {{AccumulatorLiveITCase}} and all test cases of
{{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
was:
This Jira tracks the removal of tests that start with "LegacyXXX" and covered
by a test named "XXX" and covering the same topic.
1. We already have {{JobRetrievalITCase}} and all test cases of
{{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
> Remove Legacy* Tests
>
>
> Key: FLINK-10549
> URL: https://issues.apache.org/jira/browse/FLINK-10549
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 1.7.0
>Reporter: TisonKun
>Assignee: TisonKun
>Priority: Major
> Fix For: 1.7.0
>
>
> This Jira tracks the removal of tests that start with "LegacyXXX" and covered
> by a test named "XXX" and covering the same topic.
> # We already have {{JobRetrievalITCase}} and all test cases of
> {{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
> # We already have {{AccumulatorLiveITCase}} and all test cases of
> {{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10549) Remove Legacy* Tests
[
https://issues.apache.org/jira/browse/FLINK-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
TisonKun updated FLINK-10549:
-
Description:
This Jira tracks the removal of tests that start with "LegacyXXX" and covered
by a test named "XXX" and covering the same topic.
# We already have {{JobRetrievalITCase}} and all test cases of
{{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
# We already have {{AccumulatorLiveITCase}} and all test cases of
{{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
# We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test cases
of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just simply
remove it.
was:
This Jira tracks the removal of tests that start with "LegacyXXX" and covered
by a test named "XXX" and covering the same topic.
# We already have {{JobRetrievalITCase}} and all test cases of
{{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
# We already have {{AccumulatorLiveITCase}} and all test cases of
{{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
> Remove Legacy* Tests
>
>
> Key: FLINK-10549
> URL: https://issues.apache.org/jira/browse/FLINK-10549
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 1.7.0
>Reporter: TisonKun
>Assignee: TisonKun
>Priority: Major
> Fix For: 1.7.0
>
>
> This Jira tracks the removal of tests that start with "LegacyXXX" and covered
> by a test named "XXX" and covering the same topic.
> # We already have {{JobRetrievalITCase}} and all test cases of
> {{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
> # We already have {{AccumulatorLiveITCase}} and all test cases of
> {{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
> # We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test
> cases of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just
> simply remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10549) Remove Legacy* Tests based on legacy mode
[
https://issues.apache.org/jira/browse/FLINK-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
TisonKun updated FLINK-10549:
-
Summary: Remove Legacy* Tests based on legacy mode (was: Remove Legacy*
Tests)
> Remove Legacy* Tests based on legacy mode
> -
>
> Key: FLINK-10549
> URL: https://issues.apache.org/jira/browse/FLINK-10549
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 1.7.0
>Reporter: TisonKun
>Assignee: TisonKun
>Priority: Major
> Fix For: 1.7.0
>
>
> This Jira tracks the removal of tests that start with "LegacyXXX" and covered
> by a test named "XXX" and covering the same topic.
> # We already have {{JobRetrievalITCase}} and all test cases of
> {{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
> # We already have {{AccumulatorLiveITCase}} and all test cases of
> {{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
> # We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test
> cases of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just
> simply remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10549) Remove Legacy* Tests based on legacy mode
[
https://issues.apache.org/jira/browse/FLINK-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
TisonKun updated FLINK-10549:
-
Description:
This Jira tracks the removal of tests based on legacy mode and starting with
"LegacyXXX" while covered by a test named "XXX" .
# We already have {{JobRetrievalITCase}} and all test cases of
{{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
# We already have {{AccumulatorLiveITCase}} and all test cases of
{{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
# We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test cases
of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just simply
remove it.
was:
This Jira tracks the removal of tests based on legacy mode and starting with
"LegacyXXX" while covered by a test named "XXX" and covering the same topic.
# We already have {{JobRetrievalITCase}} and all test cases of
{{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
# We already have {{AccumulatorLiveITCase}} and all test cases of
{{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
# We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test cases
of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just simply
remove it.
> Remove Legacy* Tests based on legacy mode
> -
>
> Key: FLINK-10549
> URL: https://issues.apache.org/jira/browse/FLINK-10549
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 1.7.0
>Reporter: TisonKun
>Assignee: TisonKun
>Priority: Major
> Fix For: 1.7.0
>
>
> This Jira tracks the removal of tests based on legacy mode and starting with
> "LegacyXXX" while covered by a test named "XXX" .
> # We already have {{JobRetrievalITCase}} and all test cases of
> {{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
> # We already have {{AccumulatorLiveITCase}} and all test cases of
> {{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
> # We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test
> cases of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just
> simply remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10549) Remove Legacy* Tests based on legacy mode
[
https://issues.apache.org/jira/browse/FLINK-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
TisonKun updated FLINK-10549:
-
Description:
This Jira tracks the removal of tests based on legacy mode and starting with
"LegacyXXX" while covered by a test named "XXX" .
# We already have {{JobRetrievalITCase}} and all test cases of
{{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
# We already have {{AccumulatorLiveITCase}} and all test cases of
{{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
# We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test cases
of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just simply
remove it.
# We already have {{ClassLoaderITCase}} and all test cases of
{{LegacyClassLoaderITCase}} are covered. Just simply remove it.
was:
This Jira tracks the removal of tests based on legacy mode and starting with
"LegacyXXX" while covered by a test named "XXX" .
# We already have {{JobRetrievalITCase}} and all test cases of
{{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
# We already have {{AccumulatorLiveITCase}} and all test cases of
{{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
# We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test cases
of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just simply
remove it.
> Remove Legacy* Tests based on legacy mode
> -
>
> Key: FLINK-10549
> URL: https://issues.apache.org/jira/browse/FLINK-10549
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 1.7.0
>Reporter: TisonKun
>Assignee: TisonKun
>Priority: Major
> Fix For: 1.7.0
>
>
> This Jira tracks the removal of tests based on legacy mode and starting with
> "LegacyXXX" while covered by a test named "XXX" .
> # We already have {{JobRetrievalITCase}} and all test cases of
> {{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
> # We already have {{AccumulatorLiveITCase}} and all test cases of
> {{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
> # We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test
> cases of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just
> simply remove it.
> # We already have {{ClassLoaderITCase}} and all test cases of
> {{LegacyClassLoaderITCase}} are covered. Just simply remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10549) Remove Legacy* Tests based on legacy mode
[
https://issues.apache.org/jira/browse/FLINK-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
TisonKun updated FLINK-10549:
-
Description:
This Jira tracks the removal of tests based on legacy mode and starting with
"LegacyXXX" while covered by a test named "XXX" .
# We already have {{JobRetrievalITCase}} and all test cases of
{{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
# We already have {{AccumulatorLiveITCase}} and all test cases of
{{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
# We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test cases
of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just simply
remove it.
# We already have {{ClassLoaderITCase}} and all test cases of
{{LegacyClassLoaderITCase}} are covered. Just simply remove it.
# We already have {{SlotCountExceedingParallelismTest}} and all test cases of
{{LegacySlotCountExceedingParallelismTest}} are covered. Just simply remove it.
was:
This Jira tracks the removal of tests based on legacy mode and starting with
"LegacyXXX" while covered by a test named "XXX" .
# We already have {{JobRetrievalITCase}} and all test cases of
{{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
# We already have {{AccumulatorLiveITCase}} and all test cases of
{{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
# We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test cases
of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just simply
remove it.
# We already have {{ClassLoaderITCase}} and all test cases of
{{LegacyClassLoaderITCase}} are covered. Just simply remove it.
> Remove Legacy* Tests based on legacy mode
> -
>
> Key: FLINK-10549
> URL: https://issues.apache.org/jira/browse/FLINK-10549
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 1.7.0
>Reporter: TisonKun
>Assignee: TisonKun
>Priority: Major
> Fix For: 1.7.0
>
>
> This Jira tracks the removal of tests based on legacy mode and starting with
> "LegacyXXX" while covered by a test named "XXX" .
> # We already have {{JobRetrievalITCase}} and all test cases of
> {{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
> # We already have {{AccumulatorLiveITCase}} and all test cases of
> {{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
> # We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test
> cases of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just
> simply remove it.
> # We already have {{ClassLoaderITCase}} and all test cases of
> {{LegacyClassLoaderITCase}} are covered. Just simply remove it.
> # We already have {{SlotCountExceedingParallelismTest}} and all test cases
> of {{LegacySlotCountExceedingParallelismTest}} are covered. Just simply
> remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10549) Remove Legacy* Tests based on legacy mode
[
https://issues.apache.org/jira/browse/FLINK-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
TisonKun updated FLINK-10549:
-
Description:
This Jira tracks the removal of tests based on legacy mode and starting with
"LegacyXXX" while covered by a test named "XXX" .
# We already have {{JobRetrievalITCase}} and all test cases of
{{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
# We already have {{AccumulatorLiveITCase}} and all test cases of
{{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
# We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test cases
of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just simply
remove it.
# We already have {{ClassLoaderITCase}} and all test cases of
{{LegacyClassLoaderITCase}} are covered. Just simply remove it.
# We already have {{SlotCountExceedingParallelismTest}} and all test cases of
{{LegacySlotCountExceedingParallelismTest}} are covered. Just simply remove it.
# We already have {{ScheduleOrUpdateConsumersTest}} and all test cases of
{{LegacyScheduleOrUpdateConsumersTest}} are covered. Just simply remove it.
was:
This Jira tracks the removal of tests based on legacy mode and starting with
"LegacyXXX" while covered by a test named "XXX" .
# We already have {{JobRetrievalITCase}} and all test cases of
{{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
# We already have {{AccumulatorLiveITCase}} and all test cases of
{{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
# We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test cases
of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just simply
remove it.
# We already have {{ClassLoaderITCase}} and all test cases of
{{LegacyClassLoaderITCase}} are covered. Just simply remove it.
# We already have {{SlotCountExceedingParallelismTest}} and all test cases of
{{LegacySlotCountExceedingParallelismTest}} are covered. Just simply remove it.
> Remove Legacy* Tests based on legacy mode
> -
>
> Key: FLINK-10549
> URL: https://issues.apache.org/jira/browse/FLINK-10549
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 1.7.0
>Reporter: TisonKun
>Assignee: TisonKun
>Priority: Major
> Fix For: 1.7.0
>
>
> This Jira tracks the removal of tests based on legacy mode and starting with
> "LegacyXXX" while covered by a test named "XXX" .
> # We already have {{JobRetrievalITCase}} and all test cases of
> {{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
> # We already have {{AccumulatorLiveITCase}} and all test cases of
> {{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
> # We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test
> cases of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just
> simply remove it.
> # We already have {{ClassLoaderITCase}} and all test cases of
> {{LegacyClassLoaderITCase}} are covered. Just simply remove it.
> # We already have {{SlotCountExceedingParallelismTest}} and all test cases
> of {{LegacySlotCountExceedingParallelismTest}} are covered. Just simply
> remove it.
> # We already have {{ScheduleOrUpdateConsumersTest}} and all test cases of
> {{LegacyScheduleOrUpdateConsumersTest}} are covered. Just simply remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10550) Remove LegacyYarnClusterDescriptor
vinoyang created FLINK-10550: Summary: Remove LegacyYarnClusterDescriptor Key: FLINK-10550 URL: https://issues.apache.org/jira/browse/FLINK-10550 Project: Flink Issue Type: Sub-task Reporter: vinoyang Assignee: vinoyang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-10549) Remove Legacy* Tests based on legacy mode
[
https://issues.apache.org/jira/browse/FLINK-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
TisonKun updated FLINK-10549:
-
Description:
This Jira tracks the removal of tests based on legacy mode and starting with
"LegacyXXX" while covered by a test named "XXX" .
# We already have {{JobRetrievalITCase}} and all test cases of
{{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
# We already have {{AccumulatorLiveITCase}} and all test cases of
{{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
# We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test cases
of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just simply
remove it.
# We already have {{ClassLoaderITCase}} and all test cases of
{{LegacyClassLoaderITCase}} are covered. Just simply remove it.
# We already have {{SlotCountExceedingParallelismTest}} and all test cases of
{{LegacySlotCountExceedingParallelismTest}} are covered. Just simply remove it.
# We already have {{ScheduleOrUpdateConsumersTest}} and all test cases of
{{LegacyScheduleOrUpdateConsumersTest}} are covered. Just simply remove it.
# We already have {{PartialConsumePipelinedResultTest}} and all test cases of
{{LegacyPartialConsumePipelinedResultTest}} are covered. Just simply remove it.
was:
This Jira tracks the removal of tests based on legacy mode and starting with
"LegacyXXX" while covered by a test named "XXX" .
# We already have {{JobRetrievalITCase}} and all test cases of
{{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
# We already have {{AccumulatorLiveITCase}} and all test cases of
{{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
# We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test cases
of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just simply
remove it.
# We already have {{ClassLoaderITCase}} and all test cases of
{{LegacyClassLoaderITCase}} are covered. Just simply remove it.
# We already have {{SlotCountExceedingParallelismTest}} and all test cases of
{{LegacySlotCountExceedingParallelismTest}} are covered. Just simply remove it.
# We already have {{ScheduleOrUpdateConsumersTest}} and all test cases of
{{LegacyScheduleOrUpdateConsumersTest}} are covered. Just simply remove it.
> Remove Legacy* Tests based on legacy mode
> -
>
> Key: FLINK-10549
> URL: https://issues.apache.org/jira/browse/FLINK-10549
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 1.7.0
>Reporter: TisonKun
>Assignee: TisonKun
>Priority: Major
> Fix For: 1.7.0
>
>
> This Jira tracks the removal of tests based on legacy mode and starting with
> "LegacyXXX" while covered by a test named "XXX" .
> # We already have {{JobRetrievalITCase}} and all test cases of
> {{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
> # We already have {{AccumulatorLiveITCase}} and all test cases of
> {{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
> # We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test
> cases of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just
> simply remove it.
> # We already have {{ClassLoaderITCase}} and all test cases of
> {{LegacyClassLoaderITCase}} are covered. Just simply remove it.
> # We already have {{SlotCountExceedingParallelismTest}} and all test cases
> of {{LegacySlotCountExceedingParallelismTest}} are covered. Just simply
> remove it.
> # We already have {{ScheduleOrUpdateConsumersTest}} and all test cases of
> {{LegacyScheduleOrUpdateConsumersTest}} are covered. Just simply remove it.
> # We already have {{PartialConsumePipelinedResultTest}} and all test cases
> of {{LegacyPartialConsumePipelinedResultTest}} are covered. Just simply
> remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10549) Remove Legacy* Tests based on legacy mode
[
https://issues.apache.org/jira/browse/FLINK-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
TisonKun updated FLINK-10549:
-
Description:
This Jira tracks the removal of tests based on legacy mode and starting with
"LegacyXXX" while covered by a test named "XXX" and covering the same topic.
# We already have {{JobRetrievalITCase}} and all test cases of
{{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
# We already have {{AccumulatorLiveITCase}} and all test cases of
{{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
# We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test cases
of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just simply
remove it.
was:
This Jira tracks the removal of tests that start with "LegacyXXX" and covered
by a test named "XXX" and covering the same topic.
# We already have {{JobRetrievalITCase}} and all test cases of
{{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
# We already have {{AccumulatorLiveITCase}} and all test cases of
{{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
# We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test cases
of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just simply
remove it.
> Remove Legacy* Tests based on legacy mode
> -
>
> Key: FLINK-10549
> URL: https://issues.apache.org/jira/browse/FLINK-10549
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 1.7.0
>Reporter: TisonKun
>Assignee: TisonKun
>Priority: Major
> Fix For: 1.7.0
>
>
> This Jira tracks the removal of tests based on legacy mode and starting with
> "LegacyXXX" while covered by a test named "XXX" and covering the same topic.
> # We already have {{JobRetrievalITCase}} and all test cases of
> {{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
> # We already have {{AccumulatorLiveITCase}} and all test cases of
> {{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
> # We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test
> cases of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just
> simply remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10549) Remove Legacy* Tests based on legacy mode
[
https://issues.apache.org/jira/browse/FLINK-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
TisonKun updated FLINK-10549:
-
Description:
This Jira tracks the removal of tests based on legacy mode and starting with
"LegacyXXX" while covered by a test named "XXX" .
# We already have {{JobRetrievalITCase}} and all test cases of
{{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
# We already have {{AccumulatorLiveITCase}} and all test cases of
{{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
# We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test cases
of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just simply
remove it.
# We already have {{ClassLoaderITCase}} and all test cases of
{{LegacyClassLoaderITCase}} are covered. Just simply remove it.
# We already have {{SlotCountExceedingParallelismTest}} and all test cases of
{{LegacySlotCountExceedingParallelismTest}} are covered. Just simply remove it.
# We already have {{ScheduleOrUpdateConsumersTest}} and all test cases of
{{LegacyScheduleOrUpdateConsumersTest}} are covered. Just simply remove it.
# We already have {{PartialConsumePipelinedResultTest}} and all test cases of
{{LegacyPartialConsumePipelinedResultTest}} are covered. Just simply remove it.
# We already have {{AvroExternalJarProgramITCase}} and all test cases of
{{LegacyAvroExternalJarProgramITCase}} are covered. Just simply remove it.
was:
This Jira tracks the removal of tests based on legacy mode and starting with
"LegacyXXX" while covered by a test named "XXX" .
# We already have {{JobRetrievalITCase}} and all test cases of
{{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
# We already have {{AccumulatorLiveITCase}} and all test cases of
{{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
# We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test cases
of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just simply
remove it.
# We already have {{ClassLoaderITCase}} and all test cases of
{{LegacyClassLoaderITCase}} are covered. Just simply remove it.
# We already have {{SlotCountExceedingParallelismTest}} and all test cases of
{{LegacySlotCountExceedingParallelismTest}} are covered. Just simply remove it.
# We already have {{ScheduleOrUpdateConsumersTest}} and all test cases of
{{LegacyScheduleOrUpdateConsumersTest}} are covered. Just simply remove it.
# We already have {{PartialConsumePipelinedResultTest}} and all test cases of
{{LegacyPartialConsumePipelinedResultTest}} are covered. Just simply remove it.
> Remove Legacy* Tests based on legacy mode
> -
>
> Key: FLINK-10549
> URL: https://issues.apache.org/jira/browse/FLINK-10549
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 1.7.0
>Reporter: TisonKun
>Assignee: TisonKun
>Priority: Major
> Fix For: 1.7.0
>
>
> This Jira tracks the removal of tests based on legacy mode and starting with
> "LegacyXXX" while covered by a test named "XXX" .
> # We already have {{JobRetrievalITCase}} and all test cases of
> {{LegacyJobRetrievalITCase}} are covered. Just simply remove it.
> # We already have {{AccumulatorLiveITCase}} and all test cases of
> {{LegacyAccumulatorLiveITCase}} are covered. Just simply remove it.
> # We already have {{TaskCancelAsyncProducerConsumerITCase}} and all test
> cases of {{LegacyTaskCancelAsyncProducerConsumerITCase}} are covered. Just
> simply remove it.
> # We already have {{ClassLoaderITCase}} and all test cases of
> {{LegacyClassLoaderITCase}} are covered. Just simply remove it.
> # We already have {{SlotCountExceedingParallelismTest}} and all test cases
> of {{LegacySlotCountExceedingParallelismTest}} are covered. Just simply
> remove it.
> # We already have {{ScheduleOrUpdateConsumersTest}} and all test cases of
> {{LegacyScheduleOrUpdateConsumersTest}} are covered. Just simply remove it.
> # We already have {{PartialConsumePipelinedResultTest}} and all test cases
> of {{LegacyPartialConsumePipelinedResultTest}} are covered. Just simply
> remove it.
> # We already have {{AvroExternalJarProgramITCase}} and all test cases of
> {{LegacyAvroExternalJarProgramITCase}} are covered. Just simply remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] TisonKun opened a new pull request #6842: Flink 10549
TisonKun opened a new pull request #6842: Flink 10549 URL: https://github.com/apache/flink/pull/6842 ## What is the purpose of the change This pull request covers the removal of tests based on legacy mode and starting with "LegacyXXX" while covered by a test named "XXX" . Thanks @zentol a lot for previous porting job. Also cc @tillrohrmann . ## Brief change log - We already have JobRetrievalITCase and all test cases of LegacyJobRetrievalITCase are covered. Just simply remove it. - We already have AccumulatorLiveITCase and all test cases of LegacyAccumulatorLiveITCase are covered. Just simply remove it. - We already have TaskCancelAsyncProducerConsumerITCase and all test cases of LegacyTaskCancelAsyncProducerConsumerITCase are covered. Just simply remove it. - We already have ClassLoaderITCase and all test cases of LegacyClassLoaderITCase are covered. Just simply remove it. - We already have SlotCountExceedingParallelismTest and all test cases of LegacySlotCountExceedingParallelismTest are covered. Just simply remove it. - We already have ScheduleOrUpdateConsumersTest and all test cases of LegacyScheduleOrUpdateConsumersTest are covered. Just simply remove it. - We already have PartialConsumePipelinedResultTest and all test cases of LegacyPartialConsumePipelinedResultTest are covered. Just simply remove it. - We already have AvroExternalJarProgramITCase and all test cases of LegacyAvroExternalJarProgramITCase are covered. Just simply remove it. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature?no - If yes, how is the feature documented? no This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zhijiangW commented on a change in pull request #6833: [FLINK-10537][network] Fix network small performance degradation after merging [FLINK-9913]
zhijiangW commented on a change in pull request #6833: [FLINK-10537][network]
Fix network small performance degradation after merging [FLINK-9913]
URL: https://github.com/apache/flink/pull/6833#discussion_r225030697
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/serialization/SpanningRecordSerializer.java
##
@@ -92,20 +92,9 @@ public void serializeRecord(T record) throws IOException {
*/
@Override
public SerializationResult copyToBufferBuilder(BufferBuilder
targetBuffer) {
- boolean mustCommit = false;
- if (lengthBuffer.hasRemaining()) {
Review comment:
It is really not necessary to check the conditions for every record in
normal cases, and it may bring some overheads to do so.
But if we directly remove these conditions, in the case of spanning multiple
buffers because of large records or remaining small spaces in previous
`targetBuffer`, we may encounter another cost overhead. In this condition, the
length buffer is probably not remaining but only the data buffer is remaining,
and the cost of `targetBuffer.append(lengthBuffer)` would be higher than the
cost of `lengthBuffer.hasRemaining()`.
If we can check this condition on `RecordWriter` side as the following,
maybe we can check accordingly in `RecordSerializer`.
```
private boolean copyFromSerializerToTargetChannel(int targetChannel) {
serializer.reset();
boolean pruneTriggered = false;
BufferBuilder bufferBuilder = getBufferBuilder(targetChannel);
SerializationResult result =
serializer.copyToBufferBuilder(bufferBuilder, **false**);
while (result.isFullBuffer()) {
numBytesOut.inc(bufferBuilder.finish());
numBuffersOut.inc();
if (result.isFullRecord()) {
pruneTriggered = true;
bufferBuilders[targetChannel] =
Optional.empty();
break;
}
bufferBuilder = requestNewBufferBuilder(targetChannel);
result = serializer.copyToBufferBuilder(bufferBuilder,
**true**);
}
```
But I am not sure whether it is worth doing by giving this hint in
`copyToBufferBuilder` interface. What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] zhijiangW commented on a change in pull request #6833: [FLINK-10537][network] Fix network small performance degradation after merging [FLINK-9913]
zhijiangW commented on a change in pull request #6833: [FLINK-10537][network]
Fix network small performance degradation after merging [FLINK-9913]
URL: https://github.com/apache/flink/pull/6833#discussion_r225031331
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/serialization/SpanningRecordSerializer.java
##
@@ -127,6 +116,7 @@ public void reset() {
@Override
public void prune() {
+ serializationBuffer.clear();
Review comment:
The `serializationBuffer.clear()` will be called at the beginning of
`serializeRecord(T record)` and it seems no problem currently. Do you have
other concerns to add it in this process?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-10537) Network throughput performance regression after broadcast changes
[
https://issues.apache.org/jira/browse/FLINK-10537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649804#comment-16649804
]
ASF GitHub Bot commented on FLINK-10537:
zhijiangW commented on a change in pull request #6833: [FLINK-10537][network]
Fix network small performance degradation after merging [FLINK-9913]
URL: https://github.com/apache/flink/pull/6833#discussion_r225031331
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/serialization/SpanningRecordSerializer.java
##
@@ -127,6 +116,7 @@ public void reset() {
@Override
public void prune() {
+ serializationBuffer.clear();
Review comment:
The `serializationBuffer.clear()` will be called at the beginning of
`serializeRecord(T record)` and it seems no problem currently. Do you have
other concerns to add it in this process?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Network throughput performance regression after broadcast changes
> -
>
> Key: FLINK-10537
> URL: https://issues.apache.org/jira/browse/FLINK-10537
> Project: Flink
> Issue Type: Bug
> Components: Network
>Affects Versions: 1.7.0
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
>Priority: Major
> Labels: pull-request-available
>
> There is a slight network throughput regression introduced in:
> https://issues.apache.org/jira/browse/FLINK-9913
> It is visible in the following benchmark:
> [http://codespeed.dak8s.net:8000/timeline/#/?exe=1&ben=networkThroughput.1,100ms&env=2&revs=200&equid=off&quarts=on&extr=on]
> (drop in the chart that happened since 21st September.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10537) Network throughput performance regression after broadcast changes
[
https://issues.apache.org/jira/browse/FLINK-10537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649805#comment-16649805
]
ASF GitHub Bot commented on FLINK-10537:
zhijiangW commented on a change in pull request #6833: [FLINK-10537][network]
Fix network small performance degradation after merging [FLINK-9913]
URL: https://github.com/apache/flink/pull/6833#discussion_r225030697
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/serialization/SpanningRecordSerializer.java
##
@@ -92,20 +92,9 @@ public void serializeRecord(T record) throws IOException {
*/
@Override
public SerializationResult copyToBufferBuilder(BufferBuilder
targetBuffer) {
- boolean mustCommit = false;
- if (lengthBuffer.hasRemaining()) {
Review comment:
It is really not necessary to check the conditions for every record in
normal cases, and it may bring some overheads to do so.
But if we directly remove these conditions, in the case of spanning multiple
buffers because of large records or remaining small spaces in previous
`targetBuffer`, we may encounter another cost overhead. In this condition, the
length buffer is probably not remaining but only the data buffer is remaining,
and the cost of `targetBuffer.append(lengthBuffer)` would be higher than the
cost of `lengthBuffer.hasRemaining()`.
If we can check this condition on `RecordWriter` side as the following,
maybe we can check accordingly in `RecordSerializer`.
```
private boolean copyFromSerializerToTargetChannel(int targetChannel) {
serializer.reset();
boolean pruneTriggered = false;
BufferBuilder bufferBuilder = getBufferBuilder(targetChannel);
SerializationResult result =
serializer.copyToBufferBuilder(bufferBuilder, **false**);
while (result.isFullBuffer()) {
numBytesOut.inc(bufferBuilder.finish());
numBuffersOut.inc();
if (result.isFullRecord()) {
pruneTriggered = true;
bufferBuilders[targetChannel] =
Optional.empty();
break;
}
bufferBuilder = requestNewBufferBuilder(targetChannel);
result = serializer.copyToBufferBuilder(bufferBuilder,
**true**);
}
```
But I am not sure whether it is worth doing by giving this hint in
`copyToBufferBuilder` interface. What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Network throughput performance regression after broadcast changes
> -
>
> Key: FLINK-10537
> URL: https://issues.apache.org/jira/browse/FLINK-10537
> Project: Flink
> Issue Type: Bug
> Components: Network
>Affects Versions: 1.7.0
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
>Priority: Major
> Labels: pull-request-available
>
> There is a slight network throughput regression introduced in:
> https://issues.apache.org/jira/browse/FLINK-9913
> It is visible in the following benchmark:
> [http://codespeed.dak8s.net:8000/timeline/#/?exe=1&ben=networkThroughput.1,100ms&env=2&revs=200&equid=off&quarts=on&extr=on]
> (drop in the chart that happened since 21st September.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] zhijiangW commented on issue #6833: [FLINK-10537][network] Fix network small performance degradation after merging [FLINK-9913]
zhijiangW commented on issue #6833: [FLINK-10537][network] Fix network small performance degradation after merging [FLINK-9913] URL: https://github.com/apache/flink/pull/6833#issuecomment-429729793 @pnowojski , thanks for finding this potential issue which would bring regression in some cases. I agree with your point that we actually brought more condition checks for every record in my last JIRA, and this can be indeed improved to keep the previous behavior. For the implementation I just have another concern inside the comments. :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10537) Network throughput performance regression after broadcast changes
[ https://issues.apache.org/jira/browse/FLINK-10537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649816#comment-16649816 ] ASF GitHub Bot commented on FLINK-10537: zhijiangW commented on issue #6833: [FLINK-10537][network] Fix network small performance degradation after merging [FLINK-9913] URL: https://github.com/apache/flink/pull/6833#issuecomment-429729793 @pnowojski , thanks for finding this potential issue which would bring regression in some cases. I agree with your point that we actually brought more condition checks for every record in my last JIRA, and this can be indeed improved to keep the previous behavior. For the implementation I just have another concern inside the comments. :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Network throughput performance regression after broadcast changes > - > > Key: FLINK-10537 > URL: https://issues.apache.org/jira/browse/FLINK-10537 > Project: Flink > Issue Type: Bug > Components: Network >Affects Versions: 1.7.0 >Reporter: Piotr Nowojski >Assignee: Piotr Nowojski >Priority: Major > Labels: pull-request-available > > There is a slight network throughput regression introduced in: > https://issues.apache.org/jira/browse/FLINK-9913 > It is visible in the following benchmark: > [http://codespeed.dak8s.net:8000/timeline/#/?exe=1&ben=networkThroughput.1,100ms&env=2&revs=200&equid=off&quarts=on&extr=on] > (drop in the chart that happened since 21st September.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
