[GitHub] flink pull request #4208: [FLINK-7004] Switch to Travis Trusty image
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/4208 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-7004) Switch to Travis Trusty image
[ https://issues.apache.org/jira/browse/FLINK-7004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067892#comment-16067892 ] ASF GitHub Bot commented on FLINK-7004: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/4208 > Switch to Travis Trusty image > - > > Key: FLINK-7004 > URL: https://issues.apache.org/jira/browse/FLINK-7004 > Project: Flink > Issue Type: Improvement > Components: Tests >Affects Versions: 1.2.0, 1.3.0, 1.4.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Critical > Fix For: 1.2.2, 1.4.0, 1.3.2 > > > As shown in this PR https://github.com/apache/flink/pull/4167 switching to > the Trusty image on Travis seems to stabilize the build times. > We should switch for 1.2, 1.3 and 1.4. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink pull request #4207: [FLINK-7004] Switch to Travis Trusty image
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/4207 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4213: [FLINK-7032] Overwrite inherited properties from parent p...
Github user pnowojski commented on the issue: https://github.com/apache/flink/pull/4213 Yes. Should I not do this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-7032) Intellij is constantly changing language level of sub projects back to 1.6
[
https://issues.apache.org/jira/browse/FLINK-7032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067894#comment-16067894
]
ASF GitHub Bot commented on FLINK-7032:
---
Github user pnowojski commented on the issue:
https://github.com/apache/flink/pull/4213
Yes. Should I not do this?
> Intellij is constantly changing language level of sub projects back to 1.6
> ---
>
> Key: FLINK-7032
> URL: https://issues.apache.org/jira/browse/FLINK-7032
> Project: Flink
> Issue Type: Improvement
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
>
> Every time I do maven reimport projects, Intellij is switching back to 1.6
> language level. I tracked down this issue to misconfiguration in our pom.xml
> file. It correctly configure maven-compiler-plugin:
> {code:xml}
>
>
> org.apache.maven.plugins
> maven-compiler-plugin
> 3.1
>
> ${java.version}
> ${java.version}
>
>
> -Xlint:all
>
>
> {code}
> where ${java.version} is set to 1.7 in the properties, but it forgets to
> overwrite the following properties from apache-18.pom:
> {code:xml}
>
> 1.6
> 1.6
>
> {code}
> It seems like compiling from console using maven ignores those values, but
> they are confusing Intellij.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-7004) Switch to Travis Trusty image
[ https://issues.apache.org/jira/browse/FLINK-7004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067896#comment-16067896 ] ASF GitHub Bot commented on FLINK-7004: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/4207 > Switch to Travis Trusty image > - > > Key: FLINK-7004 > URL: https://issues.apache.org/jira/browse/FLINK-7004 > Project: Flink > Issue Type: Improvement > Components: Tests >Affects Versions: 1.2.0, 1.3.0, 1.4.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Critical > Fix For: 1.2.2, 1.4.0, 1.3.2 > > > As shown in this PR https://github.com/apache/flink/pull/4167 switching to > the Trusty image on Travis seems to stabilize the build times. > We should switch for 1.2, 1.3 and 1.4. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (FLINK-7004) Switch to Travis Trusty image
[ https://issues.apache.org/jira/browse/FLINK-7004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-7004. --- Resolution: Fixed 1.2: 3fcc959dd4d3c65436b44bb22848cf2565461de2 1.3: 001df19797e5155f0c6dff06eee796a6d83af19e 1.4: 958d3762de3c84264199c3c02035ff2f674973ce > Switch to Travis Trusty image > - > > Key: FLINK-7004 > URL: https://issues.apache.org/jira/browse/FLINK-7004 > Project: Flink > Issue Type: Improvement > Components: Tests >Affects Versions: 1.2.0, 1.3.0, 1.4.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Critical > Fix For: 1.2.2, 1.4.0, 1.3.2 > > > As shown in this PR https://github.com/apache/flink/pull/4167 switching to > the Trusty image on Travis seems to stabilize the build times. > We should switch for 1.2, 1.3 and 1.4. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink pull request #4146: [FLINK-6008][blob] collection of BlobServer improv...
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/4146#discussion_r124728498
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobUtils.java ---
@@ -227,7 +227,7 @@ static File getStorageLocation(File storageDir, JobID
jobID, String key) {
private static File getJobDirectory(File storageDir, JobID jobID) {
final File jobDirectory = new File(storageDir, JOB_DIR_PREFIX +
jobID.toString());
- if (!jobDirectory.exists() && !jobDirectory.mkdirs()) {
--- End diff --
Ok, please add comment or commit message info about this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4146: [FLINK-6008][blob] collection of BlobServer improv...
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/4146#discussion_r124730014
--- Diff:
flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobServerPutTest.java
---
@@ -59,6 +60,110 @@
private final Random rnd = new Random();
+
+ // --- concurrency tests for utility methods which could fail during
the put operation ---
+
+ /**
+* Checked thread that calls {@link
BlobServer#getStorageLocation(BlobKey)}
+*/
+ public static class ContentAddressableGetStorageLocation extends
CheckedThread {
+ private final BlobServer server;
+ private final BlobKey key;
+
+ public ContentAddressableGetStorageLocation(BlobServer server,
BlobKey key) {
+ this.server = server;
+ this.key = key;
+ }
+
+ @Override
+ public void go() throws Exception {
+ server.getStorageLocation(key);
--- End diff --
So maybe you could call in a loop multiple times `gets` and `deletes` (if
there is such operation) interleaved? Otherwise I don't see a real value of
those tests and I would prefer to drop them (so that we don't have to maintain
tests that do not check for anything).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4146: [FLINK-6008][blob] collection of BlobServer improv...
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/4146#discussion_r124727496
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobClient.java ---
@@ -593,9 +594,7 @@ private void sendPutHeader(OutputStream outputStream,
JobID jobID, String key) t
* the BLOB server or if the BLOB server cannot delete the file
*/
public void delete(BlobKey key) throws IOException {
- if (key == null) {
- throw new IllegalArgumentException("BLOB key must not
be null");
- }
+ checkArgument(key != null, "BLOB key must not be null.");
--- End diff --
Yes, but it would be cleaner and less surprising for future users. However
I have no strong feelings about it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4146: [FLINK-6008][blob] collection of BlobServer improv...
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/4146#discussion_r124727900
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobServerConnection.java
---
@@ -509,21 +509,7 @@ private void delete(InputStream inputStream,
OutputStream outputStream, byte[] b
if (type == CONTENT_ADDRESSABLE) {
BlobKey key =
BlobKey.readFromInputStream(inputStream);
- File blobFile =
blobServer.getStorageLocation(key);
-
- writeLock.lock();
-
- try {
- // we should make the local and remote
file deletion atomic, otherwise we might risk not
- // removing the remote file in case of
a concurrent put operation
- if (blobFile.exists() &&
!blobFile.delete()) {
- throw new IOException("Cannot
delete BLOB file " + blobFile.getAbsolutePath());
--- End diff --
Ok, sorry, I didn't find it first time I was looking for it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-6008) collection of BlobServer improvements
[
https://issues.apache.org/jira/browse/FLINK-6008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067917#comment-16067917
]
ASF GitHub Bot commented on FLINK-6008:
---
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/4146#discussion_r124730014
--- Diff:
flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobServerPutTest.java
---
@@ -59,6 +60,110 @@
private final Random rnd = new Random();
+
+ // --- concurrency tests for utility methods which could fail during
the put operation ---
+
+ /**
+* Checked thread that calls {@link
BlobServer#getStorageLocation(BlobKey)}
+*/
+ public static class ContentAddressableGetStorageLocation extends
CheckedThread {
+ private final BlobServer server;
+ private final BlobKey key;
+
+ public ContentAddressableGetStorageLocation(BlobServer server,
BlobKey key) {
+ this.server = server;
+ this.key = key;
+ }
+
+ @Override
+ public void go() throws Exception {
+ server.getStorageLocation(key);
--- End diff --
So maybe you could call in a loop multiple times `gets` and `deletes` (if
there is such operation) interleaved? Otherwise I don't see a real value of
those tests and I would prefer to drop them (so that we don't have to maintain
tests that do not check for anything).
> collection of BlobServer improvements
> -
>
> Key: FLINK-6008
> URL: https://issues.apache.org/jira/browse/FLINK-6008
> Project: Flink
> Issue Type: Improvement
> Components: Network
>Affects Versions: 1.3.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>
> The following things should be improved around the BlobServer/BlobCache:
> * update config uptions with non-deprecated ones, e.g.
> {{high-availability.cluster-id}} and {{high-availability.storageDir}}
> * promote {{BlobStore#deleteAll(JobID)}} to the {{BlobService}}
> * extend the {{BlobService}} to work with {{NAME_ADDRESSABLE}} blobs
> (prepares FLINK-4399]
> * remove {{NAME_ADDRESSABLE}} blobs after job/task termination
> * do not fail the {{BlobServer}} when a delete operation fails
> * code style, like using {{Preconditions.checkArgument}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6008) collection of BlobServer improvements
[
https://issues.apache.org/jira/browse/FLINK-6008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067919#comment-16067919
]
ASF GitHub Bot commented on FLINK-6008:
---
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/4146#discussion_r124727900
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobServerConnection.java
---
@@ -509,21 +509,7 @@ private void delete(InputStream inputStream,
OutputStream outputStream, byte[] b
if (type == CONTENT_ADDRESSABLE) {
BlobKey key =
BlobKey.readFromInputStream(inputStream);
- File blobFile =
blobServer.getStorageLocation(key);
-
- writeLock.lock();
-
- try {
- // we should make the local and remote
file deletion atomic, otherwise we might risk not
- // removing the remote file in case of
a concurrent put operation
- if (blobFile.exists() &&
!blobFile.delete()) {
- throw new IOException("Cannot
delete BLOB file " + blobFile.getAbsolutePath());
--- End diff --
Ok, sorry, I didn't find it first time I was looking for it.
> collection of BlobServer improvements
> -
>
> Key: FLINK-6008
> URL: https://issues.apache.org/jira/browse/FLINK-6008
> Project: Flink
> Issue Type: Improvement
> Components: Network
>Affects Versions: 1.3.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>
> The following things should be improved around the BlobServer/BlobCache:
> * update config uptions with non-deprecated ones, e.g.
> {{high-availability.cluster-id}} and {{high-availability.storageDir}}
> * promote {{BlobStore#deleteAll(JobID)}} to the {{BlobService}}
> * extend the {{BlobService}} to work with {{NAME_ADDRESSABLE}} blobs
> (prepares FLINK-4399]
> * remove {{NAME_ADDRESSABLE}} blobs after job/task termination
> * do not fail the {{BlobServer}} when a delete operation fails
> * code style, like using {{Preconditions.checkArgument}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6008) collection of BlobServer improvements
[
https://issues.apache.org/jira/browse/FLINK-6008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067920#comment-16067920
]
ASF GitHub Bot commented on FLINK-6008:
---
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/4146#discussion_r124728498
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobUtils.java ---
@@ -227,7 +227,7 @@ static File getStorageLocation(File storageDir, JobID
jobID, String key) {
private static File getJobDirectory(File storageDir, JobID jobID) {
final File jobDirectory = new File(storageDir, JOB_DIR_PREFIX +
jobID.toString());
- if (!jobDirectory.exists() && !jobDirectory.mkdirs()) {
--- End diff --
Ok, please add comment or commit message info about this.
> collection of BlobServer improvements
> -
>
> Key: FLINK-6008
> URL: https://issues.apache.org/jira/browse/FLINK-6008
> Project: Flink
> Issue Type: Improvement
> Components: Network
>Affects Versions: 1.3.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>
> The following things should be improved around the BlobServer/BlobCache:
> * update config uptions with non-deprecated ones, e.g.
> {{high-availability.cluster-id}} and {{high-availability.storageDir}}
> * promote {{BlobStore#deleteAll(JobID)}} to the {{BlobService}}
> * extend the {{BlobService}} to work with {{NAME_ADDRESSABLE}} blobs
> (prepares FLINK-4399]
> * remove {{NAME_ADDRESSABLE}} blobs after job/task termination
> * do not fail the {{BlobServer}} when a delete operation fails
> * code style, like using {{Preconditions.checkArgument}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6008) collection of BlobServer improvements
[
https://issues.apache.org/jira/browse/FLINK-6008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067918#comment-16067918
]
ASF GitHub Bot commented on FLINK-6008:
---
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/4146#discussion_r124727496
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobClient.java ---
@@ -593,9 +594,7 @@ private void sendPutHeader(OutputStream outputStream,
JobID jobID, String key) t
* the BLOB server or if the BLOB server cannot delete the file
*/
public void delete(BlobKey key) throws IOException {
- if (key == null) {
- throw new IllegalArgumentException("BLOB key must not
be null");
- }
+ checkArgument(key != null, "BLOB key must not be null.");
--- End diff --
Yes, but it would be cleaner and less surprising for future users. However
I have no strong feelings about it.
> collection of BlobServer improvements
> -
>
> Key: FLINK-6008
> URL: https://issues.apache.org/jira/browse/FLINK-6008
> Project: Flink
> Issue Type: Improvement
> Components: Network
>Affects Versions: 1.3.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>
> The following things should be improved around the BlobServer/BlobCache:
> * update config uptions with non-deprecated ones, e.g.
> {{high-availability.cluster-id}} and {{high-availability.storageDir}}
> * promote {{BlobStore#deleteAll(JobID)}} to the {{BlobService}}
> * extend the {{BlobService}} to work with {{NAME_ADDRESSABLE}} blobs
> (prepares FLINK-4399]
> * remove {{NAME_ADDRESSABLE}} blobs after job/task termination
> * do not fail the {{BlobServer}} when a delete operation fails
> * code style, like using {{Preconditions.checkArgument}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #4220: [FLINK-6665] Pass a ScheduledExecutorService to th...
GitHub user zjureel opened a pull request:
https://github.com/apache/flink/pull/4220
[FLINK-6665] Pass a ScheduledExecutorService to the RestartStrategy
⦠avoid any sleeps.
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zjureel/flink FLINK-6665
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4220.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4220
commit 8de578ed761d79969c6ceaf5f696dc926396f137
Author: zjureel
Date: 2017-06-29T07:10:34Z
[FLINK-6665] schedule the restart call by ScheduledExecutorService to avoid
any sleeps.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-6584) Support multiple consecutive windows in SQL
[
https://issues.apache.org/jira/browse/FLINK-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067945#comment-16067945
]
ASF GitHub Bot commented on FLINK-6584:
---
Github user wuchong commented on a diff in the pull request:
https://github.com/apache/flink/pull/4199#discussion_r124733642
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/plan/rules/common/WindowStartEndPropertiesRule.scala
---
@@ -54,34 +56,75 @@ class WindowStartEndPropertiesRule
val project = call.rel(0).asInstanceOf[LogicalProject]
val innerProject = call.rel(1).asInstanceOf[LogicalProject]
val agg = call.rel(2).asInstanceOf[LogicalWindowAggregate]
+val window = agg.getWindow
-// Retrieve window start and end properties
+val isRowtime = isRowtimeAttribute(window.timeAttribute)
+val isProctime = isProctimeAttribute(window.timeAttribute)
+
+val startEndProperties = Seq(
+ NamedWindowProperty("w$start", WindowStart(window.aliasAttribute)),
+ NamedWindowProperty("w$end", WindowEnd(window.aliasAttribute)))
+
+// allow rowtime/proctime for rowtime windows and proctime for
proctime windows
+val timeProperties = if (isRowtime) {
+ Seq(
+NamedWindowProperty("w$rowtime",
RowtimeAttribute(window.aliasAttribute)),
--- End diff --
We have so many `w$...` here, can we create the String constants and
reference them ?
> Support multiple consecutive windows in SQL
> ---
>
> Key: FLINK-6584
> URL: https://issues.apache.org/jira/browse/FLINK-6584
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> Right now, the Table API supports multiple consecutive windows as follows:
> {code}
> val table = stream.toTable(tEnv, 'rowtime.rowtime, 'int, 'double, 'float,
> 'bigdec, 'string)
> val t = table
> .window(Tumble over 2.millis on 'rowtime as 'w)
> .groupBy('w)
> .select('w.rowtime as 'rowtime, 'int.count as 'int)
> .window(Tumble over 4.millis on 'rowtime as 'w2)
> .groupBy('w2)
> .select('w2.rowtime, 'w2.end, 'int.count)
> {code}
> Similar behavior should be supported by the SQL API as well. We need to
> introduce a new auxiliary group function, but this should happen in sync with
> Apache Calcite.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6665) Pass a ScheduledExecutorService to the RestartStrategy
[
https://issues.apache.org/jira/browse/FLINK-6665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067929#comment-16067929
]
ASF GitHub Bot commented on FLINK-6665:
---
GitHub user zjureel opened a pull request:
https://github.com/apache/flink/pull/4220
[FLINK-6665] Pass a ScheduledExecutorService to the RestartStrategy
… avoid any sleeps.
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zjureel/flink FLINK-6665
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4220.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4220
commit 8de578ed761d79969c6ceaf5f696dc926396f137
Author: zjureel
Date: 2017-06-29T07:10:34Z
[FLINK-6665] schedule the restart call by ScheduledExecutorService to avoid
any sleeps.
> Pass a ScheduledExecutorService to the RestartStrategy
> --
>
> Key: FLINK-6665
> URL: https://issues.apache.org/jira/browse/FLINK-6665
> Project: Flink
> Issue Type: Sub-task
> Components: Distributed Coordination
>Reporter: Stephan Ewen
>Assignee: Fang Yong
> Fix For: 1.4.0
>
>
> Currently, the {{RestartStrategy}} is called when the {{ExecutionGraph}}
> should be restarted.
> To facilitate delays before restarting, the strategy simply sleeps, blocking
> the thread that runs the ExecutionGraph's recovery method.
> I suggest to pass {{ScheduledExecutorService}}) to the {{RestartStrategy}}
> and let it schedule the restart call that way, avoiding any sleeps.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #4199: [FLINK-6584] [table] Support multiple consecutive ...
Github user wuchong commented on a diff in the pull request:
https://github.com/apache/flink/pull/4199#discussion_r124733642
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/plan/rules/common/WindowStartEndPropertiesRule.scala
---
@@ -54,34 +56,75 @@ class WindowStartEndPropertiesRule
val project = call.rel(0).asInstanceOf[LogicalProject]
val innerProject = call.rel(1).asInstanceOf[LogicalProject]
val agg = call.rel(2).asInstanceOf[LogicalWindowAggregate]
+val window = agg.getWindow
-// Retrieve window start and end properties
+val isRowtime = isRowtimeAttribute(window.timeAttribute)
+val isProctime = isProctimeAttribute(window.timeAttribute)
+
+val startEndProperties = Seq(
+ NamedWindowProperty("w$start", WindowStart(window.aliasAttribute)),
+ NamedWindowProperty("w$end", WindowEnd(window.aliasAttribute)))
+
+// allow rowtime/proctime for rowtime windows and proctime for
proctime windows
+val timeProperties = if (isRowtime) {
+ Seq(
+NamedWindowProperty("w$rowtime",
RowtimeAttribute(window.aliasAttribute)),
--- End diff --
We have so many `w$...` here, can we create the String constants and
reference them ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4199: [FLINK-6584] [table] Support multiple consecutive ...
Github user wuchong commented on a diff in the pull request:
https://github.com/apache/flink/pull/4199#discussion_r124725025
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/plan/rules/common/WindowStartEndPropertiesRule.scala
---
@@ -54,34 +56,75 @@ class WindowStartEndPropertiesRule
val project = call.rel(0).asInstanceOf[LogicalProject]
val innerProject = call.rel(1).asInstanceOf[LogicalProject]
val agg = call.rel(2).asInstanceOf[LogicalWindowAggregate]
+val window = agg.getWindow
-// Retrieve window start and end properties
+val isRowtime = isRowtimeAttribute(window.timeAttribute)
+val isProctime = isProctimeAttribute(window.timeAttribute)
+
+val startEndProperties = Seq(
+ NamedWindowProperty("w$start", WindowStart(window.aliasAttribute)),
+ NamedWindowProperty("w$end", WindowEnd(window.aliasAttribute)))
+
+// allow rowtime/proctime for rowtime windows and proctime for
proctime windows
+val timeProperties = if (isRowtime) {
+ Seq(
+NamedWindowProperty("w$rowtime",
RowtimeAttribute(window.aliasAttribute)),
+NamedWindowProperty("w$proctime",
ProctimeAttribute(window.aliasAttribute)))
+} else if (isProctime) {
+ Seq(NamedWindowProperty("w$proctime",
ProctimeAttribute(window.aliasAttribute)))
+} else {
+ Seq()
+}
+
+val properties = startEndProperties ++ timeProperties
+
+// retrieve window start and end properties
val transformed = call.builder()
val rexBuilder = transformed.getRexBuilder
transformed.push(LogicalWindowAggregate.create(
- agg.getWindow,
- Seq(
-NamedWindowProperty("w$start",
WindowStart(agg.getWindow.aliasAttribute)),
-NamedWindowProperty("w$end",
WindowEnd(agg.getWindow.aliasAttribute))
- ), agg)
+ window,
+ properties,
+ agg)
)
// forward window start and end properties
transformed.project(
- innerProject.getProjects ++ Seq(transformed.field("w$start"),
transformed.field("w$end")))
+ innerProject.getProjects ++ properties.map(np =>
transformed.field(np.name)))
def replaceGroupAuxiliaries(node: RexNode): RexNode = {
node match {
case c: RexCall if WindowStartEndPropertiesRule.isWindowStart(c) =>
// replace expression by access to window start
rexBuilder.makeCast(c.getType, transformed.field("w$start"),
false)
+
case c: RexCall if WindowStartEndPropertiesRule.isWindowEnd(c) =>
// replace expression by access to window end
rexBuilder.makeCast(c.getType, transformed.field("w$end"), false)
+
+case c: RexCall if WindowStartEndPropertiesRule.isWindowRowtime(c)
=>
+ if (isProctime) {
+throw ValidationException("A proctime window cannot provide a
rowtime attribute.")
+ } else if (isRowtime) {
+// replace expression by access to window rowtime
+transformed.field("w$rowtime")
+ } else {
+throw TableException("Accessing the rowtime attribute of a
window is not yet " +
+ "supported in a batch environment.")
+ }
+
+case c: RexCall if
WindowStartEndPropertiesRule.isWindowProctime(c) =>
+ if (isProctime) {
--- End diff --
add a `isRowtime` condition branch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-6584) Support multiple consecutive windows in SQL
[
https://issues.apache.org/jira/browse/FLINK-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067946#comment-16067946
]
ASF GitHub Bot commented on FLINK-6584:
---
Github user wuchong commented on a diff in the pull request:
https://github.com/apache/flink/pull/4199#discussion_r124725025
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/plan/rules/common/WindowStartEndPropertiesRule.scala
---
@@ -54,34 +56,75 @@ class WindowStartEndPropertiesRule
val project = call.rel(0).asInstanceOf[LogicalProject]
val innerProject = call.rel(1).asInstanceOf[LogicalProject]
val agg = call.rel(2).asInstanceOf[LogicalWindowAggregate]
+val window = agg.getWindow
-// Retrieve window start and end properties
+val isRowtime = isRowtimeAttribute(window.timeAttribute)
+val isProctime = isProctimeAttribute(window.timeAttribute)
+
+val startEndProperties = Seq(
+ NamedWindowProperty("w$start", WindowStart(window.aliasAttribute)),
+ NamedWindowProperty("w$end", WindowEnd(window.aliasAttribute)))
+
+// allow rowtime/proctime for rowtime windows and proctime for
proctime windows
+val timeProperties = if (isRowtime) {
+ Seq(
+NamedWindowProperty("w$rowtime",
RowtimeAttribute(window.aliasAttribute)),
+NamedWindowProperty("w$proctime",
ProctimeAttribute(window.aliasAttribute)))
+} else if (isProctime) {
+ Seq(NamedWindowProperty("w$proctime",
ProctimeAttribute(window.aliasAttribute)))
+} else {
+ Seq()
+}
+
+val properties = startEndProperties ++ timeProperties
+
+// retrieve window start and end properties
val transformed = call.builder()
val rexBuilder = transformed.getRexBuilder
transformed.push(LogicalWindowAggregate.create(
- agg.getWindow,
- Seq(
-NamedWindowProperty("w$start",
WindowStart(agg.getWindow.aliasAttribute)),
-NamedWindowProperty("w$end",
WindowEnd(agg.getWindow.aliasAttribute))
- ), agg)
+ window,
+ properties,
+ agg)
)
// forward window start and end properties
transformed.project(
- innerProject.getProjects ++ Seq(transformed.field("w$start"),
transformed.field("w$end")))
+ innerProject.getProjects ++ properties.map(np =>
transformed.field(np.name)))
def replaceGroupAuxiliaries(node: RexNode): RexNode = {
node match {
case c: RexCall if WindowStartEndPropertiesRule.isWindowStart(c) =>
// replace expression by access to window start
rexBuilder.makeCast(c.getType, transformed.field("w$start"),
false)
+
case c: RexCall if WindowStartEndPropertiesRule.isWindowEnd(c) =>
// replace expression by access to window end
rexBuilder.makeCast(c.getType, transformed.field("w$end"), false)
+
+case c: RexCall if WindowStartEndPropertiesRule.isWindowRowtime(c)
=>
+ if (isProctime) {
+throw ValidationException("A proctime window cannot provide a
rowtime attribute.")
+ } else if (isRowtime) {
+// replace expression by access to window rowtime
+transformed.field("w$rowtime")
+ } else {
+throw TableException("Accessing the rowtime attribute of a
window is not yet " +
+ "supported in a batch environment.")
+ }
+
+case c: RexCall if
WindowStartEndPropertiesRule.isWindowProctime(c) =>
+ if (isProctime) {
--- End diff --
add a `isRowtime` condition branch?
> Support multiple consecutive windows in SQL
> ---
>
> Key: FLINK-6584
> URL: https://issues.apache.org/jira/browse/FLINK-6584
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> Right now, the Table API supports multiple consecutive windows as follows:
> {code}
> val table = stream.toTable(tEnv, 'rowtime.rowtime, 'int, 'double, 'float,
> 'bigdec, 'string)
> val t = table
> .window(Tumble over 2.millis on 'rowtime as 'w)
> .groupBy('w)
> .select('w.rowtime as 'rowtime, 'int.count as 'int)
> .window(Tumble over 4.millis on 'rowtime as 'w2)
> .groupBy('w2)
> .select('w2.rowtime, 'w2.end, 'int.count)
> {code}
> Similar behavior should be supported by the SQL API as well. We need to
> introduce a new auxiliary group function, but this should happen in sync with
> Apache Calcite.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6991) Inaccessible link under Gelly document
[
https://issues.apache.org/jira/browse/FLINK-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067935#comment-16067935
]
njzhuyuqi commented on FLINK-6991:
--
Hi, [~mingleizhang] I found a solution for how to fix this issue.
This web page is from the html page of the flink-web, the source code address
is
[https://github.com/apache/flink-web/blob/asf-site/content/news/2015/08/24/introducing-flink-gelly.html].
The link of gelly documentation was written with the address
[https://ci.apache.org/projects/flink/flink-docs-master/libs/gelly_guide.html],
which referred to the newest flink docs. After flink-docs-release-0.9, the
gelly guide had been moved, redirecting to
[https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/batch/libs/gelly.html].
Then after flink-docs-release-1.1, the gelly guide had been moved to another
new address
[https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/libs/gelly/].
In order to fix this bug, we would change the link of gelly documentation to
the page address
[https://ci.apache.org/projects/flink/flink-docs-release-0.9/libs/gelly_guide.html],
which exactly match the version of flink-release-0.9 at that time.
> Inaccessible link under Gelly document
> --
>
> Key: FLINK-6991
> URL: https://issues.apache.org/jira/browse/FLINK-6991
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Reporter: mingleizhang
>Assignee: njzhuyuqi
>
> When I visited the web
> http://flink.apache.org/news/2015/08/24/introducing-flink-gelly.html#top,
> then at the bottom of the page there is a Gelly Documentation link. It
> belongs to a non-existent resources. It gives me the following style stuff.
> {noformat}
> No Such Resource
> File not found.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6008) collection of BlobServer improvements
[
https://issues.apache.org/jira/browse/FLINK-6008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067956#comment-16067956
]
ASF GitHub Bot commented on FLINK-6008:
---
Github user NicoK commented on a diff in the pull request:
https://github.com/apache/flink/pull/4146#discussion_r124734798
--- Diff:
flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobServerPutTest.java
---
@@ -59,6 +60,110 @@
private final Random rnd = new Random();
+
+ // --- concurrency tests for utility methods which could fail during
the put operation ---
+
+ /**
+* Checked thread that calls {@link
BlobServer#getStorageLocation(BlobKey)}
+*/
+ public static class ContentAddressableGetStorageLocation extends
CheckedThread {
+ private final BlobServer server;
+ private final BlobKey key;
+
+ public ContentAddressableGetStorageLocation(BlobServer server,
BlobKey key) {
+ this.server = server;
+ this.key = key;
+ }
+
+ @Override
+ public void go() throws Exception {
+ server.getStorageLocation(key);
--- End diff --
Unfortunately, concurrency with `delete` operations does not work either if
not guarded - the directory may not exist anymore between
`jobDirectory.mkdirs()` and `jobDirectory.exists()`. I was able to reproduce
the error with the existing test though - if you want to try it, just change
the order of these two commands back - the test will not hit every time, but
some times.
> collection of BlobServer improvements
> -
>
> Key: FLINK-6008
> URL: https://issues.apache.org/jira/browse/FLINK-6008
> Project: Flink
> Issue Type: Improvement
> Components: Network
>Affects Versions: 1.3.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>
> The following things should be improved around the BlobServer/BlobCache:
> * update config uptions with non-deprecated ones, e.g.
> {{high-availability.cluster-id}} and {{high-availability.storageDir}}
> * promote {{BlobStore#deleteAll(JobID)}} to the {{BlobService}}
> * extend the {{BlobService}} to work with {{NAME_ADDRESSABLE}} blobs
> (prepares FLINK-4399]
> * remove {{NAME_ADDRESSABLE}} blobs after job/task termination
> * do not fail the {{BlobServer}} when a delete operation fails
> * code style, like using {{Preconditions.checkArgument}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #4146: [FLINK-6008][blob] collection of BlobServer improv...
Github user NicoK commented on a diff in the pull request:
https://github.com/apache/flink/pull/4146#discussion_r124734798
--- Diff:
flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobServerPutTest.java
---
@@ -59,6 +60,110 @@
private final Random rnd = new Random();
+
+ // --- concurrency tests for utility methods which could fail during
the put operation ---
+
+ /**
+* Checked thread that calls {@link
BlobServer#getStorageLocation(BlobKey)}
+*/
+ public static class ContentAddressableGetStorageLocation extends
CheckedThread {
+ private final BlobServer server;
+ private final BlobKey key;
+
+ public ContentAddressableGetStorageLocation(BlobServer server,
BlobKey key) {
+ this.server = server;
+ this.key = key;
+ }
+
+ @Override
+ public void go() throws Exception {
+ server.getStorageLocation(key);
--- End diff --
Unfortunately, concurrency with `delete` operations does not work either if
not guarded - the directory may not exist anymore between
`jobDirectory.mkdirs()` and `jobDirectory.exists()`. I was able to reproduce
the error with the existing test though - if you want to try it, just change
the order of these two commands back - the test will not hit every time, but
some times.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4195: [FLINK-7008] [cep] Update NFA state only when the NFA cha...
Github user dawidwys commented on the issue: https://github.com/apache/flink/pull/4195 Thanks for the work @dianfu! It looks good now. Let's just wait for travis to give green on it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-7008) Update NFA state only when the NFA changes.
[
https://issues.apache.org/jira/browse/FLINK-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067960#comment-16067960
]
ASF GitHub Bot commented on FLINK-7008:
---
Github user dawidwys commented on the issue:
https://github.com/apache/flink/pull/4195
Thanks for the work @dianfu! It looks good now. Let's just wait for travis
to give green on it.
> Update NFA state only when the NFA changes.
> ---
>
> Key: FLINK-7008
> URL: https://issues.apache.org/jira/browse/FLINK-7008
> Project: Flink
> Issue Type: Improvement
> Components: CEP
>Affects Versions: 1.3.1
>Reporter: Kostas Kloudas
>Assignee: Dian Fu
> Fix For: 1.4.0
>
>
> Currently in the {{AbstractKeyedCEPPatternOperator.updateNFA()}} method we
> update the NFA state every time the NFA is touched. This leads to redundant
> puts/gets to the state when there are no changes to the NFA itself.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6991) Inaccessible link under Gelly document
[
https://issues.apache.org/jira/browse/FLINK-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067966#comment-16067966
]
mingleizhang commented on FLINK-6991:
-
Thanks [~njzhuyuqi] for keep watching this. I would suggest redirect the link
to the newest release though. Could you take a more look for this link
[https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/gelly/] ?
:)
> Inaccessible link under Gelly document
> --
>
> Key: FLINK-6991
> URL: https://issues.apache.org/jira/browse/FLINK-6991
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Reporter: mingleizhang
>Assignee: njzhuyuqi
>
> When I visited the web
> http://flink.apache.org/news/2015/08/24/introducing-flink-gelly.html#top,
> then at the bottom of the page there is a Gelly Documentation link. It
> belongs to a non-existent resources. It gives me the following style stuff.
> {noformat}
> No Such Resource
> File not found.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4198: [Temporary] Running Mesos ResourceManager on Travis
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4198 @tillrohrmann Can you close this PR? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4172: [FLINK-6983] [cep] Do not serialize States with NFA
Github user wuchong commented on the issue: https://github.com/apache/flink/pull/4172 Yes @twalthr , you are right, the cause is different classloader. The compiled class is under Janino's custom ClassLoader. Using the current thread's classloader or user code classloader can't load the class. BUT, is it possible to pass the Janino's classloader into state deserialization ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6983) Do not serialize States with NFA
[ https://issues.apache.org/jira/browse/FLINK-6983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067978#comment-16067978 ] ASF GitHub Bot commented on FLINK-6983: --- Github user wuchong commented on the issue: https://github.com/apache/flink/pull/4172 Yes @twalthr , you are right, the cause is different classloader. The compiled class is under Janino's custom ClassLoader. Using the current thread's classloader or user code classloader can't load the class. BUT, is it possible to pass the Janino's classloader into state deserialization ? > Do not serialize States with NFA > > > Key: FLINK-6983 > URL: https://issues.apache.org/jira/browse/FLINK-6983 > Project: Flink > Issue Type: Improvement > Components: CEP >Reporter: Dawid Wysakowicz >Assignee: Dian Fu > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #4187: [FLINK-6998][Kafka Connector] Add kafka offset commit met...
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/4187 I agree. "commits-succeeded" and "commits-failed" seems good! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6998) Kafka connector needs to expose metrics for failed/successful offset commits in the Kafka Consumer callback
[ https://issues.apache.org/jira/browse/FLINK-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067983#comment-16067983 ] ASF GitHub Bot commented on FLINK-6998: --- Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/4187 I agree. "commits-succeeded" and "commits-failed" seems good! > Kafka connector needs to expose metrics for failed/successful offset commits > in the Kafka Consumer callback > --- > > Key: FLINK-6998 > URL: https://issues.apache.org/jira/browse/FLINK-6998 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector >Reporter: Zhenzhong Xu >Assignee: Zhenzhong Xu > > Propose to add "kafkaCommitsSucceeded" and "kafkaCommitsFailed" metrics in > KafkaConsumerThread class. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #4187: [FLINK-6998][Kafka Connector] Add kafka offset commit met...
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/4187 > I'll not include the other proposed metric in this PR just for the sake of simplicity. I also have some opinions on "offset lag" metric, I think this particular metric is more useful when some external entity perform the monitoring (difference of committed offset vs log head), especially in failure situations. Yes, lets keep that apart from this PR. There is also a JIRA for exactly this feature: https://issues.apache.org/jira/browse/FLINK-6109 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6998) Kafka connector needs to expose metrics for failed/successful offset commits in the Kafka Consumer callback
[ https://issues.apache.org/jira/browse/FLINK-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067986#comment-16067986 ] ASF GitHub Bot commented on FLINK-6998: --- Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/4187 > I'll not include the other proposed metric in this PR just for the sake of simplicity. I also have some opinions on "offset lag" metric, I think this particular metric is more useful when some external entity perform the monitoring (difference of committed offset vs log head), especially in failure situations. Yes, lets keep that apart from this PR. There is also a JIRA for exactly this feature: https://issues.apache.org/jira/browse/FLINK-6109 > Kafka connector needs to expose metrics for failed/successful offset commits > in the Kafka Consumer callback > --- > > Key: FLINK-6998 > URL: https://issues.apache.org/jira/browse/FLINK-6998 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector >Reporter: Zhenzhong Xu >Assignee: Zhenzhong Xu > > Propose to add "kafkaCommitsSucceeded" and "kafkaCommitsFailed" metrics in > KafkaConsumerThread class. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-6991) Inaccessible link under Gelly document
[
https://issues.apache.org/jira/browse/FLINK-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067994#comment-16067994
]
njzhuyuqi commented on FLINK-6991:
--
hi,[~mingleizhang] i would suggest to pick the 0.9 release gelly doc for this
link, because the news was written at the time of flink release-0.9 version, it
matched the flink release-0.9 version at that time.
> Inaccessible link under Gelly document
> --
>
> Key: FLINK-6991
> URL: https://issues.apache.org/jira/browse/FLINK-6991
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Reporter: mingleizhang
>Assignee: njzhuyuqi
>
> When I visited the web
> http://flink.apache.org/news/2015/08/24/introducing-flink-gelly.html#top,
> then at the bottom of the page there is a Gelly Documentation link. It
> belongs to a non-existent resources. It gives me the following style stuff.
> {noformat}
> No Such Resource
> File not found.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6991) Inaccessible link under Gelly document
[
https://issues.apache.org/jira/browse/FLINK-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067998#comment-16067998
]
mingleizhang commented on FLINK-6991:
-
Maybe Greg Hogan could share some light on it, I guess.
> Inaccessible link under Gelly document
> --
>
> Key: FLINK-6991
> URL: https://issues.apache.org/jira/browse/FLINK-6991
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Reporter: mingleizhang
>Assignee: njzhuyuqi
>
> When I visited the web
> http://flink.apache.org/news/2015/08/24/introducing-flink-gelly.html#top,
> then at the bottom of the page there is a Gelly Documentation link. It
> belongs to a non-existent resources. It gives me the following style stuff.
> {noformat}
> No Such Resource
> File not found.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4153: [FLINK-6927] [cep] Support pattern group in CEP
Github user dawidwys commented on the issue: https://github.com/apache/flink/pull/4153 @dianfu sorry I have not reviewed it yet, but I do think this feature would benefit from reworking of the Pattern API I propose in [FLINK-3414](https://issues.apache.org/jira/browse/FLINK-3414). Instead of checking for head/tail of group pattern and caching them, we could more or less use current code for plain sequence creation that could return begin/end states. Then the code for joining groups would be much easier, cause it would operate on already translated sequences. Also the new API would make [FLINK-4641]](https://issues.apache.org/jira/browse/FLINK-4641) much easier I think. As it would require API rework I would really like to hear @kl0u opinion. If we agree though not to change the API I will go straight to reviewing this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6927) Support pattern group in CEP
[
https://issues.apache.org/jira/browse/FLINK-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068013#comment-16068013
]
ASF GitHub Bot commented on FLINK-6927:
---
Github user dawidwys commented on the issue:
https://github.com/apache/flink/pull/4153
@dianfu sorry I have not reviewed it yet, but I do think this feature would
benefit from reworking of the Pattern API I propose in
[FLINK-3414](https://issues.apache.org/jira/browse/FLINK-3414).
Instead of checking for head/tail of group pattern and caching them, we
could more or less use current code for plain sequence creation that could
return begin/end states. Then the code for joining groups would be much easier,
cause it would operate on already translated sequences.
Also the new API would make
[FLINK-4641]](https://issues.apache.org/jira/browse/FLINK-4641)
much easier I think.
As it would require API rework I would really like to hear @kl0u opinion.
If we agree though not to change the API I will go straight to reviewing this
PR.
> Support pattern group in CEP
>
>
> Key: FLINK-6927
> URL: https://issues.apache.org/jira/browse/FLINK-6927
> Project: Flink
> Issue Type: Sub-task
> Components: CEP
>Reporter: Dian Fu
>Assignee: Dian Fu
>
> We should add support for pattern group. This would enrich the set of
> supported patterns. For example, users can write patterns like this with this
> feature available:
> {code}
> A --> (B --> C.times(3)).optional() --> D
> {code}
> or
> {code}
> A --> (B --> C).times(3) --> D
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4172: [FLINK-6983] [cep] Do not serialize States with NFA
Github user kl0u commented on the issue: https://github.com/apache/flink/pull/4172 Hi @wuchong . I am not an expert in Janino and how it works but I do not think you need Janino's classloader at any point. Using the `open()` of the `CEPOperator` you just need to compile the code of the `IterativeCondition` whenever we have a `IterativeConditionWrapper`. In the code snippet you sent before, line 30 should change to take the user classloader from the runtime context. If I remember correctly it is `getRuntimeContext().getUserClassLoader()`. This will allow Janino to do its magic using the user classloader. After compiling the class, you can set the `newCondition` field of the `StateTransitions` to an object of the compiled class. After this, whenever you have a checkpoint or whenever you serialize/deserialize the condition, given that this will be a regular IterativeCondition, Flink will know how to do it. I am also including @twalthr and @fhueske on this to comment, as they have more experience with this. In addition, if you have a prototype that does all this but fails at some step with an error, you can send a link and we can try to check it out. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6983) Do not serialize States with NFA
[ https://issues.apache.org/jira/browse/FLINK-6983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068044#comment-16068044 ] ASF GitHub Bot commented on FLINK-6983: --- Github user kl0u commented on the issue: https://github.com/apache/flink/pull/4172 Hi @wuchong . I am not an expert in Janino and how it works but I do not think you need Janino's classloader at any point. Using the `open()` of the `CEPOperator` you just need to compile the code of the `IterativeCondition` whenever we have a `IterativeConditionWrapper`. In the code snippet you sent before, line 30 should change to take the user classloader from the runtime context. If I remember correctly it is `getRuntimeContext().getUserClassLoader()`. This will allow Janino to do its magic using the user classloader. After compiling the class, you can set the `newCondition` field of the `StateTransitions` to an object of the compiled class. After this, whenever you have a checkpoint or whenever you serialize/deserialize the condition, given that this will be a regular IterativeCondition, Flink will know how to do it. I am also including @twalthr and @fhueske on this to comment, as they have more experience with this. In addition, if you have a prototype that does all this but fails at some step with an error, you can send a link and we can try to check it out. > Do not serialize States with NFA > > > Key: FLINK-6983 > URL: https://issues.apache.org/jira/browse/FLINK-6983 > Project: Flink > Issue Type: Improvement > Components: CEP >Reporter: Dawid Wysakowicz >Assignee: Dian Fu > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7036) Type cast in ExpressionReducer's reduce broke CodeGen's compile stage
Aegeaner created FLINK-7036:
---
Summary: Type cast in ExpressionReducer's reduce broke CodeGen's
compile stage
Key: FLINK-7036
URL: https://issues.apache.org/jira/browse/FLINK-7036
Project: Flink
Issue Type: Bug
Reporter: Aegeaner
I am implementing a STR_TO_DATE scalar SQL function to flink, and found return
type casted from java.sql.Date to Integer in Flink’s ExpressionReducer：
[https://github.com/apache/flink/blob/master/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/codegen/ExpressionReducer.scala#L56]
{code:java}
// we need to cast here for RexBuilder.makeLiteral
case (SqlTypeName.DATE, e) =>
Some(
rexBuilder.makeCast(typeFactory.createTypeFromTypeInfo(BasicTypeInfo.INT_TYPE_INFO),
e)
)
{code}
so str_to_date('01,5,2013','%d,%m,%Y')" must return an Integer,
which conflicted with my implementation.
Also, I did see the wrong result type failed the CodeGen compile stage, it
throws out exception stack as follow, even before the ExpressionReducer
actually restores the origin return
type:
[https://github.com/apache/flink/blob/master/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/codegen/ExpressionReducer.scala#L96]
{code:java}
org.apache.flink.api.common.InvalidProgramException: Table program cannot be
compiled. This is a bug. Please file an issue.
at
org.apache.flink.table.codegen.Compiler$class.compile(Compiler.scala:36)
at
org.apache.flink.table.codegen.ExpressionReducer.compile(ExpressionReducer.scala:38)
at
org.apache.flink.table.codegen.ExpressionReducer.reduce(ExpressionReducer.scala:96)
at
org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressionsInternal(ReduceExpressionsRule.java:549)
at
org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressions(ReduceExpressionsRule.java:470)
at
org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressions(ReduceExpressionsRule.java:447)
at
org.apache.calcite.rel.rules.ReduceExpressionsRule$ProjectReduceExpressionsRule.onMatch(ReduceExpressionsRule.java:270)
at
org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:317)
at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:506)
at
org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:385)
at
org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
at
org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:125)
at
org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:210)
at
org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:197)
at
org.apache.flink.table.expressions.utils.ExpressionTestBase.addSqlTestExpr(ExpressionTestBase.scala:194)
at
org.apache.flink.table.expressions.utils.ExpressionTestBase.testSqlApi(ExpressionTestBase.scala:277)
at
org.apache.flink.table.expressions.ScalarFunctionsTest.testStrToDate(ScalarFunctionsTest.scala:1516)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.
[GitHub] flink issue #4187: [FLINK-6998][Kafka Connector] Add kafka offset commit met...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4187 Please follow the camel-case pattern that we use for other metrics. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6998) Kafka connector needs to expose metrics for failed/successful offset commits in the Kafka Consumer callback
[ https://issues.apache.org/jira/browse/FLINK-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068059#comment-16068059 ] ASF GitHub Bot commented on FLINK-6998: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/4187 Please follow the camel-case pattern that we use for other metrics. > Kafka connector needs to expose metrics for failed/successful offset commits > in the Kafka Consumer callback > --- > > Key: FLINK-6998 > URL: https://issues.apache.org/jira/browse/FLINK-6998 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector >Reporter: Zhenzhong Xu >Assignee: Zhenzhong Xu > > Propose to add "kafkaCommitsSucceeded" and "kafkaCommitsFailed" metrics in > KafkaConsumerThread class. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink pull request #4198: [Temporary] Running Mesos ResourceManager on Travi...
Github user tillrohrmann closed the pull request at: https://github.com/apache/flink/pull/4198 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Assigned] (FLINK-7037) Remove scala suffix from flink-examples module
[
https://issues.apache.org/jira/browse/FLINK-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler reassigned FLINK-7037:
---
Assignee: Chesnay Schepler
> Remove scala suffix from flink-examples module
> --
>
> Key: FLINK-7037
> URL: https://issues.apache.org/jira/browse/FLINK-7037
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>
> Exceptions such as this
> {code}
> [ERROR] Failed to execute goal on project flink-tests_2.11: Could not resolve
> dependencies for project org.apache.flink:flink-tests_2.11:jar:1.4-SNAPSHOT:
> Failed to collect dependencies at
> org.apache.flink:flink-examples-batch_2.11:jar:1.4-SNAPSHOT: Failed to read
> artifact descriptor for
> org.apache.flink:flink-examples-batch_2.11:jar:1.4-SNAPSHOT: Failure to find
> org.apache.flink:flink-examples_${scala.binary.version}:pom:1.4-SNAPSHOT in
> https://oss.sonatype.org/content/repositories/snapshots/ was cached in the
> local repository, resolution will not be reattempted until the update
> interval of sonatype-snapshots has elapsed or updates are forced -> [Help 1]
> {code}
> are caused by modules that have a parent module which have a scala suffix.
> The scala version is not properly resolved for parents. In retrospect this is
> rather obvious, as the scala version is defined in the parent; in order to
> resolve the scala version one would ahve to know who the parent is, but you
> need the scala version for that.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7037) Remove scala suffix from flink-examples module
Chesnay Schepler created FLINK-7037:
---
Summary: Remove scala suffix from flink-examples module
Key: FLINK-7037
URL: https://issues.apache.org/jira/browse/FLINK-7037
Project: Flink
Issue Type: Bug
Components: Build System
Affects Versions: 1.4.0
Reporter: Chesnay Schepler
Fix For: 1.4.0
Exceptions such as this
{code}
[ERROR] Failed to execute goal on project flink-tests_2.11: Could not resolve
dependencies for project org.apache.flink:flink-tests_2.11:jar:1.4-SNAPSHOT:
Failed to collect dependencies at
org.apache.flink:flink-examples-batch_2.11:jar:1.4-SNAPSHOT: Failed to read
artifact descriptor for
org.apache.flink:flink-examples-batch_2.11:jar:1.4-SNAPSHOT: Failure to find
org.apache.flink:flink-examples_${scala.binary.version}:pom:1.4-SNAPSHOT in
https://oss.sonatype.org/content/repositories/snapshots/ was cached in the
local repository, resolution will not be reattempted until the update interval
of sonatype-snapshots has elapsed or updates are forced -> [Help 1]
{code}
are caused by modules that have a parent module which have a scala suffix. The
scala version is not properly resolved for parents. In retrospect this is
rather obvious, as the scala version is defined in the parent; in order to
resolve the scala version one would ahve to know who the parent is, but you
need the scala version for that.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #4221: [FLINK-7037] Remove scala suffic from flink-exampl...
GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/4221 [FLINK-7037] Remove scala suffic from flink-examples module This PR removes the scala version suffix from the flink-examples module. The suffix prevented building single modules that rely on any example module. This does not cause issues when building flink completely as we traverse the modules from the parent, allowing the scala version to be resolved for all modules. When building a single module in order to resolve the scala version we have to traverse from the target module towards the parent, which isn't possible since the scala version isn't resolved. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 7037 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/4221.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4221 commit 6aadbc9237ae65313e73259021d6654115bc5461 Author: zentol Date: 2017-06-29T09:26:32Z [FLINK-7037] Remove scala suffic from flink-examples module --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-7037) Remove scala suffix from flink-examples module
[
https://issues.apache.org/jira/browse/FLINK-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068071#comment-16068071
]
ASF GitHub Bot commented on FLINK-7037:
---
GitHub user zentol opened a pull request:
https://github.com/apache/flink/pull/4221
[FLINK-7037] Remove scala suffic from flink-examples module
This PR removes the scala version suffix from the flink-examples module.
The suffix prevented building single modules that rely on any example
module. This does not cause issues when building flink completely as we
traverse the modules from the parent, allowing the scala version to be resolved
for all modules.
When building a single module in order to resolve the scala version we have
to traverse from the target module towards the parent, which isn't possible
since the scala version isn't resolved.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zentol/flink 7037
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4221.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4221
commit 6aadbc9237ae65313e73259021d6654115bc5461
Author: zentol
Date: 2017-06-29T09:26:32Z
[FLINK-7037] Remove scala suffic from flink-examples module
> Remove scala suffix from flink-examples module
> --
>
> Key: FLINK-7037
> URL: https://issues.apache.org/jira/browse/FLINK-7037
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>
> Exceptions such as this
> {code}
> [ERROR] Failed to execute goal on project flink-tests_2.11: Could not resolve
> dependencies for project org.apache.flink:flink-tests_2.11:jar:1.4-SNAPSHOT:
> Failed to collect dependencies at
> org.apache.flink:flink-examples-batch_2.11:jar:1.4-SNAPSHOT: Failed to read
> artifact descriptor for
> org.apache.flink:flink-examples-batch_2.11:jar:1.4-SNAPSHOT: Failure to find
> org.apache.flink:flink-examples_${scala.binary.version}:pom:1.4-SNAPSHOT in
> https://oss.sonatype.org/content/repositories/snapshots/ was cached in the
> local repository, resolution will not be reattempted until the update
> interval of sonatype-snapshots has elapsed or updates are forced -> [Help 1]
> {code}
> are caused by modules that have a parent module which have a scala suffix.
> The scala version is not properly resolved for parents. In retrospect this is
> rather obvious, as the scala version is defined in the parent; in order to
> resolve the scala version one would ahve to know who the parent is, but you
> need the scala version for that.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-4022) Partition discovery / regex topic subscription for the Kafka consumer
[ https://issues.apache.org/jira/browse/FLINK-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068095#comment-16068095 ] ASF GitHub Bot commented on FLINK-4022: --- Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3746 Hi @aljoscha, I've rebased this PR, fixed the previous blocking issue (discovery enabling & restoring from round-robin list state), and added documentation for the limitation. Could you have another look? Thanks! > Partition discovery / regex topic subscription for the Kafka consumer > - > > Key: FLINK-4022 > URL: https://issues.apache.org/jira/browse/FLINK-4022 > Project: Flink > Issue Type: New Feature > Components: Kafka Connector, Streaming Connectors >Affects Versions: 1.0.0 >Reporter: Tzu-Li (Gordon) Tai >Assignee: Tzu-Li (Gordon) Tai >Priority: Critical > Fix For: 1.4.0 > > > Example: allow users to subscribe to "topic-n*", so that the consumer > automatically reads from "topic-n1", "topic-n2", ... and so on as they are > added to Kafka. > I propose to implement this feature by the following description: > Since the overall list of partitions to read will change after job > submission, the main big change required for this feature will be dynamic > partition assignment to subtasks while the Kafka consumer is running. This > will mainly be accomplished using Kafka 0.9.x API > `KafkaConsumer#subscribe(java.util.regex.Pattern, > ConsumerRebalanceListener)`. Each KafkaConsumers in each subtask will be > added to the same consumer group when instantiated, and rely on Kafka to > dynamically reassign partitions to them whenever a rebalance happens. The > registered `ConsumerRebalanceListener` is a callback that is called right > before and after rebalancing happens. We'll use this callback to let each > subtask commit its last offsets of partitions its currently responsible of to > an external store (or Kafka) before a rebalance; after rebalance and the > substasks gets the new partitions it'll be reading from, they'll read from > the external store to get the last offsets for their new partitions > (partitions which don't have offset entries in the store are new partitions > causing the rebalancing). > The tricky part will be restoring Flink checkpoints when the partition > assignment is dynamic. Snapshotting will remain the same - subtasks snapshot > the offsets of partitions they are currently holding. Restoring will be a > bit different in that subtasks might not be assigned matching partitions to > the snapshot the subtask is restored with (since we're letting Kafka > dynamically assign partitions). There will need to be a coordination process > where, if a restore state exists, all subtasks first commit the offsets they > receive (as a result of the restore state) to the external store, and then > all subtasks attempt to find a last offset for the partitions it is holding. > However, if the globally merged restore state feature mentioned by > [~StephanEwen] in https://issues.apache.org/jira/browse/FLINK-3231 is > available, then the restore will be simple again, as each subtask has full > access to previous global state therefore coordination is not required. > I think changing to dynamic partition assignment is also good in the long run > for handling topic repartitioning. > Overall, > User-facing API changes: > - New constructor - FlinkKafkaConsumer09(java.util.regex.Pattern, > DeserializationSchema, Properties) > - New constructor - FlinkKafkaConsumer09(java.util.regex.Pattern, > KeyedDeserializationSchema, Properties) > Implementation changes: > 1. Dynamic partition assigning depending on KafkaConsumer#subscribe > - Remove partition list querying from constructor > - Remove static partition assigning to substasks in run() > - Instead of using KafkaConsumer#assign() in fetchers to manually assign > static partitions, use subscribe() registered with the callback > implementation explained above. > 2. Restoring from checkpointed states > - Snapshotting should remain unchanged > - Restoring requires subtasks to coordinate the restored offsets they hold > before continuing (unless we are able to have merged restore states). > 3. For previous consumer functionality (consume from fixed list of topics), > the KafkaConsumer#subscribe() has a corresponding overload method for fixed > list of topics. We can simply decide which subscribe() overload to use > depending on whether a regex Pattern or list of topics is supplied. > 4. If subtasks don't initially have any assigned partitions, we shouldn't > emit MAX_VALUE watermark, since it may hold partitions after a rebalance. > Instead, un-assigned subtasks should be running a fetcher instanc
[GitHub] flink issue #3746: [FLINK-4022] [kafka] Partition and topic discovery for Fl...
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3746 Hi @aljoscha, I've rebased this PR, fixed the previous blocking issue (discovery enabling & restoring from round-robin list state), and added documentation for the limitation. Could you have another look? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (FLINK-7032) Intellij is constantly changing language level of sub projects back to 1.6
[
https://issues.apache.org/jira/browse/FLINK-7032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aegeaner updated FLINK-7032:
Component/s: Build System
> Intellij is constantly changing language level of sub projects back to 1.6
> ---
>
> Key: FLINK-7032
> URL: https://issues.apache.org/jira/browse/FLINK-7032
> Project: Flink
> Issue Type: Improvement
> Components: Build System
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
>
> Every time I do maven reimport projects, Intellij is switching back to 1.6
> language level. I tracked down this issue to misconfiguration in our pom.xml
> file. It correctly configure maven-compiler-plugin:
> {code:xml}
>
>
> org.apache.maven.plugins
> maven-compiler-plugin
> 3.1
>
> ${java.version}
> ${java.version}
>
>
> -Xlint:all
>
>
> {code}
> where ${java.version} is set to 1.7 in the properties, but it forgets to
> overwrite the following properties from apache-18.pom:
> {code:xml}
>
> 1.6
> 1.6
>
> {code}
> It seems like compiling from console using maven ignores those values, but
> they are confusing Intellij.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-7038) Several misused "KeyedDataStream" term in docs and Javadocs
[
https://issues.apache.org/jira/browse/FLINK-7038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai updated FLINK-7038:
---
Labels: starter (was: )
> Several misused "KeyedDataStream" term in docs and Javadocs
> ---
>
> Key: FLINK-7038
> URL: https://issues.apache.org/jira/browse/FLINK-7038
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.3.1, 1.4.0
>Reporter: Tzu-Li (Gordon) Tai
>Priority: Trivial
> Labels: starter
>
> The correct term should be {{KeyedStream}}.
> For example, in
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/datastream_api.html,
> it says "See keys on how to specify keys. This transformation returns a
> *KeyedDataStream*."
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-6109) Add "consumer lag" report metric to FlinkKafkaConsumer
[
https://issues.apache.org/jira/browse/FLINK-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai updated FLINK-6109:
---
Labels: starter (was: )
> Add "consumer lag" report metric to FlinkKafkaConsumer
> --
>
> Key: FLINK-6109
> URL: https://issues.apache.org/jira/browse/FLINK-6109
> Project: Flink
> Issue Type: New Feature
> Components: Kafka Connector, Streaming Connectors
>Reporter: Tzu-Li (Gordon) Tai
>
> This is a feature discussed in this ML:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Telling-if-a-job-has-caught-up-with-Kafka-td12261.html.
> As discussed, we can expose two kinds of "consumer lag" metrics for this:
> - *current consumer lag per partition:* the current difference between the
> latest offset and the last collected record. This metric is calculated and
> updated at a configurable interval. This metric basically serves as an
> indicator of how the consumer is keeping up with the head of partitions. I
> propose to name this {{currentOffsetLag}}.
> - *Consumer lag of last checkpoint per partition:* the difference between
> the latest offset and the offset stored in the checkpoint. This metric is
> only updated when checkpoints are completed. It serves as an indicator of how
> much data may need to be replayed in case of a failure. I propose to name
> this {{lastCheckpointedOffsetLag}}.
> I don't think it is reasonable to define a metric of whether or not a
> consumer has "caught up" with the HEAD. That would imply a threshold for the
> offset difference. We should probably leave this "caught up" logic for the
> user to determine themselves when they query this metric.
> The granularity of the metric is per-FlinkKafkaConsumer, and independent of
> the consumer group.id used (the offset used to calculate consumer lag is the
> internal offset state of the FlinkKafkaConsumer, not the consumer group's
> committed offsets in Kafka).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-6109) Add "consumer lag" report metric to FlinkKafkaConsumer
[
https://issues.apache.org/jira/browse/FLINK-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai updated FLINK-6109:
---
Labels: (was: starter)
> Add "consumer lag" report metric to FlinkKafkaConsumer
> --
>
> Key: FLINK-6109
> URL: https://issues.apache.org/jira/browse/FLINK-6109
> Project: Flink
> Issue Type: New Feature
> Components: Kafka Connector, Streaming Connectors
>Reporter: Tzu-Li (Gordon) Tai
>
> This is a feature discussed in this ML:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Telling-if-a-job-has-caught-up-with-Kafka-td12261.html.
> As discussed, we can expose two kinds of "consumer lag" metrics for this:
> - *current consumer lag per partition:* the current difference between the
> latest offset and the last collected record. This metric is calculated and
> updated at a configurable interval. This metric basically serves as an
> indicator of how the consumer is keeping up with the head of partitions. I
> propose to name this {{currentOffsetLag}}.
> - *Consumer lag of last checkpoint per partition:* the difference between
> the latest offset and the offset stored in the checkpoint. This metric is
> only updated when checkpoints are completed. It serves as an indicator of how
> much data may need to be replayed in case of a failure. I propose to name
> this {{lastCheckpointedOffsetLag}}.
> I don't think it is reasonable to define a metric of whether or not a
> consumer has "caught up" with the HEAD. That would imply a threshold for the
> offset difference. We should probably leave this "caught up" logic for the
> user to determine themselves when they query this metric.
> The granularity of the metric is per-FlinkKafkaConsumer, and independent of
> the consumer group.id used (the offset used to calculate consumer lag is the
> internal offset state of the FlinkKafkaConsumer, not the consumer group's
> committed offsets in Kafka).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7038) Several misused "KeyedDataStream" term in docs and Javadocs
Tzu-Li (Gordon) Tai created FLINK-7038:
--
Summary: Several misused "KeyedDataStream" term in docs and
Javadocs
Key: FLINK-7038
URL: https://issues.apache.org/jira/browse/FLINK-7038
Project: Flink
Issue Type: Bug
Components: Documentation
Affects Versions: 1.3.1, 1.4.0
Reporter: Tzu-Li (Gordon) Tai
Priority: Trivial
The correct term should be {{KeyedStream}}.
For example, in
https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/datastream_api.html,
it says "See keys on how to specify keys. This transformation returns a
*KeyedDataStream*."
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7039) Increase forkCountTestPackage for sudo-based Trav
Greg Hogan created FLINK-7039: - Summary: Increase forkCountTestPackage for sudo-based Trav Key: FLINK-7039 URL: https://issues.apache.org/jira/browse/FLINK-7039 Project: Flink Issue Type: Bug Components: Build System Affects Versions: 1.4.0 Reporter: Greg Hogan Assignee: Greg Hogan Priority: Trivial Fix For: 1.4.0 https://docs.travis-ci.com/user/ci-environment/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-7039) Increase forkCountTestPackage for sudo-based TravisCI
[ https://issues.apache.org/jira/browse/FLINK-7039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Hogan updated FLINK-7039: -- Description: The switch from the container-based to sudo-enabled environment in TravisCI has increased available memory from 4 GB to 7.5 GB so use a forkCount of 2 in all packages including flink-test. See https://docs.travis-ci.com/user/ci-environment/ (was: https://docs.travis-ci.com/user/ci-environment/) > Increase forkCountTestPackage for sudo-based TravisCI > - > > Key: FLINK-7039 > URL: https://issues.apache.org/jira/browse/FLINK-7039 > Project: Flink > Issue Type: Bug > Components: Build System >Affects Versions: 1.4.0 >Reporter: Greg Hogan >Assignee: Greg Hogan >Priority: Trivial > Fix For: 1.4.0 > > > The switch from the container-based to sudo-enabled environment in TravisCI > has increased available memory from 4 GB to 7.5 GB so use a forkCount of 2 in > all packages including flink-test. See > https://docs.travis-ci.com/user/ci-environment/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-7039) Increase forkCountTestPackage for sudo-enabled TravisCI
[ https://issues.apache.org/jira/browse/FLINK-7039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Hogan updated FLINK-7039: -- Description: The switch from the container-based to sudo-enabled environment in TravisCI has increased available memory from 4 GB to 7.5 GB so use a forkCount of 2 in all packages including flink-test. See https://docs.travis-ci.com/user/ci-environment/ The sudo-enabled machines look to be Google Compute Engine n1-standard-2, see was:The switch from the container-based to sudo-enabled environment in TravisCI has increased available memory from 4 GB to 7.5 GB so use a forkCount of 2 in all packages including flink-test. See https://docs.travis-ci.com/user/ci-environment/ > Increase forkCountTestPackage for sudo-enabled TravisCI > --- > > Key: FLINK-7039 > URL: https://issues.apache.org/jira/browse/FLINK-7039 > Project: Flink > Issue Type: Bug > Components: Build System >Affects Versions: 1.4.0 >Reporter: Greg Hogan >Assignee: Greg Hogan >Priority: Trivial > Fix For: 1.4.0 > > > The switch from the container-based to sudo-enabled environment in TravisCI > has increased available memory from 4 GB to 7.5 GB so use a forkCount of 2 in > all packages including flink-test. See > https://docs.travis-ci.com/user/ci-environment/ > The sudo-enabled machines look to be Google Compute Engine n1-standard-2, see -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (FLINK-4968) Replace direct JobManager ActorGateways by JobManagerGateway
[
https://issues.apache.org/jira/browse/FLINK-4968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann reassigned FLINK-4968:
Assignee: (was: Till Rohrmann)
> Replace direct JobManager ActorGateways by JobManagerGateway
>
>
> Key: FLINK-4968
> URL: https://issues.apache.org/jira/browse/FLINK-4968
> Project: Flink
> Issue Type: Improvement
> Components: JobManager
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
> Labels: flip-6
>
> Components which have to interact with the {{JobManager}} do RPCs via the
> {{ActorGateway}}. This exposes the underlying communication mechanism and is
> thus not easy to swap. In order to prepare the master branch for the Flip-6
> changes, I propose to encapsulate the implementation specific communication
> details behind a {{JobManagerGateway}} interface. The implementation of this
> interface allows to use different systems for RPCs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-4968) Replace direct JobManager ActorGateways by JobManagerGateway
[
https://issues.apache.org/jira/browse/FLINK-4968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-4968:
-
Labels: flip-6 (was: )
> Replace direct JobManager ActorGateways by JobManagerGateway
>
>
> Key: FLINK-4968
> URL: https://issues.apache.org/jira/browse/FLINK-4968
> Project: Flink
> Issue Type: Improvement
> Components: JobManager
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Labels: flip-6
>
> Components which have to interact with the {{JobManager}} do RPCs via the
> {{ActorGateway}}. This exposes the underlying communication mechanism and is
> thus not easy to swap. In order to prepare the master branch for the Flip-6
> changes, I propose to encapsulate the implementation specific communication
> details behind a {{JobManagerGateway}} interface. The implementation of this
> interface allows to use different systems for RPCs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #2761: [FLINK-3551] [examples] Sync Scala Streaming Examp...
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2761#discussion_r124667797 --- Diff: flink-examples/flink-examples-streaming/src/test/java/org/apache/flink/streaming/test/StreamingExamplesITCase.java --- @@ -0,0 +1,135 @@ +package org.apache.flink.streaming.test; --- End diff -- AL2 header missing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2761: [FLINK-3551] [examples] Sync Scala Streaming Examp...
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/2761#discussion_r124658452
--- Diff:
flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/windowing/SessionWindowing.java
---
@@ -70,9 +70,7 @@ public void run(SourceContext> ctx) throws Excepti
for (Tuple3 value : input) {
ctx.collectWithTimestamp(value, value.f1);
ctx.emitWatermark(new
Watermark(value.f1 - 1));
- if (!fileOutput) {
-
System.out.println("Collected: " + value);
- }
+
System.out.println("Collected: " + value);
--- End diff --
remove the `println` as well
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2761: [FLINK-3551] [examples] Sync Scala Streaming Examp...
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2761#discussion_r124658613 --- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/windowing/GroupedProcessingTimeWindowExample.java --- @@ -34,49 +34,17 @@ import static java.util.concurrent.TimeUnit.MILLISECONDS; /** - * Example of grouped processing time windows. + * An example of grouped stream windowing into sliding time windows. + * This example uses [[RichParallelSourceFunction]] to generate a list of key-value pair. --- End diff -- +s -> `... list of key-value pairs.` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2761: [FLINK-3551] [examples] Sync Scala Streaming Examp...
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2761#discussion_r124658721 --- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/windowing/GroupedProcessingTimeWindowExample.java --- @@ -126,4 +94,40 @@ public void apply(Long key, Window window, Iterable> values, return new Tuple2<>(value1.f0, value1.f1 + value2.f1); } } + + /** +* Parallel data source that serves a list of key-value pair. --- End diff -- +s -> `... list of key-value pairs.` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (FLINK-7039) Increase forkCountTestPackage for sudo-enabled TravisCI
[ https://issues.apache.org/jira/browse/FLINK-7039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Hogan updated FLINK-7039: -- Description: The switch from the container-based to sudo-enabled environment in TravisCI has increased available memory from 4 GB to 7.5 GB so use a forkCount of 2 in all packages including flink-test. See https://docs.travis-ci.com/user/ci-environment/ The sudo-enabled machines look to be Google Compute Engine n1-standard-2 with 2 "virtual CPUs". See https://cloud.google.com/compute/pricing. was: The switch from the container-based to sudo-enabled environment in TravisCI has increased available memory from 4 GB to 7.5 GB so use a forkCount of 2 in all packages including flink-test. See https://docs.travis-ci.com/user/ci-environment/ The sudo-enabled machines look to be Google Compute Engine n1-standard-2, see > Increase forkCountTestPackage for sudo-enabled TravisCI > --- > > Key: FLINK-7039 > URL: https://issues.apache.org/jira/browse/FLINK-7039 > Project: Flink > Issue Type: Bug > Components: Build System >Affects Versions: 1.4.0 >Reporter: Greg Hogan >Assignee: Greg Hogan >Priority: Trivial > Fix For: 1.4.0 > > > The switch from the container-based to sudo-enabled environment in TravisCI > has increased available memory from 4 GB to 7.5 GB so use a forkCount of 2 in > all packages including flink-test. See > https://docs.travis-ci.com/user/ci-environment/ > The sudo-enabled machines look to be Google Compute Engine n1-standard-2 with > 2 "virtual CPUs". See https://cloud.google.com/compute/pricing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink pull request #2761: [FLINK-3551] [examples] Sync Scala Streaming Examp...
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2761#discussion_r124667752 --- Diff: flink-examples/flink-examples-streaming/src/test/scala/org/apache/flink/streaming/scala/examples/StreamingExamplesITCase.scala --- @@ -0,0 +1,136 @@ +package org.apache.flink.streaming.scala.examples --- End diff -- AL2 header missing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3551) Sync Scala and Java Streaming Examples
[ https://issues.apache.org/jira/browse/FLINK-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068149#comment-16068149 ] ASF GitHub Bot commented on FLINK-3551: --- Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2761#discussion_r124658721 --- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/windowing/GroupedProcessingTimeWindowExample.java --- @@ -126,4 +94,40 @@ public void apply(Long key, Window window, Iterable> values, return new Tuple2<>(value1.f0, value1.f1 + value2.f1); } } + + /** +* Parallel data source that serves a list of key-value pair. --- End diff -- +s -> `... list of key-value pairs.` > Sync Scala and Java Streaming Examples > -- > > Key: FLINK-3551 > URL: https://issues.apache.org/jira/browse/FLINK-3551 > Project: Flink > Issue Type: Sub-task > Components: Examples >Affects Versions: 1.0.0 >Reporter: Stephan Ewen >Assignee: Lim Chee Hau > > The Scala Examples lack behind the Java Examples -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-3551) Sync Scala and Java Streaming Examples
[ https://issues.apache.org/jira/browse/FLINK-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068150#comment-16068150 ] ASF GitHub Bot commented on FLINK-3551: --- Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2761#discussion_r124667797 --- Diff: flink-examples/flink-examples-streaming/src/test/java/org/apache/flink/streaming/test/StreamingExamplesITCase.java --- @@ -0,0 +1,135 @@ +package org.apache.flink.streaming.test; --- End diff -- AL2 header missing > Sync Scala and Java Streaming Examples > -- > > Key: FLINK-3551 > URL: https://issues.apache.org/jira/browse/FLINK-3551 > Project: Flink > Issue Type: Sub-task > Components: Examples >Affects Versions: 1.0.0 >Reporter: Stephan Ewen >Assignee: Lim Chee Hau > > The Scala Examples lack behind the Java Examples -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-3551) Sync Scala and Java Streaming Examples
[ https://issues.apache.org/jira/browse/FLINK-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068152#comment-16068152 ] ASF GitHub Bot commented on FLINK-3551: --- Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2761#discussion_r124658613 --- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/windowing/GroupedProcessingTimeWindowExample.java --- @@ -34,49 +34,17 @@ import static java.util.concurrent.TimeUnit.MILLISECONDS; /** - * Example of grouped processing time windows. + * An example of grouped stream windowing into sliding time windows. + * This example uses [[RichParallelSourceFunction]] to generate a list of key-value pair. --- End diff -- +s -> `... list of key-value pairs.` > Sync Scala and Java Streaming Examples > -- > > Key: FLINK-3551 > URL: https://issues.apache.org/jira/browse/FLINK-3551 > Project: Flink > Issue Type: Sub-task > Components: Examples >Affects Versions: 1.0.0 >Reporter: Stephan Ewen >Assignee: Lim Chee Hau > > The Scala Examples lack behind the Java Examples -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-3551) Sync Scala and Java Streaming Examples
[
https://issues.apache.org/jira/browse/FLINK-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068151#comment-16068151
]
ASF GitHub Bot commented on FLINK-3551:
---
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/2761#discussion_r124658452
--- Diff:
flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/windowing/SessionWindowing.java
---
@@ -70,9 +70,7 @@ public void run(SourceContext> ctx) throws Excepti
for (Tuple3 value : input) {
ctx.collectWithTimestamp(value, value.f1);
ctx.emitWatermark(new
Watermark(value.f1 - 1));
- if (!fileOutput) {
-
System.out.println("Collected: " + value);
- }
+
System.out.println("Collected: " + value);
--- End diff --
remove the `println` as well
> Sync Scala and Java Streaming Examples
> --
>
> Key: FLINK-3551
> URL: https://issues.apache.org/jira/browse/FLINK-3551
> Project: Flink
> Issue Type: Sub-task
> Components: Examples
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Lim Chee Hau
>
> The Scala Examples lack behind the Java Examples
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-3551) Sync Scala and Java Streaming Examples
[ https://issues.apache.org/jira/browse/FLINK-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068148#comment-16068148 ] ASF GitHub Bot commented on FLINK-3551: --- Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2761#discussion_r124667752 --- Diff: flink-examples/flink-examples-streaming/src/test/scala/org/apache/flink/streaming/scala/examples/StreamingExamplesITCase.scala --- @@ -0,0 +1,136 @@ +package org.apache.flink.streaming.scala.examples --- End diff -- AL2 header missing > Sync Scala and Java Streaming Examples > -- > > Key: FLINK-3551 > URL: https://issues.apache.org/jira/browse/FLINK-3551 > Project: Flink > Issue Type: Sub-task > Components: Examples >Affects Versions: 1.0.0 >Reporter: Stephan Ewen >Assignee: Lim Chee Hau > > The Scala Examples lack behind the Java Examples -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-7039) Increase forkCountTestPackage for sudo-based TravisCI
[ https://issues.apache.org/jira/browse/FLINK-7039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Hogan updated FLINK-7039: -- Summary: Increase forkCountTestPackage for sudo-based TravisCI (was: Increase forkCountTestPackage for sudo-based Trav) > Increase forkCountTestPackage for sudo-based TravisCI > - > > Key: FLINK-7039 > URL: https://issues.apache.org/jira/browse/FLINK-7039 > Project: Flink > Issue Type: Bug > Components: Build System >Affects Versions: 1.4.0 >Reporter: Greg Hogan >Assignee: Greg Hogan >Priority: Trivial > Fix For: 1.4.0 > > > https://docs.travis-ci.com/user/ci-environment/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-7039) Increase forkCountTestPackage for sudo-enabled TravisCI
[ https://issues.apache.org/jira/browse/FLINK-7039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Hogan updated FLINK-7039: -- Summary: Increase forkCountTestPackage for sudo-enabled TravisCI (was: Increase forkCountTestPackage for sudo-based TravisCI) > Increase forkCountTestPackage for sudo-enabled TravisCI > --- > > Key: FLINK-7039 > URL: https://issues.apache.org/jira/browse/FLINK-7039 > Project: Flink > Issue Type: Bug > Components: Build System >Affects Versions: 1.4.0 >Reporter: Greg Hogan >Assignee: Greg Hogan >Priority: Trivial > Fix For: 1.4.0 > > > The switch from the container-based to sudo-enabled environment in TravisCI > has increased available memory from 4 GB to 7.5 GB so use a forkCount of 2 in > all packages including flink-test. See > https://docs.travis-ci.com/user/ci-environment/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-7039) Increase forkCountTestPackage for sudo-enabled TravisCI
[ https://issues.apache.org/jira/browse/FLINK-7039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068189#comment-16068189 ] Chesnay Schepler commented on FLINK-7039: - I've tried this out but didn't really improve build times. > Increase forkCountTestPackage for sudo-enabled TravisCI > --- > > Key: FLINK-7039 > URL: https://issues.apache.org/jira/browse/FLINK-7039 > Project: Flink > Issue Type: Bug > Components: Build System >Affects Versions: 1.4.0 >Reporter: Greg Hogan >Assignee: Greg Hogan >Priority: Trivial > Fix For: 1.4.0 > > > The switch from the container-based to sudo-enabled environment in TravisCI > has increased available memory from 4 GB to 7.5 GB so use a forkCount of 2 in > all packages including flink-test. See > https://docs.travis-ci.com/user/ci-environment/ > The sudo-enabled machines look to be Google Compute Engine n1-standard-2 with > 2 "virtual CPUs". See https://cloud.google.com/compute/pricing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink pull request #4222: [FLINK-7039] [build] Increase forkCountTestPackage...
GitHub user greghogan opened a pull request: https://github.com/apache/flink/pull/4222 [FLINK-7039] [build] Increase forkCountTestPackage for sudo-enabled TravisCI The switch from the container-based to sudo-enabled environment in TravisCI has increased available memory from 4 GB to 7.5 GB so use a forkCount of 2 in all packages including flink-test. You can merge this pull request into a Git repository by running: $ git pull https://github.com/greghogan/flink 7039_increase_forkcounttestpackage_for_sudo_enabled_travisci Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/4222.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4222 commit 38fb54a0fdd853e0553fbe21a2148c752a32 Author: Greg Hogan Date: 2017-06-29T10:42:51Z [FLINK-7039] [build] Increase forkCountTestPackage for sudo-enabled TravisCI The switch from the container-based to sudo-enabled environment in TravisCI has increased available memory from 4 GB to 7.5 GB so use a forkCount of 2 in all packages including flink-test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-7039) Increase forkCountTestPackage for sudo-enabled TravisCI
[ https://issues.apache.org/jira/browse/FLINK-7039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068202#comment-16068202 ] Greg Hogan commented on FLINK-7039: --- Even a small improvement is worthwhile. > Increase forkCountTestPackage for sudo-enabled TravisCI > --- > > Key: FLINK-7039 > URL: https://issues.apache.org/jira/browse/FLINK-7039 > Project: Flink > Issue Type: Bug > Components: Build System >Affects Versions: 1.4.0 >Reporter: Greg Hogan >Assignee: Greg Hogan >Priority: Trivial > Fix For: 1.4.0 > > > The switch from the container-based to sudo-enabled environment in TravisCI > has increased available memory from 4 GB to 7.5 GB so use a forkCount of 2 in > all packages including flink-test. See > https://docs.travis-ci.com/user/ci-environment/ > The sudo-enabled machines look to be Google Compute Engine n1-standard-2 with > 2 "virtual CPUs". See https://cloud.google.com/compute/pricing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-7039) Increase forkCountTestPackage for sudo-enabled TravisCI
[ https://issues.apache.org/jira/browse/FLINK-7039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068203#comment-16068203 ] ASF GitHub Bot commented on FLINK-7039: --- GitHub user greghogan opened a pull request: https://github.com/apache/flink/pull/4222 [FLINK-7039] [build] Increase forkCountTestPackage for sudo-enabled TravisCI The switch from the container-based to sudo-enabled environment in TravisCI has increased available memory from 4 GB to 7.5 GB so use a forkCount of 2 in all packages including flink-test. You can merge this pull request into a Git repository by running: $ git pull https://github.com/greghogan/flink 7039_increase_forkcounttestpackage_for_sudo_enabled_travisci Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/4222.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4222 commit 38fb54a0fdd853e0553fbe21a2148c752a32 Author: Greg Hogan Date: 2017-06-29T10:42:51Z [FLINK-7039] [build] Increase forkCountTestPackage for sudo-enabled TravisCI The switch from the container-based to sudo-enabled environment in TravisCI has increased available memory from 4 GB to 7.5 GB so use a forkCount of 2 in all packages including flink-test. > Increase forkCountTestPackage for sudo-enabled TravisCI > --- > > Key: FLINK-7039 > URL: https://issues.apache.org/jira/browse/FLINK-7039 > Project: Flink > Issue Type: Bug > Components: Build System >Affects Versions: 1.4.0 >Reporter: Greg Hogan >Assignee: Greg Hogan >Priority: Trivial > Fix For: 1.4.0 > > > The switch from the container-based to sudo-enabled environment in TravisCI > has increased available memory from 4 GB to 7.5 GB so use a forkCount of 2 in > all packages including flink-test. See > https://docs.travis-ci.com/user/ci-environment/ > The sudo-enabled machines look to be Google Compute Engine n1-standard-2 with > 2 "virtual CPUs". See https://cloud.google.com/compute/pricing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #4222: [FLINK-7039] [build] Increase forkCountTestPackage for su...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4222 +1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-7039) Increase forkCountTestPackage for sudo-enabled TravisCI
[ https://issues.apache.org/jira/browse/FLINK-7039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068209#comment-16068209 ] ASF GitHub Bot commented on FLINK-7039: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/4222 +1. > Increase forkCountTestPackage for sudo-enabled TravisCI > --- > > Key: FLINK-7039 > URL: https://issues.apache.org/jira/browse/FLINK-7039 > Project: Flink > Issue Type: Bug > Components: Build System >Affects Versions: 1.4.0 >Reporter: Greg Hogan >Assignee: Greg Hogan >Priority: Trivial > Fix For: 1.4.0 > > > The switch from the container-based to sudo-enabled environment in TravisCI > has increased available memory from 4 GB to 7.5 GB so use a forkCount of 2 in > all packages including flink-test. See > https://docs.travis-ci.com/user/ci-environment/ > The sudo-enabled machines look to be Google Compute Engine n1-standard-2 with > 2 "virtual CPUs". See https://cloud.google.com/compute/pricing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-6434) There may be allocatedSlots leak in SlotPool
[ https://issues.apache.org/jira/browse/FLINK-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068237#comment-16068237 ] Till Rohrmann commented on FLINK-6434: -- Hi [~tiemsn], what's the state of the fix? Are you working on this issue? > There may be allocatedSlots leak in SlotPool > > > Key: FLINK-6434 > URL: https://issues.apache.org/jira/browse/FLINK-6434 > Project: Flink > Issue Type: Bug > Components: Cluster Management >Reporter: shuai.xu >Assignee: shuai.xu > Labels: flip-6 > > If the call allocateSlot() from Execution to Slotpool timeout, the job will > begin to failover, but the pending request are still in SlotPool, if then a > new slot register to SlotPool, it may be fulfill the outdated pending request > and be added to allocatedSlots, but it will never be used and will never be > recycled. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-5857) Recycle idle containers in time for yarn mode
[ https://issues.apache.org/jira/browse/FLINK-5857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068240#comment-16068240 ] Till Rohrmann commented on FLINK-5857: -- Can we close this issue [~tiemsn]? > Recycle idle containers in time for yarn mode > - > > Key: FLINK-5857 > URL: https://issues.apache.org/jira/browse/FLINK-5857 > Project: Flink > Issue Type: Bug > Components: YARN >Reporter: shuai.xu >Assignee: shuai.xu > Labels: flip-6 > > When we run flink batch job like map reduce, after a map is finished, the > container for it may be idle for a long time, we need to have a strategy to > recycle there container to reduce resource usage -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #3398: [FLINK-5856] [FLIP-6] return redundant containers to yarn...
Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/3398 Just wanted to check what's the state of this PR? Do you have time to address the review comments @shuai-xu? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5856) Need return redundant containers to yarn for yarn mode
[ https://issues.apache.org/jira/browse/FLINK-5856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068243#comment-16068243 ] ASF GitHub Bot commented on FLINK-5856: --- Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/3398 Just wanted to check what's the state of this PR? Do you have time to address the review comments @shuai-xu? > Need return redundant containers to yarn for yarn mode > -- > > Key: FLINK-5856 > URL: https://issues.apache.org/jira/browse/FLINK-5856 > Project: Flink > Issue Type: Bug > Components: YARN >Reporter: shuai.xu >Assignee: shuai.xu > Labels: flip-6 > > In flip6, for flink on yarn mode, RM requests container from yarn according > to the requirement of the JM. But the AMRMClientAsync used in yarn doesn't > guarantee that the number of containers returned exactly equal to the number > requested. So it need to record the number request by flink rm and return the > redundant ones to yarn. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (FLINK-5254) Implement Yarn HA Services
[
https://issues.apache.org/jira/browse/FLINK-5254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann resolved FLINK-5254.
--
Resolution: Fixed
Added via 2a7dbda79a00863a511fcf64b339770d1d00f805
> Implement Yarn HA Services
> --
>
> Key: FLINK-5254
> URL: https://issues.apache.org/jira/browse/FLINK-5254
> Project: Flink
> Issue Type: Sub-task
> Components: YARN
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Labels: flip-6
>
> The Yarn HighAvailability Services should be
> **Default**
> - This option takes the YARN Application's working directory as HA storage
> - It automatically uses that working directory for the BlobStore
> - It creates a HDFS based "RunningJobsRegistry" (see below)
> - ResourceManager leader election has a pre-configured leader, via the
> configuration, pointing to the AppMaster address.
> **ZooKeeper Based**
> - The ZooKeeper based services use ZooKeeper for the ResourceManager and
> JobManager leader election. That way, they are safe against network partition
> scenarios that otherwise lead to "split brain" situations
> A prototype for the simple "single job" {{RunningJobsRegistry}} based on HDFS
> is here:
> https://github.com/StephanEwen/incubator-flink/commit/aaa2d7758797b2d6c9b6da42be6a5c4989468e3b
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-4926) Implement FLIP-6 Flink-on-YARN jobs
[ https://issues.apache.org/jira/browse/FLINK-4926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-4926: - Labels: flip-6 (was: ) > Implement FLIP-6 Flink-on-YARN jobs > --- > > Key: FLINK-4926 > URL: https://issues.apache.org/jira/browse/FLINK-4926 > Project: Flink > Issue Type: New Feature > Components: YARN >Reporter: Stephan Ewen > Labels: flip-6 > > This is the parent issue for the effort to implement Flink jobs on YARN. > This covers individual jobs running as Yarn applications, where the > application and Job lifetime are coupled. > The scope does not contain the effort to implement "YARN sessions", which are > clusters accepting multiple jobs after another. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (FLINK-5238) Flip-6 mini cluster should start local TaskManagers if there is only one TaskManager
[
https://issues.apache.org/jira/browse/FLINK-5238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann resolved FLINK-5238.
--
Resolution: Fixed
Fixed via 440e7571a9c91fd1e01addff1b5370563db96abc
> Flip-6 mini cluster should start local TaskManagers if there is only one
> TaskManager
>
>
> Key: FLINK-5238
> URL: https://issues.apache.org/jira/browse/FLINK-5238
> Project: Flink
> Issue Type: Bug
> Components: Distributed Coordination
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Labels: flip-6
>
> MiniCluster execution becomes much faster for short jobs, if the TaskManagers
> do not start their Netty Network Environment unless needed.
> If the MiniCluster has only one TaskManager, the NetworkEnvironment should
> not be started.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-4755) Create ContainerID; Rename ResourceID into ProcessID and InstanceID into RegistrationID
[
https://issues.apache.org/jira/browse/FLINK-4755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-4755:
-
Priority: Major (was: Minor)
> Create ContainerID; Rename ResourceID into ProcessID and InstanceID into
> RegistrationID
> ---
>
> Key: FLINK-4755
> URL: https://issues.apache.org/jira/browse/FLINK-4755
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Coordination
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
> Labels: flip-6
>
> Currently, the {{ResourceID}} is used to identify a container as well as the
> process running in the container. With Flip-6, we should introduce a
> dedicated {{ContainerID}} and rename {{ResourceID}} into {{ProcessID}}. The
> {{ProcessID}} would then be constructed from the {{ContainerID}} and some
> {{UUID}}, for example.
> Additionally we could rename {{InstanceID}} into {{RegistrationID}} which
> would better reflect its true purpose.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-4927) Implement FLIP-6 YARN Resource Manager
[
https://issues.apache.org/jira/browse/FLINK-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-4927:
-
Summary: Implement FLIP-6 YARN Resource Manager (was: Implement FLI-6 YARN
Resource Manager)
> Implement FLIP-6 YARN Resource Manager
> --
>
> Key: FLINK-4927
> URL: https://issues.apache.org/jira/browse/FLINK-4927
> Project: Flink
> Issue Type: Sub-task
> Components: YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>Assignee: shuai.xu
> Labels: flip-6
>
> The Flink YARN Resource Manager communicates with YARN's Resource Manager to
> acquire and release containers.
> It is also responsible to notify the JobManager eagerly about container
> failures.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-4928) Implement FLIP-6 YARN Application Master Runner
[
https://issues.apache.org/jira/browse/FLINK-4928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-4928:
-
Labels: flip-6 (was: )
> Implement FLIP-6 YARN Application Master Runner
> ---
>
> Key: FLINK-4928
> URL: https://issues.apache.org/jira/browse/FLINK-4928
> Project: Flink
> Issue Type: Sub-task
> Components: YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>Assignee: shuai.xu
> Labels: flip-6
>
> The Application Master Runner is the master process started in a YARN
> container when submitting the Flink-on-YARN job to YARN.
> It has the following data available:
> - Flink jars
> - Job jars
> - JobGraph
> - Environment variables
> - Contextual information like security tokens and certificates
> Its responsibility is the following:
> - Read all configuration and environment variables, computing the effective
> configuration
> - Start all shared components (Rpc, HighAvailability Services)
> - Start the ResourceManager
> - Start the JobManager Runner
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-4986) Harden JobMaster logic
[
https://issues.apache.org/jira/browse/FLINK-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-4986:
-
Labels: flip-6 (was: )
> Harden JobMaster logic
> --
>
> Key: FLINK-4986
> URL: https://issues.apache.org/jira/browse/FLINK-4986
> Project: Flink
> Issue Type: Sub-task
> Components: Distributed Coordination
>Reporter: Till Rohrmann
>Assignee: Stephan Ewen
>Priority: Minor
> Labels: flip-6
>
> Some of Flip-6's {{JobMaster}} logic can be streamlined by using switch
> statements instead of if-else if cascades.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-4927) Implement FLIP-6 YARN Resource Manager
[
https://issues.apache.org/jira/browse/FLINK-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-4927:
-
Labels: flip-6 (was: )
> Implement FLIP-6 YARN Resource Manager
> --
>
> Key: FLINK-4927
> URL: https://issues.apache.org/jira/browse/FLINK-4927
> Project: Flink
> Issue Type: Sub-task
> Components: YARN
> Environment: {{flip-6}} feature branch
>Reporter: Stephan Ewen
>Assignee: shuai.xu
> Labels: flip-6
>
> The Flink YARN Resource Manager communicates with YARN's Resource Manager to
> acquire and release containers.
> It is also responsible to notify the JobManager eagerly about container
> failures.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-4344) Implement new JobManager
[
https://issues.apache.org/jira/browse/FLINK-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-4344:
-
Labels: flip-6 (was: )
> Implement new JobManager
>
>
> Key: FLINK-4344
> URL: https://issues.apache.org/jira/browse/FLINK-4344
> Project: Flink
> Issue Type: New Feature
> Components: Cluster Management
>Reporter: Stephan Ewen
> Labels: flip-6
>
> This is the parent issue for the efforts to implement the {{JobManager}}
> changes based on FLIP-6
> (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077)
> Because of the breadth of changes, we should implement a new version of the
> {{JobManager}} (let's call it {{JobMaster}}) rather than updating the current
> {{JobManager}}. That will allow us to keep a working master branch.
> At the point when the new cluster management is on par with the current
> implementation, we will drop the old {{JobManager}} and rename the
> {{JobMaster}} to {{JobManager}}.
> Here is a sketch about the new design of the JobManager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-4319) Rework Cluster Management (FLIP-6)
[ https://issues.apache.org/jira/browse/FLINK-4319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-4319: - Labels: flip-6 (was: ) > Rework Cluster Management (FLIP-6) > -- > > Key: FLINK-4319 > URL: https://issues.apache.org/jira/browse/FLINK-4319 > Project: Flink > Issue Type: Improvement > Components: Cluster Management >Affects Versions: 1.1.0 >Reporter: Stephan Ewen > Labels: flip-6 > > This is the root issue to track progress of the rework of cluster management > (FLIP-6) > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-4834) Implement unified High Availability Services Abstraction
[ https://issues.apache.org/jira/browse/FLINK-4834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-4834: - Labels: flip-6 (was: ) > Implement unified High Availability Services Abstraction > > > Key: FLINK-4834 > URL: https://issues.apache.org/jira/browse/FLINK-4834 > Project: Flink > Issue Type: New Feature > Components: Cluster Management > Environment: FLIP-6 feature branch >Reporter: Stephan Ewen > Labels: flip-6 > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-4338) Implement Slot Pool
[ https://issues.apache.org/jira/browse/FLINK-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-4338: - Labels: flip-6 (was: ) > Implement Slot Pool > --- > > Key: FLINK-4338 > URL: https://issues.apache.org/jira/browse/FLINK-4338 > Project: Flink > Issue Type: New Feature > Components: Cluster Management >Affects Versions: 1.1.0 >Reporter: Stephan Ewen > Labels: flip-6 > > Implement the slot pool as described in the FLIP-6 document: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-4343) Implement new TaskManager
[
https://issues.apache.org/jira/browse/FLINK-4343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-4343:
-
Labels: flip-6 (was: )
> Implement new TaskManager
> -
>
> Key: FLINK-4343
> URL: https://issues.apache.org/jira/browse/FLINK-4343
> Project: Flink
> Issue Type: New Feature
> Components: Cluster Management
>Reporter: Stephan Ewen
> Labels: flip-6
>
> This is the parent issue for the efforts to implement the {{TaskManager}}
> changes based on FLIP-6
> (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077)
> Because of the breadth of changes, we should implement a new version of the
> {{TaskManager}} (let's call it {{TaskExecutor}}) rather than updating the
> current {{TaskManager}}. That will allow us to keep a working master branch.
> At the point when the new cluster management is on par with the current
> implementation, we will drop the old {{TaskManager}} and rename the
> {{TaskExecutor}} to {{TaskManager}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (FLINK-4338) Implement Slot Pool
[ https://issues.apache.org/jira/browse/FLINK-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann resolved FLINK-4338. -- Resolution: Fixed I think this issue can be closed since the slot pool has been implemented. If new things come up, we'll create new issues. > Implement Slot Pool > --- > > Key: FLINK-4338 > URL: https://issues.apache.org/jira/browse/FLINK-4338 > Project: Flink > Issue Type: New Feature > Components: Cluster Management >Affects Versions: 1.1.0 >Reporter: Stephan Ewen > Labels: flip-6 > > Implement the slot pool as described in the FLIP-6 document: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-4350) interaction between new JobManager and new ResourceManager
[ https://issues.apache.org/jira/browse/FLINK-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068269#comment-16068269 ] Till Rohrmann commented on FLINK-4350: -- Hi [~jinyu.zj], what is exactly the problem you're describing here? > interaction between new JobManager and new ResourceManager > -- > > Key: FLINK-4350 > URL: https://issues.apache.org/jira/browse/FLINK-4350 > Project: Flink > Issue Type: Sub-task > Components: Cluster Management >Reporter: jingzhang > > 1. Send Slot Request to ResourceManager > 2. Cancel SlotRequest to ResourceManager > 3. Register at ResourceManager > 4. Receive notification from ResourceManager of Failure TaskManager -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-4897) Implement Dispatcher
[ https://issues.apache.org/jira/browse/FLINK-4897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-4897: - Labels: flip-6 (was: ) > Implement Dispatcher > > > Key: FLINK-4897 > URL: https://issues.apache.org/jira/browse/FLINK-4897 > Project: Flink > Issue Type: New Feature > Components: Cluster Management, Mesos > Environment: FLIP-6 feature branch >Reporter: Eron Wright >Assignee: Eron Wright > Labels: flip-6 > > This task is to implement the dispatcher service, which acts as a remote > frontend for job submission. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-4350) interaction between new JobManager and new ResourceManager
[ https://issues.apache.org/jira/browse/FLINK-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-4350: - Labels: flip-6 (was: ) > interaction between new JobManager and new ResourceManager > -- > > Key: FLINK-4350 > URL: https://issues.apache.org/jira/browse/FLINK-4350 > Project: Flink > Issue Type: Sub-task > Components: Cluster Management >Reporter: jingzhang > Labels: flip-6 > > 1. Send Slot Request to ResourceManager > 2. Cancel SlotRequest to ResourceManager > 3. Register at ResourceManager > 4. Receive notification from ResourceManager of Failure TaskManager -- This message was sent by Atlassian JIRA (v6.4.14#64029)
