[jira] [Created] (FLINK-6420) Cleaner CEP API to specify conditions between events
Elias Levy created FLINK-6420:
-
Summary: Cleaner CEP API to specify conditions between events
Key: FLINK-6420
URL: https://issues.apache.org/jira/browse/FLINK-6420
Project: Flink
Issue Type: Improvement
Components: CEP
Affects Versions: 1.3.0
Reporter: Elias Levy
Priority: Minor
Flink 1.3 will introduce so-called iterative conditions, which allow the
predicate to look up events already matched by conditions in the pattern. This
permits specifying conditions between matched events, similar to a conditional
join between tables in SQL. Alas, the API could be simplified to specify such
conditions more declaratively.
At the moment you have to do something like
{code}
Pattern.
.begin[Foo]("first")
.where( first => first.baz == 1 )
.followedBy("next")
.where({ (next, ctx) =>
val first = ctx.getEventsForPattern("first").next
first.bar == next.bar && next => next.boo = "x"
})
{code}
which is not very clean. It would friendlier if you could do something like:
{code}
Pattern.
.begin[Foo]("first")
.where( first => first.baz == 1 )
.followedBy("next")
.relatedTo("first", { (first, next) => first.bar == next.bar })
.where( next => next.boo = "x" )
{code}
Something along these lines would work well when the condition being tested
against matches a single event (single quantifier).
If the condition being tested can accept multiple events (e.g. times
quantifier) two other methods could be used {{relatedToAny}} and
{{relatedToAll}}, each of which takes a predicate function. In both cases each
previously accepted element of the requested condition is evaluated against the
predicate. In the former case if any evaluation returns true the condition is
satisfied. In the later case all evaluations must return true for the
condition to be satisfied.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6419) Better support for CEP quantified conditions in PatternSelect.select
[
https://issues.apache.org/jira/browse/FLINK-6419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989600#comment-15989600
]
ASF GitHub Bot commented on FLINK-6419:
---
Github user eliaslevy commented on the issue:
https://github.com/apache/flink/pull/3477
I created [FLINK-6419](https://issues.apache.org/jira/browse/FLINK-6419).
> Better support for CEP quantified conditions in PatternSelect.select
>
>
> Key: FLINK-6419
> URL: https://issues.apache.org/jira/browse/FLINK-6419
> Project: Flink
> Issue Type: Improvement
> Components: CEP
>Affects Versions: 1.3.0
>Reporter: Elias Levy
>Priority: Minor
>
> Flink 1.3 introduces to the API quantifer methods which allow one to
> declaratively specific how many times a condition must be matched before
> there is a state change.
> The pre-existing {{PatternSelect.select}} method does not account for this
> change very well. The selection function passed to {{select}} receives a
> {{Map[String,T]}} as an argument that permits the function to look up the
> matched events by the condition's name.
> To support the new functionality that permits a condition to match multiple
> elements, when a quantifier is greater than one, the matched events are
> stored in the map by appending the condition's name with an underscore and an
> index value.
> While functional, this is less than ideal. It would be best if conditions
> with quantifier that is a multiple returned the matched events in an array
> and if they were accessible via the condition's name, without have to
> construct keys from the condition's name and an index, and iterate querying
> the map until no more are found.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink issue #3477: [Flink-3318][cep] Add support for quantifiers to CEP's pa...
Github user eliaslevy commented on the issue: https://github.com/apache/flink/pull/3477 I created [FLINK-6419](https://issues.apache.org/jira/browse/FLINK-6419). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-6419) Better support for CEP quantified conditions in PatternSelect.select
Elias Levy created FLINK-6419:
-
Summary: Better support for CEP quantified conditions in
PatternSelect.select
Key: FLINK-6419
URL: https://issues.apache.org/jira/browse/FLINK-6419
Project: Flink
Issue Type: Improvement
Components: CEP
Affects Versions: 1.3.0
Reporter: Elias Levy
Priority: Minor
Flink 1.3 introduces to the API quantifer methods which allow one to
declaratively specific how many times a condition must be matched before there
is a state change.
The pre-existing {{PatternSelect.select}} method does not account for this
change very well. The selection function passed to {{select}} receives a
{{Map[String,T]}} as an argument that permits the function to look up the
matched events by the condition's name.
To support the new functionality that permits a condition to match multiple
elements, when a quantifier is greater than one, the matched events are stored
in the map by appending the condition's name with an underscore and an index
value.
While functional, this is less than ideal. It would be best if conditions with
quantifier that is a multiple returned the matched events in an array and if
they were accessible via the condition's name, without have to construct keys
from the condition's name and an index, and iterate querying the map until no
more are found.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (FLINK-6059) Reject DataSet and DataStream without RowTypeInformation
[
https://issues.apache.org/jira/browse/FLINK-6059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Hueske closed FLINK-6059.
Resolution: Fixed
Fix Version/s: 1.2.2
1.3.0
Fixed for 1.3.0 with c8eb55f17d64722bb600c1083a478ab99e53f4ec
Fixed for 1.2.2 with fdb3f65f2d6595b88edae849ae6c848e5bbfaa2d
> Reject DataSet and DataStream without RowTypeInformation
> --
>
> Key: FLINK-6059
> URL: https://issues.apache.org/jira/browse/FLINK-6059
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
> Fix For: 1.3.0, 1.2.2
>
>
> It is not possible to automatically extract proper type information for
> {{Row}} because it is not typed with generics and holds values in an
> {{Object[]}}.
> Consequently is handled as {{GenericType}} unless a {{RowTypeInfo}} is
> explicitly specified.
> This can lead to unexpected behavior when converting a {{DataSet}} or
> {{DataStream}} into a Table. If the data set or data stream has a
> {{GenericType}}, the rows are treated as atomic type and converted into
> a single field.
> I think we should reject input types of {{GenericType}} when converting
> data sets and data streams and request a proper {{RowTypeInfo}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-6418) Support for dynamic state changes in CEP patterns
Elias Levy created FLINK-6418:
-
Summary: Support for dynamic state changes in CEP patterns
Key: FLINK-6418
URL: https://issues.apache.org/jira/browse/FLINK-6418
Project: Flink
Issue Type: Improvement
Components: CEP
Affects Versions: 1.3.0
Reporter: Elias Levy
Flink CEP library allows one to define event pattern to match where the match
condition can be determined programmatically via the {{where}} method. Flink
1.3 will introduce so-called iterative conditions, which allow the predicate to
look up events already matched by the pattern and thus be conditional on them.
1.3 also introduces to the API quantifer methods which allow one to
declaratively specific how many times a condition must be matched before there
is a state change.
Alas, there are use cases where the quantifier must be determined dynamically
based on the events matched by the pattern so far. Therefore, I propose the
adding of a new {{Pattern}}: {{until}}.
Like the new iterative variant of {{where}}, {{until}} would take a predicate
function and a context that provides access to events already matched. But
whereas {{where}} determines if an event is accepted by the pattern, {{until}}
determines whether is pattern should move on to the next state.
In our particular use case, we have a pattern where an event is matched a
number of times, but depending on the event type, the number (threshold) for
the pattern to match is different. We could decompose the pattern into
multiple similar patterns, but that could be inefficient if we have many such
patterns. If the functionality of {{until}} were available, we could make do
with a single pattern.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink pull request #3546: [FLINK-6059] [table] Reject GenericType when ...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3546 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6059) Reject DataSet and DataStream without RowTypeInformation
[
https://issues.apache.org/jira/browse/FLINK-6059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989564#comment-15989564
]
ASF GitHub Bot commented on FLINK-6059:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/3546
> Reject DataSet and DataStream without RowTypeInformation
> --
>
> Key: FLINK-6059
> URL: https://issues.apache.org/jira/browse/FLINK-6059
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
>
> It is not possible to automatically extract proper type information for
> {{Row}} because it is not typed with generics and holds values in an
> {{Object[]}}.
> Consequently is handled as {{GenericType}} unless a {{RowTypeInfo}} is
> explicitly specified.
> This can lead to unexpected behavior when converting a {{DataSet}} or
> {{DataStream}} into a Table. If the data set or data stream has a
> {{GenericType}}, the rows are treated as atomic type and converted into
> a single field.
> I think we should reject input types of {{GenericType}} when converting
> data sets and data streams and request a proper {{RowTypeInfo}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6382) Support all numeric types for generated graphs in Gelly examples
[
https://issues.apache.org/jira/browse/FLINK-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989537#comment-15989537
]
ASF GitHub Bot commented on FLINK-6382:
---
Github user greghogan commented on a diff in the pull request:
https://github.com/apache/flink/pull/3779#discussion_r114030670
--- Diff:
flink-libraries/flink-gelly-examples/src/test/java/org/apache/flink/graph/drivers/DriverBaseITCase.java
---
@@ -41,17 +43,41 @@
@Rule
public ExpectedException expectedException = ExpectedException.none();
- protected DriverBaseITCase(TestExecutionMode mode) {
+ protected final String idType;
+
+ protected DriverBaseITCase(TestExecutionMode mode, String idType) {
super(mode);
+
+ this.idType = idType;
}
- // extend MultipleProgramsTestBase default to include object reuse mode
- @Parameterized.Parameters(name = "Execution mode = {0}")
+ @Parameterized.Parameters(name = "Execution mode = {0}, ID type = {1}")
public static Collection executionModes() {
- return Arrays.asList(
- new Object[] { TestExecutionMode.CLUSTER },
- new Object[] { TestExecutionMode.CLUSTER_OBJECT_REUSE },
- new Object[] { TestExecutionMode.COLLECTION });
+ List executionModes = new ArrayList<>();
+
+ for (TestExecutionMode executionMode :
TestExecutionMode.values()) {
+ for (String idType : new String[] {"byte",
"nativeByte", "short", "nativeShort", "char", "nativeChar",
+
"integer", "nativeInteger", "long", "nativeLong", "string", "nativeString"}) {
--- End diff --
I've also changed modified the parameters string to print the ID type
before the ExecutionMode.
> Support all numeric types for generated graphs in Gelly examples
>
>
> Key: FLINK-6382
> URL: https://issues.apache.org/jira/browse/FLINK-6382
> Project: Flink
> Issue Type: Improvement
> Components: Gelly
>Affects Versions: 1.3.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
> Fix For: 1.3.0
>
>
> The Gelly examples current support {{IntValue}}, {{LongValue}}, and
> {{StringValue}} for {{RMatGraph}}. Allow transformations and tests for all
> generated graphs for {{ByteValue}}, {{Byte}}, {{ShortValue}}, {{Short}},
> {{CharValue}}, {{Character}}, {{Integer}}, {{Long}}, and {{String}}.
> This is additionally of interest for benchmarking and testing modifications
> to Flink's internal sort.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink pull request #3779: [FLINK-6382] [gelly] Support all numeric types for...
Github user greghogan commented on a diff in the pull request:
https://github.com/apache/flink/pull/3779#discussion_r114030670
--- Diff:
flink-libraries/flink-gelly-examples/src/test/java/org/apache/flink/graph/drivers/DriverBaseITCase.java
---
@@ -41,17 +43,41 @@
@Rule
public ExpectedException expectedException = ExpectedException.none();
- protected DriverBaseITCase(TestExecutionMode mode) {
+ protected final String idType;
+
+ protected DriverBaseITCase(TestExecutionMode mode, String idType) {
super(mode);
+
+ this.idType = idType;
}
- // extend MultipleProgramsTestBase default to include object reuse mode
- @Parameterized.Parameters(name = "Execution mode = {0}")
+ @Parameterized.Parameters(name = "Execution mode = {0}, ID type = {1}")
public static Collection executionModes() {
- return Arrays.asList(
- new Object[] { TestExecutionMode.CLUSTER },
- new Object[] { TestExecutionMode.CLUSTER_OBJECT_REUSE },
- new Object[] { TestExecutionMode.COLLECTION });
+ List executionModes = new ArrayList<>();
+
+ for (TestExecutionMode executionMode :
TestExecutionMode.values()) {
+ for (String idType : new String[] {"byte",
"nativeByte", "short", "nativeShort", "char", "nativeChar",
+
"integer", "nativeInteger", "long", "nativeLong", "string", "nativeString"}) {
--- End diff --
I've also changed modified the parameters string to print the ID type
before the ExecutionMode.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-6382) Support all numeric types for generated graphs in Gelly examples
[
https://issues.apache.org/jira/browse/FLINK-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989535#comment-15989535
]
ASF GitHub Bot commented on FLINK-6382:
---
Github user greghogan commented on a diff in the pull request:
https://github.com/apache/flink/pull/3779#discussion_r114030564
--- Diff:
flink-libraries/flink-gelly-examples/src/main/java/org/apache/flink/graph/drivers/input/GridGraph.java
---
@@ -82,11 +81,27 @@ public void configure(ParameterTool parameterTool)
throws ProgramParametrization
@Override
public String getIdentity() {
- return getName() + " (" + dimensions + ")";
+ return getTypeName() + " " + getName() + " (" + dimensions +
")";
}
@Override
- public Graph
create(ExecutionEnvironment env) {
+ protected long vertexCount() {
+ // in Java 8 use Math.multiplyExact(long, long)
--- End diff --
Rather than the current implementation using `BigInteger`, Java 8's
`Math.multiplyExact` detects and throws an exception on overflow. I was noting
for the future when the Flink codebase targets Java 8.
> Support all numeric types for generated graphs in Gelly examples
>
>
> Key: FLINK-6382
> URL: https://issues.apache.org/jira/browse/FLINK-6382
> Project: Flink
> Issue Type: Improvement
> Components: Gelly
>Affects Versions: 1.3.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
> Fix For: 1.3.0
>
>
> The Gelly examples current support {{IntValue}}, {{LongValue}}, and
> {{StringValue}} for {{RMatGraph}}. Allow transformations and tests for all
> generated graphs for {{ByteValue}}, {{Byte}}, {{ShortValue}}, {{Short}},
> {{CharValue}}, {{Character}}, {{Integer}}, {{Long}}, and {{String}}.
> This is additionally of interest for benchmarking and testing modifications
> to Flink's internal sort.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink pull request #3779: [FLINK-6382] [gelly] Support all numeric types for...
Github user greghogan commented on a diff in the pull request:
https://github.com/apache/flink/pull/3779#discussion_r114030564
--- Diff:
flink-libraries/flink-gelly-examples/src/main/java/org/apache/flink/graph/drivers/input/GridGraph.java
---
@@ -82,11 +81,27 @@ public void configure(ParameterTool parameterTool)
throws ProgramParametrization
@Override
public String getIdentity() {
- return getName() + " (" + dimensions + ")";
+ return getTypeName() + " " + getName() + " (" + dimensions +
")";
}
@Override
- public Graph
create(ExecutionEnvironment env) {
+ protected long vertexCount() {
+ // in Java 8 use Math.multiplyExact(long, long)
--- End diff --
Rather than the current implementation using `BigInteger`, Java 8's
`Math.multiplyExact` detects and throws an exception on overflow. I was noting
for the future when the Flink codebase targets Java 8.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-6409) TUMBLE/HOP/SESSION_START/END do not resolve time field correctly
[ https://issues.apache.org/jira/browse/FLINK-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989518#comment-15989518 ] ASF GitHub Bot commented on FLINK-6409: --- Github user fhueske commented on the issue: https://github.com/apache/flink/pull/3799 Thanks @twalthr for taking care of this! Most of the files have been copied from the Calcite master, so I won't review those. The other changes look good. +1 to merge this PR. @twalthr, can you also create a JIRA to remove the files when Calcite 1.13 is released? > TUMBLE/HOP/SESSION_START/END do not resolve time field correctly > > > Key: FLINK-6409 > URL: https://issues.apache.org/jira/browse/FLINK-6409 > Project: Flink > Issue Type: Bug > Components: Table API & SQL >Reporter: Timo Walther >Assignee: Timo Walther > > Calcite has a bug and cannot resolve the time fields of auxiliary group > functions correctly. A discussion can be found in CALCITE-1761. > Right now this issue only affects our batch SQL API, but it is a blocker for > FLINK-5884. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] flink issue #3799: [FLINK-6409] [table] TUMBLE/HOP/SESSION_START/END do not ...
Github user fhueske commented on the issue: https://github.com/apache/flink/pull/3799 Thanks @twalthr for taking care of this! Most of the files have been copied from the Calcite master, so I won't review those. The other changes look good. +1 to merge this PR. @twalthr, can you also create a JIRA to remove the files when Calcite 1.13 is released? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6387) Flink UI support access log
[
https://issues.apache.org/jira/browse/FLINK-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989433#comment-15989433
]
ASF GitHub Bot commented on FLINK-6387:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3777#discussion_r114015255
--- Diff:
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/HttpRequestHandler.java
---
@@ -197,17 +197,16 @@ else if (currentDecoder != null && msg instanceof
HttpContent) {
/**
* Record the access log if enable configure of
- * {@link
org.apache.flink.configuration.ConfigConstants#JOB_MANAGER_WEB_ACCESSLOG_ENABLE}.
+ * {@link
org.apache.flink.configuration.JobManagerOptions#JOB_MANAGER_WEB_ACCESSLOG_ENABLE}.
* record format:
* remote_addr - [time_local] "request_method URI protocolVersion"
"http_referer" "http_user_agent"
*/
- private void accesslog(ChannelHandlerContext ctx, HttpRequest req) {
+ private void logAccess(ChannelHandlerContext ctx, HttpRequest req) {
HttpHeaders headers = req.headers();
if (headers != null) {
- String line = ctx.channel().remoteAddress() + " - [" +
new Date() + "] \""
+ LOG.info(ctx.channel().remoteAddress() + " - [" + new
Date() + "] \""
--- End diff --
Please rework this to use placeholders instead.
> Flink UI support access log
> ---
>
> Key: FLINK-6387
> URL: https://issues.apache.org/jira/browse/FLINK-6387
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: shijinkui
>Assignee: shijinkui
>
> Record the use request to the access log. Append use access to the log file.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink pull request #3777: [FLINK-6387] [webfrontend]Flink UI support access ...
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3777#discussion_r114015255
--- Diff:
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/HttpRequestHandler.java
---
@@ -197,17 +197,16 @@ else if (currentDecoder != null && msg instanceof
HttpContent) {
/**
* Record the access log if enable configure of
- * {@link
org.apache.flink.configuration.ConfigConstants#JOB_MANAGER_WEB_ACCESSLOG_ENABLE}.
+ * {@link
org.apache.flink.configuration.JobManagerOptions#JOB_MANAGER_WEB_ACCESSLOG_ENABLE}.
* record format:
* remote_addr - [time_local] "request_method URI protocolVersion"
"http_referer" "http_user_agent"
*/
- private void accesslog(ChannelHandlerContext ctx, HttpRequest req) {
+ private void logAccess(ChannelHandlerContext ctx, HttpRequest req) {
HttpHeaders headers = req.headers();
if (headers != null) {
- String line = ctx.channel().remoteAddress() + " - [" +
new Date() + "] \""
+ LOG.info(ctx.channel().remoteAddress() + " - [" + new
Date() + "] \""
--- End diff --
Please rework this to use placeholders instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-6382) Support all numeric types for generated graphs in Gelly examples
[
https://issues.apache.org/jira/browse/FLINK-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989407#comment-15989407
]
ASF GitHub Bot commented on FLINK-6382:
---
Github user heytitle commented on a diff in the pull request:
https://github.com/apache/flink/pull/3779#discussion_r114009718
--- Diff:
flink-libraries/flink-gelly-examples/src/test/java/org/apache/flink/graph/drivers/DriverBaseITCase.java
---
@@ -41,17 +43,41 @@
@Rule
public ExpectedException expectedException = ExpectedException.none();
- protected DriverBaseITCase(TestExecutionMode mode) {
+ protected final String idType;
+
+ protected DriverBaseITCase(TestExecutionMode mode, String idType) {
super(mode);
+
+ this.idType = idType;
}
- // extend MultipleProgramsTestBase default to include object reuse mode
- @Parameterized.Parameters(name = "Execution mode = {0}")
+ @Parameterized.Parameters(name = "Execution mode = {0}, ID type = {1}")
public static Collection executionModes() {
- return Arrays.asList(
- new Object[] { TestExecutionMode.CLUSTER },
- new Object[] { TestExecutionMode.CLUSTER_OBJECT_REUSE },
- new Object[] { TestExecutionMode.COLLECTION });
+ List executionModes = new ArrayList<>();
+
+ for (TestExecutionMode executionMode :
TestExecutionMode.values()) {
+ for (String idType : new String[] {"byte",
"nativeByte", "short", "nativeShort", "char", "nativeChar",
+
"integer", "nativeInteger", "long", "nativeLong", "string", "nativeString"}) {
--- End diff --
What do you think if we declare the list of `idType` before the outer loop?
IMHO, this will make formation better.
> Support all numeric types for generated graphs in Gelly examples
>
>
> Key: FLINK-6382
> URL: https://issues.apache.org/jira/browse/FLINK-6382
> Project: Flink
> Issue Type: Improvement
> Components: Gelly
>Affects Versions: 1.3.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
> Fix For: 1.3.0
>
>
> The Gelly examples current support {{IntValue}}, {{LongValue}}, and
> {{StringValue}} for {{RMatGraph}}. Allow transformations and tests for all
> generated graphs for {{ByteValue}}, {{Byte}}, {{ShortValue}}, {{Short}},
> {{CharValue}}, {{Character}}, {{Integer}}, {{Long}}, and {{String}}.
> This is additionally of interest for benchmarking and testing modifications
> to Flink's internal sort.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6382) Support all numeric types for generated graphs in Gelly examples
[
https://issues.apache.org/jira/browse/FLINK-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989409#comment-15989409
]
ASF GitHub Bot commented on FLINK-6382:
---
Github user heytitle commented on a diff in the pull request:
https://github.com/apache/flink/pull/3779#discussion_r114010412
--- Diff:
flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/library/link_analysis/HITS.java
---
@@ -60,8 +60,6 @@
*
* http://www.cs.cornell.edu/home/kleinber/auth.pdf
*
- * http://www.cs.cornell.edu/home/kleinber/auth.pdf
- *
--- End diff --
Nice catch!
> Support all numeric types for generated graphs in Gelly examples
>
>
> Key: FLINK-6382
> URL: https://issues.apache.org/jira/browse/FLINK-6382
> Project: Flink
> Issue Type: Improvement
> Components: Gelly
>Affects Versions: 1.3.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
> Fix For: 1.3.0
>
>
> The Gelly examples current support {{IntValue}}, {{LongValue}}, and
> {{StringValue}} for {{RMatGraph}}. Allow transformations and tests for all
> generated graphs for {{ByteValue}}, {{Byte}}, {{ShortValue}}, {{Short}},
> {{CharValue}}, {{Character}}, {{Integer}}, {{Long}}, and {{String}}.
> This is additionally of interest for benchmarking and testing modifications
> to Flink's internal sort.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6382) Support all numeric types for generated graphs in Gelly examples
[
https://issues.apache.org/jira/browse/FLINK-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989408#comment-15989408
]
ASF GitHub Bot commented on FLINK-6382:
---
Github user heytitle commented on a diff in the pull request:
https://github.com/apache/flink/pull/3779#discussion_r114007404
--- Diff:
flink-libraries/flink-gelly-examples/src/main/java/org/apache/flink/graph/drivers/input/GridGraph.java
---
@@ -82,11 +81,27 @@ public void configure(ParameterTool parameterTool)
throws ProgramParametrization
@Override
public String getIdentity() {
- return getName() + " (" + dimensions + ")";
+ return getTypeName() + " " + getName() + " (" + dimensions +
")";
}
@Override
- public Graph
create(ExecutionEnvironment env) {
+ protected long vertexCount() {
+ // in Java 8 use Math.multiplyExact(long, long)
--- End diff --
What is the purpose of this comment?
> Support all numeric types for generated graphs in Gelly examples
>
>
> Key: FLINK-6382
> URL: https://issues.apache.org/jira/browse/FLINK-6382
> Project: Flink
> Issue Type: Improvement
> Components: Gelly
>Affects Versions: 1.3.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
> Fix For: 1.3.0
>
>
> The Gelly examples current support {{IntValue}}, {{LongValue}}, and
> {{StringValue}} for {{RMatGraph}}. Allow transformations and tests for all
> generated graphs for {{ByteValue}}, {{Byte}}, {{ShortValue}}, {{Short}},
> {{CharValue}}, {{Character}}, {{Integer}}, {{Long}}, and {{String}}.
> This is additionally of interest for benchmarking and testing modifications
> to Flink's internal sort.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink pull request #3779: [FLINK-6382] [gelly] Support all numeric types for...
Github user heytitle commented on a diff in the pull request:
https://github.com/apache/flink/pull/3779#discussion_r114009718
--- Diff:
flink-libraries/flink-gelly-examples/src/test/java/org/apache/flink/graph/drivers/DriverBaseITCase.java
---
@@ -41,17 +43,41 @@
@Rule
public ExpectedException expectedException = ExpectedException.none();
- protected DriverBaseITCase(TestExecutionMode mode) {
+ protected final String idType;
+
+ protected DriverBaseITCase(TestExecutionMode mode, String idType) {
super(mode);
+
+ this.idType = idType;
}
- // extend MultipleProgramsTestBase default to include object reuse mode
- @Parameterized.Parameters(name = "Execution mode = {0}")
+ @Parameterized.Parameters(name = "Execution mode = {0}, ID type = {1}")
public static Collection executionModes() {
- return Arrays.asList(
- new Object[] { TestExecutionMode.CLUSTER },
- new Object[] { TestExecutionMode.CLUSTER_OBJECT_REUSE },
- new Object[] { TestExecutionMode.COLLECTION });
+ List executionModes = new ArrayList<>();
+
+ for (TestExecutionMode executionMode :
TestExecutionMode.values()) {
+ for (String idType : new String[] {"byte",
"nativeByte", "short", "nativeShort", "char", "nativeChar",
+
"integer", "nativeInteger", "long", "nativeLong", "string", "nativeString"}) {
--- End diff --
What do you think if we declare the list of `idType` before the outer loop?
IMHO, this will make formation better.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3779: [FLINK-6382] [gelly] Support all numeric types for...
Github user heytitle commented on a diff in the pull request: https://github.com/apache/flink/pull/3779#discussion_r114010412 --- Diff: flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/library/link_analysis/HITS.java --- @@ -60,8 +60,6 @@ * * http://www.cs.cornell.edu/home/kleinber/auth.pdf * - * http://www.cs.cornell.edu/home/kleinber/auth.pdf - * --- End diff -- Nice catch! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3779: [FLINK-6382] [gelly] Support all numeric types for...
Github user heytitle commented on a diff in the pull request:
https://github.com/apache/flink/pull/3779#discussion_r114007404
--- Diff:
flink-libraries/flink-gelly-examples/src/main/java/org/apache/flink/graph/drivers/input/GridGraph.java
---
@@ -82,11 +81,27 @@ public void configure(ParameterTool parameterTool)
throws ProgramParametrization
@Override
public String getIdentity() {
- return getName() + " (" + dimensions + ")";
+ return getTypeName() + " " + getName() + " (" + dimensions +
")";
}
@Override
- public Graph
create(ExecutionEnvironment env) {
+ protected long vertexCount() {
+ // in Java 8 use Math.multiplyExact(long, long)
--- End diff --
What is the purpose of this comment?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989382#comment-15989382 ] ASF GitHub Bot commented on FLINK-6013: --- Github user bowenli86 commented on the issue: https://github.com/apache/flink/pull/3736 @StephanEwen @zentol I've made the following changes: 1) Datadog (DD) itself has a bug of being unstable when users filter metrics by 'host' in dashboards if 'host' is sent as tags, details in [FLINK-6013](https://issues.apache.org/jira/browse/FLINK-6013). Communicated with DD engineers, they fixed the issue, and now, as advised by them, I'm making 'host' a separate field in metric serialization. Tested the new approach successfully, and we are able to filter metrics by 'host' consistently correctly on DD dashboard. 2) Added more unit tests for the above change. Unit tests cover both cases when 'host' is present in MetricGroup and when it isn't 3) Found Maven can only recognize and run unit tests when file name ends with 'Test' rather than 'Tests'. So I renamed my test file to 'DatadogHttpClientTest' from 'DatadogHttpClientTests' 4) Upgraded okhttp and okio to newer versions 5) Found mocking system millisec will impact the new okhttp. So I separated unit tests to two enclosed test sets in 'DatadogHttpClientTest' Having successfully tested this across my company in the past couple days, I'm now pretty confident this is production ready. I agree on @zentol 's proposal if you guys are currently busy with other things, as long as we make sure this ends up in 1.3 :) Thanks, guys! > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter
Github user bowenli86 commented on the issue: https://github.com/apache/flink/pull/3736 @StephanEwen @zentol I've made the following changes: 1) Datadog (DD) itself has a bug of being unstable when users filter metrics by 'host' in dashboards if 'host' is sent as tags, details in [FLINK-6013](https://issues.apache.org/jira/browse/FLINK-6013). Communicated with DD engineers, they fixed the issue, and now, as advised by them, I'm making 'host' a separate field in metric serialization. Tested the new approach successfully, and we are able to filter metrics by 'host' consistently correctly on DD dashboard. 2) Added more unit tests for the above change. Unit tests cover both cases when 'host' is present in MetricGroup and when it isn't 3) Found Maven can only recognize and run unit tests when file name ends with 'Test' rather than 'Tests'. So I renamed my test file to 'DatadogHttpClientTest' from 'DatadogHttpClientTests' 4) Upgraded okhttp and okio to newer versions 5) Found mocking system millisec will impact the new okhttp. So I separated unit tests to two enclosed test sets in 'DatadogHttpClientTest' Having successfully tested this across my company in the past couple days, I'm now pretty confident this is production ready. I agree on @zentol 's proposal if you guys are currently busy with other things, as long as we make sure this ends up in 1.3 :) Thanks, guys! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Closed] (FLINK-5892) Recover job state at the granularity of operator
[ https://issues.apache.org/jira/browse/FLINK-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-5892. --- Resolution: Fixed Fix Version/s: 1.3.0 Implemented in 8045fabac736cc8c6b48fda8328cf91f329dc3bf, f7980a7e29457753eb3c5b975f3bb4b59d2014f8 and 2c68085f658873c2d5836fbad6b82be76a79f0f9. > Recover job state at the granularity of operator > > > Key: FLINK-5892 > URL: https://issues.apache.org/jira/browse/FLINK-5892 > Project: Flink > Issue Type: New Feature > Components: State Backends, Checkpointing >Reporter: Guowei Ma >Assignee: Guowei Ma > Fix For: 1.3.0 > > > JobGraph has no `Operator` info so `ExecutionGraph` can only recovery at the > granularity of task. > This leads to the result that the operator of the job may not recover the > state from a save point even if the save point has the state of operator. > > https://docs.google.com/document/d/19suTRF0nh7pRgeMnIEIdJ2Fq-CcNVt5_hR3cAoICf7Q/edit#. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-5892) Recover job state at the granularity of operator
[ https://issues.apache.org/jira/browse/FLINK-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989374#comment-15989374 ] ASF GitHub Bot commented on FLINK-5892: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3770 > Recover job state at the granularity of operator > > > Key: FLINK-5892 > URL: https://issues.apache.org/jira/browse/FLINK-5892 > Project: Flink > Issue Type: New Feature > Components: State Backends, Checkpointing >Reporter: Guowei Ma >Assignee: Guowei Ma > Fix For: 1.3.0 > > > JobGraph has no `Operator` info so `ExecutionGraph` can only recovery at the > granularity of task. > This leads to the result that the operator of the job may not recover the > state from a save point even if the save point has the state of operator. > > https://docs.google.com/document/d/19suTRF0nh7pRgeMnIEIdJ2Fq-CcNVt5_hR3cAoICf7Q/edit#. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] flink pull request #3770: [FLINK-5892] Restore state on the operator level
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3770 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6414) Use scala.binary.version in place of change-scala-version.sh
[
https://issues.apache.org/jira/browse/FLINK-6414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989323#comment-15989323
]
ASF GitHub Bot commented on FLINK-6414:
---
GitHub user greghogan opened a pull request:
https://github.com/apache/flink/pull/3800
[FLINK-6414] [build] Use scala.binary.version in place of
change-scala-version.sh
Use scala.binary.version as defined in the parent POM and remove the script
to swap scala version identifiers.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/greghogan/flink
6414_use_scala_binary_version_in_place_of_change_scala_version_sh
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3800.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3800
commit 6d0ca2f9ddf3d33f9f2953242234e5e053cb00b5
Author: Greg Hogan
Date: 2017-04-27T18:43:18Z
[FLINK-6414] [build] Use scala.binary.version in place of
change-scala-version.sh
Use scala.binary.version as defined in the parent POM and remove the
script to swap scala version identifiers.
> Use scala.binary.version in place of change-scala-version.sh
>
>
> Key: FLINK-6414
> URL: https://issues.apache.org/jira/browse/FLINK-6414
> Project: Flink
> Issue Type: Improvement
> Components: Build System
>Affects Versions: 1.3.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>
> Recent commits have failed to modify {{change-scala-version.sh}} resulting in
> broken builds for {{scala-2.11}}. It looks like we can remove the need for
> this script by replacing hard-coded references to the Scala version with
> Flink's maven variable {{scala.binary.version}}.
> I had initially realized that the change script is [only used for
> building|https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/building.html#scala-versions]
> and not for switching the IDE environment.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink pull request #3800: [FLINK-6414] [build] Use scala.binary.version in p...
GitHub user greghogan opened a pull request: https://github.com/apache/flink/pull/3800 [FLINK-6414] [build] Use scala.binary.version in place of change-scala-version.sh Use scala.binary.version as defined in the parent POM and remove the script to swap scala version identifiers. You can merge this pull request into a Git repository by running: $ git pull https://github.com/greghogan/flink 6414_use_scala_binary_version_in_place_of_change_scala_version_sh Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3800.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3800 commit 6d0ca2f9ddf3d33f9f2953242234e5e053cb00b5 Author: Greg Hogan Date: 2017-04-27T18:43:18Z [FLINK-6414] [build] Use scala.binary.version in place of change-scala-version.sh Use scala.binary.version as defined in the parent POM and remove the script to swap scala version identifiers. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5810) Harden SlotManager
[
https://issues.apache.org/jira/browse/FLINK-5810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989303#comment-15989303
]
Ted Yu commented on FLINK-5810:
---
In SlotManager#suspend() :
{code}
+ // stop the timeout checks for the TaskManagers and the
SlotRequests
+ taskManagerTimeoutCheck.cancel(false);
+ slotRequestTimeoutCheck.cancel(false);
+
+ taskManagerTimeoutCheck = null;
+ slotRequestTimeoutCheck = null;
{code}
May I ask why false is passed to cancel() ?
Should we care about the possibility that checker(s) keeps running ?
> Harden SlotManager
> --
>
> Key: FLINK-5810
> URL: https://issues.apache.org/jira/browse/FLINK-5810
> Project: Flink
> Issue Type: Sub-task
> Components: Distributed Coordination
>Affects Versions: 1.3.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Fix For: 1.3.0
>
>
> Harden the {{SlotManager}} logic to better cope with lost messages.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6393) Add Evenly Graph Generator to Flink Gelly
[ https://issues.apache.org/jira/browse/FLINK-6393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989289#comment-15989289 ] ASF GitHub Bot commented on FLINK-6393: --- Github user greghogan commented on the issue: https://github.com/apache/flink/pull/3788 @fanzhidongyzby that sounds good except I would leave the `EmptyGraph` as it is. In addition we'll want to add the new generators as driver inputs in `flink-gelly-examples`, add to the list of inputs in `Runner`, and add a test each to `EdgeListITCase` (run the test to compute the checksum to create the test string). Also, on the graph name, without a reference to the literature, perhaps this could be something invoking the idea that vertices are connected by length-1 paths to "far" vertices, by length-2 paths to "near" vertices, by length-3 paths to "far" vertices, etc. Something like `EchoGraph`. > Add Evenly Graph Generator to Flink Gelly > - > > Key: FLINK-6393 > URL: https://issues.apache.org/jira/browse/FLINK-6393 > Project: Flink > Issue Type: New Feature > Components: Gelly >Reporter: FlorianFan >Assignee: FlorianFan >Priority: Minor > Original Estimate: 24h > Remaining Estimate: 24h > > Evenly graph means every vertex in the graph has the same degree, so the > graph can be treated as evenly due to all the edges in the graph are > distributed evenly. when vertex degree is 0, an empty graph will be > generated. when vertex degree is vertex count - 1, complete graph will be > generated. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] flink issue #3788: [FLINK-6393] [gelly] Add Evenly Graph Generator to Flink ...
Github user greghogan commented on the issue: https://github.com/apache/flink/pull/3788 @fanzhidongyzby that sounds good except I would leave the `EmptyGraph` as it is. In addition we'll want to add the new generators as driver inputs in `flink-gelly-examples`, add to the list of inputs in `Runner`, and add a test each to `EdgeListITCase` (run the test to compute the checksum to create the test string). Also, on the graph name, without a reference to the literature, perhaps this could be something invoking the idea that vertices are connected by length-1 paths to "far" vertices, by length-2 paths to "near" vertices, by length-3 paths to "far" vertices, etc. Something like `EchoGraph`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6416) Potential divide by zero issue in InputGateMetrics#refreshAndGetAvg()
[
https://issues.apache.org/jira/browse/FLINK-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989202#comment-15989202
]
Ted Yu commented on FLINK-6416:
---
I would think 0 is reasonable return value when both total and count are zero.
> Potential divide by zero issue in InputGateMetrics#refreshAndGetAvg()
> -
>
> Key: FLINK-6416
> URL: https://issues.apache.org/jira/browse/FLINK-6416
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> int count = 0;
> for (InputChannel channel : inputGate.getInputChannels().values()) {
> if (channel instanceof RemoteInputChannel) {
> RemoteInputChannel rc = (RemoteInputChannel) channel;
> int size = rc.unsynchronizedGetNumberOfQueuedBuffers();
> total += size;
> ++count;
> }
> }
> return total / (float) count;
> {code}
> If count is zero at the end of the loop, the division would produce exception.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6412) Stream has already been closed during job cancel
[
https://issues.apache.org/jira/browse/FLINK-6412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989189#comment-15989189
]
Stefan Richter commented on FLINK-6412:
---
I can probably fix this easily, suggest I wait until I merged the incremental
checkpointing PR.
> Stream has already been closed during job cancel
>
>
> Key: FLINK-6412
> URL: https://issues.apache.org/jira/browse/FLINK-6412
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Andrey
>
> Steps to reproduce:
> * create job with RocksDBStateBackend: env.setStateBackend(new
> RocksDBStateBackend(checkpointRoot));
> * run job
> * cancel job from the Web UI.
> In logs:
> {code}
> 2017-04-28 10:25:57,475 INFO org.apache.flink.runtime.taskmanager.Task
> - Source: Custom Source (1/2)
> (05a3888ed2a232f234a10775826395a0) switched from DEPLOYING to RUNNING.
> [Source: Custom Source (1/2)]
> 2017-04-28 10:25:57,476 INFO
> org.apache.flink.streaming.runtime.tasks.StreamTask - Using
> user-defined state backend: RocksDB State Backend {isInitialized=false,
> configuredDbBasePaths=null, initializedDbBasePaths=null,
> checkpointStreamBackend=File State Backend @
> file:/flink/checkpoints/flinktests}. [Map (2/2)]
> 2017-04-28 10:25:57,476 INFO
> org.apache.flink.streaming.runtime.tasks.StreamTask - Using
> user-defined state backend: RocksDB State Backend {isInitialized=false,
> configuredDbBasePaths=null, initializedDbBasePaths=null,
> checkpointStreamBackend=File State Backend @
> file:/flink/checkpoints/flinktests}. [Source: Custom Source (1/2)]
> ...
> 2017-04-28 10:26:29,793 INFO org.apache.flink.runtime.taskmanager.Task
> - Triggering cancellation of task code Source: Custom Source
> (1/2) (05a3888ed2a232f234a10775826395a0).
> [flink-akka.actor.default-dispatcher-2]
> 2017-04-28 10:26:29,794 INFO org.apache.flink.runtime.taskmanager.Task
> - Attempting to cancel task Map (2/2)
> (bdb982b6ef47fe79b6ff5b96153c921e). [flink-akka.actor.default-dispatcher-2]
> 2017-04-28 10:26:29,794 INFO org.apache.flink.runtime.taskmanager.Task
> - Map (2/2) (bdb982b6ef47fe79b6ff5b96153c921e) switched from
> RUNNING to CANCELING. [flink-akka.actor.default-dispatcher-2]
> 2017-04-28 10:26:29,796 INFO org.apache.flink.runtime.taskmanager.Task
> - Triggering cancellation of task code Map (2/2)
> (bdb982b6ef47fe79b6ff5b96153c921e). [flink-akka.actor.default-dispatcher-2]
> 2017-04-28 10:26:29,796 INFO org.apache.flink.runtime.taskmanager.Task
> - Source: Custom Source (1/2)
> (05a3888ed2a232f234a10775826395a0) switched from CANCELING to CANCELED.
> [Source: Custom Source (1/2)]
> 2017-04-28 10:26:29,797 INFO org.apache.flink.runtime.taskmanager.Task
> - Freeing task resources for Source: Custom Source (1/2)
> (05a3888ed2a232f234a10775826395a0). [Source: Custom Source (1/2)]
> 2017-04-28 10:26:29,798 INFO org.apache.flink.core.fs.FileSystem
> - Ensuring all FileSystem streams are closed for Source: Custom
> Source (1/2) [Source: Custom Source (1/2)]
> 2017-04-28 10:26:29,803 INFO
> org.apache.flink.runtime.taskmanager.TaskManager -
> Un-registering task and sending final execution state CANCELED to JobManager
> for task Source: Custom Source (05a3888ed2a232f234a10775826395a0)
> [flink-akka.actor.default-dispatcher-2]
> 2017-04-28 10:26:39,608 INFO org.apache.flink.runtime.taskmanager.Task
> - Attempting to fail task externally Map (2/2)
> (bdb982b6ef47fe79b6ff5b96153c921e). [pool-14-thread-1]
> 2017-04-28 10:26:39,608 WARN
> org.apache.flink.streaming.runtime.tasks.StreamTask - Could not
> properly clean up the async checkpoint runnable. [Canceler for Map (2/2)
> (bdb982b6ef47fe79b6ff5b96153c921e).]
> java.lang.Exception: Could not properly cancel managed keyed state future.
> at
> org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:91)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.cleanup(StreamTask.java:1010)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.close(StreamTask.java:995)
> at org.apache.flink.util.IOUtils.closeQuietly(IOUtils.java:262)
> at org.apache.flink.util.IOUtils.closeAllQuietly(IOUtils.java:251)
> at
> org.apache.flink.util.AbstractCloseableRegistry.close(AbstractCloseableRegistry.java:91)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.cancel(StreamTask.java:364)
> at
> org.apache.flink.runtime.taskmanager.Task$TaskCanceler.run(Task.java:1390)
> at java.lang.Thread.run(T
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989181#comment-15989181 ] Bowen Li commented on FLINK-6013: - [~StephanEwen] [~Zentol] While testing the reporter, I found Datadog is unstable in filtering metrics by host names. Contacted Datadog engineering team. They got back to me this morning saying they found the issue, and instead of putting host names as tags, Datadog opens a metric format with extra field of ‘host’. Example. Rather than "metric='my.series2', points=150, tags=["host:localhost, version:1"])", they changed to "metric='my.series', points=100, host="localhost", tags=["version:1"]" I'll make corresponding changes to my reporter > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (FLINK-6417) Wildcard support for read text file
[
https://issues.apache.org/jira/browse/FLINK-6417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Artiom Darie updated FLINK-6417:
Description:
Add wildcard support while reading from s3://, hdfs://, file://, etc.
h6. Examples:
# {code} s3://bucket-name/*.gz {code}
# {code} hdfs://path/*file-name*.csv {code}
# {code} file://tmp/**/*.* {code}
h6. Proposal
# Use the existing method: {code}environment.readFile(...){code}
# List all the files in the directories
# Read files using existing: {code}ContinuousFileReaderOperator{code}
h6. Concerns (Open for discussions)
# Have multiple DataSource(s) created for each each file and then to join them
into a single DataSource
# Have all the files into the same DataSource
# Have the listing of the files on the driver and load on each task manager
was:
Add wildcard support while reading from s3://, hdfs://, file://, etc.
h6. Examples:
# {code} s3://bucket-name/*.gz {code}
# {code} hdfs://path/*file-name*.csv {code}
# {code} file://tmp/**/*.* {code}
h6. Proposal
# Use the existing method: {code}environment.readFile(...){code}
# List all the files in the directories
# Read files using existing: {code}ContinuousFileReaderOperator{code}
h6. Concerns (Open for discussions)
# Have multiple DataSource(s) created for each each file and then to join them
into a single DataSource
# Have all the files into the same DataSource
> Wildcard support for read text file
> ---
>
> Key: FLINK-6417
> URL: https://issues.apache.org/jira/browse/FLINK-6417
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Artiom Darie
>Priority: Minor
>
> Add wildcard support while reading from s3://, hdfs://, file://, etc.
> h6. Examples:
> # {code} s3://bucket-name/*.gz {code}
> # {code} hdfs://path/*file-name*.csv {code}
> # {code} file://tmp/**/*.* {code}
> h6. Proposal
> # Use the existing method: {code}environment.readFile(...){code}
> # List all the files in the directories
> # Read files using existing: {code}ContinuousFileReaderOperator{code}
> h6. Concerns (Open for discussions)
> # Have multiple DataSource(s) created for each each file and then to join
> them into a single DataSource
> # Have all the files into the same DataSource
> # Have the listing of the files on the driver and load on each task manager
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6393) Add Evenly Graph Generator to Flink Gelly
[ https://issues.apache.org/jira/browse/FLINK-6393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989137#comment-15989137 ] ASF GitHub Bot commented on FLINK-6393: --- Github user fanzhidongyzby commented on the issue: https://github.com/apache/flink/pull/3788 @greghogan , I'm preparing to implement `CirculantGraph `, and rewrite `EmptyGraph`, `CompleteGraph` and `EvenlyGraph` with `CirculantGraph`, is this ok ? > Add Evenly Graph Generator to Flink Gelly > - > > Key: FLINK-6393 > URL: https://issues.apache.org/jira/browse/FLINK-6393 > Project: Flink > Issue Type: New Feature > Components: Gelly >Reporter: FlorianFan >Assignee: FlorianFan >Priority: Minor > Original Estimate: 24h > Remaining Estimate: 24h > > Evenly graph means every vertex in the graph has the same degree, so the > graph can be treated as evenly due to all the edges in the graph are > distributed evenly. when vertex degree is 0, an empty graph will be > generated. when vertex degree is vertex count - 1, complete graph will be > generated. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] flink issue #3788: [FLINK-6393] [gelly] Add Evenly Graph Generator to Flink ...
Github user fanzhidongyzby commented on the issue: https://github.com/apache/flink/pull/3788 @greghogan , I'm preparing to implement `CirculantGraph `, and rewrite `EmptyGraph`, `CompleteGraph` and `EvenlyGraph` with `CirculantGraph`, is this ok ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6414) Use scala.binary.version in place of change-scala-version.sh
[
https://issues.apache.org/jira/browse/FLINK-6414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989135#comment-15989135
]
Greg Hogan commented on FLINK-6414:
---
[~StephanEwen], [this|https://github.com/apache/flink/pull/885] is the genesis
for borrowing Spark's {{change-scala-version.sh}} and I'm not seeing the
referenced issues. Travis is passing and Maven is installing the artifacts
locally just fine. The one glitch I am seeing is these cases where
maven-remote-resources-plugin is downloading 2.10 resources for the 2.11 build:
{noformat}
[INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles)
@ flink-tests_2.11 ---
Downloading:
https://repository.apache.org/snapshots/org/apache/flink/flink-clients_2.10/1.3-SNAPSHOT/maven-metadata.xml
Downloaded:
https://repository.apache.org/snapshots/org/apache/flink/flink-clients_2.10/1.3-SNAPSHOT/maven-metadata.xml
(1.4 kB at 16 kB/s)
Downloading:
https://repository.apache.org/snapshots/org/apache/flink/flink-clients_2.10/1.3-SNAPSHOT/flink-clients_2.10-1.3-20170428.120310-111.pom
Downloaded:
https://repository.apache.org/snapshots/org/apache/flink/flink-clients_2.10/1.3-SNAPSHOT/flink-clients_2.10-1.3-20170428.120310-111.pom
(7.7 kB at 208 kB/s)
Downloading:
https://repository.apache.org/snapshots/org/apache/flink/flink-runtime_2.10/1.3-SNAPSHOT/maven-metadata.xml
Downloaded:
https://repository.apache.org/snapshots/org/apache/flink/flink-runtime_2.10/1.3-SNAPSHOT/maven-metadata.xml
(1.4 kB at 30 kB/s)
Downloading:
https://repository.apache.org/snapshots/org/apache/flink/flink-runtime_2.10/1.3-SNAPSHOT/flink-runtime_2.10-1.3-20170428.120229-111.pom
Downloaded:
https://repository.apache.org/snapshots/org/apache/flink/flink-runtime_2.10/1.3-SNAPSHOT/flink-runtime_2.10-1.3-20170428.120229-111.pom
(14 kB at 368 kB/s)
Downloading:
https://repository.apache.org/snapshots/org/apache/flink/flink-optimizer_2.10/1.3-SNAPSHOT/maven-metadata.xml
Downloaded:
https://repository.apache.org/snapshots/org/apache/flink/flink-optimizer_2.10/1.3-SNAPSHOT/maven-metadata.xml
(1.4 kB at 36 kB/s)
Downloading:
https://repository.apache.org/snapshots/org/apache/flink/flink-optimizer_2.10/1.3-SNAPSHOT/flink-optimizer_2.10-1.3-20170428.120253-111.pom
Downloaded:
https://repository.apache.org/snapshots/org/apache/flink/flink-optimizer_2.10/1.3-SNAPSHOT/flink-optimizer_2.10-1.3-20170428.120253-111.pom
(4.4 kB at 118 kB/s)
[INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles)
@ flink-examples-streaming_2.11 ---
Downloading:
https://repository.apache.org/snapshots/org/apache/flink/flink-optimizer_2.10/1.3-SNAPSHOT/flink-optimizer_2.10-1.3-20170428.120253-111.jar
Downloading:
https://repository.apache.org/snapshots/org/apache/flink/flink-runtime_2.10/1.3-SNAPSHOT/flink-runtime_2.10-1.3-20170428.120229-111.jar
Downloading:
https://repository.apache.org/snapshots/org/apache/flink/flink-clients_2.10/1.3-SNAPSHOT/flink-clients_2.10-1.3-20170428.120310-111.jar
Downloaded:
https://repository.apache.org/snapshots/org/apache/flink/flink-clients_2.10/1.3-SNAPSHOT/flink-clients_2.10-1.3-20170428.120310-111.jar
(94 kB at 1.5 MB/s)
Downloaded:
https://repository.apache.org/snapshots/org/apache/flink/flink-optimizer_2.10/1.3-SNAPSHOT/flink-optimizer_2.10-1.3-20170428.120253-111.jar
(2.7 MB at 14 MB/s)
Downloaded:
https://repository.apache.org/snapshots/org/apache/flink/flink-runtime_2.10/1.3-SNAPSHOT/flink-runtime_2.10-1.3-20170428.120229-111.jar
(6.6 MB at 16 MB/s)
{noformat}
On master this section is empty:
{noformat}
[INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles)
@ flink-tests_2.11 ---
{noformat}
> Use scala.binary.version in place of change-scala-version.sh
>
>
> Key: FLINK-6414
> URL: https://issues.apache.org/jira/browse/FLINK-6414
> Project: Flink
> Issue Type: Improvement
> Components: Build System
>Affects Versions: 1.3.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>
> Recent commits have failed to modify {{change-scala-version.sh}} resulting in
> broken builds for {{scala-2.11}}. It looks like we can remove the need for
> this script by replacing hard-coded references to the Scala version with
> Flink's maven variable {{scala.binary.version}}.
> I had initially realized that the change script is [only used for
> building|https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/building.html#scala-versions]
> and not for switching the IDE environment.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6412) Stream has already been closed during job cancel
[
https://issues.apache.org/jira/browse/FLINK-6412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989125#comment-15989125
]
Stefan Richter commented on FLINK-6412:
---
I think the root cause of this is a) the stream is registered in the
{{CloseableRegistry}}, which can close the stream to interrupt the async
operation and b) the stream is also closed (over some method calls) in a
finally clause at the end of the {{AbstractAsyncIOCallable::performOperation}}
implementation in the {{RocksDBKeyedStateBackend}}. So the flow is probably:
runnable is performing the checkpoint, cancel closes the output stream through
the {{CloseableRegistry}}, operation ends and tries to close the stream again
in the finally clause. This should probably be done without a finally clause,
additionally catching for exception and calling a standard close in the
exception case.
> Stream has already been closed during job cancel
>
>
> Key: FLINK-6412
> URL: https://issues.apache.org/jira/browse/FLINK-6412
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Andrey
>
> Steps to reproduce:
> * create job with RocksDBStateBackend: env.setStateBackend(new
> RocksDBStateBackend(checkpointRoot));
> * run job
> * cancel job from the Web UI.
> In logs:
> {code}
> 2017-04-28 10:25:57,475 INFO org.apache.flink.runtime.taskmanager.Task
> - Source: Custom Source (1/2)
> (05a3888ed2a232f234a10775826395a0) switched from DEPLOYING to RUNNING.
> [Source: Custom Source (1/2)]
> 2017-04-28 10:25:57,476 INFO
> org.apache.flink.streaming.runtime.tasks.StreamTask - Using
> user-defined state backend: RocksDB State Backend {isInitialized=false,
> configuredDbBasePaths=null, initializedDbBasePaths=null,
> checkpointStreamBackend=File State Backend @
> file:/flink/checkpoints/flinktests}. [Map (2/2)]
> 2017-04-28 10:25:57,476 INFO
> org.apache.flink.streaming.runtime.tasks.StreamTask - Using
> user-defined state backend: RocksDB State Backend {isInitialized=false,
> configuredDbBasePaths=null, initializedDbBasePaths=null,
> checkpointStreamBackend=File State Backend @
> file:/flink/checkpoints/flinktests}. [Source: Custom Source (1/2)]
> ...
> 2017-04-28 10:26:29,793 INFO org.apache.flink.runtime.taskmanager.Task
> - Triggering cancellation of task code Source: Custom Source
> (1/2) (05a3888ed2a232f234a10775826395a0).
> [flink-akka.actor.default-dispatcher-2]
> 2017-04-28 10:26:29,794 INFO org.apache.flink.runtime.taskmanager.Task
> - Attempting to cancel task Map (2/2)
> (bdb982b6ef47fe79b6ff5b96153c921e). [flink-akka.actor.default-dispatcher-2]
> 2017-04-28 10:26:29,794 INFO org.apache.flink.runtime.taskmanager.Task
> - Map (2/2) (bdb982b6ef47fe79b6ff5b96153c921e) switched from
> RUNNING to CANCELING. [flink-akka.actor.default-dispatcher-2]
> 2017-04-28 10:26:29,796 INFO org.apache.flink.runtime.taskmanager.Task
> - Triggering cancellation of task code Map (2/2)
> (bdb982b6ef47fe79b6ff5b96153c921e). [flink-akka.actor.default-dispatcher-2]
> 2017-04-28 10:26:29,796 INFO org.apache.flink.runtime.taskmanager.Task
> - Source: Custom Source (1/2)
> (05a3888ed2a232f234a10775826395a0) switched from CANCELING to CANCELED.
> [Source: Custom Source (1/2)]
> 2017-04-28 10:26:29,797 INFO org.apache.flink.runtime.taskmanager.Task
> - Freeing task resources for Source: Custom Source (1/2)
> (05a3888ed2a232f234a10775826395a0). [Source: Custom Source (1/2)]
> 2017-04-28 10:26:29,798 INFO org.apache.flink.core.fs.FileSystem
> - Ensuring all FileSystem streams are closed for Source: Custom
> Source (1/2) [Source: Custom Source (1/2)]
> 2017-04-28 10:26:29,803 INFO
> org.apache.flink.runtime.taskmanager.TaskManager -
> Un-registering task and sending final execution state CANCELED to JobManager
> for task Source: Custom Source (05a3888ed2a232f234a10775826395a0)
> [flink-akka.actor.default-dispatcher-2]
> 2017-04-28 10:26:39,608 INFO org.apache.flink.runtime.taskmanager.Task
> - Attempting to fail task externally Map (2/2)
> (bdb982b6ef47fe79b6ff5b96153c921e). [pool-14-thread-1]
> 2017-04-28 10:26:39,608 WARN
> org.apache.flink.streaming.runtime.tasks.StreamTask - Could not
> properly clean up the async checkpoint runnable. [Canceler for Map (2/2)
> (bdb982b6ef47fe79b6ff5b96153c921e).]
> java.lang.Exception: Could not properly cancel managed keyed state future.
> at
> org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:91)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.cleanup
[jira] [Commented] (FLINK-6416) Potential divide by zero issue in InputGateMetrics#refreshAndGetAvg()
[
https://issues.apache.org/jira/browse/FLINK-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989121#comment-15989121
]
Greg Hogan commented on FLINK-6416:
---
Wouldn't this produce {{NaN}}?
> Potential divide by zero issue in InputGateMetrics#refreshAndGetAvg()
> -
>
> Key: FLINK-6416
> URL: https://issues.apache.org/jira/browse/FLINK-6416
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> int count = 0;
> for (InputChannel channel : inputGate.getInputChannels().values()) {
> if (channel instanceof RemoteInputChannel) {
> RemoteInputChannel rc = (RemoteInputChannel) channel;
> int size = rc.unsynchronizedGetNumberOfQueuedBuffers();
> total += size;
> ++count;
> }
> }
> return total / (float) count;
> {code}
> If count is zero at the end of the loop, the division would produce exception.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6341) JobManager can go to definite message sending loop when TaskManager registered
[ https://issues.apache.org/jira/browse/FLINK-6341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989090#comment-15989090 ] ASF GitHub Bot commented on FLINK-6341: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3745 > JobManager can go to definite message sending loop when TaskManager registered > -- > > Key: FLINK-6341 > URL: https://issues.apache.org/jira/browse/FLINK-6341 > Project: Flink > Issue Type: Bug > Components: JobManager >Reporter: Tao Wang >Assignee: Tao Wang > Fix For: 1.3.0 > > > When TaskManager register to JobManager, JM will send a > "NotifyResourceStarted" message to kick off Resource Manager, then trigger a > reconnection to resource manager through sending a > "TriggerRegistrationAtJobManager". > When the ref of resource manager in JobManager is not None and the > reconnection is to same resource manager, JobManager will go to a infinite > message sending loop which will always sending himself a > "ReconnectResourceManager" every 2 seconds. > We have already observed that phonomenon. More details, check how JobManager > handles `ReconnectResourceManager`. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] flink pull request #3745: [FLINK-6341]Don't let JM fall into infinite loop
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3745 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (FLINK-6341) JobManager can go to definite message sending loop when TaskManager registered
[ https://issues.apache.org/jira/browse/FLINK-6341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann resolved FLINK-6341. -- Resolution: Fixed Fix Version/s: 1.3.0 Fixed via 238383926b762c1d47159a2b4dabe8fd59777307 > JobManager can go to definite message sending loop when TaskManager registered > -- > > Key: FLINK-6341 > URL: https://issues.apache.org/jira/browse/FLINK-6341 > Project: Flink > Issue Type: Bug > Components: JobManager >Reporter: Tao Wang >Assignee: Tao Wang > Fix For: 1.3.0 > > > When TaskManager register to JobManager, JM will send a > "NotifyResourceStarted" message to kick off Resource Manager, then trigger a > reconnection to resource manager through sending a > "TriggerRegistrationAtJobManager". > When the ref of resource manager in JobManager is not None and the > reconnection is to same resource manager, JobManager will go to a infinite > message sending loop which will always sending himself a > "ReconnectResourceManager" every 2 seconds. > We have already observed that phonomenon. More details, check how JobManager > handles `ReconnectResourceManager`. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6395) TestBases not marked as abstract
[ https://issues.apache.org/jira/browse/FLINK-6395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989086#comment-15989086 ] ASF GitHub Bot commented on FLINK-6395: --- Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/3790 Alright, now I understand. +1 for merging. > TestBases not marked as abstract > > > Key: FLINK-6395 > URL: https://issues.apache.org/jira/browse/FLINK-6395 > Project: Flink > Issue Type: Improvement > Components: Tests >Affects Versions: 1.3.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Trivial > > The following test base classes don't contain any test methods and should be > marked as abstract: > BinaryOperatorTestBase > DriverTestBase > UnaryOperatorTestBase -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] flink issue #3790: [FLINK-6395] [tests] Mark test bases as abstract
Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/3790 Alright, now I understand. +1 for merging. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-6417) Wildcard support for read text file
Artiom Darie created FLINK-6417:
---
Summary: Wildcard support for read text file
Key: FLINK-6417
URL: https://issues.apache.org/jira/browse/FLINK-6417
Project: Flink
Issue Type: New Feature
Components: Core
Reporter: Artiom Darie
Priority: Minor
Add wildcard support while reading from s3://, hdfs://, file://, etc.
h6. Examples:
# {code} s3://bucket-name/*.gz {code}
# {code} hdfs://path/*file-name*.csv {code}
# {code} file://tmp/**/*.* {code}
h6. Proposal
# Use the existing method: {code}environment.readFile(...){code}
# List all the files in the directories
# Read files using existing: {code}ContinuousFileReaderOperator{code}
h6. Concerns (Open for discussions)
# Have multiple DataSource(s) created for each each file and then to join them
into a single DataSource
# Have all the files into the same DataSource
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6360) Failed to create assembly for Scala 2.11 due to tools/change-scala-version.sh not changing flink-gelly-examples_2.10
[
https://issues.apache.org/jira/browse/FLINK-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989063#comment-15989063
]
ASF GitHub Bot commented on FLINK-6360:
---
Github user EronWright closed the pull request at:
https://github.com/apache/flink/pull/3784
> Failed to create assembly for Scala 2.11 due to tools/change-scala-version.sh
> not changing flink-gelly-examples_2.10
>
>
> Key: FLINK-6360
> URL: https://issues.apache.org/jira/browse/FLINK-6360
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.3.0
>Reporter: Jacek Laskowski
>Priority: Minor
>
> I'm building Flink for Scala 2.11 using the following command:
> {code}
> oss; cd flink; gco -- .; gl && \
> ./tools/change-scala-version.sh 2.11 && \
> mvn clean install -DskipTests -Dhadoop.version=2.7.3 -Dscala.version=2.11.7
> {code}
> For the past couple of days I've been unable to build Flink because of the
> following error:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-assembly-plugin:2.6:single (bin) on project
> flink-dist_2.11: Failed to create assembly: Error adding file to archive:
> /Users/jacek/dev/oss/flink/flink-dist/../flink-libraries/flink-gelly-examples/target/flink-gelly-examples_2.10-1.3-SNAPSHOT.jar
> -> [Help 1]
> {code}
> I've been able to trace it down to {{flink-dist/src/main/assemblies/bin.xml}}
> not being changed by {{./tools/change-scala-version.sh}} so the build tries
> to include {{flink-gelly-examples_2.10}} rather than
> {{flink-gelly-examples_2.11}}.
> Please comment if my reasoning is correct or not and I'd be happy to work on
> it. Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6360) Failed to create assembly for Scala 2.11 due to tools/change-scala-version.sh not changing flink-gelly-examples_2.10
[
https://issues.apache.org/jira/browse/FLINK-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989062#comment-15989062
]
ASF GitHub Bot commented on FLINK-6360:
---
Github user EronWright commented on the issue:
https://github.com/apache/flink/pull/3784
Fixed by 03afbca5d047988e7fd8aff5d8b83ddee19570c9
> Failed to create assembly for Scala 2.11 due to tools/change-scala-version.sh
> not changing flink-gelly-examples_2.10
>
>
> Key: FLINK-6360
> URL: https://issues.apache.org/jira/browse/FLINK-6360
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.3.0
>Reporter: Jacek Laskowski
>Priority: Minor
>
> I'm building Flink for Scala 2.11 using the following command:
> {code}
> oss; cd flink; gco -- .; gl && \
> ./tools/change-scala-version.sh 2.11 && \
> mvn clean install -DskipTests -Dhadoop.version=2.7.3 -Dscala.version=2.11.7
> {code}
> For the past couple of days I've been unable to build Flink because of the
> following error:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-assembly-plugin:2.6:single (bin) on project
> flink-dist_2.11: Failed to create assembly: Error adding file to archive:
> /Users/jacek/dev/oss/flink/flink-dist/../flink-libraries/flink-gelly-examples/target/flink-gelly-examples_2.10-1.3-SNAPSHOT.jar
> -> [Help 1]
> {code}
> I've been able to trace it down to {{flink-dist/src/main/assemblies/bin.xml}}
> not being changed by {{./tools/change-scala-version.sh}} so the build tries
> to include {{flink-gelly-examples_2.10}} rather than
> {{flink-gelly-examples_2.11}}.
> Please comment if my reasoning is correct or not and I'd be happy to work on
> it. Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink pull request #3784: FLINK-6360
Github user EronWright closed the pull request at: https://github.com/apache/flink/pull/3784 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3784: FLINK-6360
Github user EronWright commented on the issue: https://github.com/apache/flink/pull/3784 Fixed by 03afbca5d047988e7fd8aff5d8b83ddee19570c9 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6409) TUMBLE/HOP/SESSION_START/END do not resolve time field correctly
[ https://issues.apache.org/jira/browse/FLINK-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989056#comment-15989056 ] ASF GitHub Bot commented on FLINK-6409: --- GitHub user twalthr opened a pull request: https://github.com/apache/flink/pull/3799 [FLINK-6409] [table] TUMBLE/HOP/SESSION_START/END do not resolve time field correctly This is a temporary fix until the next Calcite release 1.13. It adds 5 Calcite files to the Flink code base and modifies one Calcite file in order to successfully run all tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/twalthr/flink FLINK-6409 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3799.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3799 commit 31919bb8be56c55bb601a1e2de74497c2463bdf7 Author: twalthr Date: 2017-04-28T15:17:30Z [FLINK-6409] [table] TUMBLE/HOP/SESSION_START/END do not resolve time field correctly > TUMBLE/HOP/SESSION_START/END do not resolve time field correctly > > > Key: FLINK-6409 > URL: https://issues.apache.org/jira/browse/FLINK-6409 > Project: Flink > Issue Type: Bug > Components: Table API & SQL >Reporter: Timo Walther >Assignee: Timo Walther > > Calcite has a bug and cannot resolve the time fields of auxiliary group > functions correctly. A discussion can be found in CALCITE-1761. > Right now this issue only affects our batch SQL API, but it is a blocker for > FLINK-5884. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] flink pull request #3799: [FLINK-6409] [table] TUMBLE/HOP/SESSION_START/END ...
GitHub user twalthr opened a pull request: https://github.com/apache/flink/pull/3799 [FLINK-6409] [table] TUMBLE/HOP/SESSION_START/END do not resolve time field correctly This is a temporary fix until the next Calcite release 1.13. It adds 5 Calcite files to the Flink code base and modifies one Calcite file in order to successfully run all tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/twalthr/flink FLINK-6409 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3799.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3799 commit 31919bb8be56c55bb601a1e2de74497c2463bdf7 Author: twalthr Date: 2017-04-28T15:17:30Z [FLINK-6409] [table] TUMBLE/HOP/SESSION_START/END do not resolve time field correctly --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-6416) Potential divide by zero issue in InputGateMetrics#refreshAndGetAvg()
Ted Yu created FLINK-6416:
-
Summary: Potential divide by zero issue in
InputGateMetrics#refreshAndGetAvg()
Key: FLINK-6416
URL: https://issues.apache.org/jira/browse/FLINK-6416
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
int count = 0;
for (InputChannel channel : inputGate.getInputChannels().values()) {
if (channel instanceof RemoteInputChannel) {
RemoteInputChannel rc = (RemoteInputChannel) channel;
int size = rc.unsynchronizedGetNumberOfQueuedBuffers();
total += size;
++count;
}
}
return total / (float) count;
{code}
If count is zero at the end of the loop, the division would produce exception.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6415) Make sure core Flink artifacts have no specific logger dependency
[
https://issues.apache.org/jira/browse/FLINK-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988982#comment-15988982
]
Aljoscha Krettek commented on FLINK-6415:
-
I think BEAM-567 is a manifestation of this.
> Make sure core Flink artifacts have no specific logger dependency
> -
>
> Key: FLINK-6415
> URL: https://issues.apache.org/jira/browse/FLINK-6415
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 1.3.0
>
>
> Flink's code is written against {{slf4j}}
> To make sure users can use their custom logging framework we need to have
> - no direct compile-scope dependency in any core
> - a dependency in {{flink-dist}} that is not in the fat jar
> - an explicit dependency in {{examples}} (to see logs when running in the
> IDE)
> - an explicit test dependency (for logs of test execution)
> All except point (1) are already fixed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6398) RowSerializer's duplicate should always return a new instance
[
https://issues.apache.org/jira/browse/FLINK-6398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988975#comment-15988975
]
Aljoscha Krettek commented on FLINK-6398:
-
Thanks!
> RowSerializer's duplicate should always return a new instance
> -
>
> Key: FLINK-6398
> URL: https://issues.apache.org/jira/browse/FLINK-6398
> Project: Flink
> Issue Type: Bug
> Components: Core
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Kurt Young
>Assignee: Kurt Young
> Fix For: 1.3.0, 1.2.2
>
>
> RowSerializer is stateful because of {{nullMask}}, we should always return a
> new instance when duplicating.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (FLINK-6415) Make sure core Flink artifacts have no specific logger dependency
[
https://issues.apache.org/jira/browse/FLINK-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen updated FLINK-6415:
Summary: Make sure core Flink artifacts have no specific logger dependency
(was: Make sure Flink artifacts have no specific logger dependency)
> Make sure core Flink artifacts have no specific logger dependency
> -
>
> Key: FLINK-6415
> URL: https://issues.apache.org/jira/browse/FLINK-6415
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 1.3.0
>
>
> Flink's code is written against {{slf4j}}
> To make sure users can use their custom logging framework we need to have
> - no direct compile-scope dependency in any core
> - a dependency in {{flink-dist}} that is not in the fat jar
> - an explicit dependency in {{examples}} (to see logs when running in the
> IDE)
> - an explicit test dependency (for logs of test execution)
> All except point (1) are already fixed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6337) Remove the buffer provider from PartitionRequestServerHandler
[
https://issues.apache.org/jira/browse/FLINK-6337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988946#comment-15988946
]
ASF GitHub Bot commented on FLINK-6337:
---
Github user zhijiangW commented on the issue:
https://github.com/apache/flink/pull/3785
Hi @uce, I have submitted the modifications for the above two issues.
Welcome any comments! :-)
> Remove the buffer provider from PartitionRequestServerHandler
> -
>
> Key: FLINK-6337
> URL: https://issues.apache.org/jira/browse/FLINK-6337
> Project: Flink
> Issue Type: Improvement
> Components: Network
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
>
> Currently, {{PartitionRequestServerHandler}} will create a
> {{LocalBufferPool}} when the channel is registered. The {{LocalBufferPool}}
> is only used to get segment size for creating read view in
> {{SpillableSubpartition}}, and the buffers in the pool will not be used all
> the time, so it will waste the buffer resource of global pool.
> We would like to remove the {{LocalBufferPool}} from the
> {{PartitionRequestServerHandler}}, and the {{LocalBufferPool}} in
> {{ResultPartition}} can also provide the segment size for creating sub
> partition view.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6412) Stream has already been closed during job cancel
[
https://issues.apache.org/jira/browse/FLINK-6412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988943#comment-15988943
]
Stephan Ewen commented on FLINK-6412:
-
[~srichter] What's your take on this one?
My first thought would be that the problem is in the fact that to "cancel" the
{{OperatorSnapshotResult}}, it has to be run. That seems something that could
be improved by making sure that {{OperatorSnapshotResult}} cancels properly and
needs not be executed to be discarded.
> Stream has already been closed during job cancel
>
>
> Key: FLINK-6412
> URL: https://issues.apache.org/jira/browse/FLINK-6412
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Andrey
>
> Steps to reproduce:
> * create job with RocksDBStateBackend: env.setStateBackend(new
> RocksDBStateBackend(checkpointRoot));
> * run job
> * cancel job from the Web UI.
> In logs:
> {code}
> 2017-04-28 10:25:57,475 INFO org.apache.flink.runtime.taskmanager.Task
> - Source: Custom Source (1/2)
> (05a3888ed2a232f234a10775826395a0) switched from DEPLOYING to RUNNING.
> [Source: Custom Source (1/2)]
> 2017-04-28 10:25:57,476 INFO
> org.apache.flink.streaming.runtime.tasks.StreamTask - Using
> user-defined state backend: RocksDB State Backend {isInitialized=false,
> configuredDbBasePaths=null, initializedDbBasePaths=null,
> checkpointStreamBackend=File State Backend @
> file:/flink/checkpoints/flinktests}. [Map (2/2)]
> 2017-04-28 10:25:57,476 INFO
> org.apache.flink.streaming.runtime.tasks.StreamTask - Using
> user-defined state backend: RocksDB State Backend {isInitialized=false,
> configuredDbBasePaths=null, initializedDbBasePaths=null,
> checkpointStreamBackend=File State Backend @
> file:/flink/checkpoints/flinktests}. [Source: Custom Source (1/2)]
> ...
> 2017-04-28 10:26:29,793 INFO org.apache.flink.runtime.taskmanager.Task
> - Triggering cancellation of task code Source: Custom Source
> (1/2) (05a3888ed2a232f234a10775826395a0).
> [flink-akka.actor.default-dispatcher-2]
> 2017-04-28 10:26:29,794 INFO org.apache.flink.runtime.taskmanager.Task
> - Attempting to cancel task Map (2/2)
> (bdb982b6ef47fe79b6ff5b96153c921e). [flink-akka.actor.default-dispatcher-2]
> 2017-04-28 10:26:29,794 INFO org.apache.flink.runtime.taskmanager.Task
> - Map (2/2) (bdb982b6ef47fe79b6ff5b96153c921e) switched from
> RUNNING to CANCELING. [flink-akka.actor.default-dispatcher-2]
> 2017-04-28 10:26:29,796 INFO org.apache.flink.runtime.taskmanager.Task
> - Triggering cancellation of task code Map (2/2)
> (bdb982b6ef47fe79b6ff5b96153c921e). [flink-akka.actor.default-dispatcher-2]
> 2017-04-28 10:26:29,796 INFO org.apache.flink.runtime.taskmanager.Task
> - Source: Custom Source (1/2)
> (05a3888ed2a232f234a10775826395a0) switched from CANCELING to CANCELED.
> [Source: Custom Source (1/2)]
> 2017-04-28 10:26:29,797 INFO org.apache.flink.runtime.taskmanager.Task
> - Freeing task resources for Source: Custom Source (1/2)
> (05a3888ed2a232f234a10775826395a0). [Source: Custom Source (1/2)]
> 2017-04-28 10:26:29,798 INFO org.apache.flink.core.fs.FileSystem
> - Ensuring all FileSystem streams are closed for Source: Custom
> Source (1/2) [Source: Custom Source (1/2)]
> 2017-04-28 10:26:29,803 INFO
> org.apache.flink.runtime.taskmanager.TaskManager -
> Un-registering task and sending final execution state CANCELED to JobManager
> for task Source: Custom Source (05a3888ed2a232f234a10775826395a0)
> [flink-akka.actor.default-dispatcher-2]
> 2017-04-28 10:26:39,608 INFO org.apache.flink.runtime.taskmanager.Task
> - Attempting to fail task externally Map (2/2)
> (bdb982b6ef47fe79b6ff5b96153c921e). [pool-14-thread-1]
> 2017-04-28 10:26:39,608 WARN
> org.apache.flink.streaming.runtime.tasks.StreamTask - Could not
> properly clean up the async checkpoint runnable. [Canceler for Map (2/2)
> (bdb982b6ef47fe79b6ff5b96153c921e).]
> java.lang.Exception: Could not properly cancel managed keyed state future.
> at
> org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:91)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.cleanup(StreamTask.java:1010)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.close(StreamTask.java:995)
> at org.apache.flink.util.IOUtils.closeQuietly(IOUtils.java:262)
> at org.apache.flink.util.IOUtils.closeAllQuietly(IOUtils.java:251)
> at
> org.apache.flink.util.AbstractCloseableRegistry.close(AbstractCloseableRegistry.java
[GitHub] flink issue #3785: [FLINK-6337][network] Remove the buffer provider from Par...
Github user zhijiangW commented on the issue: https://github.com/apache/flink/pull/3785 Hi @uce, I have submitted the modifications for the above two issues. Welcome any comments! :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6398) RowSerializer's duplicate should always return a new instance
[
https://issues.apache.org/jira/browse/FLINK-6398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988939#comment-15988939
]
Kurt Young commented on FLINK-6398:
---
done
> RowSerializer's duplicate should always return a new instance
> -
>
> Key: FLINK-6398
> URL: https://issues.apache.org/jira/browse/FLINK-6398
> Project: Flink
> Issue Type: Bug
> Components: Core
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Kurt Young
>Assignee: Kurt Young
> Fix For: 1.3.0, 1.2.2
>
>
> RowSerializer is stateful because of {{nullMask}}, we should always return a
> new instance when duplicating.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Reopened] (FLINK-6398) RowSerializer's duplicate should always return a new instance
[
https://issues.apache.org/jira/browse/FLINK-6398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young reopened FLINK-6398:
---
> RowSerializer's duplicate should always return a new instance
> -
>
> Key: FLINK-6398
> URL: https://issues.apache.org/jira/browse/FLINK-6398
> Project: Flink
> Issue Type: Bug
> Components: Core
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Kurt Young
>Assignee: Kurt Young
> Fix For: 1.3.0, 1.2.2
>
>
> RowSerializer is stateful because of {{nullMask}}, we should always return a
> new instance when duplicating.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (FLINK-6398) RowSerializer's duplicate should always return a new instance
[
https://issues.apache.org/jira/browse/FLINK-6398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young closed FLINK-6398.
-
Resolution: Fixed
Fix Version/s: 1.2.2
> RowSerializer's duplicate should always return a new instance
> -
>
> Key: FLINK-6398
> URL: https://issues.apache.org/jira/browse/FLINK-6398
> Project: Flink
> Issue Type: Bug
> Components: Core
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Kurt Young
>Assignee: Kurt Young
> Fix For: 1.3.0, 1.2.2
>
>
> RowSerializer is stateful because of {{nullMask}}, we should always return a
> new instance when duplicating.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6413) Add stream operator callback to notify about consumed network buffer
[
https://issues.apache.org/jira/browse/FLINK-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988933#comment-15988933
]
Stephan Ewen commented on FLINK-6413:
-
Not sure about this one. Seems quite complex to model a mechanism that is not
needed by Flink (and exists in Beam because some other runtimes may need it).
This sounds to me like making the runtime more complex to match an interface
that was designed for a different runtime.
An alternative thought: We had been thinking a while back that we should offer
an interface to operators to {{processBuffer}} rather than {{processElement}}.
I think that would be more natural than a buffer consumption listener.
Also, given that the network stack needs some quite crucial other fixes along
latency and stream alignments, I am very skeptical to introduce such added
complexity unless we have a strong case that some users need that.
> Add stream operator callback to notify about consumed network buffer
> -
>
> Key: FLINK-6413
> URL: https://issues.apache.org/jira/browse/FLINK-6413
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API
>Reporter: Aljoscha Krettek
>
> This is originally motivated by BEAM-1612. Beam has the notion of bundles and
> allows users to do work at the start/end of each bundle. This could be used
> for setting up some expensive connection or for batching accesses to some
> external system. There is also internal optimisation potential because
> accesses/updates to state could be kept in-memory per bundle/buffer and only
> afterwards be written to fault-tolerant state.
> The bundling induced by the Flink network stack (which depends on the network
> buffer size and the buffer timeout) seems like a natural fit for this. I
> propose to add an _experimental_ interface {{BufferConsumedListener}} (or
> some such name):
> {code}
> interface BufferConsumedListener {
> void notifyBufferConsumed():
> }
> {code}
> that is invoked in the input processor whenever a network buffer is
> exhausted:
> https://github.com/apache/flink/blob/922352ac35f3753334e834632e3e361fbd36336e/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamInputProcessor.java#L178-L178
> The change is very simple, three lines of code would be added:
> {code}
> if (result.isBufferConsumed()) {
> currentRecordDeserializer.getCurrentBuffer().recycle();
> currentRecordDeserializer = null;
> if (streamOperator instanceof BufferConsumedListener) {
> ((BufferConsumedListener) streamOperator).notifyBufferConsumed():
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6337) Remove the buffer provider from PartitionRequestServerHandler
[
https://issues.apache.org/jira/browse/FLINK-6337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988926#comment-15988926
]
ASF GitHub Bot commented on FLINK-6337:
---
Github user zhijiangW commented on a diff in the pull request:
https://github.com/apache/flink/pull/3785#discussion_r113946358
--- Diff:
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/SpillableSubpartitionTest.java
---
@@ -68,7 +68,11 @@ public static void shutdown() {
@Override
ResultSubpartition createSubpartition() {
- return new SpillableSubpartition(0,
mock(ResultPartition.class), ioManager);
+ ResultPartition parent = mock(ResultPartition.class);
--- End diff --
I guess you want to reuse this mocked `ResultPartition` in all the internal
tests, and I notice that there already exists `createSubpartition` override
method in this test, so I can modify the return type of `SpillableSubpartition`
instead. Do you think so?
> Remove the buffer provider from PartitionRequestServerHandler
> -
>
> Key: FLINK-6337
> URL: https://issues.apache.org/jira/browse/FLINK-6337
> Project: Flink
> Issue Type: Improvement
> Components: Network
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
>
> Currently, {{PartitionRequestServerHandler}} will create a
> {{LocalBufferPool}} when the channel is registered. The {{LocalBufferPool}}
> is only used to get segment size for creating read view in
> {{SpillableSubpartition}}, and the buffers in the pool will not be used all
> the time, so it will waste the buffer resource of global pool.
> We would like to remove the {{LocalBufferPool}} from the
> {{PartitionRequestServerHandler}}, and the {{LocalBufferPool}} in
> {{ResultPartition}} can also provide the segment size for creating sub
> partition view.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink pull request #3785: [FLINK-6337][network] Remove the buffer provider f...
Github user zhijiangW commented on a diff in the pull request:
https://github.com/apache/flink/pull/3785#discussion_r113946358
--- Diff:
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/SpillableSubpartitionTest.java
---
@@ -68,7 +68,11 @@ public static void shutdown() {
@Override
ResultSubpartition createSubpartition() {
- return new SpillableSubpartition(0,
mock(ResultPartition.class), ioManager);
+ ResultPartition parent = mock(ResultPartition.class);
--- End diff --
I guess you want to reuse this mocked `ResultPartition` in all the internal
tests, and I notice that there already exists `createSubpartition` override
method in this test, so I can modify the return type of `SpillableSubpartition`
instead. Do you think so?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-6398) RowSerializer's duplicate should always return a new instance
[
https://issues.apache.org/jira/browse/FLINK-6398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988925#comment-15988925
]
Kurt Young commented on FLINK-6398:
---
[~aljoscha] Thanks for the advise, i think it's reasonable to back port it to
release-1.2, i will do it.
> RowSerializer's duplicate should always return a new instance
> -
>
> Key: FLINK-6398
> URL: https://issues.apache.org/jira/browse/FLINK-6398
> Project: Flink
> Issue Type: Bug
> Components: Core
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Kurt Young
>Assignee: Kurt Young
> Fix For: 1.3.0
>
>
> RowSerializer is stateful because of {{nullMask}}, we should always return a
> new instance when duplicating.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6414) Use scala.binary.version in place of change-scala-version.sh
[
https://issues.apache.org/jira/browse/FLINK-6414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988923#comment-15988923
]
Stephan Ewen commented on FLINK-6414:
-
I think we tried to do that initially, but it was not possible to completely
rely on variables.
Can't remember quite why, [~uce] or [~rmetzger] might remember why...
> Use scala.binary.version in place of change-scala-version.sh
>
>
> Key: FLINK-6414
> URL: https://issues.apache.org/jira/browse/FLINK-6414
> Project: Flink
> Issue Type: Improvement
> Components: Build System
>Affects Versions: 1.3.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>
> Recent commits have failed to modify {{change-scala-version.sh}} resulting in
> broken builds for {{scala-2.11}}. It looks like we can remove the need for
> this script by replacing hard-coded references to the Scala version with
> Flink's maven variable {{scala.binary.version}}.
> I had initially realized that the change script is [only used for
> building|https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/building.html#scala-versions]
> and not for switching the IDE environment.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-6415) Make sure Flink artifacts have no specific logger dependency
Stephan Ewen created FLINK-6415:
---
Summary: Make sure Flink artifacts have no specific logger
dependency
Key: FLINK-6415
URL: https://issues.apache.org/jira/browse/FLINK-6415
Project: Flink
Issue Type: Bug
Components: Build System
Affects Versions: 1.2.1, 1.2.0
Reporter: Stephan Ewen
Assignee: Stephan Ewen
Fix For: 1.3.0
Flink's code is written against {{slf4j}}
To make sure users can use their custom logging framework we need to have
- no direct compile-scope dependency in any core
- a dependency in {{flink-dist}} that is not in the fat jar
- an explicit dependency in {{examples}} (to see logs when running in the IDE)
- an explicit test dependency (for logs of test execution)
All except point (1) are already fixed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6393) Add Evenly Graph Generator to Flink Gelly
[ https://issues.apache.org/jira/browse/FLINK-6393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988921#comment-15988921 ] ASF GitHub Bot commented on FLINK-6393: --- Github user greghogan commented on the issue: https://github.com/apache/flink/pull/3788 @fanzhidongyzby, what do you think about generalizing this to the `CirculantGraph`? The graph could be initialized with a vertex count and the configured with one or more ranges. Each range would be an offset and a length. > Add Evenly Graph Generator to Flink Gelly > - > > Key: FLINK-6393 > URL: https://issues.apache.org/jira/browse/FLINK-6393 > Project: Flink > Issue Type: New Feature > Components: Gelly >Reporter: FlorianFan >Assignee: FlorianFan >Priority: Minor > Original Estimate: 24h > Remaining Estimate: 24h > > Evenly graph means every vertex in the graph has the same degree, so the > graph can be treated as evenly due to all the edges in the graph are > distributed evenly. when vertex degree is 0, an empty graph will be > generated. when vertex degree is vertex count - 1, complete graph will be > generated. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6337) Remove the buffer provider from PartitionRequestServerHandler
[
https://issues.apache.org/jira/browse/FLINK-6337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988920#comment-15988920
]
ASF GitHub Bot commented on FLINK-6337:
---
Github user zhijiangW commented on a diff in the pull request:
https://github.com/apache/flink/pull/3785#discussion_r113945018
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/misc/SuccessAfterNetworkBuffersFailureITCase.java
---
@@ -68,7 +68,6 @@ public void testSuccessfulProgramAfterFailure() {
try {
runKMeans(cluster.getLeaderRPCPort());
- fail("This program execution should have
failed.");
--- End diff --
Yes, I misunderstand this test and you get the key point.
> Remove the buffer provider from PartitionRequestServerHandler
> -
>
> Key: FLINK-6337
> URL: https://issues.apache.org/jira/browse/FLINK-6337
> Project: Flink
> Issue Type: Improvement
> Components: Network
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
>
> Currently, {{PartitionRequestServerHandler}} will create a
> {{LocalBufferPool}} when the channel is registered. The {{LocalBufferPool}}
> is only used to get segment size for creating read view in
> {{SpillableSubpartition}}, and the buffers in the pool will not be used all
> the time, so it will waste the buffer resource of global pool.
> We would like to remove the {{LocalBufferPool}} from the
> {{PartitionRequestServerHandler}}, and the {{LocalBufferPool}} in
> {{ResultPartition}} can also provide the segment size for creating sub
> partition view.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink issue #3788: [FLINK-6393] [gelly] Add Evenly Graph Generator to Flink ...
Github user greghogan commented on the issue: https://github.com/apache/flink/pull/3788 @fanzhidongyzby, what do you think about generalizing this to the `CirculantGraph`? The graph could be initialized with a vertex count and the configured with one or more ranges. Each range would be an offset and a length. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3785: [FLINK-6337][network] Remove the buffer provider f...
Github user zhijiangW commented on a diff in the pull request:
https://github.com/apache/flink/pull/3785#discussion_r113945018
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/misc/SuccessAfterNetworkBuffersFailureITCase.java
---
@@ -68,7 +68,6 @@ public void testSuccessfulProgramAfterFailure() {
try {
runKMeans(cluster.getLeaderRPCPort());
- fail("This program execution should have
failed.");
--- End diff --
Yes, I misunderstand this test and you get the key point.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-6019) Some log4j messages do not have a loglevel field set, so they can't be suppressed
[
https://issues.apache.org/jira/browse/FLINK-6019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988913#comment-15988913
]
Stephan Ewen commented on FLINK-6019:
-
Do you see these lines in the log? I think they come from the client's sysout
printing...
You can deactivate the progress printing in the {{ExecutionConfig}} of the
program...
> Some log4j messages do not have a loglevel field set, so they can't be
> suppressed
> -
>
> Key: FLINK-6019
> URL: https://issues.apache.org/jira/browse/FLINK-6019
> Project: Flink
> Issue Type: Bug
> Components: Core
>Affects Versions: 1.2.0
> Environment: Linux
>Reporter: Luke Hutchison
>
> Some of the log messages do not appear to have a loglevel value set, so they
> can't be suppressed by setting the log4j level to WARN. There's this line at
> the beginning which doesn't even have a timestamp:
> {noformat}
> Connected to JobManager at Actor[akka://flink/user/jobmanager_1#1844933939]
> {noformat}
> And then there are numerous lines like this, missing an "INFO" field:
> {noformat}
> 03/10/2017 00:01:14 Job execution switched to status RUNNING.
> 03/10/2017 00:01:14 DataSource (at readTable(DBTableReader.java:165)
> (org.apache.flink.api.java.io.PojoCsvInputFormat))(1/8) switched to SCHEDULED
> 03/10/2017 00:01:14 DataSink (count())(1/8) switched to SCHEDULED
> 03/10/2017 00:01:14 DataSink (count())(3/8) switched to DEPLOYING
> 03/10/2017 00:01:15 DataSink (count())(3/8) switched to RUNNING
> 03/10/2017 00:01:17 DataSink (count())(6/8) switched to FINISHED
> 03/10/2017 00:01:17 DataSource (at readTable(DBTableReader.java:165)
> (org.apache.flink.api.java.io.PojoCsvInputFormat))(6/8) switched to FINISHED
> 03/10/2017 00:01:17 Job execution switched to status FINISHED.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6360) Failed to create assembly for Scala 2.11 due to tools/change-scala-version.sh not changing flink-gelly-examples_2.10
[
https://issues.apache.org/jira/browse/FLINK-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988907#comment-15988907
]
Stephan Ewen commented on FLINK-6360:
-
Sorry, did not mean to hijack the issue - simply did not check for duplicates
:-(
> Failed to create assembly for Scala 2.11 due to tools/change-scala-version.sh
> not changing flink-gelly-examples_2.10
>
>
> Key: FLINK-6360
> URL: https://issues.apache.org/jira/browse/FLINK-6360
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.3.0
>Reporter: Jacek Laskowski
>Priority: Minor
>
> I'm building Flink for Scala 2.11 using the following command:
> {code}
> oss; cd flink; gco -- .; gl && \
> ./tools/change-scala-version.sh 2.11 && \
> mvn clean install -DskipTests -Dhadoop.version=2.7.3 -Dscala.version=2.11.7
> {code}
> For the past couple of days I've been unable to build Flink because of the
> following error:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-assembly-plugin:2.6:single (bin) on project
> flink-dist_2.11: Failed to create assembly: Error adding file to archive:
> /Users/jacek/dev/oss/flink/flink-dist/../flink-libraries/flink-gelly-examples/target/flink-gelly-examples_2.10-1.3-SNAPSHOT.jar
> -> [Help 1]
> {code}
> I've been able to trace it down to {{flink-dist/src/main/assemblies/bin.xml}}
> not being changed by {{./tools/change-scala-version.sh}} so the build tries
> to include {{flink-gelly-examples_2.10}} rather than
> {{flink-gelly-examples_2.11}}.
> Please comment if my reasoning is correct or not and I'd be happy to work on
> it. Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink pull request #3770: [FLINK-5892] Restore state on the operator level
Github user StefanRRichter commented on a diff in the pull request:
https://github.com/apache/flink/pull/3770#discussion_r113922612
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorState.java
---
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint;
+
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.CompositeStateHandle;
+import org.apache.flink.runtime.state.SharedStateRegistry;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * Simple container class which contains the raw/managed/legacy operator
state and key-group state handles for the sub
+ * tasks of an operator.
+ */
+public class OperatorState implements CompositeStateHandle {
+
+ private static final long serialVersionUID = -4845578005863201810L;
+
+ /** id of the operator */
+ private final OperatorID operatorID;
+
+ /** handles to non-partitioned states, subtaskindex -> subtaskstate */
+ private final Map subtaskStates;
--- End diff --
here and in a few other places in this class, we could add the `operator`
String to the variable names to make it clear for user that we are now
dealing with state on the operator level and avoid confusing flink veterans
that have a certain mental mapping for the word `(Sub)TaskState` that they must
update.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-5892) Recover job state at the granularity of operator
[ https://issues.apache.org/jira/browse/FLINK-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988897#comment-15988897 ] ASF GitHub Bot commented on FLINK-5892: --- Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3770#discussion_r113924290 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionJobVertex.java --- @@ -139,6 +158,10 @@ public ExecutionJobVertex( this.serializedTaskInformation = null; this.taskVertices = new ExecutionVertex[numTaskVertices]; + List opIDs = jobVertex.getOperatorIDs(); + this.operatorIDs = opIDs.toArray(new OperatorID[opIDs.size()]); --- End diff -- How about making `operatorIDs` an immutable list instead of an array. I think all the operations you perform could also run on an array list and we could enforce immutability so that nobody is tempted to modify the inner state of the original array (e.g. to reverse the element order for convenience in other parts of the code). Same for the alternative Ids. > Recover job state at the granularity of operator > > > Key: FLINK-5892 > URL: https://issues.apache.org/jira/browse/FLINK-5892 > Project: Flink > Issue Type: New Feature > Components: State Backends, Checkpointing >Reporter: Guowei Ma >Assignee: Guowei Ma > > JobGraph has no `Operator` info so `ExecutionGraph` can only recovery at the > granularity of task. > This leads to the result that the operator of the job may not recover the > state from a save point even if the save point has the state of operator. > > https://docs.google.com/document/d/19suTRF0nh7pRgeMnIEIdJ2Fq-CcNVt5_hR3cAoICf7Q/edit#. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-5892) Recover job state at the granularity of operator
[
https://issues.apache.org/jira/browse/FLINK-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988899#comment-15988899
]
ASF GitHub Bot commented on FLINK-5892:
---
Github user StefanRRichter commented on a diff in the pull request:
https://github.com/apache/flink/pull/3770#discussion_r113929555
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java
---
@@ -62,252 +69,390 @@ public StateAssignmentOperation(
}
public boolean assignStates() throws Exception {
-
- // this tracks if we find missing node hash ids and already use
secondary mappings
- boolean expandedToLegacyIds = false;
-
+ Map localStates = new
HashMap<>(taskStates);
Map localTasks = this.tasks;
- for (Map.Entry taskGroupStateEntry :
taskStates.entrySet()) {
-
- TaskState taskState = taskGroupStateEntry.getValue();
-
- //find vertex
for state-
-
- ExecutionJobVertex executionJobVertex =
localTasks.get(taskGroupStateEntry.getKey());
-
- // on the first time we can not find the execution job
vertex for an id, we also consider alternative ids,
- // for example as generated from older flink versions,
to provide backwards compatibility.
- if (executionJobVertex == null && !expandedToLegacyIds)
{
- localTasks =
ExecutionJobVertex.includeLegacyJobVertexIDs(localTasks);
- executionJobVertex =
localTasks.get(taskGroupStateEntry.getKey());
- expandedToLegacyIds = true;
- logger.info("Could not find ExecutionJobVertex.
Including legacy JobVertexIDs in search.");
- }
+ Set allOperatorIDs = new HashSet<>();
+ for (ExecutionJobVertex executionJobVertex : tasks.values()) {
+
allOperatorIDs.addAll(Lists.newArrayList(executionJobVertex.getOperatorIDs()));
+ }
+ for (Map.Entry taskGroupStateEntry :
taskStates.entrySet()) {
--- End diff --
This loop looks like we could factor it out into a private precondition
method like `checkStateMappingCompleteness` or something like that. Even the
previous loop and everything working on the hash set could go there.
> Recover job state at the granularity of operator
>
>
> Key: FLINK-5892
> URL: https://issues.apache.org/jira/browse/FLINK-5892
> Project: Flink
> Issue Type: New Feature
> Components: State Backends, Checkpointing
>Reporter: Guowei Ma
>Assignee: Guowei Ma
>
> JobGraph has no `Operator` info so `ExecutionGraph` can only recovery at the
> granularity of task.
> This leads to the result that the operator of the job may not recover the
> state from a save point even if the save point has the state of operator.
>
> https://docs.google.com/document/d/19suTRF0nh7pRgeMnIEIdJ2Fq-CcNVt5_hR3cAoICf7Q/edit#.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink pull request #3770: [FLINK-5892] Restore state on the operator level
Github user StefanRRichter commented on a diff in the pull request:
https://github.com/apache/flink/pull/3770#discussion_r113928558
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java
---
@@ -62,252 +69,390 @@ public StateAssignmentOperation(
}
public boolean assignStates() throws Exception {
-
- // this tracks if we find missing node hash ids and already use
secondary mappings
- boolean expandedToLegacyIds = false;
-
+ Map localStates = new
HashMap<>(taskStates);
Map localTasks = this.tasks;
- for (Map.Entry taskGroupStateEntry :
taskStates.entrySet()) {
-
- TaskState taskState = taskGroupStateEntry.getValue();
-
- //find vertex
for state-
-
- ExecutionJobVertex executionJobVertex =
localTasks.get(taskGroupStateEntry.getKey());
-
- // on the first time we can not find the execution job
vertex for an id, we also consider alternative ids,
- // for example as generated from older flink versions,
to provide backwards compatibility.
- if (executionJobVertex == null && !expandedToLegacyIds)
{
- localTasks =
ExecutionJobVertex.includeLegacyJobVertexIDs(localTasks);
- executionJobVertex =
localTasks.get(taskGroupStateEntry.getKey());
- expandedToLegacyIds = true;
- logger.info("Could not find ExecutionJobVertex.
Including legacy JobVertexIDs in search.");
- }
+ Set allOperatorIDs = new HashSet<>();
+ for (ExecutionJobVertex executionJobVertex : tasks.values()) {
+
allOperatorIDs.addAll(Lists.newArrayList(executionJobVertex.getOperatorIDs()));
+ }
+ for (Map.Entry taskGroupStateEntry :
taskStates.entrySet()) {
--- End diff --
Renaming `taskStates` and `taskGroupStateEntry` to something that has
`operator` instead of `task` in it makes this more readable - maybe
`operatorToStateMapping`. Just some leftover from the refactoring i guess.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-5892) Recover job state at the granularity of operator
[
https://issues.apache.org/jira/browse/FLINK-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988896#comment-15988896
]
ASF GitHub Bot commented on FLINK-5892:
---
Github user StefanRRichter commented on a diff in the pull request:
https://github.com/apache/flink/pull/3770#discussion_r113922612
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorState.java
---
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint;
+
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.CompositeStateHandle;
+import org.apache.flink.runtime.state.SharedStateRegistry;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * Simple container class which contains the raw/managed/legacy operator
state and key-group state handles for the sub
+ * tasks of an operator.
+ */
+public class OperatorState implements CompositeStateHandle {
+
+ private static final long serialVersionUID = -4845578005863201810L;
+
+ /** id of the operator */
+ private final OperatorID operatorID;
+
+ /** handles to non-partitioned states, subtaskindex -> subtaskstate */
+ private final Map subtaskStates;
--- End diff --
here and in a few other places in this class, we could add the `operator`
String to the variable names to make it clear for user that we are now
dealing with state on the operator level and avoid confusing flink veterans
that have a certain mental mapping for the word `(Sub)TaskState` that they must
update.
> Recover job state at the granularity of operator
>
>
> Key: FLINK-5892
> URL: https://issues.apache.org/jira/browse/FLINK-5892
> Project: Flink
> Issue Type: New Feature
> Components: State Backends, Checkpointing
>Reporter: Guowei Ma
>Assignee: Guowei Ma
>
> JobGraph has no `Operator` info so `ExecutionGraph` can only recovery at the
> granularity of task.
> This leads to the result that the operator of the job may not recover the
> state from a save point even if the save point has the state of operator.
>
> https://docs.google.com/document/d/19suTRF0nh7pRgeMnIEIdJ2Fq-CcNVt5_hR3cAoICf7Q/edit#.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-5892) Recover job state at the granularity of operator
[
https://issues.apache.org/jira/browse/FLINK-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988894#comment-15988894
]
ASF GitHub Bot commented on FLINK-5892:
---
Github user StefanRRichter commented on a diff in the pull request:
https://github.com/apache/flink/pull/3770#discussion_r113928558
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java
---
@@ -62,252 +69,390 @@ public StateAssignmentOperation(
}
public boolean assignStates() throws Exception {
-
- // this tracks if we find missing node hash ids and already use
secondary mappings
- boolean expandedToLegacyIds = false;
-
+ Map localStates = new
HashMap<>(taskStates);
Map localTasks = this.tasks;
- for (Map.Entry taskGroupStateEntry :
taskStates.entrySet()) {
-
- TaskState taskState = taskGroupStateEntry.getValue();
-
- //find vertex
for state-
-
- ExecutionJobVertex executionJobVertex =
localTasks.get(taskGroupStateEntry.getKey());
-
- // on the first time we can not find the execution job
vertex for an id, we also consider alternative ids,
- // for example as generated from older flink versions,
to provide backwards compatibility.
- if (executionJobVertex == null && !expandedToLegacyIds)
{
- localTasks =
ExecutionJobVertex.includeLegacyJobVertexIDs(localTasks);
- executionJobVertex =
localTasks.get(taskGroupStateEntry.getKey());
- expandedToLegacyIds = true;
- logger.info("Could not find ExecutionJobVertex.
Including legacy JobVertexIDs in search.");
- }
+ Set allOperatorIDs = new HashSet<>();
+ for (ExecutionJobVertex executionJobVertex : tasks.values()) {
+
allOperatorIDs.addAll(Lists.newArrayList(executionJobVertex.getOperatorIDs()));
+ }
+ for (Map.Entry taskGroupStateEntry :
taskStates.entrySet()) {
--- End diff --
Renaming `taskStates` and `taskGroupStateEntry` to something that has
`operator` instead of `task` in it makes this more readable - maybe
`operatorToStateMapping`. Just some leftover from the refactoring i guess.
> Recover job state at the granularity of operator
>
>
> Key: FLINK-5892
> URL: https://issues.apache.org/jira/browse/FLINK-5892
> Project: Flink
> Issue Type: New Feature
> Components: State Backends, Checkpointing
>Reporter: Guowei Ma
>Assignee: Guowei Ma
>
> JobGraph has no `Operator` info so `ExecutionGraph` can only recovery at the
> granularity of task.
> This leads to the result that the operator of the job may not recover the
> state from a save point even if the save point has the state of operator.
>
> https://docs.google.com/document/d/19suTRF0nh7pRgeMnIEIdJ2Fq-CcNVt5_hR3cAoICf7Q/edit#.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-5892) Recover job state at the granularity of operator
[
https://issues.apache.org/jira/browse/FLINK-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988898#comment-15988898
]
ASF GitHub Bot commented on FLINK-5892:
---
Github user StefanRRichter commented on a diff in the pull request:
https://github.com/apache/flink/pull/3770#discussion_r113920542
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
---
@@ -1025,11 +1026,11 @@ public boolean restoreLatestCheckpointedState(
LOG.info("Restoring from latest valid checkpoint: {}.",
latest);
// re-assign the task states
-
- final Map taskStates =
latest.getTaskStates();
+ final Map operatorStates =
latest.getOperatorStates();
StateAssignmentOperation stateAssignmentOperation =
- new StateAssignmentOperation(LOG,
tasks, taskStates, allowNonRestoredState);
+ new StateAssignmentOperation(LOG,
tasks, operatorStates, allowNonRestoredState);
--- End diff --
Not sure why this is implemented in a way that a logger is passed to the
`StateAssignmentOperation`. I guess the class should simply have its own
logger. I think this could be changed. But seems like this was introduced
earlier and is unrelated to this PR. But I wouldn't to have this refactored to
the normal logger scheme before we merge.
> Recover job state at the granularity of operator
>
>
> Key: FLINK-5892
> URL: https://issues.apache.org/jira/browse/FLINK-5892
> Project: Flink
> Issue Type: New Feature
> Components: State Backends, Checkpointing
>Reporter: Guowei Ma
>Assignee: Guowei Ma
>
> JobGraph has no `Operator` info so `ExecutionGraph` can only recovery at the
> granularity of task.
> This leads to the result that the operator of the job may not recover the
> state from a save point even if the save point has the state of operator.
>
> https://docs.google.com/document/d/19suTRF0nh7pRgeMnIEIdJ2Fq-CcNVt5_hR3cAoICf7Q/edit#.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink pull request #3770: [FLINK-5892] Restore state on the operator level
Github user StefanRRichter commented on a diff in the pull request:
https://github.com/apache/flink/pull/3770#discussion_r113929555
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java
---
@@ -62,252 +69,390 @@ public StateAssignmentOperation(
}
public boolean assignStates() throws Exception {
-
- // this tracks if we find missing node hash ids and already use
secondary mappings
- boolean expandedToLegacyIds = false;
-
+ Map localStates = new
HashMap<>(taskStates);
Map localTasks = this.tasks;
- for (Map.Entry taskGroupStateEntry :
taskStates.entrySet()) {
-
- TaskState taskState = taskGroupStateEntry.getValue();
-
- //find vertex
for state-
-
- ExecutionJobVertex executionJobVertex =
localTasks.get(taskGroupStateEntry.getKey());
-
- // on the first time we can not find the execution job
vertex for an id, we also consider alternative ids,
- // for example as generated from older flink versions,
to provide backwards compatibility.
- if (executionJobVertex == null && !expandedToLegacyIds)
{
- localTasks =
ExecutionJobVertex.includeLegacyJobVertexIDs(localTasks);
- executionJobVertex =
localTasks.get(taskGroupStateEntry.getKey());
- expandedToLegacyIds = true;
- logger.info("Could not find ExecutionJobVertex.
Including legacy JobVertexIDs in search.");
- }
+ Set allOperatorIDs = new HashSet<>();
+ for (ExecutionJobVertex executionJobVertex : tasks.values()) {
+
allOperatorIDs.addAll(Lists.newArrayList(executionJobVertex.getOperatorIDs()));
+ }
+ for (Map.Entry taskGroupStateEntry :
taskStates.entrySet()) {
--- End diff --
This loop looks like we could factor it out into a private precondition
method like `checkStateMappingCompleteness` or something like that. Even the
previous loop and everything working on the hash set could go there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3770: [FLINK-5892] Restore state on the operator level
Github user StefanRRichter commented on a diff in the pull request:
https://github.com/apache/flink/pull/3770#discussion_r113928105
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java
---
@@ -62,252 +69,390 @@ public StateAssignmentOperation(
}
public boolean assignStates() throws Exception {
-
- // this tracks if we find missing node hash ids and already use
secondary mappings
- boolean expandedToLegacyIds = false;
-
+ Map localStates = new
HashMap<>(taskStates);
Map localTasks = this.tasks;
- for (Map.Entry taskGroupStateEntry :
taskStates.entrySet()) {
-
- TaskState taskState = taskGroupStateEntry.getValue();
-
- //find vertex
for state-
-
- ExecutionJobVertex executionJobVertex =
localTasks.get(taskGroupStateEntry.getKey());
-
- // on the first time we can not find the execution job
vertex for an id, we also consider alternative ids,
- // for example as generated from older flink versions,
to provide backwards compatibility.
- if (executionJobVertex == null && !expandedToLegacyIds)
{
- localTasks =
ExecutionJobVertex.includeLegacyJobVertexIDs(localTasks);
- executionJobVertex =
localTasks.get(taskGroupStateEntry.getKey());
- expandedToLegacyIds = true;
- logger.info("Could not find ExecutionJobVertex.
Including legacy JobVertexIDs in search.");
- }
+ Set allOperatorIDs = new HashSet<>();
+ for (ExecutionJobVertex executionJobVertex : tasks.values()) {
+
allOperatorIDs.addAll(Lists.newArrayList(executionJobVertex.getOperatorIDs()));
--- End diff --
I we change to immutable list instead of array, this code also saves one
converting to list
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3770: [FLINK-5892] Restore state on the operator level
Github user StefanRRichter commented on a diff in the pull request:
https://github.com/apache/flink/pull/3770#discussion_r113920542
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
---
@@ -1025,11 +1026,11 @@ public boolean restoreLatestCheckpointedState(
LOG.info("Restoring from latest valid checkpoint: {}.",
latest);
// re-assign the task states
-
- final Map taskStates =
latest.getTaskStates();
+ final Map operatorStates =
latest.getOperatorStates();
StateAssignmentOperation stateAssignmentOperation =
- new StateAssignmentOperation(LOG,
tasks, taskStates, allowNonRestoredState);
+ new StateAssignmentOperation(LOG,
tasks, operatorStates, allowNonRestoredState);
--- End diff --
Not sure why this is implemented in a way that a logger is passed to the
`StateAssignmentOperation`. I guess the class should simply have its own
logger. I think this could be changed. But seems like this was introduced
earlier and is unrelated to this PR. But I wouldn't to have this refactored to
the normal logger scheme before we merge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3770: [FLINK-5892] Restore state on the operator level
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3770#discussion_r113924290 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionJobVertex.java --- @@ -139,6 +158,10 @@ public ExecutionJobVertex( this.serializedTaskInformation = null; this.taskVertices = new ExecutionVertex[numTaskVertices]; + List opIDs = jobVertex.getOperatorIDs(); + this.operatorIDs = opIDs.toArray(new OperatorID[opIDs.size()]); --- End diff -- How about making `operatorIDs` an immutable list instead of an array. I think all the operations you perform could also run on an array list and we could enforce immutability so that nobody is tempted to modify the inner state of the original array (e.g. to reverse the element order for convenience in other parts of the code). Same for the alternative Ids. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5892) Recover job state at the granularity of operator
[
https://issues.apache.org/jira/browse/FLINK-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988895#comment-15988895
]
ASF GitHub Bot commented on FLINK-5892:
---
Github user StefanRRichter commented on a diff in the pull request:
https://github.com/apache/flink/pull/3770#discussion_r113928105
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java
---
@@ -62,252 +69,390 @@ public StateAssignmentOperation(
}
public boolean assignStates() throws Exception {
-
- // this tracks if we find missing node hash ids and already use
secondary mappings
- boolean expandedToLegacyIds = false;
-
+ Map localStates = new
HashMap<>(taskStates);
Map localTasks = this.tasks;
- for (Map.Entry taskGroupStateEntry :
taskStates.entrySet()) {
-
- TaskState taskState = taskGroupStateEntry.getValue();
-
- //find vertex
for state-
-
- ExecutionJobVertex executionJobVertex =
localTasks.get(taskGroupStateEntry.getKey());
-
- // on the first time we can not find the execution job
vertex for an id, we also consider alternative ids,
- // for example as generated from older flink versions,
to provide backwards compatibility.
- if (executionJobVertex == null && !expandedToLegacyIds)
{
- localTasks =
ExecutionJobVertex.includeLegacyJobVertexIDs(localTasks);
- executionJobVertex =
localTasks.get(taskGroupStateEntry.getKey());
- expandedToLegacyIds = true;
- logger.info("Could not find ExecutionJobVertex.
Including legacy JobVertexIDs in search.");
- }
+ Set allOperatorIDs = new HashSet<>();
+ for (ExecutionJobVertex executionJobVertex : tasks.values()) {
+
allOperatorIDs.addAll(Lists.newArrayList(executionJobVertex.getOperatorIDs()));
--- End diff --
I we change to immutable list instead of array, this code also saves one
converting to list
> Recover job state at the granularity of operator
>
>
> Key: FLINK-5892
> URL: https://issues.apache.org/jira/browse/FLINK-5892
> Project: Flink
> Issue Type: New Feature
> Components: State Backends, Checkpointing
>Reporter: Guowei Ma
>Assignee: Guowei Ma
>
> JobGraph has no `Operator` info so `ExecutionGraph` can only recovery at the
> granularity of task.
> This leads to the result that the operator of the job may not recover the
> state from a save point even if the save point has the state of operator.
>
> https://docs.google.com/document/d/19suTRF0nh7pRgeMnIEIdJ2Fq-CcNVt5_hR3cAoICf7Q/edit#.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6399) Update default Hadoop download version
[ https://issues.apache.org/jira/browse/FLINK-6399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988845#comment-15988845 ] Greg Hogan commented on FLINK-6399: --- It's in the Flink website repo: https://github.com/apache/flink-web/blob/asf-site/downloads.md > Update default Hadoop download version > -- > > Key: FLINK-6399 > URL: https://issues.apache.org/jira/browse/FLINK-6399 > Project: Flink > Issue Type: Bug > Components: Project Website >Reporter: Greg Hogan > > [Update|http://flink.apache.org/downloads.html] "If you don’t want to do > this, pick the Hadoop 1 version." since Hadoop 1 versions are no longer > provided. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6411) YarnApplicationMasterRunner should not interfere with RunningJobsRegistry
[
https://issues.apache.org/jira/browse/FLINK-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988843#comment-15988843
]
ASF GitHub Bot commented on FLINK-6411:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/3797
> YarnApplicationMasterRunner should not interfere with RunningJobsRegistry
> -
>
> Key: FLINK-6411
> URL: https://issues.apache.org/jira/browse/FLINK-6411
> Project: Flink
> Issue Type: Bug
> Components: Distributed Coordination
>Affects Versions: 1.3.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Fix For: 1.3.0
>
>
> The {{YarnApplicationMasterRunner}} removes the running job from the
> {{RunningJobsRegistry}} when it is shut down. This should not be its
> responsibility and rather be delegated to the {{JobManagerRunner}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink pull request #3797: [FLINK-6411] [flip-6] Remove job removal from Runn...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3797 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Closed] (FLINK-6411) YarnApplicationMasterRunner should not interfere with RunningJobsRegistry
[
https://issues.apache.org/jira/browse/FLINK-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-6411.
Resolution: Fixed
Fixed via c36d6b808285961505d39187408e1bf77b7b5ea1
> YarnApplicationMasterRunner should not interfere with RunningJobsRegistry
> -
>
> Key: FLINK-6411
> URL: https://issues.apache.org/jira/browse/FLINK-6411
> Project: Flink
> Issue Type: Bug
> Components: Distributed Coordination
>Affects Versions: 1.3.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Fix For: 1.3.0
>
>
> The {{YarnApplicationMasterRunner}} removes the running job from the
> {{RunningJobsRegistry}} when it is shut down. This should not be its
> responsibility and rather be delegated to the {{JobManagerRunner}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (FLINK-6411) YarnApplicationMasterRunner should not interfere with RunningJobsRegistry
[
https://issues.apache.org/jira/browse/FLINK-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-6411:
-
Priority: Minor (was: Major)
> YarnApplicationMasterRunner should not interfere with RunningJobsRegistry
> -
>
> Key: FLINK-6411
> URL: https://issues.apache.org/jira/browse/FLINK-6411
> Project: Flink
> Issue Type: Bug
> Components: Distributed Coordination
>Affects Versions: 1.3.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Fix For: 1.3.0
>
>
> The {{YarnApplicationMasterRunner}} removes the running job from the
> {{RunningJobsRegistry}} when it is shut down. This should not be its
> responsibility and rather be delegated to the {{JobManagerRunner}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (FLINK-6360) Failed to create assembly for Scala 2.11 due to tools/change-scala-version.sh not changing flink-gelly-examples_2.10
[
https://issues.apache.org/jira/browse/FLINK-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Hogan closed FLINK-6360.
-
Resolution: Fixed
Fixed in 03afbca5d047988e7fd8aff5d8b83ddee19570c9
> Failed to create assembly for Scala 2.11 due to tools/change-scala-version.sh
> not changing flink-gelly-examples_2.10
>
>
> Key: FLINK-6360
> URL: https://issues.apache.org/jira/browse/FLINK-6360
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.3.0
>Reporter: Jacek Laskowski
>Priority: Minor
>
> I'm building Flink for Scala 2.11 using the following command:
> {code}
> oss; cd flink; gco -- .; gl && \
> ./tools/change-scala-version.sh 2.11 && \
> mvn clean install -DskipTests -Dhadoop.version=2.7.3 -Dscala.version=2.11.7
> {code}
> For the past couple of days I've been unable to build Flink because of the
> following error:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-assembly-plugin:2.6:single (bin) on project
> flink-dist_2.11: Failed to create assembly: Error adding file to archive:
> /Users/jacek/dev/oss/flink/flink-dist/../flink-libraries/flink-gelly-examples/target/flink-gelly-examples_2.10-1.3-SNAPSHOT.jar
> -> [Help 1]
> {code}
> I've been able to trace it down to {{flink-dist/src/main/assemblies/bin.xml}}
> not being changed by {{./tools/change-scala-version.sh}} so the build tries
> to include {{flink-gelly-examples_2.10}} rather than
> {{flink-gelly-examples_2.11}}.
> Please comment if my reasoning is correct or not and I'd be happy to work on
> it. Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (FLINK-6410) build fails after changing Scala to 2.11
[
https://issues.apache.org/jira/browse/FLINK-6410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Hogan closed FLINK-6410.
-
Resolution: Duplicate
> build fails after changing Scala to 2.11
>
>
> Key: FLINK-6410
> URL: https://issues.apache.org/jira/browse/FLINK-6410
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.3.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The {{change-scala-version.sh}} script does not correctly adjust the
> {{bin.xml}} assembly descriptor.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-5476) Fail fast if trying to submit a job to a non-existing Flink cluster
[
https://issues.apache.org/jira/browse/FLINK-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988831#comment-15988831
]
ASF GitHub Bot commented on FLINK-5476:
---
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/3753
The code does not seem to match the PR description @DmytroShkvyra. I don't
see where you use different timeouts for the connection. The only thing which
seems to be modified is that the `JobManager` now understands the
`GetClusterStatus` message. Could you please clarify what this PR tries to
achieve?
> Fail fast if trying to submit a job to a non-existing Flink cluster
> ---
>
> Key: FLINK-5476
> URL: https://issues.apache.org/jira/browse/FLINK-5476
> Project: Flink
> Issue Type: Improvement
> Components: Client
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Till Rohrmann
>Assignee: Dmytro Shkvyra
>Priority: Minor
>
> In case of entering the wrong job manager address when submitting a job via
> {{flink run}}, the {{JobClientActor}} waits per default {{60 s}} until a
> {{JobClientActorConnectionException}}, indicating that the {{JobManager}} is
> no longer reachable, is thrown. In order to fail fast in case of wrong
> connection information, we could change it such that it uses initially a much
> lower timeout and only increases the timeout if it had at least once
> successfully connected to a {{JobManager}} before.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink issue #3753: [FLINK-5476] Fail fast if trying to submit a job to a non...
Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/3753 The code does not seem to match the PR description @DmytroShkvyra. I don't see where you use different timeouts for the connection. The only thing which seems to be modified is that the `JobManager` now understands the `GetClusterStatus` message. Could you please clarify what this PR tries to achieve? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-6414) Use scala.binary.version in place of change-scala-version.sh
Greg Hogan created FLINK-6414:
-
Summary: Use scala.binary.version in place of
change-scala-version.sh
Key: FLINK-6414
URL: https://issues.apache.org/jira/browse/FLINK-6414
Project: Flink
Issue Type: Improvement
Components: Build System
Affects Versions: 1.3.0
Reporter: Greg Hogan
Assignee: Greg Hogan
Recent commits have failed to modify {{change-scala-version.sh}} resulting in
broken builds for {{scala-2.11}}. It looks like we can remove the need for this
script by replacing hard-coded references to the Scala version with Flink's
maven variable {{scala.binary.version}}.
I had initially realized that the change script is [only used for
building|https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/building.html#scala-versions]
and not for switching the IDE environment.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6411) YarnApplicationMasterRunner should not interfere with RunningJobsRegistry
[
https://issues.apache.org/jira/browse/FLINK-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988820#comment-15988820
]
ASF GitHub Bot commented on FLINK-6411:
---
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/3797
Merging this PR since changes are trivial and Travis gave green light.
> YarnApplicationMasterRunner should not interfere with RunningJobsRegistry
> -
>
> Key: FLINK-6411
> URL: https://issues.apache.org/jira/browse/FLINK-6411
> Project: Flink
> Issue Type: Bug
> Components: Distributed Coordination
>Affects Versions: 1.3.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Fix For: 1.3.0
>
>
> The {{YarnApplicationMasterRunner}} removes the running job from the
> {{RunningJobsRegistry}} when it is shut down. This should not be its
> responsibility and rather be delegated to the {{JobManagerRunner}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink issue #3797: [FLINK-6411] [flip-6] Remove job removal from RunningJobs...
Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/3797 Merging this PR since changes are trivial and Travis gave green light. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5810) Harden SlotManager
[
https://issues.apache.org/jira/browse/FLINK-5810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988817#comment-15988817
]
ASF GitHub Bot commented on FLINK-5810:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/3394
> Harden SlotManager
> --
>
> Key: FLINK-5810
> URL: https://issues.apache.org/jira/browse/FLINK-5810
> Project: Flink
> Issue Type: Sub-task
> Components: Distributed Coordination
>Affects Versions: 1.3.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Fix For: 1.3.0
>
>
> Harden the {{SlotManager}} logic to better cope with lost messages.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
