[GitHub] flink issue #2498: [FLINK-4619] - JobManager does not answer to client when ...
Github user mproch commented on the issue: https://github.com/apache/flink/pull/2498 @StephanEwen can I retrigger this or whatever? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4619) JobManager does not answer to client when restore from savepoint fails
[ https://issues.apache.org/jira/browse/FLINK-4619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15581389#comment-15581389 ] ASF GitHub Bot commented on FLINK-4619: --- Github user mproch commented on the issue: https://github.com/apache/flink/pull/2498 @StephanEwen can I retrigger this or whatever? > JobManager does not answer to client when restore from savepoint fails > -- > > Key: FLINK-4619 > URL: https://issues.apache.org/jira/browse/FLINK-4619 > Project: Flink > Issue Type: Bug >Affects Versions: 1.1.0, 1.1.1, 1.1.2 >Reporter: Maciej Prochniak > Fix For: 1.2.0, 1.1.3 > > > When savepoint used is incompatible with currently deployed process, the job > manager never returns (jobInfo.notifyClients is not invoked in one of > try-catch blocks) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4839) JobManager handle TaskManager's slot offering
Kurt Young created FLINK-4839: - Summary: JobManager handle TaskManager's slot offering Key: FLINK-4839 URL: https://issues.apache.org/jira/browse/FLINK-4839 Project: Flink Issue Type: Sub-task Reporter: Kurt Young Assignee: Kurt Young JobManager receives the TaskManager's slot offers, and decide which slots to accept. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4838) remove STREAM keyword in StreamSQLExample
[
https://issues.apache.org/jira/browse/FLINK-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15581206#comment-15581206
]
ASF GitHub Bot commented on FLINK-4838:
---
GitHub user manuzhang opened a pull request:
https://github.com/apache/flink/pull/2645
[FLINK-4838] remote STREAM keyword in StreamSQLExample
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/manuzhang/flink remove_STREAM
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2645.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2645
commit 8f9f6f5602b723258d591e7cb84ccf66cf07f848
Author: manuzhang
Date: 2016-10-17T04:48:35Z
[FLINK-4838] remote STREAM keyword in StreamSQLExample
> remove STREAM keyword in StreamSQLExample
> -
>
> Key: FLINK-4838
> URL: https://issues.apache.org/jira/browse/FLINK-4838
> Project: Flink
> Issue Type: Bug
> Components: Examples, Table API & SQL
>Reporter: Manu Zhang
>Priority: Minor
>
> After FLINK-4546, "STREAM" keyword should be removed from SQL query in
> StreamSQLExample
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request #2645: [FLINK-4838] remote STREAM keyword in StreamSQLExa...
GitHub user manuzhang opened a pull request:
https://github.com/apache/flink/pull/2645
[FLINK-4838] remote STREAM keyword in StreamSQLExample
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/manuzhang/flink remove_STREAM
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2645.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2645
commit 8f9f6f5602b723258d591e7cb84ccf66cf07f848
Author: manuzhang
Date: 2016-10-17T04:48:35Z
[FLINK-4838] remote STREAM keyword in StreamSQLExample
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Created] (FLINK-4838) remove STREAM keyword in StreamSQLExample
Manu Zhang created FLINK-4838: - Summary: remove STREAM keyword in StreamSQLExample Key: FLINK-4838 URL: https://issues.apache.org/jira/browse/FLINK-4838 Project: Flink Issue Type: Bug Components: Examples, Table API & SQL Reporter: Manu Zhang Priority: Minor After FLINK-4546, "STREAM" keyword should be removed from SQL query in StreamSQLExample -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2644: [FLINK-4837] flink-streaming-akka source connector
GitHub user sbcd90 opened a pull request: https://github.com/apache/flink/pull/2644 [FLINK-4837] flink-streaming-akka source connector This PR is created to propose the idea of having a `flink-streaming-akka source connector`. The source connector can be used to receive messages from an Akka feeder or publisher actor & these messages can then be processed using flink streaming. The source connector has the following features. - It can supports several different message formats like iterable data, bytes array & data with timestamp. - It can send back acknowledgements to the feeder actor. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sbcd90/flink flink-akka Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2644.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2644 commit c05274058a2a7f152e668ea464e257ca9dc5aac0 Author: Subhobrata Dey Date: 2016-10-15T22:09:56Z [FLINK-4837] flink-streaming-akka source connector --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4837) flink-streaming-akka source connector
[ https://issues.apache.org/jira/browse/FLINK-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15581175#comment-15581175 ] ASF GitHub Bot commented on FLINK-4837: --- GitHub user sbcd90 opened a pull request: https://github.com/apache/flink/pull/2644 [FLINK-4837] flink-streaming-akka source connector This PR is created to propose the idea of having a `flink-streaming-akka source connector`. The source connector can be used to receive messages from an Akka feeder or publisher actor & these messages can then be processed using flink streaming. The source connector has the following features. - It can supports several different message formats like iterable data, bytes array & data with timestamp. - It can send back acknowledgements to the feeder actor. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sbcd90/flink flink-akka Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2644.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2644 commit c05274058a2a7f152e668ea464e257ca9dc5aac0 Author: Subhobrata Dey Date: 2016-10-15T22:09:56Z [FLINK-4837] flink-streaming-akka source connector > flink-streaming-akka source connector > - > > Key: FLINK-4837 > URL: https://issues.apache.org/jira/browse/FLINK-4837 > Project: Flink > Issue Type: New Feature > Components: Streaming Connectors >Reporter: Subhobrata Dey > > Hello, > This issue is created to propose the idea of having a flink-streaming-akka > source connector. > The source connector can be used to receive messages from an Akka feeder or > publisher actor & these messages can then be processed using flink streaming. > The source connector has the following features. > 1. It can supports several different message formats like iterable data, > bytes array & data with timestamp. > 2. It can send back acknowledgements to the feeder actor. > Thanks & regards, > Subhobrata -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4837) flink-streaming-akka source connector
Subhobrata Dey created FLINK-4837: - Summary: flink-streaming-akka source connector Key: FLINK-4837 URL: https://issues.apache.org/jira/browse/FLINK-4837 Project: Flink Issue Type: New Feature Components: Streaming Connectors Reporter: Subhobrata Dey Hello, This issue is created to propose the idea of having a flink-streaming-akka source connector. The source connector can be used to receive messages from an Akka feeder or publisher actor & these messages can then be processed using flink streaming. The source connector has the following features. 1. It can supports several different message formats like iterable data, bytes array & data with timestamp. 2. It can send back acknowledgements to the feeder actor. Thanks & regards, Subhobrata -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4723) Unify behaviour of committed offsets to Kafka / ZK for Kafka 0.8 and 0.9 consumer
[
https://issues.apache.org/jira/browse/FLINK-4723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15581104#comment-15581104
]
ASF GitHub Bot commented on FLINK-4723:
---
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/2580
Merging this once tests turn green.
> Unify behaviour of committed offsets to Kafka / ZK for Kafka 0.8 and 0.9
> consumer
> -
>
> Key: FLINK-4723
> URL: https://issues.apache.org/jira/browse/FLINK-4723
> Project: Flink
> Issue Type: Improvement
> Components: Kafka Connector
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
> Fix For: 1.2.0, 1.1.3
>
>
> The proper "behaviour" of the offsets committed back to Kafka / ZK should be
> "the next offset that consumers should read (in Kafka terms, the 'position')".
> This is already fixed for the 0.9 consumer by FLINK-4618, by incrementing the
> committed offsets back to Kafka by the 0.9 by 1, so that the internal
> {{KafkaConsumer}} picks up the correct start position when committed offsets
> are present. This fix was required because the start position from committed
> offsets was implicitly determined with Kafka 0.9 APIs.
> However, since the 0.8 consumer handles offset committing and start position
> using Flink's own {{ZookeeperOffsetHandler}} and not Kafka's high-level APIs,
> the 0.8 consumer did not require a fix.
> I propose to still unify the behaviour of committed offsets across 0.8 and
> 0.9 to the definition above.
> Otherwise, if users in any case first uses the 0.8 consumer to read data and
> have Flink-committed offsets in ZK, and then uses a high-level 0.8 Kafka
> consumer to read the same topic in a non-Flink application, the first record
> will be duplicate (because, like described above, Kafka high-level consumers
> expect the committed offsets to be "the next record to process" and not "the
> last processed record").
> This requires incrementing the committed ZK offsets in 0.8 to also be
> incremented by 1, and changing how Flink internal offsets are initialized
> with accordance to the acquired ZK offsets.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink issue #2580: [FLINK-4723] [kafka-connector] Unify committed offsets to...
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2580 Merging this once tests turn green. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4391) Provide support for asynchronous operations over streams
[
https://issues.apache.org/jira/browse/FLINK-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15581097#comment-15581097
]
ASF GitHub Bot commented on FLINK-4391:
---
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r83573944
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollector.java
---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+
+import java.util.List;
+
+/**
+ * {@link AsyncCollector} collects data / error in user codes while
processing async i/o.
+ *
+ * @param Input type
+ * @param Output type
+ */
+@Internal
+public class AsyncCollector {
+ private List result;
+ private Throwable error;
+
+ private boolean isDone = false;
+
+ private AsyncCollectorBuffer buffer;
+
+ public AsyncCollector(AsyncCollectorBuffer buffer) {
+ this.buffer = buffer;
+ }
+
+ public AsyncCollector(AsyncCollectorBuffer buffer, boolean
isDone) {
+ this(buffer);
+ this.isDone = isDone;
+ }
+
+ /**
+* Set result
+* @param result A list of results.
+*/
+ public void collect(List result) {
+ this.result = result;
+ isDone = true;
+ buffer.mark(this);
+ }
+
+ /**
+* Set error
+* @param error A Throwable object.
+*/
+ public void collect(Throwable error) {
+ this.error = error;
+ isDone = true;
+ buffer.mark(this);
+ }
+
+ /**
+* Get result. Throw RuntimeException while encountering an error.
+*
+* @return A List of result.
+* @throws RuntimeException RuntimeException wrapping errors from user
codes.
+*/
+ public List getResult() throws RuntimeException {
--- End diff --
That makes me clarify about how to use RuntimeException ;D
I prefer using IOException, since the error is from I/O process. I will
change RuntimeException into IOException in the codes.
> Provide support for asynchronous operations over streams
>
>
> Key: FLINK-4391
> URL: https://issues.apache.org/jira/browse/FLINK-4391
> Project: Flink
> Issue Type: New Feature
> Components: DataStream API
>Reporter: Jamie Grier
>Assignee: david.wang
>
> Many Flink users need to do asynchronous processing driven by data from a
> DataStream. The classic example would be joining against an external
> database in order to enrich a stream with extra information.
> It would be nice to add general support for this type of operation in the
> Flink API. Ideally this could simply take the form of a new operator that
> manages async operations, keeps so many of them in flight, and then emits
> results to downstream operators as the async operations complete.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r83573944
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncCollector.java
---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.async;
+
+import org.apache.flink.annotation.Internal;
+
+import java.util.List;
+
+/**
+ * {@link AsyncCollector} collects data / error in user codes while
processing async i/o.
+ *
+ * @param Input type
+ * @param Output type
+ */
+@Internal
+public class AsyncCollector {
+ private List result;
+ private Throwable error;
+
+ private boolean isDone = false;
+
+ private AsyncCollectorBuffer buffer;
+
+ public AsyncCollector(AsyncCollectorBuffer buffer) {
+ this.buffer = buffer;
+ }
+
+ public AsyncCollector(AsyncCollectorBuffer buffer, boolean
isDone) {
+ this(buffer);
+ this.isDone = isDone;
+ }
+
+ /**
+* Set result
+* @param result A list of results.
+*/
+ public void collect(List result) {
+ this.result = result;
+ isDone = true;
+ buffer.mark(this);
+ }
+
+ /**
+* Set error
+* @param error A Throwable object.
+*/
+ public void collect(Throwable error) {
+ this.error = error;
+ isDone = true;
+ buffer.mark(this);
+ }
+
+ /**
+* Get result. Throw RuntimeException while encountering an error.
+*
+* @return A List of result.
+* @throws RuntimeException RuntimeException wrapping errors from user
codes.
+*/
+ public List getResult() throws RuntimeException {
--- End diff --
That makes me clarify about how to use RuntimeException ;D
I prefer using IOException, since the error is from I/O process. I will
change RuntimeException into IOException in the codes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-4391) Provide support for asynchronous operations over streams
[
https://issues.apache.org/jira/browse/FLINK-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15581083#comment-15581083
]
ASF GitHub Bot commented on FLINK-4391:
---
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r83573419
--- Diff:
flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java
---
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Example to illustrates how to use {@link
org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+* A checkpointed source.
+*/
+ private static class SimpleSource implements SourceFunction,
Checkpointed {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long
checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ Thread.sleep(10);
+ }
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1
to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
+
+ // configurations for the job
+ String statePath = args[0];
+ String cpMode = args[1];
+ int maxCount = Integer.valueOf(args[2]);
+ final int sleepFactor = Integer.valueOf(args[3]);
+ final
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r83573419
--- Diff:
flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java
---
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Example to illustrates how to use {@link
org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+* A checkpointed source.
+*/
+ private static class SimpleSource implements SourceFunction,
Checkpointed {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long
checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ Thread.sleep(10);
+ }
+ }
+ }
+
+ @Override
+ public void cancel() {
+ isRunning = false;
+ }
+ }
+
+
+ public static void main(String[] args) throws Exception {
+
+ // obtain execution environment and set setBufferTimeout to 1
to enable
+ // continuous flushing of the output buffers (lowest latency)
+ StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment()
+ .setBufferTimeout(1);
+
+ // configurations for the job
+ String statePath = args[0];
+ String cpMode = args[1];
+ int maxCount = Integer.valueOf(args[2]);
+ final int sleepFactor = Integer.valueOf(args[3]);
+ final float failRatio = Float.valueOf(args[4]);
+ String mode = args[5];
+ int taskNum = Integer.valueOf(args[6]);
+ String timeType = args[7];
+
+ // setup state and checkpoint mode
+ en
[jira] [Commented] (FLINK-4547) when call connect method in AkkaRpcService using same address and same rpc gateway class, the returned gateways are equal with respect to equals and hashCode
[ https://issues.apache.org/jira/browse/FLINK-4547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15581039#comment-15581039 ] ASF GitHub Bot commented on FLINK-4547: --- Github user beyond1920 closed the pull request at: https://github.com/apache/flink/pull/2455 > when call connect method in AkkaRpcService using same address and same rpc > gateway class, the returned gateways are equal with respect to equals and > hashCode > - > > Key: FLINK-4547 > URL: https://issues.apache.org/jira/browse/FLINK-4547 > Project: Flink > Issue Type: Sub-task > Components: Distributed Coordination >Reporter: zhangjing >Assignee: zhangjing > > Now every time call connect method in AkkaRpcService class using same address > and same rpc gateway class, the return gateway object is totally different > with each other which equals and hashcode are not same. > Maybe it’s reasonable to have the same result (equals return true, and > hashcode is same) when using the same address and same Gateway class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2455: [FLINK-4547] [cluster management] when call connec...
Github user beyond1920 closed the pull request at: https://github.com/apache/flink/pull/2455 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4391) Provide support for asynchronous operations over streams
[
https://issues.apache.org/jira/browse/FLINK-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15581018#comment-15581018
]
ASF GitHub Bot commented on FLINK-4391:
---
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r83571326
--- Diff:
flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java
---
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Example to illustrates how to use {@link
org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+* A checkpointed source.
+*/
+ private static class SimpleSource implements SourceFunction,
Checkpointed {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long
checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ Thread.sleep(10);
--- End diff --
That is right. It will block checkpoint for extra 0.1 second.
> Provide support for asynchronous operations over streams
>
>
> Key: FLINK-4391
> URL: https://issues.apache.org/jira/browse/FLINK-4391
> Project: Flink
> Issue Type: New Feature
> Components: DataStream API
>Reporter: Jamie Grier
>Assignee: david.wang
>
> Many Flink users need to do asynchronous processing driven by data from a
> DataStream. The classic example would be joining against an external
> database in order to enrich a stream with extra information.
> It would be nice to add general support for this type of operation in the
> Flink API. Ideally this could simply take the form of a new operator that
> manages async operations, keeps so many
[GitHub] flink pull request #2629: [FLINK-4391] Provide support for asynchronous oper...
Github user bjlovegithub commented on a diff in the pull request:
https://github.com/apache/flink/pull/2629#discussion_r83571326
--- Diff:
flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java
---
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.async;
+
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.checkpoint.Checkpointed;
+import org.apache.flink.streaming.api.datastream.AsyncDataStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.async.AsyncCollector;
+import org.apache.flink.util.Collector;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Example to illustrates how to use {@link
org.apache.flink.streaming.api.functions.async.AsyncFunction}
+ */
+public class AsyncIOExample {
+
+ /**
+* A checkpointed source.
+*/
+ private static class SimpleSource implements SourceFunction,
Checkpointed {
+ private static final long serialVersionUID = 1L;
+
+ private volatile boolean isRunning = true;
+ private int counter = 0;
+ private int start = 0;
+
+ @Override
+ public void restoreState(Integer state) throws Exception {
+ this.start = state;
+ }
+
+ @Override
+ public Integer snapshotState(long checkpointId, long
checkpointTimestamp) throws Exception {
+ return start;
+ }
+
+ public SimpleSource(int maxNum) {
+ this.counter = maxNum;
+ }
+
+ @Override
+ public void run(SourceContext ctx) throws Exception {
+ while (start < counter && isRunning) {
+ synchronized (ctx.getCheckpointLock()) {
+ ctx.collect(start);
+ ++start;
+ Thread.sleep(10);
--- End diff --
That is right. It will block checkpoint for extra 0.1 second.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Assigned] (FLINK-4627) Use Flink's PropertiesUtil in Kinesis connector to extract typed values from config properties
[
https://issues.apache.org/jira/browse/FLINK-4627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neelesh Srinivas Salian reassigned FLINK-4627:
--
Assignee: Neelesh Srinivas Salian
> Use Flink's PropertiesUtil in Kinesis connector to extract typed values from
> config properties
> ---
>
> Key: FLINK-4627
> URL: https://issues.apache.org/jira/browse/FLINK-4627
> Project: Flink
> Issue Type: Improvement
> Components: Kinesis Connector
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Neelesh Srinivas Salian
>Priority: Trivial
> Fix For: 1.2.0
>
>
> Right now value extraction from config properties in the Kinesis connector is
> using the plain methods from {{java.util.Properties}} with string parsing.
> We can use Flink's utility {{PropertiesUtil}} to do this, with lesser lines
> of and more readable code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (FLINK-4659) Potential resource leak due to unclosed InputStream in SecurityContext#populateSystemSecurityProperties()
[
https://issues.apache.org/jira/browse/FLINK-4659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neelesh Srinivas Salian reassigned FLINK-4659:
--
Assignee: Neelesh Srinivas Salian
> Potential resource leak due to unclosed InputStream in
> SecurityContext#populateSystemSecurityProperties()
> -
>
> Key: FLINK-4659
> URL: https://issues.apache.org/jira/browse/FLINK-4659
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Neelesh Srinivas Salian
>Priority: Minor
>
> {code}
> try {
> Path jaasConfPath =
> Files.createTempFile(JAAS_CONF_FILENAME, "");
> InputStream jaasConfStream =
> SecurityContext.class.getClassLoader().getResourceAsStream(JAAS_CONF_FILENAME);
> Files.copy(jaasConfStream, jaasConfPath,
> StandardCopyOption.REPLACE_EXISTING);
> jaasConfFile = jaasConfPath.toFile();
> jaasConfFile.deleteOnExit();
> } catch (IOException e) {
> throw new RuntimeException("SASL auth is enabled for
> ZK but unable to " +
> {code}
> jaasConfStream should be closed in finally block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (FLINK-4753) Kafka 0.8 connector's Periodic Offset Committer should synchronize on checkpoint lock
[
https://issues.apache.org/jira/browse/FLINK-4753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai closed FLINK-4753.
--
Resolution: Not A Bug
> Kafka 0.8 connector's Periodic Offset Committer should synchronize on
> checkpoint lock
> -
>
> Key: FLINK-4753
> URL: https://issues.apache.org/jira/browse/FLINK-4753
> Project: Flink
> Issue Type: Bug
> Components: Kafka Connector
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
> Fix For: 1.2.0
>
>
> The {{PeriodicOffsetCommitter}} doesn't synchronize on the checkpoint lock
> when creating a copy of the partition offsets in the current state.
> I propose to pass the checkpoint lock into{{PeriodicOffsetCommitter}}, and
> let {{PeriodicOffsetCommitter}} use {{Kafka08Fetcher#snapshotCurrentState()}}
> to make the current state copy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request #2604: [FLINK-4753] [kafka] PeriodicOffsetCommitter shoul...
Github user tzulitai closed the pull request at: https://github.com/apache/flink/pull/2604 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4753) Kafka 0.8 connector's Periodic Offset Committer should synchronize on checkpoint lock
[
https://issues.apache.org/jira/browse/FLINK-4753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15580926#comment-15580926
]
ASF GitHub Bot commented on FLINK-4753:
---
Github user tzulitai closed the pull request at:
https://github.com/apache/flink/pull/2604
> Kafka 0.8 connector's Periodic Offset Committer should synchronize on
> checkpoint lock
> -
>
> Key: FLINK-4753
> URL: https://issues.apache.org/jira/browse/FLINK-4753
> Project: Flink
> Issue Type: Bug
> Components: Kafka Connector
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
> Fix For: 1.2.0
>
>
> The {{PeriodicOffsetCommitter}} doesn't synchronize on the checkpoint lock
> when creating a copy of the partition offsets in the current state.
> I propose to pass the checkpoint lock into{{PeriodicOffsetCommitter}}, and
> let {{PeriodicOffsetCommitter}} use {{Kafka08Fetcher#snapshotCurrentState()}}
> to make the current state copy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4753) Kafka 0.8 connector's Periodic Offset Committer should synchronize on checkpoint lock
[
https://issues.apache.org/jira/browse/FLINK-4753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15580925#comment-15580925
]
ASF GitHub Bot commented on FLINK-4753:
---
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/2604
@rmetzger No worries, it's okay ;) Closing this now.
> Kafka 0.8 connector's Periodic Offset Committer should synchronize on
> checkpoint lock
> -
>
> Key: FLINK-4753
> URL: https://issues.apache.org/jira/browse/FLINK-4753
> Project: Flink
> Issue Type: Bug
> Components: Kafka Connector
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
> Fix For: 1.2.0
>
>
> The {{PeriodicOffsetCommitter}} doesn't synchronize on the checkpoint lock
> when creating a copy of the partition offsets in the current state.
> I propose to pass the checkpoint lock into{{PeriodicOffsetCommitter}}, and
> let {{PeriodicOffsetCommitter}} use {{Kafka08Fetcher#snapshotCurrentState()}}
> to make the current state copy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4652) Don't pass credentials explicitly to AmazonClient - use credentials provider instead
[ https://issues.apache.org/jira/browse/FLINK-4652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15580923#comment-15580923 ] ASF GitHub Bot commented on FLINK-4652: --- Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2635 Thanks for fixing this. +1, looks good to me. > Don't pass credentials explicitly to AmazonClient - use credentials provider > instead > > > Key: FLINK-4652 > URL: https://issues.apache.org/jira/browse/FLINK-4652 > Project: Flink > Issue Type: Bug > Components: Kinesis Connector >Affects Versions: 1.2.0 >Reporter: Kristian Frøhlich Hansen >Assignee: Greg Hogan >Priority: Minor > Fix For: 1.2.0 > > > By using the credentials explicitly we are responsible for checking and > refreshing credentials before they expire. If no refreshment is done we will > encounter AmazonServiceException: 'The security token included in the request > is expired'. To utilize automatic refreshment of credentials pass the > AWSCredentialsProvider direclty to AmazonClient by removing the > getCredentials() call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2604: [FLINK-4753] [kafka] PeriodicOffsetCommitter should synch...
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2604 @rmetzger No worries, it's okay ;) Closing this now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2635: [FLINK-4652] [streaming connectors] Automatically refresh...
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2635 Thanks for fixing this. +1, looks good to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4659) Potential resource leak due to unclosed InputStream in SecurityContext#populateSystemSecurityProperties()
[
https://issues.apache.org/jira/browse/FLINK-4659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15580919#comment-15580919
]
Ted Yu commented on FLINK-4659:
---
Please go ahead.
> Potential resource leak due to unclosed InputStream in
> SecurityContext#populateSystemSecurityProperties()
> -
>
> Key: FLINK-4659
> URL: https://issues.apache.org/jira/browse/FLINK-4659
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> try {
> Path jaasConfPath =
> Files.createTempFile(JAAS_CONF_FILENAME, "");
> InputStream jaasConfStream =
> SecurityContext.class.getClassLoader().getResourceAsStream(JAAS_CONF_FILENAME);
> Files.copy(jaasConfStream, jaasConfPath,
> StandardCopyOption.REPLACE_EXISTING);
> jaasConfFile = jaasConfPath.toFile();
> jaasConfFile.deleteOnExit();
> } catch (IOException e) {
> throw new RuntimeException("SASL auth is enabled for
> ZK but unable to " +
> {code}
> jaasConfStream should be closed in finally block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request #2643: [hotfix] Treat taskManager's rpc address and locat...
GitHub user KurtYoung opened a pull request: https://github.com/apache/flink/pull/2643 [hotfix] Treat taskManager's rpc address and location separately I forgot that rpc address and the address from TaskManagerLocation are not the same thing, we should use rpc address to communicate. I revert some codes to Till's original version, sorry about this. You can merge this pull request into a Git repository by running: $ git pull https://github.com/KurtYoung/flink flip-6 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2643.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2643 commit 7e30eab9174ca179809ade60c47d9af54a3717de Author: Kurt Young Date: 2016-10-17T01:38:46Z [hotfix] Treat taskManager's rpc address and location separately --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3999) Rename the `running` flag in the drivers to `canceled`
[
https://issues.apache.org/jira/browse/FLINK-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15580868#comment-15580868
]
ASF GitHub Bot commented on FLINK-3999:
---
GitHub user nssalian opened a pull request:
https://github.com/apache/flink/pull/2642
[FLINK-3999]: Rename the `running` flag in the drivers to `canceled`
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-3999]
Rename the `running` flag in the drivers to `canceled`")
- Added the cancelled flag instead of the running flag to replace the
functionality since it was not truly "running".
- Modified the operators to have the cancelled instead of running even in
tests
- [ ] Documentation
- Modified the Example for the Loading Data where the import statement
needed to be changed.
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
- Ran a Travis build and local mvn tests as well.
@ggevay if you could help review. Thank you.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nssalian/flink FLINK-3999
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2642.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2642
commit 65b3ff78f100ed1b13ec2fcc727f4869823b5918
Author: Neelesh Srinivas Salian
Date: 2016-09-12T22:38:06Z
FLINK-4614: Changed the version from 1.2-SNAPSHOT to 1.2 to help correct
docs in 1.2.0
commit 2ef440161699943cb3cc38eccc6ef39a1bd512a1
Author: Neelesh Srinivas Salian
Date: 2016-10-16T20:41:00Z
FLINK-3999: Modified the running flag for the runtime module to cancelled
to depict correctness
commit 46d91b0052971e5116c674a350ff38aa692616bb
Author: Neelesh Srinivas Salian
Date: 2016-10-16T23:07:45Z
FLINK-3999: Fixed Tests
commit 1433a5db5ad13aef3f76e8ab4b2fa131272fe59d
Author: Neelesh Srinivas Salian
Date: 2016-10-16T23:58:33Z
FLINK-3999: More changes to fix tests
commit ba4f429043f3985db5ab37a3902a6fee15a7440e
Author: Neelesh Srinivas Salian
Date: 2016-10-17T01:12:43Z
FLINK-3999: Updated config version
> Rename the `running` flag in the drivers to `canceled`
> --
>
> Key: FLINK-3999
> URL: https://issues.apache.org/jira/browse/FLINK-3999
> Project: Flink
> Issue Type: Bug
> Components: Local Runtime
>Reporter: Gabor Gevay
>Assignee: Neelesh Srinivas Salian
>Priority: Trivial
>
> The name of the {{running}} flag in the drivers doesn't reflect its usage:
> when the operator just stops normally, then it is not running anymore, but
> the {{running}} flag will still be true, since the {{running}} flag is only
> set when cancelling.
> It should be renamed, and the value inverted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request #2642: [FLINK-3999]: Rename the `running` flag in the dri...
GitHub user nssalian opened a pull request:
https://github.com/apache/flink/pull/2642
[FLINK-3999]: Rename the `running` flag in the drivers to `canceled`
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-3999]
Rename the `running` flag in the drivers to `canceled`")
- Added the cancelled flag instead of the running flag to replace the
functionality since it was not truly "running".
- Modified the operators to have the cancelled instead of running even in
tests
- [ ] Documentation
- Modified the Example for the Loading Data where the import statement
needed to be changed.
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
- Ran a Travis build and local mvn tests as well.
@ggevay if you could help review. Thank you.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nssalian/flink FLINK-3999
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2642.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2642
commit 65b3ff78f100ed1b13ec2fcc727f4869823b5918
Author: Neelesh Srinivas Salian
Date: 2016-09-12T22:38:06Z
FLINK-4614: Changed the version from 1.2-SNAPSHOT to 1.2 to help correct
docs in 1.2.0
commit 2ef440161699943cb3cc38eccc6ef39a1bd512a1
Author: Neelesh Srinivas Salian
Date: 2016-10-16T20:41:00Z
FLINK-3999: Modified the running flag for the runtime module to cancelled
to depict correctness
commit 46d91b0052971e5116c674a350ff38aa692616bb
Author: Neelesh Srinivas Salian
Date: 2016-10-16T23:07:45Z
FLINK-3999: Fixed Tests
commit 1433a5db5ad13aef3f76e8ab4b2fa131272fe59d
Author: Neelesh Srinivas Salian
Date: 2016-10-16T23:58:33Z
FLINK-3999: More changes to fix tests
commit ba4f429043f3985db5ab37a3902a6fee15a7440e
Author: Neelesh Srinivas Salian
Date: 2016-10-17T01:12:43Z
FLINK-3999: Updated config version
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-4659) Potential resource leak due to unclosed InputStream in SecurityContext#populateSystemSecurityProperties()
[
https://issues.apache.org/jira/browse/FLINK-4659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15580858#comment-15580858
]
Neelesh Srinivas Salian commented on FLINK-4659:
I can work on this one [~tedyu], if no one else is already.
> Potential resource leak due to unclosed InputStream in
> SecurityContext#populateSystemSecurityProperties()
> -
>
> Key: FLINK-4659
> URL: https://issues.apache.org/jira/browse/FLINK-4659
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> try {
> Path jaasConfPath =
> Files.createTempFile(JAAS_CONF_FILENAME, "");
> InputStream jaasConfStream =
> SecurityContext.class.getClassLoader().getResourceAsStream(JAAS_CONF_FILENAME);
> Files.copy(jaasConfStream, jaasConfPath,
> StandardCopyOption.REPLACE_EXISTING);
> jaasConfFile = jaasConfPath.toFile();
> jaasConfFile.deleteOnExit();
> } catch (IOException e) {
> throw new RuntimeException("SASL auth is enabled for
> ZK but unable to " +
> {code}
> jaasConfStream should be closed in finally block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4792) Update documentation - FlinkML/QuickStart Guide
[
https://issues.apache.org/jira/browse/FLINK-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15580842#comment-15580842
]
ASF GitHub Bot commented on FLINK-4792:
---
GitHub user nssalian opened a pull request:
https://github.com/apache/flink/pull/2641
[FLINK-4792] [Docs] Update documentation - FlinkML/QuickStart Guide
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-4792]
Update documentation - FlinkML/QuickStart Guide")
- [ ] Documentation
- Modified the example to correct the import statement in Loading Data
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nssalian/flink FLINK-4792-code
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2641.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2641
commit 65b3ff78f100ed1b13ec2fcc727f4869823b5918
Author: Neelesh Srinivas Salian
Date: 2016-09-12T22:38:06Z
FLINK-4614: Changed the version from 1.2-SNAPSHOT to 1.2 to help correct
docs in 1.2.0
commit f07e5bfaea21b6c3bf562d1f167c3ec50948773c
Author: Neelesh Srinivas Salian
Date: 2016-10-17T00:54:20Z
FLINK-4792: Changed the import statement for the Loading data example
> Update documentation - FlinkML/QuickStart Guide
> ---
>
> Key: FLINK-4792
> URL: https://issues.apache.org/jira/browse/FLINK-4792
> Project: Flink
> Issue Type: Improvement
>Reporter: Thomas FOURNIER
>Priority: Trivial
>
> Hi,
> I'm going through the first steps of FlinkML/QuickStart guide:
> https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/libs/ml/quickstart.html
> When using env.readCsvFile:
> val env = ExecutionEnvironment.getExecutionEnvironment
> val survival = env.readCsvFile[(String, String, String, String)]("path/data")
> I encounter the following error:
> Error:(17, 69) could not find implicit value for evidence parameter of type
> org.apache.flink.api.common.typeinfo.TypeInformation[(String, String, String,
> String)]
> Error occurred in an application involving default arguments.
> val survival = env.readCsvFile[(String, String, String, String)]("path/data")
> To solve this issue, you need to do the following imports:
> import org.apache.flink.api.scala._ ( instead of import
> org.apache.flink.api.scala.ExecutionEnvironment as documented).
> I think it would be relevant to update the documentation, even if this point
> is mentioned in FAQ.
> Thanks
> Best
> Thomas
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request #2641: [FLINK-4792] [Docs] Update documentation - FlinkML...
GitHub user nssalian opened a pull request:
https://github.com/apache/flink/pull/2641
[FLINK-4792] [Docs] Update documentation - FlinkML/QuickStart Guide
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-4792]
Update documentation - FlinkML/QuickStart Guide")
- [ ] Documentation
- Modified the example to correct the import statement in Loading Data
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nssalian/flink FLINK-4792-code
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2641.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2641
commit 65b3ff78f100ed1b13ec2fcc727f4869823b5918
Author: Neelesh Srinivas Salian
Date: 2016-09-12T22:38:06Z
FLINK-4614: Changed the version from 1.2-SNAPSHOT to 1.2 to help correct
docs in 1.2.0
commit f07e5bfaea21b6c3bf562d1f167c3ec50948773c
Author: Neelesh Srinivas Salian
Date: 2016-10-17T00:54:20Z
FLINK-4792: Changed the import statement for the Loading data example
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Resolved] (FLINK-4351) JobManager handle TaskManager's registration
[ https://issues.apache.org/jira/browse/FLINK-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young resolved FLINK-4351. --- Resolution: Fixed resolved by a19cae3b07963776c07c0aae7bee806004f59429 > JobManager handle TaskManager's registration > > > Key: FLINK-4351 > URL: https://issues.apache.org/jira/browse/FLINK-4351 > Project: Flink > Issue Type: Sub-task > Components: Cluster Management >Reporter: zhangjing >Assignee: Kurt Young > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLINK-4689) Implement a simple slot provider for the new job manager
[ https://issues.apache.org/jira/browse/FLINK-4689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young resolved FLINK-4689. --- Resolution: Fixed resolved by e91b82d3c868e18611064f905b345906f1414f84 > Implement a simple slot provider for the new job manager > > > Key: FLINK-4689 > URL: https://issues.apache.org/jira/browse/FLINK-4689 > Project: Flink > Issue Type: Sub-task > Components: Cluster Management >Reporter: Kurt Young >Assignee: Kurt Young > > In flip-6 branch, we need to adjust existing scheduling model. In the first > step, we should introduce a simple / naive slot provider which just ignore > all the sharing or location constraint, to make whole thing work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4836) Implement Mini Cluster
[
https://issues.apache.org/jira/browse/FLINK-4836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15580584#comment-15580584
]
Stephan Ewen commented on FLINK-4836:
-
Part one implemented in the flip-6 feature branch
> Implement Mini Cluster
> --
>
> Key: FLINK-4836
> URL: https://issues.apache.org/jira/browse/FLINK-4836
> Project: Flink
> Issue Type: New Feature
> Components: Cluster Management
> Environment: FLIP-6 feature branch
>Reporter: Stephan Ewen
>
> This task is to implement the embedded mini cluster (similar to the
> {{LocalFlinkMiniCluster}} based on the new components developed in "Flink
> Improvement Proposal 6"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (FLINK-4835) Implement standalone High Availability Services
[
https://issues.apache.org/jira/browse/FLINK-4835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen closed FLINK-4835.
---
> Implement standalone High Availability Services
> ---
>
> Key: FLINK-4835
> URL: https://issues.apache.org/jira/browse/FLINK-4835
> Project: Flink
> Issue Type: Sub-task
> Components: Cluster Management
> Environment: FLIP-6 feature branch
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>
> This is a version of the {{HighAvailabilityServices}} that is in fact not
> highly available and used for setups where everything runs in one process
> (mini clusters) and simple setups (non-ha)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (FLINK-4835) Implement standalone High Availability Services
[
https://issues.apache.org/jira/browse/FLINK-4835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen resolved FLINK-4835.
-
Resolution: Implemented
Implemented in flip-6 feature branch
> Implement standalone High Availability Services
> ---
>
> Key: FLINK-4835
> URL: https://issues.apache.org/jira/browse/FLINK-4835
> Project: Flink
> Issue Type: Sub-task
> Components: Cluster Management
> Environment: FLIP-6 feature branch
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>
> This is a version of the {{HighAvailabilityServices}} that is in fact not
> highly available and used for setups where everything runs in one process
> (mini clusters) and simple setups (non-ha)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4836) Implement Mini Cluster
Stephan Ewen created FLINK-4836:
---
Summary: Implement Mini Cluster
Key: FLINK-4836
URL: https://issues.apache.org/jira/browse/FLINK-4836
Project: Flink
Issue Type: New Feature
Components: Cluster Management
Environment: FLIP-6 feature branch
Reporter: Stephan Ewen
This task is to implement the embedded mini cluster (similar to the
{{LocalFlinkMiniCluster}} based on the new components developed in "Flink
Improvement Proposal 6"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (FLINK-4657) Implement HighAvailabilityServices based on zookeeper
[ https://issues.apache.org/jira/browse/FLINK-4657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen closed FLINK-4657. --- > Implement HighAvailabilityServices based on zookeeper > - > > Key: FLINK-4657 > URL: https://issues.apache.org/jira/browse/FLINK-4657 > Project: Flink > Issue Type: Sub-task > Components: Cluster Management >Reporter: Kurt Young >Assignee: Kurt Young > > For flip-6, we will have ResourceManager and every JobManager as potential > leader contender and retriever. We should separate them by using different > zookeeper path. > For example, the path could be /leader/resource-manaeger for RM. And for each > JM, the path could be /leader/job-managers/JobID -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4835) Implement standalone High Availability Services
Stephan Ewen created FLINK-4835:
---
Summary: Implement standalone High Availability Services
Key: FLINK-4835
URL: https://issues.apache.org/jira/browse/FLINK-4835
Project: Flink
Issue Type: Sub-task
Components: Cluster Management
Environment: FLIP-6 feature branch
Reporter: Stephan Ewen
Assignee: Stephan Ewen
This is a version of the {{HighAvailabilityServices}} that is in fact not
highly available and used for setups where everything runs in one process (mini
clusters) and simple setups (non-ha)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLINK-4657) Implement HighAvailabilityServices based on zookeeper
[ https://issues.apache.org/jira/browse/FLINK-4657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen updated FLINK-4657: Issue Type: Sub-task (was: New Feature) Parent: FLINK-4834 > Implement HighAvailabilityServices based on zookeeper > - > > Key: FLINK-4657 > URL: https://issues.apache.org/jira/browse/FLINK-4657 > Project: Flink > Issue Type: Sub-task > Components: Cluster Management >Reporter: Kurt Young >Assignee: Kurt Young > > For flip-6, we will have ResourceManager and every JobManager as potential > leader contender and retriever. We should separate them by using different > zookeeper path. > For example, the path could be /leader/resource-manaeger for RM. And for each > JM, the path could be /leader/job-managers/JobID -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4834) Implement unified High Availability Services Abstraction
Stephan Ewen created FLINK-4834: --- Summary: Implement unified High Availability Services Abstraction Key: FLINK-4834 URL: https://issues.apache.org/jira/browse/FLINK-4834 Project: Flink Issue Type: New Feature Components: Cluster Management Environment: FLIP-6 feature branch Reporter: Stephan Ewen -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLINK-2345) Zero Configuration Cluster Setup
[ https://issues.apache.org/jira/browse/FLINK-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen resolved FLINK-2345. - Resolution: Unresolved This needs more concrete suggestions > Zero Configuration Cluster Setup > > > Key: FLINK-2345 > URL: https://issues.apache.org/jira/browse/FLINK-2345 > Project: Flink > Issue Type: New Feature >Reporter: Suminda Dharmasena > > Best for of local clustering is using auto discovery. Also automatic tuning > of cluster configuration parameters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-2345) Zero Configuration Cluster Setup
[ https://issues.apache.org/jira/browse/FLINK-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen closed FLINK-2345. --- > Zero Configuration Cluster Setup > > > Key: FLINK-2345 > URL: https://issues.apache.org/jira/browse/FLINK-2345 > Project: Flink > Issue Type: New Feature >Reporter: Suminda Dharmasena > > Best for of local clustering is using auto discovery. Also automatic tuning > of cluster configuration parameters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4294) Allow access of composite type fields
[
https://issues.apache.org/jira/browse/FLINK-4294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15580345#comment-15580345
]
ASF GitHub Bot commented on FLINK-4294:
---
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/2319#discussion_r83556947
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/FlinkTypeFactory.scala
---
@@ -79,14 +83,27 @@ class FlinkTypeFactory(typeSystem: RelDataTypeSystem)
extends JavaTypeFactoryImp
}
private def createAdvancedType(typeInfo: TypeInformation[_]):
RelDataType = typeInfo match {
-// TODO add specific RelDataTypes
-// for PrimitiveArrayTypeInfo, ObjectArrayTypeInfo, CompositeType
+case ct: CompositeType[_] =>
+ new CompositeRelDataType(ct, this)
+
+// TODO add specific RelDataTypes for PrimitiveArrayTypeInfo,
ObjectArrayTypeInfo
case ti: TypeInformation[_] =>
new GenericRelDataType(typeInfo,
getTypeSystem.asInstanceOf[FlinkTypeSystem])
case ti@_ =>
throw TableException(s"Unsupported type information: $ti")
}
+
+ override def createTypeWithNullability(
+ relDataType: RelDataType,
+ nullable: Boolean)
+: RelDataType = relDataType match {
+case composite: CompositeRelDataType =>
+ // at the moment we do not care about nullability
--- End diff --
`CompositeRelDataType` is the first custom type and thus not supported by
the superclass (`JavaTypeFactory`) that's why I had to override this method.
The super method does some copying logic and sets the nullability of every
field of the record. At the moment every field in the Table API can be null
anyway, so I skipped this logic here until we provide full nullabilty support
through the entire API.
> Allow access of composite type fields
> -
>
> Key: FLINK-4294
> URL: https://issues.apache.org/jira/browse/FLINK-4294
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> Currently all Flink CompositeTypes are treated as GenericRelDataTypes. It
> would be better to access individual fields of composite types, too. e.g.
> {code}
> SELECT composite.name FROM composites
> SELECT tuple.f0 FROM tuples
> 'f0.getField(0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request #2319: [FLINK-4294] [table] Allow access of composite typ...
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/2319#discussion_r83556947
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/FlinkTypeFactory.scala
---
@@ -79,14 +83,27 @@ class FlinkTypeFactory(typeSystem: RelDataTypeSystem)
extends JavaTypeFactoryImp
}
private def createAdvancedType(typeInfo: TypeInformation[_]):
RelDataType = typeInfo match {
-// TODO add specific RelDataTypes
-// for PrimitiveArrayTypeInfo, ObjectArrayTypeInfo, CompositeType
+case ct: CompositeType[_] =>
+ new CompositeRelDataType(ct, this)
+
+// TODO add specific RelDataTypes for PrimitiveArrayTypeInfo,
ObjectArrayTypeInfo
case ti: TypeInformation[_] =>
new GenericRelDataType(typeInfo,
getTypeSystem.asInstanceOf[FlinkTypeSystem])
case ti@_ =>
throw TableException(s"Unsupported type information: $ti")
}
+
+ override def createTypeWithNullability(
+ relDataType: RelDataType,
+ nullable: Boolean)
+: RelDataType = relDataType match {
+case composite: CompositeRelDataType =>
+ // at the moment we do not care about nullability
--- End diff --
`CompositeRelDataType` is the first custom type and thus not supported by
the superclass (`JavaTypeFactory`) that's why I had to override this method.
The super method does some copying logic and sets the nullability of every
field of the record. At the moment every field in the Table API can be null
anyway, so I skipped this logic here until we provide full nullabilty support
through the entire API.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-4294) Allow access of composite type fields
[
https://issues.apache.org/jira/browse/FLINK-4294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15580335#comment-15580335
]
ASF GitHub Bot commented on FLINK-4294:
---
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/2319#discussion_r83556835
--- Diff: docs/dev/table_api.md ---
@@ -1656,6 +1656,29 @@ temporalOverlaps(TIMEPOINT, TEMPORAL, TIMEPOINT,
TEMPORAL)
+
+
+{% highlight java %}
+ANY.flatten()
+{% endhighlight %}
+
+
+Converts a Flink composite type (such as Tuple, POJO, etc.) and
all of its subtypes into a flat representation where every subtype is a
separate field.
+
+
+
+
+
+{% highlight java %}
+COMPOSITE.get(STRING)
--- End diff --
I thought we only agreed that `getField` is too long. Personally I like
this approach more because it can be found in the expression DSL and has a
Scala doc; and it doesn't add further case distinction logic to the validation
layer. Since "$" is valid part of Java identifier we would also restrict the
field naming of POJOs. If there is a consensus I can change it to
"field$substring".
> Allow access of composite type fields
> -
>
> Key: FLINK-4294
> URL: https://issues.apache.org/jira/browse/FLINK-4294
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> Currently all Flink CompositeTypes are treated as GenericRelDataTypes. It
> would be better to access individual fields of composite types, too. e.g.
> {code}
> SELECT composite.name FROM composites
> SELECT tuple.f0 FROM tuples
> 'f0.getField(0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request #2319: [FLINK-4294] [table] Allow access of composite typ...
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/2319#discussion_r83556835
--- Diff: docs/dev/table_api.md ---
@@ -1656,6 +1656,29 @@ temporalOverlaps(TIMEPOINT, TEMPORAL, TIMEPOINT,
TEMPORAL)
+
+
+{% highlight java %}
+ANY.flatten()
+{% endhighlight %}
+
+
+Converts a Flink composite type (such as Tuple, POJO, etc.) and
all of its subtypes into a flat representation where every subtype is a
separate field.
+
+
+
+
+
+{% highlight java %}
+COMPOSITE.get(STRING)
--- End diff --
I thought we only agreed that `getField` is too long. Personally I like
this approach more because it can be found in the expression DSL and has a
Scala doc; and it doesn't add further case distinction logic to the validation
layer. Since "$" is valid part of Java identifier we would also restrict the
field naming of POJOs. If there is a consensus I can change it to
"field$substring".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-3931) Implement Transport Encryption (SSL/TLS)
[ https://issues.apache.org/jira/browse/FLINK-3931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15580160#comment-15580160 ] ASF GitHub Bot commented on FLINK-3931: --- Github user skrishnappa commented on the issue: https://github.com/apache/flink/pull/2518 Have included few more test cases to handle ssl failure scenarios. Also rebased the PR to the latest master > Implement Transport Encryption (SSL/TLS) > > > Key: FLINK-3931 > URL: https://issues.apache.org/jira/browse/FLINK-3931 > Project: Flink > Issue Type: New Feature >Reporter: Eron Wright >Assignee: Suresh Krishnappa > Labels: security > Original Estimate: 1,008h > Remaining Estimate: 1,008h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > To assure privacy and data integrity between Flink components, enable TLS for > all communication channels. As described in the design doc: > - Accept a configured certificate or generate a certificate. > - Enable Akka SSL > - Implement Data Transfer SSL > - Implement Blob Server SSL > - Implement Web UI HTTPS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2518: [FLINK-3931] Implement Transport Encryption (SSL/TLS)
Github user skrishnappa commented on the issue: https://github.com/apache/flink/pull/2518 Have included few more test cases to handle ssl failure scenarios. Also rebased the PR to the latest master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3931) Implement Transport Encryption (SSL/TLS)
[
https://issues.apache.org/jira/browse/FLINK-3931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15580157#comment-15580157
]
ASF GitHub Bot commented on FLINK-3931:
---
Github user skrishnappa commented on a diff in the pull request:
https://github.com/apache/flink/pull/2518#discussion_r83553892
--- Diff:
flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobClientSslTest.java
---
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.blob;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.InetSocketAddress;
+import java.security.MessageDigest;
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.core.fs.Path;
+import org.codehaus.jackson.map.DeserializerFactory;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * This class contains unit tests for the {@link BlobClient} with ssl
enabled.
+ */
+public class BlobClientSslTest {
+
+ /** The buffer size used during the tests in bytes. */
+ private static final int TEST_BUFFER_SIZE = 17 * 1000;
+
+ /** The instance of the BLOB server used during the tests. */
+ private static BlobServer BLOB_SERVER;
+
+ /** The blob service client configuration */
+ private static Configuration clientConfig;
+ /**
+* Starts the BLOB server.
+*/
+ @BeforeClass
+ public static void startServer() {
+ try {
+ Configuration config = new Configuration();
+ config.setBoolean(ConfigConstants.SECURITY_SSL_ENABLED,
true);
+ config.setString(ConfigConstants.SECURITY_SSL_KEYSTORE,
"src/test/resources/local127.keystore");
+
config.setString(ConfigConstants.SECURITY_SSL_KEYSTORE_PASSWORD, "password");
+
config.setString(ConfigConstants.SECURITY_SSL_KEY_PASSWORD, "password");
--- End diff --
Added few more testcases for failure scenarios
> Implement Transport Encryption (SSL/TLS)
>
>
> Key: FLINK-3931
> URL: https://issues.apache.org/jira/browse/FLINK-3931
> Project: Flink
> Issue Type: New Feature
>Reporter: Eron Wright
>Assignee: Suresh Krishnappa
> Labels: security
> Original Estimate: 1,008h
> Remaining Estimate: 1,008h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> To assure privacy and data integrity between Flink components, enable TLS for
> all communication channels. As described in the design doc:
> - Accept a configured certificate or generate a certificate.
> - Enable Akka SSL
> - Implement Data Transfer SSL
> - Implement Blob Server SSL
> - Implement Web UI HTTPS
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request #2518: [FLINK-3931] Implement Transport Encryption (SSL/T...
Github user skrishnappa commented on a diff in the pull request:
https://github.com/apache/flink/pull/2518#discussion_r83553892
--- Diff:
flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobClientSslTest.java
---
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.blob;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.InetSocketAddress;
+import java.security.MessageDigest;
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.core.fs.Path;
+import org.codehaus.jackson.map.DeserializerFactory;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * This class contains unit tests for the {@link BlobClient} with ssl
enabled.
+ */
+public class BlobClientSslTest {
+
+ /** The buffer size used during the tests in bytes. */
+ private static final int TEST_BUFFER_SIZE = 17 * 1000;
+
+ /** The instance of the BLOB server used during the tests. */
+ private static BlobServer BLOB_SERVER;
+
+ /** The blob service client configuration */
+ private static Configuration clientConfig;
+ /**
+* Starts the BLOB server.
+*/
+ @BeforeClass
+ public static void startServer() {
+ try {
+ Configuration config = new Configuration();
+ config.setBoolean(ConfigConstants.SECURITY_SSL_ENABLED,
true);
+ config.setString(ConfigConstants.SECURITY_SSL_KEYSTORE,
"src/test/resources/local127.keystore");
+
config.setString(ConfigConstants.SECURITY_SSL_KEYSTORE_PASSWORD, "password");
+
config.setString(ConfigConstants.SECURITY_SSL_KEY_PASSWORD, "password");
--- End diff --
Added few more testcases for failure scenarios
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Assigned] (FLINK-4351) JobManager handle TaskManager's registration
[ https://issues.apache.org/jira/browse/FLINK-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young reassigned FLINK-4351: - Assignee: Kurt Young > JobManager handle TaskManager's registration > > > Key: FLINK-4351 > URL: https://issues.apache.org/jira/browse/FLINK-4351 > Project: Flink > Issue Type: Sub-task > Components: Cluster Management >Reporter: zhangjing >Assignee: Kurt Young > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-4351) JobManager handle TaskManager's registration
[ https://issues.apache.org/jira/browse/FLINK-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-4351: -- Summary: JobManager handle TaskManager's registration (was: interaction between new JobManager and new TaskManager) > JobManager handle TaskManager's registration > > > Key: FLINK-4351 > URL: https://issues.apache.org/jira/browse/FLINK-4351 > Project: Flink > Issue Type: Sub-task > Components: Cluster Management >Reporter: zhangjing > > 1. Receive slot offer from TaskExecutor， ack it or reject it > 2. Release slot allocation to TaskExecutor > 3. Deploy task to offered slot on TaskExecutor > 4. Send Heartbeat to TaskExecutor and Receive Response which contains metrics > from TaskExecutor > 5. Other > messages(UpdateTaskExecutionState/RequestNextInputSplit/AbstractCheckpointMessage) > from TaskExecutor which is as same as current -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-4351) JobManager handle TaskManager's registration
[ https://issues.apache.org/jira/browse/FLINK-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-4351: -- Description: was: 1. Receive slot offer from TaskExecutor， ack it or reject it 2. Release slot allocation to TaskExecutor 3. Deploy task to offered slot on TaskExecutor 4. Send Heartbeat to TaskExecutor and Receive Response which contains metrics from TaskExecutor 5. Other messages(UpdateTaskExecutionState/RequestNextInputSplit/AbstractCheckpointMessage) from TaskExecutor which is as same as current > JobManager handle TaskManager's registration > > > Key: FLINK-4351 > URL: https://issues.apache.org/jira/browse/FLINK-4351 > Project: Flink > Issue Type: Sub-task > Components: Cluster Management >Reporter: zhangjing > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4129) HITSAlgorithm should test for element-wise convergence
[
https://issues.apache.org/jira/browse/FLINK-4129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15579933#comment-15579933
]
Vasia Kalavri commented on FLINK-4129:
--
I think having two HITS examples could be confusing to users. Is this
implementation showcasing some feature that no other example is or could we
simply remove it in favor of the HITS driver?
> HITSAlgorithm should test for element-wise convergence
> --
>
> Key: FLINK-4129
> URL: https://issues.apache.org/jira/browse/FLINK-4129
> Project: Flink
> Issue Type: Bug
> Components: Gelly
>Affects Versions: 1.1.0
>Reporter: Greg Hogan
>Priority: Minor
>
> {{HITSAlgorithm}} tests for convergence by summing the difference of each
> authority score minus the average score. This is simply comparing the sum of
> scores against the previous sum of scores which is not a good test for
> convergence.
> {code}
> // count the diff value of sum of authority scores
> diffSumAggregator.aggregate(previousAuthAverage -
> newAuthorityValue.getValue());
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1091) Allow joins with the solution set using key selectors
[ https://issues.apache.org/jira/browse/FLINK-1091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15579877#comment-15579877 ] Vasia Kalavri commented on FLINK-1091: -- Hi [~neelesh77], I'm not working on this. I've unassigned the issue. Do you have a use-case where you need this? > Allow joins with the solution set using key selectors > - > > Key: FLINK-1091 > URL: https://issues.apache.org/jira/browse/FLINK-1091 > Project: Flink > Issue Type: Sub-task > Components: Iterations >Reporter: Vasia Kalavri >Priority: Minor > Labels: easyfix, features > > Currently, the solution set may only be joined with using tuple field > positions. > A possible solution can be providing explicit functions "joinWithSolution" > and "coGroupWithSolution" to make sure the keys used are valid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-1091) Allow joins with the solution set using key selectors
[ https://issues.apache.org/jira/browse/FLINK-1091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vasia Kalavri updated FLINK-1091: - Assignee: (was: Vasia Kalavri) > Allow joins with the solution set using key selectors > - > > Key: FLINK-1091 > URL: https://issues.apache.org/jira/browse/FLINK-1091 > Project: Flink > Issue Type: Sub-task > Components: Iterations >Reporter: Vasia Kalavri >Priority: Minor > Labels: easyfix, features > > Currently, the solution set may only be joined with using tuple field > positions. > A possible solution can be providing explicit functions "joinWithSolution" > and "coGroupWithSolution" to make sure the keys used are valid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-4357) Implement TaskManager side of slot allocation from ResourceManager
[
https://issues.apache.org/jira/browse/FLINK-4357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-4357.
Resolution: Fixed
Done via FLINK-4360
> Implement TaskManager side of slot allocation from ResourceManager
> --
>
> Key: FLINK-4357
> URL: https://issues.apache.org/jira/browse/FLINK-4357
> Project: Flink
> Issue Type: Sub-task
> Components: Cluster Management
>Reporter: Zhijiang Wang
>Assignee: Till Rohrmann
>
> The {{ResourceManager}} may tell the {{TaskManager}} to give a slot to a
> specific {{JobManager}}.
> The slot allocation messages are fenced via ({{RmLeaderID}}, {{JobID}},
> {{AllocationID}}, {{ResourceID}}).
> The TM will ack the request ({{RmLeaderID}},{{AllocationID}}) to the RM and
> then offer that slot to the JM. If not accepted by JM, the TM notifies the RM
> that the slot is in fact available.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (FLINK-4357) Implement TaskManager side of slot allocation from ResourceManager
[
https://issues.apache.org/jira/browse/FLINK-4357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann reassigned FLINK-4357:
Assignee: Till Rohrmann (was: Zhijiang Wang)
> Implement TaskManager side of slot allocation from ResourceManager
> --
>
> Key: FLINK-4357
> URL: https://issues.apache.org/jira/browse/FLINK-4357
> Project: Flink
> Issue Type: Sub-task
> Components: Cluster Management
>Reporter: Zhijiang Wang
>Assignee: Till Rohrmann
>
> The {{ResourceManager}} may tell the {{TaskManager}} to give a slot to a
> specific {{JobManager}}.
> The slot allocation messages are fenced via ({{RmLeaderID}}, {{JobID}},
> {{AllocationID}}, {{ResourceID}}).
> The TM will ack the request ({{RmLeaderID}},{{AllocationID}}) to the RM and
> then offer that slot to the JM. If not accepted by JM, the TM notifies the RM
> that the slot is in fact available.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (FLINK-4358) Implement TaskManager deployment task into slot
[
https://issues.apache.org/jira/browse/FLINK-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-4358.
Resolution: Fixed
Done via FLINK-4738
> Implement TaskManager deployment task into slot
> ---
>
> Key: FLINK-4358
> URL: https://issues.apache.org/jira/browse/FLINK-4358
> Project: Flink
> Issue Type: Sub-task
> Components: Cluster Management
>Reporter: Zhijiang Wang
>Assignee: Zhijiang Wang
>
> The {{TaskManager}} may receive task deployment request (bound to
> slot/allocationID) from {{JobManager}}.
> The TM may decline request if slot is not allocated to that JM or slot does
> not exist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (FLINK-4360) Implement TaskManager registration slot at JobManager
[
https://issues.apache.org/jira/browse/FLINK-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-4360.
Resolution: Fixed
Added via 2486d3787b0d45a6220d3dcc96ca665986c47837
> Implement TaskManager registration slot at JobManager
> -
>
> Key: FLINK-4360
> URL: https://issues.apache.org/jira/browse/FLINK-4360
> Project: Flink
> Issue Type: Sub-task
> Components: Cluster Management
>Reporter: Zhijiang Wang
>Assignee: Till Rohrmann
>
> The {{TaskManager}} offers a slot to a {{JobManager}} at the
> {{ResourceManager}}’s behest. That slot is then tied to that JM until the JM
> releases the slot.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4343) Implement new TaskManager
[
https://issues.apache.org/jira/browse/FLINK-4343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15579610#comment-15579610
]
ASF GitHub Bot commented on FLINK-4343:
---
Github user tillrohrmann closed the pull request at:
https://github.com/apache/flink/pull/2640
> Implement new TaskManager
> -
>
> Key: FLINK-4343
> URL: https://issues.apache.org/jira/browse/FLINK-4343
> Project: Flink
> Issue Type: New Feature
> Components: Cluster Management
>Reporter: Stephan Ewen
>
> This is the parent issue for the efforts to implement the {{TaskManager}}
> changes based on FLIP-6
> (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077)
> Because of the breadth of changes, we should implement a new version of the
> {{TaskManager}} (let's call it {{TaskExecutor}}) rather than updating the
> current {{TaskManager}}. That will allow us to keep a working master branch.
> At the point when the new cluster management is on par with the current
> implementation, we will drop the old {{TaskManager}} and rename the
> {{TaskExecutor}} to {{TaskManager}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request #2640: [FLINK-4343] [tm] Implement TM -> JM registration ...
Github user tillrohrmann closed the pull request at: https://github.com/apache/flink/pull/2640 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4343) Implement new TaskManager
[
https://issues.apache.org/jira/browse/FLINK-4343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15579608#comment-15579608
]
ASF GitHub Bot commented on FLINK-4343:
---
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2640
Failing test case is unrelated. Will merge the PR into the flip-6 branch.
> Implement new TaskManager
> -
>
> Key: FLINK-4343
> URL: https://issues.apache.org/jira/browse/FLINK-4343
> Project: Flink
> Issue Type: New Feature
> Components: Cluster Management
>Reporter: Stephan Ewen
>
> This is the parent issue for the efforts to implement the {{TaskManager}}
> changes based on FLIP-6
> (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077)
> Because of the breadth of changes, we should implement a new version of the
> {{TaskManager}} (let's call it {{TaskExecutor}}) rather than updating the
> current {{TaskManager}}. That will allow us to keep a working master branch.
> At the point when the new cluster management is on par with the current
> implementation, we will drop the old {{TaskManager}} and rename the
> {{TaskExecutor}} to {{TaskManager}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink issue #2640: [FLINK-4343] [tm] Implement TM -> JM registration logic
Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/2640 Failing test case is unrelated. Will merge the PR into the flip-6 branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
