[jira] [Commented] (FLINK-6301) Flink KafkaConnector09 leaks memory on reading compressed messages due to a Kafka consumer bug
[
https://issues.apache.org/jira/browse/FLINK-6301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970699#comment-15970699
]
Tzu-Li (Gordon) Tai commented on FLINK-6301:
Thanks a lot for looking into this in detail!
It should be straight forward to just bump the Kafka 0.10 dependency in
FlinkKafkaConsumer010 (for {{master}} and {{release-1.2}}).
For FlinkKafkaConsumer09, I would say we only bump if there's a fix for the
issue in the Kafka 0.9 releases.
> Flink KafkaConnector09 leaks memory on reading compressed messages due to a
> Kafka consumer bug
> --
>
> Key: FLINK-6301
> URL: https://issues.apache.org/jira/browse/FLINK-6301
> Project: Flink
> Issue Type: Bug
> Components: Kafka Connector
>Affects Versions: 1.2.0, 1.1.3, 1.1.4
>Reporter: Rahul Yadav
> Attachments: jeprof.24611.1228.i1228.heap.svg,
> jeprof.24611.1695.i1695.heap.svg, jeprof.24611.265.i265.heap.svg,
> jeprof.24611.3138.i3138.heap.svg, jeprof.24611.595.i595.heap.svg,
> jeprof.24611.705.i705.heap.svg, jeprof.24611.81.i81.heap.svg
>
>
> Hi
> We are running Flink on a standalone cluster with 8 TaskManagers having 8
> vCPUs and 8 slots each. Each host has 16 GB of RAM.
> In our jobs,
> # We are consuming gzip compressed messages from Kafka using
> *KafkaConnector09* and use *rocksDB* backend for checkpoint storage.
> # To debug the leak, we used *jemalloc and jprof* to profile the sources of
> malloc calls from the java process and attached are the profiles generated at
> various stages of the job. As we can see, apart from the os.malloc and
> rocksDB.allocateNewBlock, there are additional malloc calls coming from
> inflate() method of java.util.zip.inflater. These calls are innocuous as long
> as the inflater.end() method is called after it's use.
> # To look for sources of inflate() method, we used Btrace on the running
> process to dump caller stack on the method call. Following is the stackTrace
> we got:
> {code}
> java.util.zip.Inflater.inflate(Inflater.java)
> java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152)
> java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117)
> java.io.DataInputStream.readFully(DataInputStream.java:195)
> org.apache.kafka.common.record.MemoryRecords$RecordsIterator.makeNext(MemoryRecords.java:253)
> org.apache.kafka.common.record.MemoryRecords$RecordsIterator.makeNext(MemoryRecords.java:210)
> org.apache.kafka.common.utils.AbstractIterator.maybeComputeNext(AbstractIterator.java:79)
> org.apache.kafka.common.utils.AbstractIterator.hasNext(AbstractIterator.java:45)
> org.apache.kafka.common.record.MemoryRecords$RecordsIterator.innerDone(MemoryRecords.java:282)
> org.apache.kafka.common.record.MemoryRecords$RecordsIterator.makeNext(MemoryRecords.java:233)
> org.apache.kafka.common.record.MemoryRecords$RecordsIterator.makeNext(MemoryRecords.java:210)
> org.apache.kafka.common.utils.AbstractIterator.maybeComputeNext(AbstractIterator.java:79)
> org.apache.kafka.common.utils.AbstractIterator.hasNext(AbstractIterator.java:45)
> org.apache.kafka.clients.consumer.internals.Fetcher.handleFetchResponse(Fetcher.java:563)
> org.apache.kafka.clients.consumer.internals.Fetcher.access$000(Fetcher.java:69)
> org.apache.kafka.clients.consumer.internals.Fetcher$1.onSuccess(Fetcher.java:139)
> org.apache.kafka.clients.consumer.internals.Fetcher$1.onSuccess(Fetcher.java:136)
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:380)
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:274)
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:320)
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:213)
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:193)
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:908)
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:853)
> org.apache.flink.streaming.connectors.kafka.internal.KafkaConsumerThread.run(KafkaConsumerThread.java:227)
> {code}
> The end() method on Inflater is called inside the close() method of
> *InflaterInputSteam* (extended by *GZIPInputStream*) but looking through the
> Kafka consumer code, we found that RecordsIterator is not closing the
> compressor stream after use and hence, causing the memory leak:
>
[jira] [Commented] (FLINK-6306) Sink for eventually consistent file systems
[ https://issues.apache.org/jira/browse/FLINK-6306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970683#comment-15970683 ] Tzu-Li (Gordon) Tai commented on FLINK-6306: This definitely needs a closer look. Thanks for picking this up! Could you briefly describe what you have in mind for the implementation? > Sink for eventually consistent file systems > --- > > Key: FLINK-6306 > URL: https://issues.apache.org/jira/browse/FLINK-6306 > Project: Flink > Issue Type: New Feature > Components: filesystem-connector >Reporter: Seth Wiesman >Assignee: Seth Wiesman > > Currently Flink provides the BucketingSink as an exactly once method for > writing out to a file system. It provides there guarantees by moving files > through several stages and deleting or truncating files that get into a bad > state. While this is a powerful abstraction, it causes issues with eventually > consistent file systems such as Amazon's S3 where must operations (ie rename, > delete, truncate) are not guaranteed to become consistent within a reasonable > amount of time. Flink should provide a sink that provides exactly once writes > to a file system where only PUT operations are considered consistent. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6311) NPE in FlinkKinesisConsumer if source was closed before run
Tzu-Li (Gordon) Tai created FLINK-6311:
--
Summary: NPE in FlinkKinesisConsumer if source was closed before
run
Key: FLINK-6311
URL: https://issues.apache.org/jira/browse/FLINK-6311
Project: Flink
Issue Type: Bug
Components: Kinesis Connector
Reporter: Tzu-Li (Gordon) Tai
This was reported by an user:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-errors-out-and-job-fails-IOException-from-CollectSink-open-td12606.html
The {{shutdownFetcher}} method of {{KinesisDataFetcher}} is not protected
against the condition when the source was closed before it started running.
Both {{mainThread}} and {{shardConsumersExecutor}} should have null checks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6130) Consider calling resourceManager#getTerminationFuture() with lock held
[
https://issues.apache.org/jira/browse/FLINK-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970649#comment-15970649
]
ASF GitHub Bot commented on FLINK-6130:
---
Github user zhangminglei commented on the issue:
https://github.com/apache/flink/pull/3726
@tedyu @zentol It seems there is a problem with travis-ci test if no output
within 5 mins then it will print the stacktrace and then kill the watchdog ,
please help to review and checkout how to solve, I am appreciate it.
```
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.213 sec -
in org.apache.flink.runtime.minicluster.MiniClusterITCase
==
Maven produced no output for 300 seconds.
==
==
The following Java processes are running (JPS)
==
1552 Launcher
12642 Jps
8045 surefirebooter5889190340308801783.jar
...
Trying to KILL watchdog (1547).
MVN exited with EXIT CODE: 143.
java.io.FileNotFoundException: build-target/lib/flink-dist*.jar (No such
file or directory)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:220)
at java.util.zip.ZipFile.(ZipFile.java:150)
at java.util.zip.ZipFile.(ZipFile.java:121)
at sun.tools.jar.Main.list(Main.java:1060)
at sun.tools.jar.Main.run(Main.java:291)
at sun.tools.jar.Main.main(Main.java:1233)
find: `./flink-yarn-tests/target/flink-yarn-tests*': No such file or
directory
PRODUCED build artifacts.
```
> Consider calling resourceManager#getTerminationFuture() with lock held
> --
>
> Key: FLINK-6130
> URL: https://issues.apache.org/jira/browse/FLINK-6130
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> In YarnFlinkApplicationMasterRunner#runApplicationMaster() :
> {code}
> synchronized (lock) {
> LOG.info("Starting High Availability Services");
> ...
> }
> // wait for resource manager to finish
> resourceManager.getTerminationFuture().get();
> {code}
> resourceManager#getTerminationFuture() is called without holding lock.
> We should store the value returned from
> resourceManager#getTerminationFuture() inside the synchronized block.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink issue #3726: [FLINK-6130] [yarn] Fix Consider calling resourceManager#...
Github user zhangminglei commented on the issue: https://github.com/apache/flink/pull/3726 @tedyu @zentol It seems there is a problem with travis-ci test if no output within 5 mins then it will print the stacktrace and then kill the watchdog , please help to review and checkout how to solve, I am appreciate it. ``` Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.213 sec - in org.apache.flink.runtime.minicluster.MiniClusterITCase == Maven produced no output for 300 seconds. == == The following Java processes are running (JPS) == 1552 Launcher 12642 Jps 8045 surefirebooter5889190340308801783.jar ... Trying to KILL watchdog (1547). MVN exited with EXIT CODE: 143. java.io.FileNotFoundException: build-target/lib/flink-dist*.jar (No such file or directory) at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.(ZipFile.java:220) at java.util.zip.ZipFile.(ZipFile.java:150) at java.util.zip.ZipFile.(ZipFile.java:121) at sun.tools.jar.Main.list(Main.java:1060) at sun.tools.jar.Main.run(Main.java:291) at sun.tools.jar.Main.main(Main.java:1233) find: `./flink-yarn-tests/target/flink-yarn-tests*': No such file or directory PRODUCED build artifacts. ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Assigned] (FLINK-6309) Memory consumer weights should be calculated in job vertex level
[
https://issues.apache.org/jira/browse/FLINK-6309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xu Pingyong reassigned FLINK-6309:
--
Assignee: Xu Pingyong
> Memory consumer weights should be calculated in job vertex level
>
>
> Key: FLINK-6309
> URL: https://issues.apache.org/jira/browse/FLINK-6309
> Project: Flink
> Issue Type: Improvement
> Components: Optimizer
>Reporter: Kurt Young
>Assignee: Xu Pingyong
>
> Currently in {{PlanFinalizer}}, we travel the whole job vertexes to calculate
> the memory consumer weights, and then assign the weights for each job vertex.
> In a case of a large job graph, e.g. with multiple joins, group reduces, the
> consumer weights will be high and the usable memory for each job vertex will
> be very low.
> I think it makes more sense to calculate the memory consumer weights in job
> vertex level (after chaining) to maximize the memory utility.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6130) Consider calling resourceManager#getTerminationFuture() with lock held
[
https://issues.apache.org/jira/browse/FLINK-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970608#comment-15970608
]
ASF GitHub Bot commented on FLINK-6130:
---
GitHub user zhangminglei opened a pull request:
https://github.com/apache/flink/pull/3726
[FLINK-6130] [yarn] Fix Consider calling resourceManager#getTerminati…
…onFuture() with lock held.
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhangminglei/flink flink-6130
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3726.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3726
commit d7c707046b1a081ceb5999cdb1f08bf26a2aeac2
Author: zhangminglei
Date: 2017-04-17T01:58:14Z
[FLINK-6130] [yarn] Fix Consider calling
resourceManager#getTerminationFuture() with lock held.
> Consider calling resourceManager#getTerminationFuture() with lock held
> --
>
> Key: FLINK-6130
> URL: https://issues.apache.org/jira/browse/FLINK-6130
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> In YarnFlinkApplicationMasterRunner#runApplicationMaster() :
> {code}
> synchronized (lock) {
> LOG.info("Starting High Availability Services");
> ...
> }
> // wait for resource manager to finish
> resourceManager.getTerminationFuture().get();
> {code}
> resourceManager#getTerminationFuture() is called without holding lock.
> We should store the value returned from
> resourceManager#getTerminationFuture() inside the synchronized block.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink pull request #3726: [FLINK-6130] [yarn] Fix Consider calling resourceM...
GitHub user zhangminglei opened a pull request:
https://github.com/apache/flink/pull/3726
[FLINK-6130] [yarn] Fix Consider calling resourceManager#getTerminatiâ¦
â¦onFuture() with lock held.
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhangminglei/flink flink-6130
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3726.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3726
commit d7c707046b1a081ceb5999cdb1f08bf26a2aeac2
Author: zhangminglei
Date: 2017-04-17T01:58:14Z
[FLINK-6130] [yarn] Fix Consider calling
resourceManager#getTerminationFuture() with lock held.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-6130) Consider calling resourceManager#getTerminationFuture() with lock held
[
https://issues.apache.org/jira/browse/FLINK-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970602#comment-15970602
]
mingleizhang commented on FLINK-6130:
-
[~tedyu] Thanks and appreciate it again. If there is no more question, I will
give a PR soon.
> Consider calling resourceManager#getTerminationFuture() with lock held
> --
>
> Key: FLINK-6130
> URL: https://issues.apache.org/jira/browse/FLINK-6130
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> In YarnFlinkApplicationMasterRunner#runApplicationMaster() :
> {code}
> synchronized (lock) {
> LOG.info("Starting High Availability Services");
> ...
> }
> // wait for resource manager to finish
> resourceManager.getTerminationFuture().get();
> {code}
> resourceManager#getTerminationFuture() is called without holding lock.
> We should store the value returned from
> resourceManager#getTerminationFuture() inside the synchronized block.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6130) Consider calling resourceManager#getTerminationFuture() with lock held
[
https://issues.apache.org/jira/browse/FLINK-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970599#comment-15970599
]
Ted Yu commented on FLINK-6130:
---
lgtm
> Consider calling resourceManager#getTerminationFuture() with lock held
> --
>
> Key: FLINK-6130
> URL: https://issues.apache.org/jira/browse/FLINK-6130
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> In YarnFlinkApplicationMasterRunner#runApplicationMaster() :
> {code}
> synchronized (lock) {
> LOG.info("Starting High Availability Services");
> ...
> }
> // wait for resource manager to finish
> resourceManager.getTerminationFuture().get();
> {code}
> resourceManager#getTerminationFuture() is called without holding lock.
> We should store the value returned from
> resourceManager#getTerminationFuture() inside the synchronized block.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6130) Consider calling resourceManager#getTerminationFuture() with lock held
[
https://issues.apache.org/jira/browse/FLINK-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970598#comment-15970598
]
mingleizhang commented on FLINK-6130:
-
[~tedyu] Yep. And the code like below, how do you think of this ? Thanks.
{code}
@Override
protected int runApplicationMaster(Configuration config) {
Future future;
synchronized (lock) {
LOG.info("Starting High Availability Services");
// wait for resource manager to finish
future = (Future) resourceManager.getTerminationFuture();
// (5) start the web monitor
// TODO: add web monitor
}
Object object = future.value().get();
// everything started, we can wait until all is done or the process is
killed
LOG.info("YARN Application Master finished" + object.toString());
{code}
> Consider calling resourceManager#getTerminationFuture() with lock held
> --
>
> Key: FLINK-6130
> URL: https://issues.apache.org/jira/browse/FLINK-6130
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> In YarnFlinkApplicationMasterRunner#runApplicationMaster() :
> {code}
> synchronized (lock) {
> LOG.info("Starting High Availability Services");
> ...
> }
> // wait for resource manager to finish
> resourceManager.getTerminationFuture().get();
> {code}
> resourceManager#getTerminationFuture() is called without holding lock.
> We should store the value returned from
> resourceManager#getTerminationFuture() inside the synchronized block.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-6310) LocalExecutor#endSession() uses wrong lock for synchronization
Ted Yu created FLINK-6310:
-
Summary: LocalExecutor#endSession() uses wrong lock for
synchronization
Key: FLINK-6310
URL: https://issues.apache.org/jira/browse/FLINK-6310
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
public void endSession(JobID jobID) throws Exception {
synchronized (LocalExecutor.class) {
LocalFlinkMiniCluster flink = this.flink;
{code}
In other places, lock field is used for synchronization:
{code}
public void start() throws Exception {
synchronized (lock) {
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-5752) Support push down projections for HBaseTableSource
[
https://issues.apache.org/jira/browse/FLINK-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970592#comment-15970592
]
ramkrishna.s.vasudevan commented on FLINK-5752:
---
I completed the patch related work but I don have my linux box to do some
tests. Once I get them I will submit a PR with test cases.
> Support push down projections for HBaseTableSource
> --
>
> Key: FLINK-5752
> URL: https://issues.apache.org/jira/browse/FLINK-5752
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>
> This is after the discussion to create NestedProjectableTableSource.
> Currently we support nested schema for the non-relational type of DBs like
> HBase.
> But this does not allow push down projection. This JIRA is to implement that.
> Once FLINK-5698 is implemented then we should be making use of the feature to
> push down the projections for a nested table. So in case of HBase if we have
> {f1:{a, b}, f2:{c, d}} as the nested structure then if we have a scan query
> that needs to select f2.c - then we should be specifically able to project
> only that column 'c' under 'f2'. FLINK-5698 plans to add a new API for such
> projections and HBaseTableSource should make use of that API to do the
> projection.
> [~fhueske], [~tonycox], [~jark]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6130) Consider calling resourceManager#getTerminationFuture() with lock held
[
https://issues.apache.org/jira/browse/FLINK-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970590#comment-15970590
]
Ted Yu commented on FLINK-6130:
---
You don't need to create another synchronized (lock) block.
You can use the existing one to obtain the future.
> Consider calling resourceManager#getTerminationFuture() with lock held
> --
>
> Key: FLINK-6130
> URL: https://issues.apache.org/jira/browse/FLINK-6130
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> In YarnFlinkApplicationMasterRunner#runApplicationMaster() :
> {code}
> synchronized (lock) {
> LOG.info("Starting High Availability Services");
> ...
> }
> // wait for resource manager to finish
> resourceManager.getTerminationFuture().get();
> {code}
> resourceManager#getTerminationFuture() is called without holding lock.
> We should store the value returned from
> resourceManager#getTerminationFuture() inside the synchronized block.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6130) Consider calling resourceManager#getTerminationFuture() with lock held
[
https://issues.apache.org/jira/browse/FLINK-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970588#comment-15970588
]
mingleizhang commented on FLINK-6130:
-
[~tedyu] Thanks and appreciate it. I will give a PR to this jira soon enough.
> Consider calling resourceManager#getTerminationFuture() with lock held
> --
>
> Key: FLINK-6130
> URL: https://issues.apache.org/jira/browse/FLINK-6130
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> In YarnFlinkApplicationMasterRunner#runApplicationMaster() :
> {code}
> synchronized (lock) {
> LOG.info("Starting High Availability Services");
> ...
> }
> // wait for resource manager to finish
> resourceManager.getTerminationFuture().get();
> {code}
> resourceManager#getTerminationFuture() is called without holding lock.
> We should store the value returned from
> resourceManager#getTerminationFuture() inside the synchronized block.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6130) Consider calling resourceManager#getTerminationFuture() with lock held
[
https://issues.apache.org/jira/browse/FLINK-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970587#comment-15970587
]
Ted Yu commented on FLINK-6130:
---
lgtm
> Consider calling resourceManager#getTerminationFuture() with lock held
> --
>
> Key: FLINK-6130
> URL: https://issues.apache.org/jira/browse/FLINK-6130
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> In YarnFlinkApplicationMasterRunner#runApplicationMaster() :
> {code}
> synchronized (lock) {
> LOG.info("Starting High Availability Services");
> ...
> }
> // wait for resource manager to finish
> resourceManager.getTerminationFuture().get();
> {code}
> resourceManager#getTerminationFuture() is called without holding lock.
> We should store the value returned from
> resourceManager#getTerminationFuture() inside the synchronized block.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (FLINK-6130) Consider calling resourceManager#getTerminationFuture() with lock held
[
https://issues.apache.org/jira/browse/FLINK-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970586#comment-15970586
]
mingleizhang edited comment on FLINK-6130 at 4/17/17 1:51 AM:
--
[~tedyu] Could you please help review the code ? How do you think of this ?
Thanks.
{code}
Future future;
synchronized (lock) {
// wait for resource manager to finish
future = (Future) resourceManager.getTerminationFuture();
}
Object object = future.get();
// everything started, we can wait until all is done or the process is
killed
LOG.info("YARN Application Master finished" + object.toString());
{code}
was (Author: mingleizhang):
[~tedyu] Could you please help review the code ? How do you think of this ?
Thanks.
{code}
Future future;
synchronized (lock) {
// wait for resource manager to finish
future = (Future)
resourceManager.getTerminationFuture();
}
Object object = future.get();
// everything started, we can wait until all is done or
the process is killed
LOG.info("YARN Application Master finished" +
object.toString());
{code}
> Consider calling resourceManager#getTerminationFuture() with lock held
> --
>
> Key: FLINK-6130
> URL: https://issues.apache.org/jira/browse/FLINK-6130
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> In YarnFlinkApplicationMasterRunner#runApplicationMaster() :
> {code}
> synchronized (lock) {
> LOG.info("Starting High Availability Services");
> ...
> }
> // wait for resource manager to finish
> resourceManager.getTerminationFuture().get();
> {code}
> resourceManager#getTerminationFuture() is called without holding lock.
> We should store the value returned from
> resourceManager#getTerminationFuture() inside the synchronized block.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6130) Consider calling resourceManager#getTerminationFuture() with lock held
[
https://issues.apache.org/jira/browse/FLINK-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970586#comment-15970586
]
mingleizhang commented on FLINK-6130:
-
[~tedyu] Could you please help review the code ? How do you think of this ?
Thanks.
{code}
Future future;
synchronized (lock) {
// wait for resource manager to finish
future = (Future)
resourceManager.getTerminationFuture();
}
Object object = future.get();
// everything started, we can wait until all is done or
the process is killed
LOG.info("YARN Application Master finished" +
object.toString());
{code}
> Consider calling resourceManager#getTerminationFuture() with lock held
> --
>
> Key: FLINK-6130
> URL: https://issues.apache.org/jira/browse/FLINK-6130
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> In YarnFlinkApplicationMasterRunner#runApplicationMaster() :
> {code}
> synchronized (lock) {
> LOG.info("Starting High Availability Services");
> ...
> }
> // wait for resource manager to finish
> resourceManager.getTerminationFuture().get();
> {code}
> resourceManager#getTerminationFuture() is called without holding lock.
> We should store the value returned from
> resourceManager#getTerminationFuture() inside the synchronized block.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-6309) Memory consumer weights should be calculated in job vertex level
Kurt Young created FLINK-6309:
-
Summary: Memory consumer weights should be calculated in job
vertex level
Key: FLINK-6309
URL: https://issues.apache.org/jira/browse/FLINK-6309
Project: Flink
Issue Type: Improvement
Components: Optimizer
Reporter: Kurt Young
Currently in {{PlanFinalizer}}, we travel the whole job vertexes to calculate
the memory consumer weights, and then assign the weights for each job vertex.
In a case of a large job graph, e.g. with multiple joins, group reduces, the
consumer weights will be high and the usable memory for each job vertex will be
very low.
I think it makes more sense to calculate the memory consumer weights in job
vertex level (after chaining) to maximize the memory utility.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-2032) Migrate integration tests from temp output files to collect()
[
https://issues.apache.org/jira/browse/FLINK-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970524#comment-15970524
]
Pattarawat Chormai commented on FLINK-2032:
---
Hi all,
I have searched on Github using [1] and found that there are several tests that
haven't been refactored to use _collect_ yet.
{code}
flink-streaming-scala/src/test/scala/org/apache/flink/streaming/api/scala/StreamingOperatorsITCase.scala
flink-tests/src/test/scala/org/apache/flink/api/scala/functions/ClosureCleanerITCase.scala
flink-tests/src/test/scala/org/apache/flink/api/scala/io/ScalaCsvReaderWithPOJOITCase.scala
flink-tests/src/test/scala/org/apache/flink/api/scala/operators/AggregateITCase.scala
flink-tests/src/test/scala/org/apache/flink/api/scala/operators/CoGroupITCase.scala
flink-tests/src/test/scala/org/apache/flink/api/scala/operators/DistinctITCase.scala
flink-tests/src/test/scala/org/apache/flink/api/scala/operators/ExamplesITCase.scala
flink-tests/src/test/scala/org/apache/flink/api/scala/operators/FilterITCase.scala
flink-tests/src/test/scala/org/apache/flink/api/scala/operators/FirstNITCase.scala
flink-tests/src/test/scala/org/apache/flink/api/scala/operators/FlatMapITCase.scala
flink-tests/src/test/scala/org/apache/flink/api/scala/operators/JoinITCase.scala
flink-tests/src/test/scala/org/apache/flink/api/scala/operators/MapITCase.scala
flink-tests/src/test/scala/org/apache/flink/api/scala/operators/OuterJoinITCase.scala
flink-tests/src/test/scala/org/apache/flink/api/scala/operators/PartitionITCase.scala
flink-tests/src/test/scala/org/apache/flink/api/scala/operators/ReduceITCase.scala
flink-tests/src/test/scala/org/apache/flink/api/scala/runtime/ScalaSpecialTypesITCase.scala
flink-connectors/flink-avro/src/test/java/org/apache/flink/api/io/avro/AvroPojoTest.java
flink-connectors/flink-hadoop-compatibility/src/test/java/org/apache/flink/test/hadoopcompatibility/mapred/HadoopMapFunctionITCase.java
flink-connectors/flink-hadoop-compatibility/src/test/java/org/apache/flink/test/hadoopcompatibility/mapred/HadoopReduceCombineFunctionITCase.java
flink-connectors/flink-hadoop-compatibility/src/test/java/org/apache/flink/test/hadoopcompatibility/mapred/HadoopReduceFunctionITCase.java
flink-libraries/flink-cep/src/test/java/org/apache/flink/cep/CEPITCase.java
flink-libraries/flink-gelly-examples/src/test/java/org/apache/flink/graph/test/examples/IncrementalSSSPITCase.java
flink-tests/src/test/java/org/apache/flink/test/iterative/aggregators/AggregatorsITCase.java
flink-tests/src/test/java/org/apache/flink/test/javaApiOperators/DataSinkITCase.java
{code}
I would suggest to create 2 additional subtasks each for Scala and Java and I
can help finishing them. What do you think?
[1]
https://github.com/apache/flink/search?p=5=TemporaryFolder+write==%E2%9C%93
> Migrate integration tests from temp output files to collect()
> -
>
> Key: FLINK-2032
> URL: https://issues.apache.org/jira/browse/FLINK-2032
> Project: Flink
> Issue Type: Task
> Components: Tests
>Affects Versions: 0.9
>Reporter: Fabian Hueske
>Priority: Minor
> Labels: starter
>
> Most of Flink's integration tests that execute full Flink programs and check
> their results are implemented by writing results to temporary output file and
> comparing the content of the file to a provided set of expected Strings.
> Flink's test utils make this quite comfortable and hide a lot of the
> complexity of this approach. Nonetheless, this approach has a few drawbacks:
> - increased latency by going through disk
> - comparison is on String representation of objects
> - depends on the file system
> Since Flink's {{collect()}} feature was added, the temp file approach is not
> the best approach anymore. Instead, tests can collect the result of a Flink
> program directly as objects and compare these against a set of expected
> objects.
> It would be good to migrate the existing test base to use {{collect()}}
> instead of temporary output files.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-4548) Refactor ContinuousFileMonitoringTest to use OneShotLatch
[
https://issues.apache.org/jira/browse/FLINK-4548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970516#comment-15970516
]
Pattarawat Chormai commented on FLINK-4548:
---
The test was removed already by
https://github.com/apache/flink/pull/2618/files#diff-abb9ef4de62be657638f23418bac12d7.
So, I will close this issue.
> Refactor ContinuousFileMonitoringTest to use OneShotLatch
> -
>
> Key: FLINK-4548
> URL: https://issues.apache.org/jira/browse/FLINK-4548
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API
>Reporter: Aljoscha Krettek
>Assignee: Pattarawat Chormai
>Priority: Minor
> Labels: easyfix, newbie, starter
>
> Right now, the test has custom code that uses {{synchronized}}, {{notify()}}
> and {{wait()}}. This could be simplified by using {{OneShotLatch}} which does
> that under the hood and makes the intended behavior more obvious in the test.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (FLINK-4548) Refactor ContinuousFileMonitoringTest to use OneShotLatch
[
https://issues.apache.org/jira/browse/FLINK-4548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pattarawat Chormai closed FLINK-4548.
-
Resolution: Fixed
> Refactor ContinuousFileMonitoringTest to use OneShotLatch
> -
>
> Key: FLINK-4548
> URL: https://issues.apache.org/jira/browse/FLINK-4548
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API
>Reporter: Aljoscha Krettek
>Assignee: Pattarawat Chormai
>Priority: Minor
> Labels: easyfix, newbie, starter
>
> Right now, the test has custom code that uses {{synchronized}}, {{notify()}}
> and {{wait()}}. This could be simplified by using {{OneShotLatch}} which does
> that under the hood and makes the intended behavior more obvious in the test.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (FLINK-6308) Task managers are not attaching to job manager on macos
[
https://issues.apache.org/jira/browse/FLINK-6308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mateusz Zakarczemny updated FLINK-6308:
---
Description:
I'm using flink 1.2.0. On macOS task managers were not joining cluster. I was
able to fix that by removing quotes from docker-compose:
{code}
diff --git a/flink-contrib/docker-flink/docker-compose-bluemix.yml
b/flink-contrib/docker-flink/docker-compose-bluemix.yml
index b667a0d89a..55c161766a 100644
--- a/flink-contrib/docker-flink/docker-compose-bluemix.yml
+++ b/flink-contrib/docker-flink/docker-compose-bluemix.yml
@@ -23,11 +23,11 @@ services:
jobmanager:
#image example: registry.eu-gb.bluemix.net/markussaidhiho/flink
image: ${IMAGENAME}
-container_name: "jobmanager"
+container_name: jobmanager
expose:
- - "6123"
+ - 6123"
ports:
- - "48081:8081"
+ - 48081:8081
command: jobmanager
taskmanager:
@@ -41,4 +41,4 @@ services:
links:
- "jobmanager:jobmanager"
environment:
- - JOB_MANAGER_RPC_ADDRESS="jobmanager"
+ - JOB_MANAGER_RPC_ADDRESS=jobmanager
diff --git a/flink-contrib/docker-flink/docker-compose.yml
b/flink-contrib/docker-flink/docker-compose.yml
index 6a1335360d..0a644681f3 100644
--- a/flink-contrib/docker-flink/docker-compose.yml
+++ b/flink-contrib/docker-flink/docker-compose.yml
@@ -27,7 +27,7 @@ services:
- "48081:8081"
command: jobmanager
environment:
- - JOB_MANAGER_RPC_ADDRESS="jobmanager"
+ - JOB_MANAGER_RPC_ADDRESS=jobmanager
taskmanager:
image: flink
@@ -40,4 +40,4 @@ services:
links:
- "jobmanager:jobmanager"
environment:
- - JOB_MANAGER_RPC_ADDRESS="jobmanager"
+ - JOB_MANAGER_RPC_ADDRESS=jobmanager
{code}
> Task managers are not attaching to job manager on macos
> ---
>
> Key: FLINK-6308
> URL: https://issues.apache.org/jira/browse/FLINK-6308
> Project: Flink
> Issue Type: Bug
> Components: Docker
>Reporter: Mateusz Zakarczemny
>
> I'm using flink 1.2.0. On macOS task managers were not joining cluster. I was
> able to fix that by removing quotes from docker-compose:
> {code}
> diff --git a/flink-contrib/docker-flink/docker-compose-bluemix.yml
> b/flink-contrib/docker-flink/docker-compose-bluemix.yml
> index b667a0d89a..55c161766a 100644
> --- a/flink-contrib/docker-flink/docker-compose-bluemix.yml
> +++ b/flink-contrib/docker-flink/docker-compose-bluemix.yml
> @@ -23,11 +23,11 @@ services:
>jobmanager:
> #image example: registry.eu-gb.bluemix.net/markussaidhiho/flink
> image: ${IMAGENAME}
> -container_name: "jobmanager"
> +container_name: jobmanager
> expose:
> - - "6123"
> + - 6123"
> ports:
> - - "48081:8081"
> + - 48081:8081
> command: jobmanager
>taskmanager:
> @@ -41,4 +41,4 @@ services:
> links:
>- "jobmanager:jobmanager"
> environment:
> - - JOB_MANAGER_RPC_ADDRESS="jobmanager"
> + - JOB_MANAGER_RPC_ADDRESS=jobmanager
> diff --git a/flink-contrib/docker-flink/docker-compose.yml
> b/flink-contrib/docker-flink/docker-compose.yml
> index 6a1335360d..0a644681f3 100644
> --- a/flink-contrib/docker-flink/docker-compose.yml
> +++ b/flink-contrib/docker-flink/docker-compose.yml
> @@ -27,7 +27,7 @@ services:
>- "48081:8081"
> command: jobmanager
> environment:
> - - JOB_MANAGER_RPC_ADDRESS="jobmanager"
> + - JOB_MANAGER_RPC_ADDRESS=jobmanager
>taskmanager:
> image: flink
> @@ -40,4 +40,4 @@ services:
> links:
>- "jobmanager:jobmanager"
> environment:
> - - JOB_MANAGER_RPC_ADDRESS="jobmanager"
> + - JOB_MANAGER_RPC_ADDRESS=jobmanager
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-6308) Task managers are not attaching to job manager on macos
Mateusz Zakarczemny created FLINK-6308: -- Summary: Task managers are not attaching to job manager on macos Key: FLINK-6308 URL: https://issues.apache.org/jira/browse/FLINK-6308 Project: Flink Issue Type: Bug Components: Docker Reporter: Mateusz Zakarczemny -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6130) Consider calling resourceManager#getTerminationFuture() with lock held
[
https://issues.apache.org/jira/browse/FLINK-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970408#comment-15970408
]
Ted Yu commented on FLINK-6130:
---
You can store the return value from getTerminationFuture in a variable inside
the synchronized block.
Outside the synchronized block, call get() on the variable.
> Consider calling resourceManager#getTerminationFuture() with lock held
> --
>
> Key: FLINK-6130
> URL: https://issues.apache.org/jira/browse/FLINK-6130
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> In YarnFlinkApplicationMasterRunner#runApplicationMaster() :
> {code}
> synchronized (lock) {
> LOG.info("Starting High Availability Services");
> ...
> }
> // wait for resource manager to finish
> resourceManager.getTerminationFuture().get();
> {code}
> resourceManager#getTerminationFuture() is called without holding lock.
> We should store the value returned from
> resourceManager#getTerminationFuture() inside the synchronized block.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-5611) Add QueryableStateException type
[
https://issues.apache.org/jira/browse/FLINK-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970385#comment-15970385
]
mingleizhang commented on FLINK-5611:
-
[~uce]Hi, I will work on this soon. Thanks ~
> Add QueryableStateException type
>
>
> Key: FLINK-5611
> URL: https://issues.apache.org/jira/browse/FLINK-5611
> Project: Flink
> Issue Type: Sub-task
> Components: Queryable State
>Reporter: Ufuk Celebi
>Assignee: mingleizhang
>Priority: Minor
>
> We currently have some exceptions like {{UnknownJobManager}} and the like
> that should be sub types of the to be introduced {{QueryableStateException}}.
> Right now, they extend checked and unchecked Exceptions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (FLINK-5611) Add QueryableStateException type
[
https://issues.apache.org/jira/browse/FLINK-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mingleizhang reassigned FLINK-5611:
---
Assignee: mingleizhang
> Add QueryableStateException type
>
>
> Key: FLINK-5611
> URL: https://issues.apache.org/jira/browse/FLINK-5611
> Project: Flink
> Issue Type: Sub-task
> Components: Queryable State
>Reporter: Ufuk Celebi
>Assignee: mingleizhang
>Priority: Minor
>
> We currently have some exceptions like {{UnknownJobManager}} and the like
> that should be sub types of the to be introduced {{QueryableStateException}}.
> Right now, they extend checked and unchecked Exceptions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6104) Resource leak in ListViaRangeSpeedMiniBenchmark
[
https://issues.apache.org/jira/browse/FLINK-6104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970358#comment-15970358
]
ASF GitHub Bot commented on FLINK-6104:
---
GitHub user zhangminglei opened a pull request:
https://github.com/apache/flink/pull/3725
[FLINK-6104] Fix resource leak in ListViaRangeSpeedMiniBenchmark.
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhangminglei/flink flink-6104
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3725.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3725
commit 0ff43539f76ec7f87b45fba6e27541487a37c78e
Author: zhangminglei
Date: 2017-04-16T12:10:21Z
[FLINK-6104] Fix resource leak in ListViaRangeSpeedMiniBenchmark.
> Resource leak in ListViaRangeSpeedMiniBenchmark
> ---
>
> Key: FLINK-6104
> URL: https://issues.apache.org/jira/browse/FLINK-6104
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> {code}
> final WriteOptions write_options = new WriteOptions()
> .setSync(false)
> .setDisableWAL(true);
> {code}
> WriteOptions ultimately extends AbstractNativeReference where:
> {code}
> public abstract class AbstractNativeReference implements AutoCloseable {
> {code}
> WriteOptions instance should be closed.
> {code}
> final RocksDB rocksDB = RocksDB.open(options, rocksDir.getAbsolutePath());
> {code}
> RocksDB should be closed as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink pull request #3725: [FLINK-6104] Fix resource leak in ListViaRangeSpee...
GitHub user zhangminglei opened a pull request:
https://github.com/apache/flink/pull/3725
[FLINK-6104] Fix resource leak in ListViaRangeSpeedMiniBenchmark.
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhangminglei/flink flink-6104
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3725.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3725
commit 0ff43539f76ec7f87b45fba6e27541487a37c78e
Author: zhangminglei
Date: 2017-04-16T12:10:21Z
[FLINK-6104] Fix resource leak in ListViaRangeSpeedMiniBenchmark.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-6286) hbase command not found error
[
https://issues.apache.org/jira/browse/FLINK-6286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970355#comment-15970355
]
ASF GitHub Bot commented on FLINK-6286:
---
Github user greghogan commented on the issue:
https://github.com/apache/flink/pull/3711
@lingjinjiang thanks for your contribution! Merging ...
> hbase command not found error
> -
>
> Key: FLINK-6286
> URL: https://issues.apache.org/jira/browse/FLINK-6286
> Project: Flink
> Issue Type: Bug
> Components: Batch Connectors and Input/Output Formats, Startup Shell
> Scripts
>Reporter: Jinjiang Ling
>Priority: Minor
> Attachments: FLINK-6286-0.patch, FLINK-6286-1.patch,
> FLINK-6286-2.patch, FLINK-6286-3.patch
>
>
> As I'm using flink with the HBASE_CONF_DIR env variable and don't install
> hbase, then I get the error message below.
> {quote}
> *bin/config.sh: line 303: hbase: command not found*
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink issue #3711: [FLINK-6286] [script] Fix the hbase command not found err...
Github user greghogan commented on the issue: https://github.com/apache/flink/pull/3711 @lingjinjiang thanks for your contribution! Merging ... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Assigned] (FLINK-6104) Resource leak in ListViaRangeSpeedMiniBenchmark
[
https://issues.apache.org/jira/browse/FLINK-6104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mingleizhang reassigned FLINK-6104:
---
Assignee: mingleizhang
> Resource leak in ListViaRangeSpeedMiniBenchmark
> ---
>
> Key: FLINK-6104
> URL: https://issues.apache.org/jira/browse/FLINK-6104
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> {code}
> final WriteOptions write_options = new WriteOptions()
> .setSync(false)
> .setDisableWAL(true);
> {code}
> WriteOptions ultimately extends AbstractNativeReference where:
> {code}
> public abstract class AbstractNativeReference implements AutoCloseable {
> {code}
> WriteOptions instance should be closed.
> {code}
> final RocksDB rocksDB = RocksDB.open(options, rocksDir.getAbsolutePath());
> {code}
> RocksDB should be closed as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6104) Resource leak in ListViaRangeSpeedMiniBenchmark
[
https://issues.apache.org/jira/browse/FLINK-6104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970341#comment-15970341
]
mingleizhang commented on FLINK-6104:
-
I will give a pr to this jira soon.
> Resource leak in ListViaRangeSpeedMiniBenchmark
> ---
>
> Key: FLINK-6104
> URL: https://issues.apache.org/jira/browse/FLINK-6104
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> {code}
> final WriteOptions write_options = new WriteOptions()
> .setSync(false)
> .setDisableWAL(true);
> {code}
> WriteOptions ultimately extends AbstractNativeReference where:
> {code}
> public abstract class AbstractNativeReference implements AutoCloseable {
> {code}
> WriteOptions instance should be closed.
> {code}
> final RocksDB rocksDB = RocksDB.open(options, rocksDir.getAbsolutePath());
> {code}
> RocksDB should be closed as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (FLINK-6130) Consider calling resourceManager#getTerminationFuture() with lock held
[
https://issues.apache.org/jira/browse/FLINK-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mingleizhang reassigned FLINK-6130:
---
Assignee: mingleizhang
> Consider calling resourceManager#getTerminationFuture() with lock held
> --
>
> Key: FLINK-6130
> URL: https://issues.apache.org/jira/browse/FLINK-6130
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> In YarnFlinkApplicationMasterRunner#runApplicationMaster() :
> {code}
> synchronized (lock) {
> LOG.info("Starting High Availability Services");
> ...
> }
> // wait for resource manager to finish
> resourceManager.getTerminationFuture().get();
> {code}
> resourceManager#getTerminationFuture() is called without holding lock.
> We should store the value returned from
> resourceManager#getTerminationFuture() inside the synchronized block.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6130) Consider calling resourceManager#getTerminationFuture() with lock held
[
https://issues.apache.org/jira/browse/FLINK-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970332#comment-15970332
]
mingleizhang commented on FLINK-6130:
-
[~tedyu] Thanks. I would ask how to use those return value that is stored from
resouceManager#getTerminationFuture() or do nothing ? It seems there is a
intersection with jira Flink-6275.
> Consider calling resourceManager#getTerminationFuture() with lock held
> --
>
> Key: FLINK-6130
> URL: https://issues.apache.org/jira/browse/FLINK-6130
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> In YarnFlinkApplicationMasterRunner#runApplicationMaster() :
> {code}
> synchronized (lock) {
> LOG.info("Starting High Availability Services");
> ...
> }
> // wait for resource manager to finish
> resourceManager.getTerminationFuture().get();
> {code}
> resourceManager#getTerminationFuture() is called without holding lock.
> We should store the value returned from
> resourceManager#getTerminationFuture() inside the synchronized block.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink issue #3348: [FLINK-5090] [network] Add metrics for details about inbo...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/3348 Did @StephanEwen provide the initial implementation, or why is the first commit by him? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5090) Expose optionally detailed metrics about network queue lengths
[ https://issues.apache.org/jira/browse/FLINK-5090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970314#comment-15970314 ] ASF GitHub Bot commented on FLINK-5090: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/3348 Did @StephanEwen provide the initial implementation, or why is the first commit by him? > Expose optionally detailed metrics about network queue lengths > -- > > Key: FLINK-5090 > URL: https://issues.apache.org/jira/browse/FLINK-5090 > Project: Flink > Issue Type: New Feature > Components: Metrics, Network >Affects Versions: 1.1.3 >Reporter: Stephan Ewen >Assignee: Stephan Ewen > > For debugging purposes, it is important to have access to more detailed > metrics about the length of network input and output queues. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-5090) Expose optionally detailed metrics about network queue lengths
[ https://issues.apache.org/jira/browse/FLINK-5090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970311#comment-15970311 ] ASF GitHub Bot commented on FLINK-5090: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/3348 Will try merging this now. > Expose optionally detailed metrics about network queue lengths > -- > > Key: FLINK-5090 > URL: https://issues.apache.org/jira/browse/FLINK-5090 > Project: Flink > Issue Type: New Feature > Components: Metrics, Network >Affects Versions: 1.1.3 >Reporter: Stephan Ewen >Assignee: Stephan Ewen > > For debugging purposes, it is important to have access to more detailed > metrics about the length of network input and output queues. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] flink issue #3348: [FLINK-5090] [network] Add metrics for details about inbo...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/3348 Will try merging this now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6195) Move gelly-examples jar from opt to examples
[
https://issues.apache.org/jira/browse/FLINK-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970305#comment-15970305
]
ASF GitHub Bot commented on FLINK-6195:
---
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/3691
Changes to the build and docs look good. I would propose moving the changes
to the EdgeList/AsmTestBase into a separate commit though.
> Move gelly-examples jar from opt to examples
>
>
> Key: FLINK-6195
> URL: https://issues.apache.org/jira/browse/FLINK-6195
> Project: Flink
> Issue Type: Sub-task
> Components: Build System
>Affects Versions: 1.3.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Trivial
> Fix For: 1.3.0
>
>
> The {{opt}} directory should be reserved for Flink JARs which users may
> optionally move to {{lib}} to be loaded by the runtime.
> {{flink-gelly-examples}} is a user program so is being moved to the
> {{examples}} folder.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink issue #3691: [FLINK-6195] [build] Move gelly-examples jar from opt to ...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/3691 Changes to the build and docs look good. I would propose moving the changes to the EdgeList/AsmTestBase into a separate commit though. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6257) Post-pass OVER windows
[
https://issues.apache.org/jira/browse/FLINK-6257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970303#comment-15970303
]
ASF GitHub Bot commented on FLINK-6257:
---
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/3697
Please provide some description as to the changes that you made, it's a
2000+ line diff after all.
> Post-pass OVER windows
> --
>
> Key: FLINK-6257
> URL: https://issues.apache.org/jira/browse/FLINK-6257
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: sunjincheng
>Priority: Critical
>
> The OVER windows have been implemented by several contributors.
> We should do a post pass over the contributed code and improve a few things.
> * Functionality
> ** Currently every time attribute is allowed as ORDER BY attribute. We must
> check that this is actually a time indicator ({{procTime()}}, {{rowTime()}})
> and that the order is ASCENDING.
> * Documentation
> ** Add documentation for OVER windows
> * Code style
> ** Consistent naming of {{ProcessFunctions}} and methods
> * Tests
> ** Move the OVER window tests out of SqlITCase into a dedicated class
> ** Move the OVER window tests out of WindowAggregateTest into a dedicated
> class
> ** Add tests based on the test harness for all {{ProcessFunctions}} similar
> to {{BoundedProcessingOverRangeProcessFunction}}. The tests should include
> exact boundary checks for range windows and check for proper parallelization
> with multiple keys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink issue #3697: [FLINK-6257][table]Optimize test cases
Github user zentol commented on the issue: https://github.com/apache/flink/pull/3697 Please provide some description as to the changes that you made, it's a 2000+ line diff after all. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6295) use LoadingCache instead of WeakHashMap to lower latency
[ https://issues.apache.org/jira/browse/FLINK-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970302#comment-15970302 ] ASF GitHub Bot commented on FLINK-6295: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/3709 It shouldn't matter for the display in the web-frontend in which data structure the cached ExecutionGraphs are being held. We are caching the actual ExecutionGraph that the JobManager works with and not some copy, thus there is simply no way for the handler to work with outdated data. This implies that the entire premise of this issue is flawed. It's more likely some web-related issue like the page not refreshing automatically or being cached by the browser for an inexplicable long time. > use LoadingCache instead of WeakHashMap to lower latency > > > Key: FLINK-6295 > URL: https://issues.apache.org/jira/browse/FLINK-6295 > Project: Flink > Issue Type: Bug > Components: Webfrontend >Reporter: Tao Wang >Assignee: Tao Wang > > Now in ExecutionGraphHolder, which is used in many handlers, we use a > WeakHashMap to cache ExecutionGraph(s), which is only sensitive to garbage > collection. > The latency is too high when JVM do GC rarely, which will make status of jobs > or its tasks unmatched with the real ones. > LoadingCache is a common used cache implementation from guava lib, we can use > its time based eviction to lower latency of status update. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] flink issue #3709: [FLINK-6295]use LoadingCache instead of WeakHashMap to lo...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/3709 It shouldn't matter for the display in the web-frontend in which data structure the cached ExecutionGraphs are being held. We are caching the actual ExecutionGraph that the JobManager works with and not some copy, thus there is simply no way for the handler to work with outdated data. This implies that the entire premise of this issue is flawed. It's more likely some web-related issue like the page not refreshing automatically or being cached by the browser for an inexplicable long time. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6275) Unprotected access to resourceManager in YarnFlinkApplicationMasterRunner#runApplicationMaster
[
https://issues.apache.org/jira/browse/FLINK-6275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970256#comment-15970256
]
ASF GitHub Bot commented on FLINK-6275:
---
GitHub user zhangminglei opened a pull request:
https://github.com/apache/flink/pull/3724
[FLINK-6275] [yarn] Fix unprotected access to resourceManage in YarnF…
…linkApplicationMasterRunner.
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhangminglei/flink flink_6275
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3724.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3724
commit f0c6971538673930bbca9a9e00ee53d4a26e972c
Author: rice.zhang
Date: 2017-04-16T06:38:22Z
[FLINK-6275] [yarn] Fix unprotected access to resourceManage in
YarnFlinkApplicationMasterRunner.
> Unprotected access to resourceManager in
> YarnFlinkApplicationMasterRunner#runApplicationMaster
> --
>
> Key: FLINK-6275
> URL: https://issues.apache.org/jira/browse/FLINK-6275
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> {code}
> // wait for resource manager to finish
> resourceManager.getTerminationFuture().get();
> {code}
> The above is outside synchronized block.
> @GuardedBy indicates that access should be protected.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink pull request #3724: [FLINK-6275] [yarn] Fix unprotected access to reso...
GitHub user zhangminglei opened a pull request:
https://github.com/apache/flink/pull/3724
[FLINK-6275] [yarn] Fix unprotected access to resourceManage in YarnFâ¦
â¦linkApplicationMasterRunner.
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhangminglei/flink flink_6275
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3724.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3724
commit f0c6971538673930bbca9a9e00ee53d4a26e972c
Author: rice.zhang
Date: 2017-04-16T06:38:22Z
[FLINK-6275] [yarn] Fix unprotected access to resourceManage in
YarnFlinkApplicationMasterRunner.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-6275) Unprotected access to resourceManager in YarnFlinkApplicationMasterRunner#runApplicationMaster
[
https://issues.apache.org/jira/browse/FLINK-6275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970251#comment-15970251
]
ASF GitHub Bot commented on FLINK-6275:
---
Github user zhangminglei commented on the issue:
https://github.com/apache/flink/pull/3707
@zentol Could you please reopen this pr again ? It seems belong to my
misoperation.
> Unprotected access to resourceManager in
> YarnFlinkApplicationMasterRunner#runApplicationMaster
> --
>
> Key: FLINK-6275
> URL: https://issues.apache.org/jira/browse/FLINK-6275
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> {code}
> // wait for resource manager to finish
> resourceManager.getTerminationFuture().get();
> {code}
> The above is outside synchronized block.
> @GuardedBy indicates that access should be protected.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink issue #3707: [FLINK-6275] [yarn] Fix unprotected access to resourceMan...
Github user zhangminglei commented on the issue: https://github.com/apache/flink/pull/3707 @zentol Could you please reopen this pr again ? It seems belong to my misoperation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6275) Unprotected access to resourceManager in YarnFlinkApplicationMasterRunner#runApplicationMaster
[
https://issues.apache.org/jira/browse/FLINK-6275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970245#comment-15970245
]
ASF GitHub Bot commented on FLINK-6275:
---
Github user zhangminglei closed the pull request at:
https://github.com/apache/flink/pull/3707
> Unprotected access to resourceManager in
> YarnFlinkApplicationMasterRunner#runApplicationMaster
> --
>
> Key: FLINK-6275
> URL: https://issues.apache.org/jira/browse/FLINK-6275
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> {code}
> // wait for resource manager to finish
> resourceManager.getTerminationFuture().get();
> {code}
> The above is outside synchronized block.
> @GuardedBy indicates that access should be protected.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] flink pull request #3707: [FLINK-6275] [yarn] Fix unprotected access to reso...
Github user zhangminglei closed the pull request at: https://github.com/apache/flink/pull/3707 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
