[jira] [Commented] (FLINK-8812) Possible resource leak in Flip6

2018-03-17 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-8812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403555#comment-16403555
 ] 

ASF GitHub Bot commented on FLINK-8812:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/5713

[FLINK-8812] [flip6] Set managed memory for TaskExecutor to 80 MB in 
MiniCluster

## What is the purpose of the change

In order to avoid problems with OOM exceptions, this commit sets the managed
memory to 80 MB for TaskExecutors started by the MiniCluster.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink findResourceLeak

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5713.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5713


commit 47f3f515ea75db20528079fef289de5120ae95c5
Author: Till Rohrmann 
Date:   2018-03-16T18:16:49Z

[FLINK-9016] [flip6] Properly unregister jobs from JobMetricGroup

This commit properly removes jobs from the JobMetricGroup once a job
has reached a terminal state.

commit e42749cea4a0d35850c9482c8b31274aa081dbfa
Author: Till Rohrmann 
Date:   2018-03-17T16:53:35Z

[FLINK-8812] [flip6] Set managed memory for TaskExecutor to 80 MB in 
MiniCluster

In order to avoid problems with OOM exceptions, this commit sets the managed
memory to 80 MB for TaskExecutors started by the MiniCluster.




> Possible resource leak in Flip6
> ---
>
> Key: FLINK-8812
> URL: https://issues.apache.org/jira/browse/FLINK-8812
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Reporter: Chesnay Schepler
>Assignee: Till Rohrmann
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> In this build (https://travis-ci.org/zentol/flink/builds/347373839) I set the 
> codebase to flip6 for half the profiles to find failing tests.
> The "libraries" job (https://travis-ci.org/zentol/flink/jobs/347373851) 
> failed with an OutOfMemoryError.
> This could mean that there is a memory-leak somewhere.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5713: [FLINK-8812] [flip6] Set managed memory for TaskEx...

2018-03-17 Thread tillrohrmann 
GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/5713

[FLINK-8812] [flip6] Set managed memory for TaskExecutor to 80 MB in 
MiniCluster

## What is the purpose of the change

In order to avoid problems with OOM exceptions, this commit sets the managed
memory to 80 MB for TaskExecutors started by the MiniCluster.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink findResourceLeak

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5713.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5713


commit 47f3f515ea75db20528079fef289de5120ae95c5
Author: Till Rohrmann 
Date:   2018-03-16T18:16:49Z

[FLINK-9016] [flip6] Properly unregister jobs from JobMetricGroup

This commit properly removes jobs from the JobMetricGroup once a job
has reached a terminal state.

commit e42749cea4a0d35850c9482c8b31274aa081dbfa
Author: Till Rohrmann 
Date:   2018-03-17T16:53:35Z

[FLINK-8812] [flip6] Set managed memory for TaskExecutor to 80 MB in 
MiniCluster

In order to avoid problems with OOM exceptions, this commit sets the managed
memory to 80 MB for TaskExecutors started by the MiniCluster.




---

[jira] [Created] (FLINK-9016) Unregister jobs from JobMetricGroup after termination

2018-03-17 Thread Till Rohrmann (JIRA) 
Till Rohrmann created FLINK-9016:


 Summary: Unregister jobs from JobMetricGroup after termination
 Key: FLINK-9016
 URL: https://issues.apache.org/jira/browse/FLINK-9016
 Project: Flink
  Issue Type: Bug
  Components: Metrics
Affects Versions: 1.5.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
 Fix For: 1.5.0


In order to free resources and unregister metrics, jobs should be properly 
unregistered from the {{JobMetricGroup}} once they have reached terminal state.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9009) Error| You are creating too many HashedWheelTimer instances. HashedWheelTimer is a shared resource that must be reused across the application, so that only a few insta

2018-03-17 Thread Pankaj (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-9009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403461#comment-16403461
 ] 

Pankaj commented on FLINK-9009:
---

Stacktrace

Sink: Cassandra Sink, Filter -> Flat Map -> Sink: Unnamed, Filter -> Flat Map 
-> Sink: Cassandra Sink, Filter -> Flat Map -> Sink: Unnamed, Filter -> Flat 
Map -> Sink: *Cassandra Sink)(1/10*) switched to RUNNING

17.03.2018 14:33:59.740 [OUT] [ERROR] [ ] [ ] 
*io.netty.util.ResourceLeakDetector LEAK: You are creating too many 
HashedWheelTimer instances. HashedWheelTimer is a shared resource that must be 
reused across the JVM,so that only a few instances are create*d. # # 
java.lang.OutOfMemoryError: Java heap space # 
-XX:OnOutOfMemoryError="/opt/tomcat/bin/shutdown.sh 5" # Executing /bin/sh -c 
"/opt/tomcat/bin/shutdown.sh 5"... Mar 17, 2018 2:34:28 PM 
org.apache.catalina.startup.Catalina stopServer SEVERE: Could not contact 
localhost:8005. Tomcat may not be running.

Mar 17, 2018 2:34:28 PM org.apache.catalina.startup.Catalina stopServer SEVERE: 
Catalina.stop: java.net.ConnectException: Connection refused (Connection 
refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) 
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at 
java.net.Socket.connect(Socket.java:589) at 
java.net.Socket.connect(Socket.java:538)

---

"cluster16-nio-worker-1" #142 prio=5 os_prio=0 tid=0x7f97fc386000 nid=0xbe 
waiting for monitor entry [0x7f9782e86000] java.lang.Thread.State: BLOCKED 
(on object monitor) at 
*com.datastax.driver.core.Connection$10.operationComplete(Connection.java:547) 
at* 
com.datastax.driver.core.Connection$10.operationComplete(Connection.java:534) 
at 
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
 at 
io.netty.util.concurrent.DefaultPromise.notifyLateListener(DefaultPromise.java:621)
 at 
io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:138) at 
io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:93)
 at 
io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:28)
 at com.datastax.driver.core.Connection$Flusher.run(Connection.java:870) at 
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:358)
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
 at java.lang.Thread.run(Thread.java:748)

 

io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:89)
 at 
io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:643)
 at 
io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:700)
 at 
io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:636)
 at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:284) 
at 
io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:643)
 at 
io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:700)
 at 
io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:636)
 at 
io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:622)
 at 
io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:939) 
at io.netty.channel.AbstractChannel.write(AbstractChannel.java:234) at 
com.datastax.driver.core.Connection$Flusher.run(Connection.java:870) at 
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:358)
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
 at java.lang.Thread.run(Thread.java:748)

> Error| You are creating too many HashedWheelTimer instances.  
> HashedWheelTimer is a shared resource that must be reused across the 
> application, so that only a few instances are created.
> -
>
> Key: FLINK-9009
> URL: https://issues.apache.org/jira/browse/FLINK-9009
> Project: Flink
>  Issue Type: Bug
> Environment: Pass platform: Openshit
>Reporter: Pankaj
>Priority: Blocker
>
> Steps to reproduce:
> 1- Flink with Kafka as a consumer -> Writing stream to Cassandra using flink 
>

[jira] [Commented] (FLINK-7151) FLINK SQL support create temporary function and table

2018-03-17 Thread Shuyi Chen (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-7151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403335#comment-16403335
 ] 

Shuyi Chen commented on FLINK-7151:
---

[~yinhua], yes, you should be able to implement your own TableSourceFactory to 
do so.

> FLINK SQL support create temporary function and table
> -
>
> Key: FLINK-7151
> URL: https://issues.apache.org/jira/browse/FLINK-7151
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API  SQL
>Reporter: yuemeng
>Assignee: yuemeng
>Priority: Major
>
> Based on create temporary function and table.we can register a udf,udaf,udtf 
> use sql:
> {code}
> CREATE TEMPORARY function 'TOPK' AS 
> 'com..aggregate.udaf.distinctUdaf.topk.ITopKUDAF';
> INSERT INTO db_sink SELECT id, TOPK(price, 5, 'DESC') FROM kafka_source GROUP 
> BY id;
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)