Re: Connection leak with flink elastic Sink

2018-12-13 Thread Vijay Bhaskar
Hi Gordon, We are using flink cluster 1.6.1, elastic search connector version: flink-connector-elasticsearch6_2.11 Attached the stack trace. Following are the max open file descriptor limit of theTask manager process and open connections to the elastic search cluster Regards Bhaskar *#lsof -p 620

Re: Connection leak with flink elastic Sink

2018-12-13 Thread Tzu-Li (Gordon) Tai
Hi, Besides the information that Chesnay requested, could you also provide a stack trace of the exception that caused the job to terminate in the first place? The Elasticsearch sink does indeed close the internally used Elasticsearch client, which should in turn properly release all resources [

Re: Connection leak with flink elastic Sink

2018-12-13 Thread Chesnay Schepler
Specifically which connector are you using, and which Flink version? On 12.12.2018 13:31, Vijay Bhaskar wrote: Hi We are using flink elastic sink which streams at the rate of 1000 events/sec, as described in https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/elasticsearch.ht

Connection leak with flink elastic Sink

2018-12-12 Thread Vijay Bhaskar
Hi We are using flink elastic sink which streams at the rate of 1000 events/sec, as described in https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/elasticsearch.html . We are observing connection leak of elastic connections. After few minutes all the open connections are exceedi