Hi,
I have a simple Flink job which is reading the data from Kafka topic and
generating minute aggregations and writing them to Elastic Search.
I am running the Flink Job (Flink Yarn Session) on EMR Cluster and the Job
runs for an hour fine and then it is getting stopped and when i checked th
Hi Mars,
A few questions..
1. What version of Flink are you using?
2. Are you using the default ES sink, or did you write your own?
3. What class of EC2 slave are you using?
4. What’s the parallelism of the ES sink?
5. To verify the actual open file limit, you need to…
* scp your private ke
Hi,
I am using 1.10.0 version of Flink on EMR.
I am not using the Default Flink Sink. I have a Sink Function on the Stream
and with in the invoke function i am creating a Data Structure (VO) and
putting it in the Map.
The EMR Step function i am running is. a Spring based FLink Job and i have
Hi Sateesh,
my suspicion would be that your custom Sink Function is leaking connections
(which also count for the file limit). Is there a reason that you cannot
use the ES connector of Flink?
I might have more ideas when you share your sink function.
Best,
Arvid
On Sun, Sep 27, 2020 at 7:16 PM
