Re: Flink cluster deployment strategy

2020-08-13 Thread sidhant gupta
Thanks, I will check it out. On Thu, 13 Aug, 2020, 7:55 PM Arvid Heise, wrote: > Hi Sidhant, > > If you are starting fresh with Flink, I strongly recommend to skip ECS and > EMR and directly go to a kubernetes-based solution. Scaling is much easier > on K8s, there will be some kind of

Re: Flink cluster deployment strategy

2020-08-13 Thread Arvid Heise
Hi Sidhant, If you are starting fresh with Flink, I strongly recommend to skip ECS and EMR and directly go to a kubernetes-based solution. Scaling is much easier on K8s, there will be some kind of autoscaling coming in the next release, and the best of it all: you even have the option to go to a

Re: Flink cluster deployment strategy

2020-08-11 Thread Till Rohrmann
Hi Sidhant, see the inline comments for answers On Tue, Aug 11, 2020 at 3:10 PM sidhant gupta wrote: > Hi Till, > > Thanks for your response. > I have few queries though as mentioned below: > (1) Can flink be used in map-reduce fashion with data streaming api ? > What do you understand as

Re: Flink cluster deployment strategy

2020-08-11 Thread sidhant gupta
Hi Till, Thanks for your response. I have few queries though as mentioned below: (1) Can flink be used in map-reduce fashion with data streaming api ? (2) Does it make sense to use aws EMR if we are not using flink in map-reduce fashion with streaming api ? (3) Can flink cluster be auto scaled

Re: Flink cluster deployment strategy

2020-08-11 Thread Till Rohrmann
Hi Sidhant, I am not an expert on AWS services but I believe that EMR might be a bit easier to start with since AWS EMR comes with Flink support out of the box [1]. On ECS I believe that you would have to set up the containers yourself. Another interesting deployment option could be to use

Flink cluster deployment strategy

2020-08-11 Thread sidhant gupta
Hi all, I'm kind of new to flink cluster deployment. I wanted to know which flink cluster deployment and which job mode in aws is better in terms of ease of deployment, maintenance, HA, cost, etc. As of now I am considering aws EMR vs ECS (docker containers). We have a usecase of setting up a