I have not found relying on partitions for parallelism as a disadvantage.
At flurry, we have several pipelines using both lower level API Kafka (for
legacy reasons) and kafka streams + kafka connect.
They process over 10B events per day at around 200k rps. We also use the
same system to send over
Thanks Peter, even with ECS we have autoscaling enabled but the issue is
during autoscaling we need to stop the job and start with new
parallelism which creates a downtime.
Thanks
On Fri, Nov 8, 2019 at 1:01 PM Peter Groesbeck
wrote:
> We use EMR instead of ECS but if that’s an option for your
We use EMR instead of ECS but if that’s an option for your team, you can
configure auto scaling rules in your cloud formation so that your task/job load
dynamically controls cluster sizing.
Sent from my iPhone
> On Nov 8, 2019, at 1:40 AM, Navneeth Krishnan
> wrote:
>
> Hello All,
>
> I
Hello All,
I have a streaming job running in production which is processing over 2
billion events per day and it does some heavy processing on each event. We
have been facing some challenges in managing flink in production like
scaling in and out, restarting the job with savepoint etc. Flink
