Hi, I suppose you apply windowing to a keyed stream or SQL time-windowed join? Globally windowed streams are non-parallel and processed/stored in one slot.
In case of keyed stream, total range of key values is distributed among slots. Each slot processes/stores only a subrange of keys. Window state is stored per key then. This implies that each slot stores its own data, not the same. The keyed state is not shared among slots in the same JVM. Best, Andrey > On 23 Dec 2018, at 21:35, Anil <anilsingh....@gmail.com> wrote: > > I have a setup for Flink(1.4.2) with YARN. I'm using Flink Yarn Client for > deploying my jobs to Yarn Cluster. > > In the current setup parallelism was directly mapped to the number of cores, > with each parallel instance of the job running in one container. So for a > parallelism of 9, there are 10 containers - 1 JM and 9 TM and each container > has 1 core. Each container(or each parallel instance) has one task manager > and each slot holds the entire pipeline for the job. > > Most of the jobs have a join with the window storing data for last ⅔ hours. > As per my understanding here, > each container will save it's own copy of the this last 2/3 hours data and > this is not shared between two container. > > Since this window data will be same across each container, I feel if I could > have one task manager with with multiple task slot that could share this > window data I could save a lot on my resources (each container won't need to > maintain it's own copy of window data). If I had 3 container each with one > TM and 3 Task Slot each, then I would need only 3 containers for my job to > achieve a parallelism of 9 (each task slot will hold the entire job > pipeline, so each container helps me achieve a parallelism of 3 > individually). I'm assuming that this window data will be shared among all > parallel instance running in different task slot in each container. Please > correct me here. > > As per flink docs - > > Having multiple slots means more subtasks share the same JVM. Tasks in the > same JVM share TCP connections (via multiplexing) and heartbeat messages. > They may also share data sets and data structures, thus reducing the > per-task overhead. > > > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
