Hi Vishnu,
I hope the following will help answer your question:
1) Elements are first split by key (apart from global windows) and then are put
into windows. In other words, windows are keyed.
2) A window belonging to a certain key is handled by a single node. In other
words, no matter how big the window is, its
state (the elements it contains) will never be split between two or
more nodes.
3) Where the state is stored, depends on your state backend. Currently Flink
supports an in-memory one, a filesystem one, and
a rocksDB one which is in the middle (first in-memory and then disk
when needed). Of course you can implement your own.
From the above, you can see that if you use the memory-backed state backend,
then your window size is limited by the memory
available at each of your nodes. If you use the fs state backend, then your
state is stored on disk. Finally, rocksDB will initially
use RAM and then spill on disk when no more memory is available.
Here I have to add that the window documentation is currently being re-written
to explain new features introduced in Flink 1.1,
which include more flexible handling of late events and more explicit state
garbage collection.
So please stay tuned!
I hope this helps at answering your question,
Kostas
> On Jun 27, 2016, at 8:22 PM, Vishnu Viswanath <vishnu.viswanat...@gmail.com>
> wrote:
>
> Hi All,
>
> - Is there any restriction on the size of a window in Flink with respect to
> the memory of the nodes?
> - What happens if a window size grows more than size of a node, will it be
> split into multiple nodes?
>
> if I am going to have a huge window, should I have fewer nodes with more
> memory.
> Is there any documentation on how memory is managed/handled in the case of
> windows and also in the case of joins.
>
> Regards,
> Vishnu
