Hi Kostas, Thank you. Yes 2) was exactly what I wanted to know.
- So if I am using RocksDB as state backend, does that mean that I don't have to worry much about the memory available per node since RocksDB will use RAM and Disk to store the window state? Regards, Vishnu On Mon, Jun 27, 2016 at 6:19 PM, Kostas Kloudas <k.klou...@data-artisans.com > wrote: > Hi Vishnu, > > I hope the following will help answer your question: > > 1) Elements are first split by key (apart from global windows) and then > are put into windows. In other words, windows are keyed. > 2) A window belonging to a certain key is handled by a single node. In > other words, no matter how big the window is, its > state (the elements it contains) will never be split between two > or more nodes. > 3) Where the state is stored, depends on your state backend. Currently > Flink supports an in-memory one, a filesystem one, and > a rocksDB one which is in the middle (first in-memory and then > disk when needed). Of course you can implement your own. > > From the above, you can see that if you use the memory-backed state > backend, then your window size is limited by the memory > available at each of your nodes. If you use the fs state backend, then > your state is stored on disk. Finally, rocksDB will initially > use RAM and then spill on disk when no more memory is available. > > Here I have to add that the window documentation is currently being > re-written to explain new features introduced in Flink 1.1, > which include more flexible handling of late events and more explicit > state garbage collection. > > So please stay tuned! > > I hope this helps at answering your question, > Kostas > > > On Jun 27, 2016, at 8:22 PM, Vishnu Viswanath < > vishnu.viswanat...@gmail.com> wrote: > > > > Hi All, > > > > - Is there any restriction on the size of a window in Flink with respect > to the memory of the nodes? > > - What happens if a window size grows more than size of a node, will it > be split into multiple nodes? > > > > if I am going to have a huge window, should I have fewer nodes with more > memory. > > Is there any documentation on how memory is managed/handled in the case > of windows and also in the case of joins. > > > > Regards, > > Vishnu > >
