Hi Amol, The memory consumption depends on the query/operation that you are doing. Time-based operations like group-window-aggregations, over-window-aggregations, or window-joins can automatically clean up their state once data is not no longer needed. Operations such as non-windowed aggregations or joins have to persist all data forever in state to guarantee absolute correctness. However, you can also configure an idle state retention time [1] to remove state that has not been accessed for a certain time.
Regarding you questions: 1) Unless you configure the idle state retention time, state is kept as long as needed to guarantee correctness, potentially forever. 2) Queries use Flink's regular state features, i.e., you can configure the RocksDBStateBackend to manage state on disk. 3) This depends on your query and the distribution of your data. Best, Fabian [1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/table/streaming.html#idle-state-retention-time 2018-07-04 7:46 GMT+02:00 Amol S - iProgrammer <am...@iprogrammer.com>: > Hello folks, > > I am using flink table api to join multiple tables and create a single > table from them. I have some doubts in my mind. > > 1. How long the query will maintain partial results per key and how it > maintains state of each key? > > 2. If it is maintains state in memory then the memory will continuously > grows and it leads to memory over head. > > 3. How much RAM server needs to handle 10,000 per seconds incoming records > of average size of 3KB. > > ----------------------------------------------- > *Amol Suryawanshi* > Java Developer > am...@iprogrammer.com > > > *iProgrammer Solutions Pvt. Ltd.* > > > > *Office 103, 104, 1st Floor Pride Portal,Shivaji Housing Society, > Bahiratwadi,Near Hotel JW Marriott, Off Senapati Bapat Road, Pune - 411016, > MH, INDIA.**Phone: +91 9689077510 | Skype: amols_iprogrammer* > www.iprogrammer.com <sac...@iprogrammer.com> > ------------------------------------------------ >
