Hello fabian, Thanks for your quick response,
According to above conversation flink will persist state forever for non windowed operations. I want to know how flink persiat the state i.e. Database or file system or in memory etc. On Wed, 4 Jul 2018 at 2:12 PM, Fabian Hueske <fhue...@gmail.com> wrote: > Hi Amol, > > The memory consumption depends on the query/operation that you are doing. > Time-based operations like group-window-aggregations, > over-window-aggregations, or window-joins can automatically clean up their > state once data is not no longer needed. > Operations such as non-windowed aggregations or joins have to persist all > data forever in state to guarantee absolute correctness. > However, you can also configure an idle state retention time [1] to remove > state that has not been accessed for a certain time. > > Regarding you questions: > > 1) Unless you configure the idle state retention time, state is kept as > long as needed to guarantee correctness, potentially forever. > 2) Queries use Flink's regular state features, i.e., you can configure the > RocksDBStateBackend to manage state on disk. > 3) This depends on your query and the distribution of your data. > > Best, Fabian > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/table/streaming.html#idle-state-retention-time > > 2018-07-04 7:46 GMT+02:00 Amol S - iProgrammer <am...@iprogrammer.com>: > >> Hello folks, >> >> I am using flink table api to join multiple tables and create a single >> table from them. I have some doubts in my mind. >> >> 1. How long the query will maintain partial results per key and how it >> maintains state of each key? >> >> 2. If it is maintains state in memory then the memory will continuously >> grows and it leads to memory over head. >> >> 3. How much RAM server needs to handle 10,000 per seconds incoming >> records >> of average size of 3KB. >> >> ----------------------------------------------- >> *Amol Suryawanshi* >> Java Developer >> am...@iprogrammer.com >> >> >> *iProgrammer Solutions Pvt. Ltd.* >> >> >> >> *Office 103, 104, 1st Floor Pride Portal,Shivaji Housing Society, >> Bahiratwadi,Near Hotel JW Marriott, Off Senapati Bapat Road, Pune - >> 411016, >> MH, INDIA.**Phone: +91 9689077510 | Skype: amols_iprogrammer* >> www.iprogrammer.com <sac...@iprogrammer.com> >> ------------------------------------------------ >> > -- ----------------------------------------------- *Amol Suryawanshi* Java Developer am...@iprogrammer.com *iProgrammer Solutions Pvt. Ltd.* *Office 103, 104, 1st Floor Pride Portal,Shivaji Housing Society, Bahiratwadi,Near Hotel JW Marriott, Off Senapati Bapat Road, Pune - 411016, MH, INDIA.**Phone: +91 9689077510 | Skype: amols_iprogrammer* www.iprogrammer.com <sac...@iprogrammer.com> ------------------------------------------------
