Re: Tuning checkpoint

2018-08-13 Thread Fabian Hueske
Hi Mingliang, let me answer your second question first: > Another question is about the alignment buffer, I thought it was only used for multiple input stream cases. But for keyed process function , what is actually aligned? When a task sends records to multiple downstream tasks (task not operat

Re: Tuning checkpoint

2018-08-13 Thread 祁明良
Thank you for this great answer, Fabian. Regarding the yarn JVM heap size, I tried to change containerized.heap-cutoff-ratio: 0.25 And it somehow looks like working, but the actually memory needed for rocksdb still looks like a blackbox to me. I see there’s already a JIRA ticket talking about t

Re: Tuning checkpoint

2018-08-13 Thread Hequn Cheng
Hi mingliang, Considering your first question. I answered it on stack overflow[1]. Hope it helps. Best, Hequn [1] https://stackoverflow.com/questions/51832577/what-may-probably-cause-large-alignment-duration-for-flink-job On Tue, Aug 14, 2018 at 10:16 AM, 祁明良 wrote: > Thank you for this grea