Re: Flink checkpointing gets stuck

2017-03-08 Thread Ufuk Celebi
s several bugs that were since fixed, specifically one where >> the socket timeout was infinite. We updated this package to version 5.0.0 >> and everything is working smoothly now. >> >> >> >> From: Stephan Ewen [mailto:se...@apache.org] >> Sent: Sunday, February

Re: Flink checkpointing gets stuck

2017-03-07 Thread Stephan Ewen
et timeout was infinite. We updated this package to > version 5.0.0 and everything is working smoothly now. > > > > *From:* Stephan Ewen [mailto:se...@apache.org] > *Sent:* Sunday, February 26, 2017 4:47 PM > *To:* user@flink.apache.org > *Subject:* Re: Flink checkpointing gets stu

Re: Flink checkpointing gets stuck

2017-02-26 Thread Stephan Ewen
Thanks! This looks like a bigger example, involving MongoDB, etc. Are you able to reproduce this issue with a smaller example? It would also help to understand the problem better if we knew the topology a bit better. The stack traces look like "phase 1&2" want to send data (but are back

Re: Flink checkpointing gets stuck

2017-02-24 Thread Stephan Ewen
hai Kaplan <shai.kap...@microsoft.com> > wrote: > >> And now it's happening again >> >> -Original Message----- >> From: Shai Kaplan [mailto:shai.kap...@microsoft.com] >> Sent: Wednesday, February 22, 2017 12:02 PM >> To: user@flink.apache

Re: Flink checkpointing gets stuck

2017-02-23 Thread Robert Metzger
lto:shai.kap...@microsoft.com] > Sent: Wednesday, February 22, 2017 12:02 PM > To: user@flink.apache.org > Subject: RE: Flink checkpointing gets stuck > > I changed the checkpoint interval to 30 minutes, and also switched RocksDB > predefined options to FLASH_SSD_OPTIMIZED, as suggest

RE: Flink checkpointing gets stuck

2017-02-23 Thread Shai Kaplan
And now it's happening again -Original Message- From: Shai Kaplan [mailto:shai.kap...@microsoft.com] Sent: Wednesday, February 22, 2017 12:02 PM To: user@flink.apache.org Subject: RE: Flink checkpointing gets stuck I changed the checkpoint interval to 30 minutes, and also switched

RE: Flink checkpointing gets stuck

2017-02-22 Thread Shai Kaplan
? -Original Message- From: Ufuk Celebi [mailto:u...@apache.org] Sent: Tuesday, February 21, 2017 4:54 PM To: user@flink.apache.org Subject: Re: Flink checkpointing gets stuck Hey Shai! Thanks for reporting this. It's hard to tell what causes this from your email, but could you check

Re: Flink checkpointing gets stuck

2017-02-21 Thread vinay patil
ect the performance, not make the job totally stuck. > > > > Thanks. > > > > *From:* vinay patil [mailto:[hidden email] > <http:///user/SendEmail.jtp?type=node=11780=0>] > *Sent:* Tuesday, February 21, 2017 3:58 PM > *To:* [hidden email] <http:///user/SendEmail

Re: Flink checkpointing gets stuck

2017-02-21 Thread Ufuk Celebi
Hey Shai! Thanks for reporting this. It's hard to tell what causes this from your email, but could you check the checkpoint interface (https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/checkpoint_monitoring.html) and report how much progress the checkpoints make before

RE: Flink checkpointing gets stuck

2017-02-21 Thread Shai Kaplan
think it should only affect the performance, not make the job totally stuck. Thanks. From: vinay patil [mailto:vinay18.pa...@gmail.com] Sent: Tuesday, February 21, 2017 3:58 PM To: user@flink.apache.org Subject: Re: Flink checkpointing gets stuck Hi Shai, I was facing similar issue , however now

Re: Flink checkpointing gets stuck

2017-02-21 Thread vinay patil
bug in Flink or RocksDB? > > > -- > If you reply to this email, your message will be added to the discussion > below: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink- > checkpointing-gets-stuck-tp11776.html &

Flink checkpointing gets stuck

2017-02-21 Thread Shai Kaplan
Hi. I'm running a Flink 1.2 job with a 10 seconds checkpoint interval. After some running time (minutes-hours) Flink fails to save checkpoints, and stops processing records (I'm not sure if the checkpointing failure is the cause of the problem or just a symptom). After several checkpoints that