Re: How to verify if checkpoints are asynchronous or sync
I evaluated performance by looking at the number of input records processed over 10min and 30mins duration. Thanks, R On Wed, Jan 8, 2020 at 2:21 AM Congxian Qiu wrote: > If you want to figure out the performance problem, maybe async-profile[1] > can be helpful > [1] https://github.com/jvm-profiling-tools/async-profiler > Best, > Congxian > > > William C 于2020年1月8日周三 上午11:37写道： > >> Hallo >> >> on 2020/1/8 11:31, RKandoji wrote: >> > I'm running my job on a EC2 instance with 32 cores and according to the >> > documentation I tried to use as many task slots the number of cores, >> > numOfTaskSlots=32 and parallelism=32. But I noticed that >> the performance >> > is slightly degrading when I'm using 32 task slots. Performance seems >> > better at 26 task slots than >26 task slots. So I was trying to >> > understand if additional CPU cores are being utilized by checkpointing >> > or any other async (or background operations, in the process I was >> > trying to verify if the checkpointing is async. >> >> How did you evaluate the performance? >> It's may due to busy IO or thread competition or something else similiar. >> You'd better dig into more details via logging. >> >> Regards. >> >
Re: How to verify if checkpoints are asynchronous or sync
If you want to figure out the performance problem, maybe async-profile[1] can be helpful [1] https://github.com/jvm-profiling-tools/async-profiler Best, Congxian William C 于2020年1月8日周三 上午11:37写道： > Hallo > > on 2020/1/8 11:31, RKandoji wrote: > > I'm running my job on a EC2 instance with 32 cores and according to the > > documentation I tried to use as many task slots the number of cores, > > numOfTaskSlots=32 and parallelism=32. But I noticed that the performance > > is slightly degrading when I'm using 32 task slots. Performance seems > > better at 26 task slots than >26 task slots. So I was trying to > > understand if additional CPU cores are being utilized by checkpointing > > or any other async (or background operations, in the process I was > > trying to verify if the checkpointing is async. > > How did you evaluate the performance? > It's may due to busy IO or thread competition or something else similiar. > You'd better dig into more details via logging. > > Regards. >
Re: How to verify if checkpoints are asynchronous or sync
Hallo on 2020/1/8 11:31, RKandoji wrote: I'm running my job on a EC2 instance with 32 cores and according to the documentation I tried to use as many task slots the number of cores, numOfTaskSlots=32 and parallelism=32. But I noticed that the performance is slightly degrading when I'm using 32 task slots. Performance seems better at 26 task slots than >26 task slots. So I was trying to understand if additional CPU cores are being utilized by checkpointing or any other async (or background operations, in the process I was trying to verify if the checkpointing is async. How did you evaluate the performance? It's may due to busy IO or thread competition or something else similiar. You'd better dig into more details via logging. Regards.
Re: How to verify if checkpoints are asynchronous or sync
Sorry for not providing the context earlier. I'm running my job on a EC2 instance with 32 cores and according to the documentation I tried to use as many task slots the number of cores, numOfTaskSlots=32 and parallelism=32. But I noticed that the performance is slightly degrading when I'm using 32 task slots. Performance seems better at 26 task slots than >26 task slots. So I was trying to understand if additional CPU cores are being utilized by checkpointing or any other async (or background operations, in the process I was trying to verify if the checkpointing is async. Thanks, RKandoji On Tue, Jan 7, 2020 at 10:16 PM Congxian Qiu wrote: > Hi > > RocksDB supports Incremental and full snapshot, both are async, do you > want to verify whether it's incremental or full snapshot? but I don't know > an easy to get this information currently > > Best, > Congxian > > > Zhijiang 于2020年1月8日周三 上午10:56写道： > >> The log way is simple for tracing and you can also grep some keywords to >> find your requirement messages to avoid skimming through the whole large >> logs. >> I am not quite sure what's your specific motivation for doing this. >> Besides the log way, you can also monitor the thread stack for confirming >> whether it is happening, but maybe it is not very convenient. >> Another possible way is via the checkpoint metrics which would record the >> sync/async duration time, maybe it can also satisfy your requirements. >> >> Best, >> Zhijiang >> >> ---------------------- >> From:RKandoji >> Send Time:2020 Jan. 8 (Wed.) 10:23 >> To:William C >> Cc:user >> Subject:Re: How to verify if checkpoints are asynchronous or sync >> >> Thanks for the reply. >> I will check and enable debug logs specifically for the class that >> contains this log. >> But in general logs are already too huge and I'm trying to suppress some >> of them, so wondering if there is any other way? >> >> Thanks, >> RKandoji >> >> >> On Tue, Jan 7, 2020 at 7:50 PM William C wrote: >> Can you enable debug log to check with that? >> >> regards. >> >> on 2020/1/8 6:36, RKandoji wrote: >> > But I'm curious if there is way to verify if the checkpoints are >> > happening asynchronously or synchronously. >> > >> >> >>
Re: How to verify if checkpoints are asynchronous or sync
Hi RocksDB supports Incremental and full snapshot, both are async, do you want to verify whether it's incremental or full snapshot? but I don't know an easy to get this information currently Best, Congxian Zhijiang 于2020年1月8日周三 上午10:56写道： > The log way is simple for tracing and you can also grep some keywords to > find your requirement messages to avoid skimming through the whole large > logs. > I am not quite sure what's your specific motivation for doing this. > Besides the log way, you can also monitor the thread stack for confirming > whether it is happening, but maybe it is not very convenient. > Another possible way is via the checkpoint metrics which would record the > sync/async duration time, maybe it can also satisfy your requirements. > > Best, > Zhijiang > > -- > From:RKandoji > Send Time:2020 Jan. 8 (Wed.) 10:23 > To:William C > Cc:user > Subject:Re: How to verify if checkpoints are asynchronous or sync > > Thanks for the reply. > I will check and enable debug logs specifically for the class that > contains this log. > But in general logs are already too huge and I'm trying to suppress some > of them, so wondering if there is any other way? > > Thanks, > RKandoji > > > On Tue, Jan 7, 2020 at 7:50 PM William C wrote: > Can you enable debug log to check with that? > > regards. > > on 2020/1/8 6:36, RKandoji wrote: > > But I'm curious if there is way to verify if the checkpoints are > > happening asynchronously or synchronously. > > > > >
Re: How to verify if checkpoints are asynchronous or sync
The log way is simple for tracing and you can also grep some keywords to find your requirement messages to avoid skimming through the whole large logs. I am not quite sure what's your specific motivation for doing this. Besides the log way, you can also monitor the thread stack for confirming whether it is happening, but maybe it is not very convenient. Another possible way is via the checkpoint metrics which would record the sync/async duration time, maybe it can also satisfy your requirements. Best, Zhijiang -- From:RKandoji Send Time:2020 Jan. 8 (Wed.) 10:23 To:William C Cc:user Subject:Re: How to verify if checkpoints are asynchronous or sync Thanks for the reply. I will check and enable debug logs specifically for the class that contains this log. But in general logs are already too huge and I'm trying to suppress some of them, so wondering if there is any other way? Thanks, RKandoji On Tue, Jan 7, 2020 at 7:50 PM William C wrote: Can you enable debug log to check with that? regards. on 2020/1/8 6:36, RKandoji wrote: > But I'm curious if there is way to verify if the checkpoints are > happening asynchronously or synchronously. >
Re: How to verify if checkpoints are asynchronous or sync
Thanks for the reply. I will check and enable debug logs specifically for the class that contains this log. But in general logs are already too huge and I'm trying to suppress some of them, so wondering if there is any other way? Thanks, RKandoji On Tue, Jan 7, 2020 at 7:50 PM William C wrote: > Can you enable debug log to check with that? > > regards. > > on 2020/1/8 6:36, RKandoji wrote: > > But I'm curious if there is way to verify if the checkpoints are > > happening asynchronously or synchronously. > > >
Re: How to verify if checkpoints are asynchronous or sync
Can you enable debug log to check with that? regards. on 2020/1/8 6:36, RKandoji wrote: But I'm curious if there is way to verify if the checkpoints are happening asynchronously or synchronously.
How to verify if checkpoints are asynchronous or sync
Hi, I'm using Flink 1.9, BlinkPlanner and rocksDB for backend with checkpointing enabled. I understand that checkpointing is async by default for RocksDB. But I'm curious if there is way to verify if the checkpoints are happening asynchronously or synchronously. Please let me know. Thanks, RKandoji
