Re: Re[2]: Checkpointing threads

[Raymond Wilson]

Thanks Zhenya.

I have seen the link you provide has a lot of good information on this
system. But it does not talk about the check point writers in any detail.

I appreciate this cannot be a bottleneck, my question is more related to:
"If I have more check pointing threads will check points take less time".
In our case we use AWS EFS so if each checkpoint thread is spending
relatively long times blocking on write I/O to the persistent store then
more check points allow more concurrent writes to take place. Of course, if
the check point threads themselves utilise async I/O tasks and
interleave I/O activities on that basis then there may not be an
opportunity for performance improvement, but I am not an expert in the
Ignite code base :)

Raymond.


On Wed, Sep 7, 2022 at 7:51 PM Zhenya Stanilovsky via user <
user@ignite.apache.org> wrote:

>
> No, there is no any log and metrics suggestions and as i told earlier
> — this place can`t became a bottleneck, if you have any performance
> problems — describe them somehow wider and interesting reading here [1]
>
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood
>
>
> Thanks Zhenya.
>
> Is there any logging or metrics that would indicate if there was value
> increasing the size of this pool?
>
>
> On Fri, 2 Sep 2022 at 8:20 PM, Zhenya Stanilovsky via user <
> user@ignite.apache.org
> > wrote:
>
> Hi Raymond
>
> checkpoint threads is responsible for dumping modified pages, so you may
> consider it as io bound only operation and pool size is amount of
> disc writing workers.
> I think that default is enough and no need for raising it, but it also up
> to you.
>
>
>
> Hi,
>
> I am looking at our configuration of the Ignite checkpointing system to
> ensure we have it tuned correctly.
>
> There is a checkpointing thread pool defined, which defaults to 4 threads
> in size. I have not been able to find much of a discussion on when/how this
> pool size should be changed to reflect the node size Ignite is running on.
>
> In our case, we are running 16 core servers with 128 GB RAM with
> persistence on an NFS storage layer.
>
> Given the number of cores, and the relative latency of NFS compared to
> local SSD, is 4 checkpointing threads appropriate, or are we likely to see
> better performance if we increased it to 8 (or more)?
>
> If there is a discussion related to this a pointer to it would be good
> (it's not really covered in the performance tuning section).
>
> Thanks,
> Raymond.
>
> --
> 
> Raymond Wilson
> Trimble Distinguished Engineer, Civil Construction Software (CCS)
> 11 Birmingham Drive
> 
>  |
> 
>  Christchurch, New Zealand
> 
> raymond_wil...@trimble.com
> 
>
>
>
> 
>
>
>
>
>
>
>
>
>
>
>


-- 

Raymond Wilson
Trimble Distinguished Engineer, Civil Construction Software (CCS)
11 Birmingham Drive | Christchurch, New Zealand
raymond_wil...@trimble.com

Re[2]: Checkpointing threads

[Zhenya Stanilovsky via user]

No, there is no any log and metrics suggestions and as i told earlier — this 
place can`t became a bottleneck, if you have any performance problems — 
describe them somehow wider and interesting reading here [1]
 
[1]  
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood
  
>Thanks Zhenya. 
> 
>Is there any logging or metrics that would indicate if there was value 
>increasing the size of this pool?
> 
> 
>On Fri, 2 Sep 2022 at 8:20 PM, Zhenya Stanilovsky via user < 
>user@ignite.apache.org > wrote:
>>Hi  Raymond
>> 
>>checkpoint threads is responsible for dumping modified pages, so you may 
>>consider it as io bound only operation and pool size is amount of disc 
>>writing workers.
>>I think that default is enough and no need for raising it, but it also up to 
>>you.
>>   
>>>Hi,
>>> 
>>>I am looking at our configuration of the Ignite checkpointing system to 
>>>ensure we have it tuned correctly.
>>> 
>>>There is a checkpointing thread pool defined, which defaults to 4 threads in 
>>>size. I have not been able to find much of a discussion on when/how this 
>>>pool size should be changed to reflect the node size Ignite is running on.
>>> 
>>>In our case, we are running 16 core servers with 128 GB RAM with persistence 
>>>on an NFS storage layer.
>>> 
>>>Given the number of cores, and the relative latency of NFS compared to local 
>>>SSD, is 4 checkpointing threads appropriate, or are we likely to see better 
>>>performance if we increased it to 8 (or more)?
>>> 
>>>If there is a discussion related to this a pointer to it would be good (it's 
>>>not really covered in the performance tuning section).
>>> 
>>>Thanks,
>>>Raymond.
>>>  --
>>>
>>>Raymond Wilson
>>>Trimble Distinguished Engineer, Civil Construction Software (CCS)
>>>11 Birmingham Drive   |   Christchurch, New Zealand
>>>raymond_wil...@trimble.com
>>>         
>>> 
>> 
>> 
>> 
>>