Re: Durable Memory and Ignite Persistence Tuning

[Denis Magda]

Thanks Ivan, *no less* sounds best for me.

Prachi, please do final editing of the doc:
https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning

—
Denis

> On Sep 15, 2017, at 12:24 AM, Ivan Rakov  wrote:
> 
> Denis,
> 
> Yes, Ignite page size should be *greater or equal* than OS page cache size 
> and SSD page size. I mentioned it in advice list:
> 
>> page size of Ignite shouldn't be less than SSD page size
>> Page size of Ignite shouldn't be less than OS page size
> 
> Sorry for vague wording.
> We should fix this in documentation.
> 
> 
> Best Regards,
> Ivan Rakov
> 
> On 14.09.2017 3:46, Denis Magda wrote:
>> Ivan,
>> 
>> Documented:
>> https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning 
>> 
>> 
>> However, I’m a bit confused with you 6. point regarding the page size 
>> calculation. Should Ignite page size be always less than the SSD’s page size 
>> and OS page cache size? Or it can be equal? For instance if the OS page 
>> cache size is 4 KB what should be Ignite’s page size? That’s the section 
>> about this: 
>> https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning#section-page-size
>>  
>> 
>> 
>> —
>> Denis
>> 
>>> On Sep 13, 2017, at 2:29 AM, Ivan Rakov  wrote:
>>> 
>>> Folks,
>>> 
>>> We had some experience of benchmarking Ignite with persistent store on SSD. 
>>> I think we can share some helpful advice. None of them require changing 
>>> configuration of Ignite or persistent store.
>>> 
>>> *Tuning advice for users*
>>> 
>>> 1) Be prepared for LFS performance decrease after several hours of 
>>> intensive load. Unfortunately, that's how SSD drives work: 
>>> http://codecapsule.com/2014/02/12/coding-for-ssds-part-2-architecture-of-an-ssd-and-benchmarking/
>>> Consider buying fast production-level SSD drives.
>>> 2) Consider using separate drives for LFS files and WAL. Ignite actively 
>>> performs writes to both LFS and WAL under intensive load, and having two 
>>> devices will double your throughput limit.
>>> 3) Over-provision your SSD. Performance of random writes on 50% filled disk 
>>> is much better than on 90% filled. SSD Over-Provisioning And Its Benefits: 
>>> http://www.seagate.com/ru/ru/tech-insights/ssd-over-provisioning-benefits-master-ti/
>>> 4) Leave free space in RAM to let OS use page cache and optimize writes. 
>>> Total size of all memory policies shouldn't exceed 70% of your RAM.
>>> 5) Make sure that OS doesn't utilize swap. If you use Unix, best option is 
>>> set vm.swappiness to 0.
>>> 6) Try to find out page size of your SSD. Ideally, page size of Ignite 
>>> shouldn't be less than SSD page size. Possible approaches:
>>> Find it in device specification (some manufacturers don't reveal it)
>>> Try running SSD benchmarks
>>> If you are not sure, just set page size to 4K. As various benchmarks use 4K 
>>> pages, manufacturers have to adapt drives for 4K random write workload. 
>>> Whitepaper from Intel showing that 4K pages are enough: 
>>> https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/ssd-server-storage-applications-paper.pdf
>>> Check your OS page cache size. Page size of Ignite shouldn't be less than 
>>> OS page size. How to check OS cache page size in Unix: 
>>> https://unix.stackexchange.com/questions/128213/how-is-page-size-determined-in-virtual-address-space
>>> 
>>> 
>>> Best Regards,
>>> Ivan Rakov
>>> 
>>> On 01.09.2017 21:08, Denis Magda wrote:
 Igniters,
 
 I see a lot of complains regarding the performance of the subj on the user 
 list. At the same time, I do believe that in most scenarios it’s a lack of 
 knowledge that we keep in secret.
 
 It's time to document Durable Memory and its Native Persistence tuning 
 parameters. Let's start doing this for Linux based deployments first. Here 
 is what we have for now (which is almost nothing): 
 https://apacheignite.readme.io/docs/durable-memory-tuning 
 
 
 Ideally, at some point we have to come up with doc like this: 
 https://access.redhat.com/sites/default/files/attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf
 
 Please share your expertise in a form of settings that have to be put on 
 the paper. We put them in JIRA and document afterwords:
 https://issues.apache.org/jira/browse/IGNITE-6246 
 
 
 —
 Denis
>> 
> 

Re: Durable Memory and Ignite Persistence Tuning

[Ivan Rakov]

Denis,

Yes, Ignite page size should be *greater or equal* than OS page cache 
size and SSD page size. I mentioned it in advice list:



page size of Ignite shouldn't be less than SSD page size
Page size of Ignite shouldn't be less than OS page size


Sorry for vague wording.
We should fix this in documentation.


Best Regards,
Ivan Rakov

On 14.09.2017 3:46, Denis Magda wrote:

Ivan,

Documented:
https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning 


However, I’m a bit confused with you 6. point regarding the page size calculation. 
Should Ignite page size be always less than the SSD’s page size and OS page cache 
size? Or it can be equal? For instance if the OS page cache size is 4 KB what should 
be Ignite’s page size? That’s the section about this: 
https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning#section-page-size 


—
Denis


On Sep 13, 2017, at 2:29 AM, Ivan Rakov  wrote:

Folks,

We had some experience of benchmarking Ignite with persistent store on SSD. I 
think we can share some helpful advice. None of them require changing 
configuration of Ignite or persistent store.

*Tuning advice for users*

1) Be prepared for LFS performance decrease after several hours of intensive 
load. Unfortunately, that's how SSD drives work: 
http://codecapsule.com/2014/02/12/coding-for-ssds-part-2-architecture-of-an-ssd-and-benchmarking/
Consider buying fast production-level SSD drives.
2) Consider using separate drives for LFS files and WAL. Ignite actively 
performs writes to both LFS and WAL under intensive load, and having two 
devices will double your throughput limit.
3) Over-provision your SSD. Performance of random writes on 50% filled disk is 
much better than on 90% filled. SSD Over-Provisioning And Its Benefits: 
http://www.seagate.com/ru/ru/tech-insights/ssd-over-provisioning-benefits-master-ti/
4) Leave free space in RAM to let OS use page cache and optimize writes. Total 
size of all memory policies shouldn't exceed 70% of your RAM.
5) Make sure that OS doesn't utilize swap. If you use Unix, best option is set 
vm.swappiness to 0.
6) Try to find out page size of your SSD. Ideally, page size of Ignite 
shouldn't be less than SSD page size. Possible approaches:
Find it in device specification (some manufacturers don't reveal it)
Try running SSD benchmarks
If you are not sure, just set page size to 4K. As various benchmarks use 4K 
pages, manufacturers have to adapt drives for 4K random write workload. 
Whitepaper from Intel showing that 4K pages are enough: 
https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/ssd-server-storage-applications-paper.pdf
Check your OS page cache size. Page size of Ignite shouldn't be less than OS 
page size. How to check OS cache page size in Unix: 
https://unix.stackexchange.com/questions/128213/how-is-page-size-determined-in-virtual-address-space


Best Regards,
Ivan Rakov

On 01.09.2017 21:08, Denis Magda wrote:

Igniters,

I see a lot of complains regarding the performance of the subj on the user 
list. At the same time, I do believe that in most scenarios it’s a lack of 
knowledge that we keep in secret.

It's time to document Durable Memory and its Native Persistence tuning parameters. 
Let's start doing this for Linux based deployments first. Here is what we have for 
now (which is almost nothing): 
https://apacheignite.readme.io/docs/durable-memory-tuning 


Ideally, at some point we have to come up with doc like this: 
https://access.redhat.com/sites/default/files/attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf

Please share your expertise in a form of settings that have to be put on the 
paper. We put them in JIRA and document afterwords:
https://issues.apache.org/jira/browse/IGNITE-6246 


—
Denis

Re: Durable Memory and Ignite Persistence Tuning

[Denis Magda]

Ivan,

Documented:
https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning 


However, I’m a bit confused with you 6. point regarding the page size 
calculation. Should Ignite page size be always less than the SSD’s page size 
and OS page cache size? Or it can be equal? For instance if the OS page cache 
size is 4 KB what should be Ignite’s page size? That’s the section about this: 
https://apacheignite.readme.io/v2.1/docs/durable-memory-tuning#section-page-size
 


—
Denis

> On Sep 13, 2017, at 2:29 AM, Ivan Rakov  wrote:
> 
> Folks,
> 
> We had some experience of benchmarking Ignite with persistent store on SSD. I 
> think we can share some helpful advice. None of them require changing 
> configuration of Ignite or persistent store.
> 
> *Tuning advice for users*
> 
> 1) Be prepared for LFS performance decrease after several hours of intensive 
> load. Unfortunately, that's how SSD drives work: 
> http://codecapsule.com/2014/02/12/coding-for-ssds-part-2-architecture-of-an-ssd-and-benchmarking/
> Consider buying fast production-level SSD drives.
> 2) Consider using separate drives for LFS files and WAL. Ignite actively 
> performs writes to both LFS and WAL under intensive load, and having two 
> devices will double your throughput limit.
> 3) Over-provision your SSD. Performance of random writes on 50% filled disk 
> is much better than on 90% filled. SSD Over-Provisioning And Its Benefits: 
> http://www.seagate.com/ru/ru/tech-insights/ssd-over-provisioning-benefits-master-ti/
> 4) Leave free space in RAM to let OS use page cache and optimize writes. 
> Total size of all memory policies shouldn't exceed 70% of your RAM.
> 5) Make sure that OS doesn't utilize swap. If you use Unix, best option is 
> set vm.swappiness to 0.
> 6) Try to find out page size of your SSD. Ideally, page size of Ignite 
> shouldn't be less than SSD page size. Possible approaches:
> Find it in device specification (some manufacturers don't reveal it)
> Try running SSD benchmarks
> If you are not sure, just set page size to 4K. As various benchmarks use 4K 
> pages, manufacturers have to adapt drives for 4K random write workload. 
> Whitepaper from Intel showing that 4K pages are enough: 
> https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/ssd-server-storage-applications-paper.pdf
> Check your OS page cache size. Page size of Ignite shouldn't be less than OS 
> page size. How to check OS cache page size in Unix: 
> https://unix.stackexchange.com/questions/128213/how-is-page-size-determined-in-virtual-address-space
> 
> 
> Best Regards,
> Ivan Rakov
> 
> On 01.09.2017 21:08, Denis Magda wrote:
>> Igniters,
>> 
>> I see a lot of complains regarding the performance of the subj on the user 
>> list. At the same time, I do believe that in most scenarios it’s a lack of 
>> knowledge that we keep in secret.
>> 
>> It's time to document Durable Memory and its Native Persistence tuning 
>> parameters. Let's start doing this for Linux based deployments first. Here 
>> is what we have for now (which is almost nothing): 
>> https://apacheignite.readme.io/docs/durable-memory-tuning 
>> 
>> 
>> Ideally, at some point we have to come up with doc like this: 
>> https://access.redhat.com/sites/default/files/attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf
>> 
>> Please share your expertise in a form of settings that have to be put on the 
>> paper. We put them in JIRA and document afterwords:
>> https://issues.apache.org/jira/browse/IGNITE-6246 
>> 
>> 
>> —
>> Denis
> 

Re: Durable Memory and Ignite Persistence Tuning

[Denis Magda]

That’s exactly the discussion around that documentation. Feel free to add these 
useful points there or wait while I’ll do this later.

—
Denis

> On Sep 13, 2017, at 12:27 PM, Dmitriy Setrakyan  wrote:
> 
> Completely agree with Nikita. Why not add this information here?
> 
> https://apacheignite.readme.io/docs/durable-memory-tuning
> 
> D.
> 
> On Wed, Sep 13, 2017 at 2:54 AM, Nikita Ivanov  wrote:
> 
>> Excellent work on this... This should be expanded and be prominently placed
>> in our docs/tutorials/javadocs/etc.
>> 
>> --
>> Nikita Ivanov
>> 
>> 
>> On Wed, Sep 13, 2017 at 12:29 PM, Ivan Rakov 
>> wrote:
>> 
>>> Folks,
>>> 
>>> We had some experience of benchmarking Ignite with persistent store on
>>> SSD. I think we can share some helpful advice. None of them require
>>> changing configuration of Ignite or persistent store.
>>> 
>>> *Tuning advice for users*
>>> 
>>> 1) Be prepared for LFS performance decrease after several hours of
>>> intensive load. Unfortunately, that's how SSD drives work:
>>> http://codecapsule.com/2014/02/12/coding-for-ssds-part-2-arc
>>> hitecture-of-an-ssd-and-benchmarking/
>>> Consider buying fast production-level SSD drives.
>>> 2) Consider using separate drives for LFS files and WAL. Ignite actively
>>> performs writes to both LFS and WAL under intensive load, and having two
>>> devices will double your throughput limit.
>>> 3) Over-provision your SSD. Performance of random writes on 50% filled
>>> disk is much better than on 90% filled. SSD Over-Provisioning And Its
>>> Benefits: http://www.seagate.com/ru/ru/tech-insights/ssd-over-provisio
>>> ning-benefits-master-ti/
>>> 4) Leave free space in RAM to let OS use page cache and optimize writes.
>>> Total size of all memory policies shouldn't exceed 70% of your RAM.
>>> 5) Make sure that OS doesn't utilize swap. If you use Unix, best option
>> is
>>> set vm.swappiness to 0.
>>> 6) Try to find out page size of your SSD. Ideally, page size of Ignite
>>> shouldn't be less than SSD page size. Possible approaches:
>>> Find it in device specification (some manufacturers don't reveal it)
>>> Try running SSD benchmarks
>>> If you are not sure, just set page size to 4K. As various benchmarks use
>>> 4K pages, manufacturers have to adapt drives for 4K random write
>> workload.
>>> Whitepaper from Intel showing that 4K pages are enough:
>>> https://www.intel.com/content/dam/www/public/us/en/documents
>>> /white-papers/ssd-server-storage-applications-paper.pdf
>>> Check your OS page cache size. Page size of Ignite shouldn't be less than
>>> OS page size. How to check OS cache page size in Unix:
>>> https://unix.stackexchange.com/questions/128213/how-is-page-
>>> size-determined-in-virtual-address-space
>>> 
>>> 
>>> Best Regards,
>>> Ivan Rakov
>>> 
>>> On 01.09.2017 21:08, Denis Magda wrote:
>>> 
 Igniters,
 
 I see a lot of complains regarding the performance of the subj on the
 user list. At the same time, I do believe that in most scenarios it’s a
 lack of knowledge that we keep in secret.
 
 It's time to document Durable Memory and its Native Persistence tuning
 parameters. Let's start doing this for Linux based deployments first.
>> Here
 is what we have for now (which is almost nothing):
 https://apacheignite.readme.io/docs/durable-memory-tuning <
 https://apacheignite.readme.io/docs/durable-memory-tuning>
 
 Ideally, at some point we have to come up with doc like this:
 https://access.redhat.com/sites/default/files/attachments/
 deploying-oracle-12c-on-rhel6_1.2_1.pdf
 
 Please share your expertise in a form of settings that have to be put on
 the paper. We put them in JIRA and document afterwords:
 https://issues.apache.org/jira/browse/IGNITE-6246 <
 https://issues.apache.org/jira/browse/IGNITE-6246>
 
 —
 Denis
 
>>> 
>>> 
>> 

Re: Durable Memory and Ignite Persistence Tuning

[Dmitriy Setrakyan]

Completely agree with Nikita. Why not add this information here?

https://apacheignite.readme.io/docs/durable-memory-tuning

D.

On Wed, Sep 13, 2017 at 2:54 AM, Nikita Ivanov  wrote:

> Excellent work on this... This should be expanded and be prominently placed
> in our docs/tutorials/javadocs/etc.
>
> --
> Nikita Ivanov
>
>
> On Wed, Sep 13, 2017 at 12:29 PM, Ivan Rakov 
> wrote:
>
> > Folks,
> >
> > We had some experience of benchmarking Ignite with persistent store on
> > SSD. I think we can share some helpful advice. None of them require
> > changing configuration of Ignite or persistent store.
> >
> > *Tuning advice for users*
> >
> > 1) Be prepared for LFS performance decrease after several hours of
> > intensive load. Unfortunately, that's how SSD drives work:
> > http://codecapsule.com/2014/02/12/coding-for-ssds-part-2-arc
> > hitecture-of-an-ssd-and-benchmarking/
> > Consider buying fast production-level SSD drives.
> > 2) Consider using separate drives for LFS files and WAL. Ignite actively
> > performs writes to both LFS and WAL under intensive load, and having two
> > devices will double your throughput limit.
> > 3) Over-provision your SSD. Performance of random writes on 50% filled
> > disk is much better than on 90% filled. SSD Over-Provisioning And Its
> > Benefits: http://www.seagate.com/ru/ru/tech-insights/ssd-over-provisio
> > ning-benefits-master-ti/
> > 4) Leave free space in RAM to let OS use page cache and optimize writes.
> > Total size of all memory policies shouldn't exceed 70% of your RAM.
> > 5) Make sure that OS doesn't utilize swap. If you use Unix, best option
> is
> > set vm.swappiness to 0.
> > 6) Try to find out page size of your SSD. Ideally, page size of Ignite
> > shouldn't be less than SSD page size. Possible approaches:
> > Find it in device specification (some manufacturers don't reveal it)
> > Try running SSD benchmarks
> > If you are not sure, just set page size to 4K. As various benchmarks use
> > 4K pages, manufacturers have to adapt drives for 4K random write
> workload.
> > Whitepaper from Intel showing that 4K pages are enough:
> > https://www.intel.com/content/dam/www/public/us/en/documents
> > /white-papers/ssd-server-storage-applications-paper.pdf
> > Check your OS page cache size. Page size of Ignite shouldn't be less than
> > OS page size. How to check OS cache page size in Unix:
> > https://unix.stackexchange.com/questions/128213/how-is-page-
> > size-determined-in-virtual-address-space
> >
> >
> > Best Regards,
> > Ivan Rakov
> >
> > On 01.09.2017 21:08, Denis Magda wrote:
> >
> >> Igniters,
> >>
> >> I see a lot of complains regarding the performance of the subj on the
> >> user list. At the same time, I do believe that in most scenarios it’s a
> >> lack of knowledge that we keep in secret.
> >>
> >> It's time to document Durable Memory and its Native Persistence tuning
> >> parameters. Let's start doing this for Linux based deployments first.
> Here
> >> is what we have for now (which is almost nothing):
> >> https://apacheignite.readme.io/docs/durable-memory-tuning <
> >> https://apacheignite.readme.io/docs/durable-memory-tuning>
> >>
> >> Ideally, at some point we have to come up with doc like this:
> >> https://access.redhat.com/sites/default/files/attachments/
> >> deploying-oracle-12c-on-rhel6_1.2_1.pdf
> >>
> >> Please share your expertise in a form of settings that have to be put on
> >> the paper. We put them in JIRA and document afterwords:
> >> https://issues.apache.org/jira/browse/IGNITE-6246 <
> >> https://issues.apache.org/jira/browse/IGNITE-6246>
> >>
> >> —
> >> Denis
> >>
> >
> >
>

Re: Durable Memory and Ignite Persistence Tuning

[Nikita Ivanov]

Excellent work on this... This should be expanded and be prominently placed
in our docs/tutorials/javadocs/etc.

--
Nikita Ivanov


On Wed, Sep 13, 2017 at 12:29 PM, Ivan Rakov  wrote:

> Folks,
>
> We had some experience of benchmarking Ignite with persistent store on
> SSD. I think we can share some helpful advice. None of them require
> changing configuration of Ignite or persistent store.
>
> *Tuning advice for users*
>
> 1) Be prepared for LFS performance decrease after several hours of
> intensive load. Unfortunately, that's how SSD drives work:
> http://codecapsule.com/2014/02/12/coding-for-ssds-part-2-arc
> hitecture-of-an-ssd-and-benchmarking/
> Consider buying fast production-level SSD drives.
> 2) Consider using separate drives for LFS files and WAL. Ignite actively
> performs writes to both LFS and WAL under intensive load, and having two
> devices will double your throughput limit.
> 3) Over-provision your SSD. Performance of random writes on 50% filled
> disk is much better than on 90% filled. SSD Over-Provisioning And Its
> Benefits: http://www.seagate.com/ru/ru/tech-insights/ssd-over-provisio
> ning-benefits-master-ti/
> 4) Leave free space in RAM to let OS use page cache and optimize writes.
> Total size of all memory policies shouldn't exceed 70% of your RAM.
> 5) Make sure that OS doesn't utilize swap. If you use Unix, best option is
> set vm.swappiness to 0.
> 6) Try to find out page size of your SSD. Ideally, page size of Ignite
> shouldn't be less than SSD page size. Possible approaches:
> Find it in device specification (some manufacturers don't reveal it)
> Try running SSD benchmarks
> If you are not sure, just set page size to 4K. As various benchmarks use
> 4K pages, manufacturers have to adapt drives for 4K random write workload.
> Whitepaper from Intel showing that 4K pages are enough:
> https://www.intel.com/content/dam/www/public/us/en/documents
> /white-papers/ssd-server-storage-applications-paper.pdf
> Check your OS page cache size. Page size of Ignite shouldn't be less than
> OS page size. How to check OS cache page size in Unix:
> https://unix.stackexchange.com/questions/128213/how-is-page-
> size-determined-in-virtual-address-space
>
>
> Best Regards,
> Ivan Rakov
>
> On 01.09.2017 21:08, Denis Magda wrote:
>
>> Igniters,
>>
>> I see a lot of complains regarding the performance of the subj on the
>> user list. At the same time, I do believe that in most scenarios it’s a
>> lack of knowledge that we keep in secret.
>>
>> It's time to document Durable Memory and its Native Persistence tuning
>> parameters. Let's start doing this for Linux based deployments first. Here
>> is what we have for now (which is almost nothing):
>> https://apacheignite.readme.io/docs/durable-memory-tuning <
>> https://apacheignite.readme.io/docs/durable-memory-tuning>
>>
>> Ideally, at some point we have to come up with doc like this:
>> https://access.redhat.com/sites/default/files/attachments/
>> deploying-oracle-12c-on-rhel6_1.2_1.pdf
>>
>> Please share your expertise in a form of settings that have to be put on
>> the paper. We put them in JIRA and document afterwords:
>> https://issues.apache.org/jira/browse/IGNITE-6246 <
>> https://issues.apache.org/jira/browse/IGNITE-6246>
>>
>> —
>> Denis
>>
>
>

Re: Durable Memory and Ignite Persistence Tuning

[Ivan Rakov]

> None of them require changing configuration of Ignite or persistent store
Disclaimer: this will be true since 2.3.

https://issues.apache.org/jira/browse/IGNITE-6182 - memory policy 
default max size set to 20% of RAM - since 2.2
https://issues.apache.org/jira/browse/IGNITE-5884 - default pageSize set 
to 4K - since 2.3


Best Regards,
Ivan Rakov


On 13.09.2017 12:29, Ivan Rakov wrote:

Folks,

We had some experience of benchmarking Ignite with persistent store on 
SSD. I think we can share some helpful advice. None of them require 
changing configuration of Ignite or persistent store.


*Tuning advice for users*

1) Be prepared for LFS performance decrease after several hours of 
intensive load. Unfortunately, that's how SSD drives work: 
http://codecapsule.com/2014/02/12/coding-for-ssds-part-2-architecture-of-an-ssd-and-benchmarking/

Consider buying fast production-level SSD drives.
2) Consider using separate drives for LFS files and WAL. Ignite 
actively performs writes to both LFS and WAL under intensive load, and 
having two devices will double your throughput limit.
3) Over-provision your SSD. Performance of random writes on 50% filled 
disk is much better than on 90% filled. SSD Over-Provisioning And Its 
Benefits: 
http://www.seagate.com/ru/ru/tech-insights/ssd-over-provisioning-benefits-master-ti/
4) Leave free space in RAM to let OS use page cache and optimize 
writes. Total size of all memory policies shouldn't exceed 70% of your 
RAM.
5) Make sure that OS doesn't utilize swap. If you use Unix, best 
option is set vm.swappiness to 0.
6) Try to find out page size of your SSD. Ideally, page size of Ignite 
shouldn't be less than SSD page size. Possible approaches:

Find it in device specification (some manufacturers don't reveal it)
Try running SSD benchmarks
If you are not sure, just set page size to 4K. As various benchmarks 
use 4K pages, manufacturers have to adapt drives for 4K random write 
workload. Whitepaper from Intel showing that 4K pages are enough: 
https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/ssd-server-storage-applications-paper.pdf
Check your OS page cache size. Page size of Ignite shouldn't be less 
than OS page size. How to check OS cache page size in Unix: 
https://unix.stackexchange.com/questions/128213/how-is-page-size-determined-in-virtual-address-space



Best Regards,
Ivan Rakov

On 01.09.2017 21:08, Denis Magda wrote:

Igniters,

I see a lot of complains regarding the performance of the subj on the 
user list. At the same time, I do believe that in most scenarios it’s 
a lack of knowledge that we keep in secret.


It's time to document Durable Memory and its Native Persistence 
tuning parameters. Let's start doing this for Linux based deployments 
first. Here is what we have for now (which is almost nothing): 
https://apacheignite.readme.io/docs/durable-memory-tuning 



Ideally, at some point we have to come up with doc like this: 
https://access.redhat.com/sites/default/files/attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf


Please share your expertise in a form of settings that have to be put 
on the paper. We put them in JIRA and document afterwords:
https://issues.apache.org/jira/browse/IGNITE-6246 



—
Denis

Re: Durable Memory and Ignite Persistence Tuning

[Ivan Rakov]

Folks,

We had some experience of benchmarking Ignite with persistent store on 
SSD. I think we can share some helpful advice. None of them require 
changing configuration of Ignite or persistent store.


*Tuning advice for users*

1) Be prepared for LFS performance decrease after several hours of 
intensive load. Unfortunately, that's how SSD drives work: 
http://codecapsule.com/2014/02/12/coding-for-ssds-part-2-architecture-of-an-ssd-and-benchmarking/

Consider buying fast production-level SSD drives.
2) Consider using separate drives for LFS files and WAL. Ignite actively 
performs writes to both LFS and WAL under intensive load, and having two 
devices will double your throughput limit.
3) Over-provision your SSD. Performance of random writes on 50% filled 
disk is much better than on 90% filled. SSD Over-Provisioning And Its 
Benefits: 
http://www.seagate.com/ru/ru/tech-insights/ssd-over-provisioning-benefits-master-ti/
4) Leave free space in RAM to let OS use page cache and optimize writes. 
Total size of all memory policies shouldn't exceed 70% of your RAM.
5) Make sure that OS doesn't utilize swap. If you use Unix, best option 
is set vm.swappiness to 0.
6) Try to find out page size of your SSD. Ideally, page size of Ignite 
shouldn't be less than SSD page size. Possible approaches:

Find it in device specification (some manufacturers don't reveal it)
Try running SSD benchmarks
If you are not sure, just set page size to 4K. As various benchmarks use 
4K pages, manufacturers have to adapt drives for 4K random write 
workload. Whitepaper from Intel showing that 4K pages are enough: 
https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/ssd-server-storage-applications-paper.pdf
Check your OS page cache size. Page size of Ignite shouldn't be less 
than OS page size. How to check OS cache page size in Unix: 
https://unix.stackexchange.com/questions/128213/how-is-page-size-determined-in-virtual-address-space



Best Regards,
Ivan Rakov

On 01.09.2017 21:08, Denis Magda wrote:

Igniters,

I see a lot of complains regarding the performance of the subj on the user 
list. At the same time, I do believe that in most scenarios it’s a lack of 
knowledge that we keep in secret.

It's time to document Durable Memory and its Native Persistence tuning parameters. 
Let's start doing this for Linux based deployments first. Here is what we have for 
now (which is almost nothing): 
https://apacheignite.readme.io/docs/durable-memory-tuning 


Ideally, at some point we have to come up with doc like this: 
https://access.redhat.com/sites/default/files/attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf

Please share your expertise in a form of settings that have to be put on the 
paper. We put them in JIRA and document afterwords:
https://issues.apache.org/jira/browse/IGNITE-6246 


—
Denis

Re: Durable Memory and Ignite Persistence Tuning

[dsetrakyan]

Cos, great point!

I think the Ignite community should start load testing with default settings, 
without any config changes. This should open a lot of holes.

⁣D.​

On Sep 1, 2017, 9:57 PM, at 9:57 PM, Konstantin Boudnik  wrote:
>Just to spice it up: in my experience, having a few hundred parameters
>one can
>tweak (I am making up the number here) is a tough UX call. In JVM, we
>had a
>team that worked on heuristics that would auto-tune a bunch of things
>during
>the VM startup. Hence, providing the best user experience in most
>cases.
>
>Do you think something like this is feasible in case of persistence
>functionality? Documenting it first is a great first step, but perhaps
>wrapping this knowledge into some soft of code, would be even better.
>
>I do have an anecdotal evidence where an experienced Ignite developer
>tried to implement a message processing system with Ignite 2.0. After a
>week
>of getting nowhere performance wise, he dropped it and implemented
>something
>custom with Java and nothing else. Take it or leave it: that's why I
>called
>this "anecdotal" in the first place.
>
>Speaking strictly for myself: when I come across blog posts about
>tuning of
>Apache Kudu or Apache Impala the skin on the back of head starts
>crawling. I
>imagine 95% of the potential users of these applications would turn
>away right
>at that point. If the system doesn't work well enough out of the box -
>screw
>it, there's 355 other alternatives available in the FOSS market.
>
>Thoughts?
>  Cos
>
>On Fri, Sep 01, 2017 at 11:08AM, Denis Magda wrote:
>> Igniters,
>>
>> I see a lot of complains regarding the performance of the subj on the
>user
>> list. At the same time, I do believe that in most scenarios it’s a
>lack of
>> knowledge that we keep in secret.
>>
>> It's time to document Durable Memory and its Native Persistence
>tuning
>> parameters. Let's start doing this for Linux based deployments first.
>Here
>> is what we have for now (which is almost nothing):
>> https://apacheignite.readme.io/docs/durable-memory-tuning
>> 
>>
>> Ideally, at some point we have to come up with doc like this:
>>
>https://access.redhat.com/sites/default/files/attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf
>>
>> Please share your expertise in a form of settings that have to be put
>on the
>> paper. We put them in JIRA and document afterwords:
>> https://issues.apache.org/jira/browse/IGNITE-6246
>> 
>>
>> —
>> Denis

Re: Durable Memory and Ignite Persistence Tuning

[Nikita Ivanov]

My points was in support of auto-tune for Ignite's internals. As far as
environment, sure, we need to have a doc.
--
Nikita Ivanov


On Fri, Sep 1, 2017 at 1:23 PM, Denis Magda  wrote:

> It doesn’t matter how many configuration parameters your platform,
> database or operating system has - the performance tuning usually takes
> place in business deployments. The performance optimizations might happen
> behind the scene by heuristic algorithms or basic checks or be explained in
> performance guides or both.
>
> This discussion is about this exact guide that is vital for DevOps and
> Admins. Make it first and you’ll reveal the spots for auto-tuning.
> Otherwise, we can spend months keeping a blind eye on this problem waiting
> for fully automated heaven and cleanest API in the world but nothing will
> happen at all.
>
> —
> Denis
>
> > On Sep 1, 2017, at 12:49 PM, Nikita Ivanov  wrote:
> >
> > 200% agree.
> >
> > UX is major problem for Ignite (especially so for 2.0 with all major
> > redev). Removing (!) configuration properties should be THE GOAL, not
> > adding new one or documenting them (which is a crutch to begin with).
> >
> > When I see a product with dozens config properties or more I desperately
> > look for ways not to use it...
> >
> > My 2 cents,
> > --
> > Nikita Ivanov
> >
> >
> > On Fri, Sep 1, 2017 at 12:02 PM, Konstantin Boudnik 
> wrote:
> >
> >> Just to spice it up: in my experience, having a few hundred parameters
> one
> >> can
> >> tweak (I am making up the number here) is a tough UX call. In JVM, we
> had a
> >> team that worked on heuristics that would auto-tune a bunch of things
> >> during
> >> the VM startup. Hence, providing the best user experience in most cases.
> >>
> >> Do you think something like this is feasible in case of persistence
> >> functionality? Documenting it first is a great first step, but perhaps
> >> wrapping this knowledge into some soft of code, would be even better.
> >>
> >> I do have an anecdotal evidence where an experienced Ignite developer
> >> tried to implement a message processing system with Ignite 2.0. After a
> >> week
> >> of getting nowhere performance wise, he dropped it and implemented
> >> something
> >> custom with Java and nothing else. Take it or leave it: that's why I
> called
> >> this "anecdotal" in the first place.
> >>
> >> Speaking strictly for myself: when I come across blog posts about
> tuning of
> >> Apache Kudu or Apache Impala the skin on the back of head starts
> crawling.
> >> I
> >> imagine 95% of the potential users of these applications would turn away
> >> right
> >> at that point. If the system doesn't work well enough out of the box -
> >> screw
> >> it, there's 355 other alternatives available in the FOSS market.
> >>
> >> Thoughts?
> >>  Cos
> >>
> >> On Fri, Sep 01, 2017 at 11:08AM, Denis Magda wrote:
> >>> Igniters,
> >>>
> >>> I see a lot of complains regarding the performance of the subj on the
> >> user
> >>> list. At the same time, I do believe that in most scenarios it’s a lack
> >> of
> >>> knowledge that we keep in secret.
> >>>
> >>> It's time to document Durable Memory and its Native Persistence tuning
> >>> parameters. Let's start doing this for Linux based deployments first.
> >> Here
> >>> is what we have for now (which is almost nothing):
> >>> https://apacheignite.readme.io/docs/durable-memory-tuning
> >>> 
> >>>
> >>> Ideally, at some point we have to come up with doc like this:
> >>> https://access.redhat.com/sites/default/files/
> >> attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf
> >>>
> >>> Please share your expertise in a form of settings that have to be put
> on
> >> the
> >>> paper. We put them in JIRA and document afterwords:
> >>> https://issues.apache.org/jira/browse/IGNITE-6246
> >>> 
> >>>
> >>> —
> >>> Denis
> >>
>
>

Re: Durable Memory and Ignite Persistence Tuning

[Denis Magda]

It doesn’t matter how many configuration parameters your platform, database or 
operating system has - the performance tuning usually takes place in business 
deployments. The performance optimizations might happen behind the scene by 
heuristic algorithms or basic checks or be explained in performance guides or 
both.

This discussion is about this exact guide that is vital for DevOps and Admins. 
Make it first and you’ll reveal the spots for auto-tuning. Otherwise, we can 
spend months keeping a blind eye on this problem waiting for fully automated 
heaven and cleanest API in the world but nothing will happen at all.

—
Denis  

> On Sep 1, 2017, at 12:49 PM, Nikita Ivanov  wrote:
> 
> 200% agree.
> 
> UX is major problem for Ignite (especially so for 2.0 with all major
> redev). Removing (!) configuration properties should be THE GOAL, not
> adding new one or documenting them (which is a crutch to begin with).
> 
> When I see a product with dozens config properties or more I desperately
> look for ways not to use it...
> 
> My 2 cents,
> --
> Nikita Ivanov
> 
> 
> On Fri, Sep 1, 2017 at 12:02 PM, Konstantin Boudnik  wrote:
> 
>> Just to spice it up: in my experience, having a few hundred parameters one
>> can
>> tweak (I am making up the number here) is a tough UX call. In JVM, we had a
>> team that worked on heuristics that would auto-tune a bunch of things
>> during
>> the VM startup. Hence, providing the best user experience in most cases.
>> 
>> Do you think something like this is feasible in case of persistence
>> functionality? Documenting it first is a great first step, but perhaps
>> wrapping this knowledge into some soft of code, would be even better.
>> 
>> I do have an anecdotal evidence where an experienced Ignite developer
>> tried to implement a message processing system with Ignite 2.0. After a
>> week
>> of getting nowhere performance wise, he dropped it and implemented
>> something
>> custom with Java and nothing else. Take it or leave it: that's why I called
>> this "anecdotal" in the first place.
>> 
>> Speaking strictly for myself: when I come across blog posts about tuning of
>> Apache Kudu or Apache Impala the skin on the back of head starts crawling.
>> I
>> imagine 95% of the potential users of these applications would turn away
>> right
>> at that point. If the system doesn't work well enough out of the box -
>> screw
>> it, there's 355 other alternatives available in the FOSS market.
>> 
>> Thoughts?
>>  Cos
>> 
>> On Fri, Sep 01, 2017 at 11:08AM, Denis Magda wrote:
>>> Igniters,
>>> 
>>> I see a lot of complains regarding the performance of the subj on the
>> user
>>> list. At the same time, I do believe that in most scenarios it’s a lack
>> of
>>> knowledge that we keep in secret.
>>> 
>>> It's time to document Durable Memory and its Native Persistence tuning
>>> parameters. Let's start doing this for Linux based deployments first.
>> Here
>>> is what we have for now (which is almost nothing):
>>> https://apacheignite.readme.io/docs/durable-memory-tuning
>>> 
>>> 
>>> Ideally, at some point we have to come up with doc like this:
>>> https://access.redhat.com/sites/default/files/
>> attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf
>>> 
>>> Please share your expertise in a form of settings that have to be put on
>> the
>>> paper. We put them in JIRA and document afterwords:
>>> https://issues.apache.org/jira/browse/IGNITE-6246
>>> 
>>> 
>>> —
>>> Denis
>> 

Re: Durable Memory and Ignite Persistence Tuning

[Nikita Ivanov]

200% agree.

UX is major problem for Ignite (especially so for 2.0 with all major
redev). Removing (!) configuration properties should be THE GOAL, not
adding new one or documenting them (which is a crutch to begin with).

When I see a product with dozens config properties or more I desperately
look for ways not to use it...

My 2 cents,
--
Nikita Ivanov


On Fri, Sep 1, 2017 at 12:02 PM, Konstantin Boudnik  wrote:

> Just to spice it up: in my experience, having a few hundred parameters one
> can
> tweak (I am making up the number here) is a tough UX call. In JVM, we had a
> team that worked on heuristics that would auto-tune a bunch of things
> during
> the VM startup. Hence, providing the best user experience in most cases.
>
> Do you think something like this is feasible in case of persistence
> functionality? Documenting it first is a great first step, but perhaps
> wrapping this knowledge into some soft of code, would be even better.
>
> I do have an anecdotal evidence where an experienced Ignite developer
> tried to implement a message processing system with Ignite 2.0. After a
> week
> of getting nowhere performance wise, he dropped it and implemented
> something
> custom with Java and nothing else. Take it or leave it: that's why I called
> this "anecdotal" in the first place.
>
> Speaking strictly for myself: when I come across blog posts about tuning of
> Apache Kudu or Apache Impala the skin on the back of head starts crawling.
> I
> imagine 95% of the potential users of these applications would turn away
> right
> at that point. If the system doesn't work well enough out of the box -
> screw
> it, there's 355 other alternatives available in the FOSS market.
>
> Thoughts?
>   Cos
>
> On Fri, Sep 01, 2017 at 11:08AM, Denis Magda wrote:
> > Igniters,
> >
> > I see a lot of complains regarding the performance of the subj on the
> user
> > list. At the same time, I do believe that in most scenarios it’s a lack
> of
> > knowledge that we keep in secret.
> >
> > It's time to document Durable Memory and its Native Persistence tuning
> > parameters. Let's start doing this for Linux based deployments first.
> Here
> > is what we have for now (which is almost nothing):
> > https://apacheignite.readme.io/docs/durable-memory-tuning
> > 
> >
> > Ideally, at some point we have to come up with doc like this:
> > https://access.redhat.com/sites/default/files/
> attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf
> >
> > Please share your expertise in a form of settings that have to be put on
> the
> > paper. We put them in JIRA and document afterwords:
> > https://issues.apache.org/jira/browse/IGNITE-6246
> > 
> >
> > —
> > Denis
>

Re: Durable Memory and Ignite Persistence Tuning

[Denis Magda]

Cos,

Sure, as I see from the @dev list a lot of efforts are put to the auto-tune 
way. We need to keep to this way whenever it’s possible.

However, it might be the case that we can’t auto-tune some parameters (Linux 
kernel behavior) or we know that setting A will be auto-tuned in release 2.x 
but for now we should explain how to set it manually.

Even more, we can review that page over the time and decide that now we’re 
ready to auto-tune parameter B.

—
Denis

> On Sep 1, 2017, at 12:02 PM, Konstantin Boudnik  wrote:
> 
> Just to spice it up: in my experience, having a few hundred parameters one can
> tweak (I am making up the number here) is a tough UX call. In JVM, we had a
> team that worked on heuristics that would auto-tune a bunch of things during
> the VM startup. Hence, providing the best user experience in most cases.
> 
> Do you think something like this is feasible in case of persistence
> functionality? Documenting it first is a great first step, but perhaps
> wrapping this knowledge into some soft of code, would be even better.
> 
> I do have an anecdotal evidence where an experienced Ignite developer
> tried to implement a message processing system with Ignite 2.0. After a week
> of getting nowhere performance wise, he dropped it and implemented something
> custom with Java and nothing else. Take it or leave it: that's why I called
> this "anecdotal" in the first place.
> 
> Speaking strictly for myself: when I come across blog posts about tuning of
> Apache Kudu or Apache Impala the skin on the back of head starts crawling. I
> imagine 95% of the potential users of these applications would turn away right
> at that point. If the system doesn't work well enough out of the box - screw
> it, there's 355 other alternatives available in the FOSS market.
> 
> Thoughts?
>  Cos
> 
> On Fri, Sep 01, 2017 at 11:08AM, Denis Magda wrote:
>> Igniters,
>> 
>> I see a lot of complains regarding the performance of the subj on the user
>> list. At the same time, I do believe that in most scenarios it’s a lack of
>> knowledge that we keep in secret.
>> 
>> It's time to document Durable Memory and its Native Persistence tuning
>> parameters. Let's start doing this for Linux based deployments first. Here
>> is what we have for now (which is almost nothing):
>> https://apacheignite.readme.io/docs/durable-memory-tuning
>> 
>> 
>> Ideally, at some point we have to come up with doc like this:
>> https://access.redhat.com/sites/default/files/attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf
>> 
>> Please share your expertise in a form of settings that have to be put on the
>> paper. We put them in JIRA and document afterwords:
>> https://issues.apache.org/jira/browse/IGNITE-6246
>> 
>> 
>> —
>> Denis

Re: Durable Memory and Ignite Persistence Tuning

[Konstantin Boudnik]

Just to spice it up: in my experience, having a few hundred parameters one can
tweak (I am making up the number here) is a tough UX call. In JVM, we had a
team that worked on heuristics that would auto-tune a bunch of things during
the VM startup. Hence, providing the best user experience in most cases.

Do you think something like this is feasible in case of persistence
functionality? Documenting it first is a great first step, but perhaps
wrapping this knowledge into some soft of code, would be even better.

I do have an anecdotal evidence where an experienced Ignite developer
tried to implement a message processing system with Ignite 2.0. After a week
of getting nowhere performance wise, he dropped it and implemented something
custom with Java and nothing else. Take it or leave it: that's why I called
this "anecdotal" in the first place.

Speaking strictly for myself: when I come across blog posts about tuning of
Apache Kudu or Apache Impala the skin on the back of head starts crawling. I
imagine 95% of the potential users of these applications would turn away right
at that point. If the system doesn't work well enough out of the box - screw
it, there's 355 other alternatives available in the FOSS market.

Thoughts?
  Cos

On Fri, Sep 01, 2017 at 11:08AM, Denis Magda wrote:
> Igniters,
> 
> I see a lot of complains regarding the performance of the subj on the user
> list. At the same time, I do believe that in most scenarios it’s a lack of
> knowledge that we keep in secret.
> 
> It's time to document Durable Memory and its Native Persistence tuning
> parameters. Let's start doing this for Linux based deployments first. Here
> is what we have for now (which is almost nothing):
> https://apacheignite.readme.io/docs/durable-memory-tuning
> 
> 
> Ideally, at some point we have to come up with doc like this:
> https://access.redhat.com/sites/default/files/attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf
> 
> Please share your expertise in a form of settings that have to be put on the
> paper. We put them in JIRA and document afterwords:
> https://issues.apache.org/jira/browse/IGNITE-6246
> 
> 
> —
> Denis


signature.asc
Description: Digital signature