Re: High commit size
I disabled the page file on Windows entirely, so it cannot swap out the unused part of the heap. Nevertheless, the system is sometimes very unresponsive when the shareable portion of the physical memory of the Java process is very high. I just don't get it. If I use standard IO, I/O Bytes and CPU usage for the Java process increase. Reading up to 500 MB per second and CPU usage is about 75%. If I use memory mapped IO, it all drops and the system runs smoothly. Rene 2012/9/10 Henrik Schröder > When we ran Cassandra on windows, we got better performance without memory > mapped IO. We had the same problems your are describing, what happens is > that Windows is rather aggressive about swapping out memory when all the > memory is used, and it starts swapping out "unused" parts of the heap, > which causes problems later when it has to swap it in again. We tried a > bunch of solutions to limit the mmap memory, to reduce swappiness, etc, but > nothing worked. > > Switching to standard disk access mode worked like a charm though, and > performance improved. You still have the disk cache instead of the mmap, so > you still use all the RAM of the machine, but you won't run into swapping > hell. > > So, disable mmap, benchmark it, and see if it outperforms running with > mmap. I'm pretty sure you will find that it does. > > > /Henrik > > > On Mon, Sep 10, 2012 at 4:02 PM, Rene Kochen > wrote: > >> For performance reasons I switched to memory mapped IO. Is there a way to >> make memory-mapped IO work in Windows? >> >> Thanks! >> >> >> >> 2012/9/10 Viktor Jevdokimov >> >>> Do not use mmap/auto on Windows, standard access mode only. In >>> cassandra.yaml: >>> >>> disk_access_mode: standard >>> >>> ** ** >>> >>> ** ** >>>Best regards / Pagarbiai >>> *Viktor Jevdokimov* >>> Senior Developer >>> >>> Email: viktor.jevdoki...@adform.com >>> Phone: +370 5 212 3063, Fax +370 5 261 0453 >>> J. Jasinskio 16C, LT-01112 Vilnius, Lithuania >>> Follow us on Twitter: @adforminsider<http://twitter.com/#!/adforminsider> >>> What is Adform: watch this short video <http://vimeo.com/adform/display> >>> [image: Adform News] <http://www.adform.com> >>> * * >>> *Visit us at Dmexco:* Hall: 7, Aisle: A, Stand: 030 >>> September 12-13 Cologne, Germany >>> [image: dmexco] <http://www.dmexco.de/> >>> >>> Disclaimer: The information contained in this message and attachments is >>> intended solely for the attention and use of the named addressee and may be >>> confidential. If you are not the intended recipient, you are reminded that >>> the information remains the property of the sender. You must not use, >>> disclose, distribute, copy, print or rely on this e-mail. If you have >>> received this message in error, please contact the sender immediately and >>> irrevocably delete this message and any copies. >>> >>> *From:* Rene Kochen [mailto:rene.koc...@emea.schange.com] >>> *Sent:* Monday, September 10, 2012 14:47 >>> *To:* user@cassandra.apache.org >>> *Subject:* Re: High commit size >>> >>> ** ** >>> >>> The problem is that the system just freezes and nodes are dying. The >>> system becomes very unresponsive and it always happens when the shareable >>> amount of RAM reaches the total number of bytes in the system. >>> >>> Is there something in Windows that I can tune in order to avoid this >>> behavior? I cannot easily migrate to Linux right now. >>> >>> Thanks, >>> >>> Rene >>> >>> >>> >>> 2012/9/10 Oleg Dulin >>> >>> It is memory-mapped I/O. I wouldn't worry about it. >>> >>> BTW, Windows might not be the best choice to run Cassandra on. My >>> experience running Cassandra on Windows has not been positive one. We no >>> longer support Windows as our production platform. >>> >>> Regards, >>> Oleg >>> >>> >>> On 2012-09-10 09:00:02 +, Rene Kochen said: >>> >>> Hi all, >>> >>> On my test cluster I have three Windows Server 2008 R2 machines running >>> Cassandra 1.0.11 >>> >>> If i use memory mapped IO (the default), then the nodes freeze after a >>> while. Paging is disabled. >>> >>> The private bytes are OK (8GB). That is the amount I use in the -Xms and >>> -Xmx arguments. The virtual size is big as expected because of the memory >>> mapped IO. However, the working set size (size in RAM) is 24 GB (my total >>> RAM usage). If I look with Process Explorer to the physical memory section >>> I see a very high value in the "WS Sharable" section. >>> >>> Anyone has a clue what is going om here? >>> >>> Many thanks! >>> >>> Rene >>> >>> >>> >>> >>> -- >>> Regards, >>> Oleg Dulin >>> NYC Java Big Data Engineer >>> http://www.olegdulin.com/ >>> >>> >>> >>> ** ** >>> >> >> > <><>
Re: High commit size
When we ran Cassandra on windows, we got better performance without memory mapped IO. We had the same problems your are describing, what happens is that Windows is rather aggressive about swapping out memory when all the memory is used, and it starts swapping out "unused" parts of the heap, which causes problems later when it has to swap it in again. We tried a bunch of solutions to limit the mmap memory, to reduce swappiness, etc, but nothing worked. Switching to standard disk access mode worked like a charm though, and performance improved. You still have the disk cache instead of the mmap, so you still use all the RAM of the machine, but you won't run into swapping hell. So, disable mmap, benchmark it, and see if it outperforms running with mmap. I'm pretty sure you will find that it does. /Henrik On Mon, Sep 10, 2012 at 4:02 PM, Rene Kochen wrote: > For performance reasons I switched to memory mapped IO. Is there a way to > make memory-mapped IO work in Windows? > > Thanks! > > > > 2012/9/10 Viktor Jevdokimov > >> Do not use mmap/auto on Windows, standard access mode only. In >> cassandra.yaml: >> >> disk_access_mode: standard >> >> ** ** >> >> ** ** >>Best regards / Pagarbiai >> *Viktor Jevdokimov* >> Senior Developer >> >> Email: viktor.jevdoki...@adform.com >> Phone: +370 5 212 3063, Fax +370 5 261 0453 >> J. Jasinskio 16C, LT-01112 Vilnius, Lithuania >> Follow us on Twitter: @adforminsider<http://twitter.com/#!/adforminsider> >> What is Adform: watch this short video <http://vimeo.com/adform/display> >> [image: Adform News] <http://www.adform.com> >> * * >> *Visit us at Dmexco:* Hall: 7, Aisle: A, Stand: 030 >> September 12-13 Cologne, Germany >> [image: dmexco] <http://www.dmexco.de/> >> >> Disclaimer: The information contained in this message and attachments is >> intended solely for the attention and use of the named addressee and may be >> confidential. If you are not the intended recipient, you are reminded that >> the information remains the property of the sender. You must not use, >> disclose, distribute, copy, print or rely on this e-mail. If you have >> received this message in error, please contact the sender immediately and >> irrevocably delete this message and any copies. >> >> *From:* Rene Kochen [mailto:rene.koc...@emea.schange.com] >> *Sent:* Monday, September 10, 2012 14:47 >> *To:* user@cassandra.apache.org >> *Subject:* Re: High commit size >> >> ** ** >> >> The problem is that the system just freezes and nodes are dying. The >> system becomes very unresponsive and it always happens when the shareable >> amount of RAM reaches the total number of bytes in the system. >> >> Is there something in Windows that I can tune in order to avoid this >> behavior? I cannot easily migrate to Linux right now. >> >> Thanks, >> >> Rene >> >> >> >> 2012/9/10 Oleg Dulin >> >> It is memory-mapped I/O. I wouldn't worry about it. >> >> BTW, Windows might not be the best choice to run Cassandra on. My >> experience running Cassandra on Windows has not been positive one. We no >> longer support Windows as our production platform. >> >> Regards, >> Oleg >> >> >> On 2012-09-10 09:00:02 +, Rene Kochen said: >> >> Hi all, >> >> On my test cluster I have three Windows Server 2008 R2 machines running >> Cassandra 1.0.11 >> >> If i use memory mapped IO (the default), then the nodes freeze after a >> while. Paging is disabled. >> >> The private bytes are OK (8GB). That is the amount I use in the -Xms and >> -Xmx arguments. The virtual size is big as expected because of the memory >> mapped IO. However, the working set size (size in RAM) is 24 GB (my total >> RAM usage). If I look with Process Explorer to the physical memory section >> I see a very high value in the "WS Sharable" section. >> >> Anyone has a clue what is going om here? >> >> Many thanks! >> >> Rene >> >> >> >> >> -- >> Regards, >> Oleg Dulin >> NYC Java Big Data Engineer >> http://www.olegdulin.com/ >> >> >> >> ** ** >> > > <><>
RE: High commit size
We used Cassandra on Windows for more than a year in production for RTB and other staff, that requires lowest possible latency. We used mmap before issues like yours, switched to mmap index only and finally to standard. No big difference in performance, standard was most stable. The huge difference is to run C* on Linux instead of Windows. Migration was pretty easy. Best regards / Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.com<mailto:viktor.jevdoki...@adform.com> Phone: +370 5 212 3063, Fax +370 5 261 0453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania Follow us on Twitter: @adforminsider<http://twitter.com/#!/adforminsider> What is Adform: watch this short video<http://vimeo.com/adform/display> [Adform News] <http://www.adform.com> Visit us at Dmexco: Hall: 7, Aisle: A, Stand: 030 September 12-13 Cologne, Germany [dmexco] <http://www.dmexco.de/> Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. From: Rene Kochen [mailto:rene.koc...@emea.schange.com] Sent: Monday, September 10, 2012 17:03 To: user@cassandra.apache.org Subject: Re: High commit size For performance reasons I switched to memory mapped IO. Is there a way to make memory-mapped IO work in Windows? Thanks! 2012/9/10 Viktor Jevdokimov mailto:viktor.jevdoki...@adform.com>> Do not use mmap/auto on Windows, standard access mode only. In cassandra.yaml: disk_access_mode: standard Best regards / Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.com<mailto:viktor.jevdoki...@adform.com> Phone: +370 5 212 3063, Fax +370 5 261 0453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania Follow us on Twitter: @adforminsider<http://twitter.com/#!/adforminsider> What is Adform: watch this short video<http://vimeo.com/adform/display> [Adform News]<http://www.adform.com> Visit us at Dmexco: Hall: 7, Aisle: A, Stand: 030 September 12-13 Cologne, Germany [dmexco]<http://www.dmexco.de/> Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. From: Rene Kochen [mailto:rene.koc...@emea.schange.com<mailto:rene.koc...@emea.schange.com>] Sent: Monday, September 10, 2012 14:47 To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: Re: High commit size The problem is that the system just freezes and nodes are dying. The system becomes very unresponsive and it always happens when the shareable amount of RAM reaches the total number of bytes in the system. Is there something in Windows that I can tune in order to avoid this behavior? I cannot easily migrate to Linux right now. Thanks, Rene 2012/9/10 Oleg Dulin mailto:oleg.du...@gmail.com>> It is memory-mapped I/O. I wouldn't worry about it. BTW, Windows might not be the best choice to run Cassandra on. My experience running Cassandra on Windows has not been positive one. We no longer support Windows as our production platform. Regards, Oleg On 2012-09-10 09:00:02 +, Rene Kochen said: Hi all, On my test cluster I have three Windows Server 2008 R2 machines running Cassandra 1.0.11 If i use memory mapped IO (the default), then the nodes freeze after a while. Paging is disabled. The private bytes are OK (8GB). That is the amount I use in the -Xms and -Xmx arguments. The virtual size is big as expected because of the memory mapped IO. However, the working set size (size in RAM) is 24 GB (my total RAM usage). If I look with Process Explorer to the physical memory section I see a very high value in the "WS Sharable" section. Anyone has a clue what is going om here? Many thanks! Rene -- Regards, Oleg Dulin NYC Java Big Data Engineer http://www.olegdulin.com/ <><><><>
Re: High commit size
For performance reasons I switched to memory mapped IO. Is there a way to make memory-mapped IO work in Windows? Thanks! 2012/9/10 Viktor Jevdokimov > Do not use mmap/auto on Windows, standard access mode only. In > cassandra.yaml: > > disk_access_mode: standard > > ** ** > > ** ** >Best regards / Pagarbiai > *Viktor Jevdokimov* > Senior Developer > > Email: viktor.jevdoki...@adform.com > Phone: +370 5 212 3063, Fax +370 5 261 0453 > J. Jasinskio 16C, LT-01112 Vilnius, Lithuania > Follow us on Twitter: @adforminsider <http://twitter.com/#!/adforminsider> > What is Adform: watch this short video <http://vimeo.com/adform/display> > [image: Adform News] <http://www.adform.com> > * * > *Visit us at Dmexco:* Hall: 7, Aisle: A, Stand: 030 > September 12-13 Cologne, Germany > [image: dmexco] <http://www.dmexco.de/> > > Disclaimer: The information contained in this message and attachments is > intended solely for the attention and use of the named addressee and may be > confidential. If you are not the intended recipient, you are reminded that > the information remains the property of the sender. You must not use, > disclose, distribute, copy, print or rely on this e-mail. If you have > received this message in error, please contact the sender immediately and > irrevocably delete this message and any copies. > > *From:* Rene Kochen [mailto:rene.koc...@emea.schange.com] > *Sent:* Monday, September 10, 2012 14:47 > *To:* user@cassandra.apache.org > *Subject:* Re: High commit size > > ** ** > > The problem is that the system just freezes and nodes are dying. The > system becomes very unresponsive and it always happens when the shareable > amount of RAM reaches the total number of bytes in the system. > > Is there something in Windows that I can tune in order to avoid this > behavior? I cannot easily migrate to Linux right now. > > Thanks, > > Rene > > > > 2012/9/10 Oleg Dulin > > It is memory-mapped I/O. I wouldn't worry about it. > > BTW, Windows might not be the best choice to run Cassandra on. My > experience running Cassandra on Windows has not been positive one. We no > longer support Windows as our production platform. > > Regards, > Oleg > > > On 2012-09-10 09:00:02 +, Rene Kochen said: > > Hi all, > > On my test cluster I have three Windows Server 2008 R2 machines running > Cassandra 1.0.11 > > If i use memory mapped IO (the default), then the nodes freeze after a > while. Paging is disabled. > > The private bytes are OK (8GB). That is the amount I use in the -Xms and > -Xmx arguments. The virtual size is big as expected because of the memory > mapped IO. However, the working set size (size in RAM) is 24 GB (my total > RAM usage). If I look with Process Explorer to the physical memory section > I see a very high value in the "WS Sharable" section. > > Anyone has a clue what is going om here? > > Many thanks! > > Rene > > > > > -- > Regards, > Oleg Dulin > NYC Java Big Data Engineer > http://www.olegdulin.com/ > > > > ** ** > <><>
RE: High commit size
Do not use mmap/auto on Windows, standard access mode only. In cassandra.yaml: disk_access_mode: standard Best regards / Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.com<mailto:viktor.jevdoki...@adform.com> Phone: +370 5 212 3063, Fax +370 5 261 0453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania Follow us on Twitter: @adforminsider<http://twitter.com/#!/adforminsider> What is Adform: watch this short video<http://vimeo.com/adform/display> [Adform News] <http://www.adform.com> Visit us at Dmexco: Hall: 7, Aisle: A, Stand: 030 September 12-13 Cologne, Germany [dmexco] <http://www.dmexco.de/> Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. From: Rene Kochen [mailto:rene.koc...@emea.schange.com] Sent: Monday, September 10, 2012 14:47 To: user@cassandra.apache.org Subject: Re: High commit size The problem is that the system just freezes and nodes are dying. The system becomes very unresponsive and it always happens when the shareable amount of RAM reaches the total number of bytes in the system. Is there something in Windows that I can tune in order to avoid this behavior? I cannot easily migrate to Linux right now. Thanks, Rene 2012/9/10 Oleg Dulin mailto:oleg.du...@gmail.com>> It is memory-mapped I/O. I wouldn't worry about it. BTW, Windows might not be the best choice to run Cassandra on. My experience running Cassandra on Windows has not been positive one. We no longer support Windows as our production platform. Regards, Oleg On 2012-09-10 09:00:02 +, Rene Kochen said: Hi all, On my test cluster I have three Windows Server 2008 R2 machines running Cassandra 1.0.11 If i use memory mapped IO (the default), then the nodes freeze after a while. Paging is disabled. The private bytes are OK (8GB). That is the amount I use in the -Xms and -Xmx arguments. The virtual size is big as expected because of the memory mapped IO. However, the working set size (size in RAM) is 24 GB (my total RAM usage). If I look with Process Explorer to the physical memory section I see a very high value in the "WS Sharable" section. Anyone has a clue what is going om here? Many thanks! Rene -- Regards, Oleg Dulin NYC Java Big Data Engineer http://www.olegdulin.com/ <><>
Re: High commit size
The problem is that the system just freezes and nodes are dying. The system becomes very unresponsive and it always happens when the shareable amount of RAM reaches the total number of bytes in the system. Is there something in Windows that I can tune in order to avoid this behavior? I cannot easily migrate to Linux right now. Thanks, Rene 2012/9/10 Oleg Dulin > It is memory-mapped I/O. I wouldn't worry about it. > > BTW, Windows might not be the best choice to run Cassandra on. My > experience running Cassandra on Windows has not been positive one. We no > longer support Windows as our production platform. > > Regards, > Oleg > > > On 2012-09-10 09:00:02 +, Rene Kochen said: > > Hi all, >> >> On my test cluster I have three Windows Server 2008 R2 machines running >> Cassandra 1.0.11 >> >> If i use memory mapped IO (the default), then the nodes freeze after a >> while. Paging is disabled. >> >> The private bytes are OK (8GB). That is the amount I use in the -Xms and >> -Xmx arguments. The virtual size is big as expected because of the memory >> mapped IO. However, the working set size (size in RAM) is 24 GB (my total >> RAM usage). If I look with Process Explorer to the physical memory section >> I see a very high value in the "WS Sharable" section. >> >> Anyone has a clue what is going om here? >> >> Many thanks! >> >> Rene >> >> > > > -- > Regards, > Oleg Dulin > NYC Java Big Data Engineer > http://www.olegdulin.com/ > > >
Re: High commit size
It is memory-mapped I/O. I wouldn't worry about it. BTW, Windows might not be the best choice to run Cassandra on. My experience running Cassandra on Windows has not been positive one. We no longer support Windows as our production platform. Regards, Oleg On 2012-09-10 09:00:02 +, Rene Kochen said: Hi all, On my test cluster I have three Windows Server 2008 R2 machines running Cassandra 1.0.11 If i use memory mapped IO (the default), then the nodes freeze after a while. Paging is disabled. The private bytes are OK (8GB). That is the amount I use in the -Xms and -Xmx arguments. The virtual size is big as expected because of the memory mapped IO. However, the working set size (size in RAM) is 24 GB (my total RAM usage). If I look with Process Explorer to the physical memory section I see a very high value in the "WS Sharable" section. Anyone has a clue what is going om here? Many thanks! Rene -- Regards, Oleg Dulin NYC Java Big Data Engineer http://www.olegdulin.com/
