Cassandra Disk storage capacity
Hi Team, We have a 3 node Apache cassandra 2.0.4 setup installed in our lab setup.We have set data directory to /var/lib/cassandra/data.What would be the maximum disk storage that will be used for cassandra data storage. Note : /var partition has a storage capacity of 40GB. My question is whether cassandra will the entire / directory for data storage ? If no, how to specify multiple directories for data storage ?? Best Regards Hari Krishnan Rajendhran Hadoop Admin DESS-ABIM ,Chennai BIGDATA Galaxy Tata Consultancy Services Cell:- 9677985515 Mailto: hari.rajendh...@tcs.com Website: http://www.tcs.com Experience certainty. IT Services Business Solutions Consulting =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you
Re: Cassandra Disk storage capacity
Hi Hari, C* will use your entire space - that is something one should monitor. Depending on your choose on compaction strategy your data_dir should not be filled up entirely - in the worst case compaction will need space as large as the sstables on disk, therefore 50% should be free space. The parameters used for on disk storage are commitlog_directory and data_file_directories and saved_caches_directory. The paramter data_file_directories is in plural, you can easily put more than one directory here (and you should do this instead of using RAID). Cheers, Jan Am 07.04.2014 12:56, schrieb Hari Rajendhran: Hi Team, We have a 3 node Apache cassandra 2.0.4 setup installed in our lab setup.We have set data directory to /var/lib/cassandra/data.What would be the maximum disk storage that will be used for cassandra data storage. Note : /var partition has a storage capacity of 40GB. My question is whether cassandra will the entire / directory for data storage ? If no, how to specify multiple directories for data storage ?? Best Regards Hari Krishnan Rajendhran Hadoop Admin DESS-ABIM ,Chennai BIGDATA Galaxy Tata Consultancy Services Cell:- 9677985515 Mailto: hari.rajendh...@tcs.com Website: http://www.tcs.com Experience certainty. IT Services Business Solutions Consulting =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you -- Jan Kesten, mailto:j.kes...@enercast.de Tel.: +49 561/4739664-0 FAX: -9 enercast GmbH Friedrich-Ebert-Str. 104 D-34119 Kassel HRB15471 http://www.enercast.de Online-Prognosen für erneuerbare Energien Geschäftsführung: Dipl. Ing. Thomas Landgraf, Bernd Kratz Diese E-Mail und etwaige Anhänge können vertrauliche und/oder rechtlich geschützte Informationen enthalten. Falls Sie nicht der angegebene Empfänger sind oder falls diese E-Mail irrtümlich an Sie adressiert wurde, benachrichtigen Sie uns bitte sofort durch Antwort-E-Mail und löschen Sie diese E-Mail nebst etwaigen Anlagen von Ihrem System. Ebenso dürfen Sie diese E-Mail oder ihre Anlagen nicht kopieren oder an Dritte weitergeben. Vielen Dank. This e-mail and any attachment may contain confidential and/or privileged information. If you are not the named addressee or if this transmission has been addressed to you in error, please notify us immediately by reply e-mail and then delete this e-mail and any attachment from your system. Please understand that you must not copy this e-mail or any attachment or disclose the contents to any other person. Thank you for your cooperation.
Re: Cassandra Disk storage capacity
you can specify multiple data directories in cassandra.yaml. ex: data_file_directories: - /var/lib.cass1 - /var/lib/cass2 -/some_other_mountpoint On Mon, Apr 7, 2014 at 12:10 PM, Jan Kesten j.kes...@enercast.de wrote: Hi Hari, C* will use your entire space - that is something one should monitor. Depending on your choose on compaction strategy your data_dir should not be filled up entirely - in the worst case compaction will need space as large as the sstables on disk, therefore 50% should be free space. The parameters used for on disk storage are commitlog_directory and data_file_directories and saved_caches_directory. The paramter data_file_directories is in plural, you can easily put more than one directory here (and you should do this instead of using RAID). Cheers, Jan Am 07.04.2014 12:56, schrieb Hari Rajendhran: Hi Team, We have a 3 node Apache cassandra 2.0.4 setup installed in our lab setup.We have set data directory to /var/lib/cassandra/data.What would be the maximum disk storage that will be used for cassandra data storage. Note : /var partition has a storage capacity of 40GB. My question is whether cassandra will the entire / directory for data storage ? If no, how to specify multiple directories for data storage ?? Best Regards Hari Krishnan Rajendhran Hadoop Admin DESS-ABIM ,Chennai BIGDATA Galaxy Tata Consultancy Services Cell:- 9677985515 Mailto: hari.rajendh...@tcs.com Website: http://www.tcs.com Experience certainty. IT Services Business Solutions Consulting =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you -- Jan Kesten, mailto:j.kes...@enercast.de j.kes...@enercast.de Tel.: +49 561/4739664-0 FAX: -9 enercast GmbH Friedrich-Ebert-Str. 104 D-34119 Kassel HRB15471http://www.enercast.de Online-Prognosen für erneuerbare Energien Geschäftsführung: Dipl. Ing. Thomas Landgraf, Bernd Kratz Diese E-Mail und etwaige Anhänge können vertrauliche und/oder rechtlich geschützte Informationen enthalten. Falls Sie nicht der angegebene Empfänger sind oder falls diese E-Mail irrtümlich an Sie adressiert wurde, benachrichtigen Sie uns bitte sofort durch Antwort-E-Mail und löschen Sie diese E-Mail nebst etwaigen Anlagen von Ihrem System. Ebenso dürfen Sie diese E-Mail oder ihre Anlagen nicht kopieren oder an Dritte weitergeben. Vielen Dank. This e-mail and any attachment may contain confidential and/or privileged information. If you are not the named addressee or if this transmission has been addressed to you in error, please notify us immediately by reply e-mail and then delete this e-mail and any attachment from your system. Please understand that you must not copy this e-mail or any attachment or disclose the contents to any other person. Thank you for your cooperation.
RE: Cassandra Disk storage capacity
Hi, See data_file_directories and commitlog_directory in the settings file cassandra.yaml. Cheers, Romain Hari Rajendhran hari.rajendh...@tcs.com a écrit sur 07/04/2014 12:56:37 : De : Hari Rajendhran hari.rajendh...@tcs.com A : user@cassandra.apache.org, Date : 07/04/2014 12:58 Objet : Cassandra Disk storage capacity Hi Team, We have a 3 node Apache cassandra 2.0.4 setup installed in our lab setup.We have set data directory to /var/lib/cassandra/data.What would be the maximum disk storage that will be used for cassandra data storage. Note : /var partition has a storage capacity of 40GB. My question is whether cassandra will the entire / directory for data storage ? If no, how to specify multiple directories for data storage ?? Best Regards Hari Krishnan Rajendhran Hadoop Admin DESS-ABIM ,Chennai BIGDATA Galaxy Tata Consultancy Services Cell:- 9677985515 Mailto: hari.rajendh...@tcs.com Website: http://www.tcs.com Experience certainty. IT Services Business Solutions Consulting =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you
Re: Cassandra Disk storage capacity
Hi, Thanks for the update Still i have few queries which needs to be clarified 1) I am confused why cassandra uses the entire disk space ( / Directory) even when we specify /var/lib/cassandra/data as the directory in Cassandra.yaml file 2) Is it only during compaction ,cassandra will use the entire Disk space ? 3) What is the best way to monitor the cassandra Disk usage ?? is there a opensource monitoring tool for this ?? Best Regards Hari Krishnan Rajendhran Hadoop Admin DESS-ABIM ,Chennai BIGDATA Galaxy Tata Consultancy Services Cell:- 9677985515 Mailto: hari.rajendh...@tcs.com Website: http://www.tcs.com Experience certainty. IT Services Business Solutions Consulting -Jan Kesten j.kes...@enercast.de wrote: - To: user@cassandra.apache.org From: Jan Kesten j.kes...@enercast.de Date: 04/07/2014 04:41PM Subject: Re: Cassandra Disk storage capacity Hi Hari, C* will use your entire space - that is something one should monitor. Depending on your choose on compaction strategy your data_dir should not be filled up entirely - in the worst case compaction will need space as large as the sstables on disk, therefore 50% should be free space. The parameters used for on disk storage are commitlog_directory and data_file_directories and saved_caches_directory. The paramter data_file_directories is in plural, you can easily put more than one directory here (and you should do this instead of using RAID). Cheers, Jan Am 07.04.2014 12:56, schrieb Hari Rajendhran: Hi Team, We have a 3 node Apache cassandra 2.0.4 setup installed in our lab setup.We have set data directory to /var/lib/cassandra/data.What would be the maximum disk storage that will be used for cassandra data storage. Note : /var partition has a storage capacity of 40GB. My question is whether cassandra will the entire / directory for data storage ? If no, how to specify multiple directories for data storage ?? Best Regards Hari Krishnan Rajendhran Hadoop Admin DESS-ABIM ,Chennai BIGDATA Galaxy Tata Consultancy Services Cell:- 9677985515 Mailto: hari.rajendh...@tcs.com Website: http://www.tcs.com Experience certainty. IT Services Business Solutions Consulting =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you -- Jan Kesten, mailto:j.kes...@enercast.de Tel.: +49 561/4739664-0 FAX: -9 enercast GmbH Friedrich-Ebert-Str. 104 D-34119 Kassel HRB15471 http://www.enercast.de Online-Prognosen für erneuerbare Energien Geschäftsführung: Dipl. Ing. Thomas Landgraf, Bernd Kratz Diese E-Mail und etwaige Anhänge können vertrauliche und/oder rechtlich geschützte Informationen enthalten. Falls Sie nicht der angegebene Empfänger sind oder falls diese E-Mail irrtümlich an Sie adressiert wurde, benachrichtigen Sie uns bitte sofort durch Antwort-E-Mail und löschen Sie diese E-Mail nebst etwaigen Anlagen von Ihrem System. Ebenso dürfen Sie diese E-Mail oder ihre Anlagen nicht kopieren oder an Dritte weitergeben. Vielen Dank. This e-mail and any attachment may contain confidential and/or privileged information. If you are not the named addressee or if this transmission has been addressed to you in error, please notify us immediately by reply e-mail and then delete this e-mail and any attachment from your system. Please understand that you must not copy this e-mail or any attachment or disclose the contents to any other person. Thank you for your cooperation.
Re: Cassandra Disk storage capacity
Am 07.04.2014 13:24, schrieb Hari Rajendhran: 1) I am confused why cassandra uses the entire disk space ( / Directory) even when we specify /var/lib/cassandra/data as the directory in Cassandra.yaml file 2) Is it only during compaction ,cassandra will use the entire Disk space ? 3) What is the best way to monitor the cassandra Disk usage ?? is there a opensource monitoring tool for this ?? Hi, if your / and /var/lib/cassandra/data are on different disks (or partitions) only /var/lib/cassandra/data will get filled entirely. Often this is not the case per default and you will have to create this mountpoints by yourself. Also keep in mind to keep commitlogs on a seperate disk from data to improve performance. The extra space is only needed during compaction - but cassandra will fire up compactions by itself, so you must keep this free space maintained all the time. This is valid for SizeTieredCompation, Leveled- or HybridCompations are cheaper on disk space. For the last point - there are many tools to monitor your servers inside your cluster. Nagios, Hyperic HQ and OpenNMS are some of them - you can define alerts which keep you up to date. Cheers, jan
Re: Cassandra Disk storage capacity
I guess there is a misunderstanding here: I am confused why cassandra uses the entire disk space ( / Directory) even when we specify /var/lib/cassandra/data as the directory in Cassandra.yaml file C* will use the entire MOUNTPOINT, which is not necessarily your entire total disk space. If you have a separate mountpoint for /var/lib/cassandra/data it is my understanding that C* will fill that, to the best of my knowledge. You still have stuff located outside of it, though. Like logs, to begin with. Cheers Bèrto On 7 April 2014 12:24, Hari Rajendhran hari.rajendh...@tcs.com wrote: Hi, Thanks for the update Still i have few queries which needs to be clarified 1) I am confused why cassandra uses the entire disk space ( / Directory) even when we specify /var/lib/cassandra/data as the directory in Cassandra.yaml file 2) Is it only during compaction ,cassandra will use the entire Disk space ? 3) What is the best way to monitor the cassandra Disk usage ?? is there a opensource monitoring tool for this ?? Best Regards Hari Krishnan Rajendhran Hadoop Admin DESS-ABIM ,Chennai BIGDATA Galaxy Tata Consultancy Services Cell:- 9677985515 Mailto: hari.rajendh...@tcs.com Website: http://www.tcs.com Experience certainty. IT Services Business Solutions Consulting -Jan Kesten j.kes...@enercast.de wrote: - To: user@cassandra.apache.org From: Jan Kesten j.kes...@enercast.de Date: 04/07/2014 04:41PM Subject: Re: Cassandra Disk storage capacity Hi Hari, C* will use your entire space - that is something one should monitor. Depending on your choose on compaction strategy your data_dir should not be filled up entirely - in the worst case compaction will need space as large as the sstables on disk, therefore 50% should be free space. The parameters used for on disk storage are commitlog_directory and data_file_directories and saved_caches_directory. The paramter data_file_directories is in plural, you can easily put more than one directory here (and you should do this instead of using RAID). Cheers, Jan Am 07.04.2014 12:56, schrieb Hari Rajendhran: Hi Team, We have a 3 node Apache cassandra 2.0.4 setup installed in our lab setup.We have set data directory to /var/lib/cassandra/data.What would be the maximum disk storage that will be used for cassandra data storage. Note : /var partition has a storage capacity of 40GB. My question is whether cassandra will the entire / directory for data storage ? If no, how to specify multiple directories for data storage ?? Best Regards Hari Krishnan Rajendhran Hadoop Admin DESS-ABIM ,Chennai BIGDATA Galaxy Tata Consultancy Services Cell:- 9677985515 Mailto: hari.rajendh...@tcs.com Website: http://www.tcs.com Experience certainty. IT Services Business Solutions Consulting =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you -- Jan Kesten, mailto:j.kes...@enercast.de j.kes...@enercast.de Tel.: +49 561/4739664-0 FAX: -9 enercast GmbH Friedrich-Ebert-Str. 104 D-34119 Kassel HRB15471 http://www.enercast.de Online-Prognosen für erneuerbare Energien Geschäftsführung: Dipl. Ing. Thomas Landgraf, Bernd Kratz Diese E-Mail und etwaige Anhänge können vertrauliche und/oder rechtlich geschützte Informationen enthalten. Falls Sie nicht der angegebene Empfänger sind oder falls diese E-Mail irrtümlich an Sie adressiert wurde, benachrichtigen Sie uns bitte sofort durch Antwort-E-Mail und löschen Sie diese E-Mail nebst etwaigen Anlagen von Ihrem System. Ebenso dürfen Sie diese E-Mail oder ihre Anlagen nicht kopieren oder an Dritte weitergeben. Vielen Dank. This e-mail and any attachment may contain confidential and/or privileged information. If you are not the named addressee or if this transmission has been addressed to you in error, please notify us immediately by reply e-mail and then delete this e-mail and any attachment from your system. Please understand that you must not copy this e-mail or any attachment or disclose the contents to any other person. Thank you for your cooperation. -- == If Pac-Man had affected us as kids, we'd all be running around in a darkened room munching pills and listening to repetitive music.
