We have the setup on Google cloud platform. Each node has 8 CPU + 30GB memory. 10 nodes for spark another 9nodes for Cassandra. We are using spark 1.3.0 and Datastax bundle 4.5.9(which has 2.0.x Cassandra). Spark master and worker daemon uses Xmx & Xms 4G. We have not changed the default setting of Cassandra, should we be increasing the JVM memory?
we have 9 streaming jobs the core usage varies from 2-6 and memory usage from 1 - 4 gb. We have budget to use higher CPU or higher memory systems hence was planning to have them together on more efficient nodes. Regards Vivek On Sat, Jan 23, 2016 at 7:13 am, Ted Yu <yuzhih...@gmail.com<mailto:yuzhih...@gmail.com>> wrote: Can you give us a bit more information ? How much memory does each node have ? What's the current heap allocation for Cassandra process and executor ? Spark / Cassandra release you are using Thanks On Fri, Jan 22, 2016 at 5:37 PM, <vivek.meghanat...@wipro.com<mailto:vivek.meghanat...@wipro.com>> wrote: Hi All, What is the right spark Cassandra cluster setup - having Cassandra cluster and spark cluster in different nodes or they should be on same nodes. We are having them in different nodes and performance test shows very bad result for the spark streaming jobs. Please let us know. Regards Vivek The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com<http://www.wipro.com> The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com
