Re: Spark Cassandra clusters

Thanks mohammed and Ted. I will try out the options and let you all know the progress. Also had posted in spark Cassandra connector community, got similar response. Regards Vivek On Sat, Jan 23, 2016 at 11:37 am, Mohammed Guller mailto:moham...@glassbeam.com>> wrote: Vivek, By default, Cassan

Re: Spark Cassandra clusters

Thanks Ted, also what is the suggested memory setting for Cassandra process? Regards Vivek On Sat, Jan 23, 2016 at 7:57 am, Ted Yu mailto:yuzhih...@gmail.com>> wrote: >From your description, putting Cassandra daemon on Spark cluster should be >feasible. One aspect to be measured is how much l

Re: Spark Cassandra clusters

+ spark standalone cluster On Sat, Jan 23, 2016 at 7:33 am, Vivek Meghanathan (WT01 - NEP) mailto:vivek.meghanat...@wipro.com>> wrote: We have the setup on Google cloud platform. Each node has 8 CPU + 30GB memory. 10 nodes for spark another 9nodes for Cassandra. We are using spark 1.3.0 and Da

Re: Spark Cassandra clusters

Thanks. We are using spark - Cassandra connector aligned for spark 1.3. Regards Vivek On Sat, Jan 23, 2016 at 7:27 am, Durgesh Verma mailto:dv21...@gmail.com>> wrote: This may be useful, you can try connectors. https://academy.datastax.com/demos/getting-started-apache-spark-and-cassandra https

Re: Spark Cassandra clusters

We have the setup on Google cloud platform. Each node has 8 CPU + 30GB memory. 10 nodes for spark another 9nodes for Cassandra. We are using spark 1.3.0 and Datastax bundle 4.5.9(which has 2.0.x Cassandra). Spark master and worker daemon uses Xmx & Xms 4G. We have not changed the default setting

Spark Cassandra clusters

Hi All, What is the right spark Cassandra cluster setup - having Cassandra cluster and spark cluster in different nodes or they should be on same nodes. We are having them in different nodes and performance test shows very bad result for the spark streaming jobs. Please let us know. Regards Vive

RE: spark 1.6.0 on ec2 doesn't work

Have you verified the spark master/slaves are started correctly? Please check using netstat command and open ports mode. Are they listening? Binds to which address etc.. From: Oleg Ruchovets [mailto:oruchov...@gmail.com] Sent: 19 January 2016 11:24 To: Peter Zhang Cc: Daniel Darabos ; user Su

Re: Spark Streaming + Kafka + scala job message read issue

All, The issue was related to apache Cassandra. I have changed the Cassandra to datastax Cassandra and the issue is resolved. Also I have changed the spark version to 1.3. There is some serious issue is there between spark Cassandra connector and apache Cassandra 2.1+ while using in spark strea

Re: spark job failure - akka error Association with remote system has failed

Mohammed, As i have mentioned in latest email, it was failing due to a communication issue with cassandra. Once i fixed that the issue is no more there. Regards, Vivek M From: Mohammed Guller Sent: Thursday, January 14, 2016 4:38 AM To: Vivek Meghanathan (WT0

RE: spark job failure - akka error Association with remote system has failed

Identified the problem - the Cassandra seed ip we use was down! From: Vivek Meghanathan (WT01 - NEP) Sent: 13 January 2016 13:06 To: 'user@spark.apache.org' Subject: RE: spark job failure - akka error Association with remote system has failed I have used master_ip as ip address and spark conf a

RE: spark job failure - akka error Association with remote system has failed

I have used master_ip as ip address and spark conf also has Ip address . But the following logs shows hostname. (The spark Ui shows master details in IP) 16/01/13 12:31:38 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkDriver@masternode1:36537] has failed, addr

spark job failure - akka error Association with remote system has failed

Hi All, I am running spark 1.3.0 standalone cluster mode, we have rebooted the cluster servers (system reboot). After that the spark jobs are failing by showing following error (it fails within 7-8 seconds). 2 of the jobs are running fine. All the jobs used to be stable before the system reboot.

RE: Spark Streaming + Kafka + scala job message read issue

Hello All, After investigating further using a test program, we were able to read the kafka input messages using spark streaming. Once we add a particular line which performs map and reduce - and groupByKey (all written in single line), we are not seeing the input message details in the logs.

RE: Spark 1.5.2 compatible spark-cassandra-connector

Thank you mwy and Sun for your response. Yes basic things are working for me using this connector(guava issue was encountered earlier but with proper exclusion of old version we have resolved it). The current issue is strange one �C we have a kafka-spark-cassandra streaming job in spark. The al

Spark 1.5.2 compatible spark-cassandra-connector

All, What is the compatible spark-cassandra-connector for spark 1.5.2? I can only find the latest connector version spark-cassandra-connector_2.10-1.5.0-M3 which has dependency with 1.5.1 spark. Can we use the same for 1.5.2? Any classpath issues needs to be handled or any jars needs to be exclu

Re: Can't submit job to stand alone cluster

+ if exists whether it has read permission for the user who tries to run the job. Regards Vivek On Tue, Dec 29, 2015 at 6:56 am, Ted Yu mailto:yuzhih...@gmail.com>> wrote: Have you verified that the following file does exist ? /home/hadoop/git/scalaspark/./target/scala-2.10/cluster-incidents_

Re: Spark Streaming + Kafka + scala job message read issue

Hi Bryan, Yes we are using only 1 thread per topic as we have only one Kafka server with 1 partition. What kind of logs will tell us what offset spark stream is reading from Kafka or is it resetting something without reading? Regards Vivek Sent using CloudMagic Email

Re: REST Api not working in spark

Which JRE version you are using? One possibility is you are running a lover version of JRE than it required. Regards Vivek Sent using CloudMagic Email On Fri, Dec 25, 2015 at 4:13 pm, aman solanki mailto:youthi

Re: Spark Streaming + Kafka + scala job message read issue

Hi Brian,PhuDuc, All 8 jobs are consuming 8 different IN topics. 8 different Scala jobs running each topic map mentioned below has only 1 thread number mentioned. In this case group should not be a problem right. Here is the complete flow, spring MVC sends in messages to Kafka , spark streamin

Re: Spark Streaming + Kafka + scala job message read issue

Any help is highly appreciated, i am completely stuck here.. From: Vivek Meghanathan (WT01 - NEP) Sent: Thursday, December 24, 2015 7:50 PM To: Bryan; user@spark.apache.org Subject: RE: Spark Streaming + Kafka + scala job message read issue We are using the olde

RE: Spark Streaming + Kafka + scala job message read issue

We are using the older receiver based approach, the number of partitions is 1 (we have a single node kafka) and we use single thread per topic still we have the problem. Please see the API we use. All 8 spark jobs use same group name – is that a problem? val topicMap = topics.split(",").map((_,

Spark Streaming + Kafka + scala job message read issue

Hi All, We are using Bitnami Kafka 0.8.2 + spark 1.5.2 in Google cloud platform. Our spark streaming job(consumer) not receiving all the messages sent to the specific topic. It receives 1 out of ~50 messages(added log in the job stream and identified). We are not seeing any errors in the kaf