Thank you mwy and Sun for your response. Yes basic things are working for me
using this connector(guava issue was encountered earlier but with proper
exclusion of old version we have resolved it).
The current issue is strange one �C we have a kafka-spark-cassandra streaming
job in spark. The all jobs are working fine in local cluster (local lab) using
1.3.0 spark. We are trying the same setup in Google cloud platform in 1.3.0 the
jobs are up but it does not seems to be processing the kafka messages. Using
1.5.2 spark with 1.5.0-M3 connector + 0.8.2.2 kafka I am able to make most of
the jobs work but one of them is processing only 1 out of 50 messages from
kafka.
We have wrote a test program(scala we use) to parse data from kafka stream �C
it is working fine and always receives the messages. It connects to Cassandra
gets some data works fine and prints the input data till that time, but when I
enable the join,map,reduce login , it starts to miss the message(not printing
above worked lines also). Please find the lines added below, once I add the
commented line to the job , it does not print any incoming data while I am able
to print till then. Any ideas how to debug?
val cqlOfferSummaryRdd = ssc.sc.cql3Cassandra[OfferSummary](casOfferSummary)
.map(summary => (summary.listingId, (summary.offerId,
summary.redeem, summary.viewed, summary.reserved)))
/**
//RDD -> ((listingId, searchId), Iterable(offerId, redeemCount,
viewCount))
val directListingOfferSummary = directViews.transform(Rdd => {
Rdd.join(cqlOfferSummaryRdd,10)}) //RDD -> ((listingId), (Direct, (offerId,
redeemCount, viewCount)))
.map(rdd => ((rdd._1, rdd._2._1.searchId, rdd._2._2._1),
(rdd._2._2._2, rdd._2._2._3, rdd._2._2._4))) //RDD -> ((listingId, searchId,
offerId), (redeemCount, viewCount))
.reduceByKey((a, b) => (a._1 + b._1, a._2 + b._2, a._3 + b._3))
.map(rdd => ((rdd._1._1, rdd._1._2), (rdd._1._3, rdd._2._1,
rdd._2._2, rdd._2._3))).groupByKey(10) //RDD -> ((listingId, searchId),
Iterable(offerId, redeemCount, viewCount))
directListingOfferSummary.print()
**/
Regards,
Vivek
From: mwy [mailto:wenyao...@dewmobile.net]
Sent: 30 December 2015 08:27
To: fightf...@163.com; Vivek Meghanathan (WT01 - NEP)
<vivek.meghanat...@wipro.com>; user <user@spark.apache.org>
Subject: Re: Spark 1.5.2 compatible spark-cassandra-connector
2.10-1.5.0-M3 & spark 1.5.2 work for me. The jar is built by sbt-assembly.
Just for reference.
发件人: "fightf...@163.com<mailto:fightf...@163.com>"
<fightf...@163.com<mailto:fightf...@163.com>>
日期: Wednesday, December 30, 2015 at 10:22
至: "vivek.meghanat...@wipro.com<mailto:vivek.meghanat...@wipro.com>"
<vivek.meghanat...@wipro.com<mailto:vivek.meghanat...@wipro.com>>, user
<user@spark.apache.org<mailto:user@spark.apache.org>>
主题: Re: Spark 1.5.2 compatible spark-cassandra-connector
Hi, Vivek M
I had ever tried 1.5.x spark-cassandra connector and indeed encounter some
classpath issues, mainly for the guaua dependency.
I believe that can be solved by some maven config, but have not tried that yet.
Best,
Sun.
________________________________
fightf...@163.com<mailto:fightf...@163.com>
From: vivek.meghanat...@wipro.com<mailto:vivek.meghanat...@wipro.com>
Date: 2015-12-29 20:40
To: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Spark 1.5.2 compatible spark-cassandra-connector
All,
What is the compatible spark-cassandra-connector for spark 1.5.2? I can only
find the latest connector version spark-cassandra-connector_2.10-1.5.0-M3 which
has dependency with 1.5.1 spark. Can we use the same for 1.5.2? Any classpath
issues needs to be handled or any jars needs to be excluded while packaging the
application jar?
http://central.maven.org/maven2/com/datastax/spark/spark-cassandra-connector_2.10/1.5.0-M3/spark-cassandra-connector_2.10-1.5.0-M3.pom
Regards,
Vivek M
The information contained in this electronic message and any attachments to
this message are intended for the exclusive use of the addressee(s) and may
contain proprietary, confidential or privileged information. If you are not the
intended recipient, you should not disseminate, distribute or copy this e-mail.
Please notify the sender immediately and destroy all copies of this message and
any attachments. WARNING: Computer viruses can be transmitted via email. The
recipient should check this email and any attachments for the presence of
viruses. The company accepts no liability for any damage caused by any virus
transmitted by this email. www.wipro.com<http://www.wipro.com>
The information contained in this electronic message and any attachments to
this message are intended for the exclusive use of the addressee(s) and may
contain proprietary, confidential or privileged information. If you are not the
intended recipient, you should not disseminate, distribute or copy this e-mail.
Please notify the sender immediately and destroy all copies of this message and
any attachments. WARNING: Computer viruses can be transmitted via email. The
recipient should check this email and any attachments for the presence of
viruses. The company accepts no liability for any damage caused by any virus
transmitted by this email. www.wipro.com
