Oh if that is the case then you can try tuning " spark.cassandra.input.split.size"
spark.cassandra.input.split.size approx number of Cassandra partitions in a Spark partition 100000 Hope this helps. Thanks Ankur On Thu, Sep 3, 2015 at 12:22 PM, Alaa Zubaidi (PDF) <alaa.zuba...@pdf.com> wrote: > Thanks Ankur, > > But I grabbed some keys from the Spark results and ran "nodetool -h.... > getendpoints ...." and it showed the data is coming from at least 2 nodes? > Regards, > Alaa > > On Thu, Sep 3, 2015 at 12:06 PM, Ankur Srivastava < > ankur.srivast...@gmail.com> wrote: > >> Hi Alaa, >> >> Partition when using CassandraRDD depends on your partition key in >> Cassandra table. >> >> If you see only 1 partition in the RDD it means all the rows you have >> selected have same partition_key in C* >> >> Thanks >> Ankur >> >> >> On Thu, Sep 3, 2015 at 11:54 AM, Alaa Zubaidi (PDF) <alaa.zuba...@pdf.com >> > wrote: >> >>> Hi, >>> >>> I testing Spark and Cassandra, Spark 1.4, Cassandra 2.1.7 cassandra >>> spark connector 1.4, running in standalone mode. >>> >>> I am getting 4000 rows from Cassandra (4mb row), where the row keys are >>> random. >>> ...... sc.cassandraTable[RES](keyspace,res_name).where(res_where).cache >>> >>> I am expecting that it will generate few partitions. >>> However, I can ONLY see 1 partition. >>> I cached the CassandraRDD and in the UI storage tab it shows ONLY 1 >>> partition. >>> >>> Any idea, why I am getting 1 partition? >>> >>> Thanks, >>> Alaa >>> >>> >>> >>> *This message may contain confidential and privileged information. If it >>> has been sent to you in error, please reply to advise the sender of the >>> error and then immediately permanently delete it and all attachments to it >>> from your systems. If you are not the intended recipient, do not read, >>> copy, disclose or otherwise use this message or any attachments to it. The >>> sender disclaims any liability for such unauthorized use. PLEASE NOTE that >>> all incoming e-mails sent to PDF e-mail accounts will be archived and may >>> be scanned by us and/or by external service providers to detect and prevent >>> threats to our systems, investigate illegal or inappropriate behavior, >>> and/or eliminate unsolicited promotional e-mails (“spam”). If you have any >>> concerns about this process, please contact us at * >>> *legal.departm...@pdf.com* <legal.departm...@pdf.com>*.* >> >> >> > > > -- > > Alaa Zubaidi > PDF Solutions, Inc. > 333 West San Carlos Street, Suite 1000 > San Jose, CA 95110 USA > Tel: 408-283-5639 > fax: 408-938-6479 > email: alaa.zuba...@pdf.com > > > *This message may contain confidential and privileged information. If it > has been sent to you in error, please reply to advise the sender of the > error and then immediately permanently delete it and all attachments to it > from your systems. If you are not the intended recipient, do not read, > copy, disclose or otherwise use this message or any attachments to it. The > sender disclaims any liability for such unauthorized use. PLEASE NOTE that > all incoming e-mails sent to PDF e-mail accounts will be archived and may > be scanned by us and/or by external service providers to detect and prevent > threats to our systems, investigate illegal or inappropriate behavior, > and/or eliminate unsolicited promotional e-mails (“spam”). If you have any > concerns about this process, please contact us at * > *legal.departm...@pdf.com* <legal.departm...@pdf.com>*.* >
