I'm querying a single cassandra partition using sqlContext and Its temView
which creates more than 2000 tasks on spark and took about 360 seconds:
sqlContext.read().format("org.apache.spark.sql.cassandra).options(ops).load.createOrReplaceTempView("tableName")
But using javaFunctions(sc).cassandraTable().where() it creates only one task
which response in 200 ms!
I'm using exactly the same where clause for both scenarios.
Spark UI shows like 60 GB input for sqlcontext scenario and only a few KBs for
javaFunctions scenario
Sent using Zoho Mail
