Hi all,
I have been struggling with Cassandra’s lack of adhoc query support (I know this is an anti-pattern of Cassandra, but sometimes management come over and ask me to run stuff and it’s impossible to explain that it will take me a while when it would take about 10 seconds in MySQL) so I have put together the following code snippet that bundles DataStax’s Cassandra Spark connector and allows you to submit Spark SQL to it, outputting the results in a text file. Does anyone spot any obvious flaws in this plan?? (I have a lot more error handling etc in my code, but removed it here for brevity) *private* *void* run(String sqlQuery) { SparkContext scc = *new* SparkContext(conf); CassandraSQLContext csql = *new* CassandraSQLContext(scc); DataFrame sql = csql.sql(sqlQuery); String folderName = "/tmp/output_" + System.*currentTimeMillis*(); *LOG*.info("Attempting to save SQL results in folder: " + folderName); sql.rdd().saveAsTextFile(folderName); *LOG*.info("SQL results saved"); } *public* *static* *void* main(String[] args) { String sparkMasterUrl = args[0]; String sparkHost = args[1]; String sqlQuery = args[2]; SparkConf conf = *new* SparkConf(); conf.setAppName("Java Spark SQL"); conf.setMaster(sparkMasterUrl); conf.set("spark.cassandra.connection.host", sparkHost); JavaSparkSQL app = *new* JavaSparkSQL(conf); app.run(sqlQuery, printToConsole); } I can then submit this to Spark with ‘spark-submit’: Ø *./spark-submit --class com.algomi.spark.JavaSparkSQL --master spark://sales3:7077 spark-on-cassandra-0.0.1-SNAPSHOT-jar-with-dependencies.jar spark://sales3:7077 sales3 "select * from mykeyspace.operationlog" * It seems to work pretty well, so I’m pretty happy, but wondering why this isn’t common practice (at least I haven’t been able to find much about it on Google) – is there something terrible that I’m missing? Thanks! Matthew