I'm looking to use spark for some ETL, which will mostly consist of "update"
statements (a column is a set, that'll be appended to, so a simple insert is
likely not going to work). As such, it seems like issuing CQL queries to import
the data is the best option. Using the Spark Cassandra Connector, I see I can
do this:
https://github.com/datastax/spark-cassandra-connector/blob/master/doc/1_connecting.md#connecting-manually-to-cassandra
Now I don't want to open a session and close it for every row in the source (am
I right in not wanting this? Usually, I have one session for the entire
process, and keep using that in "normal" apps). However, it says that the
connector is serializable, but the session is obviously not. So, wrapping the
whole import inside a single "withSessionDo" seems like it'll cause problems. I
was thinking of using something like this:
class CassandraStorage(conf:SparkConf) {
val session = CassandraConnector(conf).openSession()
def store (t:Thingy) : Unit = {
//session.execute cql goes here
}
}
Is this a good approach? Do I need to worry about closing the session? Where /
how best would I do that? Any pointers are appreciated.
Thanks,Ashic.
