Hi Dev , User
I want to store spark ml model in databases , so that I can reuse them
later on . I am
unable to pickle them . However while using scala I am able to convert them
into byte
array stream .
So for .eg I am able to do something below in scala but not in python
val modelToByteArray = new ByteArrayOutputStream()
val oos = new ObjectOutputStream(modelToByteArray)
oos.writeObject(model)
oos.close()
oos.flush()
spark.sparkContext.parallelize(Seq((model.uid, "my-neural-network-model",
modelToByteArray.toByteArray)))
.saveToCassandra("dfsdfs", "models", SomeColumns("uid", "name", "model")
But pickle.dumps(model) in pyspark throws error
cannot pickle '_thread.RLock' object
Please help on the same
Regards
Pralabh
