Good Day
I am having trouble saving a spark.ml Pipeline model to a pickle file, when
running locally on my PC.
I've tried a few ways to save the model:
1. mlflow.spark.log_model(artifact_path=experiment.artifact_location,
spark_model= model, registered_model_name="myModel")
* with error that the spark model is multiple files
2. pickle.dump(model, file): with error - TypeError: cannot pickle
'_thread.RLock' object
3. model.save('path'): with Java errors:
* at
org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:291)
* at
org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:182)
* at
org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:99)
... 67 more
Your assistance on this would be much appreciated.
Regards,
Caleb Mnisi
Consultant | Deloitte Analytics | Cognitive Advantage
Deloitte & Touche
5th floor, 5 Magwa Crescent, Waterfall City, 2090
M: +27 72 170 8779
[email protected]<mailto:[email protected]> |
www2.deloitte.com/za<mailto:[email protected]>
[cid:[email protected]]
Please consider the environment before printing.
*Disclaimer:* This email is subject to important restrictions, qualifications
and disclaimers ("the Disclaimer") that must be accessed and read by visiting
our website and viewing the webpage at the following address:
http://www.deloitte.com/za/disclaimer. The Disclaimer forms part of the content
of this email. If you cannot access the Disclaimer, please obtain a copy
thereof from us by sending an email to [email protected]. Deloitte
refers to a Deloitte member firm, one of its related entities, or Deloitte
Touche Tohmatsu Limited ("DTTL"). Each Deloitte member firm is a separate legal
entity and a member of DTTL. DTTL does not provide services to clients. Please
see www.deloitte.com/about to learn more.
