Good Day I am having trouble saving a spark.ml Pipeline model to a pickle file, when running locally on my PC. I've tried a few ways to save the model:
1. mlflow.spark.log_model(artifact_path=experiment.artifact_location, spark_model= model, registered_model_name="myModel") * with error that the spark model is multiple files 2. pickle.dump(model, file): with error - TypeError: cannot pickle '_thread.RLock' object 3. model.save('path'): with Java errors: * at org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:291) * at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:182) * at org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:99) ... 67 more Your assistance on this would be much appreciated. Regards, Caleb Mnisi Consultant | Deloitte Analytics | Cognitive Advantage Deloitte & Touche 5th floor, 5 Magwa Crescent, Waterfall City, 2090 M: +27 72 170 8779 cmn...@deloitte.co.za<mailto:cmn...@deloitte.co.za> | www2.deloitte.com/za<mailto:cmn...@deloitte.co.za> [cid:image001.png@01D95757.1241E560] Please consider the environment before printing. *Disclaimer:* This email is subject to important restrictions, qualifications and disclaimers ("the Disclaimer") that must be accessed and read by visiting our website and viewing the webpage at the following address: http://www.deloitte.com/za/disclaimer. The Disclaimer forms part of the content of this email. If you cannot access the Disclaimer, please obtain a copy thereof from us by sending an email to zaitserviced...@deloitte.co.za. Deloitte refers to a Deloitte member firm, one of its related entities, or Deloitte Touche Tohmatsu Limited ("DTTL"). Each Deloitte member firm is a separate legal entity and a member of DTTL. DTTL does not provide services to clients. Please see www.deloitte.com/about to learn more.