Pickle won't work. But the others should. I think you are specifying an
invalid path in both cases but hard to say without more detail
On Wed, Mar 15, 2023, 9:13 AM Mnisi, Caleb <cmn...@deloitte.co.za.invalid>
wrote:
> Good Day
>
>
>
> I am having trouble saving a spark.ml Pipeline model to a pickle file,
> when running locally on my PC.
>
> I’ve tried a few ways to save the model:
>
> 1. mlflow.spark.log_model(artifact_path=experiment.artifact_location,
> spark_model= model, registered_model_name="myModel")
> 1. with error that the spark model is multiple files
> 2. pickle.dump(model, file): with error - TypeError: cannot pickle
> '_thread.RLock' object
> 3. model.save(‘path’): with Java errors:
> 1. at
>
> org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:291)
> 2. at
>
> org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:182)
> 3. at
>
> org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:99)
> ... 67 more
>
>
>
> Your assistance on this would be much appreciated.
>
> Regards,
>
>
>
> *Caleb Mnisi*
>
> Consultant | Deloitte Analytics | Cognitive Advantage
>
> Deloitte & Touche
>
> 5th floor, 5 Magwa Crescent, Waterfall City, 2090
>
> M: +27 72 170 8779
>
> *cmn...@deloitte.co.za <cmn...@deloitte.co.za>* | www2.deloitte.com/za
> <cmn...@deloitte.co.za>
>
>
>
>
>
> Please consider the environment before printing.
>
>
> *Disclaimer:* This email is subject to important restrictions,
> qualifications and disclaimers ("the Disclaimer") that must be accessed and
> read by visiting our website and viewing the webpage at the following
> address: http://www.deloitte.com/za/disclaimer. The Disclaimer forms part
> of the content of this email. If you cannot access the Disclaimer, please
> obtain a copy thereof from us by sending an email to
> zaitserviced...@deloitte.co.za. Deloitte refers to a Deloitte member
> firm, one of its related entities, or Deloitte Touche Tohmatsu Limited
> (“DTTL”). Each Deloitte member firm is a separate legal entity and a member
> of DTTL. DTTL does not provide services to clients. Please see
> www.deloitte.com/about to learn more.
>
