Memory issues in 3.0.2 but works well on 2.4.4

[Praneeth Shishtla]

Hi,
I have a simple DecisionForest model and was able to train the model on
pyspark==2.4.4 without any issues.
However, when I upgraded to pyspark==3.0.2, the fit takes a lot of time and
eventually errors out saying out of memory. Even tried reducing the number
of samples for training but no luck.
Can anyone help with this?

Best,
Praneeth




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Sleep behavior

[Praneeth Shishtla]

Hi,
We have a 6 node spark cluster and have some pyspark jobs running on it.
The job is dependent on external application and to have resiliency we try a
couple of times.
Will it be fine to induce some wait time between two runs(using
time.sleep()) ? Or could there by any sync issues?
Wanted to understand the behaviour/issues if any.

Thanks,
Praneeth





--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Calibration Methods

[Praneeth Shishtla]

Hi,

I am looking for calibrating the output of a pyspark model. 
I looked for possible implementations in Spark but didn't find any.

Sklearn has CalibratedClassifierCV 
https://scikit-learn.org/stable/modules/generated/sklearn.calibration.CalibratedClassifierCV.html
Could anyone point if there is any/similar implementation of the same in
Spark?

Regards,
Praneeth



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Calibration Methods

[Praneeth Shishtla]

Hi All,

I am looking for ways to calibrate the output of a pyspark ML model. 
Could anyone share if there are any implementations around of the same
available in spark/pyspark?

Here is the implementation available in sklearn:
https://scikit-learn.org/stable/modules/generated/sklearn.calibration.CalibratedClassifierCV.html

Please feel to share your experience/ideas around the calibration methods
with spark.

Regards,
Praneeth



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org