[ https://issues.apache.org/jira/browse/SPARK-7360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiangrui Meng closed SPARK-7360. -------------------------------- Resolution: Done Fix Version/s: 1.4.0 Assignee: Xiangrui Meng (was: Nicholas Chammas) Target Version/s: 1.4.0 (was: 1.5.0) I marked this as "done". [~davies] mentioned that we need to serialize a lot of classes without useMemo, which hurts performance. So turning useMemo off globally is not a good option. > Compare Pyrolite performance affected by useMemo > ------------------------------------------------ > > Key: SPARK-7360 > URL: https://issues.apache.org/jira/browse/SPARK-7360 > Project: Spark > Issue Type: Task > Components: PySpark > Reporter: Xiangrui Meng > Assignee: Xiangrui Meng > Fix For: 1.4.0 > > > As discussed in SPARK-6288, disabling useMemo shows significant performance > on some ML tasks. We should test whether this is true across PySpark, and > consider patch Pyrolite for Spark. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org