Re: [EXTERNAL] Re: Conflicting PySpark Storage Level Defaults?

2019-09-16 Thread grp
; df.cache(). >> >> Do we have to explicitly set to … StorageLevel.MEMORY_AND_DISK … to get the >> serialized benefit in Python (which I thought was automatic)? Or is the >> Spark UI incorrect? >> >> SO post with specific example

Re: Conflicting PySpark Storage Level Defaults?

2019-09-16 Thread Jörn Franke
lized benefit in Python (which I thought was automatic)? Or is the > Spark UI incorrect? > > SO post with specific example/details => > https://stackoverflow.com/questions/56926337/conflicting-pys

Conflicting PySpark Storage Level Defaults?

2019-09-15 Thread grp
ave to explicitly set to … StorageLevel.MEMORY_AND_DISK … to get the serialized benefit in Python (which I thought was automatic)? Or is the Spark UI incorrect? SO post with specific example/details => https://stackoverflow.com/questions/56926337/conflicting-pyspark-storage-level-defaults Tha