Re: never understand

2016-05-25 Thread Andrew Ehrlich
- Try doing less in each transformation - Try using different data structures within the transformations - Try not caching anything to free up more memory On Wed, May 25, 2016 at 1:32 AM, pseudo oduesp wrote: > hi guys , > -i get this errors with pyspark 1.5.0 under

never understand

2016-05-25 Thread pseudo oduesp
hi guys , -i get this errors with pyspark 1.5.0 under cloudera CDH 5.5 (yarn) -i use yarn to deploy job on cluster. -i use hive context and parquet file to save my data. limit container 16 GB number of executor i tested befor it s 12 GB (executor memory) -i tested to increase number of