Re: spark uses too much memory maybe (binaryFiles() with more than 1 million files in HDFS), groupBy or reduceByKey()

[Konstantinos Kougios]

after 2h of running, now I got a 10GB long[], 1.3mil instances of long[]

So probably information about the files again.


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org