After 8 hours. The usage of memory become stable. Use the Top command will find it will be 75%. So means 12GB memory.
But it still do not make sense. Because my workload is very small. I use this spark to calculate on one csv file every 20 seconds. The size of the csv file is 1.3M. So spark is using almost 10 000 times of memory than my workload. Does that mean I need prepare 1TB RAM if the workload is 100M? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Will-the-HiveContext-cause-memory-leak-tp26921p26927.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
