[ https://issues.apache.org/jira/browse/SPARK-6738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Shen updated SPARK-6738: ----------------------------- Description: ExternalAppendOnlyMap spill 2.2 GB data to disk: {code} 15/04/07 20:27:37 INFO collection.ExternalAppendOnlyMap: Thread 54 spilling in-memory map of 2.2 GB to disk (61 times so far) 15/04/07 20:27:37 INFO collection.ExternalAppendOnlyMap: /data11/yarnenv/local/usercache/spark/appcache/application_1423737010718_40455651/spark-local-20150407202613-4e80/11/temp_local_fdb4a583-5d13-4394-bccb-e1217d5db812 {code} But the file size is only 2.2M. {code} ll -h /data11/yarnenv/local/usercache/spark/appcache/application_1423737010718_40455651/spark-local-20150407202613-4e80/11/ total 2.2M -rw-r----- 1 spark users 2.2M Apr 7 20:27 temp_local_fdb4a583-5d13-4394-bccb-e1217d5db812 {code} The GC log show that the jvm memory is less than 1GB. {code} 2015-04-07T20:27:08.023+0800: [GC 981981K->55363K(3961344K), 0.0341720 secs] 2015-04-07T20:27:14.483+0800: [GC 987523K->53737K(3961344K), 0.0252660 secs] 2015-04-07T20:27:20.793+0800: [GC 985897K->56370K(3961344K), 0.0606460 secs] 2015-04-07T20:27:27.553+0800: [GC 988530K->59089K(3961344K), 0.0651840 secs] 2015-04-07T20:27:34.067+0800: [GC 991249K->62153K(3961344K), 0.0288460 secs] 2015-04-07T20:27:40.180+0800: [GC 994313K->61344K(3961344K), 0.0388970 secs] 2015-04-07T20:27:46.490+0800: [GC 993504K->59915K(3961344K), 0.0235150 secs] {code} The estimateSize is hugh difference with spill file size was: ExternalAppendOnlyMap spill 1100M data to disk: {code} 15/04/07 16:39:48 INFO collection.ExternalAppendOnlyMap: Thread 51 spilling in-memory map of 1106.5 MB to disk (12 times so far) /data11/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-994b/30/temp_local_e4347165-6263-4678-9f1d-67ad4bcd8fb5 15/04/07 16:39:49 INFO collection.ExternalAppendOnlyMap: Thread 51 spilling in-memory map of 1106.3 MB to disk (13 times so far) /data6/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-1e29/26/temp_local_76f9900b-1b3d-4cef-b3a2-6afcde14bbd9 15/04/07 16:39:49 INFO collection.ExternalAppendOnlyMap: Thread 51 spilling in-memory map of 1105.8 MB to disk (14 times so far) /data7/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-f883/26/temp_local_3ade0aec-ac1d-469d-bc99-b6fa87cb649b 15/04/07 16:39:50 INFO collection.ExternalAppendOnlyMap: Thread 51 spilling in-memory map of 1106.8 MB to disk (15 times so far) {code} But the file size is only 1.1M. {code} [tdwadmin@tdw-10-215-149-231 ~/tdwenv/tdwgaia/logs/container-logs/application_1423737010718_40308573/container_1423737010718_40308573_01_000008]$ ll -h /data7/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-f883/26/temp_local_3ade0aec-ac1d-469d-bc99-b6fa87cb649b -rw-r----- 1 spark users 1.1M Apr 7 16:39 /data7/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-f883/26/temp_local_3ade0aec-ac1d-469d-bc99-b6fa87cb649b {code} Here are the other spilled files. {code} [tdwadmin@tdw-10-215-149-231 ~/tdwenv/tdwgaia/logs/container-logs/application_1423737010718_40308573/container_1423737010718_40308573_01_000008]$ ll -h /data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/* /data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/09: total 1.1M -rw-r----- 1 spark users 1.1M Apr 7 16:39 temp_local_3a568e10-3997-4d13-adf1-e8dfe4ba4727 /data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/18: total 2.2M -rw-r----- 1 spark users 1.1M Apr 7 16:41 temp_local_66c0df48-5d79-448b-8989-84ce1a5507d0 -rw-r----- 1 spark users 1.1M Apr 7 16:39 temp_local_f6870214-bfd5-47b2-b0b9-37194b55761b /data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/1a: total 1.1M -rw-r----- 1 spark users 1.1M Apr 7 16:40 temp_local_ba1712d2-0eb8-4833-9fa6-a87ee670826c /data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/1b: total 1.1M -rw-r----- 1 spark users 1.1M Apr 7 16:41 temp_local_1d1df5b7-846c-4bcd-a9de-c328e50e62db /data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/1f: total 1.1M -rw-r----- 1 spark users 1.1M Apr 7 16:41 temp_local_38c6a144-b588-49b1-b0f0-b91b31c2e85f /data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/20: total 1.1M -rw-r----- 1 spark users 1.1M Apr 7 16:40 temp_local_d816a301-7520-4b6e-8866-bc445f04f0c9 /data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/24: total 1.1M -rw-r----- 1 spark users 1.1M Apr 7 16:40 temp_local_7a3619cf-20e1-4815-8b1f-faeefde40d73 /data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/2e: total 2.2M -rw-r----- 1 spark users 1.1M Apr 7 16:39 temp_local_1e366a65-ced7-4b17-a085-5f873ff6dc43 -rw-r----- 1 spark users 1.1M Apr 7 16:41 temp_local_6772b815-11ee-413a-bc6c-d0dc9cbffc51 {code} The estimateSize is hugh difference with spill file size > EstimateSize is difference with spill file size > ------------------------------------------------ > > Key: SPARK-6738 > URL: https://issues.apache.org/jira/browse/SPARK-6738 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.2.0 > Reporter: Hong Shen > > ExternalAppendOnlyMap spill 2.2 GB data to disk: > {code} > 15/04/07 20:27:37 INFO collection.ExternalAppendOnlyMap: Thread 54 spilling > in-memory map of 2.2 GB to disk (61 times so far) > 15/04/07 20:27:37 INFO collection.ExternalAppendOnlyMap: > /data11/yarnenv/local/usercache/spark/appcache/application_1423737010718_40455651/spark-local-20150407202613-4e80/11/temp_local_fdb4a583-5d13-4394-bccb-e1217d5db812 > {code} > But the file size is only 2.2M. > {code} > ll -h > /data11/yarnenv/local/usercache/spark/appcache/application_1423737010718_40455651/spark-local-20150407202613-4e80/11/ > total 2.2M > -rw-r----- 1 spark users 2.2M Apr 7 20:27 > temp_local_fdb4a583-5d13-4394-bccb-e1217d5db812 > {code} > The GC log show that the jvm memory is less than 1GB. > {code} > 2015-04-07T20:27:08.023+0800: [GC 981981K->55363K(3961344K), 0.0341720 secs] > 2015-04-07T20:27:14.483+0800: [GC 987523K->53737K(3961344K), 0.0252660 secs] > 2015-04-07T20:27:20.793+0800: [GC 985897K->56370K(3961344K), 0.0606460 secs] > 2015-04-07T20:27:27.553+0800: [GC 988530K->59089K(3961344K), 0.0651840 secs] > 2015-04-07T20:27:34.067+0800: [GC 991249K->62153K(3961344K), 0.0288460 secs] > 2015-04-07T20:27:40.180+0800: [GC 994313K->61344K(3961344K), 0.0388970 secs] > 2015-04-07T20:27:46.490+0800: [GC 993504K->59915K(3961344K), 0.0235150 secs] > {code} > The estimateSize is hugh difference with spill file size -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org