[
https://issues.apache.org/jira/browse/SPARK-9827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14692441#comment-14692441
]
Apache Spark commented on SPARK-9827:
-------------------------------------
User 'davies' has created a pull request for this issue:
https://github.com/apache/spark/pull/8116
> Too many open files in TungstenExchange
> ---------------------------------------
>
> Key: SPARK-9827
> URL: https://issues.apache.org/jira/browse/SPARK-9827
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.0
> Reporter: Davies Liu
> Assignee: Josh Rosen
> Priority: Blocker
>
> When run q19 on TPCDS (scale=5) dataset with 8G memory, it open 10k shuffle
> files, crash many things (even Chrome).
> {code}
> davies@localhost:~/work/spark$ jps
> 95385 Jps
> 95316 SparkSubmit
> davies@localhost:~/work/spark$ lsof -p 95316 | wc -l
> 9827
> davies@localhost:~/work/spark$ lsof -p 95316 | tail
> java 95316 davies 9772r REG 1,2 9522 97350739
> /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/2a/shuffle_0_112_0.data
> java 95316 davies 9773r REG 1,2 8449 97351388
> /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/1a/shuffle_0_116_0.data
> java 95316 davies 9774r REG 1,2 8200 97351134
> /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/09/shuffle_0_113_0.data
> java 95316 davies 9775r REG 1,2 8057 97351941
> /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/05/shuffle_0_117_0.data
> java 95316 davies 9776r REG 1,2 8565 97351133
> /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/18/shuffle_0_114_0.data
> java 95316 davies 9777r REG 1,2 8185 97351942
> /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/1c/shuffle_0_118_0.data
> java 95316 davies 9778r REG 1,2 8865 97351135
> /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/07/shuffle_0_115_0.data
> java 95316 davies 9779r REG 1,2 8255 97351987
> /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/3d/shuffle_0_119_0.data
> java 95316 davies 9780r REG 1,2 8449 97351388
> /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/1a/shuffle_0_116_0.data
> java 95316 davies 9781r REG 1,2 9105 97352148
> /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/13/shuffle_0_120_0.data
> davies@localhost:~/work/spark$ ls -l
> /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-71afa3af-f2a5-4b72-8b2d-45aa70ff7466//3a/
> total 68
> -rw-r--r-- 1 davies staff 8272 Aug 11 09:57 shuffle_0_105_0.data
> -rw-r--r-- 1 davies staff 1608 Aug 11 09:57 shuffle_0_109_0.index
> -rw-r--r-- 1 davies staff 8414 Aug 11 09:57 shuffle_0_127_0.data
> -rw-r--r-- 1 davies staff 8368 Aug 11 09:57 shuffle_0_149_0.data
> -rw-r--r-- 1 davies staff 1608 Aug 11 09:57 shuffle_0_40_0.index
> -rw-r--r-- 1 davies staff 1608 Aug 11 09:57 shuffle_0_62_0.index
> -rw-r--r-- 1 davies staff 7965 Aug 11 09:57 shuffle_0_6_0.data
> -rw-r--r-- 1 davies staff 8419 Aug 11 09:57 shuffle_0_80_0.data
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
