[ https://issues.apache.org/jira/browse/SPARK-3546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kousuke Saruta updated SPARK-3546: ---------------------------------- Summary: InputStream of ManagedBuffer is not closed and causes running out of file descriptor (was: InputStream of ManagedBuffer does not close and causes running out of file descriptor) > InputStream of ManagedBuffer is not closed and causes running out of file > descriptor > ------------------------------------------------------------------------------------ > > Key: SPARK-3546 > URL: https://issues.apache.org/jira/browse/SPARK-3546 > Project: Spark > Issue Type: Bug > Affects Versions: 1.2.0 > Reporter: Kousuke Saruta > Priority: Critical > > If application has lots of shuffle blocks, resource leak (running out of file > descriptor) is occurred. > Following text is file descriptors of an Executor which has lots of blocks > {code} > ・・・ > lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9980 -> > /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/13/shuffle_0_340_0.data > lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9981 -> > /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/37/shuffle_0_355_0.data > lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9982 -> > /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/10/shuffle_0_370_0.data > lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9983 -> > /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/0c/shuffle_0_385_0.data > lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9984 -> > /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/0e/shuffle_0_390_0.data > lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9985 -> > /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/1d/shuffle_0_405_0.data > lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9986 -> > /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/36/shuffle_0_420_0.data > lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9987 -> > /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/1b/shuffle_0_425_0.data > lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9988 -> > /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/0b/shuffle_0_430_0.data > lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9989 -> > /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/0d/shuffle_0_450_0.data > lr-x------ 1 yarn yarn 64 9月 16 20:11 2014 999 -> > /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/28/shuffle_1_630_0.data > lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9990 -> > /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/29/shuffle_0_465_0.data > lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9991 -> > /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/14/shuffle_0_495_0.data > lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9992 -> > /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/3c/shuffle_0_525_0.data > lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9993 -> > /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/2a/shuffle_0_530_0.data > lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9994 -> > /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/05/shuffle_0_535_0.data > lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9995 -> > /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/15/shuffle_0_540_0.data > lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9996 -> > /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/2c/shuffle_0_550_0.data > lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9997 -> > /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/13/shuffle_0_560_0.data > lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9998 -> > /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/12/shuffle_0_570_0.data > lr-x------ 1 yarn yarn 64 9月 16 20:27 2014 9999 -> > /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/2f/shuffle_0_580_0.data > {code} > This is caused by not closing InputStream generated by ManagedBuffer. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org