The line of code which I highlighted in the screenshot is within the spark source code. Spark implements sort-based shuffle implementation and the spilled files are merged using the merge sort.
Here is the link https://issues.apache.org/jira/secure/attachment/12655884/Sort-basedshuffledesign.pdf which would convey the same. On Wed, Jan 6, 2016 at 8:19 PM, Annabel Melongo <melongo_anna...@yahoo.com> wrote: > Priya, > > It would be helpful if you put the entire trace log along with your code > to help determine the root cause of the error. > > Thanks > > > On Wednesday, January 6, 2016 4:00 AM, Priya Ch < > learnings.chitt...@gmail.com> wrote: > > > Running 'lsof' will let us know the open files but how do we come to know > the root cause behind opening too many files. > > Thanks, > Padma CH > > On Wed, Jan 6, 2016 at 8:39 AM, Hamel Kothari <hamelkoth...@gmail.com> > wrote: > > The "Too Many Files" part of the exception is just indicative of the fact > that when that call was made, too many files were already open. It doesn't > necessarily mean that that line is the source of all of the open files, > that's just the point at which it hit its limit. > > What I would recommend is to try to run this code again and use "lsof" on > one of the spark executors (perhaps run it in a for loop, writing the > output to separate files) until it fails and see which files are being > opened, if there's anything that seems to be taking up a clear majority > that might key you in on the culprit. > > On Tue, Jan 5, 2016 at 9:48 PM Priya Ch <learnings.chitt...@gmail.com> > wrote: > > Yes, the fileinputstream is closed. May be i didn't show in the screen > shot . > > As spark implements, sort-based shuffle, there is a parameter called > maximum merge factor which decides the number of files that can be merged > at once and this avoids too many open files. I am suspecting that it is > something related to this. > > Can someone confirm on this ? > > On Tue, Jan 5, 2016 at 11:19 PM, Annabel Melongo < > melongo_anna...@yahoo.com> wrote: > > Vijay, > > Are you closing the fileinputstream at the end of each loop ( in.close())? > My guess is those streams aren't close and thus the "too many open files" > exception. > > > On Tuesday, January 5, 2016 8:03 AM, Priya Ch < > learnings.chitt...@gmail.com> wrote: > > > Can some one throw light on this ? > > Regards, > Padma Ch > > On Mon, Dec 28, 2015 at 3:59 PM, Priya Ch <learnings.chitt...@gmail.com> > wrote: > > Chris, we are using spark 1.3.0 version. we have not set > spark.streaming.concurrentJobs > this parameter. It takes the default value. > > Vijay, > > From the tack trace it is evident that > org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$1.apply$mcVI$sp(ExternalSorter.scala:730) > is throwing the exception. I opened the spark source code and visited the > line which is throwing this exception i.e > > [image: Inline image 1] > > The lie which is marked in red is throwing the exception. The file is > ExternalSorter.scala in org.apache.spark.util.collection package. > > i went through the following blog > http://blog.cloudera.com/blog/2015/01/improving-sort-performance-in-apache-spark-its-a-double/ > and understood that there is merge factor which decide the number of > on-disk files that could be merged. Is it some way related to this ? > > Regards, > Padma CH > > On Fri, Dec 25, 2015 at 7:51 PM, Chris Fregly <ch...@fregly.com> wrote: > > and which version of Spark/Spark Streaming are you using? > > are you explicitly setting the spark.streaming.concurrentJobs to > something larger than the default of 1? > > if so, please try setting that back to 1 and see if the problem still > exists. > > this is a dangerous parameter to modify from the default - which is why > it's not well-documented. > > > On Wed, Dec 23, 2015 at 8:23 AM, Vijay Gharge <vijay.gha...@gmail.com> > wrote: > > Few indicators - > > 1) during execution time - check total number of open files using lsof > command. Need root permissions. If it is cluster not sure much ! > 2) which exact line in the code is triggering this error ? Can you paste > that snippet ? > > > On Wednesday 23 December 2015, Priya Ch <learnings.chitt...@gmail.com> > wrote: > > ulimit -n 65000 > > fs.file-max = 65000 ( in etc/sysctl.conf file) > > Thanks, > Padma Ch > > On Tue, Dec 22, 2015 at 6:47 PM, Yash Sharma <yash...@gmail.com> wrote: > > Could you share the ulimit for your setup please ? > - Thanks, via mobile, excuse brevity. > On Dec 22, 2015 6:39 PM, "Priya Ch" <learnings.chitt...@gmail.com> wrote: > > Jakob, > > Increased the settings like fs.file-max in /etc/sysctl.conf and also > increased user limit in /etc/security/limits.conf. But still see the same > issue. > > On Fri, Dec 18, 2015 at 12:54 AM, Jakob Odersky <joder...@gmail.com> > wrote: > > It might be a good idea to see how many files are open and try increasing > the open file limit (this is done on an os level). In some application > use-cases it is actually a legitimate need. > > If that doesn't help, make sure you close any unused files and streams in > your code. It will also be easier to help diagnose the issue if you send an > error-reproducing snippet. > > > > > > -- > Regards, > Vijay Gharge > > > > > > > -- > > *Chris Fregly* > Principal Data Solutions Engineer > IBM Spark Technology Center, San Francisco, CA > http://spark.tc | http://advancedspark.com > > > > > > > > > >
