bq. and *"session required pam_limits.so"*. What was the second file you modified ?
Did you make the change on all the nodes ? Please see the verification step in https://easyengine.io/tutorials/linux/increase-open-files-limit/ On Fri, Feb 5, 2016 at 1:42 AM, Mohamed Nadjib MAMI <m...@iai.uni-bonn.de> wrote: > Hello all, > > I'm getting the famous *java.io.FileNotFoundException: ... (Too many open > files) *exception. What seemed to have helped people out, it haven't for > me. I tried to set the ulimit via the command line *"ulimit -n"*, then I > tried to add the following lines to *"/etc/security/limits.conf"* file: > > ** - nofile 1000000* > *root soft nofile 1000000* > *root hard nofile 1000000* > *hduser soft nofile 1000000* > *hduser hard nofile 1000000* > > ...then I added this line *"session required pam_limits.so"* to the two > files* "/etc/pam.d/common-session"* and *"session required pam_limits.so"*. > The I logged-out/logged-in. First, I tried only the first line (** - > nofile 1000000**)*, then added the 2nd and the 3rd (root...), then added > the last two lines (hduser...), no effect. Weirdly enough, when I check > with the command *"ulimit -n"* it returns the correct value of 1000000. > > I then added *"ulimit -n 1000000"* to *"spark-env.sh"* in the master and > in each of my workers, no effect. > > What else could it be besides changing the ulimit setting? if it's only > that, what could cause Spark to ignore it? > > I'll appreciate any help in advance. > > -- > *PhD Student - EIS Group - Bonn University, Germany.* > *+49 1575 8482232 <%2B49%201575%208482232>* > >
