: Failed to create file system watcher service: User limit of inotify instances reached or too many open files

of issues. Caused by: java.lang.RuntimeException: Failed to create file system watcher service: User limit of inotify instances reached or too many open files Caused by: java.io.IOException: User limit of inotify instances reached or too many open files Failed to create file system watcher service: User limit o

Re: [SparkSQL] too many open files although ulimit set to 1048576

I think your sets not works try add `ulimit -n 10240 ` in spark-env.sh -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkSQL-too-many-open-files-although-ulimit-set-to-1048576-tp28490p28491.html Sent from the Apache Spark User List mailing list archive

Re: Too many open files, why changing ulimit not effecting?

>> https://easyengine.io/tutorials/linux/increase-open-files-limit/ >> >> On Fri, Feb 5, 2016 at 1:42 AM, Mohamed Nadjib MAMI <m...@iai.uni-bonn.de >> > wrote: >> >>> Hello all, >>> >>> I'm getting the famous *java.io.FileNotFoundException:

Re: Too many open files, why changing ulimit not effecting?

AM, Mohamed Nadjib MAMI <m...@iai.uni-bonn.de> > wrote: > >> Hello all, >> >> I'm getting the famous *java.io.FileNotFoundException: ... (Too many >> open files) *exception. What seemed to have helped people out, it >> haven't for me. I tried to set t

Too many open files, why changing ulimit not effecting?

Hello all, I'm getting the famous /java.io.FileNotFoundException: ... (Too many open files) /exception. What seemed to have helped people out, it haven't for me. I tried to set the ulimit via the command line /"ulimit -n"/, then I tried to add the following lines to /&quo

Re: Too many open files, why changing ulimit not effecting?

;m...@iai.uni-bonn.de> wrote: > Hello all, > > I'm getting the famous *java.io.FileNotFoundException: ... (Too many open > files) *exception. What seemed to have helped people out, it haven't for > me. I tried to set the ulimit via the command line *"ulimit -n"*,

Re: java.io.FileNotFoundException(Too many open files) in Spark streaming

t; >> melongo_anna...@yahoo.com> wrote: >> >>> Vijay, >>> >>> Are you closing the fileinputstream at the end of each loop ( >>> in.close())? My guess is those streams aren't close and thus the "too many >>> open files" exception. >

Re: java.io.FileNotFoundException(Too many open files) in Spark streaming

n't show in the screen > shot . > > As spark implements, sort-based shuffle, there is a parameter called > maximum merge factor which decides the number of files that can be merged > at once and this avoids too many open files. I am suspecting that it is > something re

Re: java.io.FileNotFoundException(Too many open files) in Spark streaming

Vijay, Are you closing the fileinputstream at the end of each loop ( in.close())? My guess is those streams aren't close and thus the "too many open files" exception. On Tuesday, January 5, 2016 8:03 AM, Priya Ch <learnings.chitt...@gmail.com> wrote: Can some

Re: java.io.FileNotFoundException(Too many open files) in Spark streaming

Yes, the fileinputstream is closed. May be i didn't show in the screen shot . As spark implements, sort-based shuffle, there is a parameter called maximum merge factor which decides the number of files that can be merged at once and this avoids too many open files. I am suspecting

Re: java.io.FileNotFoundException(Too many open files) in Spark streaming

Can some one throw light on this ? Regards, Padma Ch On Mon, Dec 28, 2015 at 3:59 PM, Priya Ch wrote: > Chris, we are using spark 1.3.0 version. we have not set > spark.streaming.concurrentJobs > this parameter. It takes the default value. > > Vijay, > > From

Re: java.io.FileNotFoundException(Too many open files) in Spark streaming

Chris, we are using spark 1.3.0 version. we have not set spark.streaming.concurrentJobs this parameter. It takes the default value. Vijay, From the tack trace it is evident that

Re: java.io.FileNotFoundException(Too many open files) in Spark streaming

and which version of Spark/Spark Streaming are you using? are you explicitly setting the spark.streaming.concurrentJobs to something larger than the default of 1? if so, please try setting that back to 1 and see if the problem still exists. this is a dangerous parameter to modify from the

java.io.FileNotFoundException(Too many open files) in Spark streaming

Few indicators - 1) during execution time - check total number of open files using lsof command. Need root permissions. If it is cluster not sure much ! 2) which exact line in the code is triggering this error ? Can you paste that snippet ? On Wednesday 23 December 2015, Priya Ch

Re: java.io.FileNotFoundException(Too many open files) in Spark streaming

ulimit -n 65000 fs.file-max = 65000 ( in etc/sysctl.conf file) Thanks, Padma Ch On Tue, Dec 22, 2015 at 6:47 PM, Yash Sharma wrote: > Could you share the ulimit for your setup please ? > > - Thanks, via mobile, excuse brevity. > On Dec 22, 2015 6:39 PM, "Priya Ch"

Re: java.io.FileNotFoundException(Too many open files) in Spark streaming

Jakob, Increased the settings like fs.file-max in /etc/sysctl.conf and also increased user limit in /etc/security/limits.conf. But still see the same issue. On Fri, Dec 18, 2015 at 12:54 AM, Jakob Odersky wrote: > It might be a good idea to see how many files are open

java.io.FileNotFoundException(Too many open files) in Spark streaming

Hi All, When running streaming application, I am seeing the below error: java.io.FileNotFoundException: /data1/yarn/nm/usercache/root/appcache/application_1450172646510_0004/blockmgr-a81f42cd-6b52-4704-83f3-2cfc12a11b86/02/temp_shuffle_589ddccf-d436-4d2c-9935-e5f8c137b54b (Too many open

Re: java.io.FileNotFoundException(Too many open files) in Spark streaming

It might be a good idea to see how many files are open and try increasing the open file limit (this is done on an os level). In some application use-cases it is actually a legitimate need. If that doesn't help, make sure you close any unused files and streams in your code. It will also be easier

Re: "Too many open files" exception on reduceByKey

It turns out the mesos can overwrite the OS ulimit -n setting. So we have increased the mesos slave ulimit -n setting. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Too-many-open-files-exception-on-reduceByKey-tp2462p25019.html Sent from the Apache Spark

Re: "Too many open files" exception on reduceByKey

laves/20150512-215537-2165010442-5050-1730-S5/frameworks/20150825-175705-2165010442-5050-13705-0338/executors/0/runs/19342849-d076-483c-88da-747896e19b93/./spark-6efa2dcd-aea7-478e-9fa9-6e0973578eb4/blockmgr-33b1e093-6dd6-4462-938c-2597516272a9/27/shuffle_535_2_0.index (Too many open files)         a

Re: "Too many open files" exception on reduceByKey

/19342849-d076-483c-88da-747896e19b93/./spark-6efa2dcd-aea7-478e-9fa9-6e0973578eb4/blockmgr-33b1e093-6dd6-4462-938c-2597516272a9/27/shuffle_535_2_0.index (Too many open files) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.(FileOutputStream.java:221

Re: "Too many open files" exception on reduceByKey

rs/0/runs/19342849-d076-483c-88da-747896e19b93/./spark-6efa2dcd-aea7-478e-9fa9-6e0973578eb4/blockmgr-33b1e093-6dd6-4462-938c-2597516272a9/27/shuffle_535_2_0.index > (Too many open files) > at java.io.FileOutputStream.open(Native Method) > at java.io.FileOutputStream.(File

Re: Too many open files issue

(tuple => { // send data to kafka } } val count = rdd.count log 'Processed $count records for $directory' log 'Finished directory processor for $directory' } This results in these log lines until the "Too many open files in system" errors star

Re: Too many open files issue

On 31 Aug 2015, at 19:49, Sigurd Knippenberg > wrote: I know I can adjust the max open files allowed by the OS but I'd rather fix the underlaying issue. bumping up the OS handle limits is step #1 of installing a hadoop cluster

Re: Too many open files issue

Here is the code in which NewHadoopRDD register close handler and be called when the task is completed ( https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala#L136 ). >From my understanding, possibly the reason is that this `foreach` code in your

Re: Too many open files issue

ah, now that does sound suspicious... On 2 Sep 2015, at 14:09, Sigurd Knippenberg > wrote: Yep. I know. It's was set to 32K when I ran this test. If I bump it to 64K the issue goes away. It still doesn't make sense to me that the Spark job

Re: Too many open files issue

Yep. I know. It's was set to 32K when I ran this test. If I bump it to 64K the issue goes away. It still doesn't make sense to me that the Spark job doesn't release its file handles until the end of the job instead of doing that while my loop iterates. Sigurd On Wed, Sep 2, 2015 at 4:33 AM,

Too many open files issue

I am running in a 'too many open files' issue and before I posted this I have searched on the web to see if anyone had a solution already to my particular problem but I did not see anything that helped. I know I can adjust the max open files allowed by the OS but I'd rather fix the underlaying

Re: Too many open files

.index (Too many open files) .. .. 15/07/29 10:37:01 INFO Executor: Executor is trying to kill task 9.0 in stage 11.0 (TID 306) org.apache.spark.SparkException: Job aborted due to stage failure: Task 20 in stage 11.0 failed 1 times, most recent failure: Lost task 20.0 in stage 11.0 (TID 317

RE: Too many open files

[mailto:yuzhih...@gmail.com] Sent: Wednesday, July 29, 2015 12:59 PM To: Ellafi, Saif A. Cc: user@spark.apache.org Subject: Re: Too many open files Please increase limit for open files: http://stackoverflow.com/questions/34588/how-do-i-change-the-number-of-open-files-limit-in-linux On Jul 29, 2015

Re: Too many open files

): java.io.FileNotFoundException: /tmp/spark-71109b28-0f89-4e07-a521-5ff0a943472a/blockmgr-eda0751d-fd21-4229-93b0-2ee2546edf5a/1b/temp_shuffle_a3a9815a-677a-4342-94a2-1e083d758bcc (Too many open files) my fs is ext4 and currently ulist –n is 1024 Thanks Saif

RE: Too many open files

consequences. 2. I will try to overlook admin access and see if I can get anything with only user rights From: Ted Yu [mailto:yuzhih...@gmail.com] Sent: Wednesday, July 29, 2015 12:59 PM To: Ellafi, Saif A. Cc: user@spark.apache.org Subject: Re: Too many open files Please increase limit for open

Re: Too many open files when using Spark to consume messages from Kafka

, Apr 29, 2015 at 4:06 PM, Tathagata Das t...@databricks.com wrote: Is the function ingestToMysql running on the driver or on the executors? Accordingly you can try debugging while running in a distributed manner, with and without calling the function. If you dont get too many open files

Re: Too many open files when using Spark to consume messages from Kafka

? Accordingly you can try debugging while running in a distributed manner, with and without calling the function. If you dont get too many open files without calling ingestToMysql(), the problem is likely to be in ingestToMysql(). If you get the problem even without calling ingestToMysql

Re: Too many open files when using Spark to consume messages from Kafka

wrote: Is the function ingestToMysql running on the driver or on the executors? Accordingly you can try debugging while running in a distributed manner, with and without calling the function. If you dont get too many open files without calling ingestToMysql(), the problem is likely

Re: Too many open files when using Spark to consume messages from Kafka

org.apache.spark.SparkException: ArrayBuffer(java.io.IOException: Too many open files, java.io.IOException: Too many open files, java.io.IOException: Too many open files, java.io.IOException: Too many open files, java.io.IOException: Too many open files

Re: Too many open files when using Spark to consume messages from Kafka

After around 12 hours, I got the following error messages in Spark log: 15/04/29 20:18:10 ERROR JobScheduler: Error generating jobs for time 143033869 ms org.apache.spark.SparkException: ArrayBuffer(java.io.IOException: Too many open files, java.io.IOException: Too many open files

Re: Too many open files when using Spark to consume messages from Kafka

/DirectKafkaWordCount.scala After around 12 hours, I got the following error messages in Spark log: 15/04/29 20:18:10 ERROR JobScheduler: Error generating jobs for time 143033869 ms org.apache.spark.SparkException: ArrayBuffer(java.io.IOException: Too many open files, java.io.IOException

Too many open files when using Spark to consume messages from Kafka

open files, java.io.IOException: Too many open files, java.io.IOException: Too many open files, java.io.IOException: Too many open files, java.io.IOException: Too many open files) at org.apache.spark.streaming.kafka.DirectKafkaInputDStream.latestLeaderOffsets(DirectKafkaInputDStream.scala

Re: Too many open files when using Spark to consume messages from Kafka

Is the function ingestToMysql running on the driver or on the executors? Accordingly you can try debugging while running in a distributed manner, with and without calling the function. If you dont get too many open files without calling ingestToMysql(), the problem is likely

Re: Too many open files when using Spark to consume messages from Kafka

running on the driver or on the executors? Accordingly you can try debugging while running in a distributed manner, with and without calling the function. If you dont get too many open files without calling ingestToMysql(), the problem is likely to be in ingestToMysql(). If you get

Re: Too many open files when using Spark to consume messages from Kafka

try debugging while running in a distributed manner, with and without calling the function. If you dont get too many open files without calling ingestToMysql(), the problem is likely to be in ingestToMysql(). If you get the problem even without calling ingestToMysql(), then the problem may

Re: Too many open files when using Spark to consume messages from Kafka

143033869 ms org.apache.spark.SparkException: ArrayBuffer(java.io.IOException: Too many open files, java.io.IOException: Too many open files, java.io.IOException: Too many open files, java.io.IOException: Too many open files, java.io.IOException: Too many open files

Re: Too many open files

. I have fixed spark.shuffle.manager to SORT. In /etc/secucity/limits.conf set the next values: * softnofile 100 * hardnofile 100 In spark-env.sh set ulimit -n 100 I've restarted the spark service and it continues crashing (Too many open files

Re: Too many open files

to SORT. In /etc/secucity/limits.conf set the next values: * softnofile 100 * hardnofile 100 In spark-env.sh set ulimit -n 100 I've restarted the spark service and it continues crashing (Too many open files) How can I resolve? I'm executing

Too many open files

and it continues crashing (Too many open files) How can I resolve? I'm executing Spark 1.2.0 in Cloudera 5.3.2 java.io.FileNotFoundException: /tmp/spark-local-20150330115312-37a7/2f/temp_shuffle_c4ba5bce-c516-4a2a-9e40-56121eb84a8c (Too many open files) at java.io.FileOutputStream.open(Native

Re: Too many open files

fixed spark.shuffle.manager to SORT. In /etc/secucity/limits.conf set the next values: * softnofile 100 * hardnofile 100 In spark-env.sh set ulimit -n 100 I've restarted the spark service and it continues crashing (Too many open files) How can

Re: Too many open files

to SORT. In /etc/secucity/limits.conf set the next values: * softnofile 100 * hardnofile 100 In spark-env.sh set ulimit -n 100 I've restarted the spark service and it continues crashing (Too many open files) How can I resolve? I'm executing Spark

RE: com.esotericsoftware.kryo.KryoException: java.io.IOException: File too large vs FileNotFoundException (Too many open files) on spark 1.2.1

@spark.apache.org Subject: RE: com.esotericsoftware.kryo.KryoException: java.io.IOException: File too large vs FileNotFoundException (Too many open files) on spark 1.2.1 Do you think the ulimit for the user running Spark on your nodes? Can you run ulimit -a under the user who is running spark

RE: com.esotericsoftware.kryo.KryoException: java.io.IOException: File too large vs FileNotFoundException (Too many open files) on spark 1.2.1

: com.esotericsoftware.kryo.KryoException: java.io.IOException: File too large vs FileNotFoundException (Too many open files) on spark 1.2.1 Date: Fri, 20 Mar 2015 15:28:26 -0400 Hi All, I try to run a simple sort by on 1.2.1. And it always give me below two errors: 1, 15/03/20 17:48:29 WARN TaskSetManager: Lost

com.esotericsoftware.kryo.KryoException: java.io.IOException: File too large vs FileNotFoundException (Too many open files) on spark 1.2.1

-58f72d53-8afc-41c2-ad6 b-e96b479b51f5/spark-fde6da79-0b51-4087-8234-2c07ac6d7586/spark-dd7d6682-19d d-4c66-8aa5-d8a4abe88ca2/16/temp_shuffle_756b59df-ef3a-4680-b3ac-437b5326782 6 (Too many open files) And then I switch to: conf.set(spark.shuffle.consolidateFiles, true) .set(spark.shuffle.manager

Re: com.esotericsoftware.kryo.KryoException: java.io.IOException: File too large vs FileNotFoundException (Too many open files) on spark 1.2.1

-d8a4abe88ca2/16/temp_shuffle_756b59df-ef3a-4680-b3ac-437b53267826 (Too many open files) And then I switch to: conf.set(spark.shuffle.consolidateFiles, true) .set(spark.shuffle.manager, SORT) Then I get the error: Exception in thread main org.apache.spark.SparkException: Job aborted

Re: Too many open files

Hi, I am having the same problem reported by Michael. I am trying to open 30 files. ulimit -n shows the limit is 1024. So I am not sure why the program is failing with Too many open files error. The total size of all the 30 files is 230 GB. I am running the job on a cluster with 10 nodes, each

Re: Too many open files

the limit is 1024. So I am not sure why the program is failing with Too many open files error. The total size of all the 30 files is 230 GB. I am running the job on a cluster with 10 nodes, each having 16 GB. The error appears to be happening at the distinct() stage. Here is my program

Sorting data large data- too many open files exception

Hello, I currently have a task always failing with java.io.FileNotFoundException: [...]/shuffle_0_257_2155 (Too many open files) when I run sorting operations such as distinct, sortByKey, or reduceByKey on a large number of partitions. Im working with 365 GB of data which is being split

Re: Too many open files exception on reduceByKey

file, stored in an HDFS cluster running on the same nodes as the Spark cluster, the job crashes with the message, Too many open files. My question is, why are so many files being created, and is there a way to configure the Spark context to avoid spawning that many files? I am already setting

Re: [incubating-0.9.0] Too Many Open Files on Workers

Hello Andy, This is a problem we have seen in using the CQL Java driver under heavy ready loads where it is using NIO and is waiting on many pending responses which causes to many open sockets and hence too many open files. Are you by any chance using async queries? I am the maintainer