Thanks Gen. How can I check if /dev/sdc is well mounted or not? In general,
the problem shows up when I submit the second or third job. The first job I
submit most likely will succeed.
Ey-Chih Chow
Date: Sun, 8 Feb 2015 18:18:03 +0100
Subject: Re: no space left at worker node
From: gen.tan...@gmail.com
To: eyc...@hotmail.com
CC: user@spark.apache.org
Hi,
In fact, /dev/sdb is /dev/xvdb. It seems that there is no problem about double
mount. However, there is no information about /mnt2. You should check whether
/dev/sdc is well mounted or not.The reply of Micheal is good solution about
this type of problem. You can check his site.
CheersGen
On Sun, Feb 8, 2015 at 5:53 PM, ey-chih chow <eyc...@hotmail.com> wrote:
Gen,
Thanks for your information. The content of /etc/fstab at the worker node
(r3.large) is:
#LABEL=/ / ext4 defaults,noatime 1 1tmpfs /dev/shm
tmpfs defaults 0 0devpts /dev/pts devpts gid=5,mode=620 0
0sysfs /sys sysfs defaults 0 0proc /proc
proc defaults 0 0/dev/sdb /mnt auto
defaults,noatime,nodiratime,comment=cloudconfig 0 0/dev/sdc /mnt2
auto defaults,noatime,nodiratime,comment=cloudconfig 0 0
There is no entry of /dev/xvdb.
Ey-Chih Chow
Date: Sun, 8 Feb 2015 12:09:37 +0100
Subject: Re: no space left at worker node
From: gen.tan...@gmail.com
To: eyc...@hotmail.com
CC: user@spark.apache.org
Hi,
I fact, I met this problem before. it is a bug of AWS. Which type of machine do
you use?
If I guess well, you can check the file /etc/fstab. There would be a double
mount of /dev/xvdb.If yes, you should1. stop hdfs2. umount /dev/xvdb at / 3.
restart hdfs
Hope this could be helpful.CheersGen
On Sun, Feb 8, 2015 at 8:16 AM, ey-chih chow <eyc...@hotmail.com> wrote:
Hi,
I submitted a spark job to an ec2 cluster, using spark-submit. At a worker
node, there is an exception of 'no space left on device' as follows.
==========================================
15/02/08 01:53:38 ERROR logging.FileAppender: Error writing stream to file
/root/spark/work/app-20150208014557-0003/0/stdout
java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:345)
at
org.apache.spark.util.logging.FileAppender.appendToFile(FileAppender.scala:92)
at
org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:72)
at
org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39)
at
org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
at
org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
at
org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1311)
at
org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38)
===========================================
The command df showed the following information at the worker node:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda1 8256920 8256456 0 100% /
tmpfs 7752012 0 7752012 0% /dev/shm
/dev/xvdb 30963708 1729652 27661192 6% /mnt
Does anybody know how to fix this? Thanks.
Ey-Chih Chow
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/no-space-left-at-worker-node-tp21545.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
