Launching with m3.2xlarge instances: /mnt and /mnt2 mounted on 7gb drive

2014-07-20 Thread Chris DuBois
Using the spark-ec2 script with m3.2xlarge instances seems to not have /mnt and /mnt2 pointing to the 80gb SSDs that come with that instance. Does anybody know whether extra steps are required when using this instance type? Thanks, Chris

Re: Launching with m3.2xlarge instances: /mnt and /mnt2 mounted on 7gb drive

2014-07-20 Thread Chris DuBois
I pulled the latest last night. I'm on commit 4da01e3. On Sun, Jul 20, 2014 at 2:08 PM, Matei Zaharia matei.zaha...@gmail.com wrote: Is this with the 1.0.0 scripts? I believe it's fixed in 1.0.1. Matei On Jul 20, 2014, at 1:22 AM, Chris DuBois chris.dub...@gmail.com wrote: Using

Re: Error: No space left on device

2014-07-17 Thread Chris DuBois
and using the default settings (don't set anything in SparkConf). Submit the application with --driver-memory 20g. The running times are slower than what I remember, but it depends on the instance type. Best, Xiangrui On Wed, Jul 16, 2014 at 10:18 PM, Chris DuBois chris.dub...@gmail.com wrote

Error: No space left on device

2014-07-16 Thread Chris DuBois
Hi all, I am encountering the following error: INFO scheduler.TaskSetManager: Loss was due to java.io.IOException: No space left on device [duplicate 4] For each slave, df -h looks roughtly like this, which makes the above error surprising. FilesystemSize Used Avail Use% Mounted

Re: Error: No space left on device

2014-07-16 Thread Chris DuBois
(df -i). The assembly build may create many small files. -Xiangrui On Tue, Jul 15, 2014 at 11:35 PM, Chris DuBois chris.dub...@gmail.com wrote: Hi all, I am encountering the following error: INFO scheduler.TaskSetManager: Loss was due to java.io.IOException: No space left on device

Re: Error: No space left on device

2014-07-16 Thread Chris DuBois
, Chris DuBois chris.dub...@gmail.com wrote: Hi all, I am encountering the following error: INFO scheduler.TaskSetManager: Loss was due to java.io.IOException: No space left on device [duplicate 4] For each slave, df -h looks roughtly like this, which makes the above error

Re: Error: No space left on device

2014-07-16 Thread Chris DuBois
this setting as the default value. Best, Xiangrui On Wed, Jul 16, 2014 at 12:02 AM, Chris DuBois chris.dub...@gmail.com wrote: Thanks for the quick responses! I used your final -Dspark.local.dir suggestion, but I see this during the initialization of the application: 14/07/16 06:56

Re: Error: No space left on device

2014-07-16 Thread Chris DuBois
you recommend? Chris On Wed, Jul 16, 2014 at 1:09 AM, Chris DuBois chris.dub...@gmail.com wrote: Hi Xiangrui, Here is the result on the master node: $ df -i FilesystemInodes IUsed IFree IUse% Mounted on /dev/xvda1524288 273997 250291 53% / tmpfs

Re: Trouble with spark-ec2 script: --ebs-vol-size

2014-07-16 Thread Chris DuBois
Hi Ben, It worked for me, but only when using the default region. Using --region=us-west-2 resulted in errors about security groups. Chris On Wed, Jul 16, 2014 at 8:53 AM, Ben Horner ben.hor...@atigeo.com wrote: please add From: Ben Horner [via Apache Spark User List] [hidden email]

Re: Error: No space left on device

2014-07-16 Thread Chris DuBois
/spark and /mnt2/spark as the default local directory because they are local hard drives. Did your last run of ALS on MovieLens 10M-100K with the default settings succeed? -Xiangrui On Wed, Jul 16, 2014 at 8:00 AM, Chris DuBois chris.dub...@gmail.com wrote: Hi Xiangrui, I accidentally did