I'm fairly new to Spark. The documentation suggests using the spark-ec2 script to launch clusters in AWS, hence I used it.
Would EMR offer any advantage? Regards, James On 18 February 2016 at 14:04, Gourav Sengupta <gourav.sengu...@gmail.com> wrote: > Hi, > > Just out of sheet curiosity why are you not using EMR to start your SPARK > cluster? > > > Regards, > Gourav > > On Thu, Feb 18, 2016 at 12:23 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> Have you seen this ? >> >> HADOOP-10988 >> >> Cheers >> >> On Thu, Feb 18, 2016 at 3:39 AM, James Hammerton <ja...@gluru.co> wrote: >> >>> HI, >>> >>> I am seeing warnings like this in the logs when I run Spark jobs: >>> >>> OpenJDK 64-Bit Server VM warning: You have loaded library >>> /root/ephemeral-hdfs/lib/native/libhadoop.so.1.0.0 which might have >>> disabled stack guard. The VM will try to fix the stack guard now. >>> It's highly recommended that you fix the library with 'execstack -c >>> <libfile>', or link it with '-z noexecstack'. >>> >>> >>> I used spark-ec2 to launch the cluster with the default AMI, Spark >>> 1.5.2, hadoop major version 2.4. I altered the jdk to be openjdk 8 as I'd >>> written some jobs in Java 8. The 6 workers nodes are m4.2xlarge and master >>> is m4.large. >>> >>> Could this contribute to any problems running the jobs? >>> >>> Regards, >>> >>> James >>> >> >> >
