Please see the last 3 posts on this thread: http://search-hadoop.com/m/q3RTtTorTf2o3UGK1&subj=Re+spark+ec2+vs+EMR
FYI On Thu, Feb 18, 2016 at 6:25 AM, Teng Qiu <teng...@gmail.com> wrote: > EMR is great, but I'm curiosity how are you dealing with security settings > with EMR, only whitelisting some IP range with security group setting is > really too weak. > > are there really many production system are using EMR? for me, i feel > using EMR means everyone in my IP range (for some ISP it may be the whole > town...) is able to see my spark web UI or use my running zepplin notebook > if they do some port scanning... > > 2016-02-18 15:04 GMT+01:00 Gourav Sengupta <gourav.sengu...@gmail.com>: > >> Hi, >> >> Just out of sheet curiosity why are you not using EMR to start your SPARK >> cluster? >> >> >> Regards, >> Gourav >> >> On Thu, Feb 18, 2016 at 12:23 PM, Ted Yu <yuzhih...@gmail.com> wrote: >> >>> Have you seen this ? >>> >>> HADOOP-10988 >>> >>> Cheers >>> >>> On Thu, Feb 18, 2016 at 3:39 AM, James Hammerton <ja...@gluru.co> wrote: >>> >>>> HI, >>>> >>>> I am seeing warnings like this in the logs when I run Spark jobs: >>>> >>>> OpenJDK 64-Bit Server VM warning: You have loaded library >>>> /root/ephemeral-hdfs/lib/native/libhadoop.so.1.0.0 which might have >>>> disabled stack guard. The VM will try to fix the stack guard now. >>>> It's highly recommended that you fix the library with 'execstack -c >>>> <libfile>', or link it with '-z noexecstack'. >>>> >>>> >>>> I used spark-ec2 to launch the cluster with the default AMI, Spark >>>> 1.5.2, hadoop major version 2.4. I altered the jdk to be openjdk 8 as I'd >>>> written some jobs in Java 8. The 6 workers nodes are m4.2xlarge and master >>>> is m4.large. >>>> >>>> Could this contribute to any problems running the jobs? >>>> >>>> Regards, >>>> >>>> James >>>> >>> >>> >> >
