Hi Ted/ Teng, Just read the content in the email which is very different from what the facts are: Just to want to add another point, spark-ec2 is nice to keep and improve because it allows users to any version of spark (nightly-build for example). EMR does not allow you to do that without manual process.
EMR does provide different version of SPARK to run, like currently SPARK versions 1.4.1, 1.5.0, 1.5.2 and 1.6 are all available. SPARK 1.6 was released in Amazon on Jan4, 2016 and EMR provided SPARK 1.6 in another 20 days, production ready, scalable, and integrated in AWS world. Regards, Gourav Sengupta On Thu, Feb 18, 2016 at 2:30 PM, Ted Yu <yuzhih...@gmail.com> wrote: > Please see the last 3 posts on this thread: > > http://search-hadoop.com/m/q3RTtTorTf2o3UGK1&subj=Re+spark+ec2+vs+EMR > > FYI > > On Thu, Feb 18, 2016 at 6:25 AM, Teng Qiu <teng...@gmail.com> wrote: > >> EMR is great, but I'm curiosity how are you dealing with security >> settings with EMR, only whitelisting some IP range with security group >> setting is really too weak. >> >> are there really many production system are using EMR? for me, i feel >> using EMR means everyone in my IP range (for some ISP it may be the whole >> town...) is able to see my spark web UI or use my running zepplin notebook >> if they do some port scanning... >> >> 2016-02-18 15:04 GMT+01:00 Gourav Sengupta <gourav.sengu...@gmail.com>: >> >>> Hi, >>> >>> Just out of sheet curiosity why are you not using EMR to start your >>> SPARK cluster? >>> >>> >>> Regards, >>> Gourav >>> >>> On Thu, Feb 18, 2016 at 12:23 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>> >>>> Have you seen this ? >>>> >>>> HADOOP-10988 >>>> >>>> Cheers >>>> >>>> On Thu, Feb 18, 2016 at 3:39 AM, James Hammerton <ja...@gluru.co> >>>> wrote: >>>> >>>>> HI, >>>>> >>>>> I am seeing warnings like this in the logs when I run Spark jobs: >>>>> >>>>> OpenJDK 64-Bit Server VM warning: You have loaded library >>>>> /root/ephemeral-hdfs/lib/native/libhadoop.so.1.0.0 which might have >>>>> disabled stack guard. The VM will try to fix the stack guard now. >>>>> It's highly recommended that you fix the library with 'execstack -c >>>>> <libfile>', or link it with '-z noexecstack'. >>>>> >>>>> >>>>> I used spark-ec2 to launch the cluster with the default AMI, Spark >>>>> 1.5.2, hadoop major version 2.4. I altered the jdk to be openjdk 8 as I'd >>>>> written some jobs in Java 8. The 6 workers nodes are m4.2xlarge and master >>>>> is m4.large. >>>>> >>>>> Could this contribute to any problems running the jobs? >>>>> >>>>> Regards, >>>>> >>>>> James >>>>> >>>> >>>> >>> >> >
