You can use Spark 1.4 on EMR AMI 3.8.0 if you install Spark as a 3rd party application using the bootstrap action directly without the native Spark inclusion with 1.3.1. See https://github.com/awslabs/emr-bootstrap-actions/tree/master/spark
Refer to https://github.com/awslabs/emr-bootstrap-actions/blob/master/spark/VersionInformation.md to determine the version build to install (for example, build 1.4.0.b includes sparkR and all builds include Kinesis). -Christopher From: Kelly, Jonathan Sent: Wednesday, June 17, 2015 6:56 AM To: Eugen Cepoi; Hideyoshi Maeda Cc: ayan guha; kamatsuoka; user Subject: Re: Spark on EMR Yes, for now it is a wrapper around the old install-spark BA, but that will change soon. The currently supported version in AMI 3.8.0 is 1.3.1, as 1.4.0 was released too late to include it in AMI 3.8.0. Spark 1.4.0 support is coming soon though, of course. Unfortunately, though install-spark is currently being used under the hood, passing "-v,1.4.0" in the options is not supported. Sent from Nine<http://www.9folders.com/> From: Eugen Cepoi <cepoi.eu...@gmail.com<mailto:cepoi.eu...@gmail.com>> Sent: Jun 17, 2015 6:37 AM To: Hideyoshi Maeda Cc: ayan guha;kamatsuoka;user Subject: Re: Spark on EMR It looks like it is a wrapper around https://github.com/awslabs/emr-bootstrap-actions/tree/master/spark So basically adding an option -v,1.4.0.a should work. https://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-spark-configure.html 2015-06-17 15:32 GMT+02:00 Hideyoshi Maeda <hideyoshi.ma...@gmail.com<mailto:hideyoshi.ma...@gmail.com>>: Any ideas what version of Spark is underneath? i.e. is it 1.4? and is SparkR supported on Amazon EMR? On Wed, Jun 17, 2015 at 12:06 AM, ayan guha <guha.a...@gmail.com<mailto:guha.a...@gmail.com>> wrote: That's great news. Can I assume spark on EMR supports kinesis to hbase pipeline? On 17 Jun 2015 05:29, "kamatsuoka" <ken...@gmail.com<mailto:ken...@gmail.com>> wrote: Spark is now officially supported on Amazon Elastic Map Reduce: http://aws.amazon.com/elasticmapreduce/details/spark/ -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-EMR-tp23343.html Sent from the Apache Spark User List mailing list archive at Nabble.com<http://Nabble.com>. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org> For additional commands, e-mail: user-h...@spark.apache.org<mailto:user-h...@spark.apache.org>
