Re: sbt publish-local fails with 2.0.0-SNAPSHOT

Thank you Saisai for the JIRA/PR; I'm glad to see it is a one-line fix, and will try this locally in the interim. Mike On 2/1/16, Saisai Shao wrote: > I think it is due to our recent changes to override the external resolvers > in sbt building profile, I just created a JIRA ( > https://issues.apa

Re: Secure multi tenancy on in stand alone mode

w.r.t. running Spark on YARN, there are a few outstanding issues. e.g. SPARK-11182 HDFS Delegation Token See also the comments under SPARK-12279 FYI On Mon, Feb 1, 2016 at 1:02 PM, eugene miretsky wrote: > When having multiple users sharing the same Spark cluster, it's a good > idea to isolat

Secure multi tenancy on in stand alone mode

When having multiple users sharing the same Spark cluster, it's a good idea to isolate the users - make sure that each users runs under a different Linux account and prevent them from accessing data in jobs submitted by other users. Is it currently possible to do with Spark? The only thing I found

Encrypting jobs submitted by the client

Spark supports client authentication via shared secret or kerberos (on YARN). However, the job itself is sent unencrypted over the network. Is there a way to encrypt the jobs the client submits to cluster? The rational for this is very similar to encrypting the HTTP file server traffic - Jars may

Re: Spark 1.6.1

We typically do not allow changes to the classpath in maintenance releases. On Mon, Feb 1, 2016 at 8:16 AM, Hamel Kothari wrote: > I noticed that the Jackson dependency was bumped to 2.5 in master for > something spark-streaming related. Is there any reason that this upgrade > can't be included

Re: Scala 2.11 default build

Yes they do. We haven't dropped 2.10 support yet. There are too many 2.10 active deployments out there. On Mon, Feb 1, 2016 at 11:33 AM, Jakob Odersky wrote: > Awesome! > +1 on Steve Loughran's question, how does this affect support for > 2.10? Do future contributions need to work with Scala 2.

Re: Scala 2.11 default build

Awesome! +1 on Steve Loughran's question, how does this affect support for 2.10? Do future contributions need to work with Scala 2.10? cheers On Mon, Feb 1, 2016 at 7:02 AM, Ted Yu wrote: > The following jobs have been established for build against Scala 2.10: > > https://amplab.cs.berkeley.edu/

Re: Spark 1.6.1

I noticed that the Jackson dependency was bumped to 2.5 in master for something spark-streaming related. Is there any reason that this upgrade can't be included with 1.6.1? According to later comments on this thread: https://issues.apache.org/jira/browse/SPARK-8332 and my personal experience using

Re: Spark 1.6.1

SPARK-12624 has been resolved. According to Wenchen, SPARK-12783 is fixed in 1.6.0 release. Are there other blockers for Spark 1.6.1 ? Thanks On Wed, Jan 13, 2016 at 5:39 PM, Michael Armbrust wrote: > Hey All, > > While I'm not aware of any critical issues with 1.6.0, there are several > corne

Re: Scala 2.11 default build

The following jobs have been established for build against Scala 2.10: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/SPARK-master-COMPILE-MAVEN-SCALA-2.10/ https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/SPARK-master-COMPILE-sbt-SCALA-2.10/ FYI On Mon,

Guidelines for writing SPARK packages

Hi, Is there any guidelines or specs to write a Spark package? I would like to implement a spark package and would like to know the way it needs to be structured (implement some interfaces etc) so that it can plug into Spark for extended functionality. Could any one help me poi

[ANNOUNCE] New SAMBA Package = Spark + AWS Lambda

Hi all, Just sharing news of the release of a newly available Spark package, SAMBA . https://github.com/onetapbeyond/lambda-spark-executor SAMBA is an Apache Spark packa

Re: Scala 2.11 default build

On 30 Jan 2016, at 08:22, Reynold Xin mailto:r...@databricks.com>> wrote: FYI - I just merged Josh's pull request to switch to Scala 2.11 as the default build. https://github.com/apache/spark/pull/10608 does this mean that Spark 2.10 compatibility & testing are no longer needed?

Spark Executor retries infinitely

Hi All, When a Spark job (Spark-1.5.2) is submitted with a single executor and if user passes some wrong JVM arguments with spark.executor.extraJavaOptions, the first executor fails. But the job keeps on retrying, creating a new executor and failing every tim*e, *until CTRL-C is pressed*. *Do we

Re: sbt publish-local fails with 2.0.0-SNAPSHOT

I think it is due to our recent changes to override the external resolvers in sbt building profile, I just created a JIRA ( https://issues.apache.org/jira/browse/SPARK-13109) to track this. On Mon, Feb 1, 2016 at 3:01 PM, Mike Hynes <91m...@gmail.com> wrote: > Hi devs, > > I used to be able to d