Disabling distributing local conf file during spark-submit

Hello, It looks like local conf archives always get copied to the target (HDFS) every time a job is submitted 1. Other files/arc

Re: Encrypting jobs submitted by the client

k driver and executor. that can use HTTPS > My understanding is that that you can use HTTPS for the jar server on the driver, and SASL for block transfer. Is there anything else I'm missing? Cheers, Eugene On Tue, Feb 2, 2016 at 7:46 AM, Steve Loughran wrote: > > > On 1 F

Secure multi tenancy on in stand alone mode

When having multiple users sharing the same Spark cluster, it's a good idea to isolate the users - make sure that each users runs under a different Linux account and prevent them from accessing data in jobs submitted by other users. Is it currently possible to do with Spark? The only thing I found

Encrypting jobs submitted by the client

Spark supports client authentication via shared secret or kerberos (on YARN). However, the job itself is sent unencrypted over the network. Is there a way to encrypt the jobs the client submits to cluster? The rational for this is very similar to encrypting the HTTP file server traffic - Jars may

Kafka consumer: Upgrading to use the the new Java Consumer

Hi, The Kafka connector currently uses the older Kafka Scala consumer. Kafka 0.9 came out with a new Java Kafka consumer. One of the main differences is that the Scala consumer uses a Decoder( kafka.serializer.decoder) trait to decode keys/values while the Java consumer uses the Deserializer int