Re: Kerberos impersonation of a Spark Context at runtime

[Abel Rincón]

Hi Mathieu,

Stratio is working on it, we have a solution running which accomplish our
use case, could you share your use case with us?

Here you have video and slides of our work on this topic

https://spark-summit.org/east-2017/events/kerberizing-spark/


Regards

Abel.

2017-05-04 15:01 GMT+02:00 Saisai Shao :

> Current Spark doesn't support impersonate different users at run-time.
> Current Spark's proxy user is application level, which means when setting
> through --proxy-user the whole application will be running with that user.
>
> On Thu, May 4, 2017 at 5:13 PM, matd  wrote:
>
>> Hi folks,
>>
>> I have a Spark application executing various jobs for different users
>> simultaneously, via several Spark sessions on several threads.
>>
>> My customer would like to kerberize his hadoop cluster. I wonder if there
>> is
>> a way to configure impersonation such as each of these jobs would be ran
>> with the different proxy users. From what I see in spark conf and code,
>> it's
>> not possible to do that at runtime for a specific context, but I'm not
>> familiar with Kerberos nor with this part of Spark.
>>
>> Anyone can confirm/infirm this ?
>>
>> Mathieu
>>
>> (also on S.O
>> http://stackoverflow.com/questions/43765044/kerberos-imperso
>> nation-of-a-spark-context-at-runtime)
>>
>>
>>
>> --
>> View this message in context: http://apache-spark-user-list.
>> 1001560.n3.nabble.com/Kerberos-impersonation-of-a-Spark-
>> Context-at-runtime-tp28651.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> -
>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>>
>>
>

Re: Spark on YARN using Java 1.8 fails

[Abel Rincón]

Hi,

There was another related question

https://mail-archives.apache.org/mod_mbox/incubator-spark-user/201506.mbox/%3CCAJ2peNeruM2Y2Tbf8-Wiras-weE586LM_o25FsN=+z1-bfw...@mail.gmail.com%3E


Some months ago, if I remember well, using spark 1.3 + YARN + Java 8 we had
the same probem.
https://issues.apache.org/jira/browse/SPARK-6388

BTW, nowadays we chose use java 7

Re: Does Spark streaming support is there with RabbitMQ

[Abel Rincón]

Hi,

We tested this receiver internally in stratio sparkta, and it works fine,
If you will try the receiver, we're open to your collaboration, your issues
will be wellcome.

Regards

A.Rincón
Stratio software architect



2015-07-22 8:15 GMT+02:00 Tathagata Das t...@databricks.com:

 You could contact the authors of the spark-packages.. maybe that will help?

 On Mon, Jul 20, 2015 at 6:41 AM, Jeetendra Gangele gangele...@gmail.com
 wrote:

 Thanks Todd,

 I m not sure whether somebody has used it or not. can somebody confirm if
 this integrate nicely with Spark streaming?



 On 20 July 2015 at 18:43, Todd Nist tsind...@gmail.com wrote:

 There is one package available on the spark-packages site,

 http://spark-packages.org/package/Stratio/RabbitMQ-Receiver

 The source is here:

 https://github.com/Stratio/RabbitMQ-Receiver

 Not sure that meets your needs or not.

 -Todd

 On Mon, Jul 20, 2015 at 8:52 AM, Jeetendra Gangele gangele...@gmail.com
  wrote:

 Does Apache spark support RabbitMQ. I have messages on RabbitMQ and I
 want to process them using Apache Spark streaming does it scale?

 Regards
 Jeetendra

Re: Spark and RabbitMQ

[Abel Rincón]

Hi,


There is a RabbitMQ reciver for spark-streaming

http://search.maven.org/#artifactdetails|com.stratio.receiver|rabbitmq|0.1.0-RELEASE|jar

https://github.com/Stratio/RabbitMQ-Receiver


2015-05-12 14:49 GMT+02:00 Dmitry Goldenberg dgoldenberg...@gmail.com:

 Thanks, Akhil. It looks like in the second example, for Rabbit they're
 doing this: https://www.rabbitmq.com/mqtt.html.

 On Tue, May 12, 2015 at 7:37 AM, Akhil Das ak...@sigmoidanalytics.com
 wrote:

 I found two examples Java version
 https://github.com/deepakkashyap/Spark-Streaming-with-RabbitMQ-/blob/master/example/Spark_project/CustomReceiver.java,
 and Scala version. https://github.com/d1eg0/spark-streaming-toy

 Thanks
 Best Regards

 On Tue, May 12, 2015 at 2:31 AM, dgoldenberg dgoldenberg...@gmail.com
 wrote:

 Are there existing or under development versions/modules for streaming
 messages out of RabbitMQ with SparkStreaming, or perhaps a RabbitMQ RDD?



 --
 View this message in context:
 http://apache-spark-user-list.1001560.n3.nabble.com/Spark-and-RabbitMQ-tp22852.html
 Sent from the Apache Spark User List mailing list archive at Nabble.com.

 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org