Re: Securing Spark Job on Cluster

2017-04-28 Thread Mark Hamstra
spark.local.dir http://spark.apache.org/docs/latest/configuration.html On Fri, Apr 28, 2017 at 8:51 AM, Shashi Vishwakarma < shashi.vish...@gmail.com> wrote: > Yes I am using HDFS .Just trying to understand couple of point. > > There would be two kind of encryption which would be required. > >

Re: Securing Spark Job on Cluster

2017-04-28 Thread Shashi Vishwakarma
Yes I am using HDFS .Just trying to understand couple of point. There would be two kind of encryption which would be required. 1. Data in Motion - This could be achieved by enabling SSL -

Re: Securing Spark Job on Cluster

2017-04-28 Thread Jörn Franke
Why don't you use whole disk encryption? Are you using HDFS? > On 28. Apr 2017, at 16:57, Shashi Vishwakarma > wrote: > > Agreed Jorn. Disk encryption is one option that will help to secure data but > how do I know at which location Spark is spilling temp file,

Re: Securing Spark Job on Cluster

2017-04-28 Thread Shashi Vishwakarma
Agreed Jorn. Disk encryption is one option that will help to secure data but how do I know at which location Spark is spilling temp file, shuffle data and application data ? Thanks Shashi On Fri, Apr 28, 2017 at 3:54 PM, Jörn Franke wrote: > You can use disk encryption as

Re: Securing Spark Job on Cluster

2017-04-28 Thread Jörn Franke
You can use disk encryption as provided by the operating system. Additionally, you may think about shredding disks after they are not used anymore. > On 28. Apr 2017, at 14:45, Shashi Vishwakarma > wrote: > > Hi All > > I was dealing with one the spark requirement

Re: Securing Spark Job on Cluster

2017-04-28 Thread Shashi Vishwakarma
Kerberos is not a apache project. Kerberos provides a way to do authentication but does not provide data security. On Fri, Apr 28, 2017 at 3:24 PM, veera satya nv Dantuluri < dvsnva...@gmail.com> wrote: > Hi Shashi, > > Based on your requirement for securing data, we can use Apache kebros, or >

Re: Securing Spark Job on Cluster

2017-04-28 Thread veera satya nv Dantuluri
Hi Shashi, Based on your requirement for securing data, we can use Apache kebros, or we could use the security feature in Spark. > On Apr 28, 2017, at 8:45 AM, Shashi Vishwakarma > wrote: > > Hi All > > I was dealing with one the spark requirement here where