Why don't you use whole disk encryption? Are you using HDFS?
> On 28. Apr 2017, at 16:57, Shashi Vishwakarma <shashi.vish...@gmail.com> > wrote: > > Agreed Jorn. Disk encryption is one option that will help to secure data but > how do I know at which location Spark is spilling temp file, shuffle data and > application data ? > > Thanks > Shashi > >> On Fri, Apr 28, 2017 at 3:54 PM, Jörn Franke <jornfra...@gmail.com> wrote: >> You can use disk encryption as provided by the operating system. >> Additionally, you may think about shredding disks after they are not used >> anymore. >> >> > On 28. Apr 2017, at 14:45, Shashi Vishwakarma <shashi.vish...@gmail.com> >> > wrote: >> > >> > Hi All >> > >> > I was dealing with one the spark requirement here where Client (like >> > Banking Client where security is major concern) needs all spark processing >> > should happen securely. >> > >> > For example all communication happening between spark client and server ( >> > driver & executor communication) should be on secure channel. Even when >> > spark spills on disk based on storage level (Mem+Disk), it should not be >> > written in un-encrypted format on local disk or there should be some >> > workaround to prevent spill. >> > >> > I did some research but could not get any concrete solution.Let me know >> > if someone has done this. >> > >> > Any guidance would be a great help. >> > >> > Thanks >> > Shashi >
