I want to add that the major hadoop distributions also offer additional encryption possibilities (for example Ranger from Hortonworks)
Roland Johann Software Developer/Data Engineer phenetic GmbH Lütticher Straße 10, 50674 Köln, Germany Mobil: +49 172 365 26 46 Mail: [email protected] Web: phenetic.io Handelsregister: Amtsgericht Köln (HRB 92595) Geschäftsführer: Roland Johann, Uwe Reimann > Am 26.08.2019 um 23:09 schrieb Roland Johann <[email protected]>: > > Hi all, > > instead of handling encryption explicit at application level, I suggest that > you investigate into the topic „encryption at rest“, for example encryption > at HDFS level > https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html > > <https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html> > - obviously if you don’t have to use PGP. Using encryption at the storage > layer simplifies your application and architecture and you don’t need to > reinvent the wheel. > > Kind Regards > > Roland Johann > Software Developer/Data Engineer > > phenetic GmbH > Lütticher Straße 10, 50674 Köln, Germany > > Mobil: +49 172 365 26 46 > Mail: [email protected] <mailto:[email protected]> > Web: phenetic.io <http://phenetic.io/> > > Handelsregister: Amtsgericht Köln (HRB 92595) > Geschäftsführer: Roland Johann, Uwe Reimann > > > >> Am 26.08.2019 um 17:47 schrieb Sachit Murarka <[email protected] >> <mailto:[email protected]>>: >> >> Hi Deepak, >> >> Thanks for reply. >> >> Yes. That is the option I am considering now because even apache camel needs >> data in local. I might need to copy data from hdfs to local if I want >> apache camel ( to get rid of shell). >> >> Thanks >> Sachit >> >> On Mon, 26 Aug 2019, 21:11 Deepak Sharma, <[email protected] >> <mailto:[email protected]>> wrote: >> Hi Schit >> PGP Encrypt is something that is not inbuilt with spark. >> I would suggest writing a shell script that would do pgp encrypt and use it >> in spark scala program , which would run from driver. >> >> Thanks >> Deepak >> >> On Mon, Aug 26, 2019 at 8:10 PM Sachit Murarka <[email protected] >> <mailto:[email protected]>> wrote: >> Hi All, >> >> I want to encrypt my files available at HDFS location using PGP Encryption >> How can I do it in spark. I saw Apache Camel but it seems camel is used >> when source files are in Local location rather than HDFS. >> >> Kind Regards, >> Sachit Murarka >> >> >> -- >> Thanks >> Deepak >> www.bigdatabig.com <http://www.bigdatabig.com/> >> www.keosha.net <http://www.keosha.net/>
