Hi all, instead of handling encryption explicit at application level, I suggest that you investigate into the topic „encryption at rest“, for example encryption at HDFS level https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html <https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html> - obviously if you don’t have to use PGP. Using encryption at the storage layer simplifies your application and architecture and you don’t need to reinvent the wheel.
Kind Regards Roland Johann Software Developer/Data Engineer phenetic GmbH Lütticher Straße 10, 50674 Köln, Germany Mobil: +49 172 365 26 46 Mail: [email protected] Web: phenetic.io Handelsregister: Amtsgericht Köln (HRB 92595) Geschäftsführer: Roland Johann, Uwe Reimann > Am 26.08.2019 um 17:47 schrieb Sachit Murarka <[email protected]>: > > Hi Deepak, > > Thanks for reply. > > Yes. That is the option I am considering now because even apache camel needs > data in local. I might need to copy data from hdfs to local if I want apache > camel ( to get rid of shell). > > Thanks > Sachit > > On Mon, 26 Aug 2019, 21:11 Deepak Sharma, <[email protected] > <mailto:[email protected]>> wrote: > Hi Schit > PGP Encrypt is something that is not inbuilt with spark. > I would suggest writing a shell script that would do pgp encrypt and use it > in spark scala program , which would run from driver. > > Thanks > Deepak > > On Mon, Aug 26, 2019 at 8:10 PM Sachit Murarka <[email protected] > <mailto:[email protected]>> wrote: > Hi All, > > I want to encrypt my files available at HDFS location using PGP Encryption > How can I do it in spark. I saw Apache Camel but it seems camel is used when > source files are in Local location rather than HDFS. > > Kind Regards, > Sachit Murarka > > > -- > Thanks > Deepak > www.bigdatabig.com <http://www.bigdatabig.com/> > www.keosha.net <http://www.keosha.net/>
