I want to add that the major hadoop distributions also offer additional
encryption possibilities (for example Ranger from Hortonworks)
Roland Johann
Software Developer/Data Engineer
phenetic GmbH
Lütticher Straße 10, 50674 Köln, Germany
Mobil: +49 172 365 26 46
Mail: roland.joh...@phenetic.io
Hi all,
instead of handling encryption explicit at application level, I suggest that
you investigate into the topic „encryption at rest“, for example encryption at
HDFS level
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html
Hi Deepak,
Thanks for reply.
Yes. That is the option I am considering now because even apache camel
needs data in local. I might need to copy data from hdfs to local if I
want apache camel ( to get rid of shell).
Thanks
Sachit
On Mon, 26 Aug 2019, 21:11 Deepak Sharma, wrote:
> Hi Schit
>
Hi Schit
PGP Encrypt is something that is not inbuilt with spark.
I would suggest writing a shell script that would do pgp encrypt and use it
in spark scala program , which would run from driver.
Thanks
Deepak
On Mon, Aug 26, 2019 at 8:10 PM Sachit Murarka
wrote:
> Hi All,
>
> I want to
Hi All,
I want to encrypt my files available at HDFS location using PGP Encryption
How can I do it in spark. I saw Apache Camel but it seems camel is used
when source files are in Local location rather than HDFS.
Kind Regards,
Sachit Murarka
Hey All,
I'm trying to run a pagerank with GraphFrames on a large graph (about 90
billion edges and 1.4TB total disk size) and I'm running into some issues.
The code is very simplistic it's just load dataframes from S3 and put them
into the GraphFrames pagerank function. But when I run it the
