[
https://issues.apache.org/jira/browse/SPARK-5682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14611527#comment-14611527
]
hujiayin edited comment on SPARK-5682 at 7/2/15 6:10 AM:
---------------------------------------------------------
Steps were added to encode and decode the data, the performance will not be
fast than before, in the same time, codes also have security issue, for example
save the plain text in configuration file and finally used as the part of the
key
If you use a better cypher solution, the performance downgrade will be
minimized. i think AES is a bit heavy.
In the same time, the feature based on hadoop 2.6, it is the limitation, that
is why i said rely on hadoop
Though the API is public stable, however, you cannot ensure if the API will not
be changed since it is not the comercial software.
was (Author: hujiayin):
Steps were added to encode and decode the data, the performance will not be
fast than before, in the same time, codes also have security issue, for example
save the plain text in configuration file and finally used as the part of the
key
In the same time, the feature based on hadoop 2.6, it is the limitation, that
is why i said rely on hadoop
Though the API is public stable, however, you cannot ensure if the API will not
be changed since it is not the comercial software.
> Add encrypted shuffle in spark
> ------------------------------
>
> Key: SPARK-5682
> URL: https://issues.apache.org/jira/browse/SPARK-5682
> Project: Spark
> Issue Type: New Feature
> Components: Shuffle
> Reporter: liyunzhang_intel
> Attachments: Design Document of Encrypted Spark
> Shuffle_20150209.docx, Design Document of Encrypted Spark
> Shuffle_20150318.docx, Design Document of Encrypted Spark
> Shuffle_20150402.docx, Design Document of Encrypted Spark
> Shuffle_20150506.docx
>
>
> Encrypted shuffle is enabled in hadoop 2.6 which make the process of shuffle
> data safer. This feature is necessary in spark. AES is a specification for
> the encryption of electronic data. There are 5 common modes in AES. CTR is
> one of the modes. We use two codec JceAesCtrCryptoCodec and
> OpensslAesCtrCryptoCodec to enable spark encrypted shuffle which is also used
> in hadoop encrypted shuffle. JceAesCtrypoCodec uses encrypted algorithms jdk
> provides while OpensslAesCtrCryptoCodec uses encrypted algorithms openssl
> provides.
> Because ugi credential info is used in the process of encrypted shuffle, we
> first enable encrypted shuffle on spark-on-yarn framework.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
