[
https://issues.apache.org/jira/browse/SPARK-5682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyunzhang_intel updated SPARK-5682:
------------------------------------
Description:
Encrypted shuffle is enabled in hadoop 2.6 which make the process of shuffle
data safer. This feature is necessary in spark. AES is a specification for the
encryption of electronic data. There are 5 common modes in AES. CTR is one of
the modes. We use two codec JceAesCtrCryptoCodec and OpensslAesCtrCryptoCodec
to enable spark encrypted shuffle which is also used in hadoop encrypted
shuffle. JceAesCtrypoCodec uses encrypted algorithms jdk provides while
OpensslAesCtrCryptoCodec uses encrypted algorithms openssl provides.
Because ugi credential info is used in the process of encrypted shuffle, we
first enable encrypted shuffle on spark-on-yarn framework.
was:Encrypted shuffle is enabled in hadoop 2.6 which make the process of
shuffle data safer. This feature is necessary in spark. We reuse hadoop
encrypted shuffle feature to spark and because ugi credential info is necessary
in encrypted shuffle, we first enable encrypted shuffle on spark-on-yarn
framework.
> Add encrypted shuffle in spark
> ------------------------------
>
> Key: SPARK-5682
> URL: https://issues.apache.org/jira/browse/SPARK-5682
> Project: Spark
> Issue Type: New Feature
> Components: Shuffle
> Reporter: liyunzhang_intel
> Attachments: Design Document of Encrypted Spark
> Shuffle_20150209.docx, Design Document of Encrypted Spark
> Shuffle_20150318.docx
>
>
> Encrypted shuffle is enabled in hadoop 2.6 which make the process of shuffle
> data safer. This feature is necessary in spark. AES is a specification for
> the encryption of electronic data. There are 5 common modes in AES. CTR is
> one of the modes. We use two codec JceAesCtrCryptoCodec and
> OpensslAesCtrCryptoCodec to enable spark encrypted shuffle which is also used
> in hadoop encrypted shuffle. JceAesCtrypoCodec uses encrypted algorithms jdk
> provides while OpensslAesCtrCryptoCodec uses encrypted algorithms openssl
> provides.
> Because ugi credential info is used in the process of encrypted shuffle, we
> first enable encrypted shuffle on spark-on-yarn framework.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
