For 1) you can set up transparent encryption at the root directory level for HDFS. However this works at file level and not volume level. For volume level encryption you will have to use something like LUKS only.
For 2), in addition to the steps mentioned in "data confidentiality", you may also need to set up Encrypted Shuffle <https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html>, depending on your use-cases for Hadoop. Thanks, Hariharan On Tue, Jan 7, 2020 at 1:16 AM Daniel Howard <danny...@toldme.com> wrote: > > Hello, > > I am working on getting Hadoop running within our organization. Our high-level use case is to be able to say we're running with end-to-end encryption. It looks like there are two major strategies for getting this done in Hadoop: > > A) HDFS Transparent Encryption: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html > B) Secure Mode: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SecureMode.html > > In our case, we are less concerned with Kerberos' user-level authentication, but we want node-to-node encryption and encryption-at-rest. With cluster applications, I typically achieve encryption-at-rest with LUKS and then enable an application's TLS settings to achieve encryption-in-motion. > > What is my best strategy for Hadoop? Here are a couple of questions: > > 1) The docs say I have to create a new directory, but can I configure HDFS Transparent Encryption to operate across an entire volume? > 2) If I just need encrypted-in-motion, can I do just the "Data confidentiality" part of the Secure Mode doc, or does that depend on setting up Kerberos? > > Thank You! > -danny > > -- > http://dannyman.toldme.com On Tue, Jan 7, 2020 at 1:16 AM Daniel Howard <danny...@toldme.com> wrote: > Hello, > > I am working on getting Hadoop running within our organization. Our > high-level use case is to be able to say we're running with end-to-end > encryption. It looks like there are two major strategies for getting this > done in Hadoop: > > A) HDFS Transparent Encryption: > https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html > B) Secure Mode: > https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SecureMode.html > > In our case, we are less concerned with Kerberos' user-level > authentication, but we want node-to-node encryption and encryption-at-rest. > With cluster applications, I typically achieve encryption-at-rest with LUKS > and then enable an application's TLS settings to achieve > encryption-in-motion. > > What is my best strategy for Hadoop? Here are a couple of questions: > > 1) The docs say I have to create a new directory, but can I configure HDFS > Transparent Encryption to operate across an entire volume? > 2) If I just need encrypted-in-motion, can I do just the "Data > confidentiality" part of the Secure Mode doc, or does that depend on > setting up Kerberos? > > Thank You! > -danny > > -- > http://dannyman.toldme.com >
