[
https://issues.apache.org/jira/browse/HDFS-5143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13758989#comment-13758989
]
Steve Loughran commented on HDFS-5143:
--------------------------------------
[~hitliuyi] -thanks; having a fixed size increase of only 16 bytes would make
that mapping from encrypted to actual length trivial, and be so small the
quotas shouldn't be a problem. Even if you encrypt a directory with 1M files,
it's only 16MB extra of data.
Storing the data in the NN tends to meet resistance from anyone who runs a
large NN; an extra 16 bytes/file would massively increase the cost of storing
file metadata, and so reduce the max #of files you can store in a single HDFS
instance. And as you say, its not portable -yet encrypting data at rest in a
blobstore is a use case I'd consider important. Maybe we need some FS
independent get/set metadata interface that blobstores could support with
in-blobstore metadata ops (which S3 & Swift both support), raw filesystems
could support with metadata file(s) alongside the data files, and HDFS could
support how it chose.
> Hadoop cryptographic file system
> --------------------------------
>
> Key: HDFS-5143
> URL: https://issues.apache.org/jira/browse/HDFS-5143
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: security
> Affects Versions: 3.0.0
> Reporter: Yi Liu
> Labels: rhino
> Fix For: 3.0.0
>
> Attachments: HADOOP cryptographic file system.pdf
>
>
> There is an increasing need for securing data when Hadoop customers use
> various upper layer applications, such as Map-Reduce, Hive, Pig, HBase and so
> on.
> HADOOP CFS (HADOOP Cryptographic File System) is used to secure data, based
> on HADOOP “FilterFileSystem” decorating DFS or other file systems, and
> transparent to upper layer applications. It’s configurable, scalable and fast.
> High level requirements:
> 1. Transparent to and no modification required for upper layer
> applications.
> 2. “Seek”, “PositionedReadable” are supported for input stream of CFS if
> the wrapped file system supports them.
> 3. Very high performance for encryption and decryption, they will not
> become bottleneck.
> 4. Can decorate HDFS and all other file systems in Hadoop, and will not
> modify existing structure of file system, such as namenode and datanode
> structure if the wrapped file system is HDFS.
> 5. Admin can configure encryption policies, such as which directory will
> be encrypted.
> 6. A robust key management framework.
> 7. Support Pread and append operations if the wrapped file system supports
> them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
