[jira] [Commented] (HDFS-8747) Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption Zones
[ https://issues.apache.org/jira/browse/HDFS-8747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15106889#comment-15106889 ] Sanjay Radia commented on HDFS-8747: The main motivation for nested EZ is root + subdirs as per Andrew's comment. Is it such a big deal for an admin to set up EZ as he creates the directories in dirs? I think nested encryption will complicate things like volumes down the road and I don't think this extra complexity is necessary. > Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption > Zones > -- > > Key: HDFS-8747 > URL: https://issues.apache.org/jira/browse/HDFS-8747 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption >Affects Versions: 2.6.0 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-8747-07092015.pdf, HDFS-8747-07152015.pdf, > HDFS-8747-07292015.pdf > > > HDFS Transparent Data Encryption At-Rest was introduced in Hadoop 2.6 to > allow create encryption zone on top of a single HDFS directory. Files under > the root directory of the encryption zone will be encrypted/decrypted > transparently upon HDFS client write or read operations. > Generally, it does not support rename(without data copying) across encryption > zones or between encryption zone and non-encryption zone because different > security settings of encryption zones. However, there are certain use cases > where efficient rename support is desired. This JIRA is to propose better > support of two such use cases “Scratch Space” (a.k.a. staging area) and “Soft > Delete” (a.k.a. trash) with HDFS encryption zones. > “Scratch Space” is widely used in Hadoop jobs, which requires efficient > rename support. Temporary files from MR jobs are usually stored in staging > area outside encryption zone such as “/tmp” directory and then rename to > targeted directories as specified once the data is ready to be further > processed. > Below is a summary of supported/unsupported cases from latest Hadoop: > * Rename within the encryption zone is supported > * Rename the entire encryption zone by moving the root directory of the zone > is allowed. > * Rename sub-directory/file from encryption zone to non-encryption zone is > not allowed. > * Rename sub-directory/file from encryption zone A to encryption zone B is > not allowed. > * Rename from non-encryption zone to encryption zone is not allowed. > “Soft delete” (a.k.a. trash) is a client-side “soft delete” feature that > helps prevent accidental deletion of files and directories. If trash is > enabled and a file or directory is deleted using the Hadoop shell, the file > is moved to the .Trash directory of the user's home directory instead of > being deleted. Deleted files are initially moved (renamed) to the Current > sub-directory of the .Trash directory with original path being preserved. > Files and directories in the trash can be restored simply by moving them to a > location outside the .Trash directory. > Due to the limited rename support, delete sub-directory/file within > encryption zone with trash feature is not allowed. Client has to use > -skipTrash option to work around this. HADOOP-10902 and HDFS-6767 improved > the error message but without a complete solution to the problem. > We propose to solve the problem by generalizing the mapping between > encryption zone and its underlying HDFS directories from 1:1 today to 1:N. > The encryption zone should allow non-overlapped directories such as scratch > space or soft delete "trash" locations to be added/removed dynamically after > creation. This way, rename for "scratch space" and "soft delete" can be > better supported without breaking the assumption that rename is only > supported "within the zone". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8747) Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption Zones
[ https://issues.apache.org/jira/browse/HDFS-8747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14680736#comment-14680736 ] Andrew Wang commented on HDFS-8747: --- I was thinking the root zone is created by an admin when / is empty, just like any other encryption zone. This means the admin needs to do this up front when the cluster is freshly formatted. > Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption > Zones > -- > > Key: HDFS-8747 > URL: https://issues.apache.org/jira/browse/HDFS-8747 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption >Affects Versions: 2.6.0 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-8747-07092015.pdf, HDFS-8747-07152015.pdf, > HDFS-8747-07292015.pdf > > > HDFS Transparent Data Encryption At-Rest was introduced in Hadoop 2.6 to > allow create encryption zone on top of a single HDFS directory. Files under > the root directory of the encryption zone will be encrypted/decrypted > transparently upon HDFS client write or read operations. > Generally, it does not support rename(without data copying) across encryption > zones or between encryption zone and non-encryption zone because different > security settings of encryption zones. However, there are certain use cases > where efficient rename support is desired. This JIRA is to propose better > support of two such use cases “Scratch Space” (a.k.a. staging area) and “Soft > Delete” (a.k.a. trash) with HDFS encryption zones. > “Scratch Space” is widely used in Hadoop jobs, which requires efficient > rename support. Temporary files from MR jobs are usually stored in staging > area outside encryption zone such as “/tmp” directory and then rename to > targeted directories as specified once the data is ready to be further > processed. > Below is a summary of supported/unsupported cases from latest Hadoop: > * Rename within the encryption zone is supported > * Rename the entire encryption zone by moving the root directory of the zone > is allowed. > * Rename sub-directory/file from encryption zone to non-encryption zone is > not allowed. > * Rename sub-directory/file from encryption zone A to encryption zone B is > not allowed. > * Rename from non-encryption zone to encryption zone is not allowed. > “Soft delete” (a.k.a. trash) is a client-side “soft delete” feature that > helps prevent accidental deletion of files and directories. If trash is > enabled and a file or directory is deleted using the Hadoop shell, the file > is moved to the .Trash directory of the user's home directory instead of > being deleted. Deleted files are initially moved (renamed) to the Current > sub-directory of the .Trash directory with original path being preserved. > Files and directories in the trash can be restored simply by moving them to a > location outside the .Trash directory. > Due to the limited rename support, delete sub-directory/file within > encryption zone with trash feature is not allowed. Client has to use > -skipTrash option to work around this. HADOOP-10902 and HDFS-6767 improved > the error message but without a complete solution to the problem. > We propose to solve the problem by generalizing the mapping between > encryption zone and its underlying HDFS directories from 1:1 today to 1:N. > The encryption zone should allow non-overlapped directories such as scratch > space or soft delete "trash" locations to be added/removed dynamically after > creation. This way, rename for "scratch space" and "soft delete" can be > better supported without breaking the assumption that rename is only > supported "within the zone". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8747) Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption Zones
[ https://issues.apache.org/jira/browse/HDFS-8747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662206#comment-14662206 ] Xiaoyu Yao commented on HDFS-8747: -- bq. Regarding nested encryption zones, one request we've had is setting "/" as an encryption zone, then subdirs as different EZs. This guarantees that all data in HDFS is encrypted, and gives the flexibility of using different EZ keys for subdirs if desired. That is not difficult to support as long as we don't allow create zone with existing files. Do you expect the root zone to be created implicitly by NN if this certain key is enabled? > Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption > Zones > -- > > Key: HDFS-8747 > URL: https://issues.apache.org/jira/browse/HDFS-8747 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption >Affects Versions: 2.6.0 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-8747-07092015.pdf, HDFS-8747-07152015.pdf, > HDFS-8747-07292015.pdf > > > HDFS Transparent Data Encryption At-Rest was introduced in Hadoop 2.6 to > allow create encryption zone on top of a single HDFS directory. Files under > the root directory of the encryption zone will be encrypted/decrypted > transparently upon HDFS client write or read operations. > Generally, it does not support rename(without data copying) across encryption > zones or between encryption zone and non-encryption zone because different > security settings of encryption zones. However, there are certain use cases > where efficient rename support is desired. This JIRA is to propose better > support of two such use cases “Scratch Space” (a.k.a. staging area) and “Soft > Delete” (a.k.a. trash) with HDFS encryption zones. > “Scratch Space” is widely used in Hadoop jobs, which requires efficient > rename support. Temporary files from MR jobs are usually stored in staging > area outside encryption zone such as “/tmp” directory and then rename to > targeted directories as specified once the data is ready to be further > processed. > Below is a summary of supported/unsupported cases from latest Hadoop: > * Rename within the encryption zone is supported > * Rename the entire encryption zone by moving the root directory of the zone > is allowed. > * Rename sub-directory/file from encryption zone to non-encryption zone is > not allowed. > * Rename sub-directory/file from encryption zone A to encryption zone B is > not allowed. > * Rename from non-encryption zone to encryption zone is not allowed. > “Soft delete” (a.k.a. trash) is a client-side “soft delete” feature that > helps prevent accidental deletion of files and directories. If trash is > enabled and a file or directory is deleted using the Hadoop shell, the file > is moved to the .Trash directory of the user's home directory instead of > being deleted. Deleted files are initially moved (renamed) to the Current > sub-directory of the .Trash directory with original path being preserved. > Files and directories in the trash can be restored simply by moving them to a > location outside the .Trash directory. > Due to the limited rename support, delete sub-directory/file within > encryption zone with trash feature is not allowed. Client has to use > -skipTrash option to work around this. HADOOP-10902 and HDFS-6767 improved > the error message but without a complete solution to the problem. > We propose to solve the problem by generalizing the mapping between > encryption zone and its underlying HDFS directories from 1:1 today to 1:N. > The encryption zone should allow non-overlapped directories such as scratch > space or soft delete "trash" locations to be added/removed dynamically after > creation. This way, rename for "scratch space" and "soft delete" can be > better supported without breaking the assumption that rename is only > supported "within the zone". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8747) Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption Zones
[ https://issues.apache.org/jira/browse/HDFS-8747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14654141#comment-14654141 ] Andrew Wang commented on HDFS-8747: --- Hi Xiaoyu, Regarding snapshots, the idea would be to take snapshots on the EZ root. Snapshots are typically scheduled on a periodic basis, so not a manual operation. Basically a cron job that runs every hour to take a snapshot. I'll note also that filesystems with snapshot support like ZFS and WAFL aren't really used with trash, since snapshots are a superior solution for data recovery. Regarding nested encryption zones, one request we've had is setting "/" as an encryption zone, then subdirs as different EZs. This guarantees that all data in HDFS is encrypted, and gives the flexibility of using different EZ keys for subdirs if desired. > Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption > Zones > -- > > Key: HDFS-8747 > URL: https://issues.apache.org/jira/browse/HDFS-8747 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption >Affects Versions: 2.6.0 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-8747-07092015.pdf, HDFS-8747-07152015.pdf, > HDFS-8747-07292015.pdf > > > HDFS Transparent Data Encryption At-Rest was introduced in Hadoop 2.6 to > allow create encryption zone on top of a single HDFS directory. Files under > the root directory of the encryption zone will be encrypted/decrypted > transparently upon HDFS client write or read operations. > Generally, it does not support rename(without data copying) across encryption > zones or between encryption zone and non-encryption zone because different > security settings of encryption zones. However, there are certain use cases > where efficient rename support is desired. This JIRA is to propose better > support of two such use cases “Scratch Space” (a.k.a. staging area) and “Soft > Delete” (a.k.a. trash) with HDFS encryption zones. > “Scratch Space” is widely used in Hadoop jobs, which requires efficient > rename support. Temporary files from MR jobs are usually stored in staging > area outside encryption zone such as “/tmp” directory and then rename to > targeted directories as specified once the data is ready to be further > processed. > Below is a summary of supported/unsupported cases from latest Hadoop: > * Rename within the encryption zone is supported > * Rename the entire encryption zone by moving the root directory of the zone > is allowed. > * Rename sub-directory/file from encryption zone to non-encryption zone is > not allowed. > * Rename sub-directory/file from encryption zone A to encryption zone B is > not allowed. > * Rename from non-encryption zone to encryption zone is not allowed. > “Soft delete” (a.k.a. trash) is a client-side “soft delete” feature that > helps prevent accidental deletion of files and directories. If trash is > enabled and a file or directory is deleted using the Hadoop shell, the file > is moved to the .Trash directory of the user's home directory instead of > being deleted. Deleted files are initially moved (renamed) to the Current > sub-directory of the .Trash directory with original path being preserved. > Files and directories in the trash can be restored simply by moving them to a > location outside the .Trash directory. > Due to the limited rename support, delete sub-directory/file within > encryption zone with trash feature is not allowed. Client has to use > -skipTrash option to work around this. HADOOP-10902 and HDFS-6767 improved > the error message but without a complete solution to the problem. > We propose to solve the problem by generalizing the mapping between > encryption zone and its underlying HDFS directories from 1:1 today to 1:N. > The encryption zone should allow non-overlapped directories such as scratch > space or soft delete "trash" locations to be added/removed dynamically after > creation. This way, rename for "scratch space" and "soft delete" can be > better supported without breaking the assumption that rename is only > supported "within the zone". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8747) Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption Zones
[ https://issues.apache.org/jira/browse/HDFS-8747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14654086#comment-14654086 ] Xiaoyu Yao commented on HDFS-8747: -- Thanks [~andrew.wang] and [~hitliuyi] for providing the feedback! Please help reviewing patch for HDFS-8830. We have customers too who want to use encryption with Trash. Adding user folder trash folder to encryption zone is cheapest solution after HDFS-8830. It is an one-time setting (per user/per zone) requiring no code changes. We get the snapshot automatically for users' Trash with default trash policy for free. This is better than manually snapshots for different users' Trash or not support it at all with "-skipTrash". We could still explore other solutions for trash in HDFS-8831. bq. I do think though that nested encryption zones would make this better yet (for reasons even besides trash), and would not be too difficult to implement. I understand nested zone helps the case where user's home folder may already in an encryption zone for trash. Can you elaborate more on how nested zone would make this better overall? We could add it if this really help. > Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption > Zones > -- > > Key: HDFS-8747 > URL: https://issues.apache.org/jira/browse/HDFS-8747 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption >Affects Versions: 2.6.0 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-8747-07092015.pdf, HDFS-8747-07152015.pdf, > HDFS-8747-07292015.pdf > > > HDFS Transparent Data Encryption At-Rest was introduced in Hadoop 2.6 to > allow create encryption zone on top of a single HDFS directory. Files under > the root directory of the encryption zone will be encrypted/decrypted > transparently upon HDFS client write or read operations. > Generally, it does not support rename(without data copying) across encryption > zones or between encryption zone and non-encryption zone because different > security settings of encryption zones. However, there are certain use cases > where efficient rename support is desired. This JIRA is to propose better > support of two such use cases “Scratch Space” (a.k.a. staging area) and “Soft > Delete” (a.k.a. trash) with HDFS encryption zones. > “Scratch Space” is widely used in Hadoop jobs, which requires efficient > rename support. Temporary files from MR jobs are usually stored in staging > area outside encryption zone such as “/tmp” directory and then rename to > targeted directories as specified once the data is ready to be further > processed. > Below is a summary of supported/unsupported cases from latest Hadoop: > * Rename within the encryption zone is supported > * Rename the entire encryption zone by moving the root directory of the zone > is allowed. > * Rename sub-directory/file from encryption zone to non-encryption zone is > not allowed. > * Rename sub-directory/file from encryption zone A to encryption zone B is > not allowed. > * Rename from non-encryption zone to encryption zone is not allowed. > “Soft delete” (a.k.a. trash) is a client-side “soft delete” feature that > helps prevent accidental deletion of files and directories. If trash is > enabled and a file or directory is deleted using the Hadoop shell, the file > is moved to the .Trash directory of the user's home directory instead of > being deleted. Deleted files are initially moved (renamed) to the Current > sub-directory of the .Trash directory with original path being preserved. > Files and directories in the trash can be restored simply by moving them to a > location outside the .Trash directory. > Due to the limited rename support, delete sub-directory/file within > encryption zone with trash feature is not allowed. Client has to use > -skipTrash option to work around this. HADOOP-10902 and HDFS-6767 improved > the error message but without a complete solution to the problem. > We propose to solve the problem by generalizing the mapping between > encryption zone and its underlying HDFS directories from 1:1 today to 1:N. > The encryption zone should allow non-overlapped directories such as scratch > space or soft delete "trash" locations to be added/removed dynamically after > creation. This way, rename for "scratch space" and "soft delete" can be > better supported without breaking the assumption that rename is only > supported "within the zone". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8747) Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption Zones
[ https://issues.apache.org/jira/browse/HDFS-8747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14652534#comment-14652534 ] Andrew Wang commented on HDFS-8747: --- >From our side, we have some customers using encryption who want Trash as a >safety mechanism. So simply using -skipTrash means they lose this safety. My >advice has been to use snapshots, since snapshots provide similar (if not >superior) properties to trash. That's also why I'm willing to accept some of >the compromises regarding the proposed design; while not perfect, it's better >than what we've got now. I do think though that nested encryption zones would make this better yet (for reasons even besides trash), and would not be too difficult to implement. > Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption > Zones > -- > > Key: HDFS-8747 > URL: https://issues.apache.org/jira/browse/HDFS-8747 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption >Affects Versions: 2.6.0 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-8747-07092015.pdf, HDFS-8747-07152015.pdf, > HDFS-8747-07292015.pdf > > > HDFS Transparent Data Encryption At-Rest was introduced in Hadoop 2.6 to > allow create encryption zone on top of a single HDFS directory. Files under > the root directory of the encryption zone will be encrypted/decrypted > transparently upon HDFS client write or read operations. > Generally, it does not support rename(without data copying) across encryption > zones or between encryption zone and non-encryption zone because different > security settings of encryption zones. However, there are certain use cases > where efficient rename support is desired. This JIRA is to propose better > support of two such use cases “Scratch Space” (a.k.a. staging area) and “Soft > Delete” (a.k.a. trash) with HDFS encryption zones. > “Scratch Space” is widely used in Hadoop jobs, which requires efficient > rename support. Temporary files from MR jobs are usually stored in staging > area outside encryption zone such as “/tmp” directory and then rename to > targeted directories as specified once the data is ready to be further > processed. > Below is a summary of supported/unsupported cases from latest Hadoop: > * Rename within the encryption zone is supported > * Rename the entire encryption zone by moving the root directory of the zone > is allowed. > * Rename sub-directory/file from encryption zone to non-encryption zone is > not allowed. > * Rename sub-directory/file from encryption zone A to encryption zone B is > not allowed. > * Rename from non-encryption zone to encryption zone is not allowed. > “Soft delete” (a.k.a. trash) is a client-side “soft delete” feature that > helps prevent accidental deletion of files and directories. If trash is > enabled and a file or directory is deleted using the Hadoop shell, the file > is moved to the .Trash directory of the user's home directory instead of > being deleted. Deleted files are initially moved (renamed) to the Current > sub-directory of the .Trash directory with original path being preserved. > Files and directories in the trash can be restored simply by moving them to a > location outside the .Trash directory. > Due to the limited rename support, delete sub-directory/file within > encryption zone with trash feature is not allowed. Client has to use > -skipTrash option to work around this. HADOOP-10902 and HDFS-6767 improved > the error message but without a complete solution to the problem. > We propose to solve the problem by generalizing the mapping between > encryption zone and its underlying HDFS directories from 1:1 today to 1:N. > The encryption zone should allow non-overlapped directories such as scratch > space or soft delete "trash" locations to be added/removed dynamically after > creation. This way, rename for "scratch space" and "soft delete" can be > better supported without breaking the assumption that rename is only > supported "within the zone". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8747) Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption Zones
[
https://issues.apache.org/jira/browse/HDFS-8747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14651474#comment-14651474
]
Yi Liu commented on HDFS-8747:
--
The design looks good overall. Besides Andrew's comments, I have few
questions/comments too:
*1.* About trash, it's to prevent accidental deletion of files and directories
if enabled, I think we may not need special support for encryption zone besides
what we have currently, since if users don't use {{-skipTrash}}, the deletion
will be failed, so people will not accidentally delete the files/directories,
they should know what they are doing if he uses {{-skipTrash}}.On the other
hand, the management is more complex as Andrew discussed for Trash. Do you see
any real problems of trash for encryption zones.
*2.* About Scratch, it's good and can solve some problems, nice work to support
it. I did see hive needs this support if people want to use transparent
encryption there.
> Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption
> Zones
> --
>
> Key: HDFS-8747
> URL: https://issues.apache.org/jira/browse/HDFS-8747
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Affects Versions: 2.6.0
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-8747-07092015.pdf, HDFS-8747-07152015.pdf,
> HDFS-8747-07292015.pdf
>
>
> HDFS Transparent Data Encryption At-Rest was introduced in Hadoop 2.6 to
> allow create encryption zone on top of a single HDFS directory. Files under
> the root directory of the encryption zone will be encrypted/decrypted
> transparently upon HDFS client write or read operations.
> Generally, it does not support rename(without data copying) across encryption
> zones or between encryption zone and non-encryption zone because different
> security settings of encryption zones. However, there are certain use cases
> where efficient rename support is desired. This JIRA is to propose better
> support of two such use cases “Scratch Space” (a.k.a. staging area) and “Soft
> Delete” (a.k.a. trash) with HDFS encryption zones.
> “Scratch Space” is widely used in Hadoop jobs, which requires efficient
> rename support. Temporary files from MR jobs are usually stored in staging
> area outside encryption zone such as “/tmp” directory and then rename to
> targeted directories as specified once the data is ready to be further
> processed.
> Below is a summary of supported/unsupported cases from latest Hadoop:
> * Rename within the encryption zone is supported
> * Rename the entire encryption zone by moving the root directory of the zone
> is allowed.
> * Rename sub-directory/file from encryption zone to non-encryption zone is
> not allowed.
> * Rename sub-directory/file from encryption zone A to encryption zone B is
> not allowed.
> * Rename from non-encryption zone to encryption zone is not allowed.
> “Soft delete” (a.k.a. trash) is a client-side “soft delete” feature that
> helps prevent accidental deletion of files and directories. If trash is
> enabled and a file or directory is deleted using the Hadoop shell, the file
> is moved to the .Trash directory of the user's home directory instead of
> being deleted. Deleted files are initially moved (renamed) to the Current
> sub-directory of the .Trash directory with original path being preserved.
> Files and directories in the trash can be restored simply by moving them to a
> location outside the .Trash directory.
> Due to the limited rename support, delete sub-directory/file within
> encryption zone with trash feature is not allowed. Client has to use
> -skipTrash option to work around this. HADOOP-10902 and HDFS-6767 improved
> the error message but without a complete solution to the problem.
> We propose to solve the problem by generalizing the mapping between
> encryption zone and its underlying HDFS directories from 1:1 today to 1:N.
> The encryption zone should allow non-overlapped directories such as scratch
> space or soft delete "trash" locations to be added/removed dynamically after
> creation. This way, rename for "scratch space" and "soft delete" can be
> better supported without breaking the assumption that rename is only
> supported "within the zone".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8747) Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption Zones
[ https://issues.apache.org/jira/browse/HDFS-8747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14649905#comment-14649905 ] Xiaoyu Yao commented on HDFS-8747: -- bq. This is maybe viable for scratch, but not for trash. There can be many users on a cluster accessing a variety of EZs, such that it's unmanageable for the super-user to set up all the Trash folders beforehand. Three solutions have been discussed in "Design->Soft Delete" section of the spec. My initial take is on "Option 1: Per User Trash Namespace", which is mostly for compatibility and simplicity. If pre-create trash folder for many users is a concern, "Option 2: Global Trash Namespace" which is similar to the idea proposed in Hadoop-7310 can be used. It will not be compatible with current Trash behavior where users find their deleted files under /user/username/.Trash/Current/ These solutions can be implemented as pluggable trash policy for admin to choose with configurable keys when the default one may not be appropriate for their deployment. bq. Another question, how would this work if a user's homedir is already an EZ? Do you plan to add support for nested encryption zones? No we don't plan to support nested encryption zones. If we take "Option 1", this will not be supported. But if we take "Option 2", it will not be a problem as the trash namespace for encryption zone will be separated from user's homedir. > Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption > Zones > -- > > Key: HDFS-8747 > URL: https://issues.apache.org/jira/browse/HDFS-8747 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption >Affects Versions: 2.6.0 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-8747-07092015.pdf, HDFS-8747-07152015.pdf, > HDFS-8747-07292015.pdf > > > HDFS Transparent Data Encryption At-Rest was introduced in Hadoop 2.6 to > allow create encryption zone on top of a single HDFS directory. Files under > the root directory of the encryption zone will be encrypted/decrypted > transparently upon HDFS client write or read operations. > Generally, it does not support rename(without data copying) across encryption > zones or between encryption zone and non-encryption zone because different > security settings of encryption zones. However, there are certain use cases > where efficient rename support is desired. This JIRA is to propose better > support of two such use cases “Scratch Space” (a.k.a. staging area) and “Soft > Delete” (a.k.a. trash) with HDFS encryption zones. > “Scratch Space” is widely used in Hadoop jobs, which requires efficient > rename support. Temporary files from MR jobs are usually stored in staging > area outside encryption zone such as “/tmp” directory and then rename to > targeted directories as specified once the data is ready to be further > processed. > Below is a summary of supported/unsupported cases from latest Hadoop: > * Rename within the encryption zone is supported > * Rename the entire encryption zone by moving the root directory of the zone > is allowed. > * Rename sub-directory/file from encryption zone to non-encryption zone is > not allowed. > * Rename sub-directory/file from encryption zone A to encryption zone B is > not allowed. > * Rename from non-encryption zone to encryption zone is not allowed. > “Soft delete” (a.k.a. trash) is a client-side “soft delete” feature that > helps prevent accidental deletion of files and directories. If trash is > enabled and a file or directory is deleted using the Hadoop shell, the file > is moved to the .Trash directory of the user's home directory instead of > being deleted. Deleted files are initially moved (renamed) to the Current > sub-directory of the .Trash directory with original path being preserved. > Files and directories in the trash can be restored simply by moving them to a > location outside the .Trash directory. > Due to the limited rename support, delete sub-directory/file within > encryption zone with trash feature is not allowed. Client has to use > -skipTrash option to work around this. HADOOP-10902 and HDFS-6767 improved > the error message but without a complete solution to the problem. > We propose to solve the problem by generalizing the mapping between > encryption zone and its underlying HDFS directories from 1:1 today to 1:N. > The encryption zone should allow non-overlapped directories such as scratch > space or soft delete "trash" locations to be added/removed dynamically after > creation. This way, rename for "scratch space" and "soft delete" can be > better supported without breaking the assumption that rename is only > supported "within the zone". -- This message was sent by Atl
[jira] [Commented] (HDFS-8747) Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption Zones
[ https://issues.apache.org/jira/browse/HDFS-8747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14649805#comment-14649805 ] Andrew Wang commented on HDFS-8747: --- bq. Encryption zone as a security concept should be managed consistently with a single entity. Based on that, support adding additional roots to encryption zone is a natural enhancement and better solution. SGTM, definitely like the idea of the EZ as a management unit. bq. The trash folder/scratch folder can be per-created and added to encryption zone by super user as needed This is maybe viable for scratch, but not for trash. There can be many users on a cluster accessing a variety of EZs, such that it's unmanageable for the super-user to set up all the Trash folders beforehand. Another question, how would this work if a user's homedir is already an EZ? Do you plan to add support for nested encryption zones? > Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption > Zones > -- > > Key: HDFS-8747 > URL: https://issues.apache.org/jira/browse/HDFS-8747 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption >Affects Versions: 2.6.0 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-8747-07092015.pdf, HDFS-8747-07152015.pdf, > HDFS-8747-07292015.pdf > > > HDFS Transparent Data Encryption At-Rest was introduced in Hadoop 2.6 to > allow create encryption zone on top of a single HDFS directory. Files under > the root directory of the encryption zone will be encrypted/decrypted > transparently upon HDFS client write or read operations. > Generally, it does not support rename(without data copying) across encryption > zones or between encryption zone and non-encryption zone because different > security settings of encryption zones. However, there are certain use cases > where efficient rename support is desired. This JIRA is to propose better > support of two such use cases “Scratch Space” (a.k.a. staging area) and “Soft > Delete” (a.k.a. trash) with HDFS encryption zones. > “Scratch Space” is widely used in Hadoop jobs, which requires efficient > rename support. Temporary files from MR jobs are usually stored in staging > area outside encryption zone such as “/tmp” directory and then rename to > targeted directories as specified once the data is ready to be further > processed. > Below is a summary of supported/unsupported cases from latest Hadoop: > * Rename within the encryption zone is supported > * Rename the entire encryption zone by moving the root directory of the zone > is allowed. > * Rename sub-directory/file from encryption zone to non-encryption zone is > not allowed. > * Rename sub-directory/file from encryption zone A to encryption zone B is > not allowed. > * Rename from non-encryption zone to encryption zone is not allowed. > “Soft delete” (a.k.a. trash) is a client-side “soft delete” feature that > helps prevent accidental deletion of files and directories. If trash is > enabled and a file or directory is deleted using the Hadoop shell, the file > is moved to the .Trash directory of the user's home directory instead of > being deleted. Deleted files are initially moved (renamed) to the Current > sub-directory of the .Trash directory with original path being preserved. > Files and directories in the trash can be restored simply by moving them to a > location outside the .Trash directory. > Due to the limited rename support, delete sub-directory/file within > encryption zone with trash feature is not allowed. Client has to use > -skipTrash option to work around this. HADOOP-10902 and HDFS-6767 improved > the error message but without a complete solution to the problem. > We propose to solve the problem by generalizing the mapping between > encryption zone and its underlying HDFS directories from 1:1 today to 1:N. > The encryption zone should allow non-overlapped directories such as scratch > space or soft delete "trash" locations to be added/removed dynamically after > creation. This way, rename for "scratch space" and "soft delete" can be > better supported without breaking the assumption that rename is only > supported "within the zone". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8747) Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption Zones
[ https://issues.apache.org/jira/browse/HDFS-8747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14649486#comment-14649486 ] Xiaoyu Yao commented on HDFS-8747: -- Thanks [~andrew.wang] for reviewing. bq. Have you thought about simply allowing rename between EZs with the same settings? This would be a much smaller and easier change with similar properties. Your proposal I think is still better in terms of ease-of-use and also ensuring security invariants around key rolling (if/when we implement that). Yes. We've discussed this simpler work around. But there are many limitations such as security invariants you mentioned above. We don't want to limit different EZs to share the same zone key just to support rename as they may have different policies. Encryption zone as a security concept should be managed consistently with a single entity. Based on that, support adding additional roots to encryption zone is a natural enhancement and better solution. bq. If we keep the APIs superuser-only, how does a normal user add their trash folder to an EZ? Same for scratch folders, e.g. if the Hive user is not a superuser. I think we should keep this API as superuser-only. It can still be useful even though we keep it as superuser only. The trash folder/scratch folder can be per-created and added to encryption zone by super user as needed. This removes the limitation for hive scratch folder, which currently has to be configured under the single root of the encryption zone. We can discuss more on this for HDFS-8831. > Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption > Zones > -- > > Key: HDFS-8747 > URL: https://issues.apache.org/jira/browse/HDFS-8747 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption >Affects Versions: 2.6.0 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-8747-07092015.pdf, HDFS-8747-07152015.pdf, > HDFS-8747-07292015.pdf > > > HDFS Transparent Data Encryption At-Rest was introduced in Hadoop 2.6 to > allow create encryption zone on top of a single HDFS directory. Files under > the root directory of the encryption zone will be encrypted/decrypted > transparently upon HDFS client write or read operations. > Generally, it does not support rename(without data copying) across encryption > zones or between encryption zone and non-encryption zone because different > security settings of encryption zones. However, there are certain use cases > where efficient rename support is desired. This JIRA is to propose better > support of two such use cases “Scratch Space” (a.k.a. staging area) and “Soft > Delete” (a.k.a. trash) with HDFS encryption zones. > “Scratch Space” is widely used in Hadoop jobs, which requires efficient > rename support. Temporary files from MR jobs are usually stored in staging > area outside encryption zone such as “/tmp” directory and then rename to > targeted directories as specified once the data is ready to be further > processed. > Below is a summary of supported/unsupported cases from latest Hadoop: > * Rename within the encryption zone is supported > * Rename the entire encryption zone by moving the root directory of the zone > is allowed. > * Rename sub-directory/file from encryption zone to non-encryption zone is > not allowed. > * Rename sub-directory/file from encryption zone A to encryption zone B is > not allowed. > * Rename from non-encryption zone to encryption zone is not allowed. > “Soft delete” (a.k.a. trash) is a client-side “soft delete” feature that > helps prevent accidental deletion of files and directories. If trash is > enabled and a file or directory is deleted using the Hadoop shell, the file > is moved to the .Trash directory of the user's home directory instead of > being deleted. Deleted files are initially moved (renamed) to the Current > sub-directory of the .Trash directory with original path being preserved. > Files and directories in the trash can be restored simply by moving them to a > location outside the .Trash directory. > Due to the limited rename support, delete sub-directory/file within > encryption zone with trash feature is not allowed. Client has to use > -skipTrash option to work around this. HADOOP-10902 and HDFS-6767 improved > the error message but without a complete solution to the problem. > We propose to solve the problem by generalizing the mapping between > encryption zone and its underlying HDFS directories from 1:1 today to 1:N. > The encryption zone should allow non-overlapped directories such as scratch > space or soft delete "trash" locations to be added/removed dynamically after > creation. This way, rename for "scratch space" and "soft delete" can be > better sup
[jira] [Commented] (HDFS-8747) Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption Zones
[ https://issues.apache.org/jira/browse/HDFS-8747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14648086#comment-14648086 ] Andrew Wang commented on HDFS-8747: --- Hi [~xyao], I like this idea overall, nice work. A few questions: * Have you thought about simply allowing rename between EZs with the same settings? This would be a much smaller and easier change with similar properties. Your proposal I think is still better in terms of ease-of-use and also ensuring security invariants around key rolling (if/when we implement that). * If we keep the APIs superuser-only, how does a normal user add their trash folder to an EZ? Same for scratch folders, e.g. if the Hive user is not a superuser. > Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption > Zones > -- > > Key: HDFS-8747 > URL: https://issues.apache.org/jira/browse/HDFS-8747 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption >Affects Versions: 2.6.0 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-8747-07092015.pdf, HDFS-8747-07152015.pdf, > HDFS-8747-07292015.pdf > > > HDFS Transparent Data Encryption At-Rest was introduced in Hadoop 2.6 to > allow create encryption zone on top of a single HDFS directory. Files under > the root directory of the encryption zone will be encrypted/decrypted > transparently upon HDFS client write or read operations. > Generally, it does not support rename(without data copying) across encryption > zones or between encryption zone and non-encryption zone because different > security settings of encryption zones. However, there are certain use cases > where efficient rename support is desired. This JIRA is to propose better > support of two such use cases “Scratch Space” (a.k.a. staging area) and “Soft > Delete” (a.k.a. trash) with HDFS encryption zones. > “Scratch Space” is widely used in Hadoop jobs, which requires efficient > rename support. Temporary files from MR jobs are usually stored in staging > area outside encryption zone such as “/tmp” directory and then rename to > targeted directories as specified once the data is ready to be further > processed. > Below is a summary of supported/unsupported cases from latest Hadoop: > * Rename within the encryption zone is supported > * Rename the entire encryption zone by moving the root directory of the zone > is allowed. > * Rename sub-directory/file from encryption zone to non-encryption zone is > not allowed. > * Rename sub-directory/file from encryption zone A to encryption zone B is > not allowed. > * Rename from non-encryption zone to encryption zone is not allowed. > “Soft delete” (a.k.a. trash) is a client-side “soft delete” feature that > helps prevent accidental deletion of files and directories. If trash is > enabled and a file or directory is deleted using the Hadoop shell, the file > is moved to the .Trash directory of the user's home directory instead of > being deleted. Deleted files are initially moved (renamed) to the Current > sub-directory of the .Trash directory with original path being preserved. > Files and directories in the trash can be restored simply by moving them to a > location outside the .Trash directory. > Due to the limited rename support, delete sub-directory/file within > encryption zone with trash feature is not allowed. Client has to use > -skipTrash option to work around this. HADOOP-10902 and HDFS-6767 improved > the error message but without a complete solution to the problem. > We propose to solve the problem by generalizing the mapping between > encryption zone and its underlying HDFS directories from 1:1 today to 1:N. > The encryption zone should allow non-overlapped directories such as scratch > space or soft delete "trash" locations to be added/removed dynamically after > creation. This way, rename for "scratch space" and "soft delete" can be > better supported without breaking the assumption that rename is only > supported "within the zone". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8747) Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption Zones
[
https://issues.apache.org/jira/browse/HDFS-8747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14621518#comment-14621518
]
Yi Liu commented on HDFS-8747:
--
Thanks Xiaoyu for working on this. Improving the {{rename}} is good for user
and upper applications, I will take some time to go through your design and
post my comments.
> Provide Better "Scratch Space" and "Soft Delete" Support for HDFS Encryption
> Zones
> --
>
> Key: HDFS-8747
> URL: https://issues.apache.org/jira/browse/HDFS-8747
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Affects Versions: 2.6.0
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-8747-07092015.pdf
>
>
> HDFS Transparent Data Encryption At-Rest was introduced in Hadoop 2.6 to
> allow create encryption zone on top of a single HDFS directory. Files under
> the root directory of the encryption zone will be encrypted/decrypted
> transparently upon HDFS client write or read operations.
> Generally, it does not support rename(without data copying) across encryption
> zones or between encryption zone and non-encryption zone because different
> security settings of encryption zones. However, there are certain use cases
> where efficient rename support is desired. This JIRA is to propose better
> support of two such use cases “Scratch Space” (a.k.a. staging area) and “Soft
> Delete” (a.k.a. trash) with HDFS encryption zones.
> “Scratch Space” is widely used in Hadoop jobs, which requires efficient
> rename support. Temporary files from MR jobs are usually stored in staging
> area outside encryption zone such as “/tmp” directory and then rename to
> targeted directories as specified once the data is ready to be further
> processed.
> Below is a summary of supported/unsupported cases from latest Hadoop:
> * Rename within the encryption zone is supported
> * Rename the entire encryption zone by moving the root directory of the zone
> is allowed.
> * Rename sub-directory/file from encryption zone to non-encryption zone is
> not allowed.
> * Rename sub-directory/file from encryption zone A to encryption zone B is
> not allowed.
> * Rename from non-encryption zone to encryption zone is not allowed.
> “Soft delete” (a.k.a. trash) is a client-side “soft delete” feature that
> helps prevent accidental deletion of files and directories. If trash is
> enabled and a file or directory is deleted using the Hadoop shell, the file
> is moved to the .Trash directory of the user's home directory instead of
> being deleted. Deleted files are initially moved (renamed) to the Current
> sub-directory of the .Trash directory with original path being preserved.
> Files and directories in the trash can be restored simply by moving them to a
> location outside the .Trash directory.
> Due to the limited rename support, delete sub-directory/file within
> encryption zone with trash feature is not allowed. Client has to use
> -skipTrash option to work around this. HADOOP-10902 and HDFS-6767 improved
> the error message but without a complete solution to the problem.
> We propose to solve the problem by generalizing the mapping between
> encryption zone and its underlying HDFS directories from 1:1 today to 1:N.
> The encryption zone should allow non-overlapped directories such as scratch
> space or soft delete "trash" locations to be added/removed dynamically after
> creation. This way, rename for "scratch space" and "soft delete" can be
> better supported without breaking the assumption that rename is only
> supported "within the zone".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
