http://git-wip-us.apache.org/repos/asf/hadoop/blob/a45ef2b6/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsQuotaAdminGuide.md ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsQuotaAdminGuide.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsQuotaAdminGuide.md new file mode 100644 index 0000000..380604b --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsQuotaAdminGuide.md @@ -0,0 +1,92 @@ +<!--- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +HDFS Quotas Guide +================= + +* [HDFS Quotas Guide](#HDFS_Quotas_Guide) + * [Overview](#Overview) + * [Name Quotas](#Name_Quotas) + * [Space Quotas](#Space_Quotas) + * [Administrative Commands](#Administrative_Commands) + * [Reporting Command](#Reporting_Command) + +Overview +-------- + +The Hadoop Distributed File System (HDFS) allows the administrator to set quotas for the number of names used and the amount of space used for individual directories. Name quotas and space quotas operate independently, but the administration and implementation of the two types of quotas are closely parallel. + +Name Quotas +----------- + +The name quota is a hard limit on the number of file and directory names in the tree rooted at that directory. File and directory creations fail if the quota would be exceeded. Quotas stick with renamed directories; the rename operation fails if operation would result in a quota violation. The attempt to set a quota will still succeed even if the directory would be in violation of the new quota. A newly created directory has no associated quota. The largest quota is Long.Max\_Value. A quota of one forces a directory to remain empty. (Yes, a directory counts against its own quota!) + +Quotas are persistent with the fsimage. When starting, if the fsimage is immediately in violation of a quota (perhaps the fsimage was surreptitiously modified), a warning is printed for each of such violations. Setting or removing a quota creates a journal entry. + +Space Quotas +------------ + +The space quota is a hard limit on the number of bytes used by files in the tree rooted at that directory. Block allocations fail if the quota would not allow a full block to be written. Each replica of a block counts against the quota. Quotas stick with renamed directories; the rename operation fails if the operation would result in a quota violation. A newly created directory has no associated quota. The largest quota is `Long.Max_Value`. A quota of zero still permits files to be created, but no blocks can be added to the files. Directories don't use host file system space and don't count against the space quota. The host file system space used to save the file meta data is not counted against the quota. Quotas are charged at the intended replication factor for the file; changing the replication factor for a file will credit or debit quotas. + +Quotas are persistent with the fsimage. When starting, if the fsimage is immediately in violation of a quota (perhaps the fsimage was surreptitiously modified), a warning is printed for each of such violations. Setting or removing a quota creates a journal entry. + +Administrative Commands +----------------------- + +Quotas are managed by a set of commands available only to the administrator. + +* `hdfs dfsadmin -setQuota <N> <directory>...<directory>` + + Set the name quota to be N for each directory. Best effort for each + directory, with faults reported if N is not a positive long + integer, the directory does not exist or it is a file, or the + directory would immediately exceed the new quota. + +* `hdfs dfsadmin -clrQuota <directory>...<directory>` + + Remove any name quota for each directory. Best effort for each + directory, with faults reported if the directory does not exist or + it is a file. It is not a fault if the directory has no quota. + +* `hdfs dfsadmin -setSpaceQuota <N> <directory>...<directory>` + + Set the space quota to be N bytes for each directory. This is a + hard limit on total size of all the files under the directory tree. + The space quota takes replication also into account, i.e. one GB of + data with replication of 3 consumes 3GB of quota. N can also be + specified with a binary prefix for convenience, for e.g. 50g for 50 + gigabytes and 2t for 2 terabytes etc. Best effort for each + directory, with faults reported if N is neither zero nor a positive + integer, the directory does not exist or it is a file, or the + directory would immediately exceed the new quota. + +* `hdfs dfsadmin -clrSpaceQuota <directory>...<directory>` + + Remove any space quota for each directory. Best effort for each + directory, with faults reported if the directory does not exist or + it is a file. It is not a fault if the directory has no quota. + +Reporting Command +----------------- + +An an extension to the count command of the HDFS shell reports quota values and the current count of names and bytes in use. + +* `hadoop fs -count -q <directory>...<directory>` + + With the -q option, also report the name quota value set for each + directory, the available name quota remaining, the space quota + value set, and the available space quota remaining. If the + directory does not have a quota set, the reported values are `none` + and `inf`. +
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a45ef2b6/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsUserGuide.md ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsUserGuide.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsUserGuide.md new file mode 100644 index 0000000..37fa4be --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsUserGuide.md @@ -0,0 +1,375 @@ +<!--- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +HDFS Users Guide +================ + +* [HDFS Users Guide](#HDFS_Users_Guide) + * [Purpose](#Purpose) + * [Overview](#Overview) + * [Prerequisites](#Prerequisites) + * [Web Interface](#Web_Interface) + * [Shell Commands](#Shell_Commands) + * [DFSAdmin Command](#DFSAdmin_Command) + * [Secondary NameNode](#Secondary_NameNode) + * [Checkpoint Node](#Checkpoint_Node) + * [Backup Node](#Backup_Node) + * [Import Checkpoint](#Import_Checkpoint) + * [Balancer](#Balancer) + * [Rack Awareness](#Rack_Awareness) + * [Safemode](#Safemode) + * [fsck](#fsck) + * [fetchdt](#fetchdt) + * [Recovery Mode](#Recovery_Mode) + * [Upgrade and Rollback](#Upgrade_and_Rollback) + * [DataNode Hot Swap Drive](#DataNode_Hot_Swap_Drive) + * [File Permissions and Security](#File_Permissions_and_Security) + * [Scalability](#Scalability) + * [Related Documentation](#Related_Documentation) + +Purpose +------- + +This document is a starting point for users working with Hadoop Distributed File System (HDFS) either as a part of a Hadoop cluster or as a stand-alone general purpose distributed file system. While HDFS is designed to "just work" in many environments, a working knowledge of HDFS helps greatly with configuration improvements and diagnostics on a specific cluster. + +Overview +-------- + +HDFS is the primary distributed storage used by Hadoop applications. A HDFS cluster primarily consists of a NameNode that manages the file system metadata and DataNodes that store the actual data. The HDFS Architecture Guide describes HDFS in detail. This user guide primarily deals with the interaction of users and administrators with HDFS clusters. The HDFS architecture diagram depicts basic interactions among NameNode, the DataNodes, and the clients. Clients contact NameNode for file metadata or file modifications and perform actual file I/O directly with the DataNodes. + +The following are some of the salient features that could be of interest to many users. + +* Hadoop, including HDFS, is well suited for distributed storage and + distributed processing using commodity hardware. It is fault + tolerant, scalable, and extremely simple to expand. MapReduce, well + known for its simplicity and applicability for large set of + distributed applications, is an integral part of Hadoop. + +* HDFS is highly configurable with a default configuration well + suited for many installations. Most of the time, configuration + needs to be tuned only for very large clusters. + +* Hadoop is written in Java and is supported on all major platforms. + +* Hadoop supports shell-like commands to interact with HDFS directly. + +* The NameNode and Datanodes have built in web servers that makes it + easy to check current status of the cluster. + +* New features and improvements are regularly implemented in HDFS. + The following is a subset of useful features in HDFS: + + * File permissions and authentication. + + * Rack awareness: to take a node's physical location into + account while scheduling tasks and allocating storage. + + * Safemode: an administrative mode for maintenance. + + * `fsck`: a utility to diagnose health of the file system, to find + missing files or blocks. + + * `fetchdt`: a utility to fetch DelegationToken and store it in a + file on the local system. + + * Balancer: tool to balance the cluster when the data is + unevenly distributed among DataNodes. + + * Upgrade and rollback: after a software upgrade, it is possible + to rollback to HDFS' state before the upgrade in case of unexpected problems. + + * Secondary NameNode: performs periodic checkpoints of the + namespace and helps keep the size of file containing log of + HDFS modifications within certain limits at the NameNode. + + * Checkpoint node: performs periodic checkpoints of the + namespace and helps minimize the size of the log stored at the + NameNode containing changes to the HDFS. Replaces the role + previously filled by the Secondary NameNode, though is not yet + battle hardened. The NameNode allows multiple Checkpoint nodes + simultaneously, as long as there are no Backup nodes + registered with the system. + + * Backup node: An extension to the Checkpoint node. In addition + to checkpointing it also receives a stream of edits from the + NameNode and maintains its own in-memory copy of the + namespace, which is always in sync with the active NameNode + namespace state. Only one Backup node may be registered with + the NameNode at once. + +Prerequisites +------------- + +The following documents describe how to install and set up a Hadoop cluster: + +* [Single Node Setup](../hadoop-common/SingleCluster.html) for first-time users. +* [Cluster Setup](../hadoop-common/ClusterSetup.html) for large, distributed clusters. + +The rest of this document assumes the user is able to set up and run a HDFS with at least one DataNode. For the purpose of this document, both the NameNode and DataNode could be running on the same physical machine. + +Web Interface +------------- + +NameNode and DataNode each run an internal web server in order to display basic information about the current status of the cluster. With the default configuration, the NameNode front page is at `http://namenode-name:50070/`. It lists the DataNodes in the cluster and basic statistics of the cluster. The web interface can also be used to browse the file system (using "Browse the file system" link on the NameNode front page). + +Shell Commands +-------------- + +Hadoop includes various shell-like commands that directly interact with HDFS and other file systems that Hadoop supports. The command `bin/hdfs dfs -help` lists the commands supported by Hadoop shell. Furthermore, the command `bin/hdfs dfs -help command-name` displays more detailed help for a command. These commands support most of the normal files system operations like copying files, changing file permissions, etc. It also supports a few HDFS specific operations like changing replication of files. For more information see [File System Shell Guide](../hadoop-common/FileSystemShell.html). + +### DFSAdmin Command + +The `bin/hdfs dfsadmin` command supports a few HDFS administration related operations. The `bin/hdfs dfsadmin -help` command lists all the commands currently supported. For e.g.: + +* `-report`: reports basic statistics of HDFS. Some of this + information is also available on the NameNode front page. + +* `-safemode`: though usually not required, an administrator can + manually enter or leave Safemode. + +* `-finalizeUpgrade`: removes previous backup of the cluster made + during last upgrade. + +* `-refreshNodes`: Updates the namenode with the set of datanodes + allowed to connect to the namenode. Namenodes re-read datanode + hostnames in the file defined by `dfs.hosts`, `dfs.hosts.exclude` + Hosts defined in `dfs.hosts` are the datanodes that are part of the + cluster. If there are entries in `dfs.hosts`, only the hosts in it + are allowed to register with the namenode. Entries in + `dfs.hosts.exclude` are datanodes that need to be decommissioned. + Datanodes complete decommissioning when all the replicas from them + are replicated to other datanodes. Decommissioned nodes are not + automatically shutdown and are not chosen for writing for new + replicas. + +* `-printTopology` : Print the topology of the cluster. Display a tree + of racks and datanodes attached to the tracks as viewed by the + NameNode. + +For command usage, see [dfsadmin](./HDFSCommands.html#dfsadmin). + +Secondary NameNode +------------------ + +The NameNode stores modifications to the file system as a log appended to a native file system file, edits. When a NameNode starts up, it reads HDFS state from an image file, fsimage, and then applies edits from the edits log file. It then writes new HDFS state to the fsimage and starts normal operation with an empty edits file. Since NameNode merges fsimage and edits files only during start up, the edits log file could get very large over time on a busy cluster. Another side effect of a larger edits file is that next restart of NameNode takes longer. + +The secondary NameNode merges the fsimage and the edits log files periodically and keeps edits log size within a limit. It is usually run on a different machine than the primary NameNode since its memory requirements are on the same order as the primary NameNode. + +The start of the checkpoint process on the secondary NameNode is controlled by two configuration parameters. + +* `dfs.namenode.checkpoint.period`, set to 1 hour by default, specifies + the maximum delay between two consecutive checkpoints, and + +* `dfs.namenode.checkpoint.txns`, set to 1 million by default, defines the + number of uncheckpointed transactions on the NameNode which will + force an urgent checkpoint, even if the checkpoint period has not + been reached. + +The secondary NameNode stores the latest checkpoint in a directory which is structured the same way as the primary NameNode's directory. So that the check pointed image is always ready to be read by the primary NameNode if necessary. + +For command usage, see [secondarynamenode](./HDFSCommands.html#secondarynamenode). + +Checkpoint Node +--------------- + +NameNode persists its namespace using two files: fsimage, which is the latest checkpoint of the namespace and edits, a journal (log) of changes to the namespace since the checkpoint. When a NameNode starts up, it merges the fsimage and edits journal to provide an up-to-date view of the file system metadata. The NameNode then overwrites fsimage with the new HDFS state and begins a new edits journal. + +The Checkpoint node periodically creates checkpoints of the namespace. It downloads fsimage and edits from the active NameNode, merges them locally, and uploads the new image back to the active NameNode. The Checkpoint node usually runs on a different machine than the NameNode since its memory requirements are on the same order as the NameNode. The Checkpoint node is started by bin/hdfs namenode -checkpoint on the node specified in the configuration file. + +The location of the Checkpoint (or Backup) node and its accompanying web interface are configured via the `dfs.namenode.backup.address` and `dfs.namenode.backup.http-address` configuration variables. + +The start of the checkpoint process on the Checkpoint node is controlled by two configuration parameters. + +* `dfs.namenode.checkpoint.period`, set to 1 hour by default, specifies + the maximum delay between two consecutive checkpoints + +* `dfs.namenode.checkpoint.txns`, set to 1 million by default, defines the + number of uncheckpointed transactions on the NameNode which will + force an urgent checkpoint, even if the checkpoint period has not + been reached. + +The Checkpoint node stores the latest checkpoint in a directory that is structured the same as the NameNode's directory. This allows the checkpointed image to be always available for reading by the NameNode if necessary. See Import checkpoint. + +Multiple checkpoint nodes may be specified in the cluster configuration file. + +For command usage, see [namenode](./HDFSCommands.html#namenode). + +Backup Node +----------- + +The Backup node provides the same checkpointing functionality as the Checkpoint node, as well as maintaining an in-memory, up-to-date copy of the file system namespace that is always synchronized with the active NameNode state. Along with accepting a journal stream of file system edits from the NameNode and persisting this to disk, the Backup node also applies those edits into its own copy of the namespace in memory, thus creating a backup of the namespace. + +The Backup node does not need to download fsimage and edits files from the active NameNode in order to create a checkpoint, as would be required with a Checkpoint node or Secondary NameNode, since it already has an up-to-date state of the namespace state in memory. The Backup node checkpoint process is more efficient as it only needs to save the namespace into the local fsimage file and reset edits. + +As the Backup node maintains a copy of the namespace in memory, its RAM requirements are the same as the NameNode. + +The NameNode supports one Backup node at a time. No Checkpoint nodes may be registered if a Backup node is in use. Using multiple Backup nodes concurrently will be supported in the future. + +The Backup node is configured in the same manner as the Checkpoint node. It is started with `bin/hdfs namenode -backup`. + +The location of the Backup (or Checkpoint) node and its accompanying web interface are configured via the `dfs.namenode.backup.address` and `dfs.namenode.backup.http-address` configuration variables. + +Use of a Backup node provides the option of running the NameNode with no persistent storage, delegating all responsibility for persisting the state of the namespace to the Backup node. To do this, start the NameNode with the `-importCheckpoint` option, along with specifying no persistent storage directories of type edits `dfs.namenode.edits.dir` for the NameNode configuration. + +For a complete discussion of the motivation behind the creation of the Backup node and Checkpoint node, see [HADOOP-4539](https://issues.apache.org/jira/browse/HADOOP-4539). For command usage, see [namenode](./HDFSCommands.html#namenode). + +Import Checkpoint +----------------- + +The latest checkpoint can be imported to the NameNode if all other copies of the image and the edits files are lost. In order to do that one should: + +* Create an empty directory specified in the `dfs.namenode.name.dir` + configuration variable; + +* Specify the location of the checkpoint directory in the + configuration variable `dfs.namenode.checkpoint.dir`; + +* and start the NameNode with `-importCheckpoint` option. + +The NameNode will upload the checkpoint from the `dfs.namenode.checkpoint.dir` directory and then save it to the NameNode directory(s) set in `dfs.namenode.name.dir`. The NameNode will fail if a legal image is contained in `dfs.namenode.name.dir`. The NameNode verifies that the image in `dfs.namenode.checkpoint.dir` is consistent, but does not modify it in any way. + +For command usage, see [namenode](./HDFSCommands.html#namenode). + +Balancer +-------- + +HDFS data might not always be be placed uniformly across the DataNode. One common reason is addition of new DataNodes to an existing cluster. While placing new blocks (data for a file is stored as a series of blocks), NameNode considers various parameters before choosing the DataNodes to receive these blocks. Some of the considerations are: + +* Policy to keep one of the replicas of a block on the same node as + the node that is writing the block. + +* Need to spread different replicas of a block across the racks so + that cluster can survive loss of whole rack. + +* One of the replicas is usually placed on the same rack as the node + writing to the file so that cross-rack network I/O is reduced. + +* Spread HDFS data uniformly across the DataNodes in the cluster. + +Due to multiple competing considerations, data might not be uniformly placed across the DataNodes. HDFS provides a tool for administrators that analyzes block placement and rebalanaces data across the DataNode. A brief administrator's guide for balancer is available at [HADOOP-1652](https://issues.apache.org/jira/browse/HADOOP-1652). + +For command usage, see [balancer](./HDFSCommands.html#balancer). + +Rack Awareness +-------------- + +Typically large Hadoop clusters are arranged in racks and network traffic between different nodes with in the same rack is much more desirable than network traffic across the racks. In addition NameNode tries to place replicas of block on multiple racks for improved fault tolerance. Hadoop lets the cluster administrators decide which rack a node belongs to through configuration variable `net.topology.script.file.name`. When this script is configured, each node runs the script to determine its rack id. A default installation assumes all the nodes belong to the same rack. This feature and configuration is further described in PDF attached to [HADOOP-692](https://issues.apache.org/jira/browse/HADOOP-692). + +Safemode +-------- + +During start up the NameNode loads the file system state from the fsimage and the edits log file. It then waits for DataNodes to report their blocks so that it does not prematurely start replicating the blocks though enough replicas already exist in the cluster. During this time NameNode stays in Safemode. Safemode for the NameNode is essentially a read-only mode for the HDFS cluster, where it does not allow any modifications to file system or blocks. Normally the NameNode leaves Safemode automatically after the DataNodes have reported that most file system blocks are available. If required, HDFS could be placed in Safemode explicitly using `bin/hdfs dfsadmin -safemode` command. NameNode front page shows whether Safemode is on or off. A more detailed description and configuration is maintained as JavaDoc for `setSafeMode()`. + +fsck +---- + +HDFS supports the fsck command to check for various inconsistencies. It it is designed for reporting problems with various files, for example, missing blocks for a file or under-replicated blocks. Unlike a traditional fsck utility for native file systems, this command does not correct the errors it detects. Normally NameNode automatically corrects most of the recoverable failures. By default fsck ignores open files but provides an option to select all files during reporting. The HDFS fsck command is not a Hadoop shell command. It can be run as `bin/hdfs fsck`. For command usage, see [fsck](./HDFSCommands.html#fsck). fsck can be run on the whole file system or on a subset of files. + +fetchdt +------- + +HDFS supports the fetchdt command to fetch Delegation Token and store it in a file on the local system. This token can be later used to access secure server (NameNode for example) from a non secure client. Utility uses either RPC or HTTPS (over Kerberos) to get the token, and thus requires kerberos tickets to be present before the run (run kinit to get the tickets). The HDFS fetchdt command is not a Hadoop shell command. It can be run as `bin/hdfs fetchdt DTfile`. After you got the token you can run an HDFS command without having Kerberos tickets, by pointing `HADOOP_TOKEN_FILE_LOCATION` environmental variable to the delegation token file. For command usage, see [fetchdt](./HDFSCommands.html#fetchdt) command. + +Recovery Mode +------------- + +Typically, you will configure multiple metadata storage locations. Then, if one storage location is corrupt, you can read the metadata from one of the other storage locations. + +However, what can you do if the only storage locations available are corrupt? In this case, there is a special NameNode startup mode called Recovery mode that may allow you to recover most of your data. + +You can start the NameNode in recovery mode like so: `namenode -recover` + +When in recovery mode, the NameNode will interactively prompt you at the command line about possible courses of action you can take to recover your data. + +If you don't want to be prompted, you can give the `-force` option. This option will force recovery mode to always select the first choice. Normally, this will be the most reasonable choice. + +Because Recovery mode can cause you to lose data, you should always back up your edit log and fsimage before using it. + +Upgrade and Rollback +-------------------- + +When Hadoop is upgraded on an existing cluster, as with any software upgrade, it is possible there are new bugs or incompatible changes that affect existing applications and were not discovered earlier. In any non-trivial HDFS installation, it is not an option to loose any data, let alone to restart HDFS from scratch. HDFS allows administrators to go back to earlier version of Hadoop and rollback the cluster to the state it was in before the upgrade. HDFS upgrade is described in more detail in [Hadoop Upgrade](http://wiki.apache.org/hadoop/Hadoop_Upgrade) Wiki page. HDFS can have one such backup at a time. Before upgrading, administrators need to remove existing backup using bin/hadoop dfsadmin `-finalizeUpgrade` command. The following briefly describes the typical upgrade procedure: + +* Before upgrading Hadoop software, finalize if there an existing + backup. `dfsadmin -upgradeProgress` status can tell if the cluster + needs to be finalized. + +* Stop the cluster and distribute new version of Hadoop. + +* Run the new version with `-upgrade` option (`bin/start-dfs.sh -upgrade`). + +* Most of the time, cluster works just fine. Once the new HDFS is + considered working well (may be after a few days of operation), + finalize the upgrade. Note that until the cluster is finalized, + deleting the files that existed before the upgrade does not free up + real disk space on the DataNodes. + +* If there is a need to move back to the old version, + + * stop the cluster and distribute earlier version of Hadoop. + + * start the cluster with rollback option. (`bin/start-dfs.sh -rollback`). + +When upgrading to a new version of HDFS, it is necessary to rename or delete any paths that are reserved in the new version of HDFS. If the NameNode encounters a reserved path during upgrade, it will print an error like the following: + +` /.reserved is a reserved path and .snapshot is a reserved path component in this version of HDFS. Please rollback and delete or rename this path, or upgrade with the -renameReserved [key-value pairs] option to automatically rename these paths during upgrade.` + +Specifying `-upgrade -renameReserved [optional key-value pairs]` causes the NameNode to automatically rename any reserved paths found during startup. For example, to rename all paths named `.snapshot` to `.my-snapshot` and `.reserved` to `.my-reserved`, a user would specify `-upgrade -renameReserved .snapshot=.my-snapshot,.reserved=.my-reserved`. + +If no key-value pairs are specified with `-renameReserved`, the NameNode will then suffix reserved paths with `.<LAYOUT-VERSION>.UPGRADE_RENAMED`, e.g. `.snapshot.-51.UPGRADE_RENAMED`. + +There are some caveats to this renaming process. It's recommended, if possible, to first `hdfs dfsadmin -saveNamespace` before upgrading. This is because data inconsistency can result if an edit log operation refers to the destination of an automatically renamed file. + +DataNode Hot Swap Drive +----------------------- + +Datanode supports hot swappable drives. The user can add or replace HDFS data volumes without shutting down the DataNode. The following briefly describes the typical hot swapping drive procedure: + +* If there are new storage directories, the user should format them and mount them + appropriately. + +* The user updates the DataNode configuration `dfs.datanode.data.dir` + to reflect the data volume directories that will be actively in use. + +* The user runs `dfsadmin -reconfig datanode HOST:PORT start` to start + the reconfiguration process. The user can use + `dfsadmin -reconfig datanode HOST:PORT status` + to query the running status of the reconfiguration task. + +* Once the reconfiguration task has completed, the user can safely `umount` + the removed data volume directories and physically remove the disks. + +File Permissions and Security +----------------------------- + +The file permissions are designed to be similar to file permissions on other familiar platforms like Linux. Currently, security is limited to simple file permissions. The user that starts NameNode is treated as the superuser for HDFS. Future versions of HDFS will support network authentication protocols like Kerberos for user authentication and encryption of data transfers. The details are discussed in the Permissions Guide. + +Scalability +----------- + +Hadoop currently runs on clusters with thousands of nodes. The [PoweredBy](http://wiki.apache.org/hadoop/PoweredBy) Wiki page lists some of the organizations that deploy Hadoop on large clusters. HDFS has one NameNode for each cluster. Currently the total memory available on NameNode is the primary scalability limitation. On very large clusters, increasing average size of files stored in HDFS helps with increasing cluster size without increasing memory requirements on NameNode. The default configuration may not suite very large clusters. The [FAQ](http://wiki.apache.org/hadoop/FAQ) Wiki page lists suggested configuration improvements for large Hadoop clusters. + +Related Documentation +--------------------- + +This user guide is a good starting point for working with HDFS. While the user guide continues to improve, there is a large wealth of documentation about Hadoop and HDFS. The following list is a starting point for further exploration: + +* [Hadoop Site](http://hadoop.apache.org): The home page for the Apache Hadoop site. +* [Hadoop Wiki](http://wiki.apache.org/hadoop/FrontPage): The home page (FrontPage) for the Hadoop Wiki. Unlike the released documentation, which is part of Hadoop source tree, Hadoop Wiki is regularly edited by Hadoop Community. +* [FAQ](http://wiki.apache.org/hadoop/FAQ): The FAQ Wiki page. +* [Hadoop JavaDoc API](../../api/index.html). +* Hadoop User Mailing List: user[at]hadoop.apache.org. +* Explore [hdfs-default.xml](./hdfs-default.xml). It includes brief description of most of the configuration variables available. +* [HDFS Commands Guide](./HDFSCommands.html): HDFS commands usage. + http://git-wip-us.apache.org/repos/asf/hadoop/blob/a45ef2b6/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/LibHdfs.md ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/LibHdfs.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/LibHdfs.md new file mode 100644 index 0000000..7b869c8 --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/LibHdfs.md @@ -0,0 +1,92 @@ +<!--- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +C API libhdfs +============= + +* [C API libhdfs](#C_API_libhdfs) + * [Overview](#Overview) + * [The APIs](#The_APIs) + * [A Sample Program](#A_Sample_Program) + * [How To Link With The Library](#How_To_Link_With_The_Library) + * [Common Problems](#Common_Problems) + * [Thread Safe](#Thread_Safe) + +Overview +-------- + +libhdfs is a JNI based C API for Hadoop's Distributed File System (HDFS). It provides C APIs to a subset of the HDFS APIs to manipulate HDFS files and the filesystem. libhdfs is part of the Hadoop distribution and comes pre-compiled in `$HADOOP_HDFS_HOME/lib/native/libhdfs.so` . libhdfs is compatible with Windows and can be built on Windows by running `mvn compile` within the `hadoop-hdfs-project/hadoop-hdfs` directory of the source tree. + +The APIs +-------- + +The libhdfs APIs are a subset of the [Hadoop FileSystem APIs](../../api/org/apache/hadoop/fs/FileSystem.html). + +The header file for libhdfs describes each API in detail and is available in `$HADOOP_HDFS_HOME/include/hdfs.h`. + +A Sample Program +---------------- +```c +#include "hdfs.h" + +int main(int argc, char **argv) { + + hdfsFS fs = hdfsConnect("default", 0); + const char* writePath = "/tmp/testfile.txt"; + hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY |O_CREAT, 0, 0, 0); + if(!writeFile) { + fprintf(stderr, "Failed to open %s for writing!\n", writePath); + exit(-1); + } + char* buffer = "Hello, World!"; + tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer, strlen(buffer)+1); + if (hdfsFlush(fs, writeFile)) { + fprintf(stderr, "Failed to 'flush' %s\n", writePath); + exit(-1); + } + hdfsCloseFile(fs, writeFile); +} +``` + +How To Link With The Library +---------------------------- + +See the CMake file for `test_libhdfs_ops.c` in the libhdfs source directory (`hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt`) or something like: `gcc above_sample.c -I$HADOOP_HDFS_HOME/include -L$HADOOP_HDFS_HOME/lib/native -lhdfs -o above_sample` + +Common Problems +--------------- + +The most common problem is the `CLASSPATH` is not set properly when calling a program that uses libhdfs. Make sure you set it to all the Hadoop jars needed to run Hadoop itself as well as the right configuration directory containing `hdfs-site.xml`. It is not valid to use wildcard syntax for specifying multiple jars. It may be useful to run `hadoop classpath --glob` or `hadoop classpath --jar <path`\> to generate the correct classpath for your deployment. See [Hadoop Commands Reference](../hadoop-common/CommandsManual.html#classpath) for more information on this command. + +Thread Safe +----------- + +libdhfs is thread safe. + +* Concurrency and Hadoop FS "handles" + + The Hadoop FS implementation includes a FS handle cache which + caches based on the URI of the namenode along with the user + connecting. So, all calls to `hdfsConnect` will return the same + handle but calls to `hdfsConnectAsUser` with different users will + return different handles. But, since HDFS client handles are + completely thread safe, this has no bearing on concurrency. + +* Concurrency and libhdfs/JNI + + The libhdfs calls to JNI should always be creating thread local + storage, so (in theory), libhdfs should be as thread safe as the + underlying calls to the Hadoop FS. + + http://git-wip-us.apache.org/repos/asf/hadoop/blob/a45ef2b6/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/SLGUserGuide.md ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/SLGUserGuide.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/SLGUserGuide.md new file mode 100644 index 0000000..48b92c2 --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/SLGUserGuide.md @@ -0,0 +1,157 @@ +<!--- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +Synthetic Load Generator Guide +============================== + +* [Synthetic Load Generator Guide](#Synthetic_Load_Generator_Guide) + * [Overview](#Overview) + * [Synopsis](#Synopsis) + * [Test Space Population](#Test_Space_Population) + * [Structure Generator](#Structure_Generator) + * [Data Generator](#Data_Generator) + +Overview +-------- + +The synthetic load generator (SLG) is a tool for testing NameNode behavior under different client loads. The user can generate different mixes of read, write, and list requests by specifying the probabilities of read and write. The user controls the intensity of the load by adjusting parameters for the number of worker threads and the delay between operations. While load generators are running, the user can profile and monitor the running of the NameNode. When a load generator exits, it prints some NameNode statistics like the average execution time of each kind of operation and the NameNode throughput. + +Synopsis +-------- + +The synopsis of the command is: + + java LoadGenerator [options] + +Options include: + +* `-readProbability` *read probability* + + The probability of the read operation; default is 0.3333. + +* `-writeProbability` *write probability* + + The probability of the write operations; default is 0.3333. + +* `-root` *test space root* + + The root of the test space; default is /testLoadSpace. + +* `-maxDelayBetweenOps` *maxDelayBetweenOpsInMillis* + + The maximum delay between two consecutive operations in a thread; + default is 0 indicating no delay. + +* `-numOfThreads` *numOfThreads* + + The number of threads to spawn; default is 200. + +* `-elapsedTime` *elapsedTimeInSecs* + + The number of seconds that the program will run; A value of zero + indicates that the program runs forever. The default value is 0. + +* `-startTime` *startTimeInMillis* + + The time that all worker threads start to run. By default it is 10 + seconds after the main program starts running.This creates a + barrier if more than one load generator is running. + +* `-seed` *seed* + + The random generator seed for repeating requests to NameNode when + running with a single thread; default is the current time. + +After command line argument parsing, the load generator traverses the test space and builds a table of all directories and another table of all files in the test space. It then waits until the start time to spawn the number of worker threads as specified by the user. Each thread sends a stream of requests to NameNode. At each iteration, it first decides if it is going to read a file, create a file, or list a directory following the read and write probabilities specified by the user. The listing probability is equal to 1-read probability-write probability. When reading, it randomly picks a file in the test space and reads the entire file. When writing, it randomly picks a directory in the test space and creates a file there. + +To avoid two threads with the same load generator or from two different load generators creating the same file, the file name consists of the current machine's host name and the thread id. The length of the file follows Gaussian distribution with an average size of 2 blocks and the standard deviation of 1. The new file is filled with byte 'a'. To avoid the test space growing indefinitely, the file is deleted immediately after the file creation completes. While listing, it randomly picks a directory in the test space and lists its content. + +After an operation completes, the thread pauses for a random amount of time in the range of [0, maxDelayBetweenOps] if the specified maximum delay is not zero. All threads are stopped when the specified elapsed time is passed. Before exiting, the program prints the average execution for each kind of NameNode operations, and the number of requests served by the NameNode per second. + +Test Space Population +--------------------- + +The user needs to populate a test space before running a load generator. The structure generator generates a random test space structure and the data generator creates the files and directories of the test space in Hadoop distributed file system. + +### Structure Generator + +This tool generates a random namespace structure with the following constraints: + +1. The number of subdirectories that a directory can have is a random + number in [minWidth, maxWidth]. + +2. The maximum depth of each subdirectory is a random number + [2\*maxDepth/3, maxDepth]. + +3. Files are randomly placed in leaf directories. The size of each + file follows Gaussian distribution with an average size of 1 block + and a standard deviation of 1. + +The generated namespace structure is described by two files in the output directory. Each line of the first file contains the full name of a leaf directory. Each line of the second file contains the full name of a file and its size, separated by a blank. + +The synopsis of the command is: + + java StructureGenerator [options] + +Options include: + +* `-maxDepth` *maxDepth* + + Maximum depth of the directory tree; default is 5. + +* `-minWidth` *minWidth* + + Minimum number of subdirectories per directories; default is 1. + +* `-maxWidth` *maxWidth* + + Maximum number of subdirectories per directories; default is 5. + +* `-numOfFiles` *\#OfFiles* + + The total number of files in the test space; default is 10. + +* `-avgFileSize` *avgFileSizeInBlocks* + + Average size of blocks; default is 1. + +* `-outDir` *outDir* + + Output directory; default is the current directory. + +* `-seed` *seed* + + Random number generator seed; default is the current time. + +### Data Generator + +This tool reads the directory structure and file structure from the input directory and creates the namespace in Hadoop distributed file system. All files are filled with byte 'a'. + +The synopsis of the command is: + + java DataGenerator [options] + +Options include: + +* `-inDir` *inDir* + + Input directory name where directory/file structures are stored; + default is the current directory. + +* `-root` *test space root* + + The name of the root directory which the new namespace is going to + be placed under; default is "/testLoadSpace". + + http://git-wip-us.apache.org/repos/asf/hadoop/blob/a45ef2b6/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ShortCircuitLocalReads.md ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ShortCircuitLocalReads.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ShortCircuitLocalReads.md new file mode 100644 index 0000000..5150a3c --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ShortCircuitLocalReads.md @@ -0,0 +1,87 @@ +<!--- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +HDFS Short-Circuit Local Reads +============================== + +* [HDFS Short-Circuit Local Reads](#HDFS_Short-Circuit_Local_Reads) + * [Short-Circuit Local Reads](#Short-Circuit_Local_Reads) + * [Background](#Background) + * [Setup](#Setup) + * [Example Configuration](#Example_Configuration) + * [Legacy HDFS Short-Circuit Local Reads](#Legacy_HDFS_Short-Circuit_Local_Reads) + +Short-Circuit Local Reads +------------------------- + +### Background + +In `HDFS`, reads normally go through the `DataNode`. Thus, when the client asks the `DataNode` to read a file, the `DataNode` reads that file off of the disk and sends the data to the client over a TCP socket. So-called "short-circuit" reads bypass the `DataNode`, allowing the client to read the file directly. Obviously, this is only possible in cases where the client is co-located with the data. Short-circuit reads provide a substantial performance boost to many applications. + +### Setup + +To configure short-circuit local reads, you will need to enable `libhadoop.so`. See [Native Libraries](../hadoop-common/NativeLibraries.html) for details on enabling this library. + +Short-circuit reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the `DataNode`s to communicate. You will need to set a path to this socket. The `DataNode` needs to be able to create this path. On the other hand, it should not be possible for any user except the HDFS user or root to create this path. For this reason, paths under `/var/run` or `/var/lib` are often used. + +The client and the `DataNode` exchange information via a shared memory segment on `/dev/shm`. + +Short-circuit local reads need to be configured on both the `DataNode` and the client. + +### Example Configuration + +Here is an example configuration. + +```xml +<configuration> + <property> + <name>dfs.client.read.shortcircuit</name> + <value>true</value> + </property> + <property> + <name>dfs.domain.socket.path</name> + <value>/var/lib/hadoop-hdfs/dn_socket</value> + </property> +</configuration> +``` + +Legacy HDFS Short-Circuit Local Reads +------------------------------------- + +Legacy implementation of short-circuit local reads on which the clients directly open the HDFS block files is still available for platforms other than the Linux. Setting the value of `dfs.client.use.legacy.blockreader.local` in addition to `dfs.client.read.shortcircuit` to true enables this feature. + +You also need to set the value of `dfs.datanode.data.dir.perm` to `750` instead of the default `700` and chmod/chown the directory tree under `dfs.datanode.data.dir` as readable to the client and the `DataNode`. You must take caution because this means that the client can read all of the block files bypassing HDFS permission. + +Because Legacy short-circuit local reads is insecure, access to this feature is limited to the users listed in the value of `dfs.block.local-path-access.user`. + +```xml +<configuration> + <property> + <name>dfs.client.read.shortcircuit</name> + <value>true</value> + </property> + <property> + <name>dfs.client.use.legacy.blockreader.local</name> + <value>true</value> + </property> + <property> + <name>dfs.datanode.data.dir.perm</name> + <value>750</value> + </property> + <property> + <name>dfs.block.local-path-access.user</name> + <value>foo,bar</value> + </property> +</configuration> +``` http://git-wip-us.apache.org/repos/asf/hadoop/blob/a45ef2b6/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/TransparentEncryption.md ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/TransparentEncryption.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/TransparentEncryption.md new file mode 100644 index 0000000..aa2acbd --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/TransparentEncryption.md @@ -0,0 +1,268 @@ +<!--- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +Transparent Encryption in HDFS +============================== + +* [Overview](#Overview) +* [Background](#Background) +* [Use Cases](#Use_Cases) +* [Architecture](#Architecture) + * [Overview](#Overview) + * [Accessing data within an encryption zone](#Accessing_data_within_an_encryption_zone) + * [Key Management Server, KeyProvider, EDEKs](#Key_Management_Server_KeyProvider_EDEKs) +* [Configuration](#Configuration) + * [Configuring the cluster KeyProvider](#Configuring_the_cluster_KeyProvider) + * [Selecting an encryption algorithm and codec](#Selecting_an_encryption_algorithm_and_codec) + * [Namenode configuration](#Namenode_configuration) +* [crypto command-line interface](#crypto_command-line_interface) + * [createZone](#createZone) + * [listZones](#listZones) +* [Example usage](#Example_usage) +* [Distcp considerations](#Distcp_considerations) + * [Running as the superuser](#Running_as_the_superuser) + * [Copying between encrypted and unencrypted locations](#Copying_between_encrypted_and_unencrypted_locations) +* [Attack vectors](#Attack_vectors) + * [Hardware access exploits](#Hardware_access_exploits) + * [Root access exploits](#Root_access_exploits) + * [HDFS admin exploits](#HDFS_admin_exploits) + * [Rogue user exploits](#Rogue_user_exploits) + +Overview +-------- + +HDFS implements *transparent*, *end-to-end* encryption. Once configured, data read from and written to special HDFS directories is *transparently* encrypted and decrypted without requiring changes to user application code. This encryption is also *end-to-end*, which means the data can only be encrypted and decrypted by the client. HDFS never stores or has access to unencrypted data or unencrypted data encryption keys. This satisfies two typical requirements for encryption: *at-rest encryption* (meaning data on persistent media, such as a disk) as well as *in-transit encryption* (e.g. when data is travelling over the network). + +Background +---------- + +Encryption can be done at different layers in a traditional data management software/hardware stack. Choosing to encrypt at a given layer comes with different advantages and disadvantages. + +* **Application-level encryption**. This is the most secure and most flexible approach. The application has ultimate control over what is encrypted and can precisely reflect the requirements of the user. However, writing applications to do this is hard. This is also not an option for customers of existing applications that do not support encryption. + +* **Database-level encryption**. Similar to application-level encryption in terms of its properties. Most database vendors offer some form of encryption. However, there can be performance issues. One example is that indexes cannot be encrypted. + +* **Filesystem-level encryption**. This option offers high performance, application transparency, and is typically easy to deploy. However, it is unable to model some application-level policies. For instance, multi-tenant applications might want to encrypt based on the end user. A database might want different encryption settings for each column stored within a single file. + +* **Disk-level encryption**. Easy to deploy and high performance, but also quite inflexible. Only really protects against physical theft. + +HDFS-level encryption fits between database-level and filesystem-level encryption in this stack. This has a lot of positive effects. HDFS encryption is able to provide good performance and existing Hadoop applications are able to run transparently on encrypted data. HDFS also has more context than traditional filesystems when it comes to making policy decisions. + +HDFS-level encryption also prevents attacks at the filesystem-level and below (so-called "OS-level attacks"). The operating system and disk only interact with encrypted bytes, since the data is already encrypted by HDFS. + +Use Cases +--------- + +Data encryption is required by a number of different government, financial, and regulatory entities. For example, the health-care industry has HIPAA regulations, the card payment industry has PCI DSS regulations, and the US government has FISMA regulations. Having transparent encryption built into HDFS makes it easier for organizations to comply with these regulations. + +Encryption can also be performed at the application-level, but by integrating it into HDFS, existing applications can operate on encrypted data without changes. This integrated architecture implies stronger encrypted file semantics and better coordination with other HDFS functions. + +Architecture +------------ + +### Overview + +For transparent encryption, we introduce a new abstraction to HDFS: the *encryption zone*. An encryption zone is a special directory whose contents will be transparently encrypted upon write and transparently decrypted upon read. Each encryption zone is associated with a single *encryption zone key* which is specified when the zone is created. Each file within an encryption zone has its own unique *data encryption key (DEK)*. DEKs are never handled directly by HDFS. Instead, HDFS only ever handles an *encrypted data encryption key (EDEK)*. Clients decrypt an EDEK, and then use the subsequent DEK to read and write data. HDFS datanodes simply see a stream of encrypted bytes. + +A new cluster service is required to manage encryption keys: the Hadoop Key Management Server (KMS). In the context of HDFS encryption, the KMS performs three basic responsibilities: + +1. Providing access to stored encryption zone keys + +2. Generating new encrypted data encryption keys for storage on the NameNode + +3. Decrypting encrypted data encryption keys for use by HDFS clients + +The KMS will be described in more detail below. + +### Accessing data within an encryption zone + +When creating a new file in an encryption zone, the NameNode asks the KMS to generate a new EDEK encrypted with the encryption zone's key. The EDEK is then stored persistently as part of the file's metadata on the NameNode. + +When reading a file within an encryption zone, the NameNode provides the client with the file's EDEK and the encryption zone key version used to encrypt the EDEK. The client then asks the KMS to decrypt the EDEK, which involves checking that the client has permission to access the encryption zone key version. Assuming that is successful, the client uses the DEK to decrypt the file's contents. + +All of the above steps for the read and write path happen automatically through interactions between the DFSClient, the NameNode, and the KMS. + +Access to encrypted file data and metadata is controlled by normal HDFS filesystem permissions. This means that if HDFS is compromised (for example, by gaining unauthorized access to an HDFS superuser account), a malicious user only gains access to ciphertext and encrypted keys. However, since access to encryption zone keys is controlled by a separate set of permissions on the KMS and key store, this does not pose a security threat. + +### Key Management Server, KeyProvider, EDEKs + +The KMS is a proxy that interfaces with a backing key store on behalf of HDFS daemons and clients. Both the backing key store and the KMS implement the Hadoop KeyProvider API. See the [KMS documentation](../../hadoop-kms/index.html) for more information. + +In the KeyProvider API, each encryption key has a unique *key name*. Because keys can be rolled, a key can have multiple *key versions*, where each key version has its own *key material* (the actual secret bytes used during encryption and decryption). An encryption key can be fetched by either its key name, returning the latest version of the key, or by a specific key version. + +The KMS implements additional functionality which enables creation and decryption of *encrypted encryption keys (EEKs)*. Creation and decryption of EEKs happens entirely on the KMS. Importantly, the client requesting creation or decryption of an EEK never handles the EEK's encryption key. To create a new EEK, the KMS generates a new random key, encrypts it with the specified key, and returns the EEK to the client. To decrypt an EEK, the KMS checks that the user has access to the encryption key, uses it to decrypt the EEK, and returns the decrypted encryption key. + +In the context of HDFS encryption, EEKs are *encrypted data encryption keys (EDEKs)*, where a *data encryption key (DEK)* is what is used to encrypt and decrypt file data. Typically, the key store is configured to only allow end users access to the keys used to encrypt DEKs. This means that EDEKs can be safely stored and handled by HDFS, since the HDFS user will not have access to unencrypted encryption keys. + +Configuration +------------- + +A necessary prerequisite is an instance of the KMS, as well as a backing key store for the KMS. See the [KMS documentation](../../hadoop-kms/index.html) for more information. + +Once a KMS has been set up and the NameNode and HDFS clients have been correctly configured, an admin can use the `hadoop key` and `hdfs crypto` command-line tools to create encryption keys and set up new encryption zones. Existing data can be encrypted by copying it into the new encryption zones using tools like distcp. + +### Configuring the cluster KeyProvider + +#### dfs.encryption.key.provider.uri + +The KeyProvider to use when interacting with encryption keys used when reading and writing to an encryption zone. + +### Selecting an encryption algorithm and codec + +#### hadoop.security.crypto.codec.classes.EXAMPLECIPHERSUITE + +The prefix for a given crypto codec, contains a comma-separated list of implementation classes for a given crypto codec (eg EXAMPLECIPHERSUITE). The first implementation will be used if available, others are fallbacks. + +#### hadoop.security.crypto.codec.classes.aes.ctr.nopadding + +Default: `org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec,org.apache.hadoop.crypto.JceAesCtrCryptoCodec` + +Comma-separated list of crypto codec implementations for AES/CTR/NoPadding. The first implementation will be used if available, others are fallbacks. + +#### hadoop.security.crypto.cipher.suite + +Default: `AES/CTR/NoPadding` + +Cipher suite for crypto codec. + +#### hadoop.security.crypto.jce.provider + +Default: None + +The JCE provider name used in CryptoCodec. + +#### hadoop.security.crypto.buffer.size + +Default: `8192` + +The buffer size used by CryptoInputStream and CryptoOutputStream. + +### Namenode configuration + +#### dfs.namenode.list.encryption.zones.num.responses + +Default: `100` + +When listing encryption zones, the maximum number of zones that will be returned in a batch. Fetching the list incrementally in batches improves namenode performance. + +`crypto` command-line interface +------------------------------- + +### createZone + +Usage: `[-createZone -keyName <keyName> -path <path>]` + +Create a new encryption zone. + +| | | +|:---- |:---- | +| *path* | The path of the encryption zone to create. It must be an empty directory. | +| *keyName* | Name of the key to use for the encryption zone. | + +### listZones + +Usage: `[-listZones]` + +List all encryption zones. Requires superuser permissions. + +Example usage +------------- + +These instructions assume that you are running as the normal user or HDFS superuser as is appropriate. Use `sudo` as needed for your environment. + + # As the normal user, create a new encryption key + hadoop key create myKey + + # As the super user, create a new empty directory and make it an encryption zone + hadoop fs -mkdir /zone + hdfs crypto -createZone -keyName myKey -path /zone + + # chown it to the normal user + hadoop fs -chown myuser:myuser /zone + + # As the normal user, put a file in, read it out + hadoop fs -put helloWorld /zone + hadoop fs -cat /zone/helloWorld + +Distcp considerations +--------------------- + +### Running as the superuser + +One common usecase for distcp is to replicate data between clusters for backup and disaster recovery purposes. This is typically performed by the cluster administrator, who is an HDFS superuser. + +To enable this same workflow when using HDFS encryption, we introduced a new virtual path prefix, `/.reserved/raw/`, that gives superusers direct access to the underlying block data in the filesystem. This allows superusers to distcp data without needing having access to encryption keys, and also avoids the overhead of decrypting and re-encrypting data. It also means the source and destination data will be byte-for-byte identical, which would not be true if the data was being re-encrypted with a new EDEK. + +When using `/.reserved/raw` to distcp encrypted data, it's important to preserve extended attributes with the [-px](#a-px) flag. This is because encrypted file attributes (such as the EDEK) are exposed through extended attributes within `/.reserved/raw`, and must be preserved to be able to decrypt the file. This means that if the distcp is initiated at or above the encryption zone root, it will automatically create an encryption zone at the destination if it does not already exist. However, it's still recommended that the admin first create identical encryption zones on the destination cluster to avoid any potential mishaps. + +### Copying between encrypted and unencrypted locations + +By default, distcp compares checksums provided by the filesystem to verify that the data was successfully copied to the destination. When copying between an unencrypted and encrypted location, the filesystem checksums will not match since the underlying block data is different. In this case, specify the [-skipcrccheck](#a-skipcrccheck) and [-update](#a-update) distcp flags to avoid verifying checksums. + +Attack vectors +-------------- + +### Hardware access exploits + +These exploits assume that attacker has gained physical access to hard drives from cluster machines, i.e. datanodes and namenodes. + +1. Access to swap files of processes containing data encryption keys. + + * By itself, this does not expose cleartext, as it also requires access to encrypted block files. + + * This can be mitigated by disabling swap, using encrypted swap, or using mlock to prevent keys from being swapped out. + +2. Access to encrypted block files. + + * By itself, this does not expose cleartext, as it also requires access to DEKs. + +### Root access exploits + +These exploits assume that attacker has gained root shell access to cluster machines, i.e. datanodes and namenodes. Many of these exploits cannot be addressed in HDFS, since a malicious root user has access to the in-memory state of processes holding encryption keys and cleartext. For these exploits, the only mitigation technique is carefully restricting and monitoring root shell access. + +1. Access to encrypted block files. + + * By itself, this does not expose cleartext, as it also requires access to encryption keys. + +2. Dump memory of client processes to obtain DEKs, delegation tokens, cleartext. + + * No mitigation. + +3. Recording network traffic to sniff encryption keys and encrypted data in transit. + + * By itself, insufficient to read cleartext without the EDEK encryption key. + +4. Dump memory of datanode process to obtain encrypted block data. + + * By itself, insufficient to read cleartext without the DEK. + +5. Dump memory of namenode process to obtain encrypted data encryption keys. + + * By itself, insufficient to read cleartext without the EDEK's encryption key and encrypted block files. + +### HDFS admin exploits + +These exploits assume that the attacker has compromised HDFS, but does not have root or `hdfs` user shell access. + +1. Access to encrypted block files. + + * By itself, insufficient to read cleartext without the EDEK and EDEK encryption key. + +2. Access to encryption zone and encrypted file metadata (including encrypted data encryption keys), via -fetchImage. + + * By itself, insufficient to read cleartext without EDEK encryption keys. + +### Rogue user exploits + +A rogue user can collect keys of files they have access to, and use them later to decrypt the encrypted data of those files. As the user had access to those files, they already had access to the file contents. This can be mitigated through periodic key rolling policies. http://git-wip-us.apache.org/repos/asf/hadoop/blob/a45ef2b6/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md new file mode 100644 index 0000000..ef5976c --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md @@ -0,0 +1,242 @@ +<!--- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +ViewFs Guide +============ + +* [ViewFs Guide](#ViewFs_Guide) + * [Introduction](#Introduction) + * [The Old World (Prior to Federation)](#The_Old_World_Prior_to_Federation) + * [Single Namenode Clusters](#Single_Namenode_Clusters) + * [Pathnames Usage Patterns](#Pathnames_Usage_Patterns) + * [Pathname Usage Best Practices](#Pathname_Usage_Best_Practices) + * [New World â Federation and ViewFs](#New_World__Federation_and_ViewFs) + * [How The Clusters Look](#How_The_Clusters_Look) + * [A Global Namespace Per Cluster Using ViewFs](#A_Global_Namespace_Per_Cluster_Using_ViewFs) + * [Pathname Usage Patterns](#Pathname_Usage_Patterns) + * [Pathname Usage Best Practices](#Pathname_Usage_Best_Practices) + * [Renaming Pathnames Across Namespaces](#Renaming_Pathnames_Across_Namespaces) + * [FAQ](#FAQ) + * [Appendix: A Mount Table Configuration Example](#Appendix:_A_Mount_Table_Configuration_Example) + +Introduction +------------ + +The View File System (ViewFs) provides a way to manage multiple Hadoop file system namespaces (or namespace volumes). It is particularly useful for clusters having multiple namenodes, and hence multiple namespaces, in [HDFS Federation](./Federation.html). ViewFs is analogous to *client side mount tables* in some Unix/Linux systems. ViewFs can be used to create personalized namespace views and also per-cluster common views. + +This guide is presented in the context of Hadoop systems that have several clusters, each cluster may be federated into multiple namespaces. It also describes how to use ViewFs in federated HDFS to provide a per-cluster global namespace so that applications can operate in a way similar to the pre-federation world. + +The Old World (Prior to Federation) +----------------------------------- + +### Single Namenode Clusters + +In the old world prior to [HDFS Federation](./Federation.html), a cluster has a single namenode which provides a single file system namespace for that cluster. Suppose there are multiple clusters. The file system namespaces of each cluster are completely independent and disjoint. Furthermore, physical storage is NOT shared across clusters (i.e. the Datanodes are not shared across clusters.) + +The `core-site.xml` of each cluster has a configuration property that sets the default file system to the namenode of that cluster: + +```xml +<property> + <name>fs.default.name</name> + <value>hdfs://namenodeOfClusterX:port</value> +</property> +``` + +Such a configuration property allows one to use slash-relative names to resolve paths relative to the cluster namenode. For example, the path `/foo/bar` is referring to `hdfs://namenodeOfClusterX:port/foo/bar` using the above configuration. + +This configuration property is set on each gateway on the clusters and also on key services of that cluster such the JobTracker and Oozie. + +### Pathnames Usage Patterns + +Hence on Cluster X where the `core-site.xml` is set as above, the typical pathnames are + +1. `/foo/bar` + + * This is equivalent to `hdfs://namenodeOfClusterX:port/foo/bar` as before. + +2. `hdfs://namenodeOfClusterX:port/foo/bar` + + * While this is a valid pathname, one is better using `/foo/bar` as it allows the application and its data to be transparently moved to another cluster when needed. + +3. `hdfs://namenodeOfClusterY:port/foo/bar` + + * It is an URI for referring a pathname on another cluster such as Cluster Y. In particular, the command for copying files from cluster Y to Cluster Z looks like: + + distcp hdfs://namenodeClusterY:port/pathSrc hdfs://namenodeClusterZ:port/pathDest + +4. `webhdfs://namenodeClusterX:http_port/foo/bar` and `hftp://namenodeClusterX:http_port/foo/bar` + + * These are file system URIs respectively for accessing files via the WebHDFS file system and the HFTP file system. Note that WebHDFS and HFTP use the HTTP port of the namenode but not the RPC port. + +5. `http://namenodeClusterX:http_port/webhdfs/v1/foo/bar` and `http://proxyClusterX:http_port/foo/bar` + + * These are HTTP URLs respectively for accessing files via [WebHDFS REST API](./WebHDFS.html) and HDFS proxy. + +### Pathname Usage Best Practices + +When one is within a cluster, it is recommended to use the pathname of type (1) above instead of a fully qualified URI like (2). Fully qualified URIs are similar to addresses and do not allow the application to move along with its data. + +New World â Federation and ViewFs +--------------------------------- + +### How The Clusters Look + +Suppose there are multiple clusters. Each cluster has one or more namenodes. Each namenode has its own namespace. A namenode belongs to one and only one cluster. The namenodes in the same cluster share the physical storage of that cluster. The namespaces across clusters are independent as before. + +Operations decide what is stored on each namenode within a cluster based on the storage needs. For example, they may put all the user data (`/user/<username>`) in one namenode, all the feed-data (`/data`) in another namenode, all the projects (`/projects`) in yet another namenode, etc. + +### A Global Namespace Per Cluster Using ViewFs + +In order to provide transparency with the old world, the ViewFs file system (i.e. client-side mount table) is used to create each cluster an independent cluster namespace view, which is similar to the namespace in the old world. The client-side mount tables like the Unix mount tables and they mount the new namespace volumes using the old naming convention. The following figure shows a mount table mounting four namespace volumes `/user`, `/data`, `/projects`, and `/tmp`: + + + +ViewFs implements the Hadoop file system interface just like HDFS and the local file system. It is a trivial file system in the sense that it only allows linking to other file systems. Because ViewFs implements the Hadoop file system interface, it works transparently Hadoop tools. For example, all the shell commands work with ViewFs as with HDFS and local file system. + +The mount points of a mount table are specified in the standard Hadoop configuration files. In the configuration of each cluster, the default file system is set to the mount table for that cluster as shown below (compare it with the configuration in [Single Namenode Clusters](#Single_Namenode_Clusters)). + +```xml +<property> + <name>fs.default.name</name> + <value>viewfs://clusterX</value> +</property> +``` + +The authority following the `viewfs://` scheme in the URI is the mount table name. It is recommanded that the mount table of a cluster should be named by the cluster name. Then Hadoop system will look for a mount table with the name "clusterX" in the Hadoop configuration files. Operations arrange all gateways and service machines to contain the mount tables for ALL clusters such that, for each cluster, the default file system is set to the ViewFs mount table for that cluster as described above. + +### Pathname Usage Patterns + +Hence on Cluster X, where the `core-site.xml` is set to make the default fs to use the mount table of that cluster, the typical pathnames are + +1. `/foo/bar` + + * This is equivalent to `viewfs://clusterX/foo/bar`. If such pathname is used in the old non-federated world, then the transition to federation world is transparent. + +2. `viewfs://clusterX/foo/bar` + + * While this a valid pathname, one is better using `/foo/bar` as it allows the application and its data to be transparently moved to another cluster when needed. + +3. `viewfs://clusterY/foo/bar` + + * It is an URI for referring a pathname on another cluster such as Cluster Y. In particular, the command for copying files from cluster Y to Cluster Z looks like: + + distcp viewfs://clusterY:/pathSrc viewfs://clusterZ/pathDest + +4. `viewfs://clusterX-webhdfs/foo/bar` and `viewfs://clusterX-hftp/foo/bar` + + * These are URIs respectively for accessing files via the WebHDFS file system and the HFTP file system. + +5. `http://namenodeClusterX:http_port/webhdfs/v1/foo/bar` and `http://proxyClusterX:http_port/foo/bar` + + * These are HTTP URLs respectively for accessing files via [WebHDFS REST API](./WebHDFS.html) and HDFS proxy. Note that they are the same as before. + +### Pathname Usage Best Practices + +When one is within a cluster, it is recommended to use the pathname of type (1) above instead of a fully qualified URI like (2). Futher, applications should not use the knowledge of the mount points and use a path like `hdfs://namenodeContainingUserDirs:port/joe/foo/bar` to refer to a file in a particular namenode. One should use `/user/joe/foo/bar` instead. + +### Renaming Pathnames Across Namespaces + +Recall that one cannot rename files or directories across namenodes or clusters in the old world. The same is true in the new world but with an additional twist. For example, in the old world one can perform the commend below. + + rename /user/joe/myStuff /data/foo/bar + +This will NOT work in the new world if `/user` and `/data` are actually stored on different namenodes within a cluster. + +### FAQ + +1. **As I move from non-federated world to the federated world, I will have to keep track of namenodes for different volumes; how do I do that?** + + No, you wonât. See the examples above â you are either using a relative name and taking advantage of the default file system, or changing your path from `hdfs://namenodeCLusterX/foo/bar` to `viewfs://clusterX/foo/bar`. + +2. **What happens of Operations move some files from one namenode to another namenode within a cluster?** + + Operations may move files from one namenode to another in order to deal with storage capacity issues. They will do this in a way to avoid applications from breaking. Let's take some examples. + + * Example 1: `/user` and `/data` were on one namenode and later they need to be on separate namenodes to deal with capacity issues. Indeed, operations would have created separate mount points for `/user` and `/data`. Prior to the change the mounts for `/user` and `/data` would have pointed to the same namenode, say `namenodeContainingUserAndData`. Operations will update the mount tables so that the mount points are changed to `namenodeContaingUser` and `namenodeContainingData`, respectively. + + * Example 2: All projects were fitted on one namenode and but later they need two or more namenodes. ViewFs allows mounts like `/project/foo` and `/project/bar`. This allows mount tables to be updated to point to the corresponding namenode. + +3. **Is the mount table in each** `core-site.xml` **or in a separate file of its own?** + + The plan is to keep the mount tables in separate files and have the `core-site.xml` [xincluding](http://www.w3.org/2001/XInclude) it. While one can keep these files on each machine locally, it is better to use HTTP to access it from a central location. + +4. **Should the configuration have the mount table definitions for only one cluster or all clusters?** + + The configuration should have the mount definitions for all clusters since one needs to have access to data in other clusters such as with distcp. + +5. **When is the mount table actually read given that Operations may change a mount table over time?** + + The mount table is read when the job is submitted to the cluster. The `XInclude` in `core-site.xml` is expanded at job submission time. This means that if the mount table are changed then the jobs need to be resubmitted. Due to this reason, we want to implement merge-mount which will greatly reduce the need to change mount tables. Further, we would like to read the mount tables via another mechanism that is initialized at job start time in the future. + +6. **Will JobTracker (or Yarnâs Resource Manager) itself use the ViewFs?** + + No, it does not need to. Neither does the NodeManager. + +7. **Does ViewFs allow only mounts at the top level?** + + No; it is more general. For example, one can mount `/user/joe` and `/user/jane`. In this case, an internal read-only directory is created for `/user` in the mount table. All operations on `/user` are valid except that `/user` is read-only. + +8. **An application works across the clusters and needs to persistently store file paths. Which paths should it store?** + + You should store `viewfs://cluster/path` type path names, the same as it uses when running applications. This insulates you from movement of data within namenodes inside a cluster as long as operations do the moves in a transparent fashion. It does not insulate you if data gets moved from one cluster to another; the older (pre-federation) world did not protect you form such data movements across clusters anyway. + +9. **What about delegation tokens?** + + Delegation tokens for the cluster to which you are submitting the job (including all mounted volumes for that clusterâs mount table), and for input and output paths to your map-reduce job (including all volumes mounted via mount tables for the specified input and output paths) are all handled automatically. In addition, there is a way to add additional delegation tokens to the base cluster configuration for special circumstances. + +Appendix: A Mount Table Configuration Example +--------------------------------------------- + +Generally, users do not have to define mount tables or the `core-site.xml` to use the mount table. This is done by operations and the correct configuration is set on the right gateway machines as is done for `core-site.xml` today. + +The mount tables can be described in `core-site.xml` but it is better to use indirection in `core-site.xml` to reference a separate configuration file, say `mountTable.xml`. Add the following configuration element to `core-site.xml` for referencing `mountTable.xml`: + +```xml +<configuration xmlns:xi="http://www.w3.org/2001/XInclude";> + <xi:include href="mountTable.xml" /> +</configuration> +``` + +In the file `mountTable.xml`, there is a definition of the mount table "ClusterX" for the hypothetical cluster that is a federation of the three namespace volumes managed by the three namenodes + +1. nn1-clusterx.example.com:8020, +2. nn2-clusterx.example.com:8020, and +3. nn3-clusterx.example.com:8020. + +Here `/home` and `/tmp` are in the namespace managed by namenode nn1-clusterx.example.com:8020, and projects `/foo` and `/bar` are hosted on the other namenodes of the federated cluster. The home directory base path is set to `/home` so that each user can access its home directory using the getHomeDirectory() method defined in [FileSystem](../../api/org/apache/hadoop/fs/FileSystem.html)/[FileContext](../../api/org/apache/hadoop/fs/FileContext.html). + +```xml +<configuration> + <property> + <name>fs.viewfs.mounttable.ClusterX.homedir</name> + <value>/home</value> + </property> + <property> + <name>fs.viewfs.mounttable.ClusterX.link./home</name> + <value>hdfs://nn1-clusterx.example.com:8020/home</value> + </property> + <property> + <name>fs.viewfs.mounttable.ClusterX.link./tmp</name> + <value>hdfs://nn1-clusterx.example.com:8020/tmp</value> + </property> + <property> + <name>fs.viewfs.mounttable.ClusterX.link./projects/foo</name> + <value>hdfs://nn2-clusterx.example.com:8020/projects/foo</value> + </property> + <property> + <name>fs.viewfs.mounttable.ClusterX.link./projects/bar</name> + <value>hdfs://nn3-clusterx.example.com:8020/projects/bar</value> + </property> +</configuration> +```
