http://git-wip-us.apache.org/repos/asf/hbase/blob/fba353df/src/main/asciidoc/_chapters/security.adoc ---------------------------------------------------------------------- diff --git a/src/main/asciidoc/_chapters/security.adoc b/src/main/asciidoc/_chapters/security.adoc index f89efcc..21698fa 100644 --- a/src/main/asciidoc/_chapters/security.adoc +++ b/src/main/asciidoc/_chapters/security.adoc @@ -31,7 +31,7 @@ HBase provides mechanisms to secure various components and aspects of HBase and == Using Secure HTTP (HTTPS) for the Web UI -A default HBase install uses insecure HTTP connections for web UIs for the master and region servers. +A default HBase install uses insecure HTTP connections for Web UIs for the master and region servers. To enable secure HTTP (HTTPS) connections instead, set `hadoop.ssl.enabled` to `true` in _hbase-site.xml_. This does not change the port used by the Web UI. To change the port for the web UI for a given HBase component, configure that port's setting in hbase-site.xml. @@ -63,8 +63,7 @@ If you know how to fix this without opening a second port for HTTPS, patches are == Secure Client Access to Apache HBase Newer releases of Apache HBase (>= 0.92) support optional SASL authentication of clients. -See also Matteo Bertozzi's article on link:http://www.cloudera.com/blog/2012/09/understanding-user-authentication-and-authorization-in-apache-hbase/[Understanding - User Authentication and Authorization in Apache HBase]. +See also Matteo Bertozzi's article on link:http://www.cloudera.com/blog/2012/09/understanding-user-authentication-and-authorization-in-apache-hbase/[Understanding User Authentication and Authorization in Apache HBase]. This describes how to set up Apache HBase and clients for connection to secure HBase resources. @@ -77,13 +76,13 @@ Hadoop Authentication Configuration:: Otherwise, you would be using strong authentication for HBase but not for the underlying HDFS, which would cancel out any benefit. Kerberos KDC:: - You need to have a working Kerberos KDC. + You need to have a working Kerberos KDC. === Server-side Configuration for Secure Operation First, refer to <<security.prerequisites,security.prerequisites>> and ensure that your underlying HDFS configuration is secure. -Add the following to the `hbase-site.xml` file on every server machine in the cluster: +Add the following to the `hbase-site.xml` file on every server machine in the cluster: [source,xml] ---- @@ -101,13 +100,13 @@ Add the following to the `hbase-site.xml` file on every server machine in the cl </property> ---- -A full shutdown and restart of HBase service is required when deploying these configuration changes. +A full shutdown and restart of HBase service is required when deploying these configuration changes. === Client-side Configuration for Secure Operation -First, refer to <<security.prerequisites,security.prerequisites>> and ensure that your underlying HDFS configuration is secure. +First, refer to <<security.prerequisites>> and ensure that your underlying HDFS configuration is secure. -Add the following to the `hbase-site.xml` file on every client: +Add the following to the `hbase-site.xml` file on every client: [source,xml] ---- @@ -117,12 +116,12 @@ Add the following to the `hbase-site.xml` file on every client: </property> ---- -The client environment must be logged in to Kerberos from KDC or keytab via the `kinit` command before communication with the HBase cluster will be possible. +The client environment must be logged in to Kerberos from KDC or keytab via the `kinit` command before communication with the HBase cluster will be possible. -Be advised that if the `hbase.security.authentication` in the client- and server-side site files do not match, the client will not be able to communicate with the cluster. +Be advised that if the `hbase.security.authentication` in the client- and server-side site files do not match, the client will not be able to communicate with the cluster. Once HBase is configured for secure RPC it is possible to optionally configure encrypted communication. -To do so, add the following to the `hbase-site.xml` file on every client: +To do so, add the following to the `hbase-site.xml` file on every client: [source,xml] ---- @@ -133,7 +132,7 @@ To do so, add the following to the `hbase-site.xml` file on every client: ---- This configuration property can also be set on a per connection basis. -Set it in the `Configuration` supplied to `HTable`: +Set it in the `Configuration` supplied to `HTable`: [source,java] ---- @@ -142,12 +141,12 @@ conf.set("hbase.rpc.protection", "privacy"); HTable table = new HTable(conf, tablename); ---- -Expect a ~10% performance penalty for encrypted communication. +Expect a ~10% performance penalty for encrypted communication. [[security.client.thrift]] === Client-side Configuration for Secure Operation - Thrift Gateway -Add the following to the `hbase-site.xml` file for every Thrift gateway: +Add the following to the `hbase-site.xml` file for every Thrift gateway: [source,xml] ---- <property> @@ -161,28 +160,28 @@ Add the following to the `hbase-site.xml` file for every Thrift gateway: You may have to put the concrete full hostname. --> </property> ----- +---- -Substitute the appropriate credential and keytab for [replaceable]_$USER_ and [replaceable]_$KEYTAB_ respectively. +Substitute the appropriate credential and keytab for _$USER_ and _$KEYTAB_ respectively. In order to use the Thrift API principal to interact with HBase, it is also necessary to add the `hbase.thrift.kerberos.principal` to the `_acl_` table. -For example, to give the Thrift API principal, `thrift_server`, administrative access, a command such as this one will suffice: +For example, to give the Thrift API principal, `thrift_server`, administrative access, a command such as this one will suffice: [source,sql] ---- grant 'thrift_server', 'RWCA' ---- -For more information about ACLs, please see the <<hbase.accesscontrol.configuration>> section +For more information about ACLs, please see the <<hbase.accesscontrol.configuration>> section The Thrift gateway will authenticate with HBase using the supplied credential. No authentication will be performed by the Thrift gateway itself. -All client access via the Thrift gateway will use the Thrift gateway's credential and have its privilege. +All client access via the Thrift gateway will use the Thrift gateway's credential and have its privilege. [[security.gateway.thrift]] === Configure the Thrift Gateway to Authenticate on Behalf of the Client -<<security.client.thrift,security.client.thrift>> describes how to authenticate a Thrift client to HBase using a fixed user. +<<security.client.thrift>> describes how to authenticate a Thrift client to HBase using a fixed user. As an alternative, you can configure the Thrift gateway to authenticate to HBase on the client's behalf, and to access HBase using a proxy user. This was implemented in link:https://issues.apache.org/jira/browse/HBASE-11349[HBASE-11349] for Thrift 1, and link:https://issues.apache.org/jira/browse/HBASE-11474[HBASE-11474] for Thrift 2. @@ -195,8 +194,8 @@ If you use framed transport, you cannot yet take advantage of this feature, beca To enable it, do the following. -. Be sure Thrift is running in secure mode, by following the procedure described in <<security.client.thrift,security.client.thrift>>. -. Be sure that HBase is configured to allow proxy users, as described in <<security.rest.gateway,security.rest.gateway>>. +. Be sure Thrift is running in secure mode, by following the procedure described in <<security.client.thrift>>. +. Be sure that HBase is configured to allow proxy users, as described in <<security.rest.gateway>>. . In _hbase-site.xml_ for each cluster node running a Thrift gateway, set the property `hbase.thrift.security.qop` to one of the following three values: + * `auth-conf` - authentication, integrity, and confidentiality checking @@ -204,14 +203,14 @@ To enable it, do the following. * `auth` - authentication checking only . Restart the Thrift gateway processes for the changes to take effect. - If a node is running Thrift, the output of the +jps+ command will list a `ThriftServer` process. - To stop Thrift on a node, run the command +bin/hbase-daemon.sh stop thrift+. - To start Thrift on a node, run the command +bin/hbase-daemon.sh start thrift+. + If a node is running Thrift, the output of the `jps` command will list a `ThriftServer` process. + To stop Thrift on a node, run the command `bin/hbase-daemon.sh stop thrift`. + To start Thrift on a node, run the command `bin/hbase-daemon.sh start thrift`. [[security.gateway.thrift.doas]] === Configure the Thrift Gateway to Use the `doAs` Feature -<<security.gateway.thrift>> describes how to configure the Thrift gateway to authenticate to HBase on the client's behalf, and to access HBase using a proxy user. The limitation of this approach is that after the client is initialized with a particular set of credentials, it cannot change these credentials during the session session. The `doAs` feature provides a flexible way to impersonate multiple principals using the same client. This feature was implemented in link:https://issues.apache.org/jira/browse/HBASE-12640[HBASE-12640] for Thrift 1, but is currently not available for Thrift 2. +<<security.gateway.thrift>> describes how to configure the Thrift gateway to authenticate to HBase on the client's behalf, and to access HBase using a proxy user. The limitation of this approach is that after the client is initialized with a particular set of credentials, it cannot change these credentials during the session. The `doAs` feature provides a flexible way to impersonate multiple principals using the same client. This feature was implemented in link:https://issues.apache.org/jira/browse/HBASE-12640[HBASE-12640] for Thrift 1, but is currently not available for Thrift 2. *To allow proxy users*, add the following to the _hbase-site.xml_ file for every HBase node: @@ -249,7 +248,7 @@ Take a look at the link:https://github.com/apache/hbase/blob/master/hbase-exampl === Client-side Configuration for Secure Operation - REST Gateway -Add the following to the `hbase-site.xml` file for every REST gateway: +Add the following to the `hbase-site.xml` file for every REST gateway: [source,xml] ---- @@ -263,24 +262,24 @@ Add the following to the `hbase-site.xml` file for every REST gateway: </property> ---- -Substitute the appropriate credential and keytab for [replaceable]_$USER_ and [replaceable]_$KEYTAB_ respectively. +Substitute the appropriate credential and keytab for _$USER_ and _$KEYTAB_ respectively. The REST gateway will authenticate with HBase using the supplied credential. No authentication will be performed by the REST gateway itself. -All client access via the REST gateway will use the REST gateway's credential and have its privilege. +All client access via the REST gateway will use the REST gateway's credential and have its privilege. In order to use the REST API principal to interact with HBase, it is also necessary to add the `hbase.rest.kerberos.principal` to the `_acl_` table. -For example, to give the REST API principal, `rest_server`, administrative access, a command such as this one will suffice: +For example, to give the REST API principal, `rest_server`, administrative access, a command such as this one will suffice: [source,sql] ---- grant 'rest_server', 'RWCA' ---- -For more information about ACLs, please see the <<hbase.accesscontrol.configuration>> section +For more information about ACLs, please see the <<hbase.accesscontrol.configuration>> section -It should be possible for clients to authenticate with the HBase cluster through the REST gateway in a pass-through manner via SPEGNO HTTP authentication. -This is future work. +It should be possible for clients to authenticate with the HBase cluster through the REST gateway in a pass-through manner via SPNEGO HTTP authentication. +This is future work. [[security.rest.gateway]] === REST Gateway Impersonation Configuration @@ -292,11 +291,11 @@ The actual users are unknown. You can turn on the impersonation support. With impersonation, the REST gateway user is a proxy user. The HBase server knows the acutal/real user of each request. -So it can apply proper authorizations. +So it can apply proper authorizations. -To turn on REST gateway impersonation, we need to configure HBase servers (masters and region servers) to allow proxy users; configure REST gateway to enable impersonation. +To turn on REST gateway impersonation, we need to configure HBase servers (masters and region servers) to allow proxy users; configure REST gateway to enable impersonation. -To allow proxy users, add the following to the `hbase-site.xml` file for every HBase server: +To allow proxy users, add the following to the `hbase-site.xml` file for every HBase server: [source,xml] ---- @@ -314,9 +313,9 @@ To allow proxy users, add the following to the `hbase-site.xml` file for every H </property> ---- -Substitute the REST gateway proxy user for $USER, and the allowed group list for $GROUPS. +Substitute the REST gateway proxy user for _$USER_, and the allowed group list for _$GROUPS_. -To enable REST gateway impersonation, add the following to the `hbase-site.xml` file for every REST gateway. +To enable REST gateway impersonation, add the following to the `hbase-site.xml` file for every REST gateway. [source,xml] ---- @@ -334,35 +333,34 @@ To enable REST gateway impersonation, add the following to the `hbase-site.xml` </property> ---- -Substitute the keytab for HTTP for $KEYTAB. +Substitute the keytab for HTTP for _$KEYTAB_. [[hbase.secure.simpleconfiguration]] == Simple User Access to Apache HBase Newer releases of Apache HBase (>= 0.92) support optional SASL authentication of clients. -See also Matteo Bertozzi's article on link:http://www.cloudera.com/blog/2012/09/understanding-user-authentication-and-authorization-in-apache-hbase/[Understanding - User Authentication and Authorization in Apache HBase]. +See also Matteo Bertozzi's article on link:http://www.cloudera.com/blog/2012/09/understanding-user-authentication-and-authorization-in-apache-hbase/[Understanding User Authentication and Authorization in Apache HBase]. This describes how to set up Apache HBase and clients for simple user access to HBase resources. -=== Simple Versus Secure Access +=== Simple versus Secure Access The following section shows how to set up simple user access. Simple user access is not a secure method of operating HBase. This method is used to prevent users from making mistakes. -It can be used to mimic the Access Control using on a development system without having to set up Kerberos. +It can be used to mimic the Access Control using on a development system without having to set up Kerberos. This method is not used to prevent malicious or hacking attempts. To make HBase secure against these types of attacks, you must configure HBase for secure operation. -Refer to the section link:[Secure Client Access to HBase] and complete all of the steps described there. +Refer to the section <<hbase.secure.configuration>> and complete all of the steps described there. === Prerequisites -None +None === Server-side Configuration for Simple User Access Operation -Add the following to the `hbase-site.xml` file on every server machine in the cluster: +Add the following to the `hbase-site.xml` file on every server machine in the cluster: [source,xml] ---- @@ -388,7 +386,7 @@ Add the following to the `hbase-site.xml` file on every server machine in the cl </property> ---- -For 0.94, add the following to the `hbase-site.xml` file on every server machine in the cluster: +For 0.94, add the following to the `hbase-site.xml` file on every server machine in the cluster: [source,xml] ---- @@ -406,11 +404,11 @@ For 0.94, add the following to the `hbase-site.xml` file on every server machine </property> ---- -A full shutdown and restart of HBase service is required when deploying these configuration changes. +A full shutdown and restart of HBase service is required when deploying these configuration changes. === Client-side Configuration for Simple User Access Operation -Add the following to the `hbase-site.xml` file on every client: +Add the following to the `hbase-site.xml` file on every client: [source,xml] ---- @@ -420,7 +418,7 @@ Add the following to the `hbase-site.xml` file on every client: </property> ---- -For 0.94, add the following to the `hbase-site.xml` file on every server machine in the cluster: +For 0.94, add the following to the `hbase-site.xml` file on every server machine in the cluster: [source,xml] ---- @@ -430,42 +428,42 @@ For 0.94, add the following to the `hbase-site.xml` file on every server machine </property> ---- -Be advised that if the `hbase.security.authentication` in the client- and server-side site files do not match, the client will not be able to communicate with the cluster. +Be advised that if the `hbase.security.authentication` in the client- and server-side site files do not match, the client will not be able to communicate with the cluster. ==== Client-side Configuration for Simple User Access Operation - Thrift Gateway The Thrift gateway user will need access. -For example, to give the Thrift API user, `thrift_server`, administrative access, a command such as this one will suffice: +For example, to give the Thrift API user, `thrift_server`, administrative access, a command such as this one will suffice: [source,sql] ---- grant 'thrift_server', 'RWCA' ---- -For more information about ACLs, please see the link:[Access Control] section +For more information about ACLs, please see the <<hbase.accesscontrol.configuration>> section The Thrift gateway will authenticate with HBase using the supplied credential. No authentication will be performed by the Thrift gateway itself. -All client access via the Thrift gateway will use the Thrift gateway's credential and have its privilege. +All client access via the Thrift gateway will use the Thrift gateway's credential and have its privilege. ==== Client-side Configuration for Simple User Access Operation - REST Gateway The REST gateway will authenticate with HBase using the supplied credential. No authentication will be performed by the REST gateway itself. -All client access via the REST gateway will use the REST gateway's credential and have its privilege. +All client access via the REST gateway will use the REST gateway's credential and have its privilege. The REST gateway user will need access. -For example, to give the REST API user, `rest_server`, administrative access, a command such as this one will suffice: +For example, to give the REST API user, `rest_server`, administrative access, a command such as this one will suffice: [source,sql] ---- grant 'rest_server', 'RWCA' ---- -For more information about ACLs, please see the link:[Access Control] section +For more information about ACLs, please see the <<hbase.accesscontrol.configuration>> section -It should be possible for clients to authenticate with the HBase cluster through the REST gateway in a pass-through manner via SPEGNO HTTP authentication. -This is future work. +It should be possible for clients to authenticate with the HBase cluster through the REST gateway in a pass-through manner via SPNEGO HTTP authentication. +This is future work. == Securing Access To Your Data @@ -496,8 +494,9 @@ When copying keys, configuration files, or other files containing sensitive stri ==== .Procedure: Basic Server-Side Configuration -. Enable HFile v3, by setting +hfile.format.version +to 3 in _hbase-site.xml_. - This is the default for HBase 1.0 and newer. + +. Enable HFile v3, by setting `hfile.format.version` to 3 in _hbase-site.xml_. + This is the default for HBase 1.0 and newer. ++ [source,xml] ---- <property> @@ -506,7 +505,7 @@ When copying keys, configuration files, or other files containing sensitive stri </property> ---- -. Enable SASL and Kerberos authentication for RPC and ZooKeeper, as described in <<security.prerequisites,security.prerequisites>> and <<zk.sasl.auth,zk.sasl.auth>>. +. Enable SASL and Kerberos authentication for RPC and ZooKeeper, as described in <<security.prerequisites,security.prerequisites>> and <<zk.sasl.auth>>. [[hbase.tags]] === Tags @@ -528,7 +527,7 @@ You can enable or disable tag encoding at the level of the column family, and it Use the `HColumnDescriptor#setCompressionTags(boolean compressTags)` method to manage encoding settings on a column family. You also need to enable the DataBlockEncoder for the column family, for encoding of tags to take effect. -You can enable compression of each tag in the WAL, if WAL compression is also enabled, by setting the value of +hbase.regionserver.wal.tags.enablecompression+ to `true` in _hbase-site.xml_. +You can enable compression of each tag in the WAL, if WAL compression is also enabled, by setting the value of `hbase.regionserver.wal.tags.enablecompression` to `true` in _hbase-site.xml_. Tag compression uses dictionary encoding. Tag compression is not supported when using WAL encryption. @@ -541,8 +540,7 @@ Tag compression is not supported when using WAL encryption. ACLs in HBase are based upon a user's membership in or exclusion from groups, and a given group's permissions to access a given resource. ACLs are implemented as a coprocessor called AccessController. -HBase does not maintain a private group mapping, but relies on a [firstterm]_Hadoop - group mapper_, which maps between entities in a directory such as LDAP or Active Directory, and HBase users. +HBase does not maintain a private group mapping, but relies on a [firstterm]_Hadoop group mapper_, which maps between entities in a directory such as LDAP or Active Directory, and HBase users. Any supported Hadoop group mapper will work. Users are then granted specific permissions (Read, Write, Execute, Create, Admin) against resources (global, namespaces, tables, cells, or endpoints). @@ -555,21 +553,21 @@ No distinction is made between an insert (new record) and update (of existing re HBase access levels are granted independently of each other and allow for different types of operations at a given scope. -* Read \(R) - can read data at the given scope -* +Write (W)+ - can write data at the given scope -* +Execute (X)+ - can execute coprocessor endpoints at the given scope -* +Create (C)+ - can create tables or drop tables (even those they did not create) at the given scope -* +Admin (A)+ - can perform cluster operations such as balancing the cluster or assigning regions at the given scope +* _Read \(R)_ - can read data at the given scope +* _Write (W)_ - can write data at the given scope +* _Execute (X)_ - can execute coprocessor endpoints at the given scope +* _Create \(C)_ - can create tables or drop tables (even those they did not create) at the given scope +* _Admin (A)_ - can perform cluster operations such as balancing the cluster or assigning regions at the given scope The possible scopes are: -* +Superuser+ - superusers can perform any operation available in HBase, to any resource. +* _Superuser_ - superusers can perform any operation available in HBase, to any resource. The user who runs HBase on your cluster is a superuser, as are any principals assigned to the configuration property `hbase.superuser` in _hbase-site.xml_ on the HMaster. -* +Global+ - permissions granted at _global_ scope allow the admin to operate on all tables of the cluster. -* +Namespace+ - permissions granted at _namespace_ scope apply to all tables within a given namespace. -* +Table+ - permissions granted at _table_ scope apply to data or metadata within a given table. -* +ColumnFamily+ - permissions granted at _ColumnFamily_ scope apply to cells within that ColumnFamily. -* +Cell+ - permissions granted at _cell_ scope apply to that exact cell coordinate (key, value, timestamp). This allows for policy evolution along with data. +* _Global_ - permissions granted at _global_ scope allow the admin to operate on all tables of the cluster. +* _Namespace_ - permissions granted at _namespace_ scope apply to all tables within a given namespace. +* _Table_ - permissions granted at _table_ scope apply to data or metadata within a given table. +* _ColumnFamily_ - permissions granted at _ColumnFamily_ scope apply to cells within that ColumnFamily. +* _Cell_ - permissions granted at _cell_ scope apply to that exact cell coordinate (key, value, timestamp). This allows for policy evolution along with data. + To change an ACL on a specific cell, write an updated cell with new ACL to the precise coordinates of the original. + @@ -587,12 +585,11 @@ In a production environment, it is useful to think of access levels in terms of The following list describes appropriate access levels for some common types of HBase users. It is important not to grant more access than is required for a given user to perform their required tasks. -* Superusers - In a production system, only the HBase user should have superuser access. +* _Superusers_ - In a production system, only the HBase user should have superuser access. In a development environment, an administrator may need superuser access in order to quickly control and manage the cluster. However, this type of administrator should usually be a Global Admin rather than a superuser. -* Global Admins - A global admin can perform tasks and access every table in HBase. +* _Global Admins_ - A global admin can perform tasks and access every table in HBase. In a typical production environment, an admin should not have Read or Write permissions to data within tables. -+ * A global admin with Admin permissions can perform cluster-wide operations on the cluster, such as balancing, assigning or unassigning regions, or calling an explicit major compaction. This is an operations role. * A global admin with Create permissions can create or drop any table within HBase. @@ -602,20 +599,20 @@ In a production environment, it is likely that different users will have only on + [WARNING] ==== -In the current implementation, a Global Admin with `Admin` permission can grant himself `Read` and `Write` permissions on a table and gain access to that table's data. +In the current implementation, a Global Admin with `Admin` permission can grant himself `Read` and `Write` permissions on a table and gain access to that table's data. For this reason, only grant `Global Admin` permissions to trusted user who actually need them. -Also be aware that a `Global Admin` with `Create` permission can perform a `Put` operation on the ACL table, simulating a `grant` or `revoke` and circumventing the authorization check for `Global Admin` permissions. +Also be aware that a `Global Admin` with `Create` permission can perform a `Put` operation on the ACL table, simulating a `grant` or `revoke` and circumventing the authorization check for `Global Admin` permissions. -Due to these issues, be cautious with granting `Global Admin` privileges. +Due to these issues, be cautious with granting `Global Admin` privileges. ==== -* +Namespace Admins+ - a namespace admin with `Create` permissions can create or drop tables within that namespace, and take and restore snapshots. +* _Namespace Admins_ - a namespace admin with `Create` permissions can create or drop tables within that namespace, and take and restore snapshots. A namespace admin with `Admin` permissions can perform operations such as splits or major compactions on tables within that namespace. -* +Table Admins+ - A table admin can perform administrative operations only on that table. +* _Table Admins_ - A table admin can perform administrative operations only on that table. A table admin with `Create` permissions can create snapshots from that table or restore that table from a snapshot. A table admin with `Admin` permissions can perform operations such as splits or major compactions on that table. -* +Users+ - Users can read or write data, or both. +* _Users_ - Users can read or write data, or both. Users can also execute coprocessor endpoints, if given `Executable` permissions. .Real-World Example of Access Levels @@ -625,17 +622,16 @@ Due to these issues, be cautious with granting `Global Admin` p | Scope | Permissions | Description + | Senior Administrator | Global | Access, Create -| Manages the cluster and gives access to Junior - Administrators. +| Manages the cluster and gives access to Junior Administrators. | Junior Administrator | Global | Create -| Creates tables and gives access to Table - Administrators. +| Creates tables and gives access to Table Administrators. | Table Administrator | Table @@ -650,8 +646,7 @@ Due to these issues, be cautious with granting `Global Admin` p | Web Application | Table | Read, Write -| Puts data into HBase and uses HBase data to perform - operations. +| Puts data into HBase and uses HBase data to perform operations. |=== .ACL Matrix @@ -659,7 +654,7 @@ For more details on how ACLs map to specific HBase operations and tasks, see <<a ===== Implementation Details -Cell-level ACLs are implemented using tags (see <<hbase.tags,hbase.tags>>). In order to use cell-level ACLs, you must be using HFile v3 and HBase 0.98 or newer. +Cell-level ACLs are implemented using tags (see <<hbase.tags>>). In order to use cell-level ACLs, you must be using HFile v3 and HBase 0.98 or newer. . Files created by HBase are owned by the operating system user running the HBase process. To interact with HBase files, you should use the API or bulk load facility. @@ -670,13 +665,12 @@ Cell-level ACLs are implemented using tags (see <<hbase.tags,hbase.tags>>). In o ===== Server-Side Configuration - -. As a prerequisite, perform the steps in <<security.data.basic.server.side,security.data.basic.server.side>>. +. As a prerequisite, perform the steps in <<security.data.basic.server.side>>. . Install and configure the AccessController coprocessor, by setting the following properties in _hbase-site.xml_. - These properties take a list of classes. + These properties take a list of classes. + NOTE: If you use the AccessController along with the VisibilityController, the AccessController must come first in the list, because with both components active, the VisibilityController will delegate access control on its system tables to the AccessController. -For an example of using both together, see <<security.example.config,security.example.config>>. +For an example of using both together, see <<security.example.config>>. + [source,xml] ---- @@ -698,7 +692,7 @@ For an example of using both together, see <<security.example.config,security.ex </property> ---- + -Optionally, you can enable transport security, by setting +hbase.rpc.protection+ to `auth-conf`. +Optionally, you can enable transport security, by setting `hbase.rpc.protection` to `auth-conf`. This requires HBase 0.98.4 or newer. . Set up the Hadoop group mapper in the Hadoop namenode's _core-site.xml_. @@ -756,11 +750,11 @@ This requires HBase 0.98.4 or newer. . Optionally, enable the early-out evaluation strategy. Prior to HBase 0.98.0, if a user was not granted access to a column family, or at least a column qualifier, an AccessDeniedException would be thrown. HBase 0.98.0 removed this exception in order to allow cell-level exceptional grants. - To restore the old behavior in HBase 0.98.0-0.98.6, set +hbase.security.access.early_out+ to `true` in _hbase-site.xml_. + To restore the old behavior in HBase 0.98.0-0.98.6, set `hbase.security.access.early_out` to `true` in _hbase-site.xml_. In HBase 0.98.6, the default has been returned to `true`. . Distribute your configuration and restart your cluster for changes to take effect. -. To test your configuration, log into HBase Shell as a given user and use the +whoami+ command to report the groups your user is part of. - In this example, the user is reported as being a member of the `services` group. +. To test your configuration, log into HBase Shell as a given user and use the `whoami` command to report the groups your user is part of. + In this example, the user is reported as being a member of the `services` group. + ---- hbase> whoami @@ -798,7 +792,7 @@ grant 'user', 'RWXCA', 'TABLE', 'CF', 'CQ' ---- + Groups and users are granted access in the same way, but groups are prefixed with an `@` symbol. -In the same way, tables and namespaces are specified in the same way, but namespaces are prefixed with an `@` symbol. +In the same way, tables and namespaces are specified in the same way, but namespaces are prefixed with an `@` symbol. + It is also possible to grant multiple permissions against the same resource in a single statement, as in this example. The first sub-clause maps users to ACLs and the second sub-clause specifies the resource. @@ -853,9 +847,9 @@ grant <table>, \ { <scanner-specification> } ---- + -* [replaceable]_<user-or-group>_ is the user or group name, prefixed with `@` in the case of a group. -* [replaceable]_<permissions>_ is a string containing any or all of "RWXCA", though only R and W are meaningful at cell scope. -* [replaceable]_<scanner-specification>_ is the scanner specification syntax and conventions used by the 'scan' shell command. +* _<user-or-group>_ is the user or group name, prefixed with `@` in the case of a group. +* _<permissions>_ is a string containing any or all of "RWXCA", though only R and W are meaningful at cell scope. +* _<scanner-specification>_ is the scanner specification syntax and conventions used by the 'scan' shell command. For some examples of scanner specifications, issue the following HBase Shell command. + ---- @@ -920,9 +914,9 @@ put.setACL(âuser1â, new Permission(Permission.Action.READ)) . Revoking Access Control From a Namespace, Table, Column Family, or Cell + -The +revoke+ command and API are twins of the grant command and API, and the syntax is exactly the same. +The `revoke` command and API are twins of the grant command and API, and the syntax is exactly the same. The only exception is that you cannot revoke permissions at the cell level. -You can only revoke access that has previously been granted, and a +revoke+ statement is not the same thing as explicit denial to a resource. +You can only revoke access that has previously been granted, and a `revoke` statement is not the same thing as explicit denial to a resource. + NOTE: HBase Shell support for granting and revoking access is for testing and verification support, and should not be employed for production use because it won't apply the permissions to cells that don't exist yet. The correct way to apply cell-level permissions is to do so in the application code when storing the values. @@ -976,12 +970,12 @@ public static void verifyAllowed(User user, AccessTestAction action, int count) if (obj != null && obj instanceof List<?>) { List<?> results = (List<?>) obj; if (results != null && results.isEmpty()) { - fail("Empty non null results from action for user '" + user.getShortName() + "'"); + fail("Empty non null results from action for user '" ` user.getShortName() ` "'"); } assertEquals(count, results.size()); } } catch (AccessDeniedException ade) { - fail("Expected action to pass for user '" + user.getShortName() + "' but was denied"); + fail("Expected action to pass for user '" ` user.getShortName() ` "' but was denied"); } } ---- @@ -1000,25 +994,25 @@ Visibility labels have no meaning on their own, and may be used to denote sensit If a user's labels do not match a cell's label or expression, the user is denied access to the cell. In HBase 0.98.6 and newer, UTF-8 encoding is supported for visibility labels and expressions. -When creating labels using the `addLabels(conf, labels)` method provided by the `org.apache.hadoop.hbase.security.visibility.VisibilityClient` class and passing labels in Authorizations via Scan or Get, labels can contain UTF-8 characters, as well as the logical operators normally used in visibility labels, with normal Java notations, without needing any escaping method. +When creating labels using the `addLabels(conf, labels)` method provided by the `org.apache.hadoop.hbase.security.visibility.VisibilityClient` class and passing labels in Authorizations via Scan or Get, labels can contain UTF-8 characters, as well as the logical operators normally used in visibility labels, with normal Java notations, without needing any escaping method. However, when you pass a CellVisibility expression via a Mutation, you must enclose the expression with the `CellVisibility.quote()` method if you use UTF-8 characters or logical operators. -See `TestExpressionParser` and the source file _hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestScan.java_. +See `TestExpressionParser` and the source file _hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestScan.java_. A user adds visibility expressions to a cell during a Put operation. -In the default configuration, the user does not need to access to a label in order to label cells with it. -This behavior is controlled by the configuration option +hbase.security.visibility.mutations.checkauths+. +In the default configuration, the user does not need to have access to a label in order to label cells with it. +This behavior is controlled by the configuration option `hbase.security.visibility.mutations.checkauths`. If you set this option to `true`, the labels the user is modifying as part of the mutation must be associated with the user, or the mutation will fail. Whether a user is authorized to read a labelled cell is determined during a Get or Scan, and results which the user is not allowed to read are filtered out. This incurs the same I/O penalty as if the results were returned, but reduces load on the network. Visibility labels can also be specified during Delete operations. -For details about visibility labels and Deletes, see link:https://issues.apache.org/jira/browse/HBASE-10885[HBASE-10885]. +For details about visibility labels and Deletes, see link:https://issues.apache.org/jira/browse/HBASE-10885[HBASE-10885]. The user's effective label set is built in the RPC context when a request is first received by the RegionServer. The way that users are associated with labels is pluggable. The default plugin passes through labels specified in Authorizations added to the Get or Scan and checks those against the calling user's authenticated labels list. When the client passes labels for which the user is not authenticated, the default plugin drops them. -You can pass a subset of user authenticated labels via the `Get#setAuthorizations(Authorizations(String,...))` and `Scan#setAuthorizations(Authorizations(String,...));` methods. +You can pass a subset of user authenticated labels via the `Get#setAuthorizations(Authorizations(String,...))` and `Scan#setAuthorizations(Authorizations(String,...));` methods. Visibility label access checking is performed by the VisibilityController coprocessor. You can use interface `VisibilityLabelService` to provide a custom implementation and/or control the way that visibility labels are stored with cells. @@ -1026,29 +1020,32 @@ See the source file _hbase-server/src/test/java/org/apache/hadoop/hbase/security Visibility labels can be used in conjunction with ACLs. +NOTE: The labels have to be explicitly defined before they can be used in visibility labels. See below for an example of how this can be done. + +NOTE: There is currently no way to determine which labels have been applied to a cell. See link:https://issues.apache.org/jira/browse/HBASE-12470[HBASE-12470] for details. + +NOTE: Visibility labels are not currently applied for superusers. + .Examples of Visibility Expressions [cols="l,1", options="header"] |=== | Expression | Interpretation + | fulltime -| Allow accesss to users associated with the - fulltime label. +| Allow accesss to users associated with the fulltime label. | !public -| Allow access to users not associated with the - public label. +| Allow access to users not associated with the public label. | ( secret \| topsecret ) & !probationary -| Allow access to users associated with either the - secret or topsecret label and not - associated with the probationary label. +| Allow access to users associated with either the secret or topsecret label and not associated with the probationary label. |=== ==== Server-Side Configuration -. As a prerequisite, perform the steps in <<security.data.basic.server.side,security.data.basic.server.side>>. +. As a prerequisite, perform the steps in <<security.data.basic.server.side>>. . Install and configure the VisibilityController coprocessor by setting the following properties in _hbase-site.xml_. These properties take a list of class names. + @@ -1070,7 +1067,7 @@ NOTE: If you use the AccessController and VisibilityController coprocessors toge + By default, users can label cells with any label, including labels they are not associated with, which means that a user can Put data that he cannot read. For example, a user could label a cell with the (hypothetical) 'topsecret' label even if the user is not associated with that label. -If you only want users to be able to label cells with labels they are associated with, set +hbase.security.visibility.mutations.checkauths+ to `true`. +If you only want users to be able to label cells with labels they are associated with, set `hbase.security.visibility.mutations.checkauths` to `true`. In that case, the mutation will fail if it makes use of labels the user is not associated with. . Distribute your configuration and restart your cluster for changes to take effect. @@ -1104,10 +1101,8 @@ hbase> add_labels [ 'admin', 'service', 'developer', 'test' ] ==== [source,java] ---- - public static void addLabels() throws Exception { - PrivilegedExceptionAction<VisibilityLabelsResponse> action = - new PrivilegedExceptionAction<VisibilityLabelsResponse>() { + PrivilegedExceptionAction<VisibilityLabelsResponse> action = new PrivilegedExceptionAction<VisibilityLabelsResponse>() { public VisibilityLabelsResponse run() throws Exception { String[] labels = { SECRET, TOPSECRET, CONFIDENTIAL, PUBLIC, PRIVATE, COPYRIGHT, ACCENT, UNICODE_VIS_TAG, UC1, UC2 }; @@ -1145,7 +1140,6 @@ hbase> set_auths 'qa', [ 'test', 'developer' ] ==== [source,java] ---- - public void testSetAndGetUserAuths() throws Throwable { final String user = "user1"; PrivilegedExceptionAction<Void> action = new PrivilegedExceptionAction<Void>() { @@ -1182,7 +1176,6 @@ hbase> clear_auths 'qa', [ 'test', 'developer' ] ==== [source,java] ---- - ... auths = new String[] { SECRET, PUBLIC, CONFIDENTIAL }; VisibilityLabelsResponse response = null; @@ -1190,7 +1183,8 @@ try { response = VisibilityClient.clearAuths(conf, auths, user); } catch (Throwable e) { fail("Should not have failed"); -... + ... +} ---- ==== @@ -1202,18 +1196,15 @@ The label is associated with a given version of the cell. .HBase Shell ==== ---- -hbase> set_visibility 'user', 'admin|service|developer', \ - { COLUMNS => 'i' } +hbase> set_visibility 'user', 'admin|service|developer', { COLUMNS => 'i' } ---- ---- -hbase> set_visibility 'user', 'admin|service', \ - { COLUMNS => ' pii' } +hbase> set_visibility 'user', 'admin|service', { COLUMNS => 'pii' } ---- ---- -hbase> COLUMNS => [ 'i', 'pii' ], \ - FILTER => "(PrefixFilter ('test'))" } +hbase> set_visibility 'user', 'test', { COLUMNS => [ 'i', 'pii' ], FILTER => "(PrefixFilter ('test'))" } ---- ==== + @@ -1257,7 +1248,7 @@ You can also configure a set of `ScanLabelGenerators` to be used by the system, ==== Replicating Visibility Tags as Strings -As mentioned in the above sections, the interface `VisibilityLabelService` could be used to implement a different way of storing the visibility expressions in the cells. Clusters with replication enabled also must replicate the visibility expressions to the peer cluster. If `DefaultVisibilityLabelServiceImpl` is used as the implementation for `VisibilityLabelService`, all the visibility expression are converted to the corresponding expression based on the ordinals for each visibility label stored in the labels table. During replication, visible cellsare also replicated with the ordinal-based expression intact. The peer cluster may not have the same `labels` table with the same ordinal mapping for the visibility labels. In that case, replicating the ordinals makes no sense. It would be better if the replication occurred with the visibility expressions transmitted as strings. To replicate the visibility expression as strings to the peer cluster, create a `RegionServerObserver` config uration which works based on the implementation of the `VisibilityLabelService` interface. The configuration below enables replication of visibility expressions to peer clusters as strings. See link:https://issues.apache.org/jira/browse/HBASE-11639[HBASE-11639] for more details. +As mentioned in the above sections, the interface `VisibilityLabelService` could be used to implement a different way of storing the visibility expressions in the cells. Clusters with replication enabled also must replicate the visibility expressions to the peer cluster. If `DefaultVisibilityLabelServiceImpl` is used as the implementation for `VisibilityLabelService`, all the visibility expression are converted to the corresponding expression based on the ordinals for each visibility label stored in the labels table. During replication, visible cells are also replicated with the ordinal-based expression intact. The peer cluster may not have the same `labels` table with the same ordinal mapping for the visibility labels. In that case, replicating the ordinals makes no sense. It would be better if the replication occurred with the visibility expressions transmitted as strings. To replicate the visibility expression as strings to the peer cluster, create a `RegionServerObserver` config uration which works based on the implementation of the `VisibilityLabelService` interface. The configuration below enables replication of visibility expressions to peer clusters as strings. See link:https://issues.apache.org/jira/browse/HBASE-11639[HBASE-11639] for more details. [source,xml] ---- @@ -1286,7 +1277,7 @@ The master key may be stored on the cluster servers, protected by a secure KeySt This master key is resolved as needed by HBase processes through the configured key provider. Next, encryption use can be specified in the schema, per column family, by creating or modifying a column descriptor to include two additional attributes: the name of the encryption algorithm to use (currently only "AES" is supported), and optionally, a data key wrapped (encrypted) with the cluster master key. -If a data key is not explictly configured for a ColumnFamily, HBase will create a random data key per HFile. +If a data key is not explicitly configured for a ColumnFamily, HBase will create a random data key per HFile. This provides an incremental improvement in security over the alternative. Unless you need to supply an explicit data key, such as in a case where you are generating encrypted HFiles for bulk import with a given data key, only specify the encryption algorithm in the ColumnFamily schema metadata and let HBase create data keys on demand. Per Column Family keys facilitate low impact incremental key rotation and reduce the scope of any external leak of key material. @@ -1338,23 +1329,23 @@ In the example below, replace [replaceable]_****_ with the password. [source,xml] ---- <property> - <name>hbase.crypto.keyprovider</name> - <value>org.apache.hadoop.hbase.io.crypto.KeyStoreKeyProvider</value> + <name>hbase.crypto.keyprovider</name> + <value>org.apache.hadoop.hbase.io.crypto.KeyStoreKeyProvider</value> </property> <property> - <name>hbase.crypto.keyprovider.parameters</name> - <value>jceks:///path/to/hbase/conf/hbase.jks?password=****</value> + <name>hbase.crypto.keyprovider.parameters</name> + <value>jceks:///path/to/hbase/conf/hbase.jks?password=****</value> </property> ---- + By default, the HBase service account name will be used to resolve the cluster master key. -However, you can store it with an arbitrary alias (in the +keytool+ command). In that case, set the following property to the alias you used. +However, you can store it with an arbitrary alias (in the `keytool` command). In that case, set the following property to the alias you used. + [source,xml] ---- <property> - <name>hbase.crypto.master.key.name</name> - <value>my-alias</value> + <name>hbase.crypto.master.key.name</name> + <value>my-alias</value> </property> ---- + @@ -1365,24 +1356,22 @@ For previous versions, set the following property in your _hbase-site.xml_ [source,xml] ---- <property> - <name>hfile.format.version</name> - <value>3</value> + <name>hfile.format.version</name> + <value>3</value> </property> ---- + -Optionally, you can use a different cipher provider, either a Java Cryptography Encryption (JCE) algorithm provider or a custom HBase cipher implementation. +Optionally, you can use a different cipher provider, either a Java Cryptography Encryption (JCE) algorithm provider or a custom HBase cipher implementation. + -* JCE: -+ -* Install a signed JCE provider (supporting ``AES/CTR/NoPadding'' mode with 128 bit keys) -* Add it with highest preference to the JCE site configuration file _$JAVA_HOME/lib/security/java.security_. -* Update +hbase.crypto.algorithm.aes.provider+ and +hbase.crypto.algorithm.rng.provider+ options in _hbase-site.xml_. +* JCE: +** Install a signed JCE provider (supporting `AES/CTR/NoPadding` mode with 128 bit keys) +** Add it with highest preference to the JCE site configuration file _$JAVA_HOME/lib/security/java.security_. +** Update `hbase.crypto.algorithm.aes.provider` and `hbase.crypto.algorithm.rng.provider` options in [path]_hbase-site.xml_. -* Custom HBase Cipher: -+ -* Implement `org.apache.hadoop.hbase.io.crypto.CipherProvider`. -* Add the implementation to the server classpath. -* Update +hbase.crypto.cipherprovider+ in _hbase-site.xml_. +* Custom HBase Cipher: +** Implement `org.apache.hadoop.hbase.io.crypto.CipherProvider`. +** Add the implementation to the server classpath. +** Update `hbase.crypto.cipherprovider` in _hbase-site.xml_. . Configure WAL encryption. @@ -1452,10 +1441,12 @@ Rotate the Master Key:: [[hbase.secure.bulkload]] === Secure Bulk Load -Bulk loading in secure mode is a bit more involved than normal setup, since the client has to transfer the ownership of the files generated from the mapreduce job to HBase. -Secure bulk loading is implemented by a coprocessor, named link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.html[SecureBulkLoadEndpoint], which uses a staging directory configured by the configuration property +hbase.bulkload.staging.dir+, which defaults to _/tmp/hbase-staging/_. +Bulk loading in secure mode is a bit more involved than normal setup, since the client has to transfer the ownership of the files generated from the MapReduce job to HBase. +Secure bulk loading is implemented by a coprocessor, named link:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.html[SecureBulkLoadEndpoint], which uses a staging directory configured by the configuration property `hbase.bulkload.staging.dir`, which defaults to _/tmp/hbase-staging/_. + +.Secure Bulk Load Algorithm -* .Secure Bulk Load AlgorithmOne time only, create a staging directory which is world-traversable and owned by the user which runs HBase (mode 711, or `rwx--x--x`). A listing of this directory will look similar to the following: +* One time only, create a staging directory which is world-traversable and owned by the user which runs HBase (mode 711, or `rwx--x--x`). A listing of this directory will look similar to the following: + [source,bash] ---- @@ -1468,7 +1459,7 @@ drwx--x--x 2 hbase hbase 68 3 Sep 14:54 /tmp/hbase-staging * Internally, HBase creates a secret staging directory which is globally readable/writable (`-rwxrwxrwx, 777`). For example, _/tmp/hbase-staging/averylongandrandomdirectoryname_. The name and location of this directory is not exposed to the user. HBase manages creation and deletion of this directory. -* The user makes the data world-readable and world-writable, moves it into the random staging directory, then calls the `SecureBulkLoadClient#bulkLoadHFiles` method. +* The user makes the data world-readable and world-writable, moves it into the random staging directory, then calls the `SecureBulkLoadClient#bulkLoadHFiles` method. The strength of the security lies in the length and randomness of the secret directory. @@ -1541,29 +1532,29 @@ All options have been discussed separately in the sections above. </property> <!-- Transparent Encryption --> <property> - <name>hbase.crypto.keyprovider</name> - <value>org.apache.hadoop.hbase.io.crypto.KeyStoreKeyProvider</value> + <name>hbase.crypto.keyprovider</name> + <value>org.apache.hadoop.hbase.io.crypto.KeyStoreKeyProvider</value> </property> <property> - <name>hbase.crypto.keyprovider.parameters</name> - <value>jceks:///path/to/hbase/conf/hbase.jks?password=***</value> + <name>hbase.crypto.keyprovider.parameters</name> + <value>jceks:///path/to/hbase/conf/hbase.jks?password=***</value> </property> <property> - <name>hbase.crypto.master.key.name</name> - <value>hbase</value> + <name>hbase.crypto.master.key.name</name> + <value>hbase</value> </property> <!-- WAL Encryption --> <property> - <name>hbase.regionserver.hlog.reader.impl</name> - <value>org.apache.hadoop.hbase.regionserver.wal.SecureProtobufLogReader</value> + <name>hbase.regionserver.hlog.reader.impl</name> + <value>org.apache.hadoop.hbase.regionserver.wal.SecureProtobufLogReader</value> </property> <property> - <name>hbase.regionserver.hlog.writer.impl</name> - <value>org.apache.hadoop.hbase.regionserver.wal.SecureProtobufLogWriter</value> + <name>hbase.regionserver.hlog.writer.impl</name> + <value>org.apache.hadoop.hbase.regionserver.wal.SecureProtobufLogWriter</value> </property> <property> - <name>hbase.regionserver.wal.encryption</name> - <value>true</value> + <name>hbase.regionserver.wal.encryption</name> + <value>true</value> </property> <!-- For key rotation --> <property>
http://git-wip-us.apache.org/repos/asf/hbase/blob/fba353df/src/main/asciidoc/_chapters/shell.adoc ---------------------------------------------------------------------- diff --git a/src/main/asciidoc/_chapters/shell.adoc b/src/main/asciidoc/_chapters/shell.adoc index 1b8d8a0..237089e 100644 --- a/src/main/asciidoc/_chapters/shell.adoc +++ b/src/main/asciidoc/_chapters/shell.adoc @@ -38,13 +38,13 @@ To run the HBase shell, do as follows: $ ./bin/hbase shell ---- -Type +help+ and then +<RETURN>+ to see a listing of shell commands and options. -Browse at least the paragraphs at the end of the help emission for the gist of how variables and command arguments are entered into the HBase shell; in particular note how table names, rows, and columns, etc., must be quoted. +Type `help` and then `<RETURN>` to see a listing of shell commands and options. +Browse at least the paragraphs at the end of the help output for the gist of how variables and command arguments are entered into the HBase shell; in particular note how table names, rows, and columns, etc., must be quoted. -See <<shell_exercises,shell exercises>> for example basic shell operation. +See <<shell_exercises,shell exercises>> for example basic shell operation. Here is a nicely formatted listing of link:http://learnhbase.wordpress.com/2013/03/02/hbase-shell-commands/[all shell - commands] by Rajeshbabu Chintaguntla. + commands] by Rajeshbabu Chintaguntla. [[scripting]] == Scripting with Ruby @@ -64,27 +64,26 @@ A new non-interactive mode has been added to the HBase Shell (link:https://issue Non-interactive mode captures the exit status (success or failure) of HBase Shell commands and passes that status back to the command interpreter. If you use the normal interactive mode, the HBase Shell will only ever return its own exit status, which will nearly always be `0` for success. -To invoke non-interactive mode, pass the +-n+ or +--non-interactive+ option to HBase Shell. +To invoke non-interactive mode, pass the `-n` or `--non-interactive` option to HBase Shell. [[hbase.shell.noninteractive]] == HBase Shell in OS Scripts You can use the HBase shell from within operating system script interpreters like the Bash shell which is the default command interpreter for most Linux and UNIX distributions. -The following guidelines use Bash syntax, but could be adjusted to work with C-style shells such as csh or tcsh, and could probably be modified to work with the Microsoft Windows script interpreter as well. -Submissions are welcome. +The following guidelines use Bash syntax, but could be adjusted to work with C-style shells such as csh or tcsh, and could probably be modified to work with the Microsoft Windows script interpreter as well. Submissions are welcome. NOTE: Spawning HBase Shell commands in this way is slow, so keep that in mind when you are deciding when combining HBase operations with the operating system command line is appropriate. .Passing Commands to the HBase Shell ==== -You can pass commands to the HBase Shell in non-interactive mode (see <<hbasee.shell.noninteractive,hbasee.shell.noninteractive>>) using the +echo+ command and the `|` (pipe) operator. +You can pass commands to the HBase Shell in non-interactive mode (see <<hbasee.shell.noninteractive,hbasee.shell.noninteractive>>) using the `echo` command and the `|` (pipe) operator. Be sure to escape characters in the HBase commands which would otherwise be interpreted by the shell. Some debug-level output has been truncated from the example below. [source,bash] ---- $ echo "describe 'test1'" | ./hbase shell -n - + Version 0.98.3-hadoop2, rd5e65a9144e315bb0a964e7730871af32f5018d5, Sat May 31 19:56:09 PDT 2014 describe 'test1' @@ -122,7 +121,7 @@ This is a naive script that shows one way to store the return value and make a d echo "describe 'test'" | ./hbase shell -n > /dev/null 2>&1 status=$? -echo "The status was " $status +echo "The status was " $status if ($status == 0); then echo "The command succeeded" else @@ -134,7 +133,7 @@ return $status === Checking for Success or Failure In Scripts -Getting an exit code of 0 means that the command you scripted definitely succeeded. +Getting an exit code of `0` means that the command you scripted definitely succeeded. However, getting a non-zero exit code does not necessarily mean the command failed. The command could have succeeded, but the client lost connectivity, or some other event obscured its success. This is because RPC commands are stateless. @@ -163,10 +162,9 @@ enable 'test' .Directing HBase Shell to Execute the Commands ==== -Pass the path to the command file as the only argument to the +hbase - shell+ command. +Pass the path to the command file as the only argument to the `hbase shell` command. Each command is executed and its output is shown. -If you do not include the +exit+ command in your script, you are returned to the HBase shell prompt. +If you do not include the `exit` command in your script, you are returned to the HBase shell prompt. There is no way to programmatically check each individual command for success or failure. Also, though you see the output for each command, the commands themselves are not echoed to the screen so it can be difficult to line up the command with its output. @@ -206,14 +204,14 @@ COLUMN CELL == Passing VM Options to the Shell -You can pass VM options to the HBase Shell using the `HBASE_SHELL_OPTS` environment variable. +You can pass VM options to the HBase Shell using the `HBASE_SHELL_OPTS` environment variable. You can set this in your environment, for instance by editing _~/.bashrc_, or set it as part of the command to launch HBase Shell. The following example sets several garbage-collection-related variables, just for the lifetime of the VM running the HBase Shell. The command should be run all on a single line, but is broken by the `\` character, for readability. [source,bash] ---- -$ HBASE_SHELL_OPTS="-verbose:gc -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps \ +$ HBASE_SHELL_OPTS="-verbose:gc -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps \ -XX:+PrintGCDetails -Xloggc:$HBASE_HOME/logs/gc-hbase.log" ./bin/hbase shell ---- @@ -221,10 +219,10 @@ $ HBASE_SHELL_OPTS="-verbose:gc -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCD === Table variables -HBase 0.95 adds shell commands that provide a jruby-style object-oriented references for tables. +HBase 0.95 adds shell commands that provides jruby-style object-oriented references for tables. Previously all of the shell commands that act upon a table have a procedural style that always took the name of the table as an argument. HBase 0.95 introduces the ability to assign a table to a jruby variable. -The table reference can be used to perform data read write operations such as puts, scans, and gets well as admin functionality such as disabling, dropping, describing tables. +The table reference can be used to perform data read write operations such as puts, scans, and gets well as admin functionality such as disabling, dropping, describing tables. For example, previously you would always specify a table name: @@ -234,17 +232,17 @@ hbase(main):000:0> create âtâ, âfâ hbase(main):001:0> put 't', 'rold', 'f', 'v' 0 row(s) in 0.0080 seconds -hbase(main):002:0> scan 't' -ROW COLUMN+CELL - rold column=f:, timestamp=1378473207660, value=v +hbase(main):002:0> scan 't' +ROW COLUMN+CELL + rold column=f:, timestamp=1378473207660, value=v 1 row(s) in 0.0130 seconds hbase(main):003:0> describe 't' -DESCRIPTION ENABLED - 't', {NAME => 'f', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_ true - SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2 - 147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false - ', BLOCKCACHE => 'true'} +DESCRIPTION ENABLED + 't', {NAME => 'f', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_ true + SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2 + 147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false + ', BLOCKCACHE => 'true'} 1 row(s) in 1.4430 seconds hbase(main):004:0> disable 't' @@ -266,15 +264,15 @@ hbase(main):007 > t = create 't', 'f' hbase(main):008 > t.put 'r', 'f', 'v' 0 row(s) in 0.0640 seconds hbase(main):009 > t.scan -ROW COLUMN+CELL - r column=f:, timestamp=1331865816290, value=v +ROW COLUMN+CELL + r column=f:, timestamp=1331865816290, value=v 1 row(s) in 0.0110 seconds hbase(main):010:0> t.describe -DESCRIPTION ENABLED - 't', {NAME => 'f', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_ true - SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2 - 147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false - ', BLOCKCACHE => 'true'} +DESCRIPTION ENABLED + 't', {NAME => 'f', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_ true + SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2 + 147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false + ', BLOCKCACHE => 'true'} 1 row(s) in 0.0210 seconds hbase(main):038:0> t.disable 0 row(s) in 6.2350 seconds @@ -293,11 +291,11 @@ hbase(main):012:0> tab = get_table 't' 0 row(s) in 0.0010 seconds => Hbase::Table - t -hbase(main):013:0> tab.put âr1â ,âfâ, âvâ +hbase(main):013:0> tab.put âr1â ,âfâ, âvâ 0 row(s) in 0.0100 seconds hbase(main):014:0> tab.scan -ROW COLUMN+CELL - r1 column=f:, timestamp=1378473876949, value=v +ROW COLUMN+CELL + r1 column=f:, timestamp=1378473876949, value=v 1 row(s) in 0.0240 seconds hbase(main):015:0> ---- @@ -308,8 +306,8 @@ The list_snapshots command also acts similarly. ---- hbase(main):016 > tables = list(ât.*â) -TABLE -t +TABLE +t 1 row(s) in 0.1040 seconds => #<#<Class:0x7677ce29>:0x21d377a4> @@ -333,7 +331,7 @@ IRB.conf[:SAVE_HISTORY] = 100 IRB.conf[:HISTORY_FILE] = "#{ENV['HOME']}/.irb-save-history" ---- -See the +ruby+ documentation of _.irbrc_ to learn about other possible configurations. +See the `ruby` documentation of _.irbrc_ to learn about other possible configurations. === LOG data to timestamp @@ -352,7 +350,7 @@ hbase(main):021:0> import java.util.Date hbase(main):022:0> Date.new(1218920189000).toString() => "Sat Aug 16 20:56:29 UTC 2008" ---- -To output in a format that is exactly like that of the HBase log format will take a little messing with link:http://download.oracle.com/javase/6/docs/api/java/text/SimpleDateFormat.html[SimpleDateFormat]. +To output in a format that is exactly like that of the HBase log format will take a little messing with link:http://download.oracle.com/javase/6/docs/api/java/text/SimpleDateFormat.html[SimpleDateFormat]. === Debug @@ -368,7 +366,7 @@ hbase> debug <RETURN> ==== DEBUG log level -To enable DEBUG level logging in the shell, launch it with the +-d+ option. +To enable DEBUG level logging in the shell, launch it with the `-d` option. [source,bash] ---- @@ -380,13 +378,13 @@ $ ./bin/hbase shell -d ==== count Count command returns the number of rows in a table. -It's quite fast when configured with the right CACHE +It's quite fast when configured with the right CACHE [source] ---- hbase> count '<tablename>', CACHE => 1000 ----- +---- The above count fetches 1000 rows at a time. Set CACHE lower if your rows are big. -Default is to fetch one row at a time. +Default is to fetch one row at a time. http://git-wip-us.apache.org/repos/asf/hbase/blob/fba353df/src/main/asciidoc/_chapters/thrift_filter_language.adoc ---------------------------------------------------------------------- diff --git a/src/main/asciidoc/_chapters/thrift_filter_language.adoc b/src/main/asciidoc/_chapters/thrift_filter_language.adoc index 46f816a..744cec6 100644 --- a/src/main/asciidoc/_chapters/thrift_filter_language.adoc +++ b/src/main/asciidoc/_chapters/thrift_filter_language.adoc @@ -31,18 +31,18 @@ Apache link:http://thrift.apache.org/[Thrift] is a cross-platform, cross-language development framework. HBase includes a Thrift API and filter language. The Thrift API relies on client and server processes. -Documentation about the HBase Thrift API is located at link:http://wiki.apache.org/hadoop/Hbase/ThriftApi. +Documentation about the HBase Thrift API is located at http://wiki.apache.org/hadoop/Hbase/ThriftApi. -You can configure Thrift for secure authentication at the server and client side, by following the procedures in <<security.client.thrift,security.client.thrift>> and <<security.gateway.thrift,security.gateway.thrift>>. +You can configure Thrift for secure authentication at the server and client side, by following the procedures in <<security.client.thrift>> and <<security.gateway.thrift>>. The rest of this chapter discusses the filter language provided by the Thrift API. [[thrift.filter_language]] == Filter Language -Thrift Filter Language was introduced in APache HBase 0.92. +Thrift Filter Language was introduced in HBase 0.92. It allows you to perform server-side filtering when accessing HBase over Thrift or in the HBase shell. -You can find out more about shell integration by using the `scan help` command in the shell. +You can find out more about shell integration by using the `scan help` command in the shell. You specify a filter as a string, which is parsed on the server to construct the filter. @@ -69,7 +69,7 @@ Keep the following syntax guidelines in mind. .Binary Operators `AND`:: - If the `AND` operator is used, the key-vallue must satisfy both the filters. + If the `AND` operator is used, the key-value must satisfy both filters. `OR`:: If the `OR` operator is used, the key-value must satisfy at least one of the filters. @@ -79,7 +79,7 @@ Keep the following syntax guidelines in mind. For a particular row, if any of the key-values fail the filter condition, the entire row is skipped. `WHILE`:: - For a particular row, key-values will be emitted until a key-value is reached t hat fails the filter condition. + For a particular row, key-values will be emitted until a key-value is reached that fails the filter condition. .Compound Operators ==== @@ -142,8 +142,7 @@ A comparator can be any of the following: The comparison is case insensitive. Only EQUAL and NOT_EQUAL comparisons are valid with this comparator -The general syntax of a comparator is:` - ComparatorType:ComparatorValue` +The general syntax of a comparator is: `ComparatorType:ComparatorValue` The ComparatorType for the various comparators is as follows: @@ -155,7 +154,7 @@ The ComparatorType for the various comparators is as follows: The ComparatorValue can be any value. .Example ComparatorValues -. `binary:abc` will match everything that is lexicographically greater than "abc" +. `binary:abc` will match everything that is lexicographically greater than "abc" . `binaryprefix:abc` will match everything whose first 3 characters are lexicographically equal to "abc" . `regexstring:ab*yz` will match everything that doesn't begin with "ab" and ends with "yz" . `substring:abc123` will match everything that begins with the substring "abc123" @@ -165,48 +164,39 @@ The ComparatorValue can be any value. [source,php] ---- -<? $_SERVER['PHP_ROOT'] = realpath(dirname(__FILE__).'/..'); - require_once $_SERVER['PHP_ROOT'].'/flib/__flib.php'; - flib_init(FLIB_CONTEXT_SCRIPT); - require_module('storage/hbase'); - $hbase = new HBase('<server_name_running_thrift_server>', <port on which thrift server is running>); - $hbase->open(); - $client = $hbase->getClient(); - $result = $client->scannerOpenWithFilterString('table_name', "(PrefixFilter ('row2') AND (QualifierFilter (>=, 'binary:xyz'))) AND (TimestampsFilter ( 123, 456))"); - $to_print = $client->scannerGetList($result,1); - while ($to_print) { - print_r($to_print); - $to_print = $client->scannerGetList($result,1); - } - $client->scannerClose($result); +<? + $_SERVER['PHP_ROOT'] = realpath(dirname(__FILE__).'/..'); + require_once $_SERVER['PHP_ROOT'].'/flib/__flib.php'; + flib_init(FLIB_CONTEXT_SCRIPT); + require_module('storage/hbase'); + $hbase = new HBase('<server_name_running_thrift_server>', <port on which thrift server is running>); + $hbase->open(); + $client = $hbase->getClient(); + $result = $client->scannerOpenWithFilterString('table_name', "(PrefixFilter ('row2') AND (QualifierFilter (>=, 'binary:xyz'))) AND (TimestampsFilter ( 123, 456))"); + $to_print = $client->scannerGetList($result,1); + while ($to_print) { + print_r($to_print); + $to_print = $client->scannerGetList($result,1); + } + $client->scannerClose($result); ?> ---- === Example Filter Strings -* `âPrefixFilter (âRowâ) AND PageFilter (1) AND FirstKeyOnlyFilter - ()â` will return all key-value pairs that match the following conditions: +* `"PrefixFilter ('Row') AND PageFilter (1) AND FirstKeyOnlyFilter ()"` will return all key-value pairs that match the following conditions: + -. The row containing the key-value should have prefix ``Row'' -. The key-value must be located in the first row of the table -. The key-value pair must be the first key-value in the row - - - -* `â(RowFilter (=, âbinary:Row 1â) AND TimeStampsFilter (74689, - 89734)) OR ColumnRangeFilter (âabcâ, true, âxyzâ, - false))â` will return all key-value pairs that match both the following conditions: +. The row containing the key-value should have prefix _Row_ +. The key-value must be located in the first row of the table +. The key-value pair must be the first key-value in the row + -* The key-value is in a row having row key ``Row 1'' -* The key-value must have a timestamp of either 74689 or 89734. -* Or it must match the following condition: +* `"(RowFilter (=, 'binary:Row 1') AND TimeStampsFilter (74689, 89734)) OR ColumnRangeFilter ('abc', true, 'xyz', false))"` will return all key-value pairs that match both the following conditions: +** The key-value is in a row having row key _Row 1_ +** The key-value must have a timestamp of either 74689 or 89734. +** Or it must match the following condition: +*** The key-value pair must be in a column that is lexicographically >= abc and < xyz + -* The key-value pair must be in a column that is lexicographically >= abc and < xyz - - - - -* `âSKIP ValueFilter (0)â` will skip the entire row if any of the values in the row is not 0 +* `"SKIP ValueFilter (0)"` will skip the entire row if any of the values in the row is not 0 [[individualfiltersyntax]] === Individual Filter Syntax @@ -279,7 +269,7 @@ SingleColumnValueFilter:: This filter takes a column family, a qualifier, a compare operator and a comparator. If the specified column is not found â all the columns of that row will be emitted. If the column is found and the comparison with the comparator returns true, all the columns of the row will be emitted. - If the condition fails, the row will not be emitted. + If the condition fails, the row will not be emitted. SingleColumnValueExcludeFilter:: This filter takes the same arguments and behaves same as SingleColumnValueFilter â however, if the column is found and the condition passes, all the columns of the row will be emitted except for the tested column value.
