http://git-wip-us.apache.org/repos/asf/hbase-site/blob/ce958bce/book.html ---------------------------------------------------------------------- diff --git a/book.html b/book.html index af2676f..ef9d462 100644 --- a/book.html +++ b/book.html @@ -238,46 +238,47 @@ <li><a href="#snapshots_azure">139. Storing Snapshots in Microsoft Azure Blob Storage</a></li> <li><a href="#ops.capacity">140. Capacity Planning and Region Sizing</a></li> <li><a href="#table.rename">141. Table Rename</a></li> +<li><a href="#rsgroup">142. RegionServer Grouping</a></li> </ul> </li> <li><a href="#developer">Building and Developing Apache HBase</a> <ul class="sectlevel1"> -<li><a href="#getting.involved">142. Getting Involved</a></li> -<li><a href="#repos">143. Apache HBase Repositories</a></li> -<li><a href="#_ides">144. IDEs</a></li> -<li><a href="#build">145. Building Apache HBase</a></li> -<li><a href="#releasing">146. Releasing Apache HBase</a></li> -<li><a href="#hbase.rc.voting">147. Voting on Release Candidates</a></li> -<li><a href="#documentation">148. Generating the HBase Reference Guide</a></li> -<li><a href="#hbase.org">149. Updating <a href="http://hbase.apache.org";>hbase.apache.org</a></a></li> -<li><a href="#hbase.tests">150. Tests</a></li> -<li><a href="#developing">151. Developer Guidelines</a></li> +<li><a href="#getting.involved">143. Getting Involved</a></li> +<li><a href="#repos">144. Apache HBase Repositories</a></li> +<li><a href="#_ides">145. IDEs</a></li> +<li><a href="#build">146. Building Apache HBase</a></li> +<li><a href="#releasing">147. Releasing Apache HBase</a></li> +<li><a href="#hbase.rc.voting">148. Voting on Release Candidates</a></li> +<li><a href="#documentation">149. Generating the HBase Reference Guide</a></li> +<li><a href="#hbase.org">150. Updating <a href="http://hbase.apache.org";>hbase.apache.org</a></a></li> +<li><a href="#hbase.tests">151. Tests</a></li> +<li><a href="#developing">152. Developer Guidelines</a></li> </ul> </li> <li><a href="#unit.tests">Unit Testing HBase Applications</a> <ul class="sectlevel1"> -<li><a href="#_junit">152. JUnit</a></li> -<li><a href="#mockito">153. Mockito</a></li> -<li><a href="#_mrunit">154. MRUnit</a></li> -<li><a href="#_integration_testing_with_an_hbase_mini_cluster">155. Integration Testing with an HBase Mini-Cluster</a></li> +<li><a href="#_junit">153. JUnit</a></li> +<li><a href="#mockito">154. Mockito</a></li> +<li><a href="#_mrunit">155. MRUnit</a></li> +<li><a href="#_integration_testing_with_an_hbase_mini_cluster">156. Integration Testing with an HBase Mini-Cluster</a></li> </ul> </li> <li><a href="#protobuf">Protobuf in HBase</a> <ul class="sectlevel1"> -<li><a href="#_protobuf">156. Protobuf</a></li> +<li><a href="#_protobuf">157. Protobuf</a></li> </ul> </li> <li><a href="#zookeeper">ZooKeeper</a> <ul class="sectlevel1"> -<li><a href="#_using_existing_zookeeper_ensemble">157. Using existing ZooKeeper ensemble</a></li> -<li><a href="#zk.sasl.auth">158. SASL Authentication with ZooKeeper</a></li> +<li><a href="#_using_existing_zookeeper_ensemble">158. Using existing ZooKeeper ensemble</a></li> +<li><a href="#zk.sasl.auth">159. SASL Authentication with ZooKeeper</a></li> </ul> </li> <li><a href="#community">Community</a> <ul class="sectlevel1"> -<li><a href="#_decisions">159. Decisions</a></li> -<li><a href="#community.roles">160. Community Roles</a></li> -<li><a href="#hbase.commit.msg.format">161. Commit Message format</a></li> +<li><a href="#_decisions">160. Decisions</a></li> +<li><a href="#community.roles">161. Community Roles</a></li> +<li><a href="#hbase.commit.msg.format">162. Commit Message format</a></li> </ul> </li> <li><a href="#_appendix">Appendix</a> @@ -287,7 +288,7 @@ <li><a href="#hbck.in.depth">Appendix C: hbck In Depth</a></li> <li><a href="#appendix_acl_matrix">Appendix D: Access Control Matrix</a></li> <li><a href="#compression">Appendix E: Compression and Data Block Encoding In HBase</a></li> -<li><a href="#data.block.encoding.enable">162. Enable Data Block Encoding</a></li> +<li><a href="#data.block.encoding.enable">163. Enable Data Block Encoding</a></li> <li><a href="#sql">Appendix F: SQL over HBase</a></li> <li><a href="#ycsb">Appendix G: YCSB</a></li> <li><a href="#_hfile_format_2">Appendix H: HFile format</a></li> @@ -296,8 +297,8 @@ <li><a href="#asf">Appendix K: HBase and the Apache Software Foundation</a></li> <li><a href="#orca">Appendix L: Apache HBase Orca</a></li> <li><a href="#tracing">Appendix M: Enabling Dapper-like Tracing in HBase</a></li> -<li><a href="#tracing.client.modifications">163. Client Modifications</a></li> -<li><a href="#tracing.client.shell">164. Tracing from HBase Shell</a></li> +<li><a href="#tracing.client.modifications">164. Client Modifications</a></li> +<li><a href="#tracing.client.shell">165. Tracing from HBase Shell</a></li> <li><a href="#hbase.rpc">Appendix N: 0.95 RPC Specification</a></li> </ul> </li> @@ -17037,13 +17038,13 @@ hcd.setMobThreshold(<span class="integer">102400L</span>); <div class="sect2"> <h3 id="_testing_mob"><a class="anchor" href="#_testing_mob"></a>74.2. Testing MOB</h3> <div class="paragraph"> -<p>The utility <code>org.apache.hadoop.hbase.IntegrationTestIngestMOB</code> is provided to assist with testing +<p>The utility <code>org.apache.hadoop.hbase.IntegrationTestIngestWithMOB</code> is provided to assist with testing the MOB feature. The utility is run as follows:</p> </div> <div class="listingblock"> <div class="content"> -<pre class="CodeRay highlight"><code data-lang="bash">$ sudo -u hbase hbase org.apache.hadoop.hbase.IntegrationTestIngestMOB \ - -threshold 102400 \ +<pre class="CodeRay highlight"><code data-lang="bash">$ sudo -u hbase hbase org.apache.hadoop.hbase.IntegrationTestIngestWithMOB \ + -threshold 1024 \ -minMobDataSize 512 \ -maxMobDataSize 5120</code></pre> </div> @@ -17121,100 +17122,19 @@ suit your environment, and restart or rolling restart the RegionServer.</p> <div class="paragraph"> <p>To manually compact MOB files, rather than waiting for the <a href="#mob.cache.configure">configuration</a> to trigger compaction, use the -<code>compact_mob</code> or <code>major_compact_mob</code> HBase shell commands. These commands -require the first argument to be the table name, and take an optional column -family as the second argument. If the column family is omitted, all MOB-enabled -column families are compacted.</p> +<code>compact</code> or <code>major_compact</code> HBase shell commands. These commands +require the first argument to be the table name, and take a column +family as the second argument. and take a compaction type as the third argument.</p> </div> <div class="listingblock"> <div class="content"> -<pre>hbase> compact_mob 't1', 'c1' -hbase> compact_mob 't1' -hbase> major_compact_mob 't1', 'c1' -hbase> major_compact_mob 't1'</pre> +<pre>hbase> compact 't1', 'c1â, âMOBâ +hbase> major_compact_mob 't1', 'c1â, âMOBâ</pre> </div> </div> <div class="paragraph"> -<p>These commands are also available via <code>Admin.compactMob</code> and -<code>Admin.majorCompactMob</code> methods.</p> -</div> -</div> -<div class="sect3"> -<h4 id="_mob_sweeper"><a class="anchor" href="#_mob_sweeper"></a>74.4.2. MOB Sweeper</h4> -<div class="paragraph"> -<p>HBase MOB a MapReduce job called the Sweeper tool for -optimization. The Sweeper tool coalesces small MOB files or MOB files with many -deletions or updates. The Sweeper tool is not required if you use native MOB compaction, which -does not rely on MapReduce.</p> -</div> -<div class="paragraph"> -<p>To configure the Sweeper tool, set the following options:</p> -</div> -<div class="listingblock"> -<div class="content"> -<pre class="CodeRay highlight"><code data-lang="xml"><span class="tag"><property></span> - <span class="tag"><name></span>hbase.mob.sweep.tool.compaction.ratio<span class="tag"></name></span> - <span class="tag"><value></span>0.5f<span class="tag"></value></span> - <span class="tag"><description></span> - If there are too many cells deleted in a mob file, it's regarded - as an invalid file and needs to be merged. - If existingCellsSize/mobFileSize is less than ratio, it's regarded - as an invalid file. The default value is 0.5f. - <span class="tag"></description></span> -<span class="tag"></property></span> -<span class="tag"><property></span> - <span class="tag"><name></span>hbase.mob.sweep.tool.compaction.mergeable.size<span class="tag"></name></span> - <span class="tag"><value></span>134217728<span class="tag"></value></span> - <span class="tag"><description></span> - If the size of a mob file is less than this value, it's regarded as a small - file and needs to be merged. The default value is 128MB. - <span class="tag"></description></span> -<span class="tag"></property></span> -<span class="tag"><property></span> - <span class="tag"><name></span>hbase.mob.sweep.tool.compaction.memstore.flush.size<span class="tag"></name></span> - <span class="tag"><value></span>134217728<span class="tag"></value></span> - <span class="tag"><description></span> - The flush size for the memstore used by sweep job. Each sweep reducer owns such a memstore. - The default value is 128MB. - <span class="tag"></description></span> -<span class="tag"></property></span> -<span class="tag"><property></span> - <span class="tag"><name></span>hbase.master.mob.ttl.cleaner.period<span class="tag"></name></span> - <span class="tag"><value></span>86400<span class="tag"></value></span> - <span class="tag"><description></span> - The period that ExpiredMobFileCleanerChore runs. The unit is second. - The default value is one day. - <span class="tag"></description></span> -<span class="tag"></property></span></code></pre> -</div> -</div> -<div class="paragraph"> -<p>Next, add the HBase install directory, <em>`$HBASE_HOME`/*</em>, and HBase library directory to -<em>yarn-site.xml</em> Adjust this example to suit your environment.</p> -</div> -<div class="listingblock"> -<div class="content"> -<pre class="CodeRay highlight"><code data-lang="xml"><span class="tag"><property></span> - <span class="tag"><description></span>Classpath for typical applications.<span class="tag"></description></span> - <span class="tag"><name></span>yarn.application.classpath<span class="tag"></name></span> - <span class="tag"><value></span> - $HADOOP_CONF_DIR, - $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*, - $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*, - $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*, - $HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*, - $HBASE_HOME/*, $HBASE_HOME/lib/* - <span class="tag"></value></span> -<span class="tag"></property></span></code></pre> -</div> -</div> -<div class="paragraph"> -<p>Finally, run the <code>sweeper</code> tool for each column which is configured for MOB.</p> -</div> -<div class="listingblock"> -<div class="content"> -<pre class="CodeRay highlight"><code data-lang="bash">$ org.apache.hadoop.hbase.mob.compactions.Sweeper _tableName_ _familyName_</code></pre> -</div> +<p>These commands are also available via <code>Admin.compact</code> and +<code>Admin.majorCompact</code> methods.</p> </div> </div> </div> @@ -27277,6 +27197,196 @@ hbase shell> drop 'tableName'</pre> </div> </div> </div> +<div class="sect1"> +<h2 id="rsgroup"><a class="anchor" href="#rsgroup"></a>142. RegionServer Grouping</h2> +<div class="sectionbody"> +<div class="paragraph"> +<p>RegionServer Grouping (A.K.A <code>rsgroup</code>) is an advanced feature for +partitioning regionservers into distinctive groups for strict isolation. It +should only be used by users who are sophisticated enough to understand the +full implications and have a sufficient background in managing HBase clusters. +It was developed by Yahoo! and they run it at scale on their large grid cluster. +See <a href="http://www.slideshare.net/HBaseCon/keynote-apache-hbase-at-yahoo-scale";>HBase at Yahoo! Scale</a>.</p> +</div> +<div class="paragraph"> +<p>RSGroups can be defined and managed with shell commands or corresponding Java +APIs. A server can be added to a group with hostname and port pair and tables +can be moved to this group so that only regionservers in the same rsgroup can +host the regions of the table. RegionServers and tables can only belong to one +rsgroup at a time. By default, all tables and regionservers belong to the +<code>default</code> rsgroup. System tables can also be put into a rsgroup using the regular +APIs. A custom balancer implementation tracks assignments per rsgroup and makes +sure to move regions to the relevant regionservers in that rsgroup. The rsgroup +information is stored in a regular HBase table, and a zookeeper-based read-only +cache is used at cluster bootstrap time.</p> +</div> +<div class="paragraph"> +<p>To enable, add the following to your hbase-site.xml and restart your Master:</p> +</div> +<div class="listingblock"> +<div class="content"> +<pre class="CodeRay highlight"><code data-lang="xml"> <span class="tag"><property></span> + <span class="tag"><name></span>hbase.coprocessor.master.classes<span class="tag"></name></span> + <span class="tag"><value></span>org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint<span class="tag"></value></span> + <span class="tag"></property></span> + <span class="tag"><property></span> + <span class="tag"><name></span>hbase.master.loadbalancer.class<span class="tag"></name></span> + <span class="tag"><value></span>org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer<span class="tag"></value></span> + <span class="tag"></property></span></code></pre> +</div> +</div> +<div class="paragraph"> +<p>Then use the shell <em>rsgroup</em> commands to create and manipulate RegionServer +groups: e.g. to add a rsgroup and then add a server to it. To see the list of +rsgroup commands available in the hbase shell type:</p> +</div> +<div class="listingblock"> +<div class="content"> +<pre class="CodeRay highlight"><code data-lang="bash"> hbase(main):008:0> help ârsgroupâ + Took 0.5610 seconds</code></pre> +</div> +</div> +<div class="paragraph"> +<p>High level, you create a rsgroup that is other than the <code>default</code> group using +<em>add_rsgroup</em> command. You then add servers and tables to this group with the +<em>move_servers_rsgroup</em> and <em>move_tables_rsgroup</em> commands. If necessary, run +a balance for the group if tables are slow to migrate to the groups dedicated +server with the <em>balance_rsgroup</em> command (Usually this is not needed). To +monitor effect of the commands, see the <code>Tables</code> tab toward the end of the +Master UI home page. If you click on a table, you can see what servers it is +deployed across. You should see here a reflection of the grouping done with +your shell commands. View the master log if issues.</p> +</div> +<div class="paragraph"> +<p>Here is example using a few of the rsgroup commands. To add a group, do as follows:</p> +</div> +<div class="listingblock"> +<div class="content"> +<pre class="CodeRay highlight"><code data-lang="bash"> hbase(main):008:0> add_rsgroup 'my_group' + Took 0.5610 seconds</code></pre> +</div> +</div> +<div class="admonitionblock note"> +<table> +<tr> +<td class="icon"> +<i class="fa icon-note" title="Note"></i> +</td> +<td class="content"> +<div class="title">RegionServer Groups must be Enabled</div> +<div class="paragraph"> +<p>If you have not enabled the rsgroup Coprocessor Endpoint in the master and +you run the any of the rsgroup shell commands, you will see an error message +like the below:</p> +</div> +<div class="listingblock"> +<div class="content"> +<pre class="CodeRay highlight"><code data-lang="java">ERROR: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No registered master coprocessor service found <span class="keyword">for</span> name RSGroupAdminService + at org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:<span class="integer">604</span>) + at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos<span class="error">$</span>MasterService<span class="error">$</span><span class="integer">2</span>.callBlockingMethod(MasterProtos.java) + at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:<span class="integer">1140</span>) + at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:<span class="integer">133</span>) + at org.apache.hadoop.hbase.ipc.RpcExecutor<span class="error">$</span><span class="predefined-type">Handler</span>.run(RpcExecutor.java:<span class="integer">277</span>) + at org.apache.hadoop.hbase.ipc.RpcExecutor<span class="error">$</span><span class="predefined-type">Handler</span>.run(RpcExecutor.java:<span class="integer">257</span>)</code></pre> +</div> +</div> +</td> +</tr> +</table> +</div> +<div class="paragraph"> +<p>Add a server (specified by hostname + port) to the just-made group using the +<em>move_servers_rsgroup</em> command as follows:</p> +</div> +<div class="listingblock"> +<div class="content"> +<pre class="CodeRay highlight"><code data-lang="bash"> hbase(main):010:0> move_servers_rsgroup 'my_group',['k.att.net:51129']</code></pre> +</div> +</div> +<div class="admonitionblock note"> +<table> +<tr> +<td class="icon"> +<i class="fa icon-note" title="Note"></i> +</td> +<td class="content"> +<div class="title">Hostname and Port vs ServerName</div> +<div class="paragraph"> +<p>The rsgroup feature refers to servers in a cluster with hostname and port only. +It does not make use of the HBase ServerName type identifying RegionServers; +i.e. hostname + port + starttime to distinguish RegionServer instances. The +rsgroup feature keeps working across RegionServer restarts so the starttime of +ServerName — and hence the ServerName type — is not appropriate. +Administration</p> +</div> +</td> +</tr> +</table> +</div> +<div class="paragraph"> +<p>Servers come and go over the lifetime of a Cluster. Currently, you must +manually align the servers referenced in rsgroups with the actual state of +nodes in the running cluster. What we mean by this is that if you decommission +a server, then you must update rsgroups as part of your server decommission +process removing references.</p> +</div> +<div class="paragraph"> +<p>But, there is no _remove_offline_servers_rsgroup_command you say!</p> +</div> +<div class="paragraph"> +<p>The way to remove a server is to move it to the <code>default</code> group. The <code>default</code> +group is special. All rsgroups, but the <code>default</code> rsgroup, are static in that +edits via the shell commands are persisted to the system <code>hbase:rsgroup</code> table. +If they reference a decommissioned server, then they need to be updated to undo +the reference.</p> +</div> +<div class="paragraph"> +<p>The <code>default</code> group is not like other rsgroups in that it is dynamic. Its server +list mirrors the current state of the cluster; i.e. if you shutdown a server that +was part of the <code>default</code> rsgroup, and then do a <em>get_rsgroup</em> <code>default</code> to list +its content in the shell, the server will no longer be listed. For non-<code>default</code> +groups, though a mode may be offline, it will persist in the non-<code>default</code> groupâs +list of servers. But if you move the offline server from the non-default rsgroup +to default, it will not show in the <code>default</code> list. It will just be dropped.</p> +</div> +<div class="sect2"> +<h3 id="_best_practice"><a class="anchor" href="#_best_practice"></a>142.1. Best Practice</h3> +<div class="paragraph"> +<p>The authors of the rsgroup feature, the Yahoo! HBase Engineering team, have been +running it on their grid for a good while now and have come up with a few best +practices informed by their experience.</p> +</div> +<div class="sect3"> +<h4 id="_isolate_system_tables"><a class="anchor" href="#_isolate_system_tables"></a>142.1.1. Isolate System Tables</h4> +<div class="paragraph"> +<p>Either have a system rsgroup where all the system tables are or just leave the +system tables in <code>default</code> rsgroup and have all user-space tables are in +non-<code>default</code> rsgroups.</p> +</div> +</div> +<div class="sect3"> +<h4 id="_dead_nodes"><a class="anchor" href="#_dead_nodes"></a>142.1.2. Dead Nodes</h4> +<div class="paragraph"> +<p>Yahoo! Have found it useful at their scale to keep a special rsgroup of dead or +questionable nodes; this is one means of keeping them out of the running until repair.</p> +</div> +<div class="paragraph"> +<p>Be careful replacing dead nodes in an rsgroup. Ensure there are enough live nodes +before you start moving out the dead. Move in good live nodes first if you have to.</p> +</div> +</div> +</div> +<div class="sect2"> +<h3 id="_troubleshooting"><a class="anchor" href="#_troubleshooting"></a>142.2. Troubleshooting</h3> +<div class="paragraph"> +<p>Viewing the Master log will give you insight on rsgroup operation.</p> +</div> +<div class="paragraph"> +<p>If it appears stuck, restart the Master process.</p> +</div> +</div> +</div> +</div> <h1 id="developer" class="sect0"><a class="anchor" href="#developer"></a>Building and Developing Apache HBase</h1> <div class="openblock partintro"> <div class="content"> @@ -27285,7 +27395,7 @@ Being familiar with these guidelines will help the HBase committers to use your </div> </div> <div class="sect1"> -<h2 id="getting.involved"><a class="anchor" href="#getting.involved"></a>142. Getting Involved</h2> +<h2 id="getting.involved"><a class="anchor" href="#getting.involved"></a>143. Getting Involved</h2> <div class="sectionbody"> <div class="paragraph"> <p>Apache HBase gets better only when people contribute! If you are looking to contribute to Apache HBase, look for <a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20labels%20in%20(beginner)">issues in JIRA tagged with the label 'beginner'</a>. @@ -27300,7 +27410,7 @@ See <a href="http://search-hadoop.com/m/DHED43re96";>What label <p>As Apache HBase is an Apache Software Foundation project, see <a href="#asf">asf</a> for more information about how the ASF functions.</p> </div> <div class="sect2"> -<h3 id="mailing.list"><a class="anchor" href="#mailing.list"></a>142.1. Mailing Lists</h3> +<h3 id="mailing.list"><a class="anchor" href="#mailing.list"></a>143.1. Mailing Lists</h3> <div class="paragraph"> <p>Sign up for the dev-list and the user-list. See the <a href="http://hbase.apache.org/mail-lists.html";>mailing lists</a> page. @@ -27308,14 +27418,14 @@ Posing questions - and helping to answer other people’s questions - is enc </div> </div> <div class="sect2"> -<h3 id="slack"><a class="anchor" href="#slack"></a>142.2. Slack</h3> +<h3 id="slack"><a class="anchor" href="#slack"></a>143.2. Slack</h3> <div class="paragraph"> <p>The Apache HBase project has its own link: <a href="http://apache-hbase.slack.com";>Slack Channel</a> for real-time questions and discussion. Mail <a href="mailto:d...@hbase.apache.org";>d...@hbase.apache.org</a> to request an invite.</p> </div> </div> <div class="sect2"> -<h3 id="irc"><a class="anchor" href="#irc"></a>142.3. Internet Relay Chat (IRC)</h3> +<h3 id="irc"><a class="anchor" href="#irc"></a>143.3. Internet Relay Chat (IRC)</h3> <div class="paragraph"> <p>(NOTE: Our IRC channel seems to have been deprecated in favor of the above Slack channel)</p> </div> @@ -27325,7 +27435,7 @@ FreeNode offers a web-based client, but most people prefer a native client, and </div> </div> <div class="sect2"> -<h3 id="_jira"><a class="anchor" href="#_jira"></a>142.4. Jira</h3> +<h3 id="_jira"><a class="anchor" href="#_jira"></a>143.4. Jira</h3> <div class="paragraph"> <p>Check for existing issues in <a href="https://issues.apache.org/jira/browse/HBASE";>Jira</a>. If it’s either a new feature request, enhancement, or a bug, file a ticket.</p> @@ -27371,7 +27481,7 @@ code snippet </div> </div> <div class="sect1"> -<h2 id="repos"><a class="anchor" href="#repos"></a>143. Apache HBase Repositories</h2> +<h2 id="repos"><a class="anchor" href="#repos"></a>144. Apache HBase Repositories</h2> <div class="sectionbody"> <div class="paragraph"> <p>There are two different repositories for Apache HBase: Subversion (SVN) and Git. @@ -27385,12 +27495,12 @@ See <a href="http://hbase.apache.org/source-repository.html";>Source Code </div> </div> <div class="sect1"> -<h2 id="_ides"><a class="anchor" href="#_ides"></a>144. IDEs</h2> +<h2 id="_ides"><a class="anchor" href="#_ides"></a>145. IDEs</h2> <div class="sectionbody"> <div class="sect2"> -<h3 id="eclipse"><a class="anchor" href="#eclipse"></a>144.1. Eclipse</h3> +<h3 id="eclipse"><a class="anchor" href="#eclipse"></a>145.1. Eclipse</h3> <div class="sect3"> -<h4 id="eclipse.code.formatting"><a class="anchor" href="#eclipse.code.formatting"></a>144.1.1. Code Formatting</h4> +<h4 id="eclipse.code.formatting"><a class="anchor" href="#eclipse.code.formatting"></a>145.1.1. Code Formatting</h4> <div class="paragraph"> <p>Under the <em>dev-support/</em> folder, you will find <em>hbase_eclipse_formatter.xml</em>. We encourage you to have this formatter in place in eclipse when editing HBase code.</p> @@ -27406,13 +27516,13 @@ edited lines' is selected.</p> </div> </div> <div class="sect3"> -<h4 id="eclipse.git.plugin"><a class="anchor" href="#eclipse.git.plugin"></a>144.1.2. Eclipse Git Plugin</h4> +<h4 id="eclipse.git.plugin"><a class="anchor" href="#eclipse.git.plugin"></a>145.1.2. Eclipse Git Plugin</h4> <div class="paragraph"> <p>If you cloned the project via git, download and install the Git plugin (EGit). Attach to your local git repo (via the <span class="label">Git Repositories</span> window) and you’ll be able to see file revision history, generate patches, etc.</p> </div> </div> <div class="sect3"> -<h4 id="eclipse.maven.setup"><a class="anchor" href="#eclipse.maven.setup"></a>144.1.3. HBase Project Setup in Eclipse using <code>m2eclipse</code></h4> +<h4 id="eclipse.maven.setup"><a class="anchor" href="#eclipse.maven.setup"></a>145.1.3. HBase Project Setup in Eclipse using <code>m2eclipse</code></h4> <div class="paragraph"> <p>The easiest way is to use the m2eclipse plugin for Eclipse. Eclipse Indigo or newer includes m2eclipse, or you can download it from <a href="http://www.eclipse.org/m2e/"; class="bare">http://www.eclipse.org/m2e/</a>. It provides Maven integration for Eclipse, and even lets you use the direct Maven commands from within Eclipse to compile and test your project.</p> @@ -27449,7 +27559,7 @@ doesn't exist</pre> </div> </div> <div class="sect3"> -<h4 id="eclipse.commandline"><a class="anchor" href="#eclipse.commandline"></a>144.1.4. HBase Project Setup in Eclipse Using the Command Line</h4> +<h4 id="eclipse.commandline"><a class="anchor" href="#eclipse.commandline"></a>145.1.4. HBase Project Setup in Eclipse Using the Command Line</h4> <div class="paragraph"> <p>Instead of using <code>m2eclipse</code>, you can generate the Eclipse files from the command line.</p> </div> @@ -27479,7 +27589,7 @@ You only need to do this once.</p> </div> </div> <div class="sect3"> -<h4 id="eclipse.maven.class"><a class="anchor" href="#eclipse.maven.class"></a>144.1.5. Maven Classpath Variable</h4> +<h4 id="eclipse.maven.class"><a class="anchor" href="#eclipse.maven.class"></a>145.1.5. Maven Classpath Variable</h4> <div class="paragraph"> <p>The <code>$M2_REPO</code> classpath variable needs to be set up for the project. This needs to be set to your local Maven repository, which is usually <em>~/.m2/repository</em></p> @@ -27498,7 +27608,7 @@ Unbound classpath variable: 'M2_REPO/com/google/protobuf/protobuf-java/2.3.0/pro </div> </div> <div class="sect3"> -<h4 id="eclipse.issues"><a class="anchor" href="#eclipse.issues"></a>144.1.6. Eclipse Known Issues</h4> +<h4 id="eclipse.issues"><a class="anchor" href="#eclipse.issues"></a>145.1.6. Eclipse Known Issues</h4> <div class="paragraph"> <p>Eclipse will currently complain about <em>Bytes.java</em>. It is not possible to turn these errors off.</p> @@ -27513,14 +27623,14 @@ Access restriction: The method getLong(Object, long) from the type Unsafe is not </div> </div> <div class="sect3"> -<h4 id="eclipse.more"><a class="anchor" href="#eclipse.more"></a>144.1.7. Eclipse - More Information</h4> +<h4 id="eclipse.more"><a class="anchor" href="#eclipse.more"></a>145.1.7. Eclipse - More Information</h4> <div class="paragraph"> <p>For additional information on setting up Eclipse for HBase development on Windows, see <a href="http://michaelmorello.blogspot.com/2011/09/hbase-subversion-eclipse-windows.html";>Michael Morello’s blog</a> on the topic.</p> </div> </div> </div> <div class="sect2"> -<h3 id="_intellij_idea"><a class="anchor" href="#_intellij_idea"></a>144.2. IntelliJ IDEA</h3> +<h3 id="_intellij_idea"><a class="anchor" href="#_intellij_idea"></a>145.2. IntelliJ IDEA</h3> <div class="paragraph"> <p>You can set up IntelliJ IDEA for similar functionality as Eclipse. Follow these steps.</p> @@ -27546,7 +27656,7 @@ required</span> is selected, and click <b class="button">Next</b>.</p> </div> </div> <div class="sect2"> -<h3 id="_other_ides"><a class="anchor" href="#_other_ides"></a>144.3. Other IDEs</h3> +<h3 id="_other_ides"><a class="anchor" href="#_other_ides"></a>145.3. Other IDEs</h3> <div class="paragraph"> <p>It would be useful to mirror the <a href="#eclipse">eclipse</a> set-up instructions for other IDEs. If you would like to assist, please have a look at <a href="https://issues.apache.org/jira/browse/HBASE-11704";>HBASE-11704</a>.</p> @@ -27555,10 +27665,10 @@ If you would like to assist, please have a look at <a href="https://issues.apach </div> </div> <div class="sect1"> -<h2 id="build"><a class="anchor" href="#build"></a>145. Building Apache HBase</h2> +<h2 id="build"><a class="anchor" href="#build"></a>146. Building Apache HBase</h2> <div class="sectionbody"> <div class="sect2"> -<h3 id="build.basic"><a class="anchor" href="#build.basic"></a>145.1. Basic Compile</h3> +<h3 id="build.basic"><a class="anchor" href="#build.basic"></a>146.1. Basic Compile</h3> <div class="paragraph"> <p>HBase is compiled using Maven. You must use at least Maven 3.0.4. @@ -27581,7 +27691,7 @@ See <a href="#java">java</a> for more complete information about supported JDK v </table> </div> <div class="sect3"> -<h4 id="maven.build.commands"><a class="anchor" href="#maven.build.commands"></a>145.1.1. Maven Build Commands</h4> +<h4 id="maven.build.commands"><a class="anchor" href="#maven.build.commands"></a>146.1.1. Maven Build Commands</h4> <div class="paragraph"> <p>All commands are executed from the local HBase project directory.</p> </div> @@ -27647,13 +27757,13 @@ To create the full installable HBase package takes a little bit more work, so re </div> </div> <div class="sect3"> -<h4 id="maven.build.commands.unitall"><a class="anchor" href="#maven.build.commands.unitall"></a>145.1.2. Running all or individual Unit Tests</h4> +<h4 id="maven.build.commands.unitall"><a class="anchor" href="#maven.build.commands.unitall"></a>146.1.2. Running all or individual Unit Tests</h4> <div class="paragraph"> <p>See the <a href="#hbase.unittests.cmds">hbase.unittests.cmds</a> section in <a href="#hbase.unittests">hbase.unittests</a></p> </div> </div> <div class="sect3"> -<h4 id="maven.build.hadoop"><a class="anchor" href="#maven.build.hadoop"></a>145.1.3. Building against various hadoop versions.</h4> +<h4 id="maven.build.hadoop"><a class="anchor" href="#maven.build.hadoop"></a>146.1.3. Building against various hadoop versions.</h4> <div class="paragraph"> <p>As of 0.96, Apache HBase supports building against Apache Hadoop versions: 1.0.3, 2.0.0-alpha and 3.0.0-SNAPSHOT. By default, in 0.96 and earlier, we will build with Hadoop-1.0.x. @@ -27701,7 +27811,7 @@ If you are running, for example HBase-0.94 and wanted to build against Hadoop 0. </div> </div> <div class="sect3"> -<h4 id="build.protobuf"><a class="anchor" href="#build.protobuf"></a>145.1.4. Build Protobuf</h4> +<h4 id="build.protobuf"><a class="anchor" href="#build.protobuf"></a>146.1.4. Build Protobuf</h4> <div class="paragraph"> <p>You may need to change the protobuf definitions that reside in the <em>hbase-protocol</em> module or other modules.</p> </div> @@ -27733,7 +27843,7 @@ You can use maven profile <code>compile-protobuf</code> to do this.</p> </div> </div> <div class="sect3"> -<h4 id="build.thrift"><a class="anchor" href="#build.thrift"></a>145.1.5. Build Thrift</h4> +<h4 id="build.thrift"><a class="anchor" href="#build.thrift"></a>146.1.5. Build Thrift</h4> <div class="paragraph"> <p>You may need to change the thrift definitions that reside in the <em>hbase-thrift</em> module or other modules.</p> </div> @@ -27757,7 +27867,7 @@ You can use maven profile <code>compile-thrift</code> to do this.</p> </div> </div> <div class="sect3"> -<h4 id="_build_a_tarball"><a class="anchor" href="#_build_a_tarball"></a>145.1.6. Build a Tarball</h4> +<h4 id="_build_a_tarball"><a class="anchor" href="#_build_a_tarball"></a>146.1.6. Build a Tarball</h4> <div class="paragraph"> <p>You can build a tarball without going through the release process described in <a href="#releasing">releasing</a>, by running the following command:</p> </div> @@ -27784,7 +27894,7 @@ You can use maven profile <code>compile-thrift</code> to do this.</p> </div> </div> <div class="sect3"> -<h4 id="build.gotchas"><a class="anchor" href="#build.gotchas"></a>145.1.7. Build Gotchas</h4> +<h4 id="build.gotchas"><a class="anchor" href="#build.gotchas"></a>146.1.7. Build Gotchas</h4> <div class="paragraph"> <p>If you see <code>Unable to find resource 'VM_global_library.vm'</code>, ignore it. It’s not an error. @@ -27796,7 +27906,7 @@ It is <a href="http://jira.codehaus.org/browse/MSITE-286";>officially </div> </div> <div class="sect1"> -<h2 id="releasing"><a class="anchor" href="#releasing"></a>146. Releasing Apache HBase</h2> +<h2 id="releasing"><a class="anchor" href="#releasing"></a>147. Releasing Apache HBase</h2> <div class="sectionbody"> <div class="admonitionblock note"> <table> @@ -27815,7 +27925,7 @@ See <a href="#java">java</a> for Java requirements per HBase release.</p> </table> </div> <div class="sect2"> -<h3 id="_building_against_hbase_0_96_0_98"><a class="anchor" href="#_building_against_hbase_0_96_0_98"></a>146.1. Building against HBase 0.96-0.98</h3> +<h3 id="_building_against_hbase_0_96_0_98"><a class="anchor" href="#_building_against_hbase_0_96_0_98"></a>147.1. Building against HBase 0.96-0.98</h3> <div class="paragraph"> <p>HBase 0.96.x will run on Hadoop 1.x or Hadoop 2.x. HBase 0.98 still runs on both, but HBase 0.98 deprecates use of Hadoop 1. @@ -27888,7 +27998,7 @@ For the build to sign them for you, you a properly configured <em>settings.xml</ </div> </div> <div class="sect2"> -<h3 id="maven.release"><a class="anchor" href="#maven.release"></a>146.2. Making a Release Candidate</h3> +<h3 id="maven.release"><a class="anchor" href="#maven.release"></a>147.2. Making a Release Candidate</h3> <div class="admonitionblock note"> <table> <tr> @@ -28228,7 +28338,7 @@ $ svn commit ...</code></pre> </div> </div> <div class="sect2"> -<h3 id="maven.snapshot"><a class="anchor" href="#maven.snapshot"></a>146.3. Publishing a SNAPSHOT to maven</h3> +<h3 id="maven.snapshot"><a class="anchor" href="#maven.snapshot"></a>147.3. Publishing a SNAPSHOT to maven</h3> <div class="paragraph"> <p>Make sure your <em>settings.xml</em> is set up properly (see <a href="#maven.settings.xml">Example <em>~/.m2/settings.xml</em> File</a>). Make sure the hbase version includes <code>-SNAPSHOT</code> as a suffix. @@ -28249,7 +28359,7 @@ It will put a snapshot up into the apache snapshot repository for you.</p> </div> </div> <div class="sect1"> -<h2 id="hbase.rc.voting"><a class="anchor" href="#hbase.rc.voting"></a>147. Voting on Release Candidates</h2> +<h2 id="hbase.rc.voting"><a class="anchor" href="#hbase.rc.voting"></a>148. Voting on Release Candidates</h2> <div class="sectionbody"> <div class="paragraph"> <p>Everyone is encouraged to try and vote on HBase release candidates. @@ -28267,7 +28377,7 @@ for how we arrived at this process.</p> </div> </div> <div class="sect1"> -<h2 id="documentation"><a class="anchor" href="#documentation"></a>148. Generating the HBase Reference Guide</h2> +<h2 id="documentation"><a class="anchor" href="#documentation"></a>149. Generating the HBase Reference Guide</h2> <div class="sectionbody"> <div class="paragraph"> <p>The manual is marked up using Asciidoc. @@ -28278,16 +28388,16 @@ See <a href="#appendix_contributing_to_documentation">appendix contributing to d </div> </div> <div class="sect1"> -<h2 id="hbase.org"><a class="anchor" href="#hbase.org"></a>149. Updating <a href="http://hbase.apache.org";>hbase.apache.org</a></h2> +<h2 id="hbase.org"><a class="anchor" href="#hbase.org"></a>150. Updating <a href="http://hbase.apache.org";>hbase.apache.org</a></h2> <div class="sectionbody"> <div class="sect2"> -<h3 id="hbase.org.site.contributing"><a class="anchor" href="#hbase.org.site.contributing"></a>149.1. Contributing to hbase.apache.org</h3> +<h3 id="hbase.org.site.contributing"><a class="anchor" href="#hbase.org.site.contributing"></a>150.1. Contributing to hbase.apache.org</h3> <div class="paragraph"> <p>See <a href="#appendix_contributing_to_documentation">appendix contributing to documentation</a> for more information on contributing to the documentation or website.</p> </div> </div> <div class="sect2"> -<h3 id="hbase.org.site.publishing"><a class="anchor" href="#hbase.org.site.publishing"></a>149.2. Publishing <a href="http://hbase.apache.org";>hbase.apache.org</a></h3> +<h3 id="hbase.org.site.publishing"><a class="anchor" href="#hbase.org.site.publishing"></a>150.2. Publishing <a href="http://hbase.apache.org";>hbase.apache.org</a></h3> <div class="paragraph"> <p>See <a href="#website_publish">Publishing the HBase Website and Documentation</a> for instructions on publishing the website and documentation.</p> </div> @@ -28295,7 +28405,7 @@ See <a href="#appendix_contributing_to_documentation">appendix contributing to d </div> </div> <div class="sect1"> -<h2 id="hbase.tests"><a class="anchor" href="#hbase.tests"></a>150. Tests</h2> +<h2 id="hbase.tests"><a class="anchor" href="#hbase.tests"></a>151. Tests</h2> <div class="sectionbody"> <div class="paragraph"> <p>Developers, at a minimum, should familiarize themselves with the unit test detail; unit tests in HBase have a character not usually seen in other projects.</p> @@ -28305,7 +28415,7 @@ See <a href="#appendix_contributing_to_documentation">appendix contributing to d For developing unit tests for your HBase applications, see <a href="#unit.tests">unit.tests</a>.</p> </div> <div class="sect2"> -<h3 id="hbase.moduletests"><a class="anchor" href="#hbase.moduletests"></a>150.1. Apache HBase Modules</h3> +<h3 id="hbase.moduletests"><a class="anchor" href="#hbase.moduletests"></a>151.1. Apache HBase Modules</h3> <div class="paragraph"> <p>As of 0.96, Apache HBase is split into multiple modules. This creates "interesting" rules for how and where tests are written. @@ -28314,7 +28424,7 @@ These tests can spin up a minicluster and will need to be categorized. For any other module, for example <code>hbase-common</code>, the tests must be strict unit tests and just test the class under test - no use of the HBaseTestingUtility or minicluster is allowed (or even possible given the dependency tree).</p> </div> <div class="sect3"> -<h4 id="hbase.moduletest.shell"><a class="anchor" href="#hbase.moduletest.shell"></a>150.1.1. Testing the HBase Shell</h4> +<h4 id="hbase.moduletest.shell"><a class="anchor" href="#hbase.moduletest.shell"></a>151.1.1. Testing the HBase Shell</h4> <div class="paragraph"> <p>The HBase shell and its tests are predominantly written in jruby. In order to make these tests run as a part of the standard build, there is a single JUnit test, <code>TestShell</code>, that takes care of loading the jruby implemented tests and running them. @@ -28355,7 +28465,7 @@ You can run all of the HBase admin related tests, including both the normal admi </div> </div> <div class="sect3"> -<h4 id="hbase.moduletest.run"><a class="anchor" href="#hbase.moduletest.run"></a>150.1.2. Running Tests in other Modules</h4> +<h4 id="hbase.moduletest.run"><a class="anchor" href="#hbase.moduletest.run"></a>151.1.2. Running Tests in other Modules</h4> <div class="paragraph"> <p>If the module you are developing in has no other dependencies on other HBase modules, then you can cd into that module and just run:</p> </div> @@ -28391,7 +28501,7 @@ For example, to skip the tests in <code>hbase-server</code> and <code>hbase-comm </div> </div> <div class="sect2"> -<h3 id="hbase.unittests"><a class="anchor" href="#hbase.unittests"></a>150.2. Unit Tests</h3> +<h3 id="hbase.unittests"><a class="anchor" href="#hbase.unittests"></a>151.2. Unit Tests</h3> <div class="paragraph"> <p>Apache HBase test cases are subdivided into four categories: small, medium, large, and integration with corresponding JUnit <a href="http://www.junit.org/node/581";>categories</a>: <code>SmallTests</code>, <code>MediumTests</code>, <code>LargeTests</code>, <code>IntegrationTests</code>. @@ -28460,9 +28570,9 @@ See <a href="#integration.tests">integration.tests</a> for more info.</p> </div> </div> <div class="sect2"> -<h3 id="hbase.unittests.cmds"><a class="anchor" href="#hbase.unittests.cmds"></a>150.3. Running tests</h3> +<h3 id="hbase.unittests.cmds"><a class="anchor" href="#hbase.unittests.cmds"></a>151.3. Running tests</h3> <div class="sect3"> -<h4 id="hbase.unittests.cmds.test"><a class="anchor" href="#hbase.unittests.cmds.test"></a>150.3.1. Default: small and medium category tests</h4> +<h4 id="hbase.unittests.cmds.test"><a class="anchor" href="#hbase.unittests.cmds.test"></a>151.3.1. Default: small and medium category tests</h4> <div class="paragraph"> <p>Running <code>mvn test</code> will execute all small tests in a single JVM (no fork) and then medium tests in a separate JVM for each test instance. Medium tests are NOT executed if there is an error in a small test. @@ -28471,7 +28581,7 @@ There is one report for small tests, and one report for medium tests if they are </div> </div> <div class="sect3"> -<h4 id="hbase.unittests.cmds.test.runalltests"><a class="anchor" href="#hbase.unittests.cmds.test.runalltests"></a>150.3.2. Running all tests</h4> +<h4 id="hbase.unittests.cmds.test.runalltests"><a class="anchor" href="#hbase.unittests.cmds.test.runalltests"></a>151.3.2. Running all tests</h4> <div class="paragraph"> <p>Running <code>mvn test -P runAllTests</code> will execute small tests in a single JVM then medium and large tests in a separate JVM for each test. Medium and large tests are NOT executed if there is an error in a small test. @@ -28480,7 +28590,7 @@ There is one report for small tests, and one report for medium and large tests i </div> </div> <div class="sect3"> -<h4 id="hbase.unittests.cmds.test.localtests.mytest"><a class="anchor" href="#hbase.unittests.cmds.test.localtests.mytest"></a>150.3.3. Running a single test or all tests in a package</h4> +<h4 id="hbase.unittests.cmds.test.localtests.mytest"><a class="anchor" href="#hbase.unittests.cmds.test.localtests.mytest"></a>151.3.3. Running a single test or all tests in a package</h4> <div class="paragraph"> <p>To run an individual test, e.g. <code>MyTest</code>, rum <code>mvn test -Dtest=MyTest</code> You can also pass multiple, individual tests as a comma-delimited list:</p> </div> @@ -28506,7 +28616,7 @@ However, you need to make sure the sum of <code>Tests run:</code> in the <code>R </div> </div> <div class="sect3"> -<h4 id="hbase.unittests.cmds.test.profiles"><a class="anchor" href="#hbase.unittests.cmds.test.profiles"></a>150.3.4. Other test invocation permutations</h4> +<h4 id="hbase.unittests.cmds.test.profiles"><a class="anchor" href="#hbase.unittests.cmds.test.profiles"></a>151.3.4. Other test invocation permutations</h4> <div class="paragraph"> <p>Running <code>mvn test -P runSmallTests</code> will execute "small" tests only, using a single JVM.</p> </div> @@ -28521,7 +28631,7 @@ However, you need to make sure the sum of <code>Tests run:</code> in the <code>R </div> </div> <div class="sect3"> -<h4 id="hbase.unittests.test.faster"><a class="anchor" href="#hbase.unittests.test.faster"></a>150.3.5. Running tests faster</h4> +<h4 id="hbase.unittests.test.faster"><a class="anchor" href="#hbase.unittests.test.faster"></a>151.3.5. Running tests faster</h4> <div class="paragraph"> <p>By default, <code>$ mvn test -P runAllTests</code> runs 5 tests in parallel. It can be increased on a developer’s machine. @@ -28563,7 +28673,7 @@ sudo mount -t tmpfs -o size=2048M tmpfs /ram2G</pre> </div> </div> <div class="sect3"> -<h4 id="hbase.unittests.cmds.test.hbasetests"><a class="anchor" href="#hbase.unittests.cmds.test.hbasetests"></a>150.3.6. hbasetests.sh</h4> +<h4 id="hbase.unittests.cmds.test.hbasetests"><a class="anchor" href="#hbase.unittests.cmds.test.hbasetests"></a>151.3.6. hbasetests.sh</h4> <div class="paragraph"> <p>It’s also possible to use the script hbasetests.sh. This script runs the medium and large tests in parallel with two maven instances, and provides a single report. @@ -28578,7 +28688,7 @@ Running ./dev-support/hbasetests.sh replayFailed will rerun the failed tests a s </div> </div> <div class="sect3"> -<h4 id="hbase.unittests.resource.checker"><a class="anchor" href="#hbase.unittests.resource.checker"></a>150.3.7. Test Resource Checker</h4> +<h4 id="hbase.unittests.resource.checker"><a class="anchor" href="#hbase.unittests.resource.checker"></a>151.3.7. Test Resource Checker</h4> <div class="paragraph"> <p>A custom Maven SureFire plugin listener checks a number of resources before and after each HBase unit test runs and logs its findings at the end of the test output files which can be found in <em>target/surefire-reports</em> per Maven module (Tests write test reports named for the test class into this directory. Check the <em>*-out.txt</em> files). The resources counted are the number of threads, the number of file descriptors, etc. @@ -28599,9 +28709,9 @@ ConnectionCount=1 (was 1)</pre> </div> </div> <div class="sect2"> -<h3 id="hbase.tests.writing"><a class="anchor" href="#hbase.tests.writing"></a>150.4. Writing Tests</h3> +<h3 id="hbase.tests.writing"><a class="anchor" href="#hbase.tests.writing"></a>151.4. Writing Tests</h3> <div class="sect3"> -<h4 id="hbase.tests.rules"><a class="anchor" href="#hbase.tests.rules"></a>150.4.1. General rules</h4> +<h4 id="hbase.tests.rules"><a class="anchor" href="#hbase.tests.rules"></a>151.4.1. General rules</h4> <div class="ulist"> <ul> <li> @@ -28622,7 +28732,7 @@ This class offers helper functions to create a temp directory and do the cleanup </div> </div> <div class="sect3"> -<h4 id="hbase.tests.categories"><a class="anchor" href="#hbase.tests.categories"></a>150.4.2. Categories and execution time</h4> +<h4 id="hbase.tests.categories"><a class="anchor" href="#hbase.tests.categories"></a>151.4.2. Categories and execution time</h4> <div class="ulist"> <ul> <li> @@ -28639,7 +28749,7 @@ This should ensure a good parallelization for people using it, and ease the anal </div> </div> <div class="sect3"> -<h4 id="hbase.tests.sleeps"><a class="anchor" href="#hbase.tests.sleeps"></a>150.4.3. Sleeps in tests</h4> +<h4 id="hbase.tests.sleeps"><a class="anchor" href="#hbase.tests.sleeps"></a>151.4.3. Sleeps in tests</h4> <div class="paragraph"> <p>Whenever possible, tests should not use <span class="method">Thread.sleep</span>, but rather waiting for the real event they need. This is faster and clearer for the reader. @@ -28652,7 +28762,7 @@ Waiting for a socket operation should be done in a 200 ms sleep loop.</p> </div> </div> <div class="sect3"> -<h4 id="hbase.tests.cluster"><a class="anchor" href="#hbase.tests.cluster"></a>150.4.4. Tests using a cluster</h4> +<h4 id="hbase.tests.cluster"><a class="anchor" href="#hbase.tests.cluster"></a>151.4.4. Tests using a cluster</h4> <div class="paragraph"> <p>Tests using a HRegion do not have to start a cluster: A region can use the local file system. Start/stopping a cluster cost around 10 seconds. @@ -28664,7 +28774,7 @@ This will allow to share the cluster later.</p> </div> </div> <div class="sect3"> -<h4 id="hbase.tests.example.code"><a class="anchor" href="#hbase.tests.example.code"></a>150.4.5. Tests Skeleton Code</h4> +<h4 id="hbase.tests.example.code"><a class="anchor" href="#hbase.tests.example.code"></a>151.4.5. Tests Skeleton Code</h4> <div class="paragraph"> <p>Here is a test skeleton code with Categorization and a Category-based timeout rule to copy and paste and use as basis for test contribution.</p> </div> @@ -28708,7 +28818,7 @@ This will allow to share the cluster later.</p> </div> </div> <div class="sect2"> -<h3 id="integration.tests"><a class="anchor" href="#integration.tests"></a>150.5. Integration Tests</h3> +<h3 id="integration.tests"><a class="anchor" href="#integration.tests"></a>151.5. Integration Tests</h3> <div class="paragraph"> <p>HBase integration/system tests are tests that are beyond HBase unit tests. They are generally long-lasting, sizeable (the test can be asked to 1M rows or 1B rows), targetable (they can take configuration that will point them at the ready-made cluster they are to run against; integration tests do not include cluster start/stop code), and verifying success, integration tests rely on public APIs only; they do not attempt to examine server internals asserting success/fail. @@ -28762,7 +28872,7 @@ For example,</p> </div> </div> <div class="sect3"> -<h4 id="maven.build.commands.integration.tests.mini"><a class="anchor" href="#maven.build.commands.integration.tests.mini"></a>150.5.1. Running integration tests against mini cluster</h4> +<h4 id="maven.build.commands.integration.tests.mini"><a class="anchor" href="#maven.build.commands.integration.tests.mini"></a>151.5.1. Running integration tests against mini cluster</h4> <div class="paragraph"> <p>HBase 0.92 added a <code>verify</code> maven target. Invoking it, for example by doing <code>mvn verify</code>, will run all the phases up to and including the verify phase via the maven <a href="http://maven.apache.org/plugins/maven-failsafe-plugin/";>failsafe @@ -28807,7 +28917,7 @@ To just run <code>IntegrationTestClassXYZ.java</code>, use: mvn </div> </div> <div class="sect3"> -<h4 id="maven.build.commands.integration.tests.distributed"><a class="anchor" href="#maven.build.commands.integration.tests.distributed"></a>150.5.2. Running integration tests against distributed cluster</h4> +<h4 id="maven.build.commands.integration.tests.distributed"><a class="anchor" href="#maven.build.commands.integration.tests.distributed"></a>151.5.2. Running integration tests against distributed cluster</h4> <div class="paragraph"> <p>If you have an already-setup HBase cluster, you can launch the integration tests by invoking the class <code>IntegrationTestsDriver</code>. You may have to run test-compile first. @@ -28844,7 +28954,7 @@ For other deployment options, a ClusterManager can be implemented and plugged in </div> </div> <div class="sect3"> -<h4 id="maven.build.commands.integration.tests.destructive"><a class="anchor" href="#maven.build.commands.integration.tests.destructive"></a>150.5.3. Destructive integration / system tests (ChaosMonkey)</h4> +<h4 id="maven.build.commands.integration.tests.destructive"><a class="anchor" href="#maven.build.commands.integration.tests.destructive"></a>151.5.3. Destructive integration / system tests (ChaosMonkey)</h4> <div class="paragraph"> <p>HBase 0.96 introduced a tool named <code>ChaosMonkey</code>, modeled after <a href="http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html";>same-named tool by Netflix’s Chaos Monkey tool</a>. @@ -28949,14 +29059,14 @@ policy, which is configured with all the available actions. It chose to run <cod </div> </div> <div class="sect3"> -<h4 id="_available_policies"><a class="anchor" href="#_available_policies"></a>150.5.4. Available Policies</h4> +<h4 id="_available_policies"><a class="anchor" href="#_available_policies"></a>151.5.4. Available Policies</h4> <div class="paragraph"> <p>HBase ships with several ChaosMonkey policies, available in the <code>hbase/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/policies/</code> directory.</p> </div> </div> <div class="sect3"> -<h4 id="chaos.monkey.properties"><a class="anchor" href="#chaos.monkey.properties"></a>150.5.5. Configuring Individual ChaosMonkey Actions</h4> +<h4 id="chaos.monkey.properties"><a class="anchor" href="#chaos.monkey.properties"></a>151.5.5. Configuring Individual ChaosMonkey Actions</h4> <div class="paragraph"> <p>Since HBase version 1.0.0 (<a href="https://issues.apache.org/jira/browse/HBASE-11348";>HBASE-11348</a>), ChaosMonkey integration tests can be configured per test run. @@ -29024,10 +29134,10 @@ properties file, which may be <code>hbase-site.xml</code> or a different propert </div> </div> <div class="sect1"> -<h2 id="developing"><a class="anchor" href="#developing"></a>151. Developer Guidelines</h2> +<h2 id="developing"><a class="anchor" href="#developing"></a>152. Developer Guidelines</h2> <div class="sectionbody"> <div class="sect2"> -<h3 id="_branches"><a class="anchor" href="#_branches"></a>151.1. Branches</h3> +<h3 id="_branches"><a class="anchor" href="#_branches"></a>152.1. Branches</h3> <div class="paragraph"> <p>We use Git for source code management and latest development happens on <code>master</code> branch. There are branches for past major/minor/maintenance releases and important features and bug fixes are often @@ -29035,7 +29145,7 @@ branches for past major/minor/maintenance releases and important features and bu </div> </div> <div class="sect2"> -<h3 id="_release_managers"><a class="anchor" href="#_release_managers"></a>151.2. Release Managers</h3> +<h3 id="_release_managers"><a class="anchor" href="#_release_managers"></a>152.2. Release Managers</h3> <div class="paragraph"> <p>Each maintained release branch has a release manager, who volunteers to coordinate new features and bug fixes are backported to that release. The release managers are <a href="https://hbase.apache.org/team-list.html";>committers</a>. @@ -29083,9 +29193,9 @@ End-of-life releases are not included in this list. </table> </div> <div class="sect2"> -<h3 id="code.standards"><a class="anchor" href="#code.standards"></a>151.3. Code Standards</h3> +<h3 id="code.standards"><a class="anchor" href="#code.standards"></a>152.3. Code Standards</h3> <div class="sect3"> -<h4 id="_interface_classifications"><a class="anchor" href="#_interface_classifications"></a>151.3.1. Interface Classifications</h4> +<h4 id="_interface_classifications"><a class="anchor" href="#_interface_classifications"></a>152.3.1. Interface Classifications</h4> <div class="paragraph"> <p>Interfaces are classified both by audience and by stability level. These labels appear at the head of a class. @@ -29165,7 +29275,7 @@ Mark your new packages if publicly accessible.</p> </div> </div> <div class="sect3"> -<h4 id="common.patch.feedback"><a class="anchor" href="#common.patch.feedback"></a>151.3.2. Code Formatting Conventions</h4> +<h4 id="common.patch.feedback"><a class="anchor" href="#common.patch.feedback"></a>152.3.2. Code Formatting Conventions</h4> <div class="paragraph"> <p>Please adhere to the following guidelines so that your patches can be reviewed more quickly. These guidelines have been developed based upon common feedback on patches from new contributors.</p> @@ -29353,7 +29463,7 @@ If you are creating a new class that needs to implement this interface, do not f </div> </div> <div class="sect3"> -<h4 id="_garbage_collection_conserving_guidelines"><a class="anchor" href="#_garbage_collection_conserving_guidelines"></a>151.3.3. Garbage-Collection Conserving Guidelines</h4> +<h4 id="_garbage_collection_conserving_guidelines"><a class="anchor" href="#_garbage_collection_conserving_guidelines"></a>152.3.3. Garbage-Collection Conserving Guidelines</h4> <div class="paragraph"> <p>The following guidelines were borrowed from <a href="http://engineering.linkedin.com/performance/linkedin-feed-faster-less-jvm-garbage"; class="bare">http://engineering.linkedin.com/performance/linkedin-feed-faster-less-jvm-garbage</a>. Keep them in mind to keep preventable garbage collection to a minimum. Have a look @@ -29385,13 +29495,13 @@ these guidelines.</p> </div> </div> <div class="sect2"> -<h3 id="design.invariants"><a class="anchor" href="#design.invariants"></a>151.4. Invariants</h3> +<h3 id="design.invariants"><a class="anchor" href="#design.invariants"></a>152.4. Invariants</h3> <div class="paragraph"> <p>We don’t have many but what we have we list below. All are subject to challenge of course but until then, please hold to the rules of the road.</p> </div> <div class="sect3"> -<h4 id="design.invariants.zk.data"><a class="anchor" href="#design.invariants.zk.data"></a>151.4.1. No permanent state in ZooKeeper</h4> +<h4 id="design.invariants.zk.data"><a class="anchor" href="#design.invariants.zk.data"></a>152.4.1. No permanent state in ZooKeeper</h4> <div class="paragraph"> <p>ZooKeeper state should transient (treat it like memory). If ZooKeeper state is deleted, hbase should be able to recover and essentially be in the same state.</p> </div> @@ -29423,7 +29533,7 @@ Follow progress on this issue at <a href="https://issues.apache.org/jira/browse/ </div> </div> <div class="sect2"> -<h3 id="run.insitu"><a class="anchor" href="#run.insitu"></a>151.5. Running In-Situ</h3> +<h3 id="run.insitu"><a class="anchor" href="#run.insitu"></a>152.5. Running In-Situ</h3> <div class="paragraph"> <p>If you are developing Apache HBase, frequently it is useful to test your changes against a more-real cluster than what you find in unit tests. In this case, HBase can be run directly from the source in local-mode. @@ -29452,7 +29562,7 @@ Generally, the above command is just a good thing to try running first, if maven </div> </div> <div class="sect2"> -<h3 id="add.metrics"><a class="anchor" href="#add.metrics"></a>151.6. Adding Metrics</h3> +<h3 id="add.metrics"><a class="anchor" href="#add.metrics"></a>152.6. Adding Metrics</h3> <div class="paragraph"> <p>After adding a new feature a developer might want to add metrics. HBase exposes metrics using the Hadoop Metrics 2 system, so adding a new metric involves exposing that metric to the hadoop system. @@ -29462,14 +29572,14 @@ To get an in-depth look at the reasoning and structure of these classes you can To add a metric to an existing MBean follow the short guide below:</p> </div> <div class="sect3"> -<h4 id="_add_metric_name_and_function_to_hadoop_compat_interface"><a class="anchor" href="#_add_metric_name_and_function_to_hadoop_compat_interface"></a>151.6.1. Add Metric name and Function to Hadoop Compat Interface.</h4> +<h4 id="_add_metric_name_and_function_to_hadoop_compat_interface"><a class="anchor" href="#_add_metric_name_and_function_to_hadoop_compat_interface"></a>152.6.1. Add Metric name and Function to Hadoop Compat Interface.</h4> <div class="paragraph"> <p>Inside of the source interface the corresponds to where the metrics are generated (eg MetricsMasterSource for things coming from HMaster) create new static strings for metric name and description. Then add a new method that will be called to add new reading.</p> </div> </div> <div class="sect3"> -<h4 id="_add_the_implementation_to_both_hadoop_1_and_hadoop_2_compat_modules"><a class="anchor" href="#_add_the_implementation_to_both_hadoop_1_and_hadoop_2_compat_modules"></a>151.6.2. Add the Implementation to Both Hadoop 1 and Hadoop 2 Compat modules.</h4> +<h4 id="_add_the_implementation_to_both_hadoop_1_and_hadoop_2_compat_modules"><a class="anchor" href="#_add_the_implementation_to_both_hadoop_1_and_hadoop_2_compat_modules"></a>152.6.2. Add the Implementation to Both Hadoop 1 and Hadoop 2 Compat modules.</h4> <div class="paragraph"> <p>Inside of the implementation of the source (eg. MetricsMasterSourceImpl in the above example) create a new histogram, counter, gauge, or stat in the init method. @@ -29482,7 +29592,7 @@ For this the MetricsAssertHelper is provided.</p> </div> </div> <div class="sect2"> -<h3 id="git.best.practices"><a class="anchor" href="#git.best.practices"></a>151.7. Git Best Practices</h3> +<h3 id="git.best.practices"><a class="anchor" href="#git.best.practices"></a>152.7. Git Best Practices</h3> <div class="dlist"> <dl> <dt class="hdlist1">Avoid git merges.</dt> @@ -29499,7 +29609,7 @@ For this the MetricsAssertHelper is provided.</p> <p>Please contribute to this document if you think of other Git best practices.</p> </div> <div class="sect3"> -<h4 id="__code_rebase_all_git_branches_sh_code"><a class="anchor" href="#__code_rebase_all_git_branches_sh_code"></a>151.7.1. <code>rebase_all_git_branches.sh</code></h4> +<h4 id="__code_rebase_all_git_branches_sh_code"><a class="anchor" href="#__code_rebase_all_git_branches_sh_code"></a>152.7.1. <code>rebase_all_git_branches.sh</code></h4> <div class="paragraph"> <p>The <em>dev-support/rebase_all_git_branches.sh</em> script is provided to help keep your Git repository clean. Use the <code>-h</code> parameter to get usage instructions. @@ -29512,7 +29622,7 @@ The script checks the directory for sub-directory called <em>.git/</em>, before </div> </div> <div class="sect2"> -<h3 id="submitting.patches"><a class="anchor" href="#submitting.patches"></a>151.8. Submitting Patches</h3> +<h3 id="submitting.patches"><a class="anchor" href="#submitting.patches"></a>152.8. Submitting Patches</h3> <div class="paragraph"> <p>If you are new to submitting patches to open source or new to submitting patches to Apache, start by reading the <a href="http://commons.apache.org/patches.html";>On Contributing Patches</a> page from @@ -29522,7 +29632,7 @@ It provides a nice overview that applies equally to the Apache HBase Project. good read to understand development workflow.</p> </div> <div class="sect3"> -<h4 id="submitting.patches.create"><a class="anchor" href="#submitting.patches.create"></a>151.8.1. Create Patch</h4> +<h4 id="submitting.patches.create"><a class="anchor" href="#submitting.patches.create"></a>152.8.1. Create Patch</h4> <div class="paragraph"> <p>Make sure you review <a href="#common.patch.feedback">common.patch.feedback</a> for code style. If your patch @@ -29628,7 +29738,7 @@ Click on <b class="button">Cancel Patch</b> and then on <b class="button">Submit </div> </div> <div class="sect3"> -<h4 id="submitting.patches.tests"><a class="anchor" href="#submitting.patches.tests"></a>151.8.2. Unit Tests</h4> +<h4 id="submitting.patches.tests"><a class="anchor" href="#submitting.patches.tests"></a>152.8.2. Unit Tests</h4> <div class="paragraph"> <p>Always add and/or update relevant unit tests when making the changes. Make sure that new/changed unit tests pass locally before submitting the patch because it is faster @@ -29645,13 +29755,13 @@ See <a href="#hbase.tests">hbase.tests</a> for more information on tests.</p> </div> </div> <div class="sect3"> -<h4 id="_integration_tests"><a class="anchor" href="#_integration_tests"></a>151.8.3. Integration Tests</h4> +<h4 id="_integration_tests"><a class="anchor" href="#_integration_tests"></a>152.8.3. Integration Tests</h4> <div class="paragraph"> <p>Significant new features should provide an integration test in addition to unit tests, suitable for exercising the new feature at different points in its configuration space.</p> </div> </div> <div class="sect3"> -<h4 id="reviewboard"><a class="anchor" href="#reviewboard"></a>151.8.4. ReviewBoard</h4> +<h4 id="reviewboard"><a class="anchor" href="#reviewboard"></a>152.8.4. ReviewBoard</h4> <div class="paragraph"> <p>Patches larger than one screen, or patches that will be tricky to review, should go through <a href="http://reviews.apache.org";>ReviewBoard</a>.</p> </div> @@ -29694,7 +29804,7 @@ This attaches the ReviewBoard to the JIRA, for easy access.</p> </div> </div> <div class="sect3"> -<h4 id="_guide_for_hbase_committers"><a class="anchor" href="#_guide_for_hbase_committers"></a>151.8.5. Guide for HBase Committers</h4> +<h4 id="_guide_for_hbase_committers"><a class="anchor" href="#_guide_for_hbase_committers"></a>152.8.5. Guide for HBase Committers</h4> <div class="sect4"> <h5 id="_new_committers"><a class="anchor" href="#_new_committers"></a>New committers</h5> <div class="paragraph"> @@ -29974,21 +30084,21 @@ A committer should.</p> </div> </div> <div class="sect3"> -<h4 id="_dialog"><a class="anchor" href="#_dialog"></a>151.8.6. Dialog</h4> +<h4 id="_dialog"><a class="anchor" href="#_dialog"></a>152.8.6. Dialog</h4> <div class="paragraph"> <p>Committers should hang out in the #hbase room on irc.freenode.net for real-time discussions. However any substantive discussion (as with any off-list project-related discussion) should be re-iterated in Jira or on the developer list.</p> </div> </div> <div class="sect3"> -<h4 id="_do_not_edit_jira_comments"><a class="anchor" href="#_do_not_edit_jira_comments"></a>151.8.7. Do not edit JIRA comments</h4> +<h4 id="_do_not_edit_jira_comments"><a class="anchor" href="#_do_not_edit_jira_comments"></a>152.8.7. Do not edit JIRA comments</h4> <div class="paragraph"> <p>Misspellings and/or bad grammar is preferable to the disruption a JIRA comment edit causes: See the discussion at <a href="http://search-hadoop.com/?q=%5BReopened%5D+%28HBASE-451%29+Remove+HTableDescriptor+from+HRegionInfo&fc_project=HBase";>Re:(HBASE-451) Remove HTableDescriptor from HRegionInfo</a></p> </div> </div> </div> <div class="sect2"> -<h3 id="hbase.archetypes.development"><a class="anchor" href="#hbase.archetypes.development"></a>151.9. Development of HBase-related Maven archetypes</h3> +<h3 id="hbase.archetypes.development"><a class="anchor" href="#hbase.archetypes.development"></a>152.9. Development of HBase-related Maven archetypes</h3> <div class="paragraph"> <p>The development of HBase-related Maven archetypes was begun with <a href="https://issues.apache.org/jira/browse/HBASE-14876";>HBASE-14876</a>. @@ -30008,7 +30118,7 @@ For information on unit tests for HBase itself, see <a href="#hbase.tests">hbase </div> </div> <div class="sect1"> -<h2 id="_junit"><a class="anchor" href="#_junit"></a>152. JUnit</h2> +<h2 id="_junit"><a class="anchor" href="#_junit"></a>153. JUnit</h2> <div class="sectionbody"> <div class="paragraph"> <p>HBase uses <a href="http://junit.org";>JUnit</a> 4 for unit tests</p> @@ -30080,7 +30190,7 @@ For an introduction to JUnit, see <a href="https://github.com/junit-team/junit/w </div> </div> <div class="sect1"> -<h2 id="mockito"><a class="anchor" href="#mockito"></a>153. Mockito</h2> +<h2 id="mockito"><a class="anchor" href="#mockito"></a>154. Mockito</h2> <div class="sectionbody"> <div class="paragraph"> <p>Mockito is a mocking framework. @@ -30158,7 +30268,7 @@ Similarly, you can now expand into other operations such as Get, Scan, or Delete </div> </div> <div class="sect1"> -<h2 id="_mrunit"><a class="anchor" href="#_mrunit"></a>154. MRUnit</h2> +<h2 id="_mrunit"><a class="anchor" href="#_mrunit"></a>155. MRUnit</h2> <div class="sectionbody"> <div class="paragraph"> <p><a href="http://mrunit.apache.org/";>Apache MRUnit</a> is a library that allows you to unit-test MapReduce jobs. @@ -30255,7 +30365,7 @@ strValue2 = <span class="string"><span class="delimiter">"</span><span clas </div> </div> <div class="sect1"> -<h2 id="_integration_testing_with_an_hbase_mini_cluster"><a class="anchor" href="#_integration_testing_with_an_hbase_mini_cluster"></a>155. Integration Testing with an HBase Mini-Cluster</h2> +<h2 id="_integration_testing_with_an_hbase_mini_cluster"><a class="anchor" href="#_integration_testing_with_an_hbase_mini_cluster"></a>156. Integration Testing with an HBase Mini-Cluster</h2> <div class="sectionbody"> <div class="paragraph"> <p>HBase ships with HBaseTestingUtility, which makes it easy to write integration tests using a <em class="firstterm">mini-cluster</em>. @@ -30345,7 +30455,7 @@ Starting the mini-cluster takes about 20-30 seconds, but that should be appropri </div> <h1 id="protobuf" class="sect0"><a class="anchor" href="#protobuf"></a>Protobuf in HBase</h1> <div class="sect1"> -<h2 id="_protobuf"><a class="anchor" href="#_protobuf"></a>156. Protobuf</h2> +<h2 id="_protobuf"><a class="anchor" href="#_protobuf"></a>157. Protobuf</h2> <div class="sectionbody"> <div class="paragraph"> <p>HBase uses Google’s <a href="http://protobuf.protobufs";>protobufs</a> wherever @@ -30400,7 +30510,7 @@ practice needs to whither. We’ll make plain why in the later <a href="#shaded.protobuf">hbase-2.0.0</a> section.</p> </div> <div class="sect2"> -<h3 id="shaded.protobuf"><a class="anchor" href="#shaded.protobuf"></a>156.1. hbase-2.0.0 and the shading of protobufs (HBASE-15638)</h3> +<h3 id="shaded.protobuf"><a class="anchor" href="#shaded.protobuf"></a>157.1. hbase-2.0.0 and the shading of protobufs (HBASE-15638)</h3> <div class="paragraph"> <p>As of hbase-2.0.0, our protobuf usage gets a little more involved. HBase core protobuf references are offset so as to refer to a private, @@ -30610,7 +30720,7 @@ zookeeper could start dropping sessions if it has to run through a directory of </div> </div> <div class="sect1"> -<h2 id="_using_existing_zookeeper_ensemble"><a class="anchor" href="#_using_existing_zookeeper_ensemble"></a>157. Using existing ZooKeeper ensemble</h2> +<h2 id="_using_existing_zookeeper_ensemble"><a class="anchor" href="#_using_existing_zookeeper_ensemble"></a>158. Using existing ZooKeeper ensemble</h2> <div class="sectionbody"> <div class="paragraph"> <p>To point HBase at an existing ZooKeeper cluster, one that is not managed by HBase, set <code>HBASE_MANAGES_ZK</code> in <em>conf/hbase-env.sh</em> to false</p> @@ -30647,7 +30757,7 @@ Additionally, see the <a href="http://wiki.apache.org/hadoop/ZooKeeper/FAQ#A7";>Z </div> </div> <div class="sect1"> -<h2 id="zk.sasl.auth"><a class="anchor" href="#zk.sasl.auth"></a>158. SASL Authentication with ZooKeeper</h2> +<h2 id="zk.sasl.auth"><a class="anchor" href="#zk.sasl.auth"></a>159. SASL Authentication with ZooKeeper</h2> <div class="sectionbody"> <div class="paragraph"> <p>Newer releases of Apache HBase (>= 0.92) will support connecting to a ZooKeeper Quorum that supports SASL authentication (which is available in ZooKeeper versions 3.4.0 or later).</p> @@ -30657,7 +30767,7 @@ Additionally, see the <a href="http://wiki.apache.org/hadoop/ZooKeeper/FAQ#A7";>Z ZooKeeper/HBase mutual authentication (<a href="https://issues.apache.org/jira/browse/HBASE-2418";>HBASE-2418</a>) is required as part of a complete secure HBase configuration (<a href="https://issues.apache.org/jira/browse/HBASE-3025";>HBASE-3025</a>). For simplicity of explication, this section ignores additional configuration required (Secure HDFS and Coprocessor configuration). It’s recommended to begin with an HBase-managed ZooKeeper configuration (as opposed to a standalone ZooKeeper quorum) for ease of learning.</p> </div> <div class="sect2"> -<h3 id="_operating_system_prerequisites"><a class="anchor" href="#_operating_system_prerequisites"></a>158.1. Operating System Prerequisites</h3> +<h3 id="_operating_system_prerequisites"><a class="anchor" href="#_operating_system_prerequisites"></a>159.1. Operating System Prerequisites</h3> <div class="paragraph"> <p>You need to have a working Kerberos KDC setup. For each <code>$HOST</code> that will run a ZooKeeper server, you should have a principle <code>zookeeper/$HOST</code>. @@ -30693,7 +30803,7 @@ The ZooKeeper client and server libraries manage their own ticket refreshment by </div> </div> <div class="sect2"> -<h3 id="_hbase_managed_zookeeper_configuration"><a class="anchor" href="#_hbase_managed_zookeeper_configuration"></a>158.2. HBase-managed ZooKeeper Configuration</h3> +<h3 id="_hbase_managed_zookeeper_configuration"><a class="anchor" href="#_hbase_managed_zookeeper_configuration"></a>159.2. HBase-managed ZooKeeper Configuration</h3> <div class="paragraph"> <p>On each node that will run a zookeeper, a master, or a regionserver, create a <a href="http://docs.oracle.com/javase/1.4.2/docs/guide/security/jgss/tutorials/LoginConfigFile.html";>JAAS</a> configuration file in the conf directory of the node’s <em>HBASE_HOME</em> directory that looks like the following:</p> </div> @@ -30785,7 +30895,7 @@ bin/hbase regionserver start</pre> </div> </div> <div class="sect2"> -<h3 id="_external_zookeeper_configuration"><a class="anchor" href="#_external_zookeeper_configuration"></a>158.3. External ZooKeeper Configuration</h3> +<h3 id="_external_zookeeper_configuration"><a class="anchor" href="#_external_zookeeper_configuration"></a>159.3. External ZooKeeper Configuration</h3> <div class="paragraph"> <p>Add a JAAS configuration file that looks like:</p> </div> @@ -30886,7 +30996,7 @@ bin/hbase regionserver start</pre> </div> </div> <div class="sect2"> -<h3 id="_zookeeper_server_authentication_log_output"><a class="anchor" href="#_zookeeper_server_authentication_log_output"></a>158.4. ZooKeeper Server Authentication Log Output</h3> +<h3 id="_zookeeper_server_authentication_log_output"><a class="anchor" href="#_zookeeper_server_authentication_log_output"></a>159.4. ZooKeeper Server Authentication Log Output</h3> <div class="paragraph"> <p>If the configuration above is successful, you should see something similar to the following in your ZooKeeper server logs:</p> </div> @@ -30908,7 +31018,7 @@ bin/hbase regionserver start</pre> </div> </div> <div class="sect2"> -<h3 id="_zookeeper_client_authentication_log_output"><a class="anchor" href="#_zookeeper_client_authentication_log_output"></a>158.5. ZooKeeper Client Authentication Log Output</h3> +<h3 id="_zookeeper_client_authentication_log_output"><a class="anchor" href="#_zookeeper_client_authentication_log_output"></a>159.5. ZooKeeper Client Authentication Log Output</h3> <div class="paragraph"> <p>On the ZooKeeper client side (HBase master or regionserver), you should see something similar to the following:</p> </div> @@ -30929,7 +31039,7 @@ bin/hbase regionserver start</pre> </div> </div> <div class="sect2"> -<h3 id="_configuration_from_scratch"><a class="anchor" href="#_configuration_from_scratch"></a>158.6. Configuration from Scratch</h3> +<h3 id="_configuration_from_scratch"><a class="anchor" href="#_configuration_from_scratch"></a>159.6. Configuration from Scratch</h3> <div class="paragraph"> <p>This has been tested on the current standard Amazon Linux AMI. First setup KDC and principals as described above. @@ -30955,9 +31065,9 @@ bin/hbase regionserver &</pre> </div> </div> <div class="sect2"> -<h3 id="_future_improvements"><a class="anchor" href="#_future_improvements"></a>158.7. Future improvements</h3> +<h3 id="_future_improvements"><a class="anchor" href="#_future_improvements"></a>159.7. Future improvements</h3> <div class="sect3"> -<h4 id="_fix_target_cached_classpath_txt"><a class="anchor" href="#_fix_target_cached_classpath_txt"></a>158.7.1. Fix target/cached_classpath.txt</h4> +<h4 id="_fix_target_cached_classpath_txt"><a class="anchor" href="#_fix_target_cached_classpath_txt"></a>159.7.1. Fix target/cached_classpath.txt</h4> <div class="paragraph"> <p>You must override the standard hadoop-core jar file from the <code>target/cached_classpath.txt</code> file with the version containing the HADOOP-7070 fix. You can use the following script to do this:</p> @@ -30970,13 +31080,13 @@ mv target/tmp.txt target/cached_classpath.txt</pre> </div> </div> <div class="sect3"> -<h4 id="_set_jaas_configuration_programmatically"><a class="anchor" href="#_set_jaas_configuration_programmatically"></a>158.7.2. Set JAAS configuration programmatically</h4> +<h4 id="_set_jaas_configuration_programmatically"><a class="anchor" href="#_set_jaas_configuration_programmatically"></a>159.7.2. Set JAAS configuration programmatically</h4> <div class="paragraph"> <p>This would avoid the need for a separate Hadoop jar that fixes <a href="https://issues.apache.org/jira/browse/HADOOP-7070";>HADOOP-7070</a>.</p> </div> </div> <div class="sect3"> -<h4 id="_elimination_of_code_kerberos_removehostfromprincipal_code_and_kerberos_removerealmfromprincipal"><a class="anchor" href="#_elimination_of_code_kerberos_removehostfromprincipal_code_and_kerberos_removerealmfromprincipal"></a>158.7.3. Elimination of <code>kerberos.removeHostFromPrincipal</code> and`kerberos.removeRealmFromPrincipal`</h4> +<h4 id="_elimination_of_code_kerberos_removehostfromprincipal_code_and_kerberos_removerealmfromprincipal"><a class="anchor" href="#_elimination_of_code_kerberos_removehostfromprincipal_code_and_kerberos_removerealmfromprincipal"></a>159.7.3. Elimination of <code>kerberos.removeHostFromPrincipal</code> and`kerberos.removeRealmFromPrincipal`</h4> </div> </div> @@ -30984,7 +31094,7 @@ mv target/tmp.txt target/cached_classpath.txt</pre> </div> <h1 id="community" class="sect0"><a class="anchor" href="#community"></a>Community</h1> <div class="sect1"> -<h2 id="_decisions"><a class="anchor" href="#_decisions"></a>159. Decisions</h2> +<h2 id="_decisions"><a class="anchor" href="#_decisions"></a>160. Decisions</h2> <div class="sectionbody"> <div class="paragraph"> <div class="title">Feature Branches</div> @@ -31056,7 +31166,7 @@ We also are currently in violation of this basic tenet — repli </div> </div> <div class="sect1"> -<h2 id="community.roles"><a class="anchor" href="#community.roles"></a>160. Community Roles</h2> +<h2 id="community.roles"><a class="anchor" href="#community.roles"></a>161. Community Roles</h2> <div class="sectionbody"> <div id="owner" class="paragraph"> <div class="title">Component Owner/Lieutenant</div> @@ -31083,7 +31193,7 @@ Owners do not need to be committers.</p> </div> </div> <div class="sect1"> -<h2 id="hbase.commit.msg.format"><a class="anchor" href="#hbase.commit.msg.format"></a>161. Commit Message format</h2> +<h2 id="hbase.commit.msg.format"><a class="anchor" href="#hbase.commit.msg.format"></a>162. Commit Message format</h2> <div class="sectionbody"> <div class="paragraph"> <p>We <a href="http://search-hadoop.com/m/Gwxwl10cFHa1";>agreed</a> to the following Git commit message format:</p> @@ -33360,7 +33470,7 @@ Options: </div> </div> <div class="sect1"> -<h2 id="data.block.encoding.enable"><a class="anchor" href="#data.block.encoding.enable"></a>162. Enable Data Block Encoding</h2> +<h2 id="data.block.encoding.enable"><a class="anchor" href="#data.block.encoding.enable"></a>163. Enable Data Block Encoding</h2> <div class="sectionbody"> <div class="paragraph"> <p>Codecs are built into HBase so no extra configuration is needed. @@ -34278,7 +34388,7 @@ The <code>LocalFileSpanReceiver</code> looks in <em>hbase-site.xml</em> for </div> </div> <div class="sect1"> -<h2 id="tracing.client.modifications"><a class="anchor" href="#tracing.client.modifications"></a>163. Client Modifications</h2> +<h2 id="tracing.client.modifications"><a class="anchor" href="#tracing.client.modifications"></a>164. Client Modifications</h2> <div class="sectionbody"> <div class="paragraph"> <p>In order to turn on tracing in your client code, you must initialize the module sending spans to receiver once per client process.</p> @@ -34336,7 +34446,7 @@ See the HTrace <em>README</em> for more information on Samplers.</p> </div> </div> <div class="sect1"> -<h2 id="tracing.client.shell"><a class="anchor" href="#tracing.client.shell"></a>164. Tracing from HBase Shell</h2> +<h2 id="tracing.client.shell"><a class="anchor" href="#tracing.client.shell"></a>165. Tracing from HBase Shell</h2> <div class="sectionbody"> <div class="paragraph"> <p>You can use <code>trace</code> command for tracing requests from HBase Shell. <code>trace 'start'</code> command turns on tracing and <code>trace 'stop'</code> command turns off tracing.</p> @@ -34646,7 +34756,7 @@ The server will return cellblocks compressed using this same compressor as long <div id="footer"> <div id="footer-text"> Version 2.0.0-SNAPSHOT<br> -Last updated 2016-11-07 14:29:11 +00:00 +Last updated 2017-02-16 14:29:53 +00:00 </div> </div> </body>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/ce958bce/bulk-loads.html ---------------------------------------------------------------------- diff --git a/bulk-loads.html b/bulk-loads.html index a5c364f..d4bbebd 100644 --- a/bulk-loads.html +++ b/bulk-loads.html @@ -7,7 +7,7 @@ <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170205" /> + <meta name="Date-Revision-yyyymmdd" content="20170217" /> <meta http-equiv="Content-Language" content="en" /> <title>Apache HBase – Bulk Loads in Apache HBase (TM) @@ -323,7 +323,7 @@ under the License. --> <a href="https://www.apache.org/";>The Apache Software Foundation</a>. All rights reserved. - <li id="publishDate" class="pull-right">Last Published: 2017-02-05</li> + <li id="publishDate" class="pull-right">Last Published: 2017-02-17</li> </p> </div>
