[Hadoop Wiki] Update of "HowToDevelopUnitTests" by Luke Lu
Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change
notification.
The "HowToDevelopUnitTests" page has been changed by LukeLu.
The comment on this change is: Let unit tests be "unit" tests.
http://wiki.apache.org/hadoop/HowToDevelopUnitTests?action=diff&rev1=6&rev2=7
--
This leaves less code around (lower maintenance costs), and ensures that any
failure gets reported with a full stack trace.
+ == Let Unit Tests be "Unit" tests ==
+
+ Avoid starting servers (including Jetty and Mini{DFS|MR}Clusters) in
'''unit''' tests, as they take tens of seconds to start for each test (HDFS and
Map``Reduce tests already take many hours mostly due to these servers starts).
Use them only in cross component '''functional''' or '''integration''' (system)
tests (cf. [[https://issues.apache.org/jira/browse/HADOOP-6399|HADOOP-6399]]).
Try to use one of the lighter weight
[[http://www.martinfowler.com/bliki/TestDouble.html|test doubles]] for
collaborating components for the component under test. Hadoop has adopted the
[[http://mockito.googlecode.com/svn/tags/latest/javadoc/org/mockito/Mockito.html|Mockito]]
library for easy mock and stub creation.
+
== References ==
* [[http://code.google.com/p/t2framework/wiki/JUnitQuickTutorial|Quick
tutorial]] on the JUnit website.
[Hadoop Wiki] Trivial Update of "Hbase/MavenPrimer" by stack
Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change
notification.
The "Hbase/MavenPrimer" page has been changed by stack.
http://wiki.apache.org/hadoop/Hbase/MavenPrimer?action=diff&rev1=2&rev2=3
--
+ == Table of Contents ==
+
+ 1. [[#whatdoineed|What do I need?]]
+ 1. [[#ide|What about IDE support?]]
+ 1. [[#jar|How do I just build the jars without going through a full unit
test cycle?]]
+ 1. [[#full|How do I do a full Distribution style build, including running
all the tests?]]
+ 1. [[#core|I don't want to build all the other modules, I'm just interested
in the base Hbase 'core' stuff, is there something simpler?]]
+ 1. [[#javadoc|How do I build javadoc only?]]
+ 1. [[#oneunittest|How do I run one unit test only?]]
+
+ <>
+
== What do I need? ==
You need Maven 2.0.9 or greater, I would recommend Maven 2.2 though.
Downloads here:
@@ -14, +26 @@
as there are a few things within the build that require more than the
standard JVM memory allows.
+ <>
+
== What about IDE support? ==
IntelliJ: Built-in awesomeness
Eclipse: Definitely recommend the M2Eclipse plugin here:
http://m2eclipse.sonatype.org/
+ <>
== How do I just build the jars without going through a full unit test cycle?
==
From the top-level directory:
@@ -27, +42 @@
you'll find the jars under the 'target' sub-directory of each sub-module, so
'hbase/core/target/hbase-core-0.20.1.jar' is effectively the original
'hbase.jar' produced previously.
+ <>
== How do I do a full Distribution style build, including running all the
tests? ==
From the top-level directory:
@@ -35, +51 @@
In the top-level 'hbase/target' directory you will find a set of tar balls
and zip files. the '-bin' contains something akin to the original HBase
release tar ball. The -'project' and -'src' are there to provide an archive of
the entire Maven project as it stood during release, and a slimmer, source-only
bundle respectively.
+ <>
== I don't want to build all the other modules, I'm just interested in the
base Hbase 'core' stuff, is there something simpler? ==
sure, just:
@@ -44, +61 @@
look in the target sub-directory.
- Enjoy!
+ <>
+ == How do I build javadoc only? ==
+
+ {{{mvn javadoc:javadoc}}}
+
+ <>
+
+ == How do I run one unit test only? ==
+
+ {{{mvn mvn test -Dtest=}}}
+
[Hadoop Wiki] Update of "SupportingProjects" by YoramKu lbak
Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification. The "SupportingProjects" page has been changed by YoramKulbak. http://wiki.apache.org/hadoop/SupportingProjects?action=diff&rev1=27&rev2=28 -- * [[http://github.com/greglu/hbase-stargate|hbase-stargate]] -- Ruby client for HBase Stargate. * [[http://github.com/ghelmling/meetup.beeno|Meetup.Beeno]] -- simple HBase Java "beans" mapping framework based on annotations. Includes a rudimentary high level query API that generates the appropriate server-side filters. * [[https://jira.springsource.org/browse/SPR-5950|Spring Framework HBase Template]] -- Provides HBase data access templates similar to what is provided in Spring for JDBC, Hibernate, iBatis, etc. If you find this useful, please vote for its inclusion in the Spring Framework. - * [[http://github.com/davidsantiago/clojure-hbase|Clojure-HBase]] -- A library for convenient access to HBase from Clojure. + * [[http://github.com/davidsantiago/clojure-hbase|Clojure-HBase]] -- A library for convenient access to HBase from Clojure. * [[http://www.lilycms.org/lily/index/sketchbook/hbaseindexes.html|HBase indexing library]] -- A library for building and querying HBase-table-based indexes. - * [[http://github.com/akkumar/hbasene|HBasene]] -- Lucene+HBase - Using HBase as the backing store for the TF-IDF representations needed by Lucene. Also, contains a library for constructing lucene indices from HBase schema. + * [[http://github.com/akkumar/hbasene|HBasene]] -- Lucene+HBase - Using HBase as the backing store for the TF-IDF representations needed by Lucene. Also, contains a library for constructing lucene indices from HBase schema. * [[http://github.com/larsgeorge/jmxtoolkit|JMXToolkit]] -- A HBase tailored JMX toolkit enabling monitoring with Cacti and checking with Nagios or similar. + * [[http://github.com/ykulbak/ihbase|IHBASE]] -- Provides faster scans by indexing regions, each region has it's own index. The indexed columns are user-defined and indexes can be intersected or joined in a single query. Example HBase Applications
svn commit: r936148 - in /hadoop/common/branches/branch-0.20: ./ ivy/ src/hdfs/org/apache/hadoop/hdfs/server/namenode/ src/test/org/apache/hadoop/hdfs/ src/test/org/apache/hadoop/hdfs/server/namenode/
Author: shv
Date: Wed Apr 21 03:05:45 2010
New Revision: 936148
URL: http://svn.apache.org/viewvc?rev=936148&view=rev
Log:
HDFS-909. Merge -r 936130:936131 from trunk to branch-0.20.
Added:
hadoop/common/branches/branch-0.20/src/test/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java
(with props)
Modified:
hadoop/common/branches/branch-0.20/CHANGES.txt
hadoop/common/branches/branch-0.20/ivy/libraries.properties
hadoop/common/branches/branch-0.20/src/hdfs/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
hadoop/common/branches/branch-0.20/src/hdfs/org/apache/hadoop/hdfs/server/namenode/FSImage.java
hadoop/common/branches/branch-0.20/src/hdfs/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
hadoop/common/branches/branch-0.20/src/test/org/apache/hadoop/hdfs/MiniDFSCluster.java
Modified: hadoop/common/branches/branch-0.20/CHANGES.txt
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20/CHANGES.txt?rev=936148&r1=936147&r2=936148&view=diff
==
--- hadoop/common/branches/branch-0.20/CHANGES.txt (original)
+++ hadoop/common/branches/branch-0.20/CHANGES.txt Wed Apr 21 03:05:45 2010
@@ -23,6 +23,9 @@ Release 0.20.3 - Unreleased
HDFS-1041. DFSClient.getFileChecksum(..) should retry if connection to
the first datanode fails. (szetszwo)
+HDFS-909. Wait until edits syncing is finishes before purging edits.
+(Todd Lipcon via shv)
+
IMPROVEMENTS
MAPREDUCE-1407. Update javadoc in mapreduce.{Mapper,Reducer} to match
Modified: hadoop/common/branches/branch-0.20/ivy/libraries.properties
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20/ivy/libraries.properties?rev=936148&r1=936147&r2=936148&view=diff
==
--- hadoop/common/branches/branch-0.20/ivy/libraries.properties (original)
+++ hadoop/common/branches/branch-0.20/ivy/libraries.properties Wed Apr 21
03:05:45 2010
@@ -58,7 +58,7 @@ kfs.version=0.1
log4j.version=1.2.15
lucene-core.version=2.3.1
-mockito-all.version=1.8.0
+mockito-all.version=1.8.2
oro.version=2.0.8
Modified:
hadoop/common/branches/branch-0.20/src/hdfs/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20/src/hdfs/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java?rev=936148&r1=936147&r2=936148&view=diff
==
---
hadoop/common/branches/branch-0.20/src/hdfs/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
(original)
+++
hadoop/common/branches/branch-0.20/src/hdfs/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
Wed Apr 21 03:05:45 2010
@@ -83,7 +83,7 @@ public class FSEditLog {
private long lastPrintTime;
// is a sync currently running?
- private boolean isSyncRunning;
+ private volatile boolean isSyncRunning;
// these are statistics counters.
private long numTransactions;// number of transactions
@@ -308,6 +308,13 @@ public class FSEditLog {
return editStreams == null ? 0 : editStreams.size();
}
+ /**
+ * Return the current edit streams. This is for use from tests only!
+ */
+ ArrayList getEditStreams() {
+return editStreams;
+ }
+
boolean isOpen() {
return getNumEditStreams() > 0;
}
@@ -338,6 +345,7 @@ public class FSEditLog {
}
public synchronized void createEditLogFile(File name) throws IOException {
+waitForSyncToFinish();
EditLogOutputStream eStream = new EditLogFileOutputStream(name);
eStream.create();
eStream.close();
@@ -347,12 +355,7 @@ public class FSEditLog {
* Shutdown the file store.
*/
public synchronized void close() throws IOException {
-while (isSyncRunning) {
- try {
-wait(1000);
- } catch (InterruptedException ie) {
- }
-}
+waitForSyncToFinish();
if (editStreams == null) {
return;
}
@@ -851,9 +854,60 @@ public class FSEditLog {
metrics.transactions.inc((end-start));
}
- //
- // Sync all modifications done by this thread.
- //
+ /**
+ * Blocks until all ongoing edits have been synced to disk.
+ * This differs from logSync in that it waits for edits that have been
+ * written by other threads, not just edits from the calling thread.
+ *
+ * NOTE: this should be done while holding the FSNamesystem lock, or
+ * else more operations can start writing while this is in progress.
+ */
+ void logSyncAll() throws IOException {
+// Record the most recent transaction ID as our own id
+synchronized (this) {
+ TransactionId id = myTransactionId.get();
+ id.txid = txid;
+}
+// Then make sure we're synced up to this point
+logSync();
+ }
+
+ synchronized void waitForSyncToFinish() {
+while (isSyncRunning) {
+ try {
+wait(1000);
+ }
