[jira] [Commented] (HBASE-5286) bin/hbase's logic of adding Hadoop jar files to the classpath is fragile when presented with split packaged Hadoop 0.23 installation
[
https://issues.apache.org/jira/browse/HBASE-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13212229#comment-13212229
]
Roman Shaposhnik commented on HBASE-5286:
-
@Todd,
I see 2 reasons to potentially prefer the full-blown execution script:
# it is guaranteed to work (while hadoop classpath could, potentially, have
bugs)
# it takes care of niatve libraries setting (and anything else for that
matter that might not be covered by the CP)
Of course, the last point is way stronger than the first one.
That said, I think making HBase script use hadoop classpath instead of the
current logic definitely qualifies as an incremental improvement. I'll submit
the patch tomorrow and we can worry about native libraries, etc later.
bin/hbase's logic of adding Hadoop jar files to the classpath is fragile when
presented with split packaged Hadoop 0.23 installation
Key: HBASE-5286
URL: https://issues.apache.org/jira/browse/HBASE-5286
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.92.0
Reporter: Roman Shaposhnik
Assignee: Roman Shaposhnik
Here's the bit from bin/hbase that might need TLC now that Hadoop can be
spotted in the wild in split-package configuration:
{noformat}
#If avail, add Hadoop to the CLASSPATH and to the JAVA_LIBRARY_PATH
if [ ! -z $HADOOP_HOME ]; then
HADOOPCPPATH=
if [ -z $HADOOP_CONF_DIR ]; then
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} ${HADOOP_HOME}/conf)
else
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} ${HADOOP_CONF_DIR})
fi
if [ `echo ${HADOOP_HOME}/hadoop-core*.jar` !=
${HADOOP_HOME}/hadoop-core*.jar ] ; then
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} `ls
${HADOOP_HOME}/hadoop-core*.jar | head -1`)
else
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} `ls
${HADOOP_HOME}/hadoop-common*.jar | head -1`)
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} `ls
${HADOOP_HOME}/hadoop-hdfs*.jar | head -1`)
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} `ls
${HADOOP_HOME}/hadoop-mapred*.jar | head -1`)
fi
{noformat}
There's a couple of issues with the above code:
0. HADOOP_HOME is now deprecated in Hadoop 0.23
1. the list of jar files added to the class-path should be revised
2. we need to figure out a more robust way to get the jar files that are
needed to the classpath (things like hadoop-mapred*.jar tend to match
src/test jars as well)
Better yet, it would be useful to look into whether we can transition HBase's
bin/hbase onto using bin/hadoop as a launcher script instead of direct JAVA
invocations (Pig, Hive, Sqoop and Mahout already do that)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5221) bin/hbase script doesn't look for Hadoop jars in the right place in trunk layout
[ https://issues.apache.org/jira/browse/HBASE-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13203784#comment-13203784 ] Roman Shaposhnik commented on HBASE-5221: - Guys, this fix will break certain HADOOP 0.23 layout. The correct fix for this one will have to come out of HBASE-5286 It would be really nice if folks can chime in on the JIRA that I quoted. Also, I don't quite understand why this is a problem. For developer's builds HBase bundles everything under lib and thus has no reason to look under HADOOP installation tree. For pacakged deployments (Bigtop, CDH) the layout of Hadoop is different anyway, which means this fix will actually break things. bin/hbase script doesn't look for Hadoop jars in the right place in trunk layout Key: HBASE-5221 URL: https://issues.apache.org/jira/browse/HBASE-5221 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Jimmy Xiang Fix For: 0.94.0 Attachments: hbase-5221.txt Running against an 0.24.0-SNAPSHOT hadoop: ls: cannot access /home/todd/ha-demo/hadoop-0.24.0-SNAPSHOT/hadoop-common*.jar: No such file or directory ls: cannot access /home/todd/ha-demo/hadoop-0.24.0-SNAPSHOT/hadoop-hdfs*.jar: No such file or directory ls: cannot access /home/todd/ha-demo/hadoop-0.24.0-SNAPSHOT/hadoop-mapred*.jar: No such file or directory The jars are rooted deeper in the heirarchy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5221) bin/hbase script doesn't look for Hadoop jars in the right place in trunk layout
[ https://issues.apache.org/jira/browse/HBASE-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13204292#comment-13204292 ] Roman Shaposhnik commented on HBASE-5221: - @stack, NP -- thanks for taking care of it on your end so quickly. I'll try to tackle HBASE-5286 tomorrow. @Jimmy, for the developer build you always have an option of unsetting the HADOOP_HOME variable and the ClassNotFound will go away. In fact, I think, for dev. builds it is desirable to unset HADOOP_HOME, since that way you can be sure that you're testing your HBase build with whatever dependencies are declared in the POM. bin/hbase script doesn't look for Hadoop jars in the right place in trunk layout Key: HBASE-5221 URL: https://issues.apache.org/jira/browse/HBASE-5221 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Jimmy Xiang Fix For: 0.94.0 Attachments: hbase-5221.txt Running against an 0.24.0-SNAPSHOT hadoop: ls: cannot access /home/todd/ha-demo/hadoop-0.24.0-SNAPSHOT/hadoop-common*.jar: No such file or directory ls: cannot access /home/todd/ha-demo/hadoop-0.24.0-SNAPSHOT/hadoop-hdfs*.jar: No such file or directory ls: cannot access /home/todd/ha-demo/hadoop-0.24.0-SNAPSHOT/hadoop-mapred*.jar: No such file or directory The jars are rooted deeper in the heirarchy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5286) bin/hbase's logic of adding Hadoop jar files to the classpath is fragile when presented with split packaged Hadoop 0.23 installation
[
https://issues.apache.org/jira/browse/HBASE-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198025#comment-13198025
]
Roman Shaposhnik commented on HBASE-5286:
-
The basic challenge the way I see it is to have bin/hbase capable of supporting
standalone mode AND distribution mode at the same time.
HBase needs to keep bundling its dependencies so that one can deploy it
straight from binary tarballs. At the same time
we have to make sure that we can point at the ZK/Hadoop dependencies available
elsewhere.
Talking with Todd off-line here's what emerged:
We could change around the hbase assembly so that
hadoop/ZK (and its transitive deps) get pushed into a
lib/[hadoop|zookeeper]-dependencies directory.
Then in the wrapper script, if HADOOP_HOME isn't defined
(or detectable at usual spots) then we add
lib/[hadoop|zookeeper]-dependencies/* to the classpath?
Otherwise we use the one set in the env?
Thoughts?
bin/hbase's logic of adding Hadoop jar files to the classpath is fragile when
presented with split packaged Hadoop 0.23 installation
Key: HBASE-5286
URL: https://issues.apache.org/jira/browse/HBASE-5286
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.92.0
Reporter: Roman Shaposhnik
Assignee: Roman Shaposhnik
Here's the bit from bin/hbase that might need TLC now that Hadoop can be
spotted in the wild in split-package configuration:
{noformat}
#If avail, add Hadoop to the CLASSPATH and to the JAVA_LIBRARY_PATH
if [ ! -z $HADOOP_HOME ]; then
HADOOPCPPATH=
if [ -z $HADOOP_CONF_DIR ]; then
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} ${HADOOP_HOME}/conf)
else
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} ${HADOOP_CONF_DIR})
fi
if [ `echo ${HADOOP_HOME}/hadoop-core*.jar` !=
${HADOOP_HOME}/hadoop-core*.jar ] ; then
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} `ls
${HADOOP_HOME}/hadoop-core*.jar | head -1`)
else
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} `ls
${HADOOP_HOME}/hadoop-common*.jar | head -1`)
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} `ls
${HADOOP_HOME}/hadoop-hdfs*.jar | head -1`)
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} `ls
${HADOOP_HOME}/hadoop-mapred*.jar | head -1`)
fi
{noformat}
There's a couple of issues with the above code:
0. HADOOP_HOME is now deprecated in Hadoop 0.23
1. the list of jar files added to the class-path should be revised
2. we need to figure out a more robust way to get the jar files that are
needed to the classpath (things like hadoop-mapred*.jar tend to match
src/test jars as well)
Better yet, it would be useful to look into whether we can transition HBase's
bin/hbase onto using bin/hadoop as a launcher script instead of direct JAVA
invocations (Pig, Hive, Sqoop and Mahout already do that)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5286) bin/hbase's logic of adding Hadoop jar files to the classpath is fragile when presented with split packaged Hadoop 0.23 installation
[
https://issues.apache.org/jira/browse/HBASE-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198102#comment-13198102
]
Roman Shaposhnik commented on HBASE-5286:
-
Frankly, in .23 there's no clarity yet. Potentially you can have as many as 4
different places to get jar files from HADOOP_COMMON_HOME, HADOOP_HDFS_HOME,
YARN_HOME, HADOOP_MAPRED_HOME.
And yes, sed-logic is possible, but I think a clean-separation is more
desirable since otherwise we'd always be at the mercy of Hadoop/ZK naming, etc.
bin/hbase's logic of adding Hadoop jar files to the classpath is fragile when
presented with split packaged Hadoop 0.23 installation
Key: HBASE-5286
URL: https://issues.apache.org/jira/browse/HBASE-5286
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.92.0
Reporter: Roman Shaposhnik
Assignee: Roman Shaposhnik
Here's the bit from bin/hbase that might need TLC now that Hadoop can be
spotted in the wild in split-package configuration:
{noformat}
#If avail, add Hadoop to the CLASSPATH and to the JAVA_LIBRARY_PATH
if [ ! -z $HADOOP_HOME ]; then
HADOOPCPPATH=
if [ -z $HADOOP_CONF_DIR ]; then
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} ${HADOOP_HOME}/conf)
else
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} ${HADOOP_CONF_DIR})
fi
if [ `echo ${HADOOP_HOME}/hadoop-core*.jar` !=
${HADOOP_HOME}/hadoop-core*.jar ] ; then
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} `ls
${HADOOP_HOME}/hadoop-core*.jar | head -1`)
else
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} `ls
${HADOOP_HOME}/hadoop-common*.jar | head -1`)
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} `ls
${HADOOP_HOME}/hadoop-hdfs*.jar | head -1`)
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} `ls
${HADOOP_HOME}/hadoop-mapred*.jar | head -1`)
fi
{noformat}
There's a couple of issues with the above code:
0. HADOOP_HOME is now deprecated in Hadoop 0.23
1. the list of jar files added to the class-path should be revised
2. we need to figure out a more robust way to get the jar files that are
needed to the classpath (things like hadoop-mapred*.jar tend to match
src/test jars as well)
Better yet, it would be useful to look into whether we can transition HBase's
bin/hbase onto using bin/hadoop as a launcher script instead of direct JAVA
invocations (Pig, Hive, Sqoop and Mahout already do that)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5286) bin/hbase's logic of adding Hadoop jar files to the classpath is fragile when presented with split packaged Hadoop 0.23 installation
[
https://issues.apache.org/jira/browse/HBASE-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198285#comment-13198285
]
Roman Shaposhnik commented on HBASE-5286:
-
@stack, sed-magic is certainly possible, but it will be quite difficult to
support all the layouts of hadoop in a single hbase script and account for all
the naming conventions
How about if we use hadoop binary to execute hbase (when we can find it) and
fall back on standalone execution when we can't. Sort of what Pig does over
here (line #234):
http://svn.apache.org/viewvc/pig/trunk/bin/pig?revision=1228547view=markup
bin/hbase's logic of adding Hadoop jar files to the classpath is fragile when
presented with split packaged Hadoop 0.23 installation
Key: HBASE-5286
URL: https://issues.apache.org/jira/browse/HBASE-5286
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.92.0
Reporter: Roman Shaposhnik
Assignee: Roman Shaposhnik
Here's the bit from bin/hbase that might need TLC now that Hadoop can be
spotted in the wild in split-package configuration:
{noformat}
#If avail, add Hadoop to the CLASSPATH and to the JAVA_LIBRARY_PATH
if [ ! -z $HADOOP_HOME ]; then
HADOOPCPPATH=
if [ -z $HADOOP_CONF_DIR ]; then
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} ${HADOOP_HOME}/conf)
else
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} ${HADOOP_CONF_DIR})
fi
if [ `echo ${HADOOP_HOME}/hadoop-core*.jar` !=
${HADOOP_HOME}/hadoop-core*.jar ] ; then
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} `ls
${HADOOP_HOME}/hadoop-core*.jar | head -1`)
else
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} `ls
${HADOOP_HOME}/hadoop-common*.jar | head -1`)
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} `ls
${HADOOP_HOME}/hadoop-hdfs*.jar | head -1`)
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} `ls
${HADOOP_HOME}/hadoop-mapred*.jar | head -1`)
fi
{noformat}
There's a couple of issues with the above code:
0. HADOOP_HOME is now deprecated in Hadoop 0.23
1. the list of jar files added to the class-path should be revised
2. we need to figure out a more robust way to get the jar files that are
needed to the classpath (things like hadoop-mapred*.jar tend to match
src/test jars as well)
Better yet, it would be useful to look into whether we can transition HBase's
bin/hbase onto using bin/hadoop as a launcher script instead of direct JAVA
invocations (Pig, Hive, Sqoop and Mahout already do that)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5284) TestLogRolling.java doesn't compile against the latest 0.23.1-SNAPSHOT
[
https://issues.apache.org/jira/browse/HBASE-5284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13194035#comment-13194035
]
Roman Shaposhnik commented on HBASE-5284:
-
Perhaps, given that the tests in this class are predicated on HDFS-826 it might
make sense to disable it for 0.23 profile.
TestLogRolling.java doesn't compile against the latest 0.23.1-SNAPSHOT
--
Key: HBASE-5284
URL: https://issues.apache.org/jira/browse/HBASE-5284
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.92.0
Reporter: Roman Shaposhnik
Here's how to reproduce:
{noformat}
$ mvn clean -DskipTests -Dhadoop.profile=23 -Dinstall site assembly:assembly
-Dmaven.repo.local=/home/rvs/.m2/repository
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile
(default-testCompile) on project hbase: Compilation failure
[ERROR]
/home/rvs/src/bigtop/output/hbase/hbase-0.92.0/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java:[341,33]
cannot find symbol
[ERROR] symbol : variable dnRegistration
[ERROR] location: class org.apache.hadoop.hdfs.server.datanode.DataNode
[ERROR] - [Help 1]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5284) TestLogRolling.java doesn't compile against the latest 0.23.1-SNAPSHOT
[
https://issues.apache.org/jira/browse/HBASE-5284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13194083#comment-13194083
]
Roman Shaposhnik commented on HBASE-5284:
-
FWIW: I believe this is also related to HDFS-1670
TestLogRolling.java doesn't compile against the latest 0.23.1-SNAPSHOT
--
Key: HBASE-5284
URL: https://issues.apache.org/jira/browse/HBASE-5284
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.92.0
Reporter: Roman Shaposhnik
Here's how to reproduce:
{noformat}
$ mvn clean -DskipTests -Dhadoop.profile=23 -Dinstall site assembly:assembly
-Dmaven.repo.local=/home/rvs/.m2/repository
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile
(default-testCompile) on project hbase: Compilation failure
[ERROR]
/home/rvs/src/bigtop/output/hbase/hbase-0.92.0/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java:[341,33]
cannot find symbol
[ERROR] symbol : variable dnRegistration
[ERROR] location: class org.apache.hadoop.hdfs.server.datanode.DataNode
[ERROR] - [Help 1]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4954) IllegalArgumentException in hfile2 blockseek
[ https://issues.apache.org/jira/browse/HBASE-4954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163344#comment-13163344 ] Roman Shaposhnik commented on HBASE-4954: - Tedd, Stack, it seems that the problem is, indeed with 0.22 snapshot. I made sure to run the tests with a manually constructed classpath pointing to the release artifacts of Hadoop 0.22 (I built them myself from Konst's RC) and everything passes. IllegalArgumentException in hfile2 blockseek Key: HBASE-4954 URL: https://issues.apache.org/jira/browse/HBASE-4954 Project: HBase Issue Type: Bug Reporter: stack Priority: Critical Fix For: 0.92.0 On Tue, Nov 29, 2011 at 10:20 PM, Stack st...@duboce.net wrote: The first hbase 0.92.0 release candidate is available for download: http://people.apache.org/~stack/hbase-0.92.0-candidate-0/ Here's another persistent issues that I'd appreciate somebody taking a quick look at: http://bigtop01.cloudera.org:8080/view/Hadoop%200.22/job/Bigtop-hadoop22-smoketest/28/testReport/org.apache.bigtop.itest.hbase.smoke/TestHFileOutputFormat/testMRIncrementalLoadWithSplit/ Caused by: java.lang.IllegalArgumentException at java.nio.Buffer.position(Buffer.java:218) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.blockSeek(HFileReaderV2.java:632) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.loadBlockAndSeekToKey(HFileReaderV2.java:545) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:503) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:511) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:475) at org.apache.hadoop.hbase.io.HalfStoreFileReader$1.seekTo(HalfStoreFileReader.java:157) at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.copyHFileHalf(LoadIncrementalHFiles.java:544) at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:516) at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:377) at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:441) at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:325) at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:323) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4920) We need a mascot, a totem
[ https://issues.apache.org/jira/browse/HBASE-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160499#comment-13160499 ] Roman Shaposhnik commented on HBASE-4920: - Orca is fine, but a bad ass honey badger now owns my heart as far as mascots for OS projects are concerned :-) http://www.youtube.com/watch?v=4r7wHMg5Yjg We need a mascot, a totem - Key: HBASE-4920 URL: https://issues.apache.org/jira/browse/HBASE-4920 Project: HBase Issue Type: Task Reporter: stack Attachments: Orca_479990801.jpg, Screen shot 2011-11-30 at 4.06.17 PM.png We need a totem for our t-shirt that is yet to be printed. O'Reilly owns the Clyesdale. We need something else. We could have a fluffy little duck that quacks 'hbase!' when you squeeze it and we could order boxes of them from some off-shore sweatshop that subcontracts to a contractor who employs child labor only. Or we could have an Orca (Big!, Fast!, Killer!, and in a poem that Marcy from Salesforce showed me, that was a bit too spiritual for me to be seen quoting here, it had the Orca as the 'Guardian of the Cosmic Memory': i.e. in translation, bigdata). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4821) A fully automated comprehensive distributed integration test for HBase
[ https://issues.apache.org/jira/browse/HBASE-4821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153223#comment-13153223 ] Roman Shaposhnik commented on HBASE-4821: - Bigtop (http://incubator.apache.org/projects/bigtop.html) aims at providing a platform for the creations of exactly the kind of tests you're talking about *across* all the projects of a Hadoop stack. Granted, we're a young project and our test management framework is nowhere near the scope and quality of something like TestNG, but I think it will be helpful to invest in improving it. To give you a quick idea of the baseline architecture, here's what Bigtop testing framework assumes: 1. all tests are implemented as code running on top of JVM. We don't care what language it is (Java, Groovy, Clojure, etc) as long as at the end of the day there's a bunch of class files generated. 2. all tests are packaged/versioned as Maven artifacts 3. all test data is packaged/versioned as Maven artifacts 4. the entry point into test execution is via Junit/TestNG-style 5. tests NEVER concern themselves with deployment (we've got Puppet for that) 6. tests NEVER concern themselves with configuration (we've got Puppet for that) If you think Bigtop can serve as a reasonable platform for what you're trying to accomplish, lets continue this discussion over at bigtop-dev@incubator (and Bigtop JIRAs). A fully automated comprehensive distributed integration test for HBase -- Key: HBASE-4821 URL: https://issues.apache.org/jira/browse/HBASE-4821 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin To properly verify that a particular version of HBase is good for production deployment we need a better way to do real cluster testing after incremental changes. Running unit tests is good, but we also need to deploy HBase to a cluster, run integration tests, load tests, Thrift server tests, kill some region servers, kill the master, and produce a report. All of this needs to happen in 20-30 minutes with minimal manual intervention. I think this way we can combine agile development with high stability of the codebase. I am envisioning a high-level framework written in a scripting language (e.g. Python) that would abstract external operations such as deploy to test cluster, kill a particular server, run load test A, run load test B (we already have a few kinds of load tests implemented in Java, and we could write a Thrift load test in Python). This tool should also produce intermediate output, allowing to catch problems early and restart the test. No implementation has yet been done. Any ideas or suggestions are welcome. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4526) special case for stopping master in hbase-daemon.sh is no longer required
[
https://issues.apache.org/jira/browse/HBASE-4526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118938#comment-13118938
]
Roman Shaposhnik commented on HBASE-4526:
-
Actually, the only thing that my patch changes is what signal gets used to stop
master. Currently it is SIGKILL which is very harsh. And after my change it'll
be SIGTERM which gives JVM a chance to shutdown. No other changes in behavior.
More to the point -- I tried this in a fully distributed case (master and
regionserver running on different nodes) and it seems that I can stop the
master without affecting the regionserver.
special case for stopping master in hbase-daemon.sh is no longer required
-
Key: HBASE-4526
URL: https://issues.apache.org/jira/browse/HBASE-4526
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.90.4
Reporter: Roman Shaposhnik
Assignee: Roman Shaposhnik
Priority: Minor
Attachments: HBASE-4526.patch.txt
Now that HBASE-4209 is finally done (many thanks to stack for help and
encouragement!) I don't think there's any reason to keep this bit of code in
hbase-daemon.sh:
{noformat}
if [ $command = master ]; then
echo `date` Killing $command $loglog
kill -9 `cat $pid` /dev/null 21
else
echo `date` Killing $command $loglog
kill `cat $pid` /dev/null 21
fi
{noformat}
I suggest we remove the special case completely, since I don't think it
serves any useful purpose (patch attached).
Now, as an additional precautionary measure. We can try to follow up with a
SIGKILL once a certain timeout expires.
Let me know if you think it is necessary to do so and I'll update the patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
