[jira] [Updated] (HBASE-7495) parallel scanner seek in StoreScanner's constructor
[
https://issues.apache.org/jira/browse/HBASE-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liang xie updated HBASE-7495:
-
Attachment: HBASE-7495.txt
parallel scanner seek in StoreScanner's constructor
---
Key: HBASE-7495
URL: https://issues.apache.org/jira/browse/HBASE-7495
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.94.3, 0.96.0
Reporter: liang xie
Assignee: liang xie
Attachments: HBASE-7495.txt, HBASE-7495.txt
seems there's a potential improvable space before doing scanner.next:
{code:title=StoreScanner.java|borderStyle=solid}
if (explicitColumnQuery lazySeekEnabledGlobally) {
for (KeyValueScanner scanner : scanners) {
scanner.requestSeek(matcher.getStartKey(), false, true);
}
} else {
for (KeyValueScanner scanner : scanners) {
scanner.seek(matcher.getStartKey());
}
}
{code}
we can do scanner.requestSeek or scanner.seek in parallel, instead of current
serialization, to reduce latency for special case.
Any ideas on it ? I'll have a try if the comments/suggestions are positive：）
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7495) parallel scanner seek in StoreScanner's constructor
[
https://issues.apache.org/jira/browse/HBASE-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liang xie updated HBASE-7495:
-
Status: Patch Available (was: Open)
parallel scanner seek in StoreScanner's constructor
---
Key: HBASE-7495
URL: https://issues.apache.org/jira/browse/HBASE-7495
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.94.3, 0.96.0
Reporter: liang xie
Assignee: liang xie
Attachments: HBASE-7495.txt, HBASE-7495.txt
seems there's a potential improvable space before doing scanner.next:
{code:title=StoreScanner.java|borderStyle=solid}
if (explicitColumnQuery lazySeekEnabledGlobally) {
for (KeyValueScanner scanner : scanners) {
scanner.requestSeek(matcher.getStartKey(), false, true);
}
} else {
for (KeyValueScanner scanner : scanners) {
scanner.seek(matcher.getStartKey());
}
}
{code}
we can do scanner.requestSeek or scanner.seek in parallel, instead of current
serialization, to reduce latency for special case.
Any ideas on it ? I'll have a try if the comments/suggestions are positive：）
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7495) parallel seek in StoreScanner
[
https://issues.apache.org/jira/browse/HBASE-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liang xie updated HBASE-7495:
-
Summary: parallel seek in StoreScanner (was: parallel scanner seek in
StoreScanner's constructor)
parallel seek in StoreScanner
-
Key: HBASE-7495
URL: https://issues.apache.org/jira/browse/HBASE-7495
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.94.3, 0.96.0
Reporter: liang xie
Assignee: liang xie
Attachments: HBASE-7495.txt, HBASE-7495.txt
seems there's a potential improvable space before doing scanner.next:
{code:title=StoreScanner.java|borderStyle=solid}
if (explicitColumnQuery lazySeekEnabledGlobally) {
for (KeyValueScanner scanner : scanners) {
scanner.requestSeek(matcher.getStartKey(), false, true);
}
} else {
for (KeyValueScanner scanner : scanners) {
scanner.seek(matcher.getStartKey());
}
}
{code}
we can do scanner.requestSeek or scanner.seek in parallel, instead of current
serialization, to reduce latency for special case.
Any ideas on it ? I'll have a try if the comments/suggestions are positive：）
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7495) parallel seek in StoreScanner
[
https://issues.apache.org/jira/browse/HBASE-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13561444#comment-13561444
]
liang xie commented on HBASE-7495:
--
IMHO, the order is maintained by the following:
{code}
heap = new KeyValueHeap(scanners, store.comparator);
{code}
parallel seek in StoreScanner
-
Key: HBASE-7495
URL: https://issues.apache.org/jira/browse/HBASE-7495
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.94.3, 0.96.0
Reporter: liang xie
Assignee: liang xie
Attachments: HBASE-7495.txt, HBASE-7495.txt
seems there's a potential improvable space before doing scanner.next:
{code:title=StoreScanner.java|borderStyle=solid}
if (explicitColumnQuery lazySeekEnabledGlobally) {
for (KeyValueScanner scanner : scanners) {
scanner.requestSeek(matcher.getStartKey(), false, true);
}
} else {
for (KeyValueScanner scanner : scanners) {
scanner.seek(matcher.getStartKey());
}
}
{code}
we can do scanner.requestSeek or scanner.seek in parallel, instead of current
serialization, to reduce latency for special case.
Any ideas on it ? I'll have a try if the comments/suggestions are positive：）
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7495) parallel seek in StoreScanner
[
https://issues.apache.org/jira/browse/HBASE-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13561475#comment-13561475
]
liang xie commented on HBASE-7495:
--
Really comfused to me, why does it break ACID testcase...
parallel seek in StoreScanner
-
Key: HBASE-7495
URL: https://issues.apache.org/jira/browse/HBASE-7495
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.94.3, 0.96.0
Reporter: liang xie
Assignee: liang xie
Attachments: HBASE-7495.txt, HBASE-7495.txt
seems there's a potential improvable space before doing scanner.next:
{code:title=StoreScanner.java|borderStyle=solid}
if (explicitColumnQuery lazySeekEnabledGlobally) {
for (KeyValueScanner scanner : scanners) {
scanner.requestSeek(matcher.getStartKey(), false, true);
}
} else {
for (KeyValueScanner scanner : scanners) {
scanner.seek(matcher.getStartKey());
}
}
{code}
we can do scanner.requestSeek or scanner.seek in parallel, instead of current
serialization, to reduce latency for special case.
Any ideas on it ? I'll have a try if the comments/suggestions are positive：）
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7495) parallel scanner seek in StoreScanner's constructor
[
https://issues.apache.org/jira/browse/HBASE-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13555951#comment-13555951
]
liang xie commented on HBASE-7495:
--
I just did a apple-to-apple comparison this morning, it shows the parallel seek
reduces latency in special scenario.
Attached is a prelim patch just for refer.
My test env : 10 dn/rs each with 12*2T SATA, hfile.block.cache.size=0,
hbase0.94.3, cdh4.1.1
My test data :
recordcount=10
fieldcount=3
fieldlength=200
hbase(main):002:0 describe 'YCSBTest'
DESCRIPTION ENABLED
{NAME = 'YCSBTest', SPLIT_POLICY =
'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy', FAMILIES
= [{NAME = 'te true
st', DATA_BLOCK_ENCODING = 'NONE', BLOOMFILTER = 'NONE', REPLICATION_SCOPE =
'1', VERSIONS = '3', COMPRESSION = 'NONE', MIN_VER
SIONS = '0', TTL = '2147483647', KEEP_DELETED_CELLS = 'false', BLOCKSIZE =
'65536', IN_MEMORY = 'false', ENCODE_ON_DISK = 'tru
e', BLOCKCACHE = 'true'}]}
$./hdfs dfs -du -s -h hdfs://lgxl-xieliang/
726.8g hdfs://lgxl-xieliang/
100 regions in total, and most of numberOfStorefiles in those regions are [0,5]
My test cmd: bin/ycsb run hbase -P ./workloads/kaka -threads 1 -p
columnfamily=test -p table=YCSBTest -s log/run.log 21
I restarted the whole hbase/hdfs cluster and clear OS cache(echo 1
/proc/sys/vm/drop_caches) before each run.
Serial seek result:
[OVERALL], RunTime(ms), 300027.0
[OVERALL], Throughput(ops/sec), 20.09819116279535
[READ], Operations, 6030
[READ], AverageLatency(us), 49739.97446102819
[READ], MinLatency(us), 2768
[READ], MaxLatency(us), 782892
[READ], 50thPercentileLatency(ms), 45
[READ], 95thPercentileLatency(ms), 90
[READ], 99thPercentileLatency(ms), 124
[READ], Return=0, 6030
Parallel seek result:
[OVERALL], RunTime(ms), 300016.0
[OVERALL], Throughput(ops/sec), 39.584555490373845
[READ], Operations, 11876
[READ], AverageLatency(us), 25249.878410239136
[READ], MinLatency(us), 3084
[READ], MaxLatency(us), 753547
[READ], 50thPercentileLatency(ms), 22
[READ], 95thPercentileLatency(ms), 43
[READ], 99thPercentileLatency(ms), 67
[READ], Return=0, 11876
parallel scanner seek in StoreScanner's constructor
---
Key: HBASE-7495
URL: https://issues.apache.org/jira/browse/HBASE-7495
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.94.3, 0.96.0
Reporter: liang xie
Assignee: liang xie
Attachments: HBASE-7495.txt
seems there's a potential improvable space before doing scanner.next:
{code:title=StoreScanner.java|borderStyle=solid}
if (explicitColumnQuery lazySeekEnabledGlobally) {
for (KeyValueScanner scanner : scanners) {
scanner.requestSeek(matcher.getStartKey(), false, true);
}
} else {
for (KeyValueScanner scanner : scanners) {
scanner.seek(matcher.getStartKey());
}
}
{code}
we can do scanner.requestSeek or scanner.seek in parallel, instead of current
serialization, to reduce latency for special case.
Any ideas on it ? I'll have a try if the comments/suggestions are positive：）
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7495) parallel scanner seek in StoreScanner's constructor
[
https://issues.apache.org/jira/browse/HBASE-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liang xie updated HBASE-7495:
-
Attachment: HBASE-7495.txt
parallel scanner seek in StoreScanner's constructor
---
Key: HBASE-7495
URL: https://issues.apache.org/jira/browse/HBASE-7495
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.94.3, 0.96.0
Reporter: liang xie
Assignee: liang xie
Attachments: HBASE-7495.txt
seems there's a potential improvable space before doing scanner.next:
{code:title=StoreScanner.java|borderStyle=solid}
if (explicitColumnQuery lazySeekEnabledGlobally) {
for (KeyValueScanner scanner : scanners) {
scanner.requestSeek(matcher.getStartKey(), false, true);
}
} else {
for (KeyValueScanner scanner : scanners) {
scanner.seek(matcher.getStartKey());
}
}
{code}
we can do scanner.requestSeek or scanner.seek in parallel, instead of current
serialization, to reduce latency for special case.
Any ideas on it ? I'll have a try if the comments/suggestions are positive：）
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7495) parallel scanner seek in StoreScanner's constructor
[
https://issues.apache.org/jira/browse/HBASE-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13555956#comment-13555956
]
liang xie commented on HBASE-7495:
--
[~lhofhansl], for get operation, the code should go to seek, not
requestSeek, please correct me if anything wrong
[~zjushch], IMHO, it's easier to see the improment if the test:1)larger test
dataset and smaller cache size 2)more numberOfStorefiles/disks 3)not too heavy
IO-bound(e.g. 100% io util)
the patch is broken some test cases, i'll look into it later
parallel scanner seek in StoreScanner's constructor
---
Key: HBASE-7495
URL: https://issues.apache.org/jira/browse/HBASE-7495
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.94.3, 0.96.0
Reporter: liang xie
Assignee: liang xie
Attachments: HBASE-7495.txt
seems there's a potential improvable space before doing scanner.next:
{code:title=StoreScanner.java|borderStyle=solid}
if (explicitColumnQuery lazySeekEnabledGlobally) {
for (KeyValueScanner scanner : scanners) {
scanner.requestSeek(matcher.getStartKey(), false, true);
}
} else {
for (KeyValueScanner scanner : scanners) {
scanner.seek(matcher.getStartKey());
}
}
{code}
we can do scanner.requestSeek or scanner.seek in parallel, instead of current
serialization, to reduce latency for special case.
Any ideas on it ? I'll have a try if the comments/suggestions are positive：）
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6031) RegionServer does not go down while aborting
[
https://issues.apache.org/jira/browse/HBASE-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545720#comment-13545720
]
liang xie commented on HBASE-6031:
--
we hit this issue today as well:)
Attached rs_shutdown_hung20130107.jstack was a thread dump during hung ocurred
I've filed HADOOP-9181 to resolve it
RegionServer does not go down while aborting
Key: HBASE-6031
URL: https://issues.apache.org/jira/browse/HBASE-6031
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0
Reporter: ramkrishna.s.vasudevan
Fix For: 0.96.0
Attachments: rs_shutdown_hung20130107.jstack, rsthread.txt
Following is the thread dump.
{code}
1997531088@qtp-716941846-5 prio=10 tid=0x7f7c5820c800 nid=0xe1b in
Object.wait() [0x7f7c56ae8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.mortbay.io.nio.SelectChannelEndPoint.blockWritable(SelectChannelEndPoint.java:279)
- locked 0x7f7cfe0616d0 (a
org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint)
at
org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:545)
at
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:639)
at
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:109)
- locked 0x7f7cfe74d758 (a
org.mortbay.util.ByteArrayOutputStream2)
at
org.mortbay.jetty.AbstractGenerator$OutputWriter.write(AbstractGenerator.java:904)
at java.io.Writer.write(Writer.java:96)
- locked 0x7f7cfca02fc0 (a
org.mortbay.jetty.HttpConnection$OutputWriter)
at java.io.PrintWriter.write(PrintWriter.java:361)
- locked 0x7f7cfca02fc0 (a
org.mortbay.jetty.HttpConnection$OutputWriter)
at org.jamon.escaping.HtmlEscaping.write(HtmlEscaping.java:43)
at
org.jamon.escaping.AbstractCharacterEscaping.write(AbstractCharacterEscaping.java:35)
at
org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmplImpl.renderNoFlush(RSStatusTmplImpl.java:222)
at
org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmpl.renderNoFlush(RSStatusTmpl.java:180)
at
org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmpl.render(RSStatusTmpl.java:171)
at
org.apache.hadoop.hbase.regionserver.RSStatusServlet.doGet(RSStatusServlet.java:48)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:932)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
1374615312@qtp-716941846-3 prio=10 tid=0x7f7c58214800 nid=0xc42 in
Object.wait() [0x7f7c55bd9000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.mortbay.io.nio.SelectChannelEndPoint.blockWritable(SelectChannelEndPoint.java:279)
[jira] [Updated] (HBASE-6031) RegionServer does not go down while aborting
[
https://issues.apache.org/jira/browse/HBASE-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liang xie updated HBASE-6031:
-
Attachment: rs_shutdown_hung20130107.jstack
RegionServer does not go down while aborting
Key: HBASE-6031
URL: https://issues.apache.org/jira/browse/HBASE-6031
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0
Reporter: ramkrishna.s.vasudevan
Fix For: 0.96.0
Attachments: rs_shutdown_hung20130107.jstack, rsthread.txt
Following is the thread dump.
{code}
1997531088@qtp-716941846-5 prio=10 tid=0x7f7c5820c800 nid=0xe1b in
Object.wait() [0x7f7c56ae8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.mortbay.io.nio.SelectChannelEndPoint.blockWritable(SelectChannelEndPoint.java:279)
- locked 0x7f7cfe0616d0 (a
org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint)
at
org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:545)
at
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:639)
at
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:109)
- locked 0x7f7cfe74d758 (a
org.mortbay.util.ByteArrayOutputStream2)
at
org.mortbay.jetty.AbstractGenerator$OutputWriter.write(AbstractGenerator.java:904)
at java.io.Writer.write(Writer.java:96)
- locked 0x7f7cfca02fc0 (a
org.mortbay.jetty.HttpConnection$OutputWriter)
at java.io.PrintWriter.write(PrintWriter.java:361)
- locked 0x7f7cfca02fc0 (a
org.mortbay.jetty.HttpConnection$OutputWriter)
at org.jamon.escaping.HtmlEscaping.write(HtmlEscaping.java:43)
at
org.jamon.escaping.AbstractCharacterEscaping.write(AbstractCharacterEscaping.java:35)
at
org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmplImpl.renderNoFlush(RSStatusTmplImpl.java:222)
at
org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmpl.renderNoFlush(RSStatusTmpl.java:180)
at
org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmpl.render(RSStatusTmpl.java:171)
at
org.apache.hadoop.hbase.regionserver.RSStatusServlet.doGet(RSStatusServlet.java:48)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:932)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
1374615312@qtp-716941846-3 prio=10 tid=0x7f7c58214800 nid=0xc42 in
Object.wait() [0x7f7c55bd9000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.mortbay.io.nio.SelectChannelEndPoint.blockWritable(SelectChannelEndPoint.java:279)
- locked 0x7f7cfdbb6cc8 (a
org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint)
at
[jira] [Commented] (HBASE-6031) RegionServer does not go down while aborting
[
https://issues.apache.org/jira/browse/HBASE-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545724#comment-13545724
]
liang xie commented on HBASE-6031:
--
Due to qtp-* was not a daemon thread, our hbase shutdownhook was not be
triggered
RegionServer does not go down while aborting
Key: HBASE-6031
URL: https://issues.apache.org/jira/browse/HBASE-6031
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0
Reporter: ramkrishna.s.vasudevan
Fix For: 0.96.0
Attachments: rs_shutdown_hung20130107.jstack, rsthread.txt
Following is the thread dump.
{code}
1997531088@qtp-716941846-5 prio=10 tid=0x7f7c5820c800 nid=0xe1b in
Object.wait() [0x7f7c56ae8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.mortbay.io.nio.SelectChannelEndPoint.blockWritable(SelectChannelEndPoint.java:279)
- locked 0x7f7cfe0616d0 (a
org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint)
at
org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:545)
at
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:639)
at
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:109)
- locked 0x7f7cfe74d758 (a
org.mortbay.util.ByteArrayOutputStream2)
at
org.mortbay.jetty.AbstractGenerator$OutputWriter.write(AbstractGenerator.java:904)
at java.io.Writer.write(Writer.java:96)
- locked 0x7f7cfca02fc0 (a
org.mortbay.jetty.HttpConnection$OutputWriter)
at java.io.PrintWriter.write(PrintWriter.java:361)
- locked 0x7f7cfca02fc0 (a
org.mortbay.jetty.HttpConnection$OutputWriter)
at org.jamon.escaping.HtmlEscaping.write(HtmlEscaping.java:43)
at
org.jamon.escaping.AbstractCharacterEscaping.write(AbstractCharacterEscaping.java:35)
at
org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmplImpl.renderNoFlush(RSStatusTmplImpl.java:222)
at
org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmpl.renderNoFlush(RSStatusTmpl.java:180)
at
org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmpl.render(RSStatusTmpl.java:171)
at
org.apache.hadoop.hbase.regionserver.RSStatusServlet.doGet(RSStatusServlet.java:48)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:932)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
1374615312@qtp-716941846-3 prio=10 tid=0x7f7c58214800 nid=0xc42 in
Object.wait() [0x7f7c55bd9000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.mortbay.io.nio.SelectChannelEndPoint.blockWritable(SelectChannelEndPoint.java:279)
- locked 0x7f7cfdbb6cc8 (a
[jira] [Commented] (HBASE-6031) RegionServer does not go down while aborting
[
https://issues.apache.org/jira/browse/HBASE-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546495#comment-13546495
]
liang xie commented on HBASE-6031:
--
we can close this issue now, since HADOOP-9181 had been resolved.
RegionServer does not go down while aborting
Key: HBASE-6031
URL: https://issues.apache.org/jira/browse/HBASE-6031
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0
Reporter: ramkrishna.s.vasudevan
Fix For: 0.96.0
Attachments: rs_shutdown_hung20130107.jstack, rsthread.txt
Following is the thread dump.
{code}
1997531088@qtp-716941846-5 prio=10 tid=0x7f7c5820c800 nid=0xe1b in
Object.wait() [0x7f7c56ae8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.mortbay.io.nio.SelectChannelEndPoint.blockWritable(SelectChannelEndPoint.java:279)
- locked 0x7f7cfe0616d0 (a
org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint)
at
org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:545)
at
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:639)
at
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:109)
- locked 0x7f7cfe74d758 (a
org.mortbay.util.ByteArrayOutputStream2)
at
org.mortbay.jetty.AbstractGenerator$OutputWriter.write(AbstractGenerator.java:904)
at java.io.Writer.write(Writer.java:96)
- locked 0x7f7cfca02fc0 (a
org.mortbay.jetty.HttpConnection$OutputWriter)
at java.io.PrintWriter.write(PrintWriter.java:361)
- locked 0x7f7cfca02fc0 (a
org.mortbay.jetty.HttpConnection$OutputWriter)
at org.jamon.escaping.HtmlEscaping.write(HtmlEscaping.java:43)
at
org.jamon.escaping.AbstractCharacterEscaping.write(AbstractCharacterEscaping.java:35)
at
org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmplImpl.renderNoFlush(RSStatusTmplImpl.java:222)
at
org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmpl.renderNoFlush(RSStatusTmpl.java:180)
at
org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmpl.render(RSStatusTmpl.java:171)
at
org.apache.hadoop.hbase.regionserver.RSStatusServlet.doGet(RSStatusServlet.java:48)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:932)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
1374615312@qtp-716941846-3 prio=10 tid=0x7f7c58214800 nid=0xc42 in
Object.wait() [0x7f7c55bd9000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.mortbay.io.nio.SelectChannelEndPoint.blockWritable(SelectChannelEndPoint.java:279)
- locked 0x7f7cfdbb6cc8 (a
[jira] [Created] (HBASE-7495) parallel scanner seek in StoreScanner's constructor
liang xie created HBASE-7495:
Summary: parallel scanner seek in StoreScanner's constructor
Key: HBASE-7495
URL: https://issues.apache.org/jira/browse/HBASE-7495
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.94.3, 0.96.0
Reporter: liang xie
Assignee: liang xie
seems there's a potential improvable space before doing scanner.next:
{code:title=StoreScanner.java|borderStyle=solid}
if (explicitColumnQuery lazySeekEnabledGlobally) {
for (KeyValueScanner scanner : scanners) {
scanner.requestSeek(matcher.getStartKey(), false, true);
}
} else {
for (KeyValueScanner scanner : scanners) {
scanner.seek(matcher.getStartKey());
}
}
{code}
we can do scanner.requestSeek or scanner.seek in parallel, instead of current
serialization, to reduce latency for special case.
Any ideas on it ? I'll have a try if the comments/suggestions are positive：）
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7495) parallel scanner seek in StoreScanner's constructor
[
https://issues.apache.org/jira/browse/HBASE-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13544589#comment-13544589
]
liang xie commented on HBASE-7495:
--
How about scanner.seek [~lhofhansl], we seek memstore and storefiles one by
one currently.
parallel scanner seek in StoreScanner's constructor
---
Key: HBASE-7495
URL: https://issues.apache.org/jira/browse/HBASE-7495
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.94.3, 0.96.0
Reporter: liang xie
Assignee: liang xie
seems there's a potential improvable space before doing scanner.next:
{code:title=StoreScanner.java|borderStyle=solid}
if (explicitColumnQuery lazySeekEnabledGlobally) {
for (KeyValueScanner scanner : scanners) {
scanner.requestSeek(matcher.getStartKey(), false, true);
}
} else {
for (KeyValueScanner scanner : scanners) {
scanner.seek(matcher.getStartKey());
}
}
{code}
we can do scanner.requestSeek or scanner.seek in parallel, instead of current
serialization, to reduce latency for special case.
Any ideas on it ? I'll have a try if the comments/suggestions are positive：）
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7495) parallel scanner seek in StoreScanner's constructor
[
https://issues.apache.org/jira/browse/HBASE-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13544598#comment-13544598
]
liang xie commented on HBASE-7495:
--
Oops, seems i have some misunderstanding on this code snippet, emm, i have to
debug it per your comments:)
parallel scanner seek in StoreScanner's constructor
---
Key: HBASE-7495
URL: https://issues.apache.org/jira/browse/HBASE-7495
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.94.3, 0.96.0
Reporter: liang xie
Assignee: liang xie
seems there's a potential improvable space before doing scanner.next:
{code:title=StoreScanner.java|borderStyle=solid}
if (explicitColumnQuery lazySeekEnabledGlobally) {
for (KeyValueScanner scanner : scanners) {
scanner.requestSeek(matcher.getStartKey(), false, true);
}
} else {
for (KeyValueScanner scanner : scanners) {
scanner.seek(matcher.getStartKey());
}
}
{code}
we can do scanner.requestSeek or scanner.seek in parallel, instead of current
serialization, to reduce latency for special case.
Any ideas on it ? I'll have a try if the comments/suggestions are positive：）
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7449) Refine UseMembar detail
liang xie created HBASE-7449: Summary: Refine UseMembar detail Key: HBASE-7449 URL: https://issues.apache.org/jira/browse/HBASE-7449 Project: HBase Issue Type: Improvement Components: documentation Affects Versions: 0.96.0 Environment: $ uname -a Linux lg-hadoop-control01.bj 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux $ java -version java version 1.6.0_37 Java(TM) SE Runtime Environment (build 1.6.0_37-b06) Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode) Reporter: liang xie Assignee: liang xie Priority: Trivial we encountered a hung issue, the thread dump shows: most of threads waiting to a lock but there isn't lock holder be found We know there were some discuss before, e.g: https://issues.apache.org/jira/browse/HBASE-3622 http://bugs.sun.com/view_bug.do?bug_id=6822370 http://cs.oswego.edu/pipermail/concurrency-interest/2012-August/009635.html https://blogs.oracle.com/dave/entry/a_race_in_locksupport_park My point is that the JVM bug is still not fixed(at least for latest 1.6.0_37, the hotspot version: 20.12-b01), it's not just a old JVM issue, let's update the trouble shooting document firstly if possible. I'm trying to reproduce with a debug openjdk version in-house. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7449) Refine UseMembar detail
[ https://issues.apache.org/jira/browse/HBASE-7449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7449: - Attachment: HBASE-7449.txt Refine UseMembar detail --- Key: HBASE-7449 URL: https://issues.apache.org/jira/browse/HBASE-7449 Project: HBase Issue Type: Improvement Components: documentation Affects Versions: 0.96.0 Environment: $ uname -a Linux lg-hadoop-control01.bj 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux $ java -version java version 1.6.0_37 Java(TM) SE Runtime Environment (build 1.6.0_37-b06) Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode) Reporter: liang xie Assignee: liang xie Priority: Trivial Attachments: HBASE-7449.txt we encountered a hung issue, the thread dump shows: most of threads waiting to a lock but there isn't lock holder be found We know there were some discuss before, e.g: https://issues.apache.org/jira/browse/HBASE-3622 http://bugs.sun.com/view_bug.do?bug_id=6822370 http://cs.oswego.edu/pipermail/concurrency-interest/2012-August/009635.html https://blogs.oracle.com/dave/entry/a_race_in_locksupport_park My point is that the JVM bug is still not fixed(at least for latest 1.6.0_37, the hotspot version: 20.12-b01), it's not just a old JVM issue, let's update the trouble shooting document firstly if possible. I'm trying to reproduce with a debug openjdk version in-house. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7336) HFileBlock.readAtOffset does not work well with multiple threads
[ https://issues.apache.org/jira/browse/HBASE-7336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13530612#comment-13530612 ] liang xie commented on HBASE-7336: -- +1, LGTM,a nice test case, Lars HFileBlock.readAtOffset does not work well with multiple threads Key: HBASE-7336 URL: https://issues.apache.org/jira/browse/HBASE-7336 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Critical Fix For: 0.96.0, 0.94.4 Attachments: 7336-0.94.txt, 7336-0.96.txt HBase grinds to a halt when many threads scan along the same set of blocks and neither read short circuit is nor block caching is enabled for the dfs client ... disabling the block cache makes sense on very large scans. It turns out that synchronizing in istream in HFileBlock.readAtOffset is the culprit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-1648) heap usage and limit reporting doesn't work as expected when using G1 GC
[
https://issues.apache.org/jira/browse/HBASE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508579#comment-13508579
]
liang xie commented on HBASE-1648:
--
[~saint@gmail.com] yes, it should be safe. we should not use G1 before
JDK7u4
And also, i tried in a hbase0.94.0 + jdk6u22 + G1GC test env(not a good choice,
aha):
hbase(main):003:0 status 'simple'
2 live servers
ha3:60020 1354522267193
requestsPerSecond=0, numberOfOnlineRegions=1, usedHeapMB=38,
maxHeapMB=200
ha2:60020 1354551060862
requestsPerSecond=0, numberOfOnlineRegions=6, usedHeapMB=88,
maxHeapMB=200
0 dead servers
Aggregate load: 0, regions: 7
the output is fine
And, i tried with jdk7u9 + G1GC as well:
hbase(main):002:0 status 'simple'
2 live servers
ha2:60020 1354552532489
requestsPerSecond=0, numberOfOnlineRegions=5, usedHeapMB=46,
maxHeapMB=200
ha3:60020 1354523738930
requestsPerSecond=0, numberOfOnlineRegions=2, usedHeapMB=44,
maxHeapMB=200
0 dead servers
Aggregate load: 0, regions: 7
still good
heap usage and limit reporting doesn't work as expected when using G1 GC
Key: HBASE-1648
URL: https://issues.apache.org/jira/browse/HBASE-1648
Project: HBase
Issue Type: Bug
Environment: Linux Centos x86_64 5.3, Sun JDK 1.6.0_14 (HotSpot
64-Bit Server VM build 14.0-b16)
Reporter: Andrew Purtell
Priority: Minor
Getting some early experience with the G1 GC.
Running with -Xmx1000m and HBASE_OPTS set to -XX:+UnlockExperimentalVMOpts
-XX:+UseG1GC. Regionserver heap use reports are not of much use:
{noformat}
hbase(main):001:0 status 'simple'
3 live servers
test3:60020 1247389283042
requests=0, regions=1, usedHeap=0, maxHeap=41
test2:60020 1247389219994
requests=0, regions=1, usedHeap=0, maxHeap=41
test4:60020 1247389324563
requests=0, regions=2, usedHeap=0, maxHeap=41
0 dead servers
{noformat}
top is about right:
{noformat}
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
23988 hadoop24 0 1492m 98m 10m S 0.0 2.5 0:02.65 java
{noformat}
Incidentally, don't try -XX:+UseG1GC and -XX:+DoEscapeAnalysis together or
the JVM will rapidly segfault.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2039) G1 GC issues
[ https://issues.apache.org/jira/browse/HBASE-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506374#comment-13506374 ] liang xie commented on HBASE-2039: -- en, it's not recommented to enable G1 before jdk7u4 we need to reevaluate this issue with jdk7u4+ again,IMHO G1 GC issues Key: HBASE-2039 URL: https://issues.apache.org/jira/browse/HBASE-2039 Project: HBase Issue Type: Sub-task Reporter: stack Lets keep an issue where we report on g1 issues. Lets keep list of crashes we see. I filed an issue up on bug parade. Lets see if it becomes actual bug. Below I note version of vm and the type of crash (Internal Error (nmethod.cpp:1981), pid=32319. Its a 'fatal error'). It happens for me after 5-10 minutes when a loading test. Same thing each time. G1 in 1.6 seems plain broke; crashes on use of stuff in concurrent utils package. Date Created: Thu Dec 10 13:33:04 MST 2009 .. Synopsis:Running G1 GC, crashes with Internal Error (nmethod.cpp:1981), pid=32319... Description: FULL PRODUCT VERSION : java version 1.7.0-ea Java(TM) SE Runtime Environment (build 1.7.0-ea-b77) Java HotSpot(TM) 64-Bit Server VM (build 17.0-b05, mixed mode) FULL OS VERSION : Fedora Core release 6 (Zod) EXTRA RELEVANT SYSTEM CONFIGURATION : Here are the JVM args: -XX:+HeapDumpOnOutOfMemoryError -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-1648) heap usage and limit reporting doesn't work as expected when using G1 GC
[
https://issues.apache.org/jira/browse/HBASE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506375#comment-13506375
]
liang xie commented on HBASE-1648:
--
emmm, it's not a hbase related bug, some jdk tools don't work well with G1 on
jdk6
heap usage and limit reporting doesn't work as expected when using G1 GC
Key: HBASE-1648
URL: https://issues.apache.org/jira/browse/HBASE-1648
Project: HBase
Issue Type: Bug
Environment: Linux Centos x86_64 5.3, Sun JDK 1.6.0_14 (HotSpot
64-Bit Server VM build 14.0-b16)
Reporter: Andrew Purtell
Priority: Minor
Getting some early experience with the G1 GC.
Running with -Xmx1000m and HBASE_OPTS set to -XX:+UnlockExperimentalVMOpts
-XX:+UseG1GC. Regionserver heap use reports are not of much use:
{noformat}
hbase(main):001:0 status 'simple'
3 live servers
test3:60020 1247389283042
requests=0, regions=1, usedHeap=0, maxHeap=41
test2:60020 1247389219994
requests=0, regions=1, usedHeap=0, maxHeap=41
test4:60020 1247389324563
requests=0, regions=2, usedHeap=0, maxHeap=41
0 dead servers
{noformat}
top is about right:
{noformat}
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
23988 hadoop24 0 1492m 98m 10m S 0.0 2.5 0:02.65 java
{noformat}
Incidentally, don't try -XX:+UseG1GC and -XX:+DoEscapeAnalysis together or
the JVM will rapidly segfault.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13503326#comment-13503326 ] liang xie commented on HBASE-5954: -- sound good for me. hope more guys have an eye on it Allow proper fsync support for HBase Key: HBASE-5954 URL: https://issues.apache.org/jira/browse/HBASE-5954 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Lars Hofhansl Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7184) pre-check table name in table.jsp
[ https://issues.apache.org/jira/browse/HBASE-7184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7184: - Status: Patch Available (was: Open) pre-check table name in table.jsp - Key: HBASE-7184 URL: https://issues.apache.org/jira/browse/HBASE-7184 Project: HBase Issue Type: Improvement Components: UI Affects Versions: 0.94.2, 0.96.0 Reporter: liang xie Assignee: liang xie Attachments: HBASE-7184.txt Currently the (table)name parameter in table.jsp isn't checked, it brings two problems at least: 1) 500 error for invalid value 2) directly written to JSP output, giving reflected XSS vulnerability we can do a parameter-checking -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7184) pre-check table name in table.jsp
[ https://issues.apache.org/jira/browse/HBASE-7184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7184: - Attachment: HBASE-7184.txt pre-check table name in table.jsp - Key: HBASE-7184 URL: https://issues.apache.org/jira/browse/HBASE-7184 Project: HBase Issue Type: Improvement Components: UI Affects Versions: 0.94.2, 0.96.0 Reporter: liang xie Assignee: liang xie Attachments: HBASE-7184.txt Currently the (table)name parameter in table.jsp isn't checked, it brings two problems at least: 1) 500 error for invalid value 2) directly written to JSP output, giving reflected XSS vulnerability we can do a parameter-checking -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7184) pre-check table name in table.jsp
liang xie created HBASE-7184: Summary: pre-check table name in table.jsp Key: HBASE-7184 URL: https://issues.apache.org/jira/browse/HBASE-7184 Project: HBase Issue Type: Improvement Components: UI Affects Versions: 0.94.2, 0.96.0 Reporter: liang xie Assignee: liang xie Attachments: HBASE-7184.txt Currently the (table)name parameter in table.jsp isn't checked, it brings two problems at least: 1) 500 error for invalid value 2) directly written to JSP output, giving reflected XSS vulnerability we can do a parameter-checking -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7184) pre-check table name in table.jsp
[ https://issues.apache.org/jira/browse/HBASE-7184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13500081#comment-13500081 ] liang xie commented on HBASE-7184: -- Before applied the attached patch, if i access host:port/table.jsp?name=... , will get : HTTP ERROR 500 Problem accessing /table.jsp. Reason: ... Caused by: org.apache.hadoop.hbase.TableNotFoundException: ... at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:999) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:864) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:133) at org.apache.hadoop.hbase.generated.master.table_jsp._jspService(table_jsp.java:67) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:98) at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1056) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Powered by Jetty:// After applied patch, i got : Illegal table name parameter! pre-check table name in table.jsp - Key: HBASE-7184 URL: https://issues.apache.org/jira/browse/HBASE-7184 Project: HBase Issue Type: Improvement Components: UI Affects Versions: 0.94.2, 0.96.0 Reporter: liang xie Assignee: liang xie Attachments: HBASE-7184.txt Currently the (table)name parameter in table.jsp isn't checked, it brings two problems at least: 1) 500 error for invalid value 2) directly written to JSP output, giving reflected XSS vulnerability we can do a parameter-checking -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7184) pre-check table name in table.jsp
[ https://issues.apache.org/jira/browse/HBASE-7184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13500112#comment-13500112 ] liang xie commented on HBASE-7184: -- emmm. seems the failure test case is unrelated with this issue, IMHO... pre-check table name in table.jsp - Key: HBASE-7184 URL: https://issues.apache.org/jira/browse/HBASE-7184 Project: HBase Issue Type: Improvement Components: UI Affects Versions: 0.94.2, 0.96.0 Reporter: liang xie Assignee: liang xie Attachments: HBASE-7184.txt Currently the (table)name parameter in table.jsp isn't checked, it brings two problems at least: 1) 500 error for invalid value 2) directly written to JSP output, giving reflected XSS vulnerability we can do a parameter-checking -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13493707#comment-13493707 ] liang xie commented on HBASE-5954: -- Hi [~lhofhansl],the HDFS-3979 has been committed, so maybe we can have a more clear target/fix version plan on HBASE-5954 now, right? Allow proper fsync support for HBase Key: HBASE-5954 URL: https://issues.apache.org/jira/browse/HBASE-5954 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Lars Hofhansl Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3925) Make Shell's -d and debug cmd behave the same
[
https://issues.apache.org/jira/browse/HBASE-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13485374#comment-13485374
]
liang xie commented on HBASE-3925:
--
[~saint@gmail.com] Before i applied the patch, we could see :
Make Shell's -d and debug cmd behave the same
-
Key: HBASE-3925
URL: https://issues.apache.org/jira/browse/HBASE-3925
Project: HBase
Issue Type: Improvement
Components: shell
Affects Versions: 0.90.3, 0.90.7, 0.92.2, 0.94.3, 0.96.0, 0.98.0
Reporter: Lars George
Assignee: liang xie
Priority: Trivial
Labels: patch
Attachments: HBASE-3925.patch, HBASE-3925-v2.txt
The -d option switches log4j to DEBUG and leaves the backtrace level at the
default. When using the supplied debug command we only switch the backtrace,
but I would think this also should set the log4j levels:
{noformat}
# Debugging method
def debug
if @shell.debug
@shell.debug = false
conf.back_trace_limit = 0
else
@shell.debug = true
conf.back_trace_limit = 100
end
debug?
end
{noformat}
could be something like
{noformat}
# Debugging method
def debug
if @shell.debug
@shell.debug = false
conf.back_trace_limit = 0
log_level = org.apache.log4j.Level::ERROR
else
@shell.debug = true
conf.back_trace_limit = 100
log_level = org.apache.log4j.Level::DEBUG
end
org.apache.log4j.Logger.getLogger(org.apache.zookeeper).setLevel(log_level)
org.apache.log4j.Logger.getLogger(org.apache.hadoop.hbase).setLevel(log_level)
debug?
end
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3925) Make Shell's -d and debug cmd behave the same
[
https://issues.apache.org/jira/browse/HBASE-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13485375#comment-13485375
]
liang xie commented on HBASE-3925:
--
[root@ha1 bin]# ./hbase shell -d
Setting DEBUG log level
..
hbase(main):001:0 debug?
Debug mode is OFF
And after applied the patch, it'll be consistence(the -d option will take
effect) :
[root@ha1 bin]# ./hbase shell -d
Setting DEBUG log level
..
hbase(main):001:0 debug?
Debug mode is ON
Make Shell's -d and debug cmd behave the same
-
Key: HBASE-3925
URL: https://issues.apache.org/jira/browse/HBASE-3925
Project: HBase
Issue Type: Improvement
Components: shell
Affects Versions: 0.90.3, 0.90.7, 0.92.2, 0.94.3, 0.96.0, 0.98.0
Reporter: Lars George
Assignee: liang xie
Priority: Trivial
Labels: patch
Attachments: HBASE-3925.patch, HBASE-3925-v2.txt
The -d option switches log4j to DEBUG and leaves the backtrace level at the
default. When using the supplied debug command we only switch the backtrace,
but I would think this also should set the log4j levels:
{noformat}
# Debugging method
def debug
if @shell.debug
@shell.debug = false
conf.back_trace_limit = 0
else
@shell.debug = true
conf.back_trace_limit = 100
end
debug?
end
{noformat}
could be something like
{noformat}
# Debugging method
def debug
if @shell.debug
@shell.debug = false
conf.back_trace_limit = 0
log_level = org.apache.log4j.Level::ERROR
else
@shell.debug = true
conf.back_trace_limit = 100
log_level = org.apache.log4j.Level::DEBUG
end
org.apache.log4j.Logger.getLogger(org.apache.zookeeper).setLevel(log_level)
org.apache.log4j.Logger.getLogger(org.apache.hadoop.hbase).setLevel(log_level)
debug?
end
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7008) Set scanner caching to a better default
[ https://issues.apache.org/jira/browse/HBASE-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13484773#comment-13484773 ] liang xie commented on HBASE-7008: -- +1 for #1, the default setting should be user-friendly for most of end-users or newbie:) we should not expect end-users tuning per doc always. Or in other words, if we leave all those things unchanged, maybe there'll be still many users complain why HBase performance so bad, or only the experts could finger the root cause out. Set scanner caching to a better default --- Key: HBASE-7008 URL: https://issues.apache.org/jira/browse/HBASE-7008 Project: HBase Issue Type: Bug Components: Client Reporter: liang xie Assignee: liang xie Fix For: 0.94.3, 0.96.0 Attachments: 7008-0.94.txt, 7008-0.94-v2.txt, 7008-v3.txt, 7008-v4.txt, HBASE-7008.patch, HBASE-7008-v2.patch per http://search-hadoop.com/m/qaRu9iM2f02/Set+scanner+caching+to+a+better+default%253Fsubj=Set+scanner+caching+to+a+better+default+ let's set to 100 by default -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6968) Several HBase write perf improvement
[
https://issues.apache.org/jira/browse/HBASE-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13482173#comment-13482173
]
liang xie commented on HBASE-6968:
--
I look through trunk code, there's no change needed, so let's set this affects
issue version on 0.90/0.92/0.94 only, right ?
Several HBase write perf improvement
Key: HBASE-6968
URL: https://issues.apache.org/jira/browse/HBASE-6968
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
Here are 2 hbase write performance improvements recently:
1) Avoid creating HBaseConfiguraiton object for each HLog. Every time when
creating a HBaseConfiguraiton object, it would parse the xml configuration
files from disk, which is not cheap operation.
In HLog.java:
orig:
{code:title=HLog.java}
newWriter = createWriter(fs, newPath, HBaseConfiguration.create(conf));
{code}
new:
{code}
newWriter = createWriter(fs, newPath, conf);
{code}
2) Change 2 hotspot synchronized functions into double locking pattern. So it
shall remove the synchronization overhead in the normal case.
orig:
{code:title=HBaseRpcMetrics.java}
public synchronized void inc(String name, int amt) {
MetricsTimeVaryingRate m = get(name);
if (m == null) {
m = create(name);
}
m.inc(amt);
}
{code}
new:
{code}
public void inc(String name, int amt) {
MetricsTimeVaryingRate m = get(name);
if (m == null) {
synchronized (this) {
if ((m = get(name)) == null) {
m = create(name);
}
}
}
m.inc(amt);
}
{code}
=
orig:
{code:title=MemStoreFlusher.java}
public synchronized void reclaimMemStoreMemory() {
if (this.server.getGlobalMemstoreSize().get() = globalMemStoreLimit) {
flushSomeRegions();
}
}
{code}
new:
{code}
public void reclaimMemStoreMemory() {
if (this.server.getGlobalMemstoreSize().get() = globalMemStoreLimit) {
flushSomeRegions();
}
}
private synchronized void flushSomeRegions() {
if (this.server.getGlobalMemstoreSize().get() globalMemStoreLimit) {
return; // double check the global memstore size inside of the
synchronized block.
}
...
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3925) Make Shell's -d and debug cmd behave the same
[
https://issues.apache.org/jira/browse/HBASE-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liang xie updated HBASE-3925:
-
Attachment: HBASE-3925-v2.txt
Attached patch addressed Ted's comment
Make Shell's -d and debug cmd behave the same
-
Key: HBASE-3925
URL: https://issues.apache.org/jira/browse/HBASE-3925
Project: HBase
Issue Type: Improvement
Components: shell
Affects Versions: 0.90.3, 0.90.7, 0.92.2, 0.94.3, 0.96.0, 0.98.0
Reporter: Lars George
Assignee: liang xie
Priority: Trivial
Labels: patch
Attachments: HBASE-3925.patch, HBASE-3925-v2.txt
The -d option switches log4j to DEBUG and leaves the backtrace level at the
default. When using the supplied debug command we only switch the backtrace,
but I would think this also should set the log4j levels:
{noformat}
# Debugging method
def debug
if @shell.debug
@shell.debug = false
conf.back_trace_limit = 0
else
@shell.debug = true
conf.back_trace_limit = 100
end
debug?
end
{noformat}
could be something like
{noformat}
# Debugging method
def debug
if @shell.debug
@shell.debug = false
conf.back_trace_limit = 0
log_level = org.apache.log4j.Level::ERROR
else
@shell.debug = true
conf.back_trace_limit = 100
log_level = org.apache.log4j.Level::DEBUG
end
org.apache.log4j.Logger.getLogger(org.apache.zookeeper).setLevel(log_level)
org.apache.log4j.Logger.getLogger(org.apache.hadoop.hbase).setLevel(log_level)
debug?
end
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7037) ReplicationPeer logs at WARN level aborting server instead of at FATAL
[ https://issues.apache.org/jira/browse/HBASE-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7037: - Attachment: HBASE-7037.txt just one line change:) ReplicationPeer logs at WARN level aborting server instead of at FATAL -- Key: HBASE-7037 URL: https://issues.apache.org/jira/browse/HBASE-7037 Project: HBase Issue Type: Bug Components: Replication Reporter: stack Labels: noob Attachments: HBASE-7037.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7037) ReplicationPeer logs at WARN level aborting server instead of at FATAL
[ https://issues.apache.org/jira/browse/HBASE-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7037: - Status: Patch Available (was: Open) ReplicationPeer logs at WARN level aborting server instead of at FATAL -- Key: HBASE-7037 URL: https://issues.apache.org/jira/browse/HBASE-7037 Project: HBase Issue Type: Bug Components: Replication Reporter: stack Labels: noob Attachments: HBASE-7037.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3925) Make Shell's -d and debug cmd behave the same
[
https://issues.apache.org/jira/browse/HBASE-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13482014#comment-13482014
]
liang xie commented on HBASE-3925:
--
Could anyone help to review it? thanks in advance
Make Shell's -d and debug cmd behave the same
-
Key: HBASE-3925
URL: https://issues.apache.org/jira/browse/HBASE-3925
Project: HBase
Issue Type: Improvement
Components: shell
Affects Versions: 0.90.3, 0.90.7, 0.92.2, 0.94.3, 0.96.0, 0.98.0
Reporter: Lars George
Assignee: liang xie
Priority: Trivial
Labels: patch
Attachments: HBASE-3925.patch
The -d option switches log4j to DEBUG and leaves the backtrace level at the
default. When using the supplied debug command we only switch the backtrace,
but I would think this also should set the log4j levels:
{noformat}
# Debugging method
def debug
if @shell.debug
@shell.debug = false
conf.back_trace_limit = 0
else
@shell.debug = true
conf.back_trace_limit = 100
end
debug?
end
{noformat}
could be something like
{noformat}
# Debugging method
def debug
if @shell.debug
@shell.debug = false
conf.back_trace_limit = 0
log_level = org.apache.log4j.Level::ERROR
else
@shell.debug = true
conf.back_trace_limit = 100
log_level = org.apache.log4j.Level::DEBUG
end
org.apache.log4j.Logger.getLogger(org.apache.zookeeper).setLevel(log_level)
org.apache.log4j.Logger.getLogger(org.apache.hadoop.hbase).setLevel(log_level)
debug?
end
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6994) minor doc update about DEFAULT_ACCEPTABLE_FACTOR
[ https://issues.apache.org/jira/browse/HBASE-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13482016#comment-13482016 ] liang xie commented on HBASE-6994: -- Could anyone help to review it? thanks in advance minor doc update about DEFAULT_ACCEPTABLE_FACTOR Key: HBASE-6994 URL: https://issues.apache.org/jira/browse/HBASE-6994 Project: HBase Issue Type: Bug Components: documentation Affects Versions: 0.96.0 Reporter: liang xie Assignee: liang xie Priority: Minor Attachments: HBASE-6994.patch Per trunk code, in LruBlockCache.java: static final float DEFAULT_ACCEPTABLE_FACTOR = 0.99f; but the site doc still : number of region servers * heap size * hfile.block.cache.size * 0.85 seems the HBASE-6312 forgot to update this doc:) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6955) Block read time should be in ms, not in ns
[ https://issues.apache.org/jira/browse/HBASE-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13482083#comment-13482083 ] liang xie commented on HBASE-6955: -- do we need to port this to trunk as well ? Revision: 1397821 in fb-89 branch has done this job Block read time should be in ms, not in ns -- Key: HBASE-6955 URL: https://issues.apache.org/jira/browse/HBASE-6955 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor Attachments: D5901.1.patch, D5901.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7001) Fix the RCN Correctness Warning in MemStoreFlusher class
[ https://issues.apache.org/jira/browse/HBASE-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13481122#comment-13481122 ] liang xie commented on HBASE-7001: -- Could any other experts have a review on this trivial issue? Thanks in advance:) Fix the RCN Correctness Warning in MemStoreFlusher class Key: HBASE-7001 URL: https://issues.apache.org/jira/browse/HBASE-7001 Project: HBase Issue Type: Bug Reporter: liang xie Assignee: liang xie Priority: Minor Attachments: HBASE-7001.patch https://builds.apache.org/job/PreCommit-HBASE-Build/3057//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html#Warnings_CORRECTNESS shows : Bug type RCN_REDUNDANT_NULLCHECK_WOULD_HAVE_BEEN_A_NPE (click for details) In class org.apache.hadoop.hbase.regionserver.MemStoreFlusher In method org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher$FlushRegionEntry) Value loaded from region Return value of org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushRegionEntry.access$000(MemStoreFlusher$FlushRegionEntry) At MemStoreFlusher.java:[line 346] Redundant null check at MemStoreFlusher.java:[line 363] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2689) Implement common gateway service daemon for Avro and Thrift servers
[ https://issues.apache.org/jira/browse/HBASE-2689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13481127#comment-13481127 ] liang xie commented on HBASE-2689: -- [~saint@gmail.com], could we close this stale issue right now ? Implement common gateway service daemon for Avro and Thrift servers --- Key: HBASE-2689 URL: https://issues.apache.org/jira/browse/HBASE-2689 Project: HBase Issue Type: Improvement Components: avro, Thrift Reporter: Jeff Hammerbacher -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7014) cleanup 4 minor findbug warings
[
https://issues.apache.org/jira/browse/HBASE-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13481150#comment-13481150
]
liang xie commented on HBASE-7014:
--
[~saint@gmail.com], sorry for late response:0
1) Before applied this patch, see
https://builds.apache.org/job/PreCommit-HBASE-Build/3087/console: Found 494
Findbugs warnings
(/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/target/findbugsXml.xml);
After applied this patch, see
https://builds.apache.org/job/PreCommit-HBASE-Build/3088/console: Found 490
Findbugs warnings
(/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/target/findbugsXml.xml)
2) Here is the findbugs output that i was addressed :
output1
{quote}
Invocation of toString on tableName in
org.apache.hadoop.hbase.security.access.AccessController.updateACL(RegionCoprocessorEnvironment,
Map)
Bug type DMI_INVOKING_TOSTRING_ON_ARRAY (click for details)
In class org.apache.hadoop.hbase.security.access.AccessController
In method
org.apache.hadoop.hbase.security.access.AccessController.updateACL(RegionCoprocessorEnvironment,
Map)
Local variable named tableName
At AccessController.java:[line 239]
{quote}
output2
{quote}
BC instanceof will always return true in
org.apache.hadoop.hbase.filter.ByteArrayComparable.areSerializedFieldsEqual(ByteArrayComparable),
since all org.apache.hadoop.hbase.filter.ByteArrayComparable are instances of
org.apache.hadoop.hbase.filter.ByteArrayComparable
Bug type BC_VACUOUS_INSTANCEOF (click for details)
In class org.apache.hadoop.hbase.filter.ByteArrayComparable
In method
org.apache.hadoop.hbase.filter.ByteArrayComparable.areSerializedFieldsEqual(ByteArrayComparable)
Actual type org.apache.hadoop.hbase.filter.ByteArrayComparable
Expected org.apache.hadoop.hbase.filter.ByteArrayComparable
At ByteArrayComparable.java:[line 80]
{quote}
output3:
{quote}
GC String is incompatible with expected argument type
org.apache.hadoop.hbase.ServerName in
org.apache.hadoop.hbase.master.ServerManager.getServerConnection(ServerName)
Bug type GC_UNRELATED_TYPES (click for details)
In class org.apache.hadoop.hbase.master.ServerManager
In method
org.apache.hadoop.hbase.master.ServerManager.getServerConnection(ServerName)
Actual type String
Expected org.apache.hadoop.hbase.ServerName
Called method java.util.Map.get(Object)
Invoked on org.apache.hadoop.hbase.master.ServerManager.serverConnections
String.equals(Object) used to determine equality
At ServerManager.java:[line 664]
{quote}
output4:
{quote}
ES Comparison of String objects using == or != in
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(Path,
String, long, boolean, boolean)
Bug type ES_COMPARING_STRINGS_WITH_EQ (click for details)
In class
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager
In method
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(Path,
String, long, boolean, boolean)
Actual type String
Value loaded from key
Return value of java.util.SortedSet.first()
At ReplicationSourceManager.java:[line 161]
{quote}
cleanup 4 minor findbug warings
---
Key: HBASE-7014
URL: https://issues.apache.org/jira/browse/HBASE-7014
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.0
Reporter: liang xie
Assignee: liang xie
Priority: Minor
Fix For: 0.96.0
Attachments: HBASE-7014.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7014) cleanup 4 minor findbug warings
[
https://issues.apache.org/jira/browse/HBASE-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13481151#comment-13481151
]
liang xie commented on HBASE-7014:
--
the serverConnections definition :
{quote}
private final MapServerName, AdminProtocol serverConnections =
new HashMapServerName, AdminProtocol();
{quote}
so the original code : AdminProtocol admin =
this.serverConnections.get(sn.toString()); is weird, though works as well...
cleanup 4 minor findbug warings
---
Key: HBASE-7014
URL: https://issues.apache.org/jira/browse/HBASE-7014
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.0
Reporter: liang xie
Assignee: liang xie
Priority: Minor
Fix For: 0.96.0
Attachments: HBASE-7014.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7014) cleanup 4 minor findbug warings
[ https://issues.apache.org/jira/browse/HBASE-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13481154#comment-13481154 ] liang xie commented on HBASE-7014: -- bq. Do you have to update the test-patch.properties under dev-support count now you've fixed a few? I'm not familiar with this file, but it happened to we don't need to change the number, please correct me if i'm wrong: From https://builds.apache.org/job/PreCommit-HBASE-Build/3088//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html; total number:27 From https://builds.apache.org/job/PreCommit-HBASE-Build/3088//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html; total number:490 490 + 27 = 517 , seems just same as OK_FINDBUGS_WARNINGS=517 in test-patch.properties ... so weird:) cleanup 4 minor findbug warings --- Key: HBASE-7014 URL: https://issues.apache.org/jira/browse/HBASE-7014 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: liang xie Assignee: liang xie Priority: Minor Fix For: 0.96.0 Attachments: HBASE-7014.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6967) Introduce a hook in StoreScanner to allow aggregation of KeyValues in a single row.
[ https://issues.apache.org/jira/browse/HBASE-6967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13481164#comment-13481164 ] liang xie commented on HBASE-6967: -- Do we need to port it to trunk ? i see Revision: 1400431 in 89-fb had checked in already. Introduce a hook in StoreScanner to allow aggregation of KeyValues in a single row. --- Key: HBASE-6967 URL: https://issues.apache.org/jira/browse/HBASE-6967 Project: HBase Issue Type: New Feature Reporter: Adela Maznikar Assignee: Adela Maznikar Priority: Minor Introduce an interface for enabling hooks in StoreScanner -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6930) [89-fb] Avoid acquiring the same row lock repeatedly
[ https://issues.apache.org/jira/browse/HBASE-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13481168#comment-13481168 ] liang xie commented on HBASE-6930: -- [~liyin], seems we need to open an issue for trunk as well, right? I compared Revision: 1393468 in fb-89 branch and current trunk code in roughly， it looks trunk code need to have the similar optimization as well. [89-fb] Avoid acquiring the same row lock repeatedly Key: HBASE-6930 URL: https://issues.apache.org/jira/browse/HBASE-6930 Project: HBase Issue Type: Bug Reporter: Liyin Tang Attachments: D5841.1.patch, D5841.2.patch When processing the multiPut, multiMutations or multiDelete operations, each IPC handler thread tries to acquire a lock for each row key in these batches. If there are duplicated row keys in these batches, previously the IPC handler thread will repeatedly acquire the same row key again and again. So the optimization is to sort each batch operation based on the row key in the client side, and skip acquiring the same row lock repeatedly in the server side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7008) Set scanner caching to a better default
[ https://issues.apache.org/jira/browse/HBASE-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7008: - Attachment: HBASE-7008-v2.patch Addressed Lars Michael's comments, thanks for having a nice review:) Set scanner caching to a better default --- Key: HBASE-7008 URL: https://issues.apache.org/jira/browse/HBASE-7008 Project: HBase Issue Type: Bug Components: Client Reporter: liang xie Assignee: liang xie Fix For: 0.94.3, 0.96.0 Attachments: HBASE-7008.patch, HBASE-7008-v2.patch per http://search-hadoop.com/m/qaRu9iM2f02/Set+scanner+caching+to+a+better+default%253Fsubj=Set+scanner+caching+to+a+better+default+ let's set to 100 by default -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6925) Change socket write size from 8K to 64K for HBaseServer
[ https://issues.apache.org/jira/browse/HBASE-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-6925: - Attachment: HBASE-6925.patch Revision1393467 shows the change has committed into 0.89-fb branch already Change socket write size from 8K to 64K for HBaseServer --- Key: HBASE-6925 URL: https://issues.apache.org/jira/browse/HBASE-6925 Project: HBase Issue Type: Sub-task Reporter: Karthik Ranganathan Assignee: Karthik Ranganathan Attachments: HBASE-6925.patch Creating a JIRA for this, but the change is trivial: change NIO_BUFFER_LIMIT from 8K to 64K in HBaseServer. This seems to increase scan throughput. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6727) [89-fb] allow HBaseServers's callqueue to be better configurable to avoid OOMs
[ https://issues.apache.org/jira/browse/HBASE-6727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479785#comment-13479785 ] liang xie commented on HBASE-6727: -- Hi Kannan, can we close it now, since i saw Revision: 1395033 had resolved it on FB-89 branch. [89-fb] allow HBaseServers's callqueue to be better configurable to avoid OOMs -- Key: HBASE-6727 URL: https://issues.apache.org/jira/browse/HBASE-6727 Project: HBase Issue Type: Bug Reporter: Kannan Muthukkaruppan Assignee: Michal Gregorczyk The callQueue size (where requests get queued up if all handlers are busy) is a LinkedBlockingQueue of size 100 * number_of_handlers. So, with say 300 handler threads, the call queue can have upto 30k entries queued up. If the requests are large enough, this can result in OOM or severe GC pauses. Ideally, we should allow this param to be separately configurable independent of the numberof handlers; perhaps an even better approach would be to specify a memory size based limit, instead of a number of entries based limit. [I have not looked at the trunk version for this issue. So it may or may not be relevant there.] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6923) Create scanner benchmark
[ https://issues.apache.org/jira/browse/HBASE-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479789#comment-13479789 ] liang xie commented on HBASE-6923: -- Hi Karthik, do you have a plan to port your Revision: 1396006 to trunk ? thanks Create scanner benchmark Key: HBASE-6923 URL: https://issues.apache.org/jira/browse/HBASE-6923 Project: HBase Issue Type: Sub-task Reporter: Karthik Ranganathan Assignee: Karthik Ranganathan Attachments: TestStorePerformance.java Create a simple program to benchmark performance/throughput of scanners, and print some results at the end. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7014) cleanup 4 minor findbug warings
liang xie created HBASE-7014: Summary: cleanup 4 minor findbug warings Key: HBASE-7014 URL: https://issues.apache.org/jira/browse/HBASE-7014 Project: HBase Issue Type: Bug Reporter: liang xie Assignee: liang xie Priority: Minor Attachments: HBASE-7014.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7014) cleanup 4 minor findbug warings
[ https://issues.apache.org/jira/browse/HBASE-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7014: - Affects Version/s: 0.96.0 Status: Patch Available (was: Open) cleanup 4 minor findbug warings --- Key: HBASE-7014 URL: https://issues.apache.org/jira/browse/HBASE-7014 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: liang xie Assignee: liang xie Priority: Minor Attachments: HBASE-7014.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7014) cleanup 4 minor findbug warings
[ https://issues.apache.org/jira/browse/HBASE-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7014: - Attachment: HBASE-7014.patch cleanup 4 minor findbug warings --- Key: HBASE-7014 URL: https://issues.apache.org/jira/browse/HBASE-7014 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: liang xie Assignee: liang xie Priority: Minor Attachments: HBASE-7014.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7008) Set scanner caching to a better default
liang xie created HBASE-7008: Summary: Set scanner caching to a better default Key: HBASE-7008 URL: https://issues.apache.org/jira/browse/HBASE-7008 Project: HBase Issue Type: Bug Components: Client Reporter: liang xie -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7008) Set scanner caching to a better default
[ https://issues.apache.org/jira/browse/HBASE-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7008: - Attachment: HBASE-7008.patch Set scanner caching to a better default --- Key: HBASE-7008 URL: https://issues.apache.org/jira/browse/HBASE-7008 Project: HBase Issue Type: Bug Components: Client Reporter: liang xie Attachments: HBASE-7008.patch per http://search-hadoop.com/m/qaRu9iM2f02/Set+scanner+caching+to+a+better+default%253Fsubj=Set+scanner+caching+to+a+better+default+ let's set to 100 by default -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7008) Set scanner caching to a better default
[ https://issues.apache.org/jira/browse/HBASE-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7008: - Description: per http://search-hadoop.com/m/qaRu9iM2f02/Set+scanner+caching+to+a+better+default%253Fsubj=Set+scanner+caching+to+a+better+default+ let's set to 100 by default Set scanner caching to a better default --- Key: HBASE-7008 URL: https://issues.apache.org/jira/browse/HBASE-7008 Project: HBase Issue Type: Bug Components: Client Reporter: liang xie Attachments: HBASE-7008.patch per http://search-hadoop.com/m/qaRu9iM2f02/Set+scanner+caching+to+a+better+default%253Fsubj=Set+scanner+caching+to+a+better+default+ let's set to 100 by default -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7008) Set scanner caching to a better default
[ https://issues.apache.org/jira/browse/HBASE-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7008: - Assignee: liang xie Status: Patch Available (was: Open) Set scanner caching to a better default --- Key: HBASE-7008 URL: https://issues.apache.org/jira/browse/HBASE-7008 Project: HBase Issue Type: Bug Components: Client Reporter: liang xie Assignee: liang xie Attachments: HBASE-7008.patch per http://search-hadoop.com/m/qaRu9iM2f02/Set+scanner+caching+to+a+better+default%253Fsubj=Set+scanner+caching+to+a+better+default+ let's set to 100 by default -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7008) Set scanner caching to a better default
[ https://issues.apache.org/jira/browse/HBASE-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479594#comment-13479594 ] liang xie commented on HBASE-7008: -- emm, IMHO, it's not very necessary:) Set scanner caching to a better default --- Key: HBASE-7008 URL: https://issues.apache.org/jira/browse/HBASE-7008 Project: HBase Issue Type: Bug Components: Client Reporter: liang xie Assignee: liang xie Attachments: HBASE-7008.patch per http://search-hadoop.com/m/qaRu9iM2f02/Set+scanner+caching+to+a+better+default%253Fsubj=Set+scanner+caching+to+a+better+default+ let's set to 100 by default -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7000) Fix the INT_VACUOUS_COMPARISON WARNING in KeyValue class
liang xie created HBASE-7000: Summary: Fix the INT_VACUOUS_COMPARISON WARNING in KeyValue class Key: HBASE-7000 URL: https://issues.apache.org/jira/browse/HBASE-7000 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: liang xie Assignee: liang xie Priority: Minor -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7000) Fix the INT_VACUOUS_COMPARISON WARNING in KeyValue class
[
https://issues.apache.org/jira/browse/HBASE-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liang xie updated HBASE-7000:
-
Attachment: HBASE-7000.patch
there're two choices here.
1 attached file shows change to some value not the old Integer.MAX_VALUE
2 modify KeyValue.java， remove the following code:
if (vlength HConstants.MAXIMUM_VALUE_LENGTH) { // FindBugs
INT_VACUOUS_COMPARISON
throw new IllegalArgumentException(Value length + vlength ++
HConstants.MAXIMUM_VALUE_LENGTH);
}
Fix the INT_VACUOUS_COMPARISON WARNING in KeyValue class
--
Key: HBASE-7000
URL: https://issues.apache.org/jira/browse/HBASE-7000
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.0
Reporter: liang xie
Assignee: liang xie
Priority: Minor
Attachments: HBASE-7000.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7000) Fix the INT_VACUOUS_COMPARISON WARNING in KeyValue class
[ https://issues.apache.org/jira/browse/HBASE-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477794#comment-13477794 ] liang xie commented on HBASE-7000: -- IMHO, the HConstants.MAXIMUM_VALUE_LENGTH maybe changed in future, so the check statement should be always be there for safety Fix the INT_VACUOUS_COMPARISON WARNING in KeyValue class -- Key: HBASE-7000 URL: https://issues.apache.org/jira/browse/HBASE-7000 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: liang xie Assignee: liang xie Priority: Minor Attachments: HBASE-7000.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7000) Fix the INT_VACUOUS_COMPARISON WARNING in KeyValue class
[ https://issues.apache.org/jira/browse/HBASE-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477795#comment-13477795 ] liang xie commented on HBASE-7000: -- BTW, the WARING was caused by : an integer comparison(vlength Integer.MAX_VALUE) that always returns the same value Fix the INT_VACUOUS_COMPARISON WARNING in KeyValue class -- Key: HBASE-7000 URL: https://issues.apache.org/jira/browse/HBASE-7000 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: liang xie Assignee: liang xie Priority: Minor Attachments: HBASE-7000.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7000) Fix the INT_VACUOUS_COMPARISON WARNING in KeyValue class
[ https://issues.apache.org/jira/browse/HBASE-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7000: - Status: Patch Available (was: Open) Fix the INT_VACUOUS_COMPARISON WARNING in KeyValue class -- Key: HBASE-7000 URL: https://issues.apache.org/jira/browse/HBASE-7000 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: liang xie Assignee: liang xie Priority: Minor Attachments: HBASE-7000.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7001) Fix the RCN Correctness Warning in MemStoreFlusher class
liang xie created HBASE-7001: Summary: Fix the RCN Correctness Warning in MemStoreFlusher class Key: HBASE-7001 URL: https://issues.apache.org/jira/browse/HBASE-7001 Project: HBase Issue Type: Bug Reporter: liang xie Priority: Minor -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7001) Fix the RCN Correctness Warning in MemStoreFlusher class
[ https://issues.apache.org/jira/browse/HBASE-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7001: - Description: https://builds.apache.org/job/PreCommit-HBASE-Build/3057//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html#Warnings_CORRECTNESS shows : Bug type RCN_REDUNDANT_NULLCHECK_WOULD_HAVE_BEEN_A_NPE (click for details) In class org.apache.hadoop.hbase.regionserver.MemStoreFlusher In method org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher$FlushRegionEntry) Value loaded from region Return value of org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushRegionEntry.access$000(MemStoreFlusher$FlushRegionEntry) At MemStoreFlusher.java:[line 346] Redundant null check at MemStoreFlusher.java:[line 363] Fix the RCN Correctness Warning in MemStoreFlusher class Key: HBASE-7001 URL: https://issues.apache.org/jira/browse/HBASE-7001 Project: HBase Issue Type: Bug Reporter: liang xie Priority: Minor https://builds.apache.org/job/PreCommit-HBASE-Build/3057//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html#Warnings_CORRECTNESS shows : Bug type RCN_REDUNDANT_NULLCHECK_WOULD_HAVE_BEEN_A_NPE (click for details) In class org.apache.hadoop.hbase.regionserver.MemStoreFlusher In method org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher$FlushRegionEntry) Value loaded from region Return value of org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushRegionEntry.access$000(MemStoreFlusher$FlushRegionEntry) At MemStoreFlusher.java:[line 346] Redundant null check at MemStoreFlusher.java:[line 363] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7001) Fix the RCN Correctness Warning in MemStoreFlusher class
[ https://issues.apache.org/jira/browse/HBASE-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7001: - Assignee: liang xie Status: Patch Available (was: Open) Fix the RCN Correctness Warning in MemStoreFlusher class Key: HBASE-7001 URL: https://issues.apache.org/jira/browse/HBASE-7001 Project: HBase Issue Type: Bug Reporter: liang xie Assignee: liang xie Priority: Minor Attachments: HBASE-7001.patch https://builds.apache.org/job/PreCommit-HBASE-Build/3057//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html#Warnings_CORRECTNESS shows : Bug type RCN_REDUNDANT_NULLCHECK_WOULD_HAVE_BEEN_A_NPE (click for details) In class org.apache.hadoop.hbase.regionserver.MemStoreFlusher In method org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher$FlushRegionEntry) Value loaded from region Return value of org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushRegionEntry.access$000(MemStoreFlusher$FlushRegionEntry) At MemStoreFlusher.java:[line 346] Redundant null check at MemStoreFlusher.java:[line 363] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7001) Fix the RCN Correctness Warning in MemStoreFlusher class
[ https://issues.apache.org/jira/browse/HBASE-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7001: - Attachment: HBASE-7001.patch Fix the RCN Correctness Warning in MemStoreFlusher class Key: HBASE-7001 URL: https://issues.apache.org/jira/browse/HBASE-7001 Project: HBase Issue Type: Bug Reporter: liang xie Priority: Minor Attachments: HBASE-7001.patch https://builds.apache.org/job/PreCommit-HBASE-Build/3057//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html#Warnings_CORRECTNESS shows : Bug type RCN_REDUNDANT_NULLCHECK_WOULD_HAVE_BEEN_A_NPE (click for details) In class org.apache.hadoop.hbase.regionserver.MemStoreFlusher In method org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher$FlushRegionEntry) Value loaded from region Return value of org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushRegionEntry.access$000(MemStoreFlusher$FlushRegionEntry) At MemStoreFlusher.java:[line 346] Redundant null check at MemStoreFlusher.java:[line 363] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7002) Fix all 4 findbug performance warnings
liang xie created HBASE-7002: Summary: Fix all 4 findbug performance warnings Key: HBASE-7002 URL: https://issues.apache.org/jira/browse/HBASE-7002 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: liang xie Priority: Minor Attachments: HBASE-7002.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7002) Fix all 4 findbug performance warnings
[ https://issues.apache.org/jira/browse/HBASE-7002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7002: - Description: Fix the perf warning from this report : https://builds.apache.org/job/PreCommit-HBASE-Build/3057//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html#Warnings_PERFORMANCE Fix all 4 findbug performance warnings -- Key: HBASE-7002 URL: https://issues.apache.org/jira/browse/HBASE-7002 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: liang xie Priority: Minor Attachments: HBASE-7002.patch Fix the perf warning from this report : https://builds.apache.org/job/PreCommit-HBASE-Build/3057//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html#Warnings_PERFORMANCE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7002) Fix all 4 findbug performance warnings
[ https://issues.apache.org/jira/browse/HBASE-7002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7002: - Attachment: HBASE-7002.patch Fix all 4 findbug performance warnings -- Key: HBASE-7002 URL: https://issues.apache.org/jira/browse/HBASE-7002 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: liang xie Priority: Minor Attachments: HBASE-7002.patch Fix the perf warning from this report : https://builds.apache.org/job/PreCommit-HBASE-Build/3057//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html#Warnings_PERFORMANCE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7002) Fix all 4 findbug performance warnings
[ https://issues.apache.org/jira/browse/HBASE-7002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7002: - Assignee: liang xie Status: Patch Available (was: Open) Fix all 4 findbug performance warnings -- Key: HBASE-7002 URL: https://issues.apache.org/jira/browse/HBASE-7002 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: liang xie Assignee: liang xie Priority: Minor Attachments: HBASE-7002.patch Fix the perf warning from this report : https://builds.apache.org/job/PreCommit-HBASE-Build/3057//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html#Warnings_PERFORMANCE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7000) Fix the INT_VACUOUS_COMPARISON WARNING in KeyValue class
[ https://issues.apache.org/jira/browse/HBASE-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7000: - Attachment: HBASE-7000-v2.patch Attached v2 patch addressed Ted's comments Fix the INT_VACUOUS_COMPARISON WARNING in KeyValue class -- Key: HBASE-7000 URL: https://issues.apache.org/jira/browse/HBASE-7000 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: liang xie Assignee: liang xie Priority: Minor Attachments: HBASE-7000.patch, HBASE-7000-v2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6411) Move Master Metrics to metrics 2
[ https://issues.apache.org/jira/browse/HBASE-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-6411: - Fix Version/s: 0.96.0 Move Master Metrics to metrics 2 Key: HBASE-6411 URL: https://issues.apache.org/jira/browse/HBASE-6411 Project: HBase Issue Type: Sub-task Reporter: Elliott Clark Assignee: Alex Baranau Fix For: 0.96.0 Attachments: HBASE-6411-0.patch, HBASE-6411-1.patch, HBASE-6411-2.patch, HBASE-6411-3.patch, HBASE-6411-4_2.patch, HBASE-6411-4.patch, HBASE-6411-5.patch, HBASE-6411_concept.patch Move Master Metrics to metrics 2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7005) Upgrade Thrift lib to 0.9.0
[ https://issues.apache.org/jira/browse/HBASE-7005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478618#comment-13478618 ] liang xie commented on HBASE-7005: -- +1 Upgrade Thrift lib to 0.9.0 --- Key: HBASE-7005 URL: https://issues.apache.org/jira/browse/HBASE-7005 Project: HBase Issue Type: Bug Components: Thrift Reporter: Jake Farrell Priority: Minor Attachments: Hbase-7005.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6998) Uncatched exception in main() makes the HMaster/HRegionServer process suspend
liang xie created HBASE-6998: Summary: Uncatched exception in main() makes the HMaster/HRegionServer process suspend Key: HBASE-6998 URL: https://issues.apache.org/jira/browse/HBASE-6998 Project: HBase Issue Type: Bug Components: master, regionserver Affects Versions: 0.94.2, 0.96.0 Environment: CentOS6.2 + CDH4.1 HDFS + hbase0.94.2 Reporter: liang xie Assignee: liang xie I am trying HDFS QJM feature in our test env. after a misconfig, i found the HMaster/HRegionServer process still up if the main thread is dead. Here is the stack trace: xception in thread main java.net.UnknownHostException: unknown host: cluster1 at org.apache.hadoop.ipc.Client$Connection.init(Client.java:214) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1196) at org.apache.hadoop.ipc.Client.call(Client.java:1050) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) at $Proxy8.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379) at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:238) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:203) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123) at org.apache.hadoop.hbase.regionserver.HRegionServer.startRegionServer(HRegionServer.java:3647) at org.apache.hadoop.hbase.regionserver.HRegionServer.startRegionServer(HRegionServer.java:3631) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:61) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:75) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76) at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:3691) Then i need to kill the process manually each time, so annoyed. After applied the attached patch, the process will exist as expected, then i am happy again :) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6998) Uncatched exception in main() makes the HMaster/HRegionServer process suspend
[ https://issues.apache.org/jira/browse/HBASE-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-6998: - Attachment: HBASE-6998.patch Uncatched exception in main() makes the HMaster/HRegionServer process suspend - Key: HBASE-6998 URL: https://issues.apache.org/jira/browse/HBASE-6998 Project: HBase Issue Type: Bug Components: master, regionserver Affects Versions: 0.94.2, 0.96.0 Environment: CentOS6.2 + CDH4.1 HDFS + hbase0.94.2 Reporter: liang xie Assignee: liang xie Attachments: HBASE-6998.patch I am trying HDFS QJM feature in our test env. after a misconfig, i found the HMaster/HRegionServer process still up if the main thread is dead. Here is the stack trace: xception in thread main java.net.UnknownHostException: unknown host: cluster1 at org.apache.hadoop.ipc.Client$Connection.init(Client.java:214) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1196) at org.apache.hadoop.ipc.Client.call(Client.java:1050) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) at $Proxy8.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379) at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:238) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:203) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123) at org.apache.hadoop.hbase.regionserver.HRegionServer.startRegionServer(HRegionServer.java:3647) at org.apache.hadoop.hbase.regionserver.HRegionServer.startRegionServer(HRegionServer.java:3631) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:61) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:75) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76) at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:3691) Then i need to kill the process manually each time, so annoyed. After applied the attached patch, the process will exist as expected, then i am happy again :) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6998) Uncatched exception in main() makes the HMaster/HRegionServer process suspend
[ https://issues.apache.org/jira/browse/HBASE-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-6998: - Description: I am trying HDFS QJM feature in our test env. after a misconfig, i found the HMaster/HRegionServer process still up if the main thread is dead. Here is the stack trace: xception in thread main java.net.UnknownHostException: unknown host: cluster1 at org.apache.hadoop.ipc.Client$Connection.init(Client.java:214) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1196) at org.apache.hadoop.ipc.Client.call(Client.java:1050) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) at $Proxy8.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379) at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:238) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:203) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123) at org.apache.hadoop.hbase.regionserver.HRegionServer.startRegionServer(HRegionServer.java:3647) at org.apache.hadoop.hbase.regionserver.HRegionServer.startRegionServer(HRegionServer.java:3631) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:61) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:75) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76) at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:3691) Then i need to kill the process manually to cleanup each time, so annoyed. After applied the attached patch, the process will exist as expected, then i am happy again :) was: I am trying HDFS QJM feature in our test env. after a misconfig, i found the HMaster/HRegionServer process still up if the main thread is dead. Here is the stack trace: xception in thread main java.net.UnknownHostException: unknown host: cluster1 at org.apache.hadoop.ipc.Client$Connection.init(Client.java:214) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1196) at org.apache.hadoop.ipc.Client.call(Client.java:1050) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) at $Proxy8.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379) at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:238) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:203) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123) at org.apache.hadoop.hbase.regionserver.HRegionServer.startRegionServer(HRegionServer.java:3647) at org.apache.hadoop.hbase.regionserver.HRegionServer.startRegionServer(HRegionServer.java:3631) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:61) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:75) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76) at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:3691) Then i need to kill the process manually each time, so annoyed. After applied the attached patch, the process will exist as expected, then i am happy again :) Uncatched exception in main() makes the HMaster/HRegionServer process suspend - Key: HBASE-6998 URL: https://issues.apache.org/jira/browse/HBASE-6998 Project:
[jira] [Updated] (HBASE-6998) Uncatched exception in main() makes the HMaster/HRegionServer process suspend
[ https://issues.apache.org/jira/browse/HBASE-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-6998: - Attachment: (was: HBASE-6998.patch) Uncatched exception in main() makes the HMaster/HRegionServer process suspend - Key: HBASE-6998 URL: https://issues.apache.org/jira/browse/HBASE-6998 Project: HBase Issue Type: Bug Components: master, regionserver Affects Versions: 0.94.2, 0.96.0 Environment: CentOS6.2 + CDH4.1 HDFS + hbase0.94.2 Reporter: liang xie Assignee: liang xie Attachments: HBASE-6998.patch I am trying HDFS QJM feature in our test env. after a misconfig, i found the HMaster/HRegionServer process still up if the main thread is dead. Here is the stack trace: xception in thread main java.net.UnknownHostException: unknown host: cluster1 at org.apache.hadoop.ipc.Client$Connection.init(Client.java:214) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1196) at org.apache.hadoop.ipc.Client.call(Client.java:1050) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) at $Proxy8.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379) at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:238) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:203) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123) at org.apache.hadoop.hbase.regionserver.HRegionServer.startRegionServer(HRegionServer.java:3647) at org.apache.hadoop.hbase.regionserver.HRegionServer.startRegionServer(HRegionServer.java:3631) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:61) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:75) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76) at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:3691) Then i need to kill the process manually to cleanup each time, so annoyed. After applied the attached patch, the process will exist as expected, then i am happy again :) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6998) Uncatched exception in main() makes the HMaster/HRegionServer process suspend
[ https://issues.apache.org/jira/browse/HBASE-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-6998: - Attachment: HBASE-6998.patch Uncatched exception in main() makes the HMaster/HRegionServer process suspend - Key: HBASE-6998 URL: https://issues.apache.org/jira/browse/HBASE-6998 Project: HBase Issue Type: Bug Components: master, regionserver Affects Versions: 0.94.2, 0.96.0 Environment: CentOS6.2 + CDH4.1 HDFS + hbase0.94.2 Reporter: liang xie Assignee: liang xie Attachments: HBASE-6998.patch I am trying HDFS QJM feature in our test env. after a misconfig, i found the HMaster/HRegionServer process still up if the main thread is dead. Here is the stack trace: xception in thread main java.net.UnknownHostException: unknown host: cluster1 at org.apache.hadoop.ipc.Client$Connection.init(Client.java:214) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1196) at org.apache.hadoop.ipc.Client.call(Client.java:1050) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) at $Proxy8.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379) at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:238) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:203) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123) at org.apache.hadoop.hbase.regionserver.HRegionServer.startRegionServer(HRegionServer.java:3647) at org.apache.hadoop.hbase.regionserver.HRegionServer.startRegionServer(HRegionServer.java:3631) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:61) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:75) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76) at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:3691) Then i need to kill the process manually to cleanup each time, so annoyed. After applied the attached patch, the process will exist as expected, then i am happy again :) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6998) Uncatched exception in main() makes the HMaster/HRegionServer process suspend
[ https://issues.apache.org/jira/browse/HBASE-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-6998: - Status: Patch Available (was: Open) Uncatched exception in main() makes the HMaster/HRegionServer process suspend - Key: HBASE-6998 URL: https://issues.apache.org/jira/browse/HBASE-6998 Project: HBase Issue Type: Bug Components: master, regionserver Affects Versions: 0.94.2, 0.96.0 Environment: CentOS6.2 + CDH4.1 HDFS + hbase0.94.2 Reporter: liang xie Assignee: liang xie Attachments: HBASE-6998.patch I am trying HDFS QJM feature in our test env. after a misconfig, i found the HMaster/HRegionServer process still up if the main thread is dead. Here is the stack trace: xception in thread main java.net.UnknownHostException: unknown host: cluster1 at org.apache.hadoop.ipc.Client$Connection.init(Client.java:214) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1196) at org.apache.hadoop.ipc.Client.call(Client.java:1050) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) at $Proxy8.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379) at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:238) at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:203) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123) at org.apache.hadoop.hbase.regionserver.HRegionServer.startRegionServer(HRegionServer.java:3647) at org.apache.hadoop.hbase.regionserver.HRegionServer.startRegionServer(HRegionServer.java:3631) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:61) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:75) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76) at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:3691) Then i need to kill the process manually to cleanup each time, so annoyed. After applied the attached patch, the process will exist as expected, then i am happy again :) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6994) minor doc update about DEFAULT_ACCEPTABLE_FACTOR
[ https://issues.apache.org/jira/browse/HBASE-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-6994: - Description: Per ttk code, in LruBlockCache.java: static final float DEFAULT_ACCEPTABLE_FACTOR = 0.99f; but the site doc still : number of region servers * heap size * hfile.block.cache.size * 0.85 seems the HBASE-6312 forgot to update this doc:) was: Per turnk code, in LruBlockCache.java: static final float DEFAULT_ACCEPTABLE_FACTOR = 0.99f; but the site doc still : number of region servers * heap size * hfile.block.cache.size * 0.85 seems the HBASE-6312 forgot to update this doc:) minor doc update about DEFAULT_ACCEPTABLE_FACTOR Key: HBASE-6994 URL: https://issues.apache.org/jira/browse/HBASE-6994 Project: HBase Issue Type: Bug Components: documentation Affects Versions: 0.96.0 Reporter: liang xie Assignee: liang xie Priority: Minor Per ttk code, in LruBlockCache.java: static final float DEFAULT_ACCEPTABLE_FACTOR = 0.99f; but the site doc still : number of region servers * heap size * hfile.block.cache.size * 0.85 seems the HBASE-6312 forgot to update this doc:) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6994) minor doc update about DEFAULT_ACCEPTABLE_FACTOR
liang xie created HBASE-6994: Summary: minor doc update about DEFAULT_ACCEPTABLE_FACTOR Key: HBASE-6994 URL: https://issues.apache.org/jira/browse/HBASE-6994 Project: HBase Issue Type: Bug Components: documentation Affects Versions: 0.96.0 Reporter: liang xie Assignee: liang xie Priority: Minor Per turnk code, in LruBlockCache.java: static final float DEFAULT_ACCEPTABLE_FACTOR = 0.99f; but the site doc still : number of region servers * heap size * hfile.block.cache.size * 0.85 seems the HBASE-6312 forgot to update this doc:) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6994) minor doc update about DEFAULT_ACCEPTABLE_FACTOR
[ https://issues.apache.org/jira/browse/HBASE-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-6994: - Attachment: HBASE-6994.patch minor doc update about DEFAULT_ACCEPTABLE_FACTOR Key: HBASE-6994 URL: https://issues.apache.org/jira/browse/HBASE-6994 Project: HBase Issue Type: Bug Components: documentation Affects Versions: 0.96.0 Reporter: liang xie Assignee: liang xie Priority: Minor Attachments: HBASE-6994.patch Per trunk code, in LruBlockCache.java: static final float DEFAULT_ACCEPTABLE_FACTOR = 0.99f; but the site doc still : number of region servers * heap size * hfile.block.cache.size * 0.85 seems the HBASE-6312 forgot to update this doc:) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6994) minor doc update about DEFAULT_ACCEPTABLE_FACTOR
[ https://issues.apache.org/jira/browse/HBASE-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-6994: - Description: Per trunk code, in LruBlockCache.java: static final float DEFAULT_ACCEPTABLE_FACTOR = 0.99f; but the site doc still : number of region servers * heap size * hfile.block.cache.size * 0.85 seems the HBASE-6312 forgot to update this doc:) was: Per ttk code, in LruBlockCache.java: static final float DEFAULT_ACCEPTABLE_FACTOR = 0.99f; but the site doc still : number of region servers * heap size * hfile.block.cache.size * 0.85 seems the HBASE-6312 forgot to update this doc:) minor doc update about DEFAULT_ACCEPTABLE_FACTOR Key: HBASE-6994 URL: https://issues.apache.org/jira/browse/HBASE-6994 Project: HBase Issue Type: Bug Components: documentation Affects Versions: 0.96.0 Reporter: liang xie Assignee: liang xie Priority: Minor Attachments: HBASE-6994.patch Per trunk code, in LruBlockCache.java: static final float DEFAULT_ACCEPTABLE_FACTOR = 0.99f; but the site doc still : number of region servers * heap size * hfile.block.cache.size * 0.85 seems the HBASE-6312 forgot to update this doc:) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6994) minor doc update about DEFAULT_ACCEPTABLE_FACTOR
[ https://issues.apache.org/jira/browse/HBASE-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-6994: - Status: Patch Available (was: Open) minor doc update about DEFAULT_ACCEPTABLE_FACTOR Key: HBASE-6994 URL: https://issues.apache.org/jira/browse/HBASE-6994 Project: HBase Issue Type: Bug Components: documentation Affects Versions: 0.96.0 Reporter: liang xie Assignee: liang xie Priority: Minor Attachments: HBASE-6994.patch Per trunk code, in LruBlockCache.java: static final float DEFAULT_ACCEPTABLE_FACTOR = 0.99f; but the site doc still : number of region servers * heap size * hfile.block.cache.size * 0.85 seems the HBASE-6312 forgot to update this doc:) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6994) minor doc update about DEFAULT_ACCEPTABLE_FACTOR
[ https://issues.apache.org/jira/browse/HBASE-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13475949#comment-13475949 ] liang xie commented on HBASE-6994: -- thanks my colleague honghua for observing doc inconsistency firstly minor doc update about DEFAULT_ACCEPTABLE_FACTOR Key: HBASE-6994 URL: https://issues.apache.org/jira/browse/HBASE-6994 Project: HBase Issue Type: Bug Components: documentation Affects Versions: 0.96.0 Reporter: liang xie Assignee: liang xie Priority: Minor Attachments: HBASE-6994.patch Per trunk code, in LruBlockCache.java: static final float DEFAULT_ACCEPTABLE_FACTOR = 0.99f; but the site doc still : number of region servers * heap size * hfile.block.cache.size * 0.85 seems the HBASE-6312 forgot to update this doc:) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3577) enables Thrift client to get the Region location
[
https://issues.apache.org/jira/browse/HBASE-3577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13463683#comment-13463683
]
liang xie commented on HBASE-3577:
--
getRegionInfo function can be found on 0.940.96 codebase; This issue seems
only exists on 0.92 and before.
enables Thrift client to get the Region location
Key: HBASE-3577
URL: https://issues.apache.org/jira/browse/HBASE-3577
Project: HBase
Issue Type: Improvement
Components: Thrift
Reporter: Kazuki Ohta
Fix For: 0.96.0
Attachments: HBASE3577-1.patch, HBASE3577-2.patch
The current thrift interface has the getTableRegions() interface like below.
{code}
listTRegionInfo getTableRegions(
/** table name */
1:Text tableName)
throws (1:IOError io)
{code}
{code}
struct TRegionInfo {
1:Text startKey,
2:Text endKey,
3:i64 id,
4:Text name,
5:byte version
}
{code}
But the method don't have the region location information (where the region
is located).
I want to add the Thrift interfaces like below in HTable.java.
{code}
public MapHRegionInfo, HServerAddress getRegionsInfo() throws IOException
{code}
{code}
public HRegionLocation getRegionLocation(final String row)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2689) Implement common gateway service daemon for Avro and Thrift servers
[ https://issues.apache.org/jira/browse/HBASE-2689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13463691#comment-13463691 ] liang xie commented on HBASE-2689: -- Seems it can be closed ... Implement common gateway service daemon for Avro and Thrift servers --- Key: HBASE-2689 URL: https://issues.apache.org/jira/browse/HBASE-2689 Project: HBase Issue Type: Improvement Components: avro, Thrift Reporter: Jeff Hammerbacher -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5835) [hbck] Catch and handle NotServingRegionException when close region attempt fails
[
https://issues.apache.org/jira/browse/HBASE-5835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liang xie updated HBASE-5835:
-
Status: Patch Available (was: Open)
seems i forgot to click submit patch...
[hbck] Catch and handle NotServingRegionException when close region attempt
fails
-
Key: HBASE-5835
URL: https://issues.apache.org/jira/browse/HBASE-5835
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.0, 0.90.7, 0.92.2, 0.96.0
Reporter: Jonathan Hsieh
Attachments: HBASE-5835.patch
Currently, if hbck attempts to close a region and catches a
NotServerRegionException, hbck may hang outputting a stack trace. Since the
goal is to close the region at a particular server, and since it is not
serving the region, the region is closed, and we should just warn and eat
this exception.
{code}
Exception in thread main org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hbase.NotServingRegionException: Received close for
regionid but we are not serving it
at
org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2162)
at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy5.closeRegion(Unknown Source)
at
org.apache.hadoop.hbase.util.HBaseFsckRepair.closeRegionSilentlyAndWait(HBaseFsckRepair.java:165)
at org.apache.hadoop.hbase.util.HBaseFsck.closeRegion(HBaseFsck.java:1185)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1302)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1065)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:351)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:370)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3001)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6882) Thrift IOError should include exception class
[
https://issues.apache.org/jira/browse/HBASE-6882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13463503#comment-13463503
]
liang xie commented on HBASE-6882:
--
Hi Mikhail, seems attached file is not for current community TRUNK version?
since i saw :
{code:title=Hbase.thrift|borderStyle=solid}
exception IOError {
1: string message,
- 2: i64 backoffTimeMillis
+ 2: i64 backoffTimeMillis,
+ 3: string exceptionClass
}
{code}
there is no backoffTimeMillis parameter in struct IOError on current trunk code
and another thing, do we encourage using thrift2 more than thrift right now ?
if that's right, maybe changing thrift2's TIOError is great ?
Thrift IOError should include exception class
-
Key: HBASE-6882
URL: https://issues.apache.org/jira/browse/HBASE-6882
Project: HBase
Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments: D5679.1.patch
Return exception class as part of IOError thrown from the Thrift proxy or the
embedded Thrift server in the regionserver.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6882) Thrift IOError should include exception class
[
https://issues.apache.org/jira/browse/HBASE-6882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13463504#comment-13463504
]
liang xie commented on HBASE-6882:
--
Hi Mikhail, seems attached file is not for current community TRUNK version?
since i saw :
{code:title=Hbase.thrift|borderStyle=solid}
exception IOError {
1: string message,
- 2: i64 backoffTimeMillis
+ 2: i64 backoffTimeMillis,
+ 3: string exceptionClass
}
{code}
there is no backoffTimeMillis parameter in struct IOError on current trunk code
and another thing, do we encourage using thrift2 more than thrift right now ?
if that's right, maybe changing thrift2's TIOError is great ?
Thrift IOError should include exception class
-
Key: HBASE-6882
URL: https://issues.apache.org/jira/browse/HBASE-6882
Project: HBase
Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments: D5679.1.patch
Return exception class as part of IOError thrown from the Thrift proxy or the
embedded Thrift server in the regionserver.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6882) Thrift IOError should include exception class
[ https://issues.apache.org/jira/browse/HBASE-6882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13463511#comment-13463511 ] liang xie commented on HBASE-6882: -- Got it, [~saint@gmail.com] I'd like to have a try:) Thrift IOError should include exception class - Key: HBASE-6882 URL: https://issues.apache.org/jira/browse/HBASE-6882 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D5679.1.patch Return exception class as part of IOError thrown from the Thrift proxy or the embedded Thrift server in the regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6882) Thrift IOError should include exception class
[ https://issues.apache.org/jira/browse/HBASE-6882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13463532#comment-13463532 ] liang xie commented on HBASE-6882: -- Thanks [~saint@gmail.com] for nice guiding ! My plan is to resolve some outstanding thrift related issues firstly, afterwards i could know more details, then maybe i'll have a good feeling on how to fuse thriftthrift2. Don't worry, i'll send a design note before making any big change:) Thrift IOError should include exception class - Key: HBASE-6882 URL: https://issues.apache.org/jira/browse/HBASE-6882 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D5679.1.patch Return exception class as part of IOError thrown from the Thrift proxy or the embedded Thrift server in the regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6871) HFileBlockIndex Write Error BlockIndex in HFile V2
[ https://issues.apache.org/jira/browse/HBASE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461710#comment-13461710 ] liang xie commented on HBASE-6871: -- Hi Feng, attached patch is not well-formed, please refer to http://hbase.apache.org/book.html#submitting.patches to generate a valid diff file, thanks HFileBlockIndex Write Error BlockIndex in HFile V2 -- Key: HBASE-6871 URL: https://issues.apache.org/jira/browse/HBASE-6871 Project: HBase Issue Type: Bug Components: HFile Affects Versions: 0.94.1 Environment: redhat 5u4 Reporter: Feng Wang Priority: Critical Attachments: HBASE-6871.patch After writing some data, compaction and scan operation both failure, the exception message is below: 2012-09-18 06:32:26,227 ERROR org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: Compaction failed regionName=hfile_test,,1347778722498.d220df43fb9d8af4633bd7f547613f9e., storeName=page_info, fileCount=7, fileSize=1.3m (188.0k, 188.0k, 188.0k, 188.0k, 188.0k, 185.8k, 223.3k), priority=9, time=45826250816757428java.io.IOException: Could not reseek StoreFileScanner[HFileScanner for reader reader=hdfs://hadoopdev1.cm6:9000/hbase/hfile_test/d220df43fb9d8af4633bd7f547613f9e/page_info/b0f6118f58de47ad9d87cac438ee0895, compression=lzo, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], firstKey=http://com.truereligionbrandjeans.www/Womens_Dresses/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/4010.html/page_info:anchor_sig/1347764439449/DeleteColumn, lastKey=http://com.trura.www//page_info:page_type/1347763395089/Put, avgKeyLen=776, avgValueLen=4, entries=12853, length=228611, cur=http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/1347764003865/Put/vlen=1/ts=0] to key http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/OLDEST_TIMESTAMP/Minimum/vlen=0/ts=0 at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:178) at org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:299) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:244) at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:521) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:402) at org.apache.hadoop.hbase.regionserver.Store.compactStore(Store.java:1570) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:997) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1216) at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:250) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Expected block type LEAF_INDEX, but got INTERMEDIATE_INDEX: blockType=INTERMEDIATE_INDEX, onDiskSizeWithoutHeader=8514, uncompressedSizeWithoutHeader=131837, prevBlockOffset=-1, dataBeginsWith=\x00\x00\x00\x9B\x00\x00\x00\x00\x00\x00\x03#\x00\x00\x050\x00\x00\x08\xB7\x00\x00\x0Cr\x00\x00\x0F\xFA\x00\x00\x120, fileOffset=218942at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.validateBlockType(HFileReaderV2.java:378) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:331) at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:213) at
[jira] [Assigned] (HBASE-3137) Optimize CAS operations in ZKAssign by being optimistic rather than always doing a sync()
[ https://issues.apache.org/jira/browse/HBASE-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie reassigned HBASE-3137: Assignee: liang xie Optimize CAS operations in ZKAssign by being optimistic rather than always doing a sync() - Key: HBASE-3137 URL: https://issues.apache.org/jira/browse/HBASE-3137 Project: HBase Issue Type: Improvement Components: Zookeeper Reporter: Jonathan Gray Assignee: liang xie HBASE-3136 introduces a fix to the CAS operations in ZKAssign by doing a call to ZK sync() before doing the operation. This gives correct behavior but requires doing a sync every time, even if it isn't necessary. Instead, we should be optimistic and only do retries/syncing if necessary. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3137) Optimize CAS operations in ZKAssign by being optimistic rather than always doing a sync()
[ https://issues.apache.org/jira/browse/HBASE-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461082#comment-13461082 ] liang xie commented on HBASE-3137: -- let me pick it up [~Jonathan Gray] Please feel free to reassign to yourself if you are working on it right now Optimize CAS operations in ZKAssign by being optimistic rather than always doing a sync() - Key: HBASE-3137 URL: https://issues.apache.org/jira/browse/HBASE-3137 Project: HBase Issue Type: Improvement Components: Zookeeper Reporter: Jonathan Gray Assignee: liang xie HBASE-3136 introduces a fix to the CAS operations in ZKAssign by doing a call to ZK sync() before doing the operation. This gives correct behavior but requires doing a sync every time, even if it isn't necessary. Instead, we should be optimistic and only do retries/syncing if necessary. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6852) SchemaMetrics.updateOnCacheHit costs too much while full scanning a table with all of its fields
[ https://issues.apache.org/jira/browse/HBASE-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13460275#comment-13460275 ] liang xie commented on HBASE-6852: -- Hi Cheng, for running time, could you exclude the system resouce factor ？ e.g. you ran the original version with many physical IOs, but reran the patched version without similar physical IO requests due to hitting OS page cache. In other words, could the reduced running time symptom be reproduced always, even you run patched version first, then rerun the original version ? It'd better if you can issue echo 1 /proc/sys/vm/drop_caches to free pagecache between each test. SchemaMetrics.updateOnCacheHit costs too much while full scanning a table with all of its fields Key: HBASE-6852 URL: https://issues.apache.org/jira/browse/HBASE-6852 Project: HBase Issue Type: Improvement Components: metrics Affects Versions: 0.94.0 Reporter: Cheng Hao Priority: Minor Labels: performance Fix For: 0.94.2, 0.96.0 Attachments: onhitcache-trunk.patch The SchemaMetrics.updateOnCacheHit costs too much while I am doing the full table scanning. Here is the top 5 hotspots within regionserver while full scanning a table: (Sorry for the less-well-format) CPU: Intel Westmere microarchitecture, speed 2.262e+06 MHz (estimated) Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 500 samples %image name symbol name --- 9844713.4324 14033.jo void org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, boolean) 98447100.000 14033.jo void org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, boolean) [self] --- 45814 6.2510 14033.jo int org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, byte[], int, int) 45814100.000 14033.jo int org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, byte[], int, int) [self] --- 43523 5.9384 14033.jo boolean org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) 43523100.000 14033.jo boolean org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) [self] --- 42548 5.8054 14033.jo int org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, byte[], int, int) 42548100.000 14033.jo int org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, byte[], int, int) [self] --- 40572 5.5358 14033.jo int org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 40572100.000 14033.jo int org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 [self] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6677) Random ZooKeeper port in test can overrun max port
[
https://issues.apache.org/jira/browse/HBASE-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liang xie updated HBASE-6677:
-
Attachment: HBASE-6677.patch
one line change, it should be safe, i didn't run any test case...
Random ZooKeeper port in test can overrun max port
--
Key: HBASE-6677
URL: https://issues.apache.org/jira/browse/HBASE-6677
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.96.0
Reporter: Gregory Chanan
Priority: Trivial
Labels: noob
Attachments: HBASE-6677.patch
{code}
while (true) {
try {
standaloneServerFactory = new NIOServerCnxnFactory();
standaloneServerFactory.configure(
new InetSocketAddress(tentativePort),
configuration.getInt(HConstants.ZOOKEEPER_MAX_CLIENT_CNXNS,
1000));
} catch (BindException e) {
LOG.debug(Failed binding ZK Server to client port: +
tentativePort);
// This port is already in use, try to use another.
tentativePort++;
continue;
}
break;
}
{code}
In the case of failure and all the above ports have already been binded, you
can extend past the max port. Need to check against a max value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6677) Random ZooKeeper port in test can overrun max port
[
https://issues.apache.org/jira/browse/HBASE-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liang xie updated HBASE-6677:
-
Status: Patch Available (was: Open)
Random ZooKeeper port in test can overrun max port
--
Key: HBASE-6677
URL: https://issues.apache.org/jira/browse/HBASE-6677
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.96.0
Reporter: Gregory Chanan
Priority: Trivial
Labels: noob
Attachments: HBASE-6677.patch
{code}
while (true) {
try {
standaloneServerFactory = new NIOServerCnxnFactory();
standaloneServerFactory.configure(
new InetSocketAddress(tentativePort),
configuration.getInt(HConstants.ZOOKEEPER_MAX_CLIENT_CNXNS,
1000));
} catch (BindException e) {
LOG.debug(Failed binding ZK Server to client port: +
tentativePort);
// This port is already in use, try to use another.
tentativePort++;
continue;
}
break;
}
{code}
In the case of failure and all the above ports have already been binded, you
can extend past the max port. Need to check against a max value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3546) XSS in the WebUI
[
https://issues.apache.org/jira/browse/HBASE-3546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13459586#comment-13459586
]
liang xie commented on HBASE-3546:
--
I tried to repro on 0.94 but failed, seems jamon(RegionListTmpl.jamon) did the
HTML escape. I saw the scriptalert('js')/script text in Start/End Key
columns from WebUI, w/o JS code running.
On 0.90, there's no escape in regionserver.jsp, this issue only exists on
0.90(or before), seems didn't worth to fix it?
Please correct me if i am wrong:)
XSS in the WebUI
Key: HBASE-3546
URL: https://issues.apache.org/jira/browse/HBASE-3546
Project: HBase
Issue Type: Bug
Reporter: Takuya Ueshin
Priority: Minor
There are possibilities of XSS in the WebUI.
If ColumnFamily or Region splitting keys are like
Bytes.toBytes(scriptalert('js')/script)
then browsers run the JavaScript code.
I tested on HBase-0.90.0 .
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3834) Store ignores checksum errors when opening files
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-3834: - Attachment: HBASE-3834.patch Hi ToddStack, actually, this issue was caused by the CATCH statement in loadStoreFiles(). After applied the attached patch, i could observe just the same behivor as 0.920.94. and i also ensure there's no CATCH in 0.920.94's loadStoreFiles(). Store ignores checksum errors when opening files Key: HBASE-3834 URL: https://issues.apache.org/jira/browse/HBASE-3834 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.2 Reporter: Todd Lipcon Assignee: liang xie Priority: Critical Fix For: 0.90.8 Attachments: HBASE-3834.patch, hbase-3834.tar.gz2 If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
