[jira] [Created] (HBASE-12201) Close the writers in the MOB sweep tool
Jingcheng Du created HBASE-12201: Summary: Close the writers in the MOB sweep tool Key: HBASE-12201 URL: https://issues.apache.org/jira/browse/HBASE-12201 Project: HBase Issue Type: Bug Reporter: Jingcheng Du Assignee: Jingcheng Du Priority: Critical Fix For: hbase-11339 When running the sweep tool, we encountered such an exception. org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /hbase/mobdir/.tmp/mobcompaction/SweepJob-SweepMapper-SweepReducer-testSweepToolExpiredNoMinVersion-data/working/names/all (inode 46500): File does not exist. Holder DFSClient_NONMAPREDUCE_-1863270027_1 does not have any open files. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3319) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3407) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3377) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:673) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.complete(AuthorizationProviderProxyClientProtocol.java:219) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:520) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) at org.apache.hadoop.ipc.Client.call(Client.java:1411) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy15.complete(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:435) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy16.complete(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2180) at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2164) at org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:908) at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:925) at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:861) at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2687) at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2704) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) This is because we have several writers opened by fs.create(path, true) are not closed properly. Meanwhile, in the current implementation, we save the temp files under sweepJobDir/working/..., and when we remove the directory of the sweep job only the working is deleted. We should remove the whole sweepJobDir instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Compactions nice to have features
please also see https://issues.apache.org/jira/browse/HBASE-11368. it looks to me the multi-CF bulkload could use a lower granularity region level lock so that compaction would not block bulkload and subsequent read/scans. On Mon, Oct 6, 2014 at 6:01 AM, lars hofhansl la...@apache.org wrote: - rack IO throttle. We should add that to accommodate for over subscription at the ToR level. Can you decipher that, Lars? ToR is Top of Rack switch. Over subscription means that a ToR switch usually does not have enough bandwidth to serve traffic in and out of rack at full speed. For example if you had 40 machines in a rack with 1ge links each, and the ToR switch has a 10ge uplink, you'd say the ToR switch is 4 to 1 over subsctribed. Was just trying to say: Yeah, we need that :) - cluster wide compaction storms. Yeah, that's bad. Can be alleviated by spreading timed major compactions out. (in our clusters we set the interval to 1 week and the jitter to 1/2 week) I think we have some JIRAs for that? That you can already do: hbase.hregion.majorcompaction defaults to day in 0.94 (8640ms). Means *all* data is rewritten *every single* day. We set it to 60480ms (1 week) hbase.hregion.majorcompaction.jitter defaults to 20% (0.2). We set this to 0.5 (so we spread out the timed major compactions over 1 week, to avoid storms) Just checked 0.98. Turns out these are exactly the defaults there (1 week +- 1/2 week). Cool, forgot about that (see HBASE-8450). So in 0.98+ the defaults should be pretty good on this end. Compaction storms can still happen during normal load when write is equally spread out of very many regions. I that case it's not unlikely that many regions decide to compact at the same time. -- Lars From: Vladimir Rodionov vladrodio...@gmail.com To: dev@hbase.apache.org dev@hbase.apache.org; lars hofhansl la...@apache.org Sent: Sunday, October 5, 2014 2:11 PM Subject: Re: Compactions nice to have features A few comments: - bulkload - you mean not by loading pre-created HFiles? If you do that there would be no compaction during the import as the files are simply moved into place. Bulk load is not always convenient or feasible, we have batched mutations support in API but still compaction is serious issue. Cassandra allows to disable/enable compactions (I think its cluster-wide, not sure though), why do should not we have? - local compaction IO limit. Limiting the number of compaction threads (1 by default) is not good enough ... ? You can cause too much harm even with a single thread compacting per region server? This is I am not sure about myself. The idea is to make compaction more I/O nicer. For example, read operations and memstore flushes should have higher priority than compaction I/O. One way is to limit (throttle) compaction bandwidth locally, there are some other approaches as well. - rack IO throttle. We should add that to accommodate for over subscription at the ToR level. Can you decipher that, Lars? - cluster wide compaction storms. Yeah, that's bad. Can be alleviated by spreading timed major compactions out. (in our clusters we set the interval to 1 week and the jitter to 1/2 week) I think we have some JIRAs for that? - what do you think about off-peak compaction? We have that in part as the compaction ratio can be set differently for off peak hours Off peak compactions can have higher limits or even different policies. Generally I like the idea of being able to pace compaction better. Do you want to file jiras for these? Yeah, will do that. On Sat, Oct 4, 2014 at 10:31 AM, lars hofhansl la...@apache.org wrote: Hi Vladimir, these are very interesting. A few comments: - bulkload - you mean not by loading pre-created HFiles? If you do that there would be no compaction during the import as the files are simply moved into place. - local compaction IO limit. Limiting the number of compaction threads (1 by default) is not good enough ... ? You can cause too much harm even with a single thread compacting per region server? - rack IO throttle. We should add that to accommodate for over subscription at the ToR level. - cluster wide compaction storms. Yeah, that's bad. Can be alleviated by spreading timed major compactions out. (in our clusters we set the interval to 1 week and the jitter to 1/2 week) - what do you think about off-peak compaction? We have that in part as the compaction ratio can be set differently for off peak hours Generally I like the idea of being able to pace compaction better. Do you want to file jiras for these? Doesn't mean you have to do all the work :) -- Lars From: Vladimir Rodionov vladrodio...@gmail.com To: dev@hbase.apache.org dev@hbase.apache.org Sent: Friday, October 3, 2014 10:34 PM Subject: Compactions nice to have features I am thinking
[jira] [Created] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock
Anoop Sam John created HBASE-12202: -- Summary: Support DirectByteBuffer usage in HFileBlock Key: HBASE-12202 URL: https://issues.apache.org/jira/browse/HBASE-12202 Project: HBase Issue Type: Sub-task Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 0.99.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12203) Remove Master from table status query path.
Andrey Stepachev created HBASE-12203: Summary: Remove Master from table status query path. Key: HBASE-12203 URL: https://issues.apache.org/jira/browse/HBASE-12203 Project: HBase Issue Type: Improvement Components: Client, master Affects Versions: 2.0.0 Reporter: Andrey Stepachev Fix For: 1.0.0, 2.0.0 With patch HBASE-7767 we moved table statuses from ZK to HDFS. That was a good cleanup, but we put additional (substantial) load on master. Some client requests use checks for table state (for example HBASE-12035). Thats is why patch was not back ported to branch1 (HBASE-11978) Lets replicate state back to zk, but as a mirror of table states. What can be done: 1. TableStateManager would push table state changes to zk 2. Return back ZKTableStateClientSideReader. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12204) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1
Ted Yu created HBASE-12204: -- Summary: Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1 Key: HBASE-12204 URL: https://issues.apache.org/jira/browse/HBASE-12204 Project: HBase Issue Type: Improvement Reporter: Ted Yu Priority: Minor The patch from HBASE-12016 doesn't apply to branch-1 due to the fact that HBASE-7767 was applied to master only. This JIRA is to backport HBASE-12016 to branch-1. This can be done when HBASE-12035 is resolved and branch-1 is brought to parity with master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12205) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to 0.98
Andrew Purtell created HBASE-12205: -- Summary: Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to 0.98 Key: HBASE-12205 URL: https://issues.apache.org/jira/browse/HBASE-12205 Project: HBase Issue Type: Task Reporter: Andrew Purtell Priority: Minor Fix For: 0.98.8 Adapt the changes made in HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to 0.98. Not a backport as much as a functionally equivalent set of changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12206) NPE in RSRpcServices
Jimmy Xiang created HBASE-12206:
---
Summary: NPE in RSRpcServices
Key: HBASE-12206
URL: https://issues.apache.org/jira/browse/HBASE-12206
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
Looks leases is null, which is possible since the region server is not open
yet. Will add a check.
{noformat}
2014-10-08 08:38:17,985 ERROR [B.defaultRpcServer.handler=0,queue=0,port=20020]
ipc.RpcServer: Unexpected throwable object
java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:1957)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:30422)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2020)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:724)
2014-10-08 08:38:17,988 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=20020]
ipc.RpcServer: B.defaultRpcServer.handler=0,queue=0,port=20020: callId: 645
service: ClientService methodName: Scan size: 22 connection: 10.20.212.36:53810
java.io.IOException
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2054)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:1957)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:30422)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2020)
... 4 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HBASE-8936) Fixing ZKSlipLog#Counters
[ https://issues.apache.org/jira/browse/HBASE-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manukranth Kolloju reopened HBASE-8936: --- Fixing ZKSlipLog#Counters - Key: HBASE-8936 URL: https://issues.apache.org/jira/browse/HBASE-8936 Project: HBase Issue Type: Bug Affects Versions: 0.89-fb Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Priority: Trivial Fix For: 0.89-fb Original Estimate: 24h Remaining Estimate: 24h Avoiding usage of reflection in ZKSplitLog#Counters class. This use of reflection is failing the jacoco code coverage annotation and execution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-12148) Remove TimeRangeTracker as point of contention when many threads writing a Store
[ https://issues.apache.org/jira/browse/HBASE-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell reopened HBASE-12148: Reopened. Looks like consensus is an addendum is needed. I'm going to sink the 0.98.7RC0 since it has only the original change and one other reason. Remove TimeRangeTracker as point of contention when many threads writing a Store Key: HBASE-12148 URL: https://issues.apache.org/jira/browse/HBASE-12148 Project: HBase Issue Type: Sub-task Components: Performance Affects Versions: 2.0.0, 0.99.1 Reporter: stack Assignee: stack Fix For: 2.0.0, 0.98.7, 0.99.1 Attachments: 12148.addendum.txt, 12148.txt, 12148.txt, 12148v2.txt, 12148v2.txt, Screen Shot 2014-10-01 at 3.39.46 PM.png, Screen Shot 2014-10-01 at 3.41.07 PM.png -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[CANCELLED] The 1st HBase 0.98.7 release candidate (RC0) is available, vote closing 10/13/2014
Let me roll another RC very soon. There are two changes in RC0 that have proven not quite finished: HBASE-8808 (fixed by HBASE-8936) and HBASE-12148 -1 On Tue, Oct 7, 2014 at 1:43 AM, Andrew Purtell apurt...@apache.org wrote: The 1st HBase 0.98.7 release candidate (RC0) is available for download at http://people.apache.org/~apurtell/0.98.7RC0/ and Maven artifacts are also available in the temporary repository https://repository.apache.org/content/repositories/orgapachehbase-1041/ Signed with my code signing key D5365CCD. The issues resolved in this release can be found here: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753version=12327560 Please try out the candidate and vote +1/-1 by midnight Pacific Time (00:00 -0800 GMT) on October 1 3 on whether or not we should release this as 0.98. 7 . Three +1 votes from PMC will be required to release. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: Compactions nice to have features
On Oct 5, 2014, at 11:01 PM, lars hofhansl la...@apache.org wrote: - rack IO throttle. We should add that to accommodate for over subscription at the ToR level. Can you decipher that, Lars? ToR is Top of Rack switch. Over subscription means that a ToR switch usually does not have enough bandwidth to serve traffic in and out of rack at full speed. For example if you had 40 machines in a rack with 1ge links each, and the ToR switch has a 10ge uplink, you'd say the ToR switch is 4 to 1 over subsctribed. Was just trying to say: Yeah, we need that :) Hmmm. Rough math… using 3.5” SATA II (7200 RPM) drives … 4 drives would max out 1GbE. So then a server with 12 drives would max out 3Gb/S. Assuming 3.5” drives. 2.5” drives and SATAIII would push this up. So in theory you could get 5Gb/S or more from a node. 16 serves per rack… (again YMMV based on power, heat, etc … ) thats 48Gb/S and up. If you had 20 servers and they had smaller (2.5” drives) 5Gb/S x 20 = 100Gb/S. So what’s the width of the fabric? (YMMV based on ToR) I don’t know why you’d want to ‘throttle’ because the limits of the ToR would throttle you already. Of course I’m assuming that you’re running a M/R job that’s going full bore. Are you seeing this? I would imagine that you’d have a long running job maxing out the I/O and seeing a jump in wait CPU over time. And what’s the core to spindle ratio?
The 2nd HBase 0.98.7 release candidate (RC1) is available, vote closing 10/15/2014
The 2nd HBase 0.98.7 release candidate (RC1) is available for download at http://people.apache.org/~apurtell/0.98.7RC1/ and Maven artifacts are also available in the temporary repository https://repository.apache.org/content/repositories/orgapachehbase-1042/ Signed with my code signing key D5365CCD. The issues resolved in this release can be found here: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753version=12327560 Please try out the candidate and vote +1/-1 by midnight Pacific Time (00:00 -0800 GMT) on October 15 on whether or not we should release this as 0.98.7. Three +1 votes from PMC will be required to release. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
[jira] [Created] (HBASE-12207) A script to help keep your Git repo fresh
Misty Stanley-Jones created HBASE-12207: --- Summary: A script to help keep your Git repo fresh Key: HBASE-12207 URL: https://issues.apache.org/jira/browse/HBASE-12207 Project: HBase Issue Type: Improvement Components: documentation, scripts Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones I have a script that does a git pull --rebase on each tracking branch, and then attempts an automatic rebase of each local branch against its tracking branch. It also prompts you to delete local branches for HBASE- JIRAs that have been closed. I think this script may help to enforce good Git practices. It may be a good candidate to be included in dev-support/. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12208) Integerating Jacoco code coverage report into the hadoop qa bot
Manukranth Kolloju created HBASE-12208: -- Summary: Integerating Jacoco code coverage report into the hadoop qa bot Key: HBASE-12208 URL: https://issues.apache.org/jira/browse/HBASE-12208 Project: HBase Issue Type: New Feature Components: test Affects Versions: 2.0.0 Reporter: Manukranth Kolloju Fix For: 2.0.0 HBASE-8936 introduced jacoco code coverage report into the pom, but its disabled by default. The code coverage tool would help in finding the extent of code coverage we have in new patches. The hadoop qa bot should be able to stamp a -1 on diffs that decrease the coverage. This jira is to do the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12209) NPE in HRegionServer#getLastSequenceId
Jimmy Xiang created HBASE-12209:
---
Summary: NPE in HRegionServer#getLastSequenceId
Key: HBASE-12209
URL: https://issues.apache.org/jira/browse/HBASE-12209
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
Fix For: 2.0.0, 0.99.1
The region server got the logging splitting task, but the master is gone.
{noformat}
2014-10-08 08:31:22,089 ERROR [RS_LOG_REPLAY_OPS-a2428:20020-1]
executor.EventHandler: Caught throwable while processing event RS_LOG_REPLAY
java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getLastSequenceId(HRegionServer.java:2113)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:317)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:218)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:103)
at
org.apache.hadoop.hbase.regionserver.handler.HLogSplitterHandler.process(HLogSplitterHandler.java:72)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12210) Avoid KeyValue in Prefix Tree
ramkrishna.s.vasudevan created HBASE-12210: -- Summary: Avoid KeyValue in Prefix Tree Key: HBASE-12210 URL: https://issues.apache.org/jira/browse/HBASE-12210 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12211) hbase-daemon.sh should direct the ulimit output to logout instead of loglog
stanley shi created HBASE-12211:
---
Summary: hbase-daemon.sh should direct the ulimit output to logout
instead of loglog
Key: HBASE-12211
URL: https://issues.apache.org/jira/browse/HBASE-12211
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: stanley shi
Priority: Minor
In the hbase-daemon.sh file line 199, it has the content:
{quote}echo `ulimit -a` $loglog 21{quote}
The variable loglog is defined as:
{quote}
logout=$HBASE_LOG_DIR/$HBASE_LOG_PREFIX.out
loggc=$HBASE_LOG_DIR/$HBASE_LOG_PREFIX.gc
loglog=${HBASE_LOG_DIR}/${HBASE_LOGFILE}
{quote}
For my understanding, this information should be printed to the logout
variable; we should not mix this ulimit information with the actual log
printed by hbase java program.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12212) HBaseTestingUtility#waitUntilAllRegionsAssigned should wait for RegionStates
Dima Spivak created HBASE-12212: --- Summary: HBaseTestingUtility#waitUntilAllRegionsAssigned should wait for RegionStates Key: HBASE-12212 URL: https://issues.apache.org/jira/browse/HBASE-12212 Project: HBase Issue Type: Improvement Affects Versions: 0.98.6, 1.0.0, 2.0.0 Reporter: Dima Spivak At the moment, #waitUntilAllRegionsAssigned checks to see if each region has a value for info:server in the meta table, but I've encountered a few instances of tests that fail because RegionStates#getRegionsOfTable returns an empty list, despite #waitUntilAllRegionsAssigned being called after the table was created. Let's fix this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-11937) Update patch submission guidelines to stop using --no-prefix
[
https://issues.apache.org/jira/browse/HBASE-11937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misty Stanley-Jones resolved HBASE-11937.
-
Resolution: Duplicate
Update patch submission guidelines to stop using --no-prefix
Key: HBASE-11937
URL: https://issues.apache.org/jira/browse/HBASE-11937
Project: HBase
Issue Type: Improvement
Components: documentation
Reporter: Sean Busbey
Assignee: Misty Stanley-Jones
Priority: Minor
Right now the submission guidelines include the use of --no-prefix in the
Methods to Create Patches section:
{quote}
Git
git format-patch is preferred because it preserves commit messages. Use git
squash first, to combine smaller commits into a single larger one.
* git format-patch --no-prefix origin/master --stdout HBASE-.patch
* git diff --no-prefix origin/master HBASE-.patch
{quote}
The use of --no-prefix means that users of {{git apply}} and {{git am}} have
to know to specify {{-p0}} and (which strips 0 levels of prefix). Both of
those tools default to stripping the 1 level of prefix git makes by default.
The guide for HBase committers section on reviewing doesn't give any
suggestions on how to apply patches from contributors. It should probably
give examples for using {{git am}} and {{git apply}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Compactions nice to have features
Hi Michael, your math is right. I think the issue is that it actually is easy to max out the ToR switch (and hence starve out other traffic), so we might want to protect the ToR switch from prolonged heavy compaction traffic in order to keep some of the bandwidth free for other traffic. Vladimir issues were around slowing other traffic while compactions are running. -- Lars - Original Message - From: Michael Segel michael_se...@hotmail.com To: dev@hbase.apache.org; lars hofhansl la...@apache.org Cc: Vladimir Rodionov vladrodio...@gmail.com Sent: Wednesday, October 8, 2014 12:30 PM Subject: Re: Compactions nice to have features On Oct 5, 2014, at 11:01 PM, lars hofhansl la...@apache.org wrote: - rack IO throttle. We should add that to accommodate for over subscription at the ToR level. Can you decipher that, Lars? ToR is Top of Rack switch. Over subscription means that a ToR switch usually does not have enough bandwidth to serve traffic in and out of rack at full speed. For example if you had 40 machines in a rack with 1ge links each, and the ToR switch has a 10ge uplink, you'd say the ToR switch is 4 to 1 over subsctribed. Was just trying to say: Yeah, we need that :) Hmmm. Rough math… using 3.5” SATA II (7200 RPM) drives … 4 drives would max out 1GbE. So then a server with 12 drives would max out 3Gb/S. Assuming 3.5” drives. 2.5” drives and SATAIII would push this up. So in theory you could get 5Gb/S or more from a node. 16 serves per rack… (again YMMV based on power, heat, etc … ) thats 48Gb/S and up. If you had 20 servers and they had smaller (2.5” drives) 5Gb/S x 20 = 100Gb/S. So what’s the width of the fabric? (YMMV based on ToR) I don’t know why you’d want to ‘throttle’ because the limits of the ToR would throttle you already. Of course I’m assuming that you’re running a M/R job that’s going full bore. Are you seeing this? I would imagine that you’d have a long running job maxing out the I/O and seeing a jump in wait CPU over time. And what’s the core to spindle ratio?
[jira] [Created] (HBASE-12213) HFileBlock backed by Array of ByteBuffers
Anoop Sam John created HBASE-12213: -- Summary: HFileBlock backed by Array of ByteBuffers Key: HBASE-12213 URL: https://issues.apache.org/jira/browse/HBASE-12213 Project: HBase Issue Type: Sub-task Reporter: Anoop Sam John Assignee: Anoop Sam John In L2 cache (offheap) an HFile block might have been cached into multiple chunks of buffers. If HFileBlock need single BB, we will end up in recreation of bigger BB and copying. Instead we can make HFileBlock to serve data from an array of BBs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
