date:20110519

[jira] [Updated] (HBASE-3899) enhance HBase RPC to support free-ing up server handler threads even if response is not ready

2011-05-19 Thread dhruba borthakur (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

dhruba borthakur updated HBASE-3899:

Attachment: asyncRpc.txt

enhance HBase RPC to support free-ing up server handler threads even if
response is not ready
-

Key: HBASE-3899
URL: https://issues.apache.org/jira/browse/HBASE-3899
Project: HBase
Issue Type: Improvement
Components: ipc
Reporter: dhruba borthakur
Assignee: dhruba borthakur
Attachments: asyncRpc.txt, asyncRpc.txt

In the current implementation, the server handler thread picks up an item
from the incoming callqueue, processes it and then wraps the response as a
Writable and sends it back to the IPC server module. This wastes
thread-resources when the thread is blocked for disk IO (transaction logging,
read into block cache, etc).
It would be nice if we can make the RPC Server Handler threads pick up a call
from the IPC queue, hand it over to the application (e.g. HRegion), the
application can queue it to be processed asynchronously and send a response
back to the IPC server module saying that the response is not ready. The RPC
Server Handler thread is now ready to pick up another request from the
incoming callqueue. When the queued call is processed by the application, it
indicates to the IPC module that the response is now ready to be sent back to
the client.
The RPC client continues to experience the same behaviour as before. A RPC
client is synchronous and blocks till the response arrives.
This RPC enhancement allows us to do very powerful things with the
RegionServer. In future, we can make enhance the RegionServer's threading
model to a message-passing model for better performance. We will not be
limited by the number of threads in the RegionServer.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3820) Splitlog() executed while the namenode was in safemode may cause data-loss

2011-05-19 Thread Jieshan Bean (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036039#comment-13036039
 ] 

Jieshan Bean commented on HBASE-3820:
-

Stack, if you have some time available, can you review the patch again? so I 
could know whether something still exist which I need to do about the 
patch...Thanks a lot :)

 Splitlog() executed while the namenode was in safemode may cause data-loss
 --

 Key: HBASE-3820
 URL: https://issues.apache.org/jira/browse/HBASE-3820
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: Jieshan Bean
 Fix For: 0.90.4

 Attachments: HBASE-3820-90-V3.patch, HBASE-3820-MFSFix-90-V2.patch, 
 HBASE-3820-MFSFix-90.patch


 I found this problem while the namenode went into safemode due to some 
 unclear reasons. 
 There's one patch about this problem:
try {
   HLogSplitter splitter = HLogSplitter.createLogSplitter(
 conf, rootdir, logDir, oldLogDir, this.fs);
   try {
 splitter.splitLog();
   } catch (OrphanHLogAfterSplitException e) {
 LOG.warn(Retrying splitting because of:, e);
 // An HLogSplitter instance can only be used once.  Get new instance.
 splitter = HLogSplitter.createLogSplitter(conf, rootdir, logDir,
   oldLogDir, this.fs);
 splitter.splitLog();
   }
   splitTime = splitter.getTime();
   splitLogSize = splitter.getSize();
 } catch (IOException e) {
   checkFileSystem();
   LOG.error(Failed splitting  + logDir.toString(), e);
   master.abort(Shutting down HBase cluster: Failed splitting hlog 
 files..., e);
 } finally {
   this.splitLogLock.unlock();
 }
 And it was really give some useful help to some extent, while the namenode 
 process exited or been killed, but not considered the Namenode safemode 
 exception.
I think the root reason is the method of checkFileSystem().
It gives out an method to check whether the HDFS works normally(Read and 
 write could be success), and that maybe the original propose of this method. 
 This's how this method implements:
 DistributedFileSystem dfs = (DistributedFileSystem) fs;
 try {
   if (dfs.exists(new Path(/))) {  
 return;
   }
 } catch (IOException e) {
   exception = RemoteExceptionHandler.checkIOException(e);
 }

I have check the hdfs code, and learned that while the namenode was in 
 safemode ,the dfs.exists(new Path(/)) returned true. Because the file 
 system could provide read-only service. So this method just checks the dfs 
 whether could be read. I think it's not reasonable.
 


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-3900) Expose progress of a major compaction in UI and/or in shell

2011-05-19 Thread stack (JIRA)

Expose progress of a major compaction in UI and/or in shell
---

 Key: HBASE-3900
 URL: https://issues.apache.org/jira/browse/HBASE-3900
 Project: HBase
  Issue Type: Task
Reporter: stack


A general recommendation is to turn off major compactions and run them 
externally only currently the only way to follow progress of a major compaction 
is by study of regionserver logs.  Add a feature that gives a percentage 
complete of major compaction up in the UI or in shell.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-3901) Update documentation for ImportTsv to reflect recent features

2011-05-19 Thread Bill Graham (JIRA)

Update documentation for ImportTsv to reflect recent features
-

 Key: HBASE-3901
 URL: https://issues.apache.org/jira/browse/HBASE-3901
 Project: HBase
  Issue Type: Improvement
Reporter: Bill Graham
Assignee: Bill Graham
 Fix For: 0.92.0


HBASE-3880 added new features to ImportTsv. Here's a patch to update 
documentation for these and other recent features.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3901) Update documentation for ImportTsv to reflect recent features

2011-05-19 Thread Bill Graham (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Graham updated HBASE-3901:
---

Attachment: HBASE-3901_1.patch

Adding first patch. Please review this clarification for accuracy:

{quote}
By default importtsv will load data directly into HBase. To instead generate
HFiles of data to prepare for a bulk data load, pass the option:
  -Dimporttsv.bulk.output=/path/for/output
{quote}

 Update documentation for ImportTsv to reflect recent features
 -

 Key: HBASE-3901
 URL: https://issues.apache.org/jira/browse/HBASE-3901
 Project: HBase
  Issue Type: Improvement
Reporter: Bill Graham
Assignee: Bill Graham
 Fix For: 0.92.0

 Attachments: HBASE-3901_1.patch


 HBASE-3880 added new features to ImportTsv. Here's a patch to update 
 documentation for these and other recent features.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3901) Update documentation for ImportTsv to reflect recent features

2011-05-19 Thread Bill Graham (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Graham updated HBASE-3901:
---

Status: Patch Available  (was: Open)

 Update documentation for ImportTsv to reflect recent features
 -

 Key: HBASE-3901
 URL: https://issues.apache.org/jira/browse/HBASE-3901
 Project: HBase
  Issue Type: Improvement
Reporter: Bill Graham
Assignee: Bill Graham
 Fix For: 0.92.0

 Attachments: HBASE-3901_1.patch


 HBASE-3880 added new features to ImportTsv. Here's a patch to update 
 documentation for these and other recent features.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3813) Change RPC callQueue size from handlerCount * MAX_QUEUE_SIZE_PER_HANDLER;

2011-05-19 Thread stack (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036386#comment-13036386
]

stack commented on HBASE-3813:
--

@Ted Sounds good if you can figure the size of a Call object in a non-intrusive
way (as per Gary's comment above).

Change RPC callQueue size from handlerCount * MAX_QUEUE_SIZE_PER_HANDLER;
---

Key: HBASE-3813
URL: https://issues.apache.org/jira/browse/HBASE-3813
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: stack
Priority: Critical
Attachments: 3813.txt

Yesterday debugging w/ Jack we noticed that with few handlers on a big box,
he was seeing stats like this:
{code}
2011-04-21 11:54:49,451 DEBUG org.apache.hadoop.ipc.HBaseServer: Server
connection from X.X.X.X:60931; # active connections: 11; # queued calls: 2500
{code}
We had 2500 items in the rpc queue waiting to be processed.
Turns out he had too few handlers for number of clients (but also, it seems
like he figured hw issues in that his RAM bus was running at 1/4 the rate
that it should have been running at).
Chatting w/ J-D this morning, he asked if the queues hold 'data'. The queues
hold 'Calls'. Calls are the client request. They contain data.
Jack had 2500 items queued. If each item to insert was 1MB, thats 25k * 1MB
of memory that is outside of our generally accounting.
Currently the queue size is handlers * MAX_QUEUE_SIZE_PER_HANDLER where
MAX_QUEUE_SIZE_PER_HANDLER is hardcoded to be 100.
If the queue is full we block (LinkedBlockingQueue).
Going to change the queue size from 100 to 10 by default -- but also will
make it configurable and will doc. this as possible cause of OOME. Will try
it on production here before committing patch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-3902) Add Bytes.toBigDecimal and Bytes.toBytes(BigDecimal)

2011-05-19 Thread Vaibhav Puranik (JIRA)

Add Bytes.toBigDecimal and Bytes.toBytes(BigDecimal) 
-

 Key: HBASE-3902
 URL: https://issues.apache.org/jira/browse/HBASE-3902
 Project: HBase
  Issue Type: Improvement
  Components: util
Affects Versions: 0.90.2, 0.90.1
Reporter: Vaibhav Puranik
 Fix For: 0.90.4


Bytes.toBigDecimal and Bytes.toBytes were removed in 0.90.x. Please add it 
back. We have data encoded using these methods. I don't think BigDecimal class 
as getBytes/toBytes methods. And even if it had it, if the logic of encoding it 
into bytes is different, it wouldn't work with the existing data. I am sure 
that lot of people might face this issue. 

I will submit the patch in a day or two.



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3902) Add Bytes.toBigDecimal and Bytes.toBytes(BigDecimal)

2011-05-19 Thread Vaibhav Puranik (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Puranik updated HBASE-3902:
---

Attachment: big-decimal-methods-patch.txt

Patch attached. Methods copied from 0.20.6 branch

 Add Bytes.toBigDecimal and Bytes.toBytes(BigDecimal) 
 -

 Key: HBASE-3902
 URL: https://issues.apache.org/jira/browse/HBASE-3902
 Project: HBase
  Issue Type: Improvement
  Components: util
Affects Versions: 0.90.1, 0.90.2
Reporter: Vaibhav Puranik
 Fix For: 0.90.4

 Attachments: big-decimal-methods-patch.txt


 Bytes.toBigDecimal and Bytes.toBytes were removed in 0.90.x. Please add it 
 back. We have data encoded using these methods. I don't think BigDecimal 
 class as getBytes/toBytes methods. And even if it had it, if the logic of 
 encoding it into bytes is different, it wouldn't work with the existing data. 
 I am sure that lot of people might face this issue. 
 I will submit the patch in a day or two.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-3902) Add Bytes.toBigDecimal and Bytes.toBytes(BigDecimal)

2011-05-19 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-3902.
--

  Resolution: Fixed
Hadoop Flags: [Reviewed]

Committed to branch and trunk.  Thanks for the patch Vaibhav.

 Add Bytes.toBigDecimal and Bytes.toBytes(BigDecimal) 
 -

 Key: HBASE-3902
 URL: https://issues.apache.org/jira/browse/HBASE-3902
 Project: HBase
  Issue Type: Improvement
  Components: util
Affects Versions: 0.90.1, 0.90.2
Reporter: Vaibhav Puranik
 Fix For: 0.90.4

 Attachments: big-decimal-methods-patch.txt


 Bytes.toBigDecimal and Bytes.toBytes were removed in 0.90.x. Please add it 
 back. We have data encoded using these methods. I don't think BigDecimal 
 class as getBytes/toBytes methods. And even if it had it, if the logic of 
 encoding it into bytes is different, it wouldn't work with the existing data. 
 I am sure that lot of people might face this issue. 
 I will submit the patch in a day or two.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3813) Change RPC callQueue size from handlerCount * MAX_QUEUE_SIZE_PER_HANDLER;

2011-05-19 Thread Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036463#comment-13036463
 ] 

Ted Yu commented on HBASE-3813:
---

My proposal doesn't involve moving deserialization overhead into the handlers.
Primary reason is that we should determine the actual size of the parameter 
object for the Call.

So in processData(), we would have:
{code}
HbaseObjectWritable objectWritable = new HbaseObjectWritable();
Writable param = HbaseObjectWritable.readObject(dis, objectWritable, conf);
{code}

I have cloned LinkedBlockingQueueBySize off of LinkedBlockingQueue. Its 
declaration is:
{code}
public class LinkedBlockingQueueBySizeE extends WritableWithSize extends 
AbstractQueueE
implements BlockingQueueE, java.io.Serializable {
{code}
Then we can utilize this method in HbaseObjectWritable:
{code}
  public static long getWritableSize(Object instance, Class declaredClass,
 Configuration conf) {
{code}


 Change RPC callQueue size from handlerCount * MAX_QUEUE_SIZE_PER_HANDLER;
 ---

 Key: HBASE-3813
 URL: https://issues.apache.org/jira/browse/HBASE-3813
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0
Reporter: stack
Priority: Critical
 Attachments: 3813.txt


 Yesterday debugging w/ Jack we noticed that with few handlers on a big box, 
 he was seeing stats like this:
 {code}
 2011-04-21 11:54:49,451 DEBUG org.apache.hadoop.ipc.HBaseServer: Server 
 connection from X.X.X.X:60931; # active connections: 11; # queued calls: 2500
 {code}
 We had 2500 items in the rpc queue waiting to be processed.
 Turns out he had too few handlers for number of clients (but also, it seems 
 like he figured hw issues in that his RAM bus was running at 1/4 the rate 
 that it should have been running at).
 Chatting w/ J-D this morning, he asked if the queues hold 'data'.  The queues 
 hold 'Calls'.  Calls are the client request.  They contain data.
 Jack had 2500 items queued.  If each item to insert was 1MB, thats 25k * 1MB 
 of memory that is outside of our generally accounting.
 Currently the queue size is handlers * MAX_QUEUE_SIZE_PER_HANDLER where 
 MAX_QUEUE_SIZE_PER_HANDLER is hardcoded to be 100.
 If the queue is full we block (LinkedBlockingQueue).
 Going to change the queue size from 100 to 10 by default -- but also will 
 make it configurable and will doc. this as possible cause of OOME.  Will try 
 it on production here before committing patch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-3820) Splitlog() executed while the namenode was in safemode may cause data-loss

2011-05-19 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-3820.
--

  Resolution: Fixed
Hadoop Flags: [Reviewed]

Applied to trunk and branch.  Thanks for the patch Jieshan.

 Splitlog() executed while the namenode was in safemode may cause data-loss
 --

 Key: HBASE-3820
 URL: https://issues.apache.org/jira/browse/HBASE-3820
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: Jieshan Bean
 Fix For: 0.90.4

 Attachments: HBASE-3820-90-V3.patch, HBASE-3820-MFSFix-90-V2.patch, 
 HBASE-3820-MFSFix-90.patch


 I found this problem while the namenode went into safemode due to some 
 unclear reasons. 
 There's one patch about this problem:
try {
   HLogSplitter splitter = HLogSplitter.createLogSplitter(
 conf, rootdir, logDir, oldLogDir, this.fs);
   try {
 splitter.splitLog();
   } catch (OrphanHLogAfterSplitException e) {
 LOG.warn(Retrying splitting because of:, e);
 // An HLogSplitter instance can only be used once.  Get new instance.
 splitter = HLogSplitter.createLogSplitter(conf, rootdir, logDir,
   oldLogDir, this.fs);
 splitter.splitLog();
   }
   splitTime = splitter.getTime();
   splitLogSize = splitter.getSize();
 } catch (IOException e) {
   checkFileSystem();
   LOG.error(Failed splitting  + logDir.toString(), e);
   master.abort(Shutting down HBase cluster: Failed splitting hlog 
 files..., e);
 } finally {
   this.splitLogLock.unlock();
 }
 And it was really give some useful help to some extent, while the namenode 
 process exited or been killed, but not considered the Namenode safemode 
 exception.
I think the root reason is the method of checkFileSystem().
It gives out an method to check whether the HDFS works normally(Read and 
 write could be success), and that maybe the original propose of this method. 
 This's how this method implements:
 DistributedFileSystem dfs = (DistributedFileSystem) fs;
 try {
   if (dfs.exists(new Path(/))) {  
 return;
   }
 } catch (IOException e) {
   exception = RemoteExceptionHandler.checkIOException(e);
 }

I have check the hdfs code, and learned that while the namenode was in 
 safemode ,the dfs.exists(new Path(/)) returned true. Because the file 
 system could provide read-only service. So this method just checks the dfs 
 whether could be read. I think it's not reasonable.
 


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3861) MiniZooKeeperCluster.startup() should refer to hbase.zookeeper.property.maxClientCnxns

2011-05-19 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-3861:
-

Fix Version/s: (was: 0.90.3)
   0.90.4

 MiniZooKeeperCluster.startup() should refer to 
 hbase.zookeeper.property.maxClientCnxns
 --

 Key: HBASE-3861
 URL: https://issues.apache.org/jira/browse/HBASE-3861
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.3
Reporter: Eugene Koontz
Assignee: Eugene Koontz
 Fix For: 0.90.4, 0.92.0

 Attachments: HBASE-3861.patch, HBASE-3861.patch

   Original Estimate: 1h
  Remaining Estimate: 1h

 Currently the number of the client connections is hard-wired to 1000:
 {{{
 standaloneServerFactory = new NIOServerCnxnFactory();
 standaloneServerFactory.configure(new 
 InetSocketAddress(clientPort),1000);
   } catch (BindException e) {
  
 }}}
 This should be set according to the test environment's hbase configuration. 
 The property in 
 question is : hbase.zookeeper.property.maxClientCnxns.
 Currently some tests such as org.apache.hadoop.hbase.client.TestHCM fail 
 because the number of connections used by the HBase client exceeds 1000. 
 Recently MAX_CACHED_HBASE_INSTANCES increased from 31 to 2000 on 0.90 branch:
 http://svn.apache.org/viewvc/hbase/branches/0.90/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java?p2=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fclient%2FHConnectionManager.javap1=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fclient%2FHConnectionManager.javar1=1096818r2=1096817view=diffpathrev=1096818
 and correspondingly the hbase config on the Zookeeper server-side also 
 increased in hbase-default.xml:
 http://svn.apache.org/viewvc/hbase/branches/0.90/src/main/resources/hbase-default.xml?p2=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fresources%2Fhbase-default.xmlp1=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fresources%2Fhbase-default.xmlr1=1091594r2=1091593view=diffpathrev=1091594
 So if MiniZKCluster looks at this setting, the test won't have this failure.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3861) MiniZooKeeperCluster.startup() should refer to hbase.zookeeper.property.maxClientCnxns

2011-05-19 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-3861:
-

Fix Version/s: (was: 0.90.4)

Moving this to 0.92.0.  It was committed for 0.90.3.

 MiniZooKeeperCluster.startup() should refer to 
 hbase.zookeeper.property.maxClientCnxns
 --

 Key: HBASE-3861
 URL: https://issues.apache.org/jira/browse/HBASE-3861
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.3
Reporter: Eugene Koontz
Assignee: Eugene Koontz
 Fix For: 0.92.0

 Attachments: HBASE-3861.patch, HBASE-3861.patch

   Original Estimate: 1h
  Remaining Estimate: 1h

 Currently the number of the client connections is hard-wired to 1000:
 {{{
 standaloneServerFactory = new NIOServerCnxnFactory();
 standaloneServerFactory.configure(new 
 InetSocketAddress(clientPort),1000);
   } catch (BindException e) {
  
 }}}
 This should be set according to the test environment's hbase configuration. 
 The property in 
 question is : hbase.zookeeper.property.maxClientCnxns.
 Currently some tests such as org.apache.hadoop.hbase.client.TestHCM fail 
 because the number of connections used by the HBase client exceeds 1000. 
 Recently MAX_CACHED_HBASE_INSTANCES increased from 31 to 2000 on 0.90 branch:
 http://svn.apache.org/viewvc/hbase/branches/0.90/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java?p2=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fclient%2FHConnectionManager.javap1=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fclient%2FHConnectionManager.javar1=1096818r2=1096817view=diffpathrev=1096818
 and correspondingly the hbase config on the Zookeeper server-side also 
 increased in hbase-default.xml:
 http://svn.apache.org/viewvc/hbase/branches/0.90/src/main/resources/hbase-default.xml?p2=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fresources%2Fhbase-default.xmlp1=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fresources%2Fhbase-default.xmlr1=1091594r2=1091593view=diffpathrev=1091594
 So if MiniZKCluster looks at this setting, the test won't have this failure.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3811) Allow adding attributes to Scan

2011-05-19 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036513#comment-13036513
 ] 

stack commented on HBASE-3811:
--

Why do you extend TestCase Alex?  Its a new test.  Junit4 doesn't use TestCase? 
 Or does it?

Do isEmpty rather than '+  if (attributes.size() == 0) {'

It seems strange we would catch an EOFE on deserialization when we have the 
version available to us.  Can we use the SCAN_VERSION?  Up it to 2 and have 
version 2 able to deserialize both versions?

Otherwise patch looks great Alex.



 Allow adding attributes to Scan
 ---

 Key: HBASE-3811
 URL: https://issues.apache.org/jira/browse/HBASE-3811
 Project: HBase
  Issue Type: Improvement
  Components: client
Reporter: Alex Baranau
Priority: Minor
 Attachments: HBASE-3811.patch, HBASE-3811.patch


 There's sometimes a need to add custom attribute to Scan object so that it 
 can be accessed on server side.
 Example of the case where it is needed discussed here: 
 http://search-hadoop.com/m/v3Jtb2GkiO. There might be other cases where it is 
 useful, which are mostly about logging/gathering stats on server side.
 Alternative to allowing adding any custom attributes to scan could be adding 
 some fixed field, like type to the class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3826) Minor compaction needs to check if still over compactionThreshold after compacting

2011-05-19 Thread stack (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

stack updated HBASE-3826:
-

Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)

Committed the 0.92 patch to TRUNK. Thanks Nicolas.

Minor compaction needs to check if still over compactionThreshold after
compacting
--

Key: HBASE-3826
URL: https://issues.apache.org/jira/browse/HBASE-3826
Project: HBase
Issue Type: Improvement
Components: regionserver
Affects Versions: 0.90.1
Environment: hbase-0.90.1
hbase-0.90.1-cdh3u0
Reporter: Schubert Zhang
Assignee: Nicolas Spiegelberg
Labels: compaction
Fix For: 0.92.0

Attachments: HBASE-3826.patch, HBASE-3826_0.92.patch

I have a busy region, and there are 43 StoreFiles (compactionThreshold=8) in
this region.
Now, I stopped the client and stopped putting new data into it. I expect
these StoreFiles to be compacted later.

But, almost one day later, these 43 StoreFiles are still there.
(Note: in my hbase instance, I disabled the major compaction.)

It seems the minor compaction does not be started continuiously to compact
remaining storefiles.
And I checked the code, it is true.
-
After more test, a obvious issue/problem is, the complete of a minor
compaction does not check if current storefiles need more minor compaction.

I think this may be a bug or leak.

Try this test:

1. Put many data to a region, then there are 30 storefiles accumulated,
because the backend compaction cannot catch up with the fast puts.
(hbase.hstore.compactionThreshold=8, base.hstore.compaction.max=12)

2. Then stop put.

3. Then, these 30 storefiles are still there for a long time, (no automatic
minor compaction)

4. Submit a compaction on this region, then, only 12 files are compaction,
now, we have 19 storefiles. The minor compaction stopped.

I think, when a minor compaction complete, it should check if the number of
storefiles still many, if so, another minor compaction should start
continuiously.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3065) Retry all 'retryable' zk operations; e.g. connection loss

2011-05-19 Thread stack (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036526#comment-13036526
]

stack commented on HBASE-3065:
--

Thanks Liyin. I just tried the patch again to see if I could figure it but it
doesn't apply now (ugh). This patch I think important. Tests used to fail
sporadically up on apache jenkins from time to time because of a lack of retry.
You've done the work, would be nice to bring it home. Let me know if I can
help out.

Retry all 'retryable' zk operations; e.g. connection loss
-

Key: HBASE-3065
URL: https://issues.apache.org/jira/browse/HBASE-3065
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: Liyin Tang
Fix For: 0.92.0

Attachments: 3065-v3.txt, HBase-3065[r1088475]_1.patch,
hbase3065_2.patch

The 'new' master refactored our zk code tidying up all zk accesses and
coralling them behind nice zk utility classes. One improvement was letting
out all KeeperExceptions letting the client deal. Thats good generally
because in old days, we'd suppress important state zk changes in state. But
there is at least one case the new zk utility could handle for the
application and thats the class of retryable KeeperExceptions. The one that
comes to mind is conection loss. On connection loss we should retry the
just-failed operation. Usually the retry will just work. At worse, on
reconnect, we'll pick up the expired session event.
Adding in this change shouldn't be too bad given the refactor of zk corralled
all zk access into one or two classes only.
One thing to consider though is how much we should retry. We could retry on
a timer or we could retry for ever as long as the Stoppable interface is
passed so if another thread has stopped or aborted the hosting service, we'll
notice and give up trying. Doing the latter is probably better than some
kinda timeout.
HBASE-3062 adds a timed retry on the first zk operation. This issue is about
generalizing what is over there across all zk access.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3065) Retry all 'retryable' zk operations; e.g. connection loss

2011-05-19 Thread Liyin Tang (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036529#comment-13036529
]

Liyin Tang commented on HBASE-3065:
---

Thanks stack. If you cannot patch it, I can submit a new patch later. Since the
89 has a different way to use zk, I need more more time to debug the failure of
the unit tests:)

Thanks Liyin

Retry all 'retryable' zk operations; e.g. connection loss
-

Key: HBASE-3065
URL: https://issues.apache.org/jira/browse/HBASE-3065
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: Liyin Tang
Fix For: 0.92.0

Attachments: 3065-v3.txt, HBase-3065[r1088475]_1.patch,
hbase3065_2.patch

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3732) New configuration option for client-side compression

2011-05-19 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036534#comment-13036534
 ] 

stack commented on HBASE-3732:
--

Just to say that the notion of adding a compressed flag to KV is pretty 
invasive with ripples across the code base.  Messy is how we know what codec to 
used undoing the value.  This info will not be in the KV.

 New configuration option for client-side compression
 

 Key: HBASE-3732
 URL: https://issues.apache.org/jira/browse/HBASE-3732
 Project: HBase
  Issue Type: New Feature
Reporter: Jean-Daniel Cryans
 Fix For: 0.92.0

 Attachments: compressed_streams.jar


 We have a case here where we have to store very fat cells (arrays of 
 integers) which can amount into the hundreds of KBs that we need to read 
 often, concurrently, and possibly keep in cache. Compressing the values on 
 the client using java.util.zip's Deflater before sending them to HBase proved 
 to be in our case almost an order of magnitude faster.
 There reasons are evident: less data sent to hbase, memstore contains 
 compressed data, block cache contains compressed data too, etc.
 I was thinking that it might be something useful to add to a family schema, 
 so that Put/Result do the conversion for you. The actual compression algo 
 should also be configurable.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-3903) A successful write to client write-buffer may be lost or not visible

2011-05-19 Thread Tallat (JIRA)

A successful write to client write-buffer may be lost or not visible


 Key: HBASE-3903
 URL: https://issues.apache.org/jira/browse/HBASE-3903
 Project: HBase
  Issue Type: Bug
  Components: documentation
 Environment: Any.
Reporter: Tallat


A client can do a write to a client side 'write buffer' if enabled via 
hTable.setAutoFlush(false). Now, assume a client puts value v under key k. Two 
wrongs things can happen, violating the ACID semantics  of Hbase given at: 
http://hbase.apache.org/acid-semantics.html

1) Say the client fails immediately after the put succeeds. In this case, the 
put will be lost, violating the durability property:

quote Any operation that returns a success code (eg does not throw an 
exception) will be made durable. /quote
 
2) Say the client issues a read for k immediately after writing k. The put will 
be stored in the client side write buffer, while the read will go to the region 
server, returning an older value, instead of v, violating the visibility 
property:

quote
When a client receives a success response for any mutation, that mutation
is immediately visible to both that client and any client with whom it later
communicates through side channels.
/quote

Thanks,
Tallat



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-3904) HConnection.isTableAvailable returns true even with not all regions available.

2011-05-19 Thread Vidhyashankar Venkataraman (JIRA)

HConnection.isTableAvailable returns true even with not all regions available.
--

 Key: HBASE-3904
 URL: https://issues.apache.org/jira/browse/HBASE-3904
 Project: HBase
  Issue Type: Bug
  Components: client
Reporter: Vidhyashankar Venkataraman
Priority: Minor


This function as per the java doc is supposed to return true iff all the 
regions in the table are available. But if the table is still being created 
this function may return inconsistent results (For example, when a table with a 
large number of split keys is created). 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3691) Add compressor support for 'snappy', google's compressor

2011-05-19 Thread stack (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

stack updated HBASE-3691:
-

Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)

Committed to TRUNK. Thanks Nicolas and Nichole for the patches. Nicolas, I
added your howto above to the book into the compression appendix. Thanks.

Add compressor support for 'snappy', google's compressor

Key: HBASE-3691
URL: https://issues.apache.org/jira/browse/HBASE-3691
Project: HBase
Issue Type: Task
Reporter: stack
Priority: Critical
Fix For: 0.92.0

Attachments: hbase-snappy-3691-trunk-002.patch,
hbase-snappy-3691-trunk-003.patch, hbase-snappy-3691-trunk-004.patch,
hbase-snappy-3691-trunk.patch

http://code.google.com/p/snappy/ is apache licensed.
bq. Snappy is a compression/decompression library. It does not aim for
maximum compression, or compatibility with any other compression library;
instead, it aims for very high speeds and reasonable compression. For
instance, compared to the fastest mode of zlib, Snappy is an order of
magnitude faster for most inputs, but the resulting compressed files are
anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in
64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses
at about 500 MB/sec or more.
bq. Snappy is widely used inside Google, in everything from BigTable and
MapReduce to our internal RPC systems. (Snappy has previously been referred
to as Zippy in some presentations and the likes.)
Lets get it in.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2937) Facilitate Timeouts In HBase Client

2011-05-19 Thread jirapos...@reviews.apache.org (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036564#comment-13036564
]

jirapos...@reviews.apache.org commented on HBASE-2937:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/755/
---

(Updated 2011-05-19 23:14:57.474820)

Review request for hbase.

Summary
---

Thanks to HBASE-3154, users now have the ability to specify a timeout for
client-side RPC calls. However, it doesn't go far enough in terms of how low
that timeout can go. Set the RPC timeout to too low a value and you run the
risk of timing out on calls to the meta tables, which are preconditions to
calling the {{HRegionInterface}} proxy.

Given that, I believe the motivation at work in HBASE-2937 still hold true. In
this patch, I add a operation-level timeout, configurable through
hbase.client.operation.timeout, which will override the value specified by
hbase.rpc.timeout, if any, within the scope of the {{ServerCallable#call}}
method. In other words, the operation-level timeout does not apply to calls to
the meta tables.

Furthermore, the patch treats an RPC timeout as a non-fatal event, in that it
will not cause the {{HBaseClient#Connection}} instance to be closed. Last but
not the least, users will also have the ability to set the operation timeout on
the {{HTable}} on the fly.

This addresses bug HBASE-2937.
https://issues.apache.org/jira/browse/HBASE-2937

Diffs (updated)
-

src/main/java/org/apache/hadoop/hbase/HConstants.java e9e3694
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java b26f41e
src/main/java/org/apache/hadoop/hbase/client/HTable.java 61e151a
src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java 6f22123
src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 470e741
src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java dbb57d9
src/main/java/org/apache/hadoop/hbase/util/PoolMap.java 354d49a

Diff: https://reviews.apache.org/r/755/diff

Testing
---

mvn test

Thanks,

Karthick

Facilitate Timeouts In HBase Client
---

Key: HBASE-2937
URL: https://issues.apache.org/jira/browse/HBASE-2937
Project: HBase
Issue Type: New Feature
Components: client
Affects Versions: 0.89.20100621
Reporter: Karthick Sankarachary
Assignee: Karthick Sankarachary
Priority: Critical
Fix For: 0.92.0

Attachments: HBASE-2937.patch, HBASE-2937.patch

Currently, there is no way to force an operation on the HBase client (viz.
HTable) to time out if a certain amount of time has elapsed. In other words,
all invocations on the HTable class are veritable blocking calls, which will
not return until a response (successful or otherwise) is received.
In general, there are two ways to handle timeouts: (a) call the operation in
a separate thread, until it returns a response or the wait on the thread
times out and (b) have the underlying socket unblock the operation if the
read times out. The downside of the former approach is that it consumes more
resources in terms of threads and callables.
Here, we describe a way to specify and handle timeouts on the HTable client,
which relies on the latter approach (i.e., socket timeouts). Right now, the
HBaseClient sets the socket timeout to the value of the ipc.ping.interval
parameter, which is also how long it waits before pinging the server in case
of a failure. The goal is to allow clients to set that timeout on the fly
through HTable. Rather than adding an optional timeout argument to every
HTable operation, we chose to make it a property of HTable which effectively
applies to every method that involves a remote operation.
In order to propagate the timeout from HTable to HBaseClient, we replaced
all occurrences of ServerCallable in HTable with an extension called
ClientCallable, which sets the timeout on the region server interface, once
it has been instantiated, through the HConnection object. The latter, in
turn, asks HBaseRPC to pass that timeout to the corresponding Invoker, so
that it may inject the timeout at the time the invocation is made on the
region server proxy. Right before the request is sent to the server, we set
the timeout specified by the client on the underlying socket.
In conclusion, this patch will afford clients the option of performing an
HBase operation until it completes or a specified timeout elapses. Note that
a timeout of zero is interpreted as an infinite timeout.

--
This message is automatically generated by JIRA.
For more information on JIRA, see:

[jira] [Commented] (HBASE-2937) Facilitate Timeouts In HBase Client

2011-05-19 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036565#comment-13036565
 ] 

jirapos...@reviews.apache.org commented on HBASE-2937:
--



bq.  On 2011-05-19 06:11:23, Michael Stack wrote:
bq.   This seems like a bunch of functionality for a relatively small change.  
Nice one Karthick.  A few questions in the below.

Yes, it does seem like a big change for a relatively small feature, but an 
important one nevertheless. The complexity stems from the fact the scope of the 
operation timeout has to be limited to the {{ServerCallable#call}} method. 

By way of motivation, if you run the TestFromClientSide test with the following 
patch (which sets the hbase.rpc.timeout to 10ms), you'll see that 39 out of 
the 44 test cases will fail.

 24 --- 
a/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 25 +++ 
b/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 26 @@ -94,6 +94,8 @@ public class TestFromClientSide {
 27@BeforeClass
 28public static void setUpBeforeClass() throws Exception {
 29  TEST_UTIL.startMiniCluster(3);
 30 +
TEST_UTIL.getConfiguration().setInt(HConstants.HBASE_RPC_TIMEOUT_KEY, 10);
 32}

On the other hand, if you run it with the default hbase.rpc.timeout but a 
hbase.client.operation.timeout set to 10ms, then you should see the test pass.

 24 --- 
a/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 25 +++ 
b/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 26 @@ -94,6 +94,8 @@ public class TestFromClientSide {
 27@BeforeClass
 28public static void setUpBeforeClass() throws Exception {
 29  TEST_UTIL.startMiniCluster(3);
 30 +
TEST_UTIL.getConfiguration().setInt(HConstants.HBASE_RPC_TIMEOUT_KEY, 6);
 31 +
TEST_UTIL.getConfiguration().setInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT, 
10);
 32}


bq.  On 2011-05-19 06:11:23, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/HTable.java, line 184
bq.   https://reviews.apache.org/r/755/diff/1/?file=19382#file19382line184
bq.  
bq.   What if table is -ROOT-?

Good point. I will use the HTableDescriptor.isMetaTable(tableName) instead. 
When I was debugging it, I noticed that the HTable constructor ends up creating 
a MetaScanner, which in turns creates a HTable on .META., but we do need to 
check -ROOT- as well.


bq.  On 2011-05-19 06:11:23, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java, line 53
bq.   https://reviews.apache.org/r/755/diff/1/?file=19383#file19383line53
bq.  
bq.   Should this use the constant you added above? The deafault timeout?

Okay.


bq.  On 2011-05-19 06:11:23, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java, line 96
bq.   https://reviews.apache.org/r/755/diff/1/?file=19383#file19383line96
bq.  
bq.   Will this work?  What happens if lots of concurrent threads going 
against lots of different tables each with a different timeout?  Will a meta 
table call pick up a short timeout that was put in place by a near-concurrent 
edit?

Good question. Because the {{HBaseRPC#rpcTimeout}} is defined to be 
{{ThreadLocal}}, it should only apply to the user thread that is performing the 
{{HTable}} operation. Also, we take care to reset that thread-specific timeout 
(make it the default) after the {{ServerCallable#call} is done. To be strictly 
correct, I now do the reset in a finally block in 
{{HConnectionImplementation#getRegionServerWith*Retries}}.


bq.  On 2011-05-19 06:11:23, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java, line 
106
bq.   https://reviews.apache.org/r/755/diff/1/?file=19383#file19383line106
bq.  
bq.   Are there other exceptions you think we should rethrow?  Connection 
Exception?

How about we do what HBaseClient does, which is wrap the SocketTimeoutException 
inside another one, along with a context-specific error message?


- Karthick


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/755/#review683
---


On 2011-05-18 23:56:26, Karthick Sankarachary wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/755/
bq.  ---
bq.  
bq.  (Updated 2011-05-18 23:56:26)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Thanks to HBASE-3154, users now have the ability to specify a timeout for 
client-side RPC calls. However, it doesn't go far

[jira] [Assigned] (HBASE-3903) A successful write to client write-buffer may be lost or not visible

2011-05-19 Thread Doug Meil (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil reassigned HBASE-3903:


Assignee: Doug Meil

 A successful write to client write-buffer may be lost or not visible
 

 Key: HBASE-3903
 URL: https://issues.apache.org/jira/browse/HBASE-3903
 Project: HBase
  Issue Type: Bug
  Components: documentation
 Environment: Any.
Reporter: Tallat
Assignee: Doug Meil
  Labels: documentation
 Attachments: acid-semantics_HBASE_3903.xml.patch


 A client can do a write to a client side 'write buffer' if enabled via 
 hTable.setAutoFlush(false). Now, assume a client puts value v under key k. 
 Two wrongs things can happen, violating the ACID semantics  of Hbase given 
 at: http://hbase.apache.org/acid-semantics.html
 1) Say the client fails immediately after the put succeeds. In this case, the 
 put will be lost, violating the durability property:
 quote Any operation that returns a success code (eg does not throw an 
 exception) will be made durable. /quote
  
 2) Say the client issues a read for k immediately after writing k. The put 
 will be stored in the client side write buffer, while the read will go to the 
 region server, returning an older value, instead of v, violating the 
 visibility property:
 quote
 When a client receives a success response for any mutation, that mutation
 is immediately visible to both that client and any client with whom it later
 communicates through side channels.
 /quote
 Thanks,
 Tallat

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3903) A successful write to client write-buffer may be lost or not visible

2011-05-19 Thread Doug Meil (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Doug Meil updated HBASE-3903:
-

Attachment: acid-semantics_HBASE_3903.xml.patch

I had a reference to Delete being in the write-buffer, which isn't correct.

A successful write to client write-buffer may be lost or not visible

Key: HBASE-3903
URL: https://issues.apache.org/jira/browse/HBASE-3903
Project: HBase
Issue Type: Bug
Components: documentation
Environment: Any.
Reporter: Tallat
Assignee: Doug Meil
Labels: documentation
Attachments: acid-semantics_HBASE_3903.xml.patch

A client can do a write to a client side 'write buffer' if enabled via
hTable.setAutoFlush(false). Now, assume a client puts value v under key k.
Two wrongs things can happen, violating the ACID semantics of Hbase given
at: http://hbase.apache.org/acid-semantics.html
1) Say the client fails immediately after the put succeeds. In this case, the
put will be lost, violating the durability property:
quote Any operation that returns a success code (eg does not throw an
exception) will be made durable. /quote

2) Say the client issues a read for k immediately after writing k. The put
will be stored in the client side write buffer, while the read will go to the
region server, returning an older value, instead of v, violating the
visibility property:
quote
When a client receives a success response for any mutation, that mutation
is immediately visible to both that client and any client with whom it later
communicates through side channels.
/quote
Thanks,
Tallat

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3903) A successful write to client write-buffer may be lost or not visible

2011-05-19 Thread Doug Meil (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-3903:
-

Attachment: (was: acid-semantics_HBASE_3903.xml.patch)

 A successful write to client write-buffer may be lost or not visible
 

 Key: HBASE-3903
 URL: https://issues.apache.org/jira/browse/HBASE-3903
 Project: HBase
  Issue Type: Bug
  Components: documentation
 Environment: Any.
Reporter: Tallat
Assignee: Doug Meil
  Labels: documentation
 Attachments: acid-semantics_HBASE_3903.xml.patch


 A client can do a write to a client side 'write buffer' if enabled via 
 hTable.setAutoFlush(false). Now, assume a client puts value v under key k. 
 Two wrongs things can happen, violating the ACID semantics  of Hbase given 
 at: http://hbase.apache.org/acid-semantics.html
 1) Say the client fails immediately after the put succeeds. In this case, the 
 put will be lost, violating the durability property:
 quote Any operation that returns a success code (eg does not throw an 
 exception) will be made durable. /quote
  
 2) Say the client issues a read for k immediately after writing k. The put 
 will be stored in the client side write buffer, while the read will go to the 
 region server, returning an older value, instead of v, violating the 
 visibility property:
 quote
 When a client receives a success response for any mutation, that mutation
 is immediately visible to both that client and any client with whom it later
 communicates through side channels.
 /quote
 Thanks,
 Tallat

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3903) A successful write to client write-buffer may be lost or not visible

2011-05-19 Thread Doug Meil (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Doug Meil updated HBASE-3903:
-

Priority: Minor (was: Major)

changing priority to minor, since this is just a doc bug.

A successful write to client write-buffer may be lost or not visible

Key: HBASE-3903
URL: https://issues.apache.org/jira/browse/HBASE-3903
Project: HBase
Issue Type: Bug
Components: documentation
Environment: Any.
Reporter: Tallat
Assignee: Doug Meil
Priority: Minor
Labels: documentation
Attachments: acid-semantics_HBASE_3903.xml.patch

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-3905) HBaseAdmin.createTableAsync() should check for invalid split keys.

2011-05-19 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu reassigned HBASE-3905:
-

Assignee: Ted Yu

 HBaseAdmin.createTableAsync() should check for invalid split keys.
 --

 Key: HBASE-3905
 URL: https://issues.apache.org/jira/browse/HBASE-3905
 Project: HBase
  Issue Type: Bug
 Environment: Considering this function is open to users, this 
 function should validate the split key array. For example, I had tried 
 creating a table with keys that had duplicate entries. The master (sometimes) 
 crashed with a KeeperException.
 2011-05-14 01:23:33,196 FATAL org.apache.hadoop.hbase.master.HMaster: 
 Unexpected ZK exception creating/setting node OFFLINE 
 org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = 
 BadVersion for /hbase/unassigned/39c3c2f26c777f9d2da8076d9b058c9b
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:106)  
   
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)   
  
 at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1038)
 at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:708)
 at 
 org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:248)
 
 at 
 org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:936)
 
 at 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:887)
 
 at 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:729)
 
 at 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:709)
 
 at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:805)   
  
 at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:773)   
  
 at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:740)   
  
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)   
 at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
 
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
 at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1036)
 2011-05-14 01:23:33,197 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
 And just before exiting:
 2011-05-14 01:23:34,048 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server 
 Responder, call createTable({BLAH BLAH BLAH}, [[B@244e3ce5) from 
 67.195.46.34:36335: output error
Reporter: Vidhyashankar Venkataraman
Assignee: Ted Yu
Priority: Minor
 Attachments: 3905.txt




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-3905) HBaseAdmin.createTableAsync() should check for invalid split keys.

2011-05-19 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-3905.
--

   Resolution: Fixed
Fix Version/s: 0.90.4
 Hadoop Flags: [Reviewed]

Applied trunk and branch.  Thanks for the fix Ted.

 HBaseAdmin.createTableAsync() should check for invalid split keys.
 --

 Key: HBASE-3905
 URL: https://issues.apache.org/jira/browse/HBASE-3905
 Project: HBase
  Issue Type: Bug
 Environment: Considering this function is open to users, this 
 function should validate the split key array. For example, I had tried 
 creating a table with keys that had duplicate entries. The master (sometimes) 
 crashed with a KeeperException.
 2011-05-14 01:23:33,196 FATAL org.apache.hadoop.hbase.master.HMaster: 
 Unexpected ZK exception creating/setting node OFFLINE 
 org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = 
 BadVersion for /hbase/unassigned/39c3c2f26c777f9d2da8076d9b058c9b
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:106)  
   
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)   
  
 at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1038)
 at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:708)
 at 
 org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:248)
 
 at 
 org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:936)
 
 at 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:887)
 
 at 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:729)
 
 at 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:709)
 
 at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:805)   
  
 at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:773)   
  
 at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:740)   
  
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)   
 at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
 
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
 at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1036)
 2011-05-14 01:23:33,197 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
 And just before exiting:
 2011-05-14 01:23:34,048 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server 
 Responder, call createTable({BLAH BLAH BLAH}, [[B@244e3ce5) from 
 67.195.46.34:36335: output error
Reporter: Vidhyashankar Venkataraman
Assignee: Ted Yu
Priority: Minor
 Fix For: 0.90.4

 Attachments: 3905.txt




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3903) A successful write to client write-buffer may be lost or not visible

2011-05-19 Thread stack (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036672#comment-13036672
]

stack commented on HBASE-3903:
--

@Tallat Would you mind reviewing Doug's patch to give it a +1 or -1 on commit?
Thanks.

A successful write to client write-buffer may be lost or not visible

Key: HBASE-3903
URL: https://issues.apache.org/jira/browse/HBASE-3903
Project: HBase
Issue Type: Bug
Components: documentation
Environment: Any.
Reporter: Tallat
Assignee: Doug Meil
Priority: Minor
Labels: documentation
Attachments: acid-semantics_HBASE_3903.xml.patch

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3903) A successful write to client write-buffer may be lost or not visible

2011-05-19 Thread stack (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036673#comment-13036673
]

stack commented on HBASE-3903:
--

I am +1 on commit.

A successful write to client write-buffer may be lost or not visible

Key: HBASE-3903
URL: https://issues.apache.org/jira/browse/HBASE-3903
Project: HBase
Issue Type: Bug
Components: documentation
Environment: Any.
Reporter: Tallat
Assignee: Doug Meil
Priority: Minor
Labels: documentation
Attachments: acid-semantics_HBASE_3903.xml.patch

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2937) Facilitate Timeouts In HBase Client

2011-05-19 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036677#comment-13036677
 ] 

jirapos...@reviews.apache.org commented on HBASE-2937:
--



bq.  On 2011-05-19 06:11:23, Michael Stack wrote:
bq.   This seems like a bunch of functionality for a relatively small change.  
Nice one Karthick.  A few questions in the below.
bq.  
bq.  Karthick Sankarachary wrote:
bq.  Yes, it does seem like a big change for a relatively small feature, 
but an important one nevertheless. The complexity stems from the fact the scope 
of the operation timeout has to be limited to the {{ServerCallable#call}} 
method. 
bq.  
bq.  By way of motivation, if you run the TestFromClientSide test with the 
following patch (which sets the hbase.rpc.timeout to 10ms), you'll see that 
39 out of the 44 test cases will fail.
bq.  
bq.   24 --- 
a/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
bq.   25 +++ 
b/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
bq.   26 @@ -94,6 +94,8 @@ public class TestFromClientSide {
bq.   27@BeforeClass
bq.   28public static void setUpBeforeClass() throws Exception {
bq.   29  TEST_UTIL.startMiniCluster(3);
bq.   30 +
TEST_UTIL.getConfiguration().setInt(HConstants.HBASE_RPC_TIMEOUT_KEY, 10);
bq.   32}
bq.  
bq.  On the other hand, if you run it with the default hbase.rpc.timeout 
but a hbase.client.operation.timeout set to 10ms, then you should see the 
test pass.
bq.  
bq.   24 --- 
a/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
bq.   25 +++ 
b/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
bq.   26 @@ -94,6 +94,8 @@ public class TestFromClientSide {
bq.   27@BeforeClass
bq.   28public static void setUpBeforeClass() throws Exception {
bq.   29  TEST_UTIL.startMiniCluster(3);
bq.   30 +
TEST_UTIL.getConfiguration().setInt(HConstants.HBASE_RPC_TIMEOUT_KEY, 6);
bq.   31 +
TEST_UTIL.getConfiguration().setInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT, 
10);
bq.   32}
bq.

Actually I was saying the opposite.  I'm surprised at how little code had to 
change to make this fix.

So, I don't recall if there is good documentation in this patch on the 
difference between hbase.rpc.timeout and hbase.client.operation.timeout?  
If not, we need it.

Does the TestFromClientSide complete in shorter time if I set a 
hbase.client.operation.timeout of 10ms?


bq.  On 2011-05-19 06:11:23, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java, line 96
bq.   https://reviews.apache.org/r/755/diff/1/?file=19383#file19383line96
bq.  
bq.   Will this work?  What happens if lots of concurrent threads going 
against lots of different tables each with a different timeout?  Will a meta 
table call pick up a short timeout that was put in place by a near-concurrent 
edit?
bq.  
bq.  Karthick Sankarachary wrote:
bq.  Good question. Because the {{HBaseRPC#rpcTimeout}} is defined to be 
{{ThreadLocal}}, it should only apply to the user thread that is performing the 
{{HTable}} operation. Also, we take care to reset that thread-specific timeout 
(make it the default) after the {{ServerCallable#call} is done. To be strictly 
correct, I now do the reset in a finally block in 
{{HConnectionImplementation#getRegionServerWith*Retries}}.

Ok.  If ThreadLocal we should be fine.


bq.  On 2011-05-19 06:11:23, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java, line 
106
bq.   https://reviews.apache.org/r/755/diff/1/?file=19383#file19383line106
bq.  
bq.   Are there other exceptions you think we should rethrow?  Connection 
Exception?
bq.  
bq.  Karthick Sankarachary wrote:
bq.  How about we do what HBaseClient does, which is wrap the 
SocketTimeoutException inside another one, along with a context-specific error 
message?

I was more wondering if there were exceptions we should treat like 
SocketTimeoutException?


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/755/#review683
---


On 2011-05-19 23:14:57, Karthick Sankarachary wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/755/
bq.  ---
bq.  
bq.  (Updated 2011-05-19 23:14:57)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Thanks to HBASE-3154, users now have the ability to

[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable returns true even with not all regions available.

2011-05-19 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036678#comment-13036678
 ] 

stack commented on HBASE-3904:
--

So, how do you think this should work Vidhya?  isTableAvailable won't ever know 
how many regions there are supposed to be in a table.  Checking for  0 is 
probably the best it could ever do.  It sounds to me like we need something 
else or that after isTableAvailable returns true, that then you should get an 
HTable instance and call 
http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html#getRegionsInfo()
 in a loop with a bit of wait until the number of regions comes up to close the 
number you passed to create table?  (FYI: I haven't tried this to see if it 
will work).

 HConnection.isTableAvailable returns true even with not all regions available.
 --

 Key: HBASE-3904
 URL: https://issues.apache.org/jira/browse/HBASE-3904
 Project: HBase
  Issue Type: Bug
  Components: client
Reporter: Vidhyashankar Venkataraman
Priority: Minor

 This function as per the java doc is supposed to return true iff all the 
 regions in the table are available. But if the table is still being created 
 this function may return inconsistent results (For example, when a table with 
 a large number of split keys is created). 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3867) when cluster is stopped and server which hosted meta region is removed from cluster, master breaks down after restarting cluster.

2011-05-19 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036680#comment-13036680
 ] 

stack commented on HBASE-3867:
--

Do you have a patch to fix it Liu?

 when cluster is stopped and server which hosted meta region is removed from 
 cluster, master breaks down after restarting cluster.
 -

 Key: HBASE-3867
 URL: https://issues.apache.org/jira/browse/HBASE-3867
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.1, 0.90.2
Reporter: Liu Jia
Priority: Critical
   Original Estimate: 24h
  Remaining Estimate: 24h

 When cluster stopped and romove server from cluster which contains meta 
 region, then restart cluster,
 From the following code throws NoRouteToHostException
 package org.apache.hadoop.hbase.catalog;
 public class CatalogTracker 
  private HRegionInterface getMetaServerConnection(boolean refresh)
   throws IOException, InterruptedException {
 synchronized (metaAvailable) {
   if (metaAvailable.get()) {
 HRegionInterface current = getCachedConnection(metaLocation);
 if (!refresh) {
   return current;
 }
 if (verifyRegionLocation(current, this.metaLocation, META_REGION)) {
   return current;
 }
 resetMetaLocation();
   }
   HRegionInterface rootConnection = getRootServerConnection();
   if (rootConnection == null) {
 return null;
   }
   HServerAddress newLocation = 
 MetaReader.readMetaLocation(rootConnection);
   if (newLocation == null) {
 return null;
   }
   the following line throws the exception
 HRegionInterface newConnection = getCachedConnection(newLocation);
   if (verifyRegionLocation(newConnection, this.metaLocation, 
 META_REGION)) {
 setMetaLocation(newLocation);
 return newConnection;
   }
   return null;
 }
   }
 /the following method don't handle the exception.
 public class CatalogTracker 
   public boolean verifyMetaRegionLocation(final long timeout)
   throws InterruptedException, IOException {
 return getMetaServerConnection(true) != null;
   }
 //master call the CatalogTracker's method and don't handle 
 the problem too.
 package org.apache.hadoop.hbase.master;
 public class HMaster
 int assignRootAndMeta()
   throws InterruptedException, IOException, KeeperException {
 int assigned = 0;
 long timeout = this.conf.getLong(hbase.catalog.verification.timeout, 
 1000);
 // Work on ROOT region.  Is it in zk in transition?
 boolean rit = this.assignmentManager.
   
 processRegionInTransitionAndBlockUntilAssigned(HRegionInfo.ROOT_REGIONINFO);
 if (!catalogTracker.verifyRootRegionLocation(timeout)) {
   this.assignmentManager.assignRoot();
   this.catalogTracker.waitForRoot();
   assigned++;
 }
 LOG.info(-ROOT- assigned= + assigned + , rit= + rit +
   , location= + catalogTracker.getRootLocation());
 // Work on meta region
 rit = this.assignmentManager.
   
 processRegionInTransitionAndBlockUntilAssigned(HRegionInfo.FIRST_META_REGIONINFO);
 ///
 when restart cluster master break down here.
 
 if (!this.catalogTracker.verifyMetaRegionLocation(timeout)) {
   this.assignmentManager.assignMeta();
   this.catalogTracker.waitForMeta();
   // Above check waits for general meta availability but this does not
   // guarantee that the transition has completed
   
 this.assignmentManager.waitForAssignment(HRegionInfo.FIRST_META_REGIONINFO);
   assigned++;
 }
 LOG.info(.META. assigned= + assigned + , rit= + rit +
   , location= + catalogTracker.getMetaLocation());
 return assigned;
   }
 Thanks to JunQiang Yuan in www.alipay.com  for providing information about 
 this bug. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3866) Script to add regions gradually to a new regionserver.

2011-05-19 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-3866:
-

Hadoop Flags: [Reviewed]
  Status: Patch Available  (was: Open)

Marking patch available.  I'm not committing just yet.  The original script is 
a hackup that might be useful to others.  The patch Ted adds should be factored 
into any balancer improvements.

 Script to add regions gradually to a new regionserver.
 --

 Key: HBASE-3866
 URL: https://issues.apache.org/jira/browse/HBASE-3866
 Project: HBase
  Issue Type: Improvement
  Components: scripts
Affects Versions: 0.90.2
Reporter: Aravind Gottipati
Priority: Minor
 Attachments: 3866-max-regions-per-iteration.patch, slow_balancer.rb, 
 slow_balancer.rb


 When a new region server is brought online, the current balancer kicks off a 
 whole bunch of region moves and causes a lot of regions to be un-available 
 right away.  A slower balancer that gradually balances the cluster is 
 probably a good script to have.  I have an initial version that mooches off 
 the region_mover script to do this.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3859) Increment a counter when a Scanner lease expires

2011-05-19 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-3859:
-

Component/s: metrics

 Increment a counter when a Scanner lease expires
 

 Key: HBASE-3859
 URL: https://issues.apache.org/jira/browse/HBASE-3859
 Project: HBase
  Issue Type: Improvement
  Components: metrics, regionserver
Affects Versions: 0.90.2
Reporter: Benoit Sigoure
Priority: Minor

 Whenever a Scanner lease expires, the RegionServer will close it 
 automatically and log a message to complain.  I would like the RegionServer 
 to increment a counter whenever this happens and expose this counter through 
 the metrics system, so we can plug this into our monitoring system (OpenTSDB) 
 and keep track of how frequently this happens.  It's not supposed to happen 
 frequently so it's good to keep an eye on it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3763) Add Bloom Block Index Support

2011-05-19 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-3763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036683#comment-13036683
 ] 

stack commented on HBASE-3763:
--

@Mikhail Do you want to close this issue?  Your hfile2 subsumes this one?  I 
don't recall your design making note of Joydeeps second suggestion?  It seems 
like a nice little optimization.

 Add Bloom Block Index Support
 -

 Key: HBASE-3763
 URL: https://issues.apache.org/jira/browse/HBASE-3763
 Project: HBase
  Issue Type: Improvement
  Components: io, regionserver
Affects Versions: 0.89.20100924, 0.90.0, 0.90.1, 0.90.2
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Priority: Minor
  Labels: hbase, performance
 Fix For: 0.89.20100924

   Original Estimate: 0h
  Remaining Estimate: 0h

 Add a way to save HBase Bloom filters into an array of Meta blocks instead of 
 one big Meta block, and load only the blocks required to answer a query.  
 This will allow us faster bloom load times for large StoreFiles  pave the 
 path for adding Bloom Filter support to HFileOutputFormat bulk load.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3854) broken examples

2011-05-19 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-3854:
-

Fix Version/s: 0.92.0

Lets fix as part of thrift revamp.

 broken examples
 ---

 Key: HBASE-3854
 URL: https://issues.apache.org/jira/browse/HBASE-3854
 Project: HBase
  Issue Type: Bug
  Components: thrift
Affects Versions: 0.20.0
Reporter: Alexey Diomin
Priority: Minor
 Fix For: 0.92.0


 We introduce NotFound exception in HBASE-1292, but we drop it in HBASE-1367.
 In result:
 1. incorrect doc in Hbase.thrift in as result in generated java and java-doc
 2. broken examples in src/examples/thrift/

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3725) HBase increments from old value after delete and write to disk

2011-05-19 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-3725:
-

Fix Version/s: 0.92.0

 HBase increments from old value after delete and write to disk
 --

 Key: HBASE-3725
 URL: https://issues.apache.org/jira/browse/HBASE-3725
 Project: HBase
  Issue Type: Bug
  Components: io, regionserver
Affects Versions: 0.90.1
Reporter: Nathaniel Cook
Assignee: Jonathan Gray
 Fix For: 0.92.0

 Attachments: HBASE-3725-Test-v1.patch, HBASE-3725-v3.patch, 
 HBASE-3725.patch


 Deleted row values are sometimes used for starting points on new increments.
 To reproduce:
 Create a row r. Set column x to some default value.
 Force hbase to write that value to the file system (such as restarting the 
 cluster).
 Delete the row.
 Call table.incrementColumnValue with some_value
 Get the row.
 The returned value in the column was incremented from the old value before 
 the row was deleted instead of being initialized to some_value.
 Code to reproduce:
 {code}
 import java.io.IOException;
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hbase.HBaseConfiguration;
 import org.apache.hadoop.hbase.HColumnDescriptor;
 import org.apache.hadoop.hbase.HTableDescriptor;
 import org.apache.hadoop.hbase.client.Delete;
 import org.apache.hadoop.hbase.client.Get;
 import org.apache.hadoop.hbase.client.HBaseAdmin;
 import org.apache.hadoop.hbase.client.HTableInterface;
 import org.apache.hadoop.hbase.client.HTablePool;
 import org.apache.hadoop.hbase.client.Increment;
 import org.apache.hadoop.hbase.client.Result;
 import org.apache.hadoop.hbase.util.Bytes;
 public class HBaseTestIncrement
 {
   static String tableName  = testIncrement;
   static byte[] infoCF = Bytes.toBytes(info);
   static byte[] rowKey = Bytes.toBytes(test-rowKey);
   static byte[] newInc = Bytes.toBytes(new);
   static byte[] oldInc = Bytes.toBytes(old);
   /**
* This code reproduces a bug with increment column values in hbase
* Usage: First run part one by passing '1' as the first arg
*Then restart the hbase cluster so it writes everything to disk
*Run part two by passing '2' as the first arg
*
* This will result in the old deleted data being found and used for 
 the increment calls
*
* @param args
* @throws IOException
*/
   public static void main(String[] args) throws IOException
   {
   if(1.equals(args[0]))
   partOne();
   if(2.equals(args[0]))
   partTwo();
   if (both.equals(args[0]))
   {
   partOne();
   partTwo();
   }
   }
   /**
* Creates a table and increments a column value 10 times by 10 each 
 time.
* Results in a value of 100 for the column
*
* @throws IOException
*/
   static void partOne()throws IOException
   {
   Configuration conf = HBaseConfiguration.create();
   HBaseAdmin admin = new HBaseAdmin(conf);
   HTableDescriptor tableDesc = new HTableDescriptor(tableName);
   tableDesc.addFamily(new HColumnDescriptor(infoCF));
   if(admin.tableExists(tableName))
   {
   admin.disableTable(tableName);
   admin.deleteTable(tableName);
   }
   admin.createTable(tableDesc);
   HTablePool pool = new HTablePool(conf, Integer.MAX_VALUE);
   HTableInterface table = pool.getTable(Bytes.toBytes(tableName));
   //Increment unitialized column
   for (int j = 0; j  10; j++)
   {
   table.incrementColumnValue(rowKey, infoCF, oldInc, 
 (long)10);
   Increment inc = new Increment(rowKey);
   inc.addColumn(infoCF, newInc, (long)10);
   table.increment(inc);
   }
   Get get = new Get(rowKey);
   Result r = table.get(get);
   System.out.println(initial values: new  + 
 Bytes.toLong(r.getValue(infoCF, newInc)) +  old  + 
 Bytes.toLong(r.getValue(infoCF, oldInc)));
   }
   /**
* First deletes the data then increments the column 10 times by 1 each 
 time
*
* Should result in a value of 10 but it doesn't, it results in a 
 values of 110
*
* @throws IOException
*/
   static void partTwo()throws IOException
   {
   Configuration conf = HBaseConfiguration.create();
   HTablePool pool = new HTablePool(conf, Integer.MAX_VALUE);

[jira] [Commented] (HBASE-2938) Add Thread-Local Behavior To HTable Pool

2011-05-19 Thread stack (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036686#comment-13036686
]

stack commented on HBASE-2938:
--

The patch looks good to me. Adds being able to choose the type of pool --
reusable vs thread local. I will commit tomorrow unless objection. I am
running tests and they are passing so far.

Add Thread-Local Behavior To HTable Pool

Key: HBASE-2938
URL: https://issues.apache.org/jira/browse/HBASE-2938
Project: HBase
Issue Type: Improvement
Components: client
Affects Versions: 0.89.20100621
Reporter: Karthick Sankarachary
Attachments: HBASE-2938-V2.patch, HBASE-2938.patch

It is a well-documented fact that the HBase table client (viz., HTable) is
not thread-safe. Hence, the recommendation has been to use a HTablePool or a
ThreadLocal to manage access to tables. The downside of the latter is that it
(a) requires the user to reinvent the wheel in terms of mapping table names
to tables and (b) forces the user to maintain the thread-local objects.
Ideally, it would be nice if we could make the HTablePool handle thread-local
objects as well. That way, it not only becomes the one stop shop for all
client-side tables, but also insulates the user from the ThreadLocal object.

Here, we propose a way to generalize the HTablePool so that the underlying
pool type is either reusable or thread-local. To make this possible, we
introdudce the concept of a SharedMap, which essentially, maps a key to a
collection of values, the elements of which are managed by a pool. In effect,
that collection acts as a shared pool of resources, access to which is
closely controlled as dictated by the particular semantics of the pool.
Furthermore, to simplify the construction of HTablePools, we added a couple
of parameters (viz. hbase.client.htable.pool.type and
hbase.client.hbase.pool.size) to control the default behavior of a
HTablePool.

In case the size of the pool is set to a non-zero positive number, that is
used to cap the number of resources that a pool may contain for any given
key. A size of Integer#MAX_VALUE is interpreted to mean an unbounded pool.

Currently, the SharedMap supports the following types of pools:
* A ThreadLocalPool, which represents a pool that builds on the
ThreadLocal class. It essentially binds the resource to the thread from which
it is accessed.
* A ReusablePool, which represents a pool that builds on the LinkedList
class. It essentially allows resources to be checked out, at which point it
is (temporarily) removed from the pool. When the resource is no longer
required, it should be returned to the pool in order to be reused.
* A RoundRobinPool, which represents a pool that stores its resources in
an ArrayList. It load-balances access to its resources by returning a
different resource every time a given key is looked up.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3906) When HMaster is running,there are a lot of RegionLoad instances（far greater than the regions）,it has risk of OOME.

2011-05-19 Thread jian zhang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jian zhang updated HBASE-3906:
--

Description: 
1、Start hbase cluster;
2、After hmaster finish regions assignement,use jmap to dump the memory of 
hmaster;
3、Use MAT to analyse the dump file,there are too many RegionLoad instances,and 
these instances occupy more than 3G memory;

  was:
1、Start hbase cluster;
2、After hmaster finish regions assignement,use jmap to dump the memory of 
hmaster;
3、Use MAT to analyse the dump file,there are too many HServerLoad instances,and 
these instances occupy more than 3G memory;

Summary: When HMaster is running,there are a lot of RegionLoad 
instances（far greater than the regions）,it has risk of OOME.  (was: When 
HMaster is running,there are a lot of HServerLoad instances（far greater than 
the regions）,it has risk of OOME.)

 When HMaster is running,there are a lot of RegionLoad instances（far greater 
 than the regions）,it has risk of OOME.
 --

 Key: HBASE-3906
 URL: https://issues.apache.org/jira/browse/HBASE-3906
 Project: HBase
  Issue Type: Improvement
  Components: master
Affects Versions: 0.90.2
 Environment: 1 hmaster,4 regionserver,about 100,000 regions.
Reporter: jian zhang
 Fix For: 0.90.4

   Original Estimate: 168h
  Remaining Estimate: 168h

 1、Start hbase cluster;
 2、After hmaster finish regions assignement,use jmap to dump the memory of 
 hmaster;
 3、Use MAT to analyse the dump file,there are too many RegionLoad 
 instances,and these instances occupy more than 3G memory;

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable returns true even with not all regions available.

2011-05-19 Thread Vidhyashankar Venkataraman (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036687#comment-13036687
 ] 

Vidhyashankar Venkataraman commented on HBASE-3904:
---

That's exactly what I did with my table existence check! And getRegionsInfo 
works accurately.

 I think we have kind of misunderstood what is actually happening: Or I am 
wrong this time :)

 From what I read in the isTableAvailable function, the Metascanvisitor ensures 
that if there is at least one region not assigned, then the function will 
return false. This isn't enough since the createTable function in master 
assigns one region after another. (Refer to HMAster.createTable(final 
HRegionInfo [] newRegions, boolean sync)) Hence there might be a case when all 
regions are indeed fully assigned in META but it is just that the master is yet 
to populate META with the rest of the regions. I believe that's the reason why 
I got those inconsistent responses from isTableAvailable.

Therefor for isTableAvailable to work correctly with createTable(splitkeys), 
the master will have to populate all the regions in meta first before assigning 
them. I think that will do the trick.

 HConnection.isTableAvailable returns true even with not all regions available.
 --

 Key: HBASE-3904
 URL: https://issues.apache.org/jira/browse/HBASE-3904
 Project: HBase
  Issue Type: Bug
  Components: client
Reporter: Vidhyashankar Venkataraman
Priority: Minor

 This function as per the java doc is supposed to return true iff all the 
 regions in the table are available. But if the table is still being created 
 this function may return inconsistent results (For example, when a table with 
 a large number of split keys is created). 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

43 matches

Mail list logo