date:20120523

[
https://issues.apache.org/jira/browse/HBASE-6059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281502#comment-13281502
]

Hadoop QA commented on HBASE-6059:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12528713/HBASE-6059v5.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 9 new or modified tests.

+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 34 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplication
org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
org.apache.hadoop.hbase.replication.TestMasterReplication

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1963//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1963//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1963//console

This message is automatically generated.

Replaying recovered edits would make deleted data exist again
-

Key: HBASE-6059
URL: https://issues.apache.org/jira/browse/HBASE-6059
Project: HBase
Issue Type: Bug
Components: regionserver
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: HBASE-6059-testcase.patch, HBASE-6059.patch,
HBASE-6059v2.patch, HBASE-6059v3.patch, HBASE-6059v4.patch, HBASE-6059v5.patch

When we replay recovered edits, we used the minSeqId of Store, It may cause
deleted data appeared again.
Let's see how it happens. Suppose the region with two families(cf1,cf2)
1.put one data to the region (put r1,cf1:q1,v1)
2.move the region from server A to server B.
3.delete the data put by step 1(delete r1)
4.flush this region.
5.make major compaction for this region
6.move the region from server B to server A.
7.Abort server A
8.After the region is online, we could get the deleted data(r1,cf1:q1,v1)
(When we replay recovered edits, we used the minSeqId of Store, because cf2
has no store files, so its seqId is 0, so the edit log of put data will be
replayed to the region)

[jira] [Commented] (HBASE-6047) Put.has() can't determine result correctly


[ 
https://issues.apache.org/jira/browse/HBASE-6047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281537#comment-13281537
 ] 

Hudson commented on HBASE-6047:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #14 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/14/])
HBASE-6047 Put.has() can't determine result correctly (Alex Newman) 
(Revision 1341737)

 Result = FAILURE
tedyu : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestPutDotHas.java


 Put.has() can't determine result correctly
 --

 Key: HBASE-6047
 URL: https://issues.apache.org/jira/browse/HBASE-6047
 Project: HBase
  Issue Type: Bug
  Components: client
Affects Versions: 0.92.1
Reporter: Wang Qiang
Assignee: Alex Newman
 Fix For: 0.92.2, 0.96.0, 0.94.1

 Attachments: 
 0001-HBASE-6047.-Put.has-can-t-determine-result-correctly-v2.patch, 
 0001-HBASE-6047.-Put.has-can-t-determine-result-correctly.patch, 6047-92.txt, 
 PutTest.java


 the public method 'has(byte [] family, byte [] qualifier)' internally invoked 
 the private method 'has(byte [] family, byte [] qualifier, long ts, byte [] 
 value, boolean ignoreTS, boolean ignoreValue)' with 'value=new byte[0], 
 ignoreTS=true, ignoreValue=true', but there's a logical error in the body, 
 it'll enter the block
 {code}
 else if (ignoreValue) {
   for (KeyValue kv: list) {
 if (Arrays.equals(kv.getFamily(), family)  
 Arrays.equals(kv.getQualifier(), qualifier)
  kv.getTimestamp() == ts) {
   return true;
 }
   }
 }
 {code}
 the expression 'kv.getTimestamp() == ts' in the if conditions should only 
 exist when 'ignoreTS=false', otherwise, the following code will return false!
 {code}
 Put put = new Put(Bytes.toBytes(row-01));
 put.add(Bytes.toBytes(family-01), Bytes.toBytes(qualifier-01),
   1234567L, Bytes.toBytes(value-01));
 System.out.println(put.has(Bytes.toBytes(family-01),
   Bytes.toBytes(qualifier-01)));
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6069) TableInputFormatBase#createRecordReader() doesn't initialize TableRecordReader which causes NPE

2012-05-23 Thread Jie Huang (JIRA)

Jie Huang created HBASE-6069:


 Summary: TableInputFormatBase#createRecordReader() doesn't 
initialize TableRecordReader which causes NPE
 Key: HBASE-6069
 URL: https://issues.apache.org/jira/browse/HBASE-6069
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 0.94.0
Reporter: Jie Huang
Priority: Critical


While running Hive(0.9.0) query over HBase(0.94.0) with hive-hbase-handler, 
there always throws a Null Pointer Exception on Scanner object. Since the 
TableInputFormatBase#createRecordReader() missed the initialization of 
TableRecordReader object. The scanner will be null in that case. This issue 
causes Hive query fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6069) TableInputFormatBase#createRecordReader() doesn't initialize TableRecordReader which causes NPE

2012-05-23 Thread Jie Huang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jie Huang updated HBASE-6069:
-

Attachment: HBASE-6069.patch

Adding the initialization part in the 
TableInputFormatBase#createRecordReader(), the problem is fixed.

 TableInputFormatBase#createRecordReader() doesn't initialize 
 TableRecordReader which causes NPE
 ---

 Key: HBASE-6069
 URL: https://issues.apache.org/jira/browse/HBASE-6069
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 0.94.0
Reporter: Jie Huang
Priority: Critical
 Attachments: HBASE-6069.patch


 While running Hive(0.9.0) query over HBase(0.94.0) with hive-hbase-handler, 
 there always throws a Null Pointer Exception on Scanner object. Since the 
 TableInputFormatBase#createRecordReader() missed the initialization of 
 TableRecordReader object. The scanner will be null in that case. This issue 
 causes Hive query fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5573) Replace client ZooKeeper watchers by simple ZooKeeper reads

2012-05-23 Thread nkeywal (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-5573:
---

   Resolution: Fixed
Fix Version/s: 0.96.0
   Status: Resolved  (was: Patch Available)

 Replace client ZooKeeper watchers by simple ZooKeeper reads
 ---

 Key: HBASE-5573
 URL: https://issues.apache.org/jira/browse/HBASE-5573
 Project: HBase
  Issue Type: Improvement
  Components: client, zookeeper
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Fix For: 0.96.0

 Attachments: 5573.v1.patch, 5573.v2.patch, 5573.v4.patch, 
 5573.v6.patch, 5573.v7.patch, 5573.v8.patch


 Some code in the package needs to read data in ZK. This could be done by a 
 simple read, but is actually implemented with a watcher. This holds ZK 
 resources.
 Fixing this could also be an opportunity to remove the need for the client to 
 provide the master address and port.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6069) TableInputFormatBase#createRecordReader() doesn't initialize TableRecordReader which causes NPE


 [ 
https://issues.apache.org/jira/browse/HBASE-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-6069:
--

Fix Version/s: 0.94.1
   0.96.0
 Hadoop Flags: Reviewed

 TableInputFormatBase#createRecordReader() doesn't initialize 
 TableRecordReader which causes NPE
 ---

 Key: HBASE-6069
 URL: https://issues.apache.org/jira/browse/HBASE-6069
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 0.94.0
Reporter: Jie Huang
Assignee: Jie Huang
Priority: Critical
 Fix For: 0.96.0, 0.94.1

 Attachments: HBASE-6069.patch


 While running Hive(0.9.0) query over HBase(0.94.0) with hive-hbase-handler, 
 there always throws a Null Pointer Exception on Scanner object. Since the 
 TableInputFormatBase#createRecordReader() missed the initialization of 
 TableRecordReader object. The scanner will be null in that case. This issue 
 causes Hive query fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5916) RS restart just before master intialization we make the cluster non operative

2012-05-23 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281645#comment-13281645
 ] 

ramkrishna.s.vasudevan commented on HBASE-5916:
---

Am trying to check all your comments, but coming to this
{code}
if (services.isServerShutdownHandlerEnabled()) {
+// master has completed the initialization
+throw new PleaseHoldException(message);
+  }
{code}
Anyway as i mentioned there is a chance of HLog file getting deleted.  See my 
comments in 
https://issues.apache.org/jira/browse/HBASE-5916?focusedCommentId=13267205page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13267205

That is also one reason why i did not want to go with only that change.  Wanted 
to handle most of the cases.  But there are many scenarios here :(

 RS restart just before master intialization we make the cluster non operative
 -

 Key: HBASE-5916
 URL: https://issues.apache.org/jira/browse/HBASE-5916
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
 Fix For: 0.94.1

 Attachments: HBASE-5916_trunk.patch, HBASE-5916_trunk_1.patch, 
 HBASE-5916_trunk_1.patch, HBASE-5916_trunk_2.patch, HBASE-5916_trunk_3.patch, 
 HBASE-5916_trunk_4.patch, HBASE-5916_trunk_v5.patch


 Consider a case where my master is getting restarted.  RS that was alive when 
 the master restart started, gets restarted before the master initializes the 
 ServerShutDownHandler.
 {code}
 serverShutdownHandlerEnabled = true;
 {code}
 In this case when the RS tries to register with the master, the master will 
 try to expire the server but the server cannot be expired as still the 
 serverShutdownHandler is not enabled.
 This case may happen when i have only one RS gets restarted or all the RS 
 gets restarted at the same time.(before assignRootandMeta).
 {code}
 LOG.info(message);
   if (existingServer.getStartcode()  serverName.getStartcode()) {
 LOG.info(Triggering server recovery; existingServer  +
   existingServer +  looks stale, new server: + serverName);
 expireServer(existingServer);
   }
 {code}
 If another RS is brought up then the cluster comes back to normalcy.
 May be a very corner case.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6069) TableInputFormatBase#createRecordReader() doesn't initialize TableRecordReader which causes NPE


 [ 
https://issues.apache.org/jira/browse/HBASE-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-6069:
--

Attachment: 6069-v2.txt

Patch v2 fixes formatting and throws InterruptedIOException when 
InterruptedException is caught.

 TableInputFormatBase#createRecordReader() doesn't initialize 
 TableRecordReader which causes NPE
 ---

 Key: HBASE-6069
 URL: https://issues.apache.org/jira/browse/HBASE-6069
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 0.94.0
Reporter: Jie Huang
Assignee: Jie Huang
Priority: Critical
 Fix For: 0.96.0, 0.94.1

 Attachments: 6069-v2.txt, HBASE-6069.patch


 While running Hive(0.9.0) query over HBase(0.94.0) with hive-hbase-handler, 
 there always throws a Null Pointer Exception on Scanner object. Since the 
 TableInputFormatBase#createRecordReader() missed the initialization of 
 TableRecordReader object. The scanner will be null in that case. This issue 
 causes Hive query fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6070) AM.nodeDeleted and SSH races creating problems for regions under SPLIT

2012-05-23 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281678#comment-13281678
 ] 

ramkrishna.s.vasudevan commented on HBASE-6070:
---

I plan to make the following change in AM.nodeDeleted.  Currently as SSH is 
trying to handle the RIT in splitting state doing the same in AM.nodeDeleted 
leads to race.  
{code}
-if (rs.isSplitting() || rs.isSplit()) {
+if (rs.isSplit()) {
   LOG.debug(Ephemeral node deleted, regionserver crashed?,  +
 clearing from RIT; rs= + rs);
   regionOffline(rs.getRegion());
{code}
Pls provide your suggestions.

 AM.nodeDeleted and SSH races creating problems for regions under SPLIT
 --

 Key: HBASE-6070
 URL: https://issues.apache.org/jira/browse/HBASE-6070
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: ramkrishna.s.vasudevan
 Fix For: 0.92.2, 0.96.0, 0.94.1


 We tried to address the problems in Master restart and RS restart while SPLIT 
 region is in progress as part of HBASE-5806.
 While doing some more we found still there is one race condition.
 - Split has just started and the znode is in RS_SPLIT state.
 - RS goes down.
 - First call back for SSH comes.
 - As part of the fix for HBASE-5806 SSH knows that some region is in RIT.
 - But now nodeDeleted event comes for the SPLIt node and there we try to 
 delete the RIT.
 - After this we try to see in the SSH whether any node is in RIT.  As we 
 dont find the region in RIT the region is never assigned.
 When we fixed HBASE-5806 step 6 happened first and then step 5 happened.  So 
 we missed it.  Now we found that. Will come up with a patch shortly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6050) HLogSplitter renaming recovered.edits and CJ removing the parent directory races, making the HBCK to think cluster is inconsistent.

2012-05-23 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281680#comment-13281680
 ] 

ramkrishna.s.vasudevan commented on HBASE-6050:
---

Pls share your comments on this patch? If it is ok i can prepare for other 
versions also.

 HLogSplitter renaming recovered.edits and CJ removing the parent directory 
 races, making the HBCK to think cluster is inconsistent.
 ---

 Key: HBASE-6050
 URL: https://issues.apache.org/jira/browse/HBASE-6050
 Project: HBase
  Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
 Attachments: HBASE-6050.patch


 The scenario is like this
 - A region is getting splitted.
 - The master is still not processed the split .
 - Region server goes down.
 - Split log manager starts splitting the logs and creates the 
 recovered.edits in the splitlog path.
 - CJ starts and deletes the entry from META and also just completes the 
 deletion of the region dir.
 - in hlogSplitter on final step we rename the recovered.edits to come under 
 the regiondir.
 There if the regiondir doesnot exist we tend to create and then add the 
 recovered.edits.
 Because of this HBCK thinks it to be an orphan region because we have the 
 regiondir but with no regioninfo.
 Ideally cluster is fine but we it is misleading.
 {code}
 } else {
   Path dstdir = dst.getParent();
   if (!fs.exists(dstdir)) {
 if (!fs.mkdirs(dstdir)) LOG.warn(mkdir failed on  + dstdir);
   }
 }
 fs.rename(src, dst);
 LOG.debug( moved  + src +  =  + dst);
   } else {
 LOG.debug(Could not move recovered edits from  + src +
  as it doesn't exist);
   }
 }
 archiveLogs(null, corruptedLogs, processedLogs,
 oldLogDir, fs, conf);
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4655) Document architecture of backups

2012-05-23 Thread Karthik Ranganathan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281684#comment-13281684
 ] 

Karthik Ranganathan commented on HBASE-4655:


Marking as resolved, feel free to send more comments my way in case something 
is not clear.

 Document architecture of backups
 

 Key: HBASE-4655
 URL: https://issues.apache.org/jira/browse/HBASE-4655
 Project: HBase
  Issue Type: Sub-task
  Components: documentation, regionserver
Reporter: Karthik Ranganathan
Assignee: Karthik Ranganathan
 Attachments: HBase Backups Architecture v2.docx, HBase Backups 
 Architecture.docx


 Basic idea behind the backup architecture for HBase

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4655) Document architecture of backups

2012-05-23 Thread Karthik Ranganathan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Ranganathan resolved HBASE-4655.


Resolution: Fixed

 Document architecture of backups
 

 Key: HBASE-4655
 URL: https://issues.apache.org/jira/browse/HBASE-4655
 Project: HBase
  Issue Type: Sub-task
  Components: documentation, regionserver
Reporter: Karthik Ranganathan
Assignee: Karthik Ranganathan
 Attachments: HBase Backups Architecture v2.docx, HBase Backups 
 Architecture.docx


 Basic idea behind the backup architecture for HBase

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4663) MR based copier for copying HFiles

2012-05-23 Thread Karthik Ranganathan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281685#comment-13281685
 ] 

Karthik Ranganathan commented on HBASE-4663:


See https://reviews.facebook.net/D1965 for the diff. Also, see HBASE-5509 for 
the trunk version.

 MR based copier for copying HFiles
 --

 Key: HBASE-4663
 URL: https://issues.apache.org/jira/browse/HBASE-4663
 Project: HBase
  Issue Type: Sub-task
  Components: documentation, regionserver
Reporter: Karthik Ranganathan
Assignee: Karthik Ranganathan

 This copier is a modification of the distcp tool in HDFS. It does the 
 following:
 1. List out all the regions in the HBase cluster for the required table
 2. Write the above out to a file
 3. Each mapper 
3.1 lists all the HFiles for a given region by querying the regionserver
3.2 copies all the HFiles
3.3 outputs success if the copy succeeded, failure otherwise. Failed 
 regions are retried in another loop
 4. Mappers are placed on nodes which have maximum locality for a given region 
 to speed up copying

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6050) HLogSplitter renaming recovered.edits and CJ removing the parent directory races, making the HBCK to think cluster is inconsistent.


[ 
https://issues.apache.org/jira/browse/HBASE-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281686#comment-13281686
 ] 

Zhihong Yu commented on HBASE-6050:
---

Patch looks good.
Minor:
Please insert spaces around regionDir:
{code}
+ to destination  +regionDir+  as it doesn't exist.);
{code}

 HLogSplitter renaming recovered.edits and CJ removing the parent directory 
 races, making the HBCK to think cluster is inconsistent.
 ---

 Key: HBASE-6050
 URL: https://issues.apache.org/jira/browse/HBASE-6050
 Project: HBase
  Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
 Attachments: HBASE-6050.patch


 The scenario is like this
 - A region is getting splitted.
 - The master is still not processed the split .
 - Region server goes down.
 - Split log manager starts splitting the logs and creates the 
 recovered.edits in the splitlog path.
 - CJ starts and deletes the entry from META and also just completes the 
 deletion of the region dir.
 - in hlogSplitter on final step we rename the recovered.edits to come under 
 the regiondir.
 There if the regiondir doesnot exist we tend to create and then add the 
 recovered.edits.
 Because of this HBCK thinks it to be an orphan region because we have the 
 regiondir but with no regioninfo.
 Ideally cluster is fine but we it is misleading.
 {code}
 } else {
   Path dstdir = dst.getParent();
   if (!fs.exists(dstdir)) {
 if (!fs.mkdirs(dstdir)) LOG.warn(mkdir failed on  + dstdir);
   }
 }
 fs.rename(src, dst);
 LOG.debug( moved  + src +  =  + dst);
   } else {
 LOG.debug(Could not move recovered edits from  + src +
  as it doesn't exist);
   }
 }
 archiveLogs(null, corruptedLogs, processedLogs,
 oldLogDir, fs, conf);
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6069) TableInputFormatBase#createRecordReader() doesn't initialize TableRecordReader which causes NPE


[ 
https://issues.apache.org/jira/browse/HBASE-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281694#comment-13281694
 ] 

Zhihong Yu commented on HBASE-6069:
---

Integrated to 0.94 and trunk.

Thanks for the patch, Jie.

 TableInputFormatBase#createRecordReader() doesn't initialize 
 TableRecordReader which causes NPE
 ---

 Key: HBASE-6069
 URL: https://issues.apache.org/jira/browse/HBASE-6069
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 0.94.0
Reporter: Jie Huang
Assignee: Jie Huang
Priority: Critical
 Fix For: 0.96.0, 0.94.1

 Attachments: 6069-v2.txt, HBASE-6069.patch


 While running Hive(0.9.0) query over HBase(0.94.0) with hive-hbase-handler, 
 there always throws a Null Pointer Exception on Scanner object. Since the 
 TableInputFormatBase#createRecordReader() missed the initialization of 
 TableRecordReader object. The scanner will be null in that case. This issue 
 causes Hive query fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6072) Make TableRecordReaderImpl more easily extended

2012-05-23 Thread Dave Latham (JIRA)

Dave Latham created HBASE-6072:
--

 Summary: Make TableRecordReaderImpl more easily extended
 Key: HBASE-6072
 URL: https://issues.apache.org/jira/browse/HBASE-6072
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Reporter: Dave Latham
Priority: Minor


We have a MR job that is very memory bound.  It reads a potentially large row 
from hbase, then deserializes it into an (even larger) object representation, 
then does a fair amount of computation requiring memory.  After converting the 
Result into our object representation we want to free the memory holding the 
Result to be available for the actual computation of output values.

Currently we have our own custom modified copy of TableRecordReaderImpl to be 
able to set the Result value to null after reading it, but it's almost entirely 
a duplicate of hbase's TableRecordReaderImpl so we have to manually keep it up 
to date with changes to the hbase version.  If the value field of 
TableRecordReaderImpl were protected instead of private we could use a very 
simple subclass instead.

Are there any philosophical guidelines about what parts of HBase should or 
should not be easily extensible?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6069) TableInputFormatBase#createRecordReader() doesn't initialize TableRecordReader which causes NPE

[
https://issues.apache.org/jira/browse/HBASE-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281711#comment-13281711
]

Hadoop QA commented on HBASE-6069:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12528745/6069-v2.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.

+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 33 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1965//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1965//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1965//console

This message is automatically generated.

TableInputFormatBase#createRecordReader() doesn't initialize
TableRecordReader which causes NPE
---

Key: HBASE-6069
URL: https://issues.apache.org/jira/browse/HBASE-6069
Project: HBase
Issue Type: Bug
Components: mapreduce
Affects Versions: 0.94.0
Reporter: Jie Huang
Assignee: Jie Huang
Priority: Critical
Fix For: 0.96.0, 0.94.1

Attachments: 6069-v2.txt, HBASE-6069.patch

While running Hive(0.9.0) query over HBase(0.94.0) with hive-hbase-handler,
there always throws a Null Pointer Exception on Scanner object. Since the
TableInputFormatBase#createRecordReader() missed the initialization of
TableRecordReader object. The scanner will be null in that case. This issue
causes Hive query fails.

[jira] [Commented] (HBASE-6055) Snapshots in HBase 0.96


[ 
https://issues.apache.org/jira/browse/HBASE-6055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281715#comment-13281715
 ] 

Jimmy Xiang commented on HBASE-6055:


I have a concern.  Why should we do two phases?  I think the prepare phase is 
not needed.  We have row level atomicity.  We don't need every region server to 
be on the same page.  Since it is distributed, it is arguable about the meaning 
of point-in-time. That means it is hard to say it is consistent/inconsistent 
point-in-time.

I think we each region server can try to create the snapshot at first.  If 
anyone fails, partial snapshot can be just deleted.

 Snapshots in HBase 0.96
 ---

 Key: HBASE-6055
 URL: https://issues.apache.org/jira/browse/HBASE-6055
 Project: HBase
  Issue Type: New Feature
  Components: client, master, regionserver, zookeeper
Reporter: Jesse Yates
Assignee: Jesse Yates
 Fix For: 0.96.0

 Attachments: Snapshots in HBase.docx


 Continuation of HBASE-50 for the current trunk. Since the implementation has 
 drastically changed, opening as a new ticket.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6072) Make TableRecordReaderImpl more easily extended


[ 
https://issues.apache.org/jira/browse/HBASE-6072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281723#comment-13281723
 ] 

Zhihong Yu commented on HBASE-6072:
---

Instead of making value field protected, we can also add method resetValue() 
which clears the field.

@Dave:
What do you think ?

 Make TableRecordReaderImpl more easily extended
 ---

 Key: HBASE-6072
 URL: https://issues.apache.org/jira/browse/HBASE-6072
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Reporter: Dave Latham
Priority: Minor

 We have a MR job that is very memory bound.  It reads a potentially large row 
 from hbase, then deserializes it into an (even larger) object representation, 
 then does a fair amount of computation requiring memory.  After converting 
 the Result into our object representation we want to free the memory holding 
 the Result to be available for the actual computation of output values.
 Currently we have our own custom modified copy of TableRecordReaderImpl to be 
 able to set the Result value to null after reading it, but it's almost 
 entirely a duplicate of hbase's TableRecordReaderImpl so we have to manually 
 keep it up to date with changes to the hbase version.  If the value field of 
 TableRecordReaderImpl were protected instead of private we could use a very 
 simple subclass instead.
 Are there any philosophical guidelines about what parts of HBase should or 
 should not be easily extensible?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6065) Log for flush would append a non-sequential edit in the hlog， may cause data loss


[ 
https://issues.apache.org/jira/browse/HBASE-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281726#comment-13281726
 ] 

Zhihong Yu commented on HBASE-6065:
---

Will integrate patch v2 later today if there is no objection.

 Log for flush would append a non-sequential edit in the hlog， may cause data 
 loss
 -

 Key: HBASE-6065
 URL: https://issues.apache.org/jira/browse/HBASE-6065
 Project: HBase
  Issue Type: Bug
  Components: wal
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.96.0, 0.94.1

 Attachments: HBASE-6065.patch, HBASE-6065v2.patch


 After completing flush region, we will append a log edit in the hlog file 
 through HLog#completeCacheFlush.
 {code}
 public void completeCacheFlush(final byte [] encodedRegionName,
   final byte [] tableName, final long logSeqId, final boolean 
 isMetaRegion)
 {
 ...
 HLogKey key = makeKey(encodedRegionName, tableName, logSeqId,
 System.currentTimeMillis(), HConstants.DEFAULT_CLUSTER_ID);
 ...
 }
 {code}
 when we make the hlog key, we use the seqId from the parameter, and it is 
 generated by HLog#startCacheFlush,
 Here, we may append a lower seq id edit than the last edit in the hlog file.
 If it is the last edit log in the file, it may cause data loss.
 because 
 {code}
 HRegion#replayRecoveredEditsIfAny{
 ...
 maxSeqId = Math.abs(Long.parseLong(fileName));
   if (maxSeqId = minSeqId) {
 String msg = Maximum sequenceid for this log is  + maxSeqId
 +  and minimum sequenceid for the region is  + minSeqId
 + , skipped the whole file, path= + edits;
 LOG.debug(msg);
 continue;
   }
 ...
 }
 {code}
 We may skip the splitted log file, because we use the lase edit's seq id as 
 its file name, and consider this seqId as the max seq id in this log file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6072) Make TableRecordReaderImpl more easily extended

2012-05-23 Thread Dave Latham (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281740#comment-13281740
 ] 

Dave Latham commented on HBASE-6072:


That works great for my case.

I am still curious about the more general question of what should be 
extensible, but am content with a specific fix for me if people feel its 
appropriate to belong in the shared code.

 Make TableRecordReaderImpl more easily extended
 ---

 Key: HBASE-6072
 URL: https://issues.apache.org/jira/browse/HBASE-6072
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Reporter: Dave Latham
Priority: Minor

 We have a MR job that is very memory bound.  It reads a potentially large row 
 from hbase, then deserializes it into an (even larger) object representation, 
 then does a fair amount of computation requiring memory.  After converting 
 the Result into our object representation we want to free the memory holding 
 the Result to be available for the actual computation of output values.
 Currently we have our own custom modified copy of TableRecordReaderImpl to be 
 able to set the Result value to null after reading it, but it's almost 
 entirely a duplicate of hbase's TableRecordReaderImpl so we have to manually 
 keep it up to date with changes to the hbase version.  If the value field of 
 TableRecordReaderImpl were protected instead of private we could use a very 
 simple subclass instead.
 Are there any philosophical guidelines about what parts of HBase should or 
 should not be easily extensible?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6073) Add support for scan filters in Thrift2

2012-05-23 Thread Jay Talreja (JIRA)

Jay Talreja created HBASE-6073:
--

 Summary: Add support for scan filters in Thrift2
 Key: HBASE-6073
 URL: https://issues.apache.org/jira/browse/HBASE-6073
 Project: HBase
  Issue Type: New Feature
  Components: thrift
Affects Versions: 0.94.0
Reporter: Jay Talreja


With HBase 0.94 a new thrift API was added (thrift2). This API is more akin to 
the Java HBase API. Thrift (version1) had added filterString to the TScan 
struct as part of HBase release 0.92 . Thrift2 TScan object doesn't have 
filterString parameter. Hence executing server side filters using thrift2 API 
is currently not possible. 

It would be great to have filtering capabilities added to TScan struct in 
thrift2 as well to maintain feature compatibility between two thrift versions. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6067) HBase won't start when hbase.rootdir uses ViewFileSystem

2012-05-23 Thread Eli Collins (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281745#comment-13281745
 ] 

Eli Collins commented on HBASE-6067:


Should mentioned that I considered defining hbase.regionserver.hlog.blocksize 
and hbase.regionserver.hlog.tolerable.lowreplication in hbase-default.xml and 
use constants in the code like the other parameters but then (1) you can have a 
mismatch between hbase and hdfs' parameter values and (2) HBase supports 
non-HDFS file systems which may want different default values.

 HBase won't start when hbase.rootdir uses ViewFileSystem
 

 Key: HBASE-6067
 URL: https://issues.apache.org/jira/browse/HBASE-6067
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Eli Collins

 HBase currently doesn't work with HDFS federation (hbase.rootdir with a 
 client that uses viewfs) because HLog#init uses 
 FileSystem#getDefaultBlockSize and getDefaultReplication. These throw an 
 exception because there is no default filesystem in a viewfs client so 
 there's no way to determine a default block size or replication factor. They 
 could use the versions of these methods that take a path, however these were 
 introduced in HADOOP-8014 and are not yet available in Hadoop 1.x.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6069) TableInputFormatBase#createRecordReader() doesn't initialize TableRecordReader which causes NPE


[ 
https://issues.apache.org/jira/browse/HBASE-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281769#comment-13281769
 ] 

Hudson commented on HBASE-6069:
---

Integrated in HBase-0.94 #210 (See 
[https://builds.apache.org/job/HBase-0.94/210/])
HBASE-6069 TableInputFormatBase#createRecordReader() doesn't initialize 
TableRecordReader which causes NPE (Jie Huang) (Revision 1341919)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java


 TableInputFormatBase#createRecordReader() doesn't initialize 
 TableRecordReader which causes NPE
 ---

 Key: HBASE-6069
 URL: https://issues.apache.org/jira/browse/HBASE-6069
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 0.94.0
Reporter: Jie Huang
Assignee: Jie Huang
Priority: Critical
 Fix For: 0.96.0, 0.94.1

 Attachments: 6069-v2.txt, HBASE-6069.patch


 While running Hive(0.9.0) query over HBase(0.94.0) with hive-hbase-handler, 
 there always throws a Null Pointer Exception on Scanner object. Since the 
 TableInputFormatBase#createRecordReader() missed the initialization of 
 TableRecordReader object. The scanner will be null in that case. This issue 
 causes Hive query fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6069) TableInputFormatBase#createRecordReader() doesn't initialize TableRecordReader which causes NPE


[ 
https://issues.apache.org/jira/browse/HBASE-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281772#comment-13281772
 ] 

Hudson commented on HBASE-6069:
---

Integrated in HBase-TRUNK #2916 (See 
[https://builds.apache.org/job/HBase-TRUNK/2916/])
HBASE-6069 TableInputFormatBase#createRecordReader() doesn't initialize 
TableRecordReader which causes NPE (Jie Huang) (Revision 1341922)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java


 TableInputFormatBase#createRecordReader() doesn't initialize 
 TableRecordReader which causes NPE
 ---

 Key: HBASE-6069
 URL: https://issues.apache.org/jira/browse/HBASE-6069
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 0.94.0
Reporter: Jie Huang
Assignee: Jie Huang
Priority: Critical
 Fix For: 0.96.0, 0.94.1

 Attachments: 6069-v2.txt, HBASE-6069.patch


 While running Hive(0.9.0) query over HBase(0.94.0) with hive-hbase-handler, 
 there always throws a Null Pointer Exception on Scanner object. Since the 
 TableInputFormatBase#createRecordReader() missed the initialization of 
 TableRecordReader object. The scanner will be null in that case. This issue 
 causes Hive query fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6074) TestHLog is flaky

2012-05-23 Thread Devaraj Das (JIRA)

Devaraj Das created HBASE-6074:
--

 Summary: TestHLog is flaky
 Key: HBASE-6074
 URL: https://issues.apache.org/jira/browse/HBASE-6074
 Project: HBase
  Issue Type: Test
  Components: test
Affects Versions: 0.92.0
Reporter: Devaraj Das


When I run TestHLog in a loop, I see failures.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6067) HBase won't start when hbase.rootdir uses ViewFileSystem

2012-05-23 Thread Daryn Sharp (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281807#comment-13281807
 ] 

Daryn Sharp commented on HBASE-6067:


Does {{hbase.rootdir}} point to just the root directory of viewfs, or is it a 
lower level directory?  If the latter, I'm assuming this is how you could use 
the path-based variants after we fix 1.x?

 HBase won't start when hbase.rootdir uses ViewFileSystem
 

 Key: HBASE-6067
 URL: https://issues.apache.org/jira/browse/HBASE-6067
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Eli Collins

 HBase currently doesn't work with HDFS federation (hbase.rootdir with a 
 client that uses viewfs) because HLog#init uses 
 FileSystem#getDefaultBlockSize and getDefaultReplication. These throw an 
 exception because there is no default filesystem in a viewfs client so 
 there's no way to determine a default block size or replication factor. They 
 could use the versions of these methods that take a path, however these were 
 introduced in HADOOP-8014 and are not yet available in Hadoop 1.x.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6043) Add Increment Coalescing in thrift.


 [ 
https://issues.apache.org/jira/browse/HBASE-6043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-6043:
-

Attachment: HBASE-6043-4.patch

Smaller sleep for the tests.  I ran it 20 times and never saw a failure.  Not 
sure why I needed the 6 seconds before but seems like this is long enough.

 Add Increment Coalescing in thrift.
 ---

 Key: HBASE-6043
 URL: https://issues.apache.org/jira/browse/HBASE-6043
 Project: HBase
  Issue Type: Improvement
Reporter: Elliott Clark
Assignee: Elliott Clark
 Attachments: HBASE-6043-0.patch, HBASE-6043-1.patch, 
 HBASE-6043-2.patch, HBASE-6043-3.patch, HBASE-6043-4.patch


 Since the thrift server uses the client api reducing the number of rpc's 
 greatly speeds up increments.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6059) Replaying recovered edits would make deleted data exist again


 [ 
https://issues.apache.org/jira/browse/HBASE-6059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-6059:
--

Attachment: 6059v6.txt

Patch v6 modifies the comment in TestStore.java

 Replaying recovered edits would make deleted data exist again
 -

 Key: HBASE-6059
 URL: https://issues.apache.org/jira/browse/HBASE-6059
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Reporter: chunhui shen
Assignee: chunhui shen
 Attachments: 6059v6.txt, HBASE-6059-testcase.patch, HBASE-6059.patch, 
 HBASE-6059v2.patch, HBASE-6059v3.patch, HBASE-6059v4.patch, HBASE-6059v5.patch


 When we replay recovered edits, we used the minSeqId of Store, It may cause 
 deleted data appeared again.
 Let's see how it happens. Suppose the region with two families(cf1,cf2)
 1.put one data to the region (put r1,cf1:q1,v1)
 2.move the region from server A to server B.
 3.delete the data put by step 1(delete r1)
 4.flush this region.
 5.make major compaction for this region
 6.move the region from server B to server A.
 7.Abort server A
 8.After the region is online, we could get the deleted data(r1,cf1:q1,v1)
 (When we replay recovered edits, we used the minSeqId of Store, because cf2 
 has no store files, so its seqId is 0, so the edit log of put data will be 
 replayed to the region)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6065) Log for flush would append a non-sequential edit in the hlog, leading to possible data loss


 [ 
https://issues.apache.org/jira/browse/HBASE-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-6065:
--

Summary: Log for flush would append a non-sequential edit in the hlog, 
leading to possible data loss  (was: Log for flush would append a 
non-sequential edit in the hlog， may cause data loss)

 Log for flush would append a non-sequential edit in the hlog, leading to 
 possible data loss
 ---

 Key: HBASE-6065
 URL: https://issues.apache.org/jira/browse/HBASE-6065
 Project: HBase
  Issue Type: Bug
  Components: wal
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.96.0, 0.94.1

 Attachments: HBASE-6065.patch, HBASE-6065v2.patch


 After completing flush region, we will append a log edit in the hlog file 
 through HLog#completeCacheFlush.
 {code}
 public void completeCacheFlush(final byte [] encodedRegionName,
   final byte [] tableName, final long logSeqId, final boolean 
 isMetaRegion)
 {
 ...
 HLogKey key = makeKey(encodedRegionName, tableName, logSeqId,
 System.currentTimeMillis(), HConstants.DEFAULT_CLUSTER_ID);
 ...
 }
 {code}
 when we make the hlog key, we use the seqId from the parameter, and it is 
 generated by HLog#startCacheFlush,
 Here, we may append a lower seq id edit than the last edit in the hlog file.
 If it is the last edit log in the file, it may cause data loss.
 because 
 {code}
 HRegion#replayRecoveredEditsIfAny{
 ...
 maxSeqId = Math.abs(Long.parseLong(fileName));
   if (maxSeqId = minSeqId) {
 String msg = Maximum sequenceid for this log is  + maxSeqId
 +  and minimum sequenceid for the region is  + minSeqId
 + , skipped the whole file, path= + edits;
 LOG.debug(msg);
 continue;
   }
 ...
 }
 {code}
 We may skip the splitted log file, because we use the lase edit's seq id as 
 its file name, and consider this seqId as the max seq id in this log file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6065) Log for flush would append a non-sequential edit in the hlog, leading to possible data loss


[ 
https://issues.apache.org/jira/browse/HBASE-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281868#comment-13281868
 ] 

Zhihong Yu commented on HBASE-6065:
---

Integrated to 0.94 and trunk.

Thanks for the patch Chunhui.

Thanks for the review, Ramkrishna.

 Log for flush would append a non-sequential edit in the hlog, leading to 
 possible data loss
 ---

 Key: HBASE-6065
 URL: https://issues.apache.org/jira/browse/HBASE-6065
 Project: HBase
  Issue Type: Bug
  Components: wal
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.96.0, 0.94.1

 Attachments: HBASE-6065.patch, HBASE-6065v2.patch


 After completing flush region, we will append a log edit in the hlog file 
 through HLog#completeCacheFlush.
 {code}
 public void completeCacheFlush(final byte [] encodedRegionName,
   final byte [] tableName, final long logSeqId, final boolean 
 isMetaRegion)
 {
 ...
 HLogKey key = makeKey(encodedRegionName, tableName, logSeqId,
 System.currentTimeMillis(), HConstants.DEFAULT_CLUSTER_ID);
 ...
 }
 {code}
 when we make the hlog key, we use the seqId from the parameter, and it is 
 generated by HLog#startCacheFlush,
 Here, we may append a lower seq id edit than the last edit in the hlog file.
 If it is the last edit log in the file, it may cause data loss.
 because 
 {code}
 HRegion#replayRecoveredEditsIfAny{
 ...
 maxSeqId = Math.abs(Long.parseLong(fileName));
   if (maxSeqId = minSeqId) {
 String msg = Maximum sequenceid for this log is  + maxSeqId
 +  and minimum sequenceid for the region is  + minSeqId
 + , skipped the whole file, path= + edits;
 LOG.debug(msg);
 continue;
   }
 ...
 }
 {code}
 We may skip the splitted log file, because we use the lase edit's seq id as 
 its file name, and consider this seqId as the max seq id in this log file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6067) HBase won't start when hbase.rootdir uses ViewFileSystem

2012-05-23 Thread Eli Collins (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281878#comment-13281878
 ] 

Eli Collins commented on HBASE-6067:


It points to a the hbase directory on a viewfs mount (the URI is 
viewfs://cluster2/hbase).

Per HADOOP-8422 I think we should backport the new methods to branch-1 and 
modify HBase here to use them. This means the next HBase release with this 
patch would require Hadoop 1.1 or later. This might be reasonable given that 
Hadoop 1.1 is the first 1.x release that will support HBase out of the box, 
obviously would like the HBase guys input on this before proceeding.

 HBase won't start when hbase.rootdir uses ViewFileSystem
 

 Key: HBASE-6067
 URL: https://issues.apache.org/jira/browse/HBASE-6067
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Eli Collins

 HBase currently doesn't work with HDFS federation (hbase.rootdir with a 
 client that uses viewfs) because HLog#init uses 
 FileSystem#getDefaultBlockSize and getDefaultReplication. These throw an 
 exception because there is no default filesystem in a viewfs client so 
 there's no way to determine a default block size or replication factor. They 
 could use the versions of these methods that take a path, however these were 
 introduced in HADOOP-8014 and are not yet available in Hadoop 1.x.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-6067) HBase won't start when hbase.rootdir uses ViewFileSystem

2012-05-23 Thread Eli Collins (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eli Collins reassigned HBASE-6067:
--

Assignee: Eli Collins

 HBase won't start when hbase.rootdir uses ViewFileSystem
 

 Key: HBASE-6067
 URL: https://issues.apache.org/jira/browse/HBASE-6067
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Eli Collins
Assignee: Eli Collins

 HBase currently doesn't work with HDFS federation (hbase.rootdir with a 
 client that uses viewfs) because HLog#init uses 
 FileSystem#getDefaultBlockSize and getDefaultReplication. These throw an 
 exception because there is no default filesystem in a viewfs client so 
 there's no way to determine a default block size or replication factor. They 
 could use the versions of these methods that take a path, however these were 
 introduced in HADOOP-8014 and are not yet available in Hadoop 1.x.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6067) HBase won't start when hbase.rootdir uses ViewFileSystem


[ 
https://issues.apache.org/jira/browse/HBASE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281894#comment-13281894
 ] 

Zhihong Yu commented on HBASE-6067:
---

+1 on backporting new methods.

We can use reflection to detect the existence of the new method:
{code}
+  public long getDefaultBlockSize(Path f) {
{code}
If the above method exists, we pass hbase rootdir to it. Otherwise fall back to 
calling getDefaultBlockSize().

 HBase won't start when hbase.rootdir uses ViewFileSystem
 

 Key: HBASE-6067
 URL: https://issues.apache.org/jira/browse/HBASE-6067
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Eli Collins
Assignee: Eli Collins

 HBase currently doesn't work with HDFS federation (hbase.rootdir with a 
 client that uses viewfs) because HLog#init uses 
 FileSystem#getDefaultBlockSize and getDefaultReplication. These throw an 
 exception because there is no default filesystem in a viewfs client so 
 there's no way to determine a default block size or replication factor. They 
 could use the versions of these methods that take a path, however these were 
 introduced in HADOOP-8014 and are not yet available in Hadoop 1.x.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6074) TestHLog is flaky


[ 
https://issues.apache.org/jira/browse/HBASE-6074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281901#comment-13281901
 ] 

Zhihong Yu commented on HBASE-6074:
---

@Devaraj:
Did you observe failure based on hadoop 1.0 ?

What sub-tests failed ?

Thanks

 TestHLog is flaky
 -

 Key: HBASE-6074
 URL: https://issues.apache.org/jira/browse/HBASE-6074
 Project: HBase
  Issue Type: Test
  Components: test
Affects Versions: 0.92.0
Reporter: Devaraj Das

 When I run TestHLog in a loop, I see failures.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6069) TableInputFormatBase#createRecordReader() doesn't initialize TableRecordReader which causes NPE


 [ 
https://issues.apache.org/jira/browse/HBASE-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-6069:
--

Resolution: Fixed
Status: Resolved  (was: Patch Available)

 TableInputFormatBase#createRecordReader() doesn't initialize 
 TableRecordReader which causes NPE
 ---

 Key: HBASE-6069
 URL: https://issues.apache.org/jira/browse/HBASE-6069
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 0.94.0
Reporter: Jie Huang
Assignee: Jie Huang
Priority: Critical
 Fix For: 0.96.0, 0.94.1

 Attachments: 6069-v2.txt, HBASE-6069.patch


 While running Hive(0.9.0) query over HBase(0.94.0) with hive-hbase-handler, 
 there always throws a Null Pointer Exception on Scanner object. Since the 
 TableInputFormatBase#createRecordReader() missed the initialization of 
 TableRecordReader object. The scanner will be null in that case. This issue 
 causes Hive query fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6043) Add Increment Coalescing in thrift.

[
https://issues.apache.org/jira/browse/HBASE-6043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281905#comment-13281905
]

Hadoop QA commented on HBASE-6043:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12528771/HBASE-6043-4.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 35 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1966//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1966//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1966//console

This message is automatically generated.

Add Increment Coalescing in thrift.
---

Key: HBASE-6043
URL: https://issues.apache.org/jira/browse/HBASE-6043
Project: HBase
Issue Type: Improvement
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HBASE-6043-0.patch, HBASE-6043-1.patch,
HBASE-6043-2.patch, HBASE-6043-3.patch, HBASE-6043-4.patch

Since the thrift server uses the client api reducing the number of rpc's
greatly speeds up increments.

[jira] [Updated] (HBASE-6043) Add Increment Coalescing in thrift.


 [ 
https://issues.apache.org/jira/browse/HBASE-6043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-6043:
-

Attachment: HBASE-6043-5.patch

Added comments about potential data loss if thrift server crashes with 
coalescing increment turned on.

 Add Increment Coalescing in thrift.
 ---

 Key: HBASE-6043
 URL: https://issues.apache.org/jira/browse/HBASE-6043
 Project: HBase
  Issue Type: Improvement
Reporter: Elliott Clark
Assignee: Elliott Clark
 Attachments: HBASE-6043-0.patch, HBASE-6043-1.patch, 
 HBASE-6043-2.patch, HBASE-6043-3.patch, HBASE-6043-4.patch, HBASE-6043-5.patch


 Since the thrift server uses the client api reducing the number of rpc's 
 greatly speeds up increments.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5959) Add other load balancers


 [ 
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-5959:
-

Attachment: HBASE-5959-8.patch

Added configurable cache timeouts for region location.

 Add other load balancers
 

 Key: HBASE-5959
 URL: https://issues.apache.org/jira/browse/HBASE-5959
 Project: HBase
  Issue Type: New Feature
  Components: master
Affects Versions: 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, 
 HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, 
 HBASE-5959-7.patch, HBASE-5959-8.patch, HBASE-5959.D3189.1.patch, 
 HBASE-5959.D3189.2.patch, HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch, 
 HBASE-5959.D3189.5.patch, HBASE-5959.D3189.6.patch, HBASE-5959.D3189.7.patch


 Now that balancers are pluggable we should give some options.b

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5565) Refactoring doMiniBatchPut()

2012-05-23 Thread Lars Hofhansl (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281915#comment-13281915
 ] 

Lars Hofhansl commented on HBASE-5565:
--

@Scott: Does you offer still stand. I haven't gotten to this, yet.

 Refactoring doMiniBatchPut()
 

 Key: HBASE-5565
 URL: https://issues.apache.org/jira/browse/HBASE-5565
 Project: HBase
  Issue Type: Improvement
Reporter: Scott Chen
Assignee: Lars Hofhansl
 Fix For: 0.96.0


 HBASE-5542 introduce a RowProcessor interface that can handle multi-row 
 transaction.
 It will make the code cleaner if we can make doMiniBatchPut() to be covered 
 by RowProcessor.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6059) Replaying recovered edits would make deleted data exist again

[
https://issues.apache.org/jira/browse/HBASE-6059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281917#comment-13281917
]

Hadoop QA commented on HBASE-6059:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12528774/6059v6.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 9 new or modified tests.

+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 34 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplication
org.apache.hadoop.hbase.master.TestSplitLogManager
org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
org.apache.hadoop.hbase.replication.TestMasterReplication

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1967//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1967//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1967//console

This message is automatically generated.

Replaying recovered edits would make deleted data exist again
-

Key: HBASE-6059
URL: https://issues.apache.org/jira/browse/HBASE-6059
Project: HBase
Issue Type: Bug
Components: regionserver
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: 6059v6.txt, HBASE-6059-testcase.patch, HBASE-6059.patch,
HBASE-6059v2.patch, HBASE-6059v3.patch, HBASE-6059v4.patch, HBASE-6059v5.patch

[jira] [Commented] (HBASE-5959) Add other load balancers

[
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281918#comment-13281918
]

Hadoop QA commented on HBASE-5959:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12528782/HBASE-5959-8.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 16 new or modified tests.

-1 patch. The patch command could not apply the patch.

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1968//console

This message is automatically generated.

Add other load balancers

Key: HBASE-5959
URL: https://issues.apache.org/jira/browse/HBASE-5959
Project: HBase
Issue Type: New Feature
Components: master
Affects Versions: 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch,
HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch,
HBASE-5959-7.patch, HBASE-5959-8.patch, HBASE-5959.D3189.1.patch,
HBASE-5959.D3189.2.patch, HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch,
HBASE-5959.D3189.5.patch, HBASE-5959.D3189.6.patch, HBASE-5959.D3189.7.patch

Now that balancers are pluggable we should give some options.b

[jira] [Commented] (HBASE-6059) Replaying recovered edits would make deleted data exist again


[ 
https://issues.apache.org/jira/browse/HBASE-6059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281919#comment-13281919
 ] 

Zhihong Yu commented on HBASE-6059:
---

I ran TestSplitLogManager with patch v6 and it passed.

 Replaying recovered edits would make deleted data exist again
 -

 Key: HBASE-6059
 URL: https://issues.apache.org/jira/browse/HBASE-6059
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Reporter: chunhui shen
Assignee: chunhui shen
 Attachments: 6059v6.txt, HBASE-6059-testcase.patch, HBASE-6059.patch, 
 HBASE-6059v2.patch, HBASE-6059v3.patch, HBASE-6059v4.patch, HBASE-6059v5.patch


 When we replay recovered edits, we used the minSeqId of Store, It may cause 
 deleted data appeared again.
 Let's see how it happens. Suppose the region with two families(cf1,cf2)
 1.put one data to the region (put r1,cf1:q1,v1)
 2.move the region from server A to server B.
 3.delete the data put by step 1(delete r1)
 4.flush this region.
 5.make major compaction for this region
 6.move the region from server B to server A.
 7.Abort server A
 8.After the region is online, we could get the deleted data(r1,cf1:q1,v1)
 (When we replay recovered edits, we used the minSeqId of Store, because cf2 
 has no store files, so its seqId is 0, so the edit log of put data will be 
 replayed to the region)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6033) Adding some fuction to check if a table/region is in compaction


 [ 
https://issues.apache.org/jira/browse/HBASE-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-6033:
---

Status: Open  (was: Patch Available)

 Adding some fuction to check if a table/region is in compaction
 ---

 Key: HBASE-6033
 URL: https://issues.apache.org/jira/browse/HBASE-6033
 Project: HBase
  Issue Type: New Feature
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: hbase-6033_v2.patch, table_ui.png


 This feature will be helpful to find out if a major compaction is going on.
 We can show if it is in any minor compaction too.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6033) Adding some fuction to check if a table/region is in compaction


 [ 
https://issues.apache.org/jira/browse/HBASE-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-6033:
---

Attachment: hbase-6033_v3.patch

Now unit test should be green.

 Adding some fuction to check if a table/region is in compaction
 ---

 Key: HBASE-6033
 URL: https://issues.apache.org/jira/browse/HBASE-6033
 Project: HBase
  Issue Type: New Feature
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: hbase-6033_v2.patch, hbase-6033_v3.patch, table_ui.png


 This feature will be helpful to find out if a major compaction is going on.
 We can show if it is in any minor compaction too.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6033) Adding some fuction to check if a table/region is in compaction


 [ 
https://issues.apache.org/jira/browse/HBASE-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-6033:
---

Status: Patch Available  (was: Open)

 Adding some fuction to check if a table/region is in compaction
 ---

 Key: HBASE-6033
 URL: https://issues.apache.org/jira/browse/HBASE-6033
 Project: HBase
  Issue Type: New Feature
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: hbase-6033_v2.patch, hbase-6033_v3.patch, table_ui.png


 This feature will be helpful to find out if a major compaction is going on.
 We can show if it is in any minor compaction too.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6075) Improve delete(Latest-timestamp) performance: consider adding a delete_next type

2012-05-23 Thread Amitanand Aiyer (JIRA)

Amitanand Aiyer created HBASE-6075:
--

 Summary: Improve delete(Latest-timestamp) performance: consider 
adding a delete_next type
 Key: HBASE-6075
 URL: https://issues.apache.org/jira/browse/HBASE-6075
 Project: HBase
  Issue Type: Brainstorming
Reporter: Amitanand Aiyer
Priority: Minor


Disclaimer: this will only work correctly if the application is not taking 
control of the timestamp. 

We have a version of deleteVersion, which deletes the last version, if no 
specific timestamp is specified  (i.e. timestamp is left as Long.MAX_VALUE)

On the server side, this translates to deleting the largest timestamped cell in 
the specified column. Which entails doing a get, and then a delete.

We don't seem to use this api a whole lot, so not a very high pri task. 

But, for systems that use the api. We might be able to make this much faster 
(as fast as the puts) by introducing a new delete type (say 
DELETE_NEXT_VERSION) that sorts right after put in the column, and just adding 
it as a put. The deleteTracker can be updated to keep track of this delete_next 
and accordingly delete the nextKV asked for.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6076) Improve h.r.global.memstore.upper(lower)Limit description

Alex Baranau created HBASE-6076:
---

 Summary: Improve h.r.global.memstore.upper(lower)Limit description
 Key: HBASE-6076
 URL: https://issues.apache.org/jira/browse/HBASE-6076
 Project: HBase
  Issue Type: Bug
  Components: documentation
Reporter: Alex Baranau
Assignee: Alex Baranau
Priority: Trivial


hbase.regionserver.global.memstore.upper(lower)Limit settings documentation 
(hbase-default.xml) may be misleading. It mentions that:
* flushes are forced *and updates are blocked* when memstore size reaches 
hbase.regionserver.global.memstore.upperLimit. In this case flushes are forced 
and updates are blocked until memstore size is less than 
hbase.regionserver.global.memstore.lowerLimit.

But it doesn't mention this:
* flushes are forced when memstore size hits 
hbase.regionserver.global.memstore.lowerLimit

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6077) Document the most common secure RPC troubleshooting resolutions

2012-05-23 Thread Andrew Purtell (JIRA)

Andrew Purtell created HBASE-6077:
-

 Summary: Document the most common secure RPC troubleshooting 
resolutions
 Key: HBASE-6077
 URL: https://issues.apache.org/jira/browse/HBASE-6077
 Project: HBase
  Issue Type: Task
  Components: documentation, security
Affects Versions: 0.92.2, 0.96.0, 0.94.1
Reporter: Andrew Purtell
Assignee: Andrew Purtell


See attached manual troubleshooting section update.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6076) Improve h.r.global.memstore.upper(lower)Limit description


 [ 
https://issues.apache.org/jira/browse/HBASE-6076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Baranau updated HBASE-6076:


Attachment: HBASE-6076.patch

Attached patch.

 Improve h.r.global.memstore.upper(lower)Limit description
 -

 Key: HBASE-6076
 URL: https://issues.apache.org/jira/browse/HBASE-6076
 Project: HBase
  Issue Type: Bug
  Components: documentation
Reporter: Alex Baranau
Assignee: Alex Baranau
Priority: Trivial
 Attachments: HBASE-6076.patch


 hbase.regionserver.global.memstore.upper(lower)Limit settings documentation 
 (hbase-default.xml) may be misleading. It mentions that:
 * flushes are forced *and updates are blocked* when memstore size reaches 
 hbase.regionserver.global.memstore.upperLimit. In this case flushes are 
 forced and updates are blocked until memstore size is less than 
 hbase.regionserver.global.memstore.lowerLimit.
 But it doesn't mention this:
 * flushes are forced when memstore size hits 
 hbase.regionserver.global.memstore.lowerLimit

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6065) Log for flush would append a non-sequential edit in the hlog, leading to possible data loss


[ 
https://issues.apache.org/jira/browse/HBASE-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281927#comment-13281927
 ] 

Hudson commented on HBASE-6065:
---

Integrated in HBase-0.94 #212 (See 
[https://builds.apache.org/job/HBase-0.94/212/])
HBASE-6065 Log for flush would append a non-sequential edit in the hlog, 
leading to possible data loss (Chunhui) (Revision 1342017)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java


 Log for flush would append a non-sequential edit in the hlog, leading to 
 possible data loss
 ---

 Key: HBASE-6065
 URL: https://issues.apache.org/jira/browse/HBASE-6065
 Project: HBase
  Issue Type: Bug
  Components: wal
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.96.0, 0.94.1

 Attachments: HBASE-6065.patch, HBASE-6065v2.patch


 After completing flush region, we will append a log edit in the hlog file 
 through HLog#completeCacheFlush.
 {code}
 public void completeCacheFlush(final byte [] encodedRegionName,
   final byte [] tableName, final long logSeqId, final boolean 
 isMetaRegion)
 {
 ...
 HLogKey key = makeKey(encodedRegionName, tableName, logSeqId,
 System.currentTimeMillis(), HConstants.DEFAULT_CLUSTER_ID);
 ...
 }
 {code}
 when we make the hlog key, we use the seqId from the parameter, and it is 
 generated by HLog#startCacheFlush,
 Here, we may append a lower seq id edit than the last edit in the hlog file.
 If it is the last edit log in the file, it may cause data loss.
 because 
 {code}
 HRegion#replayRecoveredEditsIfAny{
 ...
 maxSeqId = Math.abs(Long.parseLong(fileName));
   if (maxSeqId = minSeqId) {
 String msg = Maximum sequenceid for this log is  + maxSeqId
 +  and minimum sequenceid for the region is  + minSeqId
 + , skipped the whole file, path= + edits;
 LOG.debug(msg);
 continue;
   }
 ...
 }
 {code}
 We may skip the splitted log file, because we use the lase edit's seq id as 
 its file name, and consider this seqId as the max seq id in this log file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6077) Document the most common secure RPC troubleshooting resolutions

2012-05-23 Thread Andrew Purtell (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-6077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-6077:
--

Attachment: 6077.patch

 Document the most common secure RPC troubleshooting resolutions
 ---

 Key: HBASE-6077
 URL: https://issues.apache.org/jira/browse/HBASE-6077
 Project: HBase
  Issue Type: Task
  Components: documentation, security
Affects Versions: 0.92.2, 0.96.0, 0.94.1
Reporter: Andrew Purtell
Assignee: Andrew Purtell
 Attachments: 6077.patch


 See attached manual troubleshooting section update.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6076) Improve h.r.global.memstore.upper(lower)Limit description


 [ 
https://issues.apache.org/jira/browse/HBASE-6076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Baranau updated HBASE-6076:


Attachment: (was: HBASE-6076.patch)

 Improve h.r.global.memstore.upper(lower)Limit description
 -

 Key: HBASE-6076
 URL: https://issues.apache.org/jira/browse/HBASE-6076
 Project: HBase
  Issue Type: Bug
  Components: documentation
Reporter: Alex Baranau
Assignee: Alex Baranau
Priority: Trivial

 hbase.regionserver.global.memstore.upper(lower)Limit settings documentation 
 (hbase-default.xml) may be misleading. It mentions that:
 * flushes are forced *and updates are blocked* when memstore size reaches 
 hbase.regionserver.global.memstore.upperLimit. In this case flushes are 
 forced and updates are blocked until memstore size is less than 
 hbase.regionserver.global.memstore.lowerLimit.
 But it doesn't mention this:
 * flushes are forced when memstore size hits 
 hbase.regionserver.global.memstore.lowerLimit

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6076) Improve h.r.global.memstore.upper(lower)Limit description


 [ 
https://issues.apache.org/jira/browse/HBASE-6076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Baranau updated HBASE-6076:


Attachment: HBASE-6076.patch

 Improve h.r.global.memstore.upper(lower)Limit description
 -

 Key: HBASE-6076
 URL: https://issues.apache.org/jira/browse/HBASE-6076
 Project: HBase
  Issue Type: Bug
  Components: documentation
Reporter: Alex Baranau
Assignee: Alex Baranau
Priority: Trivial
 Attachments: HBASE-6076.patch


 hbase.regionserver.global.memstore.upper(lower)Limit settings documentation 
 (hbase-default.xml) may be misleading. It mentions that:
 * flushes are forced *and updates are blocked* when memstore size reaches 
 hbase.regionserver.global.memstore.upperLimit. In this case flushes are 
 forced and updates are blocked until memstore size is less than 
 hbase.regionserver.global.memstore.lowerLimit.
 But it doesn't mention this:
 * flushes are forced when memstore size hits 
 hbase.regionserver.global.memstore.lowerLimit

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5959) Add other load balancers


 [ 
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-5959:
-

Attachment: HBASE-5959-9.patch

git rebase.

 Add other load balancers
 

 Key: HBASE-5959
 URL: https://issues.apache.org/jira/browse/HBASE-5959
 Project: HBase
  Issue Type: New Feature
  Components: master
Affects Versions: 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, 
 HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, 
 HBASE-5959-7.patch, HBASE-5959-8.patch, HBASE-5959-9.patch, 
 HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, HBASE-5959.D3189.3.patch, 
 HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch, HBASE-5959.D3189.6.patch, 
 HBASE-5959.D3189.7.patch


 Now that balancers are pluggable we should give some options.b

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5969) HRI.getRegionName/AsString are inconsistent for regions with the old encoding scheme

2012-05-23 Thread Jean-Daniel Cryans (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281945#comment-13281945
 ] 

Jean-Daniel Cryans commented on HBASE-5969:
---

If it doesn't break anything, sure.

 HRI.getRegionName/AsString are inconsistent for regions with the old encoding 
 scheme
 

 Key: HBASE-5969
 URL: https://issues.apache.org/jira/browse/HBASE-5969
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6, 0.92.1
Reporter: Jean-Daniel Cryans
Assignee: Alex Newman

 HRI.getRegionName and getRegionNameAsString don't give consistent results for 
 regions created with the old encoding scheme. See Aravind's comment in 
 HBASE-5929 on how a region was appended with .1290583321 when its HRI was 
 queried with getRegionNameAsString and, once passed to HBA, wasn't able to 
 compact it (I checked .META. and it doesn't have that last part so it must 
 come from getRegionName).
 This one might be a little hard to fix if we're already dependent on the 
 broken behavior of getRegionName.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5948) Deprecate and remove the Avro gateway


 [ 
https://issues.apache.org/jira/browse/HBASE-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-5948:
-

Attachment: HBASE-5948-0.patch

Patch to deprecate.  Includes a small note in the api.

 Deprecate and remove the Avro gateway
 -

 Key: HBASE-5948
 URL: https://issues.apache.org/jira/browse/HBASE-5948
 Project: HBase
  Issue Type: Task
Affects Versions: 0.96.0
Reporter: Andrew Purtell
Priority: Blocker
 Attachments: HBASE-5948-0.patch


 Deprecate the Avro gateway in 0.94. Remove in 0.96. Made a blocker against 
 that release. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6043) Add Increment Coalescing in thrift.

[
https://issues.apache.org/jira/browse/HBASE-6043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281961#comment-13281961
]

Hadoop QA commented on HBASE-6043:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12528780/HBASE-6043-5.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 35 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1969//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1969//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1969//console

This message is automatically generated.

Add Increment Coalescing in thrift.
---

Since the thrift server uses the client api reducing the number of rpc's
greatly speeds up increments.

[jira] [Commented] (HBASE-6033) Adding some fuction to check if a table/region is in compaction

[
https://issues.apache.org/jira/browse/HBASE-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281963#comment-13281963
]

Hadoop QA commented on HBASE-6033:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12528784/hbase-6033_v3.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 2 new or modified tests.

+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 33 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplication

org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
org.apache.hadoop.hbase.replication.TestMasterReplication

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1970//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1970//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1970//console

This message is automatically generated.

Adding some fuction to check if a table/region is in compaction
---

Key: HBASE-6033
URL: https://issues.apache.org/jira/browse/HBASE-6033
Project: HBase
Issue Type: New Feature
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Attachments: hbase-6033_v2.patch, hbase-6033_v3.patch, table_ui.png

This feature will be helpful to find out if a major compaction is going on.
We can show if it is in any minor compaction too.

[jira] [Created] (HBASE-6078) [refGuide] ported and refactored bulk loading docs into RefGuide

Doug Meil created HBASE-6078:


 Summary: [refGuide] ported and refactored bulk loading docs into 
RefGuide
 Key: HBASE-6078
 URL: https://issues.apache.org/jira/browse/HBASE-6078
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil


* bulk-loading.xml
Deprecated stand-alone HTML page that contained bulk-loading docs.  Left the 
page in the repo for backward compatibility since so many people have linked to 
it over the years, but points to new Arch section.

* book.xml
Added new Arch section for bulk loading

* ops_mgt.xml
Significantly expanded documentation on ImportTsv.  
Added entry for completebulkload.
Added short entry on the Driver class (for MR jobs)


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6078) [refGuide] ported and refactored bulk loading docs into RefGuide


 [ 
https://issues.apache.org/jira/browse/HBASE-6078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-6078:
-

Attachment: hbase_hbase_6078.patch

 [refGuide] ported and refactored bulk loading docs into RefGuide
 

 Key: HBASE-6078
 URL: https://issues.apache.org/jira/browse/HBASE-6078
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
 Attachments: hbase_hbase_6078.patch


 * bulk-loading.xml
 Deprecated stand-alone HTML page that contained bulk-loading docs.  Left the 
 page in the repo for backward compatibility since so many people have linked 
 to it over the years, but points to new Arch section.
 * book.xml
 Added new Arch section for bulk loading
 * ops_mgt.xml
 Significantly expanded documentation on ImportTsv.  
 Added entry for completebulkload.
 Added short entry on the Driver class (for MR jobs)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6078) [refGuide] ported and refactored bulk loading docs into RefGuide


 [ 
https://issues.apache.org/jira/browse/HBASE-6078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-6078:
-

Status: Patch Available  (was: Open)

 [refGuide] ported and refactored bulk loading docs into RefGuide
 

 Key: HBASE-6078
 URL: https://issues.apache.org/jira/browse/HBASE-6078
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
 Attachments: hbase_hbase_6078.patch


 * bulk-loading.xml
 Deprecated stand-alone HTML page that contained bulk-loading docs.  Left the 
 page in the repo for backward compatibility since so many people have linked 
 to it over the years, but points to new Arch section.
 * book.xml
 Added new Arch section for bulk loading
 * ops_mgt.xml
 Significantly expanded documentation on ImportTsv.  
 Added entry for completebulkload.
 Added short entry on the Driver class (for MR jobs)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6078) [refGuide] ported and refactored bulk loading docs into RefGuide


 [ 
https://issues.apache.org/jira/browse/HBASE-6078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-6078:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

 [refGuide] ported and refactored bulk loading docs into RefGuide
 

 Key: HBASE-6078
 URL: https://issues.apache.org/jira/browse/HBASE-6078
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
 Attachments: hbase_hbase_6078.patch


 * bulk-loading.xml
 Deprecated stand-alone HTML page that contained bulk-loading docs.  Left the 
 page in the repo for backward compatibility since so many people have linked 
 to it over the years, but points to new Arch section.
 * book.xml
 Added new Arch section for bulk loading
 * ops_mgt.xml
 Significantly expanded documentation on ImportTsv.  
 Added entry for completebulkload.
 Added short entry on the Driver class (for MR jobs)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6065) Log for flush would append a non-sequential edit in the hlog, leading to possible data loss


[ 
https://issues.apache.org/jira/browse/HBASE-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281977#comment-13281977
 ] 

Hudson commented on HBASE-6065:
---

Integrated in HBase-TRUNK #2917 (See 
[https://builds.apache.org/job/HBase-TRUNK/2917/])
HBASE-6065 Log for flush would append a non-sequential edit in the hlog, 
leading to possible data loss (Chunhui) (Revision 1342019)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java


 Log for flush would append a non-sequential edit in the hlog, leading to 
 possible data loss
 ---

 Key: HBASE-6065
 URL: https://issues.apache.org/jira/browse/HBASE-6065
 Project: HBase
  Issue Type: Bug
  Components: wal
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.96.0, 0.94.1

 Attachments: HBASE-6065.patch, HBASE-6065v2.patch


 After completing flush region, we will append a log edit in the hlog file 
 through HLog#completeCacheFlush.
 {code}
 public void completeCacheFlush(final byte [] encodedRegionName,
   final byte [] tableName, final long logSeqId, final boolean 
 isMetaRegion)
 {
 ...
 HLogKey key = makeKey(encodedRegionName, tableName, logSeqId,
 System.currentTimeMillis(), HConstants.DEFAULT_CLUSTER_ID);
 ...
 }
 {code}
 when we make the hlog key, we use the seqId from the parameter, and it is 
 generated by HLog#startCacheFlush,
 Here, we may append a lower seq id edit than the last edit in the hlog file.
 If it is the last edit log in the file, it may cause data loss.
 because 
 {code}
 HRegion#replayRecoveredEditsIfAny{
 ...
 maxSeqId = Math.abs(Long.parseLong(fileName));
   if (maxSeqId = minSeqId) {
 String msg = Maximum sequenceid for this log is  + maxSeqId
 +  and minimum sequenceid for the region is  + minSeqId
 + , skipped the whole file, path= + edits;
 LOG.debug(msg);
 continue;
   }
 ...
 }
 {code}
 We may skip the splitted log file, because we use the lase edit's seq id as 
 its file name, and consider this seqId as the max seq id in this log file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6073) Add support for scan filters in Thrift2


[ 
https://issues.apache.org/jira/browse/HBASE-6073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281981#comment-13281981
 ] 

Zhihong Yu commented on HBASE-6073:
---

Please refer to HBASE-4176 for the patch for thrift.

 Add support for scan filters in Thrift2
 ---

 Key: HBASE-6073
 URL: https://issues.apache.org/jira/browse/HBASE-6073
 Project: HBase
  Issue Type: New Feature
  Components: thrift
Affects Versions: 0.94.0
Reporter: Jay Talreja
  Labels: thrift2

 With HBase 0.94 a new thrift API was added (thrift2). This API is more akin 
 to the Java HBase API. Thrift (version1) had added filterString to the TScan 
 struct as part of HBase release 0.92 . Thrift2 TScan object doesn't have 
 filterString parameter. Hence executing server side filters using thrift2 API 
 is currently not possible. 
 It would be great to have filtering capabilities added to TScan struct in 
 thrift2 as well to maintain feature compatibility between two thrift 
 versions. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-1801) Tool to clean regions on DFS

2012-05-23 Thread Jean-Daniel Cryans (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-1801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Daniel Cryans resolved HBASE-1801.
---

Resolution: Won't Fix

It's old as hell, closing.

 Tool to clean regions on DFS
 

 Key: HBASE-1801
 URL: https://issues.apache.org/jira/browse/HBASE-1801
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.19.3
Reporter: Jean-Daniel Cryans
 Attachments: clean_table.rb


 If you have a cluster running for some time, you probably have more regions 
 on DFS than in META. Here is a tool to remove them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-6076) Improve h.r.global.memstore.upper(lower)Limit description

2012-05-23 Thread Jean-Daniel Cryans (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-6076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Daniel Cryans resolved HBASE-6076.
---

Resolution: Fixed

Committed to trunk, thanks for taking care of this Alex.

 Improve h.r.global.memstore.upper(lower)Limit description
 -

 Key: HBASE-6076
 URL: https://issues.apache.org/jira/browse/HBASE-6076
 Project: HBase
  Issue Type: Bug
  Components: documentation
Reporter: Alex Baranau
Assignee: Alex Baranau
Priority: Trivial
 Attachments: HBASE-6076.patch


 hbase.regionserver.global.memstore.upper(lower)Limit settings documentation 
 (hbase-default.xml) may be misleading. It mentions that:
 * flushes are forced *and updates are blocked* when memstore size reaches 
 hbase.regionserver.global.memstore.upperLimit. In this case flushes are 
 forced and updates are blocked until memstore size is less than 
 hbase.regionserver.global.memstore.lowerLimit.
 But it doesn't mention this:
 * flushes are forced when memstore size hits 
 hbase.regionserver.global.memstore.lowerLimit

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6079) [refGuide] Bulk load porting cleanup

Doug Meil created HBASE-6079:


 Summary: [refGuide] Bulk load porting cleanup
 Key: HBASE-6079
 URL: https://issues.apache.org/jira/browse/HBASE-6079
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor


performance.xml
* changed link that pointed to external page on bulk loading to new section in 
Arch chapter

site.xml
* changed bulk load menu option to point to RefGuide instead of external page.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6079) [refGuide] Bulk load porting cleanup


 [ 
https://issues.apache.org/jira/browse/HBASE-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-6079:
-

Attachment: src_hbase_6079.patch

 [refGuide] Bulk load porting cleanup
 

 Key: HBASE-6079
 URL: https://issues.apache.org/jira/browse/HBASE-6079
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
 Attachments: src_hbase_6079.patch


 performance.xml
 * changed link that pointed to external page on bulk loading to new section 
 in Arch chapter
 site.xml
 * changed bulk load menu option to point to RefGuide instead of external page.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6079) [refGuide] Bulk load porting cleanup


 [ 
https://issues.apache.org/jira/browse/HBASE-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-6079:
-

Status: Patch Available  (was: Open)

 [refGuide] Bulk load porting cleanup
 

 Key: HBASE-6079
 URL: https://issues.apache.org/jira/browse/HBASE-6079
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
 Attachments: src_hbase_6079.patch


 performance.xml
 * changed link that pointed to external page on bulk loading to new section 
 in Arch chapter
 site.xml
 * changed bulk load menu option to point to RefGuide instead of external page.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5986) Clients can see holes in the META table when regions are being split


 [ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5986:
--

Attachment: HBASE-5986_v3.patch

Patch v3 from Enis.

 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Attachments: 5986-v2.txt, HBASE-5986-test_v1.patch, 
 HBASE-5986_v3.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6079) [refGuide] Bulk load porting cleanup


 [ 
https://issues.apache.org/jira/browse/HBASE-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-6079:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

 [refGuide] Bulk load porting cleanup
 

 Key: HBASE-6079
 URL: https://issues.apache.org/jira/browse/HBASE-6079
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
 Attachments: src_hbase_6079.patch


 performance.xml
 * changed link that pointed to external page on bulk loading to new section 
 in Arch chapter
 site.xml
 * changed bulk load menu option to point to RefGuide instead of external page.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6080) site.xml - adding ReviewBoard to main page left-hand nav

Doug Meil created HBASE-6080:


 Summary: site.xml - adding ReviewBoard to main page left-hand nav
 Key: HBASE-6080
 URL: https://issues.apache.org/jira/browse/HBASE-6080
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Trivial


By request, adding ReviewBoard to left-hand nav on website

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5959) Add other load balancers

[
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13282002#comment-13282002
]

Hadoop QA commented on HBASE-5959:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12528790/HBASE-5959-9.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 16 new or modified tests.

+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 35 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestMasterObserver
org.apache.hadoop.hbase.replication.TestReplication
org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
org.apache.hadoop.hbase.replication.TestMasterReplication

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1971//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1971//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1971//console

This message is automatically generated.

Add other load balancers

Key: HBASE-5959
URL: https://issues.apache.org/jira/browse/HBASE-5959
Project: HBase
Issue Type: New Feature
Components: master
Affects Versions: 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch,
HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch,
HBASE-5959-7.patch, HBASE-5959-8.patch, HBASE-5959-9.patch,
HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, HBASE-5959.D3189.3.patch,
HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch, HBASE-5959.D3189.6.patch,
HBASE-5959.D3189.7.patch

Now that balancers are pluggable we should give some options.b

[jira] [Updated] (HBASE-6080) site.xml - adding ReviewBoard to main page left-hand nav


 [ 
https://issues.apache.org/jira/browse/HBASE-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-6080:
-

Attachment: src_hbase_6080.patch

 site.xml - adding ReviewBoard to main page left-hand nav
 

 Key: HBASE-6080
 URL: https://issues.apache.org/jira/browse/HBASE-6080
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Trivial
 Attachments: src_hbase_6080.patch


 By request, adding ReviewBoard to left-hand nav on website

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6080) site.xml - adding ReviewBoard to main page left-hand nav


 [ 
https://issues.apache.org/jira/browse/HBASE-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-6080:
-

Status: Patch Available  (was: Open)

 site.xml - adding ReviewBoard to main page left-hand nav
 

 Key: HBASE-6080
 URL: https://issues.apache.org/jira/browse/HBASE-6080
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Trivial
 Attachments: src_hbase_6080.patch


 By request, adding ReviewBoard to left-hand nav on website

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6080) site.xml - adding ReviewBoard to main page left-hand nav


[ 
https://issues.apache.org/jira/browse/HBASE-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13282016#comment-13282016
 ] 

Hadoop QA commented on HBASE-6080:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12528807/src_hbase_6080.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

-1 patch.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1974//console

This message is automatically generated.

 site.xml - adding ReviewBoard to main page left-hand nav
 

 Key: HBASE-6080
 URL: https://issues.apache.org/jira/browse/HBASE-6080
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Trivial
 Attachments: src_hbase_6080.patch


 By request, adding ReviewBoard to left-hand nav on website

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6081) [refGuide] compression appendix - adding section on changing codecs

Doug Meil created HBASE-6081:


 Summary: [refGuide] compression appendix - adding section on 
changing codecs
 Key: HBASE-6081
 URL: https://issues.apache.org/jira/browse/HBASE-6081
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor


book.xml
* adding section in compression appendix about changing compression codecs. 
* A frequent question on the dist-list is whether people will have to copy the 
data into a new table, etc.,  You don't.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6043) Add Increment Coalescing in thrift.


 [ 
https://issues.apache.org/jira/browse/HBASE-6043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-6043:
-

Attachment: HBASE-6043-6.patch

More comments and turn the default to off.

 Add Increment Coalescing in thrift.
 ---

 Key: HBASE-6043
 URL: https://issues.apache.org/jira/browse/HBASE-6043
 Project: HBase
  Issue Type: Improvement
Reporter: Elliott Clark
Assignee: Elliott Clark
 Attachments: HBASE-6043-0.patch, HBASE-6043-1.patch, 
 HBASE-6043-2.patch, HBASE-6043-3.patch, HBASE-6043-4.patch, 
 HBASE-6043-5.patch, HBASE-6043-6.patch


 Since the thrift server uses the client api reducing the number of rpc's 
 greatly speeds up increments.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6081) [refGuide] compression appendix - adding section on changing codecs


 [ 
https://issues.apache.org/jira/browse/HBASE-6081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-6081:
-

Attachment: book_hbase_6081.xml.patch

 [refGuide] compression appendix - adding section on changing codecs
 ---

 Key: HBASE-6081
 URL: https://issues.apache.org/jira/browse/HBASE-6081
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
 Attachments: book_hbase_6081.xml.patch


 book.xml
 * adding section in compression appendix about changing compression codecs. 
 * A frequent question on the dist-list is whether people will have to copy 
 the data into a new table, etc.,  You don't.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6081) [refGuide] compression appendix - adding section on changing codecs


 [ 
https://issues.apache.org/jira/browse/HBASE-6081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-6081:
-

Status: Patch Available  (was: Open)

 [refGuide] compression appendix - adding section on changing codecs
 ---

 Key: HBASE-6081
 URL: https://issues.apache.org/jira/browse/HBASE-6081
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
 Attachments: book_hbase_6081.xml.patch


 book.xml
 * adding section in compression appendix about changing compression codecs. 
 * A frequent question on the dist-list is whether people will have to copy 
 the data into a new table, etc.,  You don't.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6081) [refGuide] compression appendix - adding section on changing codecs


 [ 
https://issues.apache.org/jira/browse/HBASE-6081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-6081:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

 [refGuide] compression appendix - adding section on changing codecs
 ---

 Key: HBASE-6081
 URL: https://issues.apache.org/jira/browse/HBASE-6081
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
 Attachments: book_hbase_6081.xml.patch


 book.xml
 * adding section in compression appendix about changing compression codecs. 
 * A frequent question on the dist-list is whether people will have to copy 
 the data into a new table, etc.,  You don't.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5948) Deprecate and remove the Avro gateway

2012-05-23 Thread Andrew Purtell (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13282026#comment-13282026
 ] 

Andrew Purtell commented on HBASE-5948:
---

+1 Thanks Elliott.

 Deprecate and remove the Avro gateway
 -

 Key: HBASE-5948
 URL: https://issues.apache.org/jira/browse/HBASE-5948
 Project: HBase
  Issue Type: Task
Affects Versions: 0.96.0
Reporter: Andrew Purtell
Priority: Blocker
 Attachments: HBASE-5948-0.patch


 Deprecate the Avro gateway in 0.94. Remove in 0.96. Made a blocker against 
 that release. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6063) Replication related failures on trunk after HBASE-5453

2012-05-23 Thread Lars Hofhansl (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13282033#comment-13282033
 ] 

Lars Hofhansl commented on HBASE-6063:
--

+1

Going to commit soon, unless there are objections.

 Replication related failures on trunk after HBASE-5453
 --

 Key: HBASE-6063
 URL: https://issues.apache.org/jira/browse/HBASE-6063
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.96.0
Reporter: Gregory Chanan
Assignee: Gregory Chanan
 Attachments: HBASE-6063.patch


 HBASE-5453 added this line:
 {code}
 return ClusterId.parseFrom(data).toString();
 {code}
 in function:
 public static String readClusterIdZNode(ZooKeeperWatcher watcher)
 but this is not implemented, so you get log messages like:
 2012-05-21 16:46:31,256 ERROR 
 [RegionServer:0;cloudera-vm,60456,1337643971995-EventThread] 
 zookeeper.ClientCnxn$EventThread(523): Error while calling watcher 
 java.lang.IllegalArgumentException: Invalid UUID string: 
 org.apache.hadoop.hbase.ClusterId@5563d208
   at java.util.UUID.fromString(UUID.java:204)
   at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.init(ReplicationSource.java:192)
   at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.getReplicationSource(ReplicationSourceManager.java:328)
   at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.addSource(ReplicationSourceManager.java:206)
   at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$PeersWatcher.nodeChildrenChanged(ReplicationSourceManager.java:505)
   at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:300)
   at 
 org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
   at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
 2012-05-21 16:46:31,256 ERROR 
 [RegionServer:0;cloudera-vm,50926,1337643981835-EventThread] 
 zookeeper.ClientCnxn$EventThread(523): Error while calling watcher 
 and replication fails because the ClusterId does not match what is expected.  
 Patch coming soon.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6080) site.xml - adding ReviewBoard to main page left-hand nav

2012-05-23 Thread David S. Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13282035#comment-13282035
 ] 

David S. Wang commented on HBASE-6080:
--

+1

 site.xml - adding ReviewBoard to main page left-hand nav
 

 Key: HBASE-6080
 URL: https://issues.apache.org/jira/browse/HBASE-6080
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Trivial
 Attachments: src_hbase_6080.patch


 By request, adding ReviewBoard to left-hand nav on website

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6063) Replication related failures on trunk after HBASE-5453


[ 
https://issues.apache.org/jira/browse/HBASE-6063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13282034#comment-13282034
 ] 

Zhihong Yu commented on HBASE-6063:
---

+1 on patch.

 Replication related failures on trunk after HBASE-5453
 --

 Key: HBASE-6063
 URL: https://issues.apache.org/jira/browse/HBASE-6063
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.96.0
Reporter: Gregory Chanan
Assignee: Gregory Chanan
 Attachments: HBASE-6063.patch


 HBASE-5453 added this line:
 {code}
 return ClusterId.parseFrom(data).toString();
 {code}
 in function:
 public static String readClusterIdZNode(ZooKeeperWatcher watcher)
 but this is not implemented, so you get log messages like:
 2012-05-21 16:46:31,256 ERROR 
 [RegionServer:0;cloudera-vm,60456,1337643971995-EventThread] 
 zookeeper.ClientCnxn$EventThread(523): Error while calling watcher 
 java.lang.IllegalArgumentException: Invalid UUID string: 
 org.apache.hadoop.hbase.ClusterId@5563d208
   at java.util.UUID.fromString(UUID.java:204)
   at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.init(ReplicationSource.java:192)
   at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.getReplicationSource(ReplicationSourceManager.java:328)
   at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.addSource(ReplicationSourceManager.java:206)
   at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$PeersWatcher.nodeChildrenChanged(ReplicationSourceManager.java:505)
   at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:300)
   at 
 org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
   at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
 2012-05-21 16:46:31,256 ERROR 
 [RegionServer:0;cloudera-vm,50926,1337643981835-EventThread] 
 zookeeper.ClientCnxn$EventThread(523): Error while calling watcher 
 and replication fails because the ClusterId does not match what is expected.  
 Patch coming soon.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13282036#comment-13282036
]

Hadoop QA commented on HBASE-5986:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12528806/HBASE-5986_v3.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 34 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplication
org.apache.hadoop.hbase.master.TestAssignmentManager
org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
org.apache.hadoop.hbase.replication.TestMasterReplication

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1972//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1972//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1972//console

This message is automatically generated.

Clients can see holes in the META table when regions are being split

Key: HBASE-5986
URL: https://issues.apache.org/jira/browse/HBASE-5986
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Attachments: 5986-v2.txt, HBASE-5986-test_v1.patch,
HBASE-5986_v3.patch

We found this issue when running large scale ingestion tests for HBASE-5754.
The problem is that the .META. table updates are not atomic while splitting a
region. In SplitTransaction, there is a time lap between the marking the
parent offline, and adding of daughters to the META table. This can result in
clients using MetaScanner, of HTable.getStartEndKeys (used by the
TableInputFormat) missing regions which are made just offline, but the
daughters are not added yet.
This is also related to HBASE-4335.

[jira] [Updated] (HBASE-6043) Add Increment Coalescing in thrift.


 [ 
https://issues.apache.org/jira/browse/HBASE-6043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-6043:
-

Attachment: HBASE-6043-7.patch

Missed a default

 Add Increment Coalescing in thrift.
 ---

 Key: HBASE-6043
 URL: https://issues.apache.org/jira/browse/HBASE-6043
 Project: HBase
  Issue Type: Improvement
Reporter: Elliott Clark
Assignee: Elliott Clark
 Attachments: HBASE-6043-0.patch, HBASE-6043-1.patch, 
 HBASE-6043-2.patch, HBASE-6043-3.patch, HBASE-6043-4.patch, 
 HBASE-6043-5.patch, HBASE-6043-6.patch, HBASE-6043-7.patch


 Since the thrift server uses the client api reducing the number of rpc's 
 greatly speeds up increments.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6043) Add Increment Coalescing in thrift.


 [ 
https://issues.apache.org/jira/browse/HBASE-6043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-6043:
-

Attachment: HBASE-6043-8.patch

 Add Increment Coalescing in thrift.
 ---

 Key: HBASE-6043
 URL: https://issues.apache.org/jira/browse/HBASE-6043
 Project: HBase
  Issue Type: Improvement
Reporter: Elliott Clark
Assignee: Elliott Clark
 Attachments: HBASE-6043-0.patch, HBASE-6043-1.patch, 
 HBASE-6043-2.patch, HBASE-6043-3.patch, HBASE-6043-4.patch, 
 HBASE-6043-5.patch, HBASE-6043-6.patch, HBASE-6043-7.patch, HBASE-6043-8.patch


 Since the thrift server uses the client api reducing the number of rpc's 
 greatly speeds up increments.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split


[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13282039#comment-13282039
 ] 

Zhihong Yu commented on HBASE-5986:
---

I ran TestAssignmentManager with patch v3 and it passed:
{code}
Running org.apache.hadoop.hbase.master.TestAssignmentManager
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.703 sec
...
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 17.004s
{code}

 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Attachments: 5986-v2.txt, HBASE-5986-test_v1.patch, 
 HBASE-5986_v3.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6082) [refGuide] adding HBck docs in RefGuide appendix

Doug Meil created HBASE-6082:


 Summary: [refGuide] adding HBck docs in RefGuide appendix
 Key: HBASE-6082
 URL: https://issues.apache.org/jira/browse/HBASE-6082
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil


Jon pointed me to a PDF that he had put together and attached in HBASE-5634.  

I ported the attachment to the refGuide in the Appendix.

Also, added link in Ops_Mgt.xml Tools chapter to point to the appendix.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6082) [refGuide] adding HBck docs in RefGuide appendix


 [ 
https://issues.apache.org/jira/browse/HBASE-6082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-6082:
-

Attachment: src_hbase_6082.patch

 [refGuide] adding HBck docs in RefGuide appendix
 

 Key: HBASE-6082
 URL: https://issues.apache.org/jira/browse/HBASE-6082
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
 Attachments: src_hbase_6082.patch


 Jon pointed me to a PDF that he had put together and attached in HBASE-5634.  
 I ported the attachment to the refGuide in the Appendix.
 Also, added link in Ops_Mgt.xml Tools chapter to point to the appendix.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6082) [refGuide] adding HBck docs in RefGuide appendix


 [ 
https://issues.apache.org/jira/browse/HBASE-6082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-6082:
-

Status: Patch Available  (was: Open)

 [refGuide] adding HBck docs in RefGuide appendix
 

 Key: HBASE-6082
 URL: https://issues.apache.org/jira/browse/HBASE-6082
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
 Attachments: src_hbase_6082.patch


 Jon pointed me to a PDF that he had put together and attached in HBASE-5634.  
 I ported the attachment to the refGuide in the Appendix.
 Also, added link in Ops_Mgt.xml Tools chapter to point to the appendix.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6082) [refGuide] adding HBck docs in RefGuide appendix


 [ 
https://issues.apache.org/jira/browse/HBASE-6082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-6082:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

 [refGuide] adding HBck docs in RefGuide appendix
 

 Key: HBASE-6082
 URL: https://issues.apache.org/jira/browse/HBASE-6082
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
 Attachments: src_hbase_6082.patch


 Jon pointed me to a PDF that he had put together and attached in HBASE-5634.  
 I ported the attachment to the refGuide in the Appendix.
 Also, added link in Ops_Mgt.xml Tools chapter to point to the appendix.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6043) Add Increment Coalescing in thrift.