[jira] [Updated] (HBASE-7579) HTableDescriptor equals method fails if results are returned in a different order

2013-04-03 Thread stack (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-7579:
-

Fix Version/s: (was: 0.95.0)
   0.95.1

 HTableDescriptor equals method fails if results are returned in a different 
 order
 -

 Key: HBASE-7579
 URL: https://issues.apache.org/jira/browse/HBASE-7579
 Project: HBase
  Issue Type: Bug
  Components: Admin
Affects Versions: 0.95.0, 0.94.6
Reporter: Aleksandr Shulman
Assignee: Aleksandr Shulman
Priority: Minor
 Fix For: 0.95.1, 0.94.7

 Attachments: HBASE-7579-0.94.patch, HBASE-7579-v1.patch, 
 HBASE-7579-v2.patch, HBASE-7579-v3.patch, HBASE-7579-v4.patch, 
 HBASE-7579-v5.patch


 HTableDescriptor's compareTo function compares a set of HColumnDescriptors 
 against another set of HColumnDescriptors. It iterates through both, relying 
 on the fact that they will be in the same order.
 In my testing, I may have seen this issue come up, so I decided to fix it.
 It's a straightforward fix. I convert the sets into a hashset for O(1) 
 lookups (at least in theory), then I check that all items in the first set 
 are found in the second.
 Since the sizes are the same, we know that if all elements showed up in the 
 second set, then they must be equal.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5583) Master restart on create table with splitkeys does not recreate table with all the splitkey regions

2013-04-03 Thread Enis Soztutar (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620654#comment-13620654
 ] 

Enis Soztutar commented on HBASE-5583:
--

Thanks Ram. Yes, the design in HBASE-5487 is not finalized yet unfortunately. 
We wanted to bring FATE, which will only handle the client initiated 
operations, not stuff like region assignment. Then the other WAL approaches 
came, and now Sergey's design. In any case, the issue probably won't cut it for 
0.94.

I would not object to getting the patch in this issue in, given that the design 
is still in discussion. My only concern would be that the table state is 
checked in a lot of places, so we should be careful. If you think that solving 
this issue only for create table with splits is important we can go with your 
approach. Otherwise, I would wait for a resolution on HBASE-5487.  

 Master restart on create table with splitkeys does not recreate table with 
 all the splitkey regions
 ---

 Key: HBASE-5583
 URL: https://issues.apache.org/jira/browse/HBASE-5583
 Project: HBase
  Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.95.0

 Attachments: HBASE-5583_new_1.patch, HBASE-5583_new_1_review.patch, 
 HBASE-5583_new_2.patch, HBASE-5583_new_4_WIP.patch, 
 HBASE-5583_new_5_WIP_using_tableznode.patch


 - Create table using splitkeys
 - MAster goes down before all regions are added to meta
 - On master restart the table is again enabled but with less number of 
 regions than specified in splitkeys
 Anyway client will get an exception if i had called sync create table.  But 
 table exists or not check will say table exists. 
 Is this scenario to be handled by client only or can we have some mechanism 
 on the master side for this? Pls suggest.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7525) A canary monitoring program specifically for regionserver

2013-04-03 Thread stack (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-7525:
-

Fix Version/s: (was: 0.95.0)

Moving out 0.95.  Lets commit to 0.95 if you get a chance to merge it in 
Takeshi  Thanks.

 A canary monitoring program specifically for regionserver
 -

 Key: HBASE-7525
 URL: https://issues.apache.org/jira/browse/HBASE-7525
 Project: HBase
  Issue Type: New Feature
  Components: monitoring
Affects Versions: 0.94.0
Reporter: takeshi.miao
Priority: Minor
 Attachments: HBASE-7525-v0.patch, RegionServerCanary.java


 *Motivation*
 This ticket is to provide a canary monitoring tool specifically for 
 HRegionserver, details as follows
 1. This tool is required by operation team due to they thought that the 
 canary for each region of a HBase is too many for them, so I implemented this 
 coarse-granular one based on the original o.a.h.h.tool.Canary for them
 2. And this tool is implemented by multi-threading, which means the each Get 
 request sent by a thread. the reason I use this way is due to we suffered the 
 region server hung issue by now the root cause is still not clear. so this 
 tool can help operation team to detect hung region server if any.
 *example*
 1. the tool docs
 ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -help
 Usage: [opts] [regionServerName 1 [regionServrName 2...]]
  regionServerName - FQDN serverName, can use linux command:hostname -f to 
 check your serverName
  where [-opts] are:
-help Show this help and exit.
-eUse regionServerName as regular expression
   which means the regionServerName is regular expression pattern
-f B stop whole program if first error occurs, default is true
-t N timeout for a check, default is 60 (milisecs)
-daemonContinuous check at defined intervals.
-interval N  Interval between checks (sec)
 2. Will send a request to each regionserver in a HBase cluster
 ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary
 3. Will send a request to a regionserver by given name
 ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary rs1.domainname
 4. Will send a request to regionserver(s) by given regular-expression
 /opt/trend/circus-opstool/bin/hbase-canary-monitor-each-regionserver.sh -e 
 rs1.domainname.pattern
 // another example
 ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -e 
 tw-poc-tm-puppet-hdn[0-9]\{1,2\}.client.tw.trendnet.org
 5. Will send a request to a regionserver and also set a timeout limit for 
 this test
 // query regionserver:rs1.domainname with timeout limit 10sec
 // -f false, means that will not exit this program even test failed
 ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -f false -t 1 
 rs1.domainname
 // echo 1 if timeout
 echo $?
 6. Will run as daemon mode, which means it will send request to each 
 regionserver periodically
 ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -daemon

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6990) Pretty print TTL

2013-04-03 Thread stack (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-6990:
-

Fix Version/s: (was: 0.95.0)

Moving this feature out lads.  If you have a patch, will commit no problem.

 Pretty print TTL
 

 Key: HBASE-6990
 URL: https://issues.apache.org/jira/browse/HBASE-6990
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.95.0, 0.94.6
Reporter: Jean-Daniel Cryans
Assignee: Kevin Odell
Priority: Minor

 I've seen a lot of users getting confused by the TTL configuration and I 
 think that if we just pretty printed it it would solve most of the issues. 
 For example, let's say a user wanted to set a TTL of 90 days. That would be 
 7776000. But let's say that it was typo'd to 7776 instead, it gives you 
 900 days!
 So when we print the TTL we could do something like x days, x hours, x 
 minutes, x seconds (real_ttl_value). This would also help people when they 
 use ms instead of seconds as they would see really big values in there.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-8251) enable SSH before assign META on Master startup

2013-04-03 Thread Jieshan Bean (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jieshan Bean reassigned HBASE-8251:
---

Assignee: Jieshan Bean

 enable SSH before assign META on Master startup
 ---

 Key: HBASE-8251
 URL: https://issues.apache.org/jira/browse/HBASE-8251
 Project: HBase
  Issue Type: Bug
  Components: master, Region Assignment
Affects Versions: 0.94.6
Reporter: Jieshan Bean
Assignee: Jieshan Bean

 I think HBASE-5918 could not fix this issue. In HMaster#assignRootAndMeta:
 1. Assign ROOT.
 2. Block until ROOT be opened.
 3. Assign META.
 4. Block until META be opened.
 SSH is enabled after step 4. So if the RS who host ROOT dies before step 4, 
 master will be blocked.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6919) Remove unnecessary cast from Bytes.readVLong

2013-04-03 Thread stack (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-6919:
-

Fix Version/s: (was: 0.95.0)

Moving out.  No movement.  [~giacomotaylor] There is a question for you on end 
of this issue.  Put up a patch and we'll commit it.

 Remove unnecessary cast from Bytes.readVLong
 

 Key: HBASE-6919
 URL: https://issues.apache.org/jira/browse/HBASE-6919
 Project: HBase
  Issue Type: Bug
Reporter: James Taylor
Priority: Minor

 Remove the throws IOException so that caller doesn't have to catch and ignore.
 {code}
   public static long readVLong(final byte [] buffer, final int offset)
   throws IOException
 {code}
 Also, add
 {code}
   public static int readVInt(final byte [] buffer, final int offset)
   throws IOException {
 return (int)readVLong(buffer,offset);
   }
 {code}
 and these are useful too:
 {code}
 /**
  * Put long as variable length encoded number at the offset in
  * the result byte array.
  * @param vint Integer to make a vint of.
  * @param result buffer to put vint into
  * @return Vint length in bytes of vint
  */
 public static int vintToBytes(byte[] result, int offset, final long vint) 
 {
   long i = vint;
   if (i = -112  i = 127) {
 result[offset] = (byte) i;
 return 1;
   }
   int len = -112;
   if (i  0) {
 i ^= -1L; // take one's complement'
 len = -120;
   }
   long tmp = i;
   while (tmp != 0) {
 tmp = tmp  8;
 len--;
   }
   result[offset++] = (byte) len;
   len = (len  -120) ? -(len + 120) : -(len + 112);
   for (int idx = len; idx != 0; idx--) {
 int shiftbits = (idx - 1) * 8;
 long mask = 0xFFL  shiftbits;
 result[offset++] = (byte)((i  mask)  shiftbits);
   }
   return len + 1;
 }
 /**
  * Decode a vint from the buffer pointed at to by ptr and
  * increment the offset of the ptr by the length of the
  * vint.
  * @param ptr a pointer to a byte array buffer
  * @return the decoded vint value as an int
  */
 public static int vintFromBytes(ImmutableBytesWritable ptr) {
 return (int) vlongFromBytes(ptr);
 }
 
 /**
  * Decode a vint from the buffer pointed at to by ptr and
  * increment the offset of the ptr by the length of the
  * vint.
  * @param ptr a pointer to a byte array buffer
  * @return the decoded vint value as a long
  */
 public static long vlongFromBytes(ImmutableBytesWritable ptr) {
 final byte [] buffer = ptr.get();
 final int offset = ptr.getOffset();
 byte firstByte = buffer[offset];
 int len = WritableUtils.decodeVIntSize(firstByte);
 if (len == 1) {
 ptr.set(buffer, offset+1, ptr.getLength());
 return firstByte;
 }
 long i = 0;
 for (int idx = 0; idx  len-1; idx++) {
 byte b = buffer[offset + 1 + idx];
 i = i  8;
 i = i | (b  0xFF);
 }
 ptr.set(buffer, offset+len, ptr.getLength());
 return (WritableUtils.isNegativeVInt(firstByte) ? ~i : i);
 }
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-8251) enable SSH before assign META on Master startup

2013-04-03 Thread Jieshan Bean (JIRA) 
Jieshan Bean created HBASE-8251:
---

 Summary: enable SSH before assign META on Master startup
 Key: HBASE-8251
 URL: https://issues.apache.org/jira/browse/HBASE-8251
 Project: HBase
  Issue Type: Bug
  Components: master, Region Assignment
Affects Versions: 0.94.6
Reporter: Jieshan Bean


I think HBASE-5918 could not fix this issue. In HMaster#assignRootAndMeta:
1. Assign ROOT.
2. Block until ROOT be opened.
3. Assign META.
4. Block until META be opened.

SSH is enabled after step 4. So if the RS who host ROOT dies before step 4, 
master will be blocked.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6735) [89-fb] Remove reflection from HBaseClient

2013-04-03 Thread stack (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-6735:
-

   Resolution: Fixed
Fix Version/s: (was: 0.95.0)
   0.89-fb
   Status: Resolved  (was: Patch Available)

Resolving as fixed in 0.89fb.  In trunk, covered by HBASE-8214

 [89-fb] Remove reflection from HBaseClient
 --

 Key: HBASE-6735
 URL: https://issues.apache.org/jira/browse/HBASE-6735
 Project: HBase
  Issue Type: Test
Reporter: Michal Gregorczyk
Assignee: Michal Gregorczyk
Priority: Minor
 Fix For: 0.89-fb

 Attachments: 
 0002-jira-HBASE-6735-89-fb-Remove-reflection-from-HBaseCl.patch


 Every time we read response in HBaseClient we create instance of valueClass 
 using reflection and Hadoop APIs. valueClass is set in a constructor of 
 HBaseClient to one of parameters. The parameter is always HbaseObjectWritable.
 We can remove reflection by hardcoding HbaseObjectWritable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8211) Support for NN HA for 0.94

2013-04-03 Thread Himanshu Vashishtha (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-8211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620662#comment-13620662
 ] 

Himanshu Vashishtha commented on HBASE-8211:


yea, did some minor fixes too.

 Support for NN HA for 0.94
 --

 Key: HBASE-8211
 URL: https://issues.apache.org/jira/browse/HBASE-8211
 Project: HBase
  Issue Type: Bug
  Components: master, regionserver
Affects Versions: 0.94.6
Reporter: Himanshu Vashishtha
Assignee: Himanshu Vashishtha
 Fix For: 0.94.7

 Attachments: HBase-8211-v1.patch, HBase-8211-v2.patch


 HBase-8156 is for adding support for retrying non-idempotent operations. This 
 is useful in case NN is suffering from n/w hiccups, etc. This jira is to add 
 similar support for 0.94.x branch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5583) Master restart on create table with splitkeys does not recreate table with all the splitkey regions

2013-04-03 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620663#comment-13620663
 ] 

Hadoop QA commented on HBASE-5583:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12576723/HBASE-5583_new_1_review.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 6 new 
or modified tests.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces lines longer than 
100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   org.apache.hadoop.hbase.master.TestAssignmentManager
  
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5112//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5112//console

This message is automatically generated.

 Master restart on create table with splitkeys does not recreate table with 
 all the splitkey regions
 ---

 Key: HBASE-5583
 URL: https://issues.apache.org/jira/browse/HBASE-5583
 Project: HBase
  Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.95.0

 Attachments: HBASE-5583_new_1.patch, HBASE-5583_new_1_review.patch, 
 HBASE-5583_new_2.patch, HBASE-5583_new_4_WIP.patch, 
 HBASE-5583_new_5_WIP_using_tableznode.patch


 - Create table using splitkeys
 - MAster goes down before all regions are added to meta
 - On master restart the table is again enabled but with less number of 
 regions than specified in splitkeys
 Anyway client will get an exception if i had called sync create table.  But 
 table exists or not check will say table exists. 
 Is this scenario to be handled by client only or can we have some mechanism 
 on the master side for this? Pls suggest.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8248) HConnectionManager moved region tracking can cause the region server to redirect to itself; load balancer uses incorrect servername compare

2013-04-03 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620664#comment-13620664
 ] 

stack commented on HBASE-8248:
--

Patch looks good except for

-unassign(regionInfo, rs, expectedVersion, sn, true, null);
+unassign(regionInfo, rs, expectedVersion, null, true, 
null);

The above test failure is suspicious Sergey.

 HConnectionManager moved region tracking can cause the region server to 
 redirect to itself; load balancer uses incorrect servername compare
 ---

 Key: HBASE-8248
 URL: https://issues.apache.org/jira/browse/HBASE-8248
 Project: HBase
  Issue Type: Bug
  Components: Client
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
 Fix For: 0.95.0

 Attachments: HBASE-8248-v0.patch


 hbase-hbase-regionserver-ip-10-152-131-248.log:2013-04-02 16:48:24,499 INFO 
 org.apache.hadoop.hbase.regionserver.HRegionServer: Adding moved region 
 record: c52aa9838fe14b28b34f1e4e5fa14be9 to 
 ip-10-152-131-248.ec2.internal,60020,1364935613234:60020 as of 51312
 The consequences are obvious...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8028) Append, Increment: Adding rollback support

2013-04-03 Thread Himanshu Vashishtha (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620665#comment-13620665
 ] 

Himanshu Vashishtha commented on HBASE-8028:


[~jxiang] Very good points.
As Ted said, readpoint is not changed. The readpoint which is being used for 
the rollback is not committed yet, so none of the scanner is reading it at this 
point.

For the snapshots, the patch needs to be altered a bit. Basically, if the kv is 
present in snapshot then it will not be in memstore kv, and vice versa. 
So, while deleting the new kv, if it is deleted from snapshot, then the old kv 
should be only added to the snapshot. Similarly, if the new kv is deleted from 
memstore's kv, then the old kv should be added to the memstore's kv.
I'll upload a revised patch.

 Append, Increment: Adding rollback support
 --

 Key: HBASE-8028
 URL: https://issues.apache.org/jira/browse/HBASE-8028
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.5
Reporter: Himanshu Vashishtha
Assignee: Himanshu Vashishtha
 Fix For: 0.95.0

 Attachments: HBase-8028-v1.patch, HBase-8028-v2.patch, 
 HBase-8028-with-Increments-v1.patch, HBase-8028-with-Increments-v2.patch


 In case there is an exception while doing the log-sync, the memstore is not 
 rollbacked, while the mvcc is _always_ forwarded to the writeentry created at 
 the beginning of the operation. This may lead to scanners seeing results 
 which are not synched to the fs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8229) Replication code logs like crazy if a target table cannot be found.

2013-04-03 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-8229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620666#comment-13620666
 ] 

stack commented on HBASE-8229:
--

+1 on first version of patch.  Its an improvment (What you think [~jeason]?)

[~himan...@cloudera.com] Nice idea on exposing replication state in UI.

 Replication code logs like crazy if a target table cannot be found.
 ---

 Key: HBASE-8229
 URL: https://issues.apache.org/jira/browse/HBASE-8229
 Project: HBase
  Issue Type: Bug
  Components: Replication
Reporter: Lars Hofhansl
 Fix For: 0.95.0, 0.98.0, 0.94.7

 Attachments: 8229-0.94.txt, 8229-0.94-V2.txt


 One of our RS/DN machines ran out of diskspace on the partition to which we 
 write the log files.
 It turns out we still had a table in our source cluster with 
 REPLICATION_SCOPE=1 that did not have a matching table in the remote cluster.
 In then logged a long stack trace every 50ms or so, over a few days that 
 filled up our log partition.
 Since ReplicationSource cannot make any progress in this case anyway, it 
 should probably sleep a bit before retrying (or at least limit the rate at 
 which it spews out these exceptions to the log).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8229) Replication code logs like crazy if a target table cannot be found.

2013-04-03 Thread stack (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-8229:
-

Fix Version/s: (was: 0.95.0)
   0.95.1

 Replication code logs like crazy if a target table cannot be found.
 ---

 Key: HBASE-8229
 URL: https://issues.apache.org/jira/browse/HBASE-8229
 Project: HBase
  Issue Type: Bug
  Components: Replication
Reporter: Lars Hofhansl
 Fix For: 0.95.1, 0.98.0, 0.94.7

 Attachments: 8229-0.94.txt, 8229-0.94-V2.txt


 One of our RS/DN machines ran out of diskspace on the partition to which we 
 write the log files.
 It turns out we still had a table in our source cluster with 
 REPLICATION_SCOPE=1 that did not have a matching table in the remote cluster.
 In then logged a long stack trace every 50ms or so, over a few days that 
 filled up our log partition.
 Since ReplicationSource cannot make any progress in this case anyway, it 
 should probably sleep a bit before retrying (or at least limit the rate at 
 which it spews out these exceptions to the log).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8208) Data could not be replicated to slaves when deferredLogSync is enabled

2013-04-03 Thread stack (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-8208:
-

Fix Version/s: (was: 0.95.0)
   0.95.1

 Data could not be replicated to slaves when deferredLogSync is enabled
 --

 Key: HBASE-8208
 URL: https://issues.apache.org/jira/browse/HBASE-8208
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.95.0, 0.98.0, 0.94.6
Reporter: Jeffrey Zhong
Assignee: Jeffrey Zhong
 Fix For: 0.95.1, 0.98.0, 0.94.7

 Attachments: hbase-8208.patch, hbase-8208-v1.patch, 
 hbase-8208_v2.patch


 This is a subtle issue. When deferredLogSync is enabled, there are chances we 
 could flush data before syncing all HLog entries. Assuming we just flush the 
 internal cache and the server dies with some unsynced hlog entries. 
 Data is not lost at the source cluster while replication is based on WAL 
 files and some changes we flushed at the source won't be replicated the slave 
 clusters. 
 Although enabling deferredLogSync with tolerances of data loss, it breaks the 
 replication assumption that whatever persisted in the source should be 
 replicated to its slave clusters. 
 In short, the slave cluster could end up with double losses: the data loss in 
 the source and some data stored in source cluster may not be replicated to 
 slaves either.
 The fix of the issue isn't hard. Basically we can invoke sync during each 
 flush when replication is enabled for a region server. Since sync returns 
 immediately when nothing to sync so there should be no performance impact.
 Please let me know what you think!
 Thanks,
 -Jeffrey

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8229) Replication code logs like crazy if a target table cannot be found.

2013-04-03 Thread Jieshan Bean (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-8229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620667#comment-13620667
 ] 

Jieshan Bean commented on HBASE-8229:
-

+1 on first version of patch.

 Replication code logs like crazy if a target table cannot be found.
 ---

 Key: HBASE-8229
 URL: https://issues.apache.org/jira/browse/HBASE-8229
 Project: HBase
  Issue Type: Bug
  Components: Replication
Reporter: Lars Hofhansl
 Fix For: 0.95.1, 0.98.0, 0.94.7

 Attachments: 8229-0.94.txt, 8229-0.94-V2.txt


 One of our RS/DN machines ran out of diskspace on the partition to which we 
 write the log files.
 It turns out we still had a table in our source cluster with 
 REPLICATION_SCOPE=1 that did not have a matching table in the remote cluster.
 In then logged a long stack trace every 50ms or so, over a few days that 
 filled up our log partition.
 Since ReplicationSource cannot make any progress in this case anyway, it 
 should probably sleep a bit before retrying (or at least limit the rate at 
 which it spews out these exceptions to the log).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8165) Update our protobuf to 2.5 from 2.4.1

2013-04-03 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620669#comment-13620669
 ] 

stack commented on HBASE-8165:
--

[~andrew.purt...@gmail.com] What we do here?  Not upgrade until hadoop does?  
And since we support multiple hadoops?  The issue over HADOOP-9348 is just the 
glue code that was expecting a 2.5 not finding code that was generated by a 2.5 
protoc... easy enough to recognize and an issue we shouldn't have  (Code 
generated by a protoc 2.5 might be useable by a glue code that was expecting 
2.4 generated pbs).

 Update our protobuf to 2.5 from 2.4.1
 -

 Key: HBASE-8165
 URL: https://issues.apache.org/jira/browse/HBASE-8165
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
 Fix For: 0.95.0

 Attachments: 8165.txt


 Update to new 2.5 pb.  Some speedups and a new PARSER idiom that bypasses 
 making a builder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8139) Allow job names to be overridden

2013-04-03 Thread stack (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-8139:
-

Fix Version/s: (was: 0.95.0)

Moving out till a patch shows up.

 Allow job names to be overridden
 

 Key: HBASE-8139
 URL: https://issues.apache.org/jira/browse/HBASE-8139
 Project: HBase
  Issue Type: Sub-task
  Components: mapreduce, Usability
Reporter: Nick Dimiduk
 Fix For: 0.98.0


 As a general feature, we should allow mr job names to be overridden by the 
 user. See also HBASE-8077.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8165) Update our protobuf to 2.5 from 2.4.1

2013-04-03 Thread Andrew Purtell (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620675#comment-13620675
 ] 

Andrew Purtell commented on HBASE-8165:
---

Ok, since I'm worried about a possibility, let's try it and see. 

 Update our protobuf to 2.5 from 2.4.1
 -

 Key: HBASE-8165
 URL: https://issues.apache.org/jira/browse/HBASE-8165
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
 Fix For: 0.95.0

 Attachments: 8165.txt


 Update to new 2.5 pb.  Some speedups and a new PARSER idiom that bypasses 
 making a builder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-8252) Regions by Region Server table in Master's table view needs styling

2013-04-03 Thread Elliott Clark (JIRA) 
Elliott Clark created HBASE-8252:


 Summary: Regions by Region Server table in Master's table view 
needs styling
 Key: HBASE-8252
 URL: https://issues.apache.org/jira/browse/HBASE-8252
 Project: HBase
  Issue Type: Bug
  Components: UI
Reporter: Elliott Clark
Priority: Trivial


Need to add class=table to the table for region counts per region server on 
table display page.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8252) Regions by Region Server table in Master's table view needs styling

2013-04-03 Thread Elliott Clark (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-8252:
-

Affects Version/s: 0.95.0

 Regions by Region Server table in Master's table view needs styling
 ---

 Key: HBASE-8252
 URL: https://issues.apache.org/jira/browse/HBASE-8252
 Project: HBase
  Issue Type: Bug
  Components: UI
Affects Versions: 0.95.0
Reporter: Elliott Clark
Priority: Trivial

 Need to add class=table to the table for region counts per region server on 
 table display page.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8252) Regions by Region Server table in Master's table view needs styling

2013-04-03 Thread Elliott Clark (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-8252:
-

Labels: noob  (was: )

 Regions by Region Server table in Master's table view needs styling
 ---

 Key: HBASE-8252
 URL: https://issues.apache.org/jira/browse/HBASE-8252
 Project: HBase
  Issue Type: Bug
  Components: UI
Affects Versions: 0.95.0
Reporter: Elliott Clark
Priority: Trivial
  Labels: noob

 Need to add class=table to the table for region counts per region server on 
 table display page.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-8253) A corrupted log blocked ReplicationSource

2013-04-03 Thread Jieshan Bean (JIRA) 
Jieshan Bean created HBASE-8253:
---

 Summary: A corrupted log blocked ReplicationSource
 Key: HBASE-8253
 URL: https://issues.apache.org/jira/browse/HBASE-8253
 Project: HBase
  Issue Type: Bug
  Components: Replication
Affects Versions: 0.94.6
Reporter: Jieshan Bean
Assignee: Jieshan Bean


A writting log got corrupted when we forcely power down one node. Only partial 
of last WALEdit was written into that log. And that log was not the last one in 
replication queue. 
ReplicationSource was blocked under this scenario. A lot of logs like below 
were printed:
{noformat}
2013-03-30 06:53:48,628 WARN  
[regionserver26003-EventThread.replicationSource,1] 1 Got:  
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:334)
java.io.EOFException: 
hdfs://hacluster/hbase/.logs/master11,26003,1364530862620/master11%2C26003%2C1364530862620.1364553936510,
 entryStart=40434738, pos=40450048, end=40450048, edit=0
at sun.reflect.GeneratedConstructorAccessor42.newInstance(Unknown 
Source)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at 
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.addFileInfoToException(SequenceFileLogReader.java:295)
at 
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:240)
at 
org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:84)
at 
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:412)
at 
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:330)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at 
org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:68)
at 
org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:106)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2282)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2181)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2227)
at 
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:238)
... 3 more
..  
2013-03-30 06:54:38,899 WARN  
[regionserver26003-EventThread.replicationSource,1] 1 Got:  
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:334)
java.io.EOFException: 
hdfs://hacluster/hbase/.logs/master11,26003,1364530862620/master11%2C26003%2C1364530862620.1364553936510,
 entryStart=40434738, pos=40450048, end=40450048, edit=0
at sun.reflect.GeneratedConstructorAccessor42.newInstance(Unknown 
Source)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at 
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.addFileInfoToException(SequenceFileLogReader.java:295)
at 
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:240)
at 
org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:84)
at 
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:412)
at 
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:330)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at 
org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:68)
at 
org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:106)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2282)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2181)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2227)
at 
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:238)
... 3 more
... 
{noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-8254) Add lz4 to test compression util

2013-04-03 Thread Elliott Clark (JIRA) 
Elliott Clark created HBASE-8254:


 Summary: Add lz4 to test compression util
 Key: HBASE-8254
 URL: https://issues.apache.org/jira/browse/HBASE-8254
 Project: HBase
  Issue Type: Bug
  Components: util
Affects Versions: 0.95.0
Reporter: Elliott Clark


Add lz4 to the list of compression codecs on test compression util.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8253) A corrupted log blocked ReplicationSource

2013-04-03 Thread Jieshan Bean (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jieshan Bean updated HBASE-8253:


Attachment: HBASE-8253-94.patch

Patch for discussion.

In ReplicationSource#readAllEntriesToReplicateOrNextFile, only read for the 
first edit may throw EOF. So when we get EOF, currentNbEntries should be 0. No 
other case.
Please correct me if I am wrong.

 A corrupted log blocked ReplicationSource
 -

 Key: HBASE-8253
 URL: https://issues.apache.org/jira/browse/HBASE-8253
 Project: HBase
  Issue Type: Bug
  Components: Replication
Affects Versions: 0.94.6
Reporter: Jieshan Bean
Assignee: Jieshan Bean
 Attachments: HBASE-8253-94.patch


 A writting log got corrupted when we forcely power down one node. Only 
 partial of last WALEdit was written into that log. And that log was not the 
 last one in replication queue. 
 ReplicationSource was blocked under this scenario. A lot of logs like below 
 were printed:
 {noformat}
 2013-03-30 06:53:48,628 WARN  
 [regionserver26003-EventThread.replicationSource,1] 1 Got:  
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:334)
 java.io.EOFException: 
 hdfs://hacluster/hbase/.logs/master11,26003,1364530862620/master11%2C26003%2C1364530862620.1364553936510,
  entryStart=40434738, pos=40450048, end=40450048, edit=0
   at sun.reflect.GeneratedConstructorAccessor42.newInstance(Unknown 
 Source)
   at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.addFileInfoToException(SequenceFileLogReader.java:295)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:240)
   at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:84)
   at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:412)
   at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:330)
 Caused by: java.io.EOFException
   at java.io.DataInputStream.readFully(DataInputStream.java:180)
   at 
 org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:68)
   at 
 org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:106)
   at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2282)
   at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2181)
   at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2227)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:238)
   ... 3 more
 ..
 2013-03-30 06:54:38,899 WARN  
 [regionserver26003-EventThread.replicationSource,1] 1 Got:  
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:334)
 java.io.EOFException: 
 hdfs://hacluster/hbase/.logs/master11,26003,1364530862620/master11%2C26003%2C1364530862620.1364553936510,
  entryStart=40434738, pos=40450048, end=40450048, edit=0
   at sun.reflect.GeneratedConstructorAccessor42.newInstance(Unknown 
 Source)
   at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.addFileInfoToException(SequenceFileLogReader.java:295)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:240)
   at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:84)
   at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:412)
   at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:330)
 Caused by: java.io.EOFException
   at java.io.DataInputStream.readFully(DataInputStream.java:180)
   at 
 org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:68)
   at 
 org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:106)
   at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2282)
   at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2181)
   at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2227)
   at

[jira] [Updated] (HBASE-8254) Add lz4 to test compression util

2013-04-03 Thread Elliott Clark (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-8254:
-

Labels: noob  (was: )

 Add lz4 to test compression util
 

 Key: HBASE-8254
 URL: https://issues.apache.org/jira/browse/HBASE-8254
 Project: HBase
  Issue Type: Bug
  Components: util
Affects Versions: 0.95.0
Reporter: Elliott Clark
  Labels: noob

 Add lz4 to the list of compression codecs on test compression util.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8251) enable SSH before assign META on Master startup

2013-04-03 Thread rajeshbabu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-8251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620701#comment-13620701
 ] 

rajeshbabu commented on HBASE-8251:
---

[~jeason]
As part of HBASE-5875 we fixed this.After enabling SSH only we are wating for 
META(before 4th step).
{code}
  assignmentManager.assignMeta();
  enableSSHandWaitForMeta();
{code}
{code}
  private void enableSSHandWaitForMeta() throws IOException,
  InterruptedException {
enableServerShutdownHandler();
this.catalogTracker.waitForMeta();
// Above check waits for general meta availability but this does not
// guarantee that the transition has completed
this.assignmentManager
.waitForAssignment(HRegionInfo.FIRST_META_REGIONINFO);
  }
{code}

 enable SSH before assign META on Master startup
 ---

 Key: HBASE-8251
 URL: https://issues.apache.org/jira/browse/HBASE-8251
 Project: HBase
  Issue Type: Bug
  Components: master, Region Assignment
Affects Versions: 0.94.6
Reporter: Jieshan Bean
Assignee: Jieshan Bean

 I think HBASE-5918 could not fix this issue. In HMaster#assignRootAndMeta:
 1. Assign ROOT.
 2. Block until ROOT be opened.
 3. Assign META.
 4. Block until META be opened.
 SSH is enabled after step 4. So if the RS who host ROOT dies before step 4, 
 master will be blocked.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8251) enable SSH before assign META on Master startup

2013-04-03 Thread Jieshan Bean (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-8251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620703#comment-13620703
 ] 

Jieshan Bean commented on HBASE-8251:
-

[~rajesh23]I don't think so, Master was blocked at 
AM#processRegionInTransitionAndBlockUntilAssigned:
{code}
// Work on meta region
status.setStatus(Assigning META region);
rit = this.assignmentManager.
  
processRegionInTransitionAndBlockUntilAssigned(HRegionInfo.FIRST_META_REGIONINFO);
boolean metaRegionLocation = 
this.catalogTracker.verifyMetaRegionLocation(timeout);
{code}


 enable SSH before assign META on Master startup
 ---

 Key: HBASE-8251
 URL: https://issues.apache.org/jira/browse/HBASE-8251
 Project: HBase
  Issue Type: Bug
  Components: master, Region Assignment
Affects Versions: 0.94.6
Reporter: Jieshan Bean
Assignee: Jieshan Bean

 I think HBASE-5918 could not fix this issue. In HMaster#assignRootAndMeta:
 1. Assign ROOT.
 2. Block until ROOT be opened.
 3. Assign META.
 4. Block until META be opened.
 SSH is enabled after step 4. So if the RS who host ROOT dies before step 4, 
 master will be blocked.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7462) TestDrainingServer is an integration test. It should be a unit test instead

2013-04-03 Thread Nicolas Liochon (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-7462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620718#comment-13620718
 ] 

Nicolas Liochon commented on HBASE-7462:


The stask seems to say that ZK is not started?

I would expect this to be an error, as it's likely to need a starter mini 
cluster to work
HMaster hMaster = TEST_UTIL.getMiniHBaseCluster().getMaster();
CollectionServerName servers = 
TEST_UTIL.getMiniHBaseCluster().getInitialClusterStatus().getServers();
Long sizeServedRegions = TEST_UTIL.getMiniHBaseCluster().countServedRegions();



 TestDrainingServer is an integration test. It should be a unit test instead
 ---

 Key: HBASE-7462
 URL: https://issues.apache.org/jira/browse/HBASE-7462
 Project: HBase
  Issue Type: Wish
  Components: test
Affects Versions: 0.96.0
Reporter: Nicolas Liochon
Priority: Trivial
  Labels: noob

 TestDrainingServer tests the function that allows to say that a regionserver 
 should not get new regions.
 As it is written today, it's an integration test: it starts  stops a cluster.
 The test would be more efficient if it would just check that the 
 AssignmentManager does not use the drained region server; whatever the 
 circumstances (bulk assign or not for example).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7871) HBase can be stuck when closing regions concurrently

2013-04-03 Thread Anoop Sam John (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-7871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620739#comment-13620739
 ] 

Anoop Sam John commented on HBASE-7871:
---

Yes Ram. This fix should solve this test failure also.

 HBase can be stuck when closing regions concurrently 
 -

 Key: HBASE-7871
 URL: https://issues.apache.org/jira/browse/HBASE-7871
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.96.0
Reporter: Nicolas Liochon
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.1, 0.98.0

 Attachments: 7871.patch, 7871-v2.patch, 7871-v3.txt, 7871-v4.txt, 
 s1.txt, TestStartStop.java


 The attached test fails ~1% of the the time on 0.96. It seems it does not 
 fail on 0.94.5. It's simple: a table creation and some puts.
 I attach the stack. Logs says nothing it seems.
 The suspicious part is:
 {noformat}
 RS_CLOSE_REGION-localhost,57575,1361197489166-2 prio=10 
 tid=0x7fb0c8775800 nid=0x61ac runnable [0x7fb09f272000]
java.lang.Thread.State: RUNNABLE
 at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2193)
 at java.util.TreeMap.deleteEntry(TreeMap.java:2151)
 at java.util.TreeMap.remove(TreeMap.java:585)
 at java.util.TreeSet.remove(TreeSet.java:259)
 at 
 org.apache.hadoop.hbase.regionserver.MetricsRegionAggregateSourceImpl.deregister(MetricsRegionAggregateSourceImpl.java:55)
 at 
 org.apache.hadoop.hbase.regionserver.MetricsRegionSourceImpl.close(MetricsRegionSourceImpl.java:86)
 at 
 org.apache.hadoop.hbase.regionserver.MetricsRegion.close(MetricsRegion.java:40)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1063)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:969)
 - locked 0x0006944e2558 (a java.lang.Object)
 at 
 org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:146)
 at 
 org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:203)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5583) Master restart on create table with splitkeys does not recreate table with all the splitkey regions

2013-04-03 Thread ramkrishna.s.vasudevan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620779#comment-13620779
 ] 

ramkrishna.s.vasudevan commented on HBASE-5583:
---

bq.I would not object to getting the patch in this issue in, given that the 
design is still in discussion. My only concern would be that the table state is 
checked in a lot of places, so we should be careful. If you think that solving 
this issue only for create table with splits is important we can go with your 
approach. Otherwise, I would wait for a resolution on HBASE-5487.
Enis concern here is valid.  
What other folks think on this, i can work based on that.
[~saint@gmail.com],[~te...@apache.org],[~jxiang],[~rajesh23],[~mbertozzi] - 
Your thoughts?

 Master restart on create table with splitkeys does not recreate table with 
 all the splitkey regions
 ---

 Key: HBASE-5583
 URL: https://issues.apache.org/jira/browse/HBASE-5583
 Project: HBase
  Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.95.0

 Attachments: HBASE-5583_new_1.patch, HBASE-5583_new_1_review.patch, 
 HBASE-5583_new_2.patch, HBASE-5583_new_4_WIP.patch, 
 HBASE-5583_new_5_WIP_using_tableznode.patch


 - Create table using splitkeys
 - MAster goes down before all regions are added to meta
 - On master restart the table is again enabled but with less number of 
 regions than specified in splitkeys
 Anyway client will get an exception if i had called sync create table.  But 
 table exists or not check will say table exists. 
 Is this scenario to be handled by client only or can we have some mechanism 
 on the master side for this? Pls suggest.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8251) enable SSH before assign META on Master startup

2013-04-03 Thread Jieshan Bean (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jieshan Bean updated HBASE-8251:


Attachment: HBASE-8251-94.patch

Patch for 94.

 enable SSH before assign META on Master startup
 ---

 Key: HBASE-8251
 URL: https://issues.apache.org/jira/browse/HBASE-8251
 Project: HBase
  Issue Type: Bug
  Components: master, Region Assignment
Affects Versions: 0.94.6
Reporter: Jieshan Bean
Assignee: Jieshan Bean
 Attachments: HBASE-8251-94.patch


 I think HBASE-5918 could not fix this issue. In HMaster#assignRootAndMeta:
 1. Assign ROOT.
 2. Block until ROOT be opened.
 3. Assign META.
 4. Block until META be opened.
 SSH is enabled after step 4. So if the RS who host ROOT dies before step 4, 
 master will be blocked.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7615) Add metrics for snapshots

2013-04-03 Thread Matteo Bertozzi (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-7615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620824#comment-13620824
 ] 

Matteo Bertozzi commented on HBASE-7615:


[~eclark] what do you suggest instead? I just want to keep track of min/max/avg 
time of snapshots... 

As far as I know, we don't have a way to store metrics like [snapshot: time]
so at the moment just having an idea of how long on avg it takes taking a 
snapshot or restoring is a step forward.

I've used the histogram because is the same used by the log splitting, that is 
quite similar to the snapshot workload... do you think that also the log split 
metrics doesn't make sense?

 Add metrics for snapshots
 -

 Key: HBASE-7615
 URL: https://issues.apache.org/jira/browse/HBASE-7615
 Project: HBase
  Issue Type: Sub-task
Reporter: Ted Yu
Assignee: Matteo Bertozzi
 Attachments: HBASE-7615-0.94.patch, HBASE-7615-0.94-v1.patch, 
 HBASE-7615.patch, HBASE-7615.patch, HBASE-7615-v0.patch, 
 HBASE-7615-v0-ui-corrupted.png, HBASE-7615-v0-ui.png, HBASE-7615-v1.patch, 
 HBASE-7615-v2.patch, HBASE-7615-v3.patch


 Metrics should be added for snapshot.
 From Matteo: Output that we have in SnapshotInfo should be covered:
 %d HFiles (%d in archive), total size %s (%.2f%% %s shared with the source 
 table)
 Jesse mentioned snaphot counts, average time to completion.
 I think we should have counts for successful / failed snapshots.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-8255) Merge issue in HBASE-7807 on 0.95 (not trunk): postOpen coprocessor is now called twice

2013-04-03 Thread Nicolas Liochon (JIRA) 
Nicolas Liochon created HBASE-8255:
--

 Summary: Merge issue in HBASE-7807 on 0.95 (not trunk): postOpen 
coprocessor is now called twice
 Key: HBASE-8255
 URL: https://issues.apache.org/jira/browse/HBASE-8255
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.95.0
Reporter: Nicolas Liochon


For HRegion.java

On trunk:
svn diff -r 1451024:1451025 . | grep detritus
+   * Clean up any temp detritus that may have been left around from previous 
operation attempts.
+   * Clean up any split detritus that may have been left around from previous 
split attempts.
+   * Clean up any merge detritus that may have been left around from previous 
merge attempts.
+status.setStatus(Cleaning up detritus from prior splits);
-status.setStatus(Cleaning up detritus from prior splits);

On 0.95:
svn diff -r 1460610:1460611 . | grep detritus
+   * Clean up any temp detritus that may have been left around from previous 
operation attempts.
+   * Clean up any split detritus that may have been left around from previous 
split attempts.
+   * Clean up any merge detritus that may have been left around from previous 
merge attempts.
+status.setStatus(Cleaning up detritus from prior splits);
 status.setStatus(Cleaning up detritus from prior splits);


For whatever reason, some code was not deleted on the 0.95 branch.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7871) HBase can be stuck when closing regions concurrently

2013-04-03 Thread Nicolas Liochon (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-7871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620871#comment-13620871
 ] 

Nicolas Liochon commented on HBASE-7871:


Tested, no error on 250 tries = +1 for me.

 HBase can be stuck when closing regions concurrently 
 -

 Key: HBASE-7871
 URL: https://issues.apache.org/jira/browse/HBASE-7871
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.96.0
Reporter: Nicolas Liochon
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.1, 0.98.0

 Attachments: 7871.patch, 7871-v2.patch, 7871-v3.txt, 7871-v4.txt, 
 s1.txt, TestStartStop.java


 The attached test fails ~1% of the the time on 0.96. It seems it does not 
 fail on 0.94.5. It's simple: a table creation and some puts.
 I attach the stack. Logs says nothing it seems.
 The suspicious part is:
 {noformat}
 RS_CLOSE_REGION-localhost,57575,1361197489166-2 prio=10 
 tid=0x7fb0c8775800 nid=0x61ac runnable [0x7fb09f272000]
java.lang.Thread.State: RUNNABLE
 at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2193)
 at java.util.TreeMap.deleteEntry(TreeMap.java:2151)
 at java.util.TreeMap.remove(TreeMap.java:585)
 at java.util.TreeSet.remove(TreeSet.java:259)
 at 
 org.apache.hadoop.hbase.regionserver.MetricsRegionAggregateSourceImpl.deregister(MetricsRegionAggregateSourceImpl.java:55)
 at 
 org.apache.hadoop.hbase.regionserver.MetricsRegionSourceImpl.close(MetricsRegionSourceImpl.java:86)
 at 
 org.apache.hadoop.hbase.regionserver.MetricsRegion.close(MetricsRegion.java:40)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1063)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:969)
 - locked 0x0006944e2558 (a java.lang.Object)
 at 
 org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:146)
 at 
 org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:203)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8246) Backport HBASE-6318 to 0.94 where SplitLogWorker exits due to ConcurrentModificationException

2013-04-03 Thread Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-8246:
--

Resolution: Fixed
Status: Resolved  (was: Patch Available)

 Backport HBASE-6318 to 0.94 where SplitLogWorker exits due to 
 ConcurrentModificationException
 -

 Key: HBASE-8246
 URL: https://issues.apache.org/jira/browse/HBASE-8246
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.6
Reporter: Jeffrey Zhong
Assignee: Ted Yu
 Fix For: 0.94.7

 Attachments: 8246-0.94.txt, 8246-0.94-v2.txt


 Today we found the following error in our tests. Later I found we already 
 fixed the issue in trunk. I think we should backpor the fix because the 
 consequence of the issue is high and the fix isn't complicated.
 {code}
 2013-04-01 21:23:21,864 INFO 
 org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker 
 ip-10-143-160-121.ec2.internal,60020,1364849529986 done with task 
 /hbase/splitlog/hdfs%3A%2F%2Fip-10-137-16-140.ec2.internal%3A8020%2Fapps%2Fhbase%2Fdata%2F.logs%2Fip-10-137-20-188.ec2.internal%2C60020%2C1364849530779-splitting%2Fip-10-137-20-188.ec2.internal%252C60020%252C1364849530779.1364865556657
  in 67129ms
 2013-04-01 21:23:21,864 ERROR 
 org.apache.hadoop.hbase.regionserver.SplitLogWorker: unexpected error
 java.util.ConcurrentModificationException
 at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
 at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145)
 at 
 org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$OutputSink.closeLogWriters(HLogSplitter.java:1279)
 at 
 org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$OutputSink.finishWritingAndClose(HLogSplitter.java:1170)
 at 
 org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:475)
 at 
 org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:403)
 at 
 org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:111)
 at 
 org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:264)
 at 
 org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:195)
 at 
 org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:163)
 at java.lang.Thread.run(Thread.java:662)
 2013-04-01 21:23:21,865 INFO 
 org.apache.hadoop.hbase.regionserver.SplitLogWorker: SplitLogWorker 
 ip-10-143-160-121.ec2.internal,60020,1364849529986 exiting
 {code}
 The impact of this issue is that SplitLogWorker exits so does the region 
 server recovering mechanism of HBase. If any RS failed after all 
 SplitLogWorkers in te cluster exit due to the issue, you'll see a hang log 
 splitting job and the failed RS won't be recovered.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Work started] (HBASE-8246) Backport HBASE-6318 to 0.94 where SplitLogWorker exits due to ConcurrentModificationException

2013-04-03 Thread Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-8246 started by Ted Yu.

 Backport HBASE-6318 to 0.94 where SplitLogWorker exits due to 
 ConcurrentModificationException
 -

 Key: HBASE-8246
 URL: https://issues.apache.org/jira/browse/HBASE-8246
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.6
Reporter: Jeffrey Zhong
Assignee: Ted Yu
 Fix For: 0.94.7

 Attachments: 8246-0.94.txt, 8246-0.94-v2.txt


 Today we found the following error in our tests. Later I found we already 
 fixed the issue in trunk. I think we should backpor the fix because the 
 consequence of the issue is high and the fix isn't complicated.
 {code}
 2013-04-01 21:23:21,864 INFO 
 org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker 
 ip-10-143-160-121.ec2.internal,60020,1364849529986 done with task 
 /hbase/splitlog/hdfs%3A%2F%2Fip-10-137-16-140.ec2.internal%3A8020%2Fapps%2Fhbase%2Fdata%2F.logs%2Fip-10-137-20-188.ec2.internal%2C60020%2C1364849530779-splitting%2Fip-10-137-20-188.ec2.internal%252C60020%252C1364849530779.1364865556657
  in 67129ms
 2013-04-01 21:23:21,864 ERROR 
 org.apache.hadoop.hbase.regionserver.SplitLogWorker: unexpected error
 java.util.ConcurrentModificationException
 at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
 at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145)
 at 
 org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$OutputSink.closeLogWriters(HLogSplitter.java:1279)
 at 
 org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$OutputSink.finishWritingAndClose(HLogSplitter.java:1170)
 at 
 org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:475)
 at 
 org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:403)
 at 
 org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:111)
 at 
 org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:264)
 at 
 org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:195)
 at 
 org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:163)
 at java.lang.Thread.run(Thread.java:662)
 2013-04-01 21:23:21,865 INFO 
 org.apache.hadoop.hbase.regionserver.SplitLogWorker: SplitLogWorker 
 ip-10-143-160-121.ec2.internal,60020,1364849529986 exiting
 {code}
 The impact of this issue is that SplitLogWorker exits so does the region 
 server recovering mechanism of HBase. If any RS failed after all 
 SplitLogWorkers in te cluster exit due to the issue, you'll see a hang log 
 splitting job and the failed RS won't be recovered.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8246) Backport HBASE-6318 to 0.94 where SplitLogWorker exits due to ConcurrentModificationException

2013-04-03 Thread Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-8246:
--

Hadoop Flags: Reviewed
  Status: Patch Available  (was: In Progress)

 Backport HBASE-6318 to 0.94 where SplitLogWorker exits due to 
 ConcurrentModificationException
 -

 Key: HBASE-8246
 URL: https://issues.apache.org/jira/browse/HBASE-8246
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.6
Reporter: Jeffrey Zhong
Assignee: Ted Yu
 Fix For: 0.94.7

 Attachments: 8246-0.94.txt, 8246-0.94-v2.txt


 Today we found the following error in our tests. Later I found we already 
 fixed the issue in trunk. I think we should backpor the fix because the 
 consequence of the issue is high and the fix isn't complicated.
 {code}
 2013-04-01 21:23:21,864 INFO 
 org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker 
 ip-10-143-160-121.ec2.internal,60020,1364849529986 done with task 
 /hbase/splitlog/hdfs%3A%2F%2Fip-10-137-16-140.ec2.internal%3A8020%2Fapps%2Fhbase%2Fdata%2F.logs%2Fip-10-137-20-188.ec2.internal%2C60020%2C1364849530779-splitting%2Fip-10-137-20-188.ec2.internal%252C60020%252C1364849530779.1364865556657
  in 67129ms
 2013-04-01 21:23:21,864 ERROR 
 org.apache.hadoop.hbase.regionserver.SplitLogWorker: unexpected error
 java.util.ConcurrentModificationException
 at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
 at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145)
 at 
 org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$OutputSink.closeLogWriters(HLogSplitter.java:1279)
 at 
 org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$OutputSink.finishWritingAndClose(HLogSplitter.java:1170)
 at 
 org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:475)
 at 
 org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:403)
 at 
 org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:111)
 at 
 org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:264)
 at 
 org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:195)
 at 
 org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:163)
 at java.lang.Thread.run(Thread.java:662)
 2013-04-01 21:23:21,865 INFO 
 org.apache.hadoop.hbase.regionserver.SplitLogWorker: SplitLogWorker 
 ip-10-143-160-121.ec2.internal,60020,1364849529986 exiting
 {code}
 The impact of this issue is that SplitLogWorker exits so does the region 
 server recovering mechanism of HBase. If any RS failed after all 
 SplitLogWorkers in te cluster exit due to the issue, you'll see a hang log 
 splitting job and the failed RS won't be recovered.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7871) HBase can be stuck when closing regions concurrently

2013-04-03 Thread Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-7871:
--

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Integrated to 0.95 and trunk.

Thanks for the reviews.

 HBase can be stuck when closing regions concurrently 
 -

 Key: HBASE-7871
 URL: https://issues.apache.org/jira/browse/HBASE-7871
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.96.0
Reporter: Nicolas Liochon
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.1, 0.98.0

 Attachments: 7871.patch, 7871-v2.patch, 7871-v3.txt, 7871-v4.txt, 
 s1.txt, TestStartStop.java


 The attached test fails ~1% of the the time on 0.96. It seems it does not 
 fail on 0.94.5. It's simple: a table creation and some puts.
 I attach the stack. Logs says nothing it seems.
 The suspicious part is:
 {noformat}
 RS_CLOSE_REGION-localhost,57575,1361197489166-2 prio=10 
 tid=0x7fb0c8775800 nid=0x61ac runnable [0x7fb09f272000]
java.lang.Thread.State: RUNNABLE
 at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2193)
 at java.util.TreeMap.deleteEntry(TreeMap.java:2151)
 at java.util.TreeMap.remove(TreeMap.java:585)
 at java.util.TreeSet.remove(TreeSet.java:259)
 at 
 org.apache.hadoop.hbase.regionserver.MetricsRegionAggregateSourceImpl.deregister(MetricsRegionAggregateSourceImpl.java:55)
 at 
 org.apache.hadoop.hbase.regionserver.MetricsRegionSourceImpl.close(MetricsRegionSourceImpl.java:86)
 at 
 org.apache.hadoop.hbase.regionserver.MetricsRegion.close(MetricsRegion.java:40)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1063)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:969)
 - locked 0x0006944e2558 (a java.lang.Object)
 at 
 org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:146)
 at 
 org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:203)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8241) Fix the bad dependency on netty from HDFS

2013-04-03 Thread Nicolas Liochon (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-8241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620951#comment-13620951
 ] 

Nicolas Liochon commented on HBASE-8241:


Checked, we had two netty versions in the assembly, and now we have only one. 
Committed, thanks for the review!

 Fix the bad dependency on netty from HDFS
 -

 Key: HBASE-8241
 URL: https://issues.apache.org/jira/browse/HBASE-8241
 Project: HBase
  Issue Type: Bug
  Components: build, Client, master, regionserver
Affects Versions: 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.95.1, 0.96.0

 Attachments: HBASE-8241.v1.patch


 Even if it's fixed on trunk  branch-2, the current version of hdfs still has 
 a previous version of netty, with a different group id. Let's fix this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8241) Fix the bad dependency on netty from HDFS

2013-04-03 Thread Nicolas Liochon (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Liochon updated HBASE-8241:
---

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

 Fix the bad dependency on netty from HDFS
 -

 Key: HBASE-8241
 URL: https://issues.apache.org/jira/browse/HBASE-8241
 Project: HBase
  Issue Type: Bug
  Components: build, Client, master, regionserver
Affects Versions: 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.95.1, 0.96.0

 Attachments: HBASE-8241.v1.patch


 Even if it's fixed on trunk  branch-2, the current version of hdfs still has 
 a previous version of netty, with a different group id. Let's fix this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8204) Don't use hdfs append during lease recovery

2013-04-03 Thread Nicolas Liochon (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Liochon updated HBASE-8204:
---

   Resolution: Fixed
Fix Version/s: 0.95.1
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

 Don't use hdfs append during lease recovery
 ---

 Key: HBASE-8204
 URL: https://issues.apache.org/jira/browse/HBASE-8204
 Project: HBase
  Issue Type: Bug
  Components: MTTR, wal
Affects Versions: 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.95.1, 0.96.0

 Attachments: 8204.v2.patch, 8204.v2.patch, 8204.v3.patch, 
 8204.v4.patch, HBASE-8204.v1.patch


 See patch :-).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6870) HTable#coprocessorExec always scan the whole table

2013-04-03 Thread Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-6870:
--

Fix Version/s: 0.98.0
   0.95.1

 HTable#coprocessorExec always scan the whole table 
 ---

 Key: HBASE-6870
 URL: https://issues.apache.org/jira/browse/HBASE-6870
 Project: HBase
  Issue Type: Improvement
  Components: Coprocessors
Affects Versions: 0.94.1, 0.95.0, 0.96.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.95.1, 0.98.0

 Attachments: HBASE-6870.patch, HBASE-6870-testPerformance.patch, 
 HBASE-6870v2.patch, HBASE-6870v3.patch


 In current logic, HTable#coprocessorExec always scan the whole table, its 
 efficiency is low and will affect the Regionserver carrying .META. under 
 large coprocessorExec requests

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5583) Master restart on create table with splitkeys does not recreate table with all the splitkey regions

2013-04-03 Thread Jimmy Xiang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621011#comment-13621011
 ] 

Jimmy Xiang commented on HBASE-5583:


To me, I think we can wait for HBASE-5487, if this is not so urgent.

 Master restart on create table with splitkeys does not recreate table with 
 all the splitkey regions
 ---

 Key: HBASE-5583
 URL: https://issues.apache.org/jira/browse/HBASE-5583
 Project: HBase
  Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.95.0

 Attachments: HBASE-5583_new_1.patch, HBASE-5583_new_1_review.patch, 
 HBASE-5583_new_2.patch, HBASE-5583_new_4_WIP.patch, 
 HBASE-5583_new_5_WIP_using_tableznode.patch


 - Create table using splitkeys
 - MAster goes down before all regions are added to meta
 - On master restart the table is again enabled but with less number of 
 regions than specified in splitkeys
 Anyway client will get an exception if i had called sync create table.  But 
 table exists or not check will say table exists. 
 Is this scenario to be handled by client only or can we have some mechanism 
 on the master side for this? Pls suggest.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7745) Import uses System.out.println instead of logging

2013-04-03 Thread Nick Dimiduk (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621031#comment-13621031
 ] 

Nick Dimiduk commented on HBASE-7745:
-

Thanks for the patch!

 Import uses System.out.println instead of logging
 -

 Key: HBASE-7745
 URL: https://issues.apache.org/jira/browse/HBASE-7745
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Reporter: Nick Dimiduk
Assignee: Gustavo Anatoly
Priority: Trivial
 Fix For: 0.95.1, 0.98.0

 Attachments: HBASE-7745.patch


 Per the title.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7745) Import uses System.out.println instead of logging

2013-04-03 Thread Nick Dimiduk (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621030#comment-13621030
 ] 

Nick Dimiduk commented on HBASE-7745:
-

+1

 Import uses System.out.println instead of logging
 -

 Key: HBASE-7745
 URL: https://issues.apache.org/jira/browse/HBASE-7745
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Reporter: Nick Dimiduk
Assignee: Gustavo Anatoly
Priority: Trivial
 Attachments: HBASE-7745.patch


 Per the title.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7745) Import uses System.out.println instead of logging

2013-04-03 Thread Nick Dimiduk (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk updated HBASE-7745:


Fix Version/s: 0.98.0
   0.95.1

 Import uses System.out.println instead of logging
 -

 Key: HBASE-7745
 URL: https://issues.apache.org/jira/browse/HBASE-7745
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Reporter: Nick Dimiduk
Assignee: Gustavo Anatoly
Priority: Trivial
 Fix For: 0.95.1, 0.98.0

 Attachments: HBASE-7745.patch


 Per the title.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8158) Backport HBASE-8140 TableMapReduceUtils#addDependencyJar fails when nested inside another MR job

2013-04-03 Thread Nick Dimiduk (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk updated HBASE-8158:


Affects Version/s: (was: 0.94.7)
   0.94.6

 Backport HBASE-8140 TableMapReduceUtils#addDependencyJar fails when nested 
 inside another MR job
 --

 Key: HBASE-8158
 URL: https://issues.apache.org/jira/browse/HBASE-8158
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 0.94.6
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
 Attachments: 8158-port-jarfinder-0.94.patch, 
 8158-port-jarfinder-0.94.patch, 8158-port-jarfinder-0.94.patch, 
 8158-port-jarfinder-0.94.patch, 8158-port-jarfinder-0.94.patch, 
 8158-trunk-parity.patch


 Assess the stability of the JarFinder util class from Hadoop-2, as a 
 candidate for backporting to 0.94.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8158) Backport HBASE-8140 TableMapReduceUtils#addDependencyJar fails when nested inside another MR job

2013-04-03 Thread Nick Dimiduk (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk updated HBASE-8158:


Fix Version/s: 0.94.7
   0.98.0
   0.95.1

 Backport HBASE-8140 TableMapReduceUtils#addDependencyJar fails when nested 
 inside another MR job
 --

 Key: HBASE-8158
 URL: https://issues.apache.org/jira/browse/HBASE-8158
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 0.94.6
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
 Fix For: 0.95.1, 0.98.0, 0.94.7

 Attachments: 8158-port-jarfinder-0.94.patch, 
 8158-port-jarfinder-0.94.patch, 8158-port-jarfinder-0.94.patch, 
 8158-port-jarfinder-0.94.patch, 8158-port-jarfinder-0.94.patch, 
 8158-trunk-parity.patch


 Assess the stability of the JarFinder util class from Hadoop-2, as a 
 candidate for backporting to 0.94.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8240) CompoundConfiguration should implement Iterable

2013-04-03 Thread Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-8240:
--

Attachment: 8240-v4.txt

Patch v4 adds tests for iterator.

 CompoundConfiguration should implement Iterable
 ---

 Key: HBASE-8240
 URL: https://issues.apache.org/jira/browse/HBASE-8240
 Project: HBase
  Issue Type: Improvement
Reporter: Ted Yu
 Attachments: 8240-v1.txt, 8240-v2.txt, 8240-v3.txt, 8240-v4.txt


 Here is from hadoop Configuration class:
 {code}
 public class Configuration implements IterableMap.EntryString,String,
 {code}
 There're 3 addXX() methods for CompoundConfiguration:
 {code}
   public CompoundConfiguration add(final Configuration conf) {
   public CompoundConfiguration addWritableMap(
   final MapImmutableBytesWritable, ImmutableBytesWritable map) {
   public CompoundConfiguration addStringMap(final MapString, String map) {
 {code}
 Parameters to these methods all support iteration.
 We can enhance ImmutableConfigMap with the following new method:
 {code}
   public abstract java.util.Iterator iterator();
 {code}
 Then the following method of CompoundConfiguration can be implemented:
 {code}
   public IteratorMap.EntryString, String iterator() {
 {code}
 This enhancement would be useful in scenario where a mutable Configuration is 
 required.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8240) CompoundConfiguration should implement Iterable

2013-04-03 Thread Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-8240:
--

Status: Patch Available  (was: Open)

 CompoundConfiguration should implement Iterable
 ---

 Key: HBASE-8240
 URL: https://issues.apache.org/jira/browse/HBASE-8240
 Project: HBase
  Issue Type: Improvement
Reporter: Ted Yu
 Attachments: 8240-v1.txt, 8240-v2.txt, 8240-v3.txt, 8240-v4.txt


 Here is from hadoop Configuration class:
 {code}
 public class Configuration implements IterableMap.EntryString,String,
 {code}
 There're 3 addXX() methods for CompoundConfiguration:
 {code}
   public CompoundConfiguration add(final Configuration conf) {
   public CompoundConfiguration addWritableMap(
   final MapImmutableBytesWritable, ImmutableBytesWritable map) {
   public CompoundConfiguration addStringMap(final MapString, String map) {
 {code}
 Parameters to these methods all support iteration.
 We can enhance ImmutableConfigMap with the following new method:
 {code}
   public abstract java.util.Iterator iterator();
 {code}
 Then the following method of CompoundConfiguration can be implemented:
 {code}
   public IteratorMap.EntryString, String iterator() {
 {code}
 This enhancement would be useful in scenario where a mutable Configuration is 
 required.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8235) Adding inmemory CF attribute to LoadTest and PerformanceEvaluation tool

2013-04-03 Thread ramkrishna.s.vasudevan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-8235:
--

Attachment: HBASE-8235.patch

Patch for trunk.  

 Adding inmemory CF attribute to LoadTest and PerformanceEvaluation tool
 ---

 Key: HBASE-8235
 URL: https://issues.apache.org/jira/browse/HBASE-8235
 Project: HBase
  Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.98.0

 Attachments: HBASE-8235.patch


 Adding the inmemory CF attribute to the LoadTestTool and PerfEvaluation tool 
 will make it extensible to test such inmemory scenarios also.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8235) Adding inmemory CF attribute to LoadTest and PerformanceEvaluation tool

2013-04-03 Thread ramkrishna.s.vasudevan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-8235:
--

Attachment: HBASE-8235.patch

Patch for trunk

 Adding inmemory CF attribute to LoadTest and PerformanceEvaluation tool
 ---

 Key: HBASE-8235
 URL: https://issues.apache.org/jira/browse/HBASE-8235
 Project: HBase
  Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.98.0

 Attachments: HBASE-8235.patch, HBASE-8235.patch


 Adding the inmemory CF attribute to the LoadTestTool and PerfEvaluation tool 
 will make it extensible to test such inmemory scenarios also.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8235) Adding inmemory CF attribute to LoadTest and PerformanceEvaluation tool

2013-04-03 Thread ramkrishna.s.vasudevan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-8235:
--

Attachment: (was: HBASE-8235.patch)

 Adding inmemory CF attribute to LoadTest and PerformanceEvaluation tool
 ---

 Key: HBASE-8235
 URL: https://issues.apache.org/jira/browse/HBASE-8235
 Project: HBase
  Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.98.0

 Attachments: HBASE-8235.patch


 Adding the inmemory CF attribute to the LoadTestTool and PerfEvaluation tool 
 will make it extensible to test such inmemory scenarios also.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8235) Adding inmemory CF attribute to LoadTest and PerformanceEvaluation tool

2013-04-03 Thread Jean-Marc Spaggiari (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621070#comment-13621070
 ] 

Jean-Marc Spaggiari commented on HBASE-8235:


Pretty straightforward patch. Since inMemory is there also in 0.94 will it be 
possible to apply this patch on all the version to allow comparisons?

 Adding inmemory CF attribute to LoadTest and PerformanceEvaluation tool
 ---

 Key: HBASE-8235
 URL: https://issues.apache.org/jira/browse/HBASE-8235
 Project: HBase
  Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.98.0

 Attachments: HBASE-8235.patch


 Adding the inmemory CF attribute to the LoadTestTool and PerfEvaluation tool 
 will make it extensible to test such inmemory scenarios also.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8240) CompoundConfiguration should implement Iterable

2013-04-03 Thread Sergey Shelukhin (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-8240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621082#comment-13621082
 ] 

Sergey Shelukhin commented on HBASE-8240:
-

Test could just check values by getting the keys iterator returns and checking 
that value is the same from iterator and from get.
Not sure if we need to copy entire map, could make iterator of iterators; 
adding the comment to say that this is an expensive operation might be enough.
Would it be better to return UnmodifiableIteratorE (from guava) given that 
remove on this iterator will have no effect on configuration?

 CompoundConfiguration should implement Iterable
 ---

 Key: HBASE-8240
 URL: https://issues.apache.org/jira/browse/HBASE-8240
 Project: HBase
  Issue Type: Improvement
Reporter: Ted Yu
 Attachments: 8240-v1.txt, 8240-v2.txt, 8240-v3.txt, 8240-v4.txt


 Here is from hadoop Configuration class:
 {code}
 public class Configuration implements IterableMap.EntryString,String,
 {code}
 There're 3 addXX() methods for CompoundConfiguration:
 {code}
   public CompoundConfiguration add(final Configuration conf) {
   public CompoundConfiguration addWritableMap(
   final MapImmutableBytesWritable, ImmutableBytesWritable map) {
   public CompoundConfiguration addStringMap(final MapString, String map) {
 {code}
 Parameters to these methods all support iteration.
 We can enhance ImmutableConfigMap with the following new method:
 {code}
   public abstract java.util.Iterator iterator();
 {code}
 Then the following method of CompoundConfiguration can be implemented:
 {code}
   public IteratorMap.EntryString, String iterator() {
 {code}
 This enhancement would be useful in scenario where a mutable Configuration is 
 required.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk (with changes)

2013-04-03 Thread Sergey Shelukhin (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621103#comment-13621103
 ] 

Sergey Shelukhin commented on HBASE-7055:
-

Some of that is mentioned in the doc, and some in the JIRA(s) somewhere (not 
numbers though).
This only makes sense for certain workloads so I'd guess there will be no 
numbers.

 port HBASE-6371 tier-based compaction from 0.89-fb to trunk (with changes)
 --

 Key: HBASE-7055
 URL: https://issues.apache.org/jira/browse/HBASE-7055
 Project: HBase
  Issue Type: Task
  Components: Compaction
Affects Versions: 0.96.0
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
 Fix For: 0.95.0

 Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch, 
 HBASE-6371-v3-refactor-only-squashed.patch, 
 HBASE-6371-v4-refactor-only-squashed.patch, 
 HBASE-6371-v5-refactor-only-squashed.patch, HBASE-7055-v0.patch, 
 HBASE-7055-v1.patch, HBASE-7055-v2.patch, HBASE-7055-v3.patch, 
 HBASE-7055-v4.patch, HBASE-7055-v5.patch, HBASE-7055-v6.patch, 
 HBASE-7055-v7.patch, HBASE-7055-v7.patch


 See HBASE-6371 for details.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-8239) ChaosMonkey actions for root and meta don't work

2013-04-03 Thread Sergey Shelukhin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HBASE-8239:
---

Assignee: Sergey Shelukhin

 ChaosMonkey actions for root and meta don't work
 

 Key: HBASE-8239
 URL: https://issues.apache.org/jira/browse/HBASE-8239
 Project: HBase
  Issue Type: Bug
  Components: test
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Priority: Minor

 1) Meta action doesn't work, it always fails to find the server holding meta, 
 which has been the case for some time now.
 2) Root action is no longer relevant:
 13/04/01 20:36:25 INFO util.ChaosMonkey: Performing action: Restart region 
 server holding ROOT
 org.apache.hadoop.hbase.exceptions.TableNotFoundException: Cannot find row in 
 .META. for table: -ROOT-, row=-ROOT-,,99
   at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:164)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7401) Remove warning message about running 'hbase migrate'

2013-04-03 Thread Jonathan Hsieh (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621108#comment-13621108
 ] 

Jonathan Hsieh commented on HBASE-7401:
---

[~aleksshulman], thanks for pinging on this, I'll close the loop on this.

 Remove warning message about running 'hbase migrate'
 

 Key: HBASE-7401
 URL: https://issues.apache.org/jira/browse/HBASE-7401
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.3, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: hbase-7401.patch


 I was switching version to 0.94 (from 0.95-SNAPSHOT) and got this message 
 which refers to a program that does not exist.
 {code}
 org.apache.hadoop.hbase.util.FileSystemVersionException: File system needs to 
 be upgraded.  You have version null and I want version 7.  Run the 
 '${HBASE_HOME}/bin/hbase migrate' script.
 at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:324)
 at 
 org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:352)
 at 
 org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:128)
 at 
 org.apache.hadoop.hbase.master.MasterFileSystem.init(MasterFileSystem.java:113)
 at 
 org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:505)
 at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:362)
 at 
 org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.run(HMasterCommandLine.java:226)
 at java.lang.Thread.run(Thread.java:662)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7745) Import uses System.out.println instead of logging

2013-04-03 Thread stack (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-7745:
-

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Committed to trunk and 0.95 branch.  Thanks Gustavo for the patch (and Nick for 
shepherding)

 Import uses System.out.println instead of logging
 -

 Key: HBASE-7745
 URL: https://issues.apache.org/jira/browse/HBASE-7745
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Reporter: Nick Dimiduk
Assignee: Gustavo Anatoly
Priority: Trivial
 Fix For: 0.95.1, 0.98.0

 Attachments: HBASE-7745.patch


 Per the title.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8240) CompoundConfiguration should implement Iterable

2013-04-03 Thread Ted Yu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-8240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621112#comment-13621112
 ] 

Ted Yu commented on HBASE-8240:
---

Thanks for the review, Sergey.
I tried the following assertion:
{code}
  assertEquals(compoundConf.get(entry.getKey()), entry.getValue());
{code}
I got:
{code}
junit.framework.ComparisonFailure: 
expected:[/Users/tyu]/hadoop-http-auth-si... but 
was:[${user.home}]/hadoop-http-auth-si...
at junit.framework.Assert.assertEquals(Assert.java:100)
at junit.framework.Assert.assertEquals(Assert.java:107)
at junit.framework.TestCase.assertEquals(TestCase.java:269)
at 
org.apache.hadoop.hbase.TestCompoundConfiguration.testLaterConfigsOverrideEarlier(TestCompoundConfiguration.java:183)
{code}
There is logic in Configuration.get() that does substitution which I don't want 
to get involved.

 CompoundConfiguration should implement Iterable
 ---

 Key: HBASE-8240
 URL: https://issues.apache.org/jira/browse/HBASE-8240
 Project: HBase
  Issue Type: Improvement
Reporter: Ted Yu
 Attachments: 8240-v1.txt, 8240-v2.txt, 8240-v3.txt, 8240-v4.txt


 Here is from hadoop Configuration class:
 {code}
 public class Configuration implements IterableMap.EntryString,String,
 {code}
 There're 3 addXX() methods for CompoundConfiguration:
 {code}
   public CompoundConfiguration add(final Configuration conf) {
   public CompoundConfiguration addWritableMap(
   final MapImmutableBytesWritable, ImmutableBytesWritable map) {
   public CompoundConfiguration addStringMap(final MapString, String map) {
 {code}
 Parameters to these methods all support iteration.
 We can enhance ImmutableConfigMap with the following new method:
 {code}
   public abstract java.util.Iterator iterator();
 {code}
 Then the following method of CompoundConfiguration can be implemented:
 {code}
   public IteratorMap.EntryString, String iterator() {
 {code}
 This enhancement would be useful in scenario where a mutable Configuration is 
 required.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7401) Remove warning message about running 'hbase migrate'

2013-04-03 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621113#comment-13621113
 ] 

stack commented on HBASE-7401:
--

+1

 Remove warning message about running 'hbase migrate'
 

 Key: HBASE-7401
 URL: https://issues.apache.org/jira/browse/HBASE-7401
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.3, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: hbase-7401.patch


 I was switching version to 0.94 (from 0.95-SNAPSHOT) and got this message 
 which refers to a program that does not exist.
 {code}
 org.apache.hadoop.hbase.util.FileSystemVersionException: File system needs to 
 be upgraded.  You have version null and I want version 7.  Run the 
 '${HBASE_HOME}/bin/hbase migrate' script.
 at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:324)
 at 
 org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:352)
 at 
 org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:128)
 at 
 org.apache.hadoop.hbase.master.MasterFileSystem.init(MasterFileSystem.java:113)
 at 
 org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:505)
 at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:362)
 at 
 org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.run(HMasterCommandLine.java:226)
 at java.lang.Thread.run(Thread.java:662)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8251) enable SSH before assign META on Master startup

2013-04-03 Thread rajeshbabu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-8251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621117#comment-13621117
 ] 

rajeshbabu commented on HBASE-8251:
---

[~jeason]
Correct, If a meta region in transition during master startup, then not 
enabling SSH until META assigned.

Patch looks good. Nice test.

 enable SSH before assign META on Master startup
 ---

 Key: HBASE-8251
 URL: https://issues.apache.org/jira/browse/HBASE-8251
 Project: HBase
  Issue Type: Bug
  Components: master, Region Assignment
Affects Versions: 0.94.6
Reporter: Jieshan Bean
Assignee: Jieshan Bean
 Attachments: HBASE-8251-94.patch


 I think HBASE-5918 could not fix this issue. In HMaster#assignRootAndMeta:
 1. Assign ROOT.
 2. Block until ROOT be opened.
 3. Assign META.
 4. Block until META be opened.
 SSH is enabled after step 4. So if the RS who host ROOT dies before step 4, 
 master will be blocked.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7401) Remove warning message about running 'hbase migrate'

2013-04-03 Thread Jonathan Hsieh (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-7401:
--

   Resolution: Fixed
Fix Version/s: 0.94.7
   0.98.0
   0.96.0
   0.95.1
   Status: Resolved  (was: Patch Available)

 Remove warning message about running 'hbase migrate'
 

 Key: HBASE-7401
 URL: https://issues.apache.org/jira/browse/HBASE-7401
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.3, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.95.1, 0.96.0, 0.98.0, 0.94.7

 Attachments: hbase-7401.patch


 I was switching version to 0.94 (from 0.95-SNAPSHOT) and got this message 
 which refers to a program that does not exist.
 {code}
 org.apache.hadoop.hbase.util.FileSystemVersionException: File system needs to 
 be upgraded.  You have version null and I want version 7.  Run the 
 '${HBASE_HOME}/bin/hbase migrate' script.
 at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:324)
 at 
 org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:352)
 at 
 org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:128)
 at 
 org.apache.hadoop.hbase.master.MasterFileSystem.init(MasterFileSystem.java:113)
 at 
 org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:505)
 at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:362)
 at 
 org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.run(HMasterCommandLine.java:226)
 at java.lang.Thread.run(Thread.java:662)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8240) CompoundConfiguration should implement Iterable

2013-04-03 Thread Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-8240:
--

Attachment: 8240-v6.txt

Patch v6 adds a warning for using iterator.

Also introduced UnmodifiableIterator as Sergey suggested.

I tried CompositeIterator approach but its semantics is not correct: iterator 
would return multiple values for the same key.

 CompoundConfiguration should implement Iterable
 ---

 Key: HBASE-8240
 URL: https://issues.apache.org/jira/browse/HBASE-8240
 Project: HBase
  Issue Type: Improvement
Reporter: Ted Yu
 Attachments: 8240-v1.txt, 8240-v2.txt, 8240-v3.txt, 8240-v4.txt, 
 8240-v6.txt


 Here is from hadoop Configuration class:
 {code}
 public class Configuration implements IterableMap.EntryString,String,
 {code}
 There're 3 addXX() methods for CompoundConfiguration:
 {code}
   public CompoundConfiguration add(final Configuration conf) {
   public CompoundConfiguration addWritableMap(
   final MapImmutableBytesWritable, ImmutableBytesWritable map) {
   public CompoundConfiguration addStringMap(final MapString, String map) {
 {code}
 Parameters to these methods all support iteration.
 We can enhance ImmutableConfigMap with the following new method:
 {code}
   public abstract java.util.Iterator iterator();
 {code}
 Then the following method of CompoundConfiguration can be implemented:
 {code}
   public IteratorMap.EntryString, String iterator() {
 {code}
 This enhancement would be useful in scenario where a mutable Configuration is 
 required.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-8256) Add category Flaky for tests which are flaky

2013-04-03 Thread Jimmy Xiang (JIRA) 
Jimmy Xiang created HBASE-8256:
--

 Summary: Add category Flaky for tests which are flaky
 Key: HBASE-8256
 URL: https://issues.apache.org/jira/browse/HBASE-8256
 Project: HBase
  Issue Type: Bug
  Components: build, test
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang


To make the Jenkin build more useful, it is good to keep it blue/green. We can 
mark those flaky tests flaky, and don't run them by default.  However, people 
can still run them.  We can also set up a Jekin build just for those flaky 
tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8251) enable SSH before assign META on Master startup

2013-04-03 Thread Ted Yu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-8251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621130#comment-13621130
 ] 

Ted Yu commented on HBASE-8251:
---

@Jieshan:

TestMasterStartup.java is not in trunk.
Can you come up with patch for trunk with TestMasterStartup ?

 enable SSH before assign META on Master startup
 ---

 Key: HBASE-8251
 URL: https://issues.apache.org/jira/browse/HBASE-8251
 Project: HBase
  Issue Type: Bug
  Components: master, Region Assignment
Affects Versions: 0.94.6
Reporter: Jieshan Bean
Assignee: Jieshan Bean
 Attachments: HBASE-8251-94.patch


 I think HBASE-5918 could not fix this issue. In HMaster#assignRootAndMeta:
 1. Assign ROOT.
 2. Block until ROOT be opened.
 3. Assign META.
 4. Block until META be opened.
 SSH is enabled after step 4. So if the RS who host ROOT dies before step 4, 
 master will be blocked.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8256) Add category Flaky for tests which are flaky

2013-04-03 Thread Jimmy Xiang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-8256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621134#comment-13621134
 ] 

Jimmy Xiang commented on HBASE-8256:


From Andy on dev@hbase:

{quote}
Could we introduce a junit category for flakey tests such that they don't
run unless we activate them with some flag to maven?
{quote}


 Add category Flaky for tests which are flaky
 

 Key: HBASE-8256
 URL: https://issues.apache.org/jira/browse/HBASE-8256
 Project: HBase
  Issue Type: Bug
  Components: build, test
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang

 To make the Jenkin build more useful, it is good to keep it blue/green. We 
 can mark those flaky tests flaky, and don't run them by default.  However, 
 people can still run them.  We can also set up a Jekin build just for those 
 flaky tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7401) Remove warning message about running 'hbase migrate'

2013-04-03 Thread Jonathan Hsieh (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-7401:
--

Component/s: Usability

 Remove warning message about running 'hbase migrate'
 

 Key: HBASE-7401
 URL: https://issues.apache.org/jira/browse/HBASE-7401
 Project: HBase
  Issue Type: Bug
  Components: Usability
Affects Versions: 0.94.3, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.95.1, 0.96.0, 0.98.0, 0.94.7

 Attachments: hbase-7401.patch


 I was switching version to 0.94 (from 0.95-SNAPSHOT) and got this message 
 which refers to a program that does not exist.
 {code}
 org.apache.hadoop.hbase.util.FileSystemVersionException: File system needs to 
 be upgraded.  You have version null and I want version 7.  Run the 
 '${HBASE_HOME}/bin/hbase migrate' script.
 at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:324)
 at 
 org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:352)
 at 
 org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:128)
 at 
 org.apache.hadoop.hbase.master.MasterFileSystem.init(MasterFileSystem.java:113)
 at 
 org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:505)
 at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:362)
 at 
 org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.run(HMasterCommandLine.java:226)
 at java.lang.Thread.run(Thread.java:662)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7649) client retry timeout doesn't need to do x2 fallback when going to different server

2013-04-03 Thread Sergey Shelukhin (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-7649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621139#comment-13621139
 ] 

Sergey Shelukhin commented on HBASE-7649:
-

Note - r is not updated, I will rebase patch and update it soon

 client retry timeout doesn't need to do x2 fallback when going to different 
 server
 --

 Key: HBASE-7649
 URL: https://issues.apache.org/jira/browse/HBASE-7649
 Project: HBase
  Issue Type: Improvement
  Components: Client, MTTR
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
 Fix For: 0.95.0

 Attachments: HBASE-7649-v0.patch, HBASE-7649-v1.patch, 
 HBASE-7649-v2.patch, HBASE-7649-v2.patch, HBASE-7649-v2.patch, 
 HBASE-7649-v3.patch, HBASE-7649-v4.patch, HBASE-7649-v5.patch, 
 HBASE-7649-v6.patch, HBASE-7649-v7.patch


 See HBASE-7520. When we go to server A, get a bunch of failures, then finally 
 learn the region is on B it doesn't make sense to wait for 30 seconds before 
 going to B.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8220) can we record the count opened HTable for HTablePool

2013-04-03 Thread cuijianwei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

cuijianwei updated HBASE-8220:
--

Attachment: HBASE-8220-0.94.3-v4.txt

 can we record the count opened HTable for HTablePool
 

 Key: HBASE-8220
 URL: https://issues.apache.org/jira/browse/HBASE-8220
 Project: HBase
  Issue Type: Improvement
  Components: Client
Affects Versions: 0.94.3
Reporter: cuijianwei
 Attachments: HBASE-8220-0.94.3.txt, HBASE-8220-0.94.3.txt, 
 HBASE-8220-0.94.3.txt-v2, HBASE-8220-0.94.3-v2.txt, HBASE-8220-0.94.3-v3.txt, 
 HBASE-8220-0.94.3-v4.txt


 In HTablePool, we have a method getCurrentPoolSize(...) to get how many 
 opened HTable has been pooled. However, we don't know ConcurrentOpenedHTable 
 which means the count of HTable get from HTablePool.getTable(...) and don't 
 return to HTablePool by PooledTable.close(). The ConcurrentOpenedHTable may 
 be meaningful because it indicates how many HTables should be opened for the 
 application which may help us set the appropriate MaxSize of HTablePool. 
 Therefore, we can and a ConcurrentOpenedHTable as a counter in HTablePool.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8220) can we record the count opened HTable for HTablePool

2013-04-03 Thread cuijianwei (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-8220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621144#comment-13621144
 ] 

cuijianwei commented on HBASE-8220:
---

Thanks for your comment. I make a new patch to fix the bug, however, it's a 
little slow to compile the project, I will run the test locally and given the 
test result.

 can we record the count opened HTable for HTablePool
 

 Key: HBASE-8220
 URL: https://issues.apache.org/jira/browse/HBASE-8220
 Project: HBase
  Issue Type: Improvement
  Components: Client
Affects Versions: 0.94.3
Reporter: cuijianwei
 Attachments: HBASE-8220-0.94.3.txt, HBASE-8220-0.94.3.txt, 
 HBASE-8220-0.94.3.txt-v2, HBASE-8220-0.94.3-v2.txt, HBASE-8220-0.94.3-v3.txt, 
 HBASE-8220-0.94.3-v4.txt


 In HTablePool, we have a method getCurrentPoolSize(...) to get how many 
 opened HTable has been pooled. However, we don't know ConcurrentOpenedHTable 
 which means the count of HTable get from HTablePool.getTable(...) and don't 
 return to HTablePool by PooledTable.close(). The ConcurrentOpenedHTable may 
 be meaningful because it indicates how many HTables should be opened for the 
 application which may help us set the appropriate MaxSize of HTablePool. 
 Therefore, we can and a ConcurrentOpenedHTable as a counter in HTablePool.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8165) Update our protobuf to 2.5 from 2.4.1

2013-04-03 Thread stack (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-8165:
-

Attachment: 8165v2.txt

Rebase.

Seems good locally but let me run it through hadoopqa.

 Update our protobuf to 2.5 from 2.4.1
 -

 Key: HBASE-8165
 URL: https://issues.apache.org/jira/browse/HBASE-8165
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
 Fix For: 0.95.0

 Attachments: 8165.txt, 8165v2.txt


 Update to new 2.5 pb.  Some speedups and a new PARSER idiom that bypasses 
 making a builder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-8257) [refGuide] adding object design section in Schema Design

2013-04-03 Thread Doug Meil (JIRA) 
Doug Meil created HBASE-8257:


 Summary: [refGuide] adding object design section in Schema Design 
 Key: HBASE-8257
 URL: https://issues.apache.org/jira/browse/HBASE-8257
 Project: HBase
  Issue Type: New Feature
Reporter: Doug Meil
Assignee: Doug Meil
 Attachments: schema_design_hbase_8257.xml.patch

Added an object design sub-section in the Customer/Order Case Study in the 
Schema Design chapter.

Renamed the Customer / Sales Case Study to Customer/Order because I found I 
was using both sales and order interchangeably and I think that was 
confusing, so now I'm only referring to Order

Minor cleanup:  fixed some of the section-ids in the Case Study section (from 
yesterday)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8220) can we record the count opened HTable for HTablePool

2013-04-03 Thread cuijianwei (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-8220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621152#comment-13621152
 ] 

cuijianwei commented on HBASE-8220:
---

I have run TestHTablePool.java, all uts in TestHTabelPool have passed locally 
with HBASE-8220-0.94.3-v4.txt. Could please help to make sure 
HBASE-8220-0.94.3-v4.txt is OK?

 can we record the count opened HTable for HTablePool
 

 Key: HBASE-8220
 URL: https://issues.apache.org/jira/browse/HBASE-8220
 Project: HBase
  Issue Type: Improvement
  Components: Client
Affects Versions: 0.94.3
Reporter: cuijianwei
 Attachments: HBASE-8220-0.94.3.txt, HBASE-8220-0.94.3.txt, 
 HBASE-8220-0.94.3.txt-v2, HBASE-8220-0.94.3-v2.txt, HBASE-8220-0.94.3-v3.txt, 
 HBASE-8220-0.94.3-v4.txt


 In HTablePool, we have a method getCurrentPoolSize(...) to get how many 
 opened HTable has been pooled. However, we don't know ConcurrentOpenedHTable 
 which means the count of HTable get from HTablePool.getTable(...) and don't 
 return to HTablePool by PooledTable.close(). The ConcurrentOpenedHTable may 
 be meaningful because it indicates how many HTables should be opened for the 
 application which may help us set the appropriate MaxSize of HTablePool. 
 Therefore, we can and a ConcurrentOpenedHTable as a counter in HTablePool.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8257) [refGuide] adding object design section in Schema Design

2013-04-03 Thread Doug Meil (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-8257:
-

Attachment: schema_design_hbase_8257.xml.patch

 [refGuide] adding object design section in Schema Design 
 -

 Key: HBASE-8257
 URL: https://issues.apache.org/jira/browse/HBASE-8257
 Project: HBase
  Issue Type: New Feature
Reporter: Doug Meil
Assignee: Doug Meil
 Attachments: schema_design_hbase_8257.xml.patch


 Added an object design sub-section in the Customer/Order Case Study in the 
 Schema Design chapter.
 Renamed the Customer / Sales Case Study to Customer/Order because I found 
 I was using both sales and order interchangeably and I think that was 
 confusing, so now I'm only referring to Order
 Minor cleanup:  fixed some of the section-ids in the Case Study section (from 
 yesterday)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8257) [refGuide] adding object design section in Schema Design

2013-04-03 Thread Doug Meil (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-8257:
-

Status: Patch Available  (was: Open)

 [refGuide] adding object design section in Schema Design 
 -

 Key: HBASE-8257
 URL: https://issues.apache.org/jira/browse/HBASE-8257
 Project: HBase
  Issue Type: New Feature
Reporter: Doug Meil
Assignee: Doug Meil
 Attachments: schema_design_hbase_8257.xml.patch


 Added an object design sub-section in the Customer/Order Case Study in the 
 Schema Design chapter.
 Renamed the Customer / Sales Case Study to Customer/Order because I found 
 I was using both sales and order interchangeably and I think that was 
 confusing, so now I'm only referring to Order
 Minor cleanup:  fixed some of the section-ids in the Case Study section (from 
 yesterday)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8257) [refGuide] adding object design section in Schema Design

2013-04-03 Thread Doug Meil (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-8257:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

 [refGuide] adding object design section in Schema Design 
 -

 Key: HBASE-8257
 URL: https://issues.apache.org/jira/browse/HBASE-8257
 Project: HBase
  Issue Type: New Feature
Reporter: Doug Meil
Assignee: Doug Meil
 Attachments: schema_design_hbase_8257.xml.patch


 Added an object design sub-section in the Customer/Order Case Study in the 
 Schema Design chapter.
 Renamed the Customer / Sales Case Study to Customer/Order because I found 
 I was using both sales and order interchangeably and I think that was 
 confusing, so now I'm only referring to Order
 Minor cleanup:  fixed some of the section-ids in the Case Study section (from 
 yesterday)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7615) Add metrics for snapshots

2013-04-03 Thread Elliott Clark (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-7615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621165#comment-13621165
 ] 

Elliott Clark commented on HBASE-7615:
--

MetricMutableStat/MutableStat would have been what I used.

On splits, yeah you are probably correct.  I'll file a jira to change those.

 Add metrics for snapshots
 -

 Key: HBASE-7615
 URL: https://issues.apache.org/jira/browse/HBASE-7615
 Project: HBase
  Issue Type: Sub-task
Reporter: Ted Yu
Assignee: Matteo Bertozzi
 Attachments: HBASE-7615-0.94.patch, HBASE-7615-0.94-v1.patch, 
 HBASE-7615.patch, HBASE-7615.patch, HBASE-7615-v0.patch, 
 HBASE-7615-v0-ui-corrupted.png, HBASE-7615-v0-ui.png, HBASE-7615-v1.patch, 
 HBASE-7615-v2.patch, HBASE-7615-v3.patch


 Metrics should be added for snapshot.
 From Matteo: Output that we have in SnapshotInfo should be covered:
 %d HFiles (%d in archive), total size %s (%.2f%% %s shared with the source 
 table)
 Jesse mentioned snaphot counts, average time to completion.
 I think we should have counts for successful / failed snapshots.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7704) migration tool that checks presence of HFile V1 files

2013-04-03 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-7704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621166#comment-13621166
 ] 

stack commented on HBASE-7704:
--

Here is more on how I think this would work (if anyone is listening).

We want a script that we can run that goes through all the hfiles in an hbase 
install and returns code 0 if no v1 files found and non-zero if any v1 files 
are found (It should fail out as soon as it finds a v1 file).

This script can be run while an hbase cluster is up.

Script should be called something like v1Free or noV1HFiles

Script can be in java or jruby since a one-off just needed to check an hbase 
install BEFORE we do an upgrade.

Script should implement 
http://hadoop.apache.org/docs/r2.0.3-alpha/api/org/apache/hadoop/util/Tool.html 
so it can pick up configuration to find hbase to run against (I suppose this 
means it is a java script).

Ideally, script should be done in a manner such that if we need to use it in a 
mapreduce job, it'd be easy to do (we do not need the mapreduce job as part of 
this JIRA I would say).  I think this means that in the script there is a 
method which takes fully qualified hfile Path and returns true/false.

Would suggest that script have an executor service and take on the command line 
how many concurrent threads to run w/ a reasonable default so that the checking 
is done in parallel.

So, the script would walk the hbase.rootdir looking for hfiles.  Look at hbase 
file utils because will need to skip over special files and directories.  Per 
file found, it would read in its metadata and check for v1.  See hfile for how 
it finds metadata at end of file.


 migration tool that checks presence of HFile V1 files
 -

 Key: HBASE-7704
 URL: https://issues.apache.org/jira/browse/HBASE-7704
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Priority: Blocker
 Fix For: 0.95.1


 Below was Stack's comment from HBASE-7660:
 Regards the migration 'tool', or 'tool' to check for presence of v1 files, I 
 imagine it as an addition to the hfile tool 
 http://hbase.apache.org/book.html#hfile_tool2 The hfile tool already takes a 
 bunch of args including printing out meta. We could add an option to print 
 out version only – or return 1 if version 1 or some such – and then do a bit 
 of code to just list all hfiles and run this script against each. Could MR it 
 if too many files.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-8258) Make mapreduce tests pass on hadoop2

2013-04-03 Thread stack (JIRA) 
stack created HBASE-8258:


 Summary: Make mapreduce tests pass on hadoop2
 Key: HBASE-8258
 URL: https://issues.apache.org/jira/browse/HBASE-8258
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Reporter: stack
Priority: Blocker
 Fix For: 0.95.1


HBASE-7904 was a first attempt at making this work but it got lost in the weeds.

This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o 
breaking mapreduce on hadoop1)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7704) migration tool that checks presence of HFile V1 files

2013-04-03 Thread Himanshu Vashishtha (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-7704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621187#comment-13621187
 ] 

Himanshu Vashishtha commented on HBASE-7704:


I have started working on this. I am using the approach of a java standalone 
program (aka tool), which has an executor service. It takes in a hbase 
directory path, and look for table directories (checking the presence of 
.tableinfo file). Then, it dives in to the table directory and look for any v1 
file, and list it out. I was not thinking to make it a failfast one, as it is 
good to see all such files in one go (they may be in different region/tables)?

Yes, this should be done before user actually migrate. As we have removed all 
v1 related code from 0.95, it can't compact on its own. Therefore, user will 
run this script and check for the presence of any v1 file, and run compaction 
on it.?

 migration tool that checks presence of HFile V1 files
 -

 Key: HBASE-7704
 URL: https://issues.apache.org/jira/browse/HBASE-7704
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Priority: Blocker
 Fix For: 0.95.1


 Below was Stack's comment from HBASE-7660:
 Regards the migration 'tool', or 'tool' to check for presence of v1 files, I 
 imagine it as an addition to the hfile tool 
 http://hbase.apache.org/book.html#hfile_tool2 The hfile tool already takes a 
 bunch of args including printing out meta. We could add an option to print 
 out version only – or return 1 if version 1 or some such – and then do a bit 
 of code to just list all hfiles and run this script against each. Could MR it 
 if too many files.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5608) MR testcases are failing in QA builds

2013-04-03 Thread Nick Dimiduk (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621190#comment-13621190
 ] 

Nick Dimiduk commented on HBASE-5608:
-

All of these tests pass in recent jenkins builds on EC2. They're not reported 
in test results [0] for some reason, but they're there in the console output 
[1]. Resolve as not-a-bug?

[0]: 
http://54.241.6.143/job/HBase-TRUNK/lastBuild/org.apache.hbase$hbase-server/testReport/org.apache.hadoop.hbase.mapreduce/
[1]: http://54.241.6.143/job/HBase-TRUNK/lastBuild/consoleText

 MR testcases are failing in QA builds
 -

 Key: HBASE-5608
 URL: https://issues.apache.org/jira/browse/HBASE-5608
 Project: HBase
  Issue Type: Bug
  Components: build, mapreduce, test
Affects Versions: 0.92.2
 Environment: Hadoop QA - precommit builds
Reporter: Laxman
Priority: Blocker
  Labels: build-failure, mapreduce, test-fail

 Many of the MR testcases are failing in PreCommit builds (triggered by Hadoop 
 QA).
 Failing testcases are
 a) TestImportTsv
 b) TestHFileOutputFormat
 c) TestTableMapReduce

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha

2013-04-03 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621198#comment-13621198
 ] 

stack commented on HBASE-7904:
--

[~sseth]

bq. Will pull that into the 2.0.4-alpha branch.

Thanks.  I added an ask to MAPREDUCE-5083.  I could commit it but sounds like 
you are more on it... thanks for taking care of us.

I made HBASE-8258 for making hbase-side changes.


 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
  Components: mapreduce
Reporter: Ted Yu
Priority: Critical
 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8119) Optimize StochasticLoadBalancer

2013-04-03 Thread Elliott Clark (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621200#comment-13621200
 ] 

Elliott Clark commented on HBASE-8119:
--

+1.  Lets try this out on 0.95 releases.  I'm still concerned about the one 
multiplier, but real world clusters will be the best judge.

 Optimize StochasticLoadBalancer
 ---

 Key: HBASE-8119
 URL: https://issues.apache.org/jira/browse/HBASE-8119
 Project: HBase
  Issue Type: Bug
  Components: Region Assignment
Affects Versions: 0.95.0
Reporter: Enis Soztutar
Priority: Critical
 Fix For: 0.95.1


 On a 5 node trunk cluster, I ran into a weird problem with 
 StochasticLoadBalancer:
 server1   Thu Mar 14 03:42:50 UTC 20130.0 33
 server2   Thu Mar 14 03:47:53 UTC 20130.0 34
 server3   Thu Mar 14 03:46:53 UTC 2013465.0   42
 server4   Thu Mar 14 03:47:53 UTC 201311455.0 282
 server5   Thu Mar 14 03:47:53 UTC 20130.0 34
 Total:5   11920   425
 Notice that server4 has 282 regions, while the others have much less. Plus 
 for one table with 260 regions has been super imbalanced:
 {code}
 Regions by Region Server
 Region Server Region Count
 http://server3:60030/ 10
 http://server4:60030/ 250
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7704) migration tool that checks presence of HFile V1 files

2013-04-03 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-7704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621212#comment-13621212
 ] 

stack commented on HBASE-7704:
--

[~himan...@cloudera.com] Thanks.

bq. It takes in a hbase directory path, and look for table directories 
(checking the presence of .tableinfo file). 

If it is a 'Tool', then, it will read Configurations off the CLASSPATH.  If so, 
it will have access to the hbase-default hbase.rootdir configuration. Given 
this, does it need to take a directory as input?  Also, since a Tool, you could 
override the property with -Dhbase.rootdir.

Or are you thinking that you should be able to point the tool at a subset of 
the hbase.rootdir content, say the a table directory?

That could be good to have (if not directory specified, use hbase.rootdir from 
the Configuration, else use what is passed).

Would suggest you Options parser that is in the HFilePRettyPrint.

Ok on NOT failing fast.  Lets go your route.  Going your route, seeing the 
paths to the v1 files will let operator know what tables need compacting.

Thanks boss.

 migration tool that checks presence of HFile V1 files
 -

 Key: HBASE-7704
 URL: https://issues.apache.org/jira/browse/HBASE-7704
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Priority: Blocker
 Fix For: 0.95.1


 Below was Stack's comment from HBASE-7660:
 Regards the migration 'tool', or 'tool' to check for presence of v1 files, I 
 imagine it as an addition to the hfile tool 
 http://hbase.apache.org/book.html#hfile_tool2 The hfile tool already takes a 
 bunch of args including printing out meta. We could add an option to print 
 out version only – or return 1 if version 1 or some such – and then do a bit 
 of code to just list all hfiles and run this script against each. Could MR it 
 if too many files.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7615) Add metrics for snapshots

2013-04-03 Thread Matteo Bertozzi (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-7615:
---

Attachment: HBASE-7615.patch
HBASE-7615-v4.patch

 Add metrics for snapshots
 -

 Key: HBASE-7615
 URL: https://issues.apache.org/jira/browse/HBASE-7615
 Project: HBase
  Issue Type: Sub-task
Reporter: Ted Yu
Assignee: Matteo Bertozzi
 Attachments: HBASE-7615-0.94.patch, HBASE-7615-0.94-v1.patch, 
 HBASE-7615.patch, HBASE-7615.patch, HBASE-7615.patch, HBASE-7615-v0.patch, 
 HBASE-7615-v0-ui-corrupted.png, HBASE-7615-v0-ui.png, HBASE-7615-v1.patch, 
 HBASE-7615-v2.patch, HBASE-7615-v3.patch, HBASE-7615-v4.patch


 Metrics should be added for snapshot.
 From Matteo: Output that we have in SnapshotInfo should be covered:
 %d HFiles (%d in archive), total size %s (%.2f%% %s shared with the source 
 table)
 Jesse mentioned snaphot counts, average time to completion.
 I think we should have counts for successful / failed snapshots.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7649) client retry timeout doesn't need to do x2 fallback when going to different server

2013-04-03 Thread Sergey Shelukhin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HBASE-7649:


Attachment: HBASE-7649-v8.patch

 client retry timeout doesn't need to do x2 fallback when going to different 
 server
 --

 Key: HBASE-7649
 URL: https://issues.apache.org/jira/browse/HBASE-7649
 Project: HBase
  Issue Type: Improvement
  Components: Client, MTTR
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
 Fix For: 0.95.0

 Attachments: HBASE-7649-v0.patch, HBASE-7649-v1.patch, 
 HBASE-7649-v2.patch, HBASE-7649-v2.patch, HBASE-7649-v2.patch, 
 HBASE-7649-v3.patch, HBASE-7649-v4.patch, HBASE-7649-v5.patch, 
 HBASE-7649-v6.patch, HBASE-7649-v7.patch, HBASE-7649-v8.patch


 See HBASE-7520. When we go to server A, get a bunch of failures, then finally 
 learn the region is on B it doesn't make sense to wait for 30 seconds before 
 going to B.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7649) client retry timeout doesn't need to do x2 fallback when going to different server

2013-04-03 Thread Sergey Shelukhin (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-7649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621272#comment-13621272
 ] 

Sergey Shelukhin commented on HBASE-7649:
-

Some feedback from r, also updated r

 client retry timeout doesn't need to do x2 fallback when going to different 
 server
 --

 Key: HBASE-7649
 URL: https://issues.apache.org/jira/browse/HBASE-7649
 Project: HBase
  Issue Type: Improvement
  Components: Client, MTTR
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
 Fix For: 0.95.0

 Attachments: HBASE-7649-v0.patch, HBASE-7649-v1.patch, 
 HBASE-7649-v2.patch, HBASE-7649-v2.patch, HBASE-7649-v2.patch, 
 HBASE-7649-v3.patch, HBASE-7649-v4.patch, HBASE-7649-v5.patch, 
 HBASE-7649-v6.patch, HBASE-7649-v7.patch, HBASE-7649-v8.patch


 See HBASE-7520. When we go to server A, get a bunch of failures, then finally 
 learn the region is on B it doesn't make sense to wait for 30 seconds before 
 going to B.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-8240) CompoundConfiguration should implement Iterable

2013-04-03 Thread Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu reassigned HBASE-8240:
-

Assignee: Ted Yu

 CompoundConfiguration should implement Iterable
 ---

 Key: HBASE-8240
 URL: https://issues.apache.org/jira/browse/HBASE-8240
 Project: HBase
  Issue Type: Improvement
Reporter: Ted Yu
Assignee: Ted Yu
 Attachments: 8240-v1.txt, 8240-v2.txt, 8240-v3.txt, 8240-v4.txt, 
 8240-v6.txt


 Here is from hadoop Configuration class:
 {code}
 public class Configuration implements IterableMap.EntryString,String,
 {code}
 There're 3 addXX() methods for CompoundConfiguration:
 {code}
   public CompoundConfiguration add(final Configuration conf) {
   public CompoundConfiguration addWritableMap(
   final MapImmutableBytesWritable, ImmutableBytesWritable map) {
   public CompoundConfiguration addStringMap(final MapString, String map) {
 {code}
 Parameters to these methods all support iteration.
 We can enhance ImmutableConfigMap with the following new method:
 {code}
   public abstract java.util.Iterator iterator();
 {code}
 Then the following method of CompoundConfiguration can be implemented:
 {code}
   public IteratorMap.EntryString, String iterator() {
 {code}
 This enhancement would be useful in scenario where a mutable Configuration is 
 required.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7649) client retry timeout doesn't need to do x2 fallback when going to different server

2013-04-03 Thread Nicolas Liochon (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-7649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621292#comment-13621292
 ] 

Nicolas Liochon commented on HBASE-7649:


I read the code, looking for stuff like multithreading issue. Haven't found 
any. Code is clean. I'm ok with what the patch is doing, there is a unit test. 
So +1 for commit :-).

 client retry timeout doesn't need to do x2 fallback when going to different 
 server
 --

 Key: HBASE-7649
 URL: https://issues.apache.org/jira/browse/HBASE-7649
 Project: HBase
  Issue Type: Improvement
  Components: Client, MTTR
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
 Fix For: 0.95.0

 Attachments: HBASE-7649-v0.patch, HBASE-7649-v1.patch, 
 HBASE-7649-v2.patch, HBASE-7649-v2.patch, HBASE-7649-v2.patch, 
 HBASE-7649-v3.patch, HBASE-7649-v4.patch, HBASE-7649-v5.patch, 
 HBASE-7649-v6.patch, HBASE-7649-v7.patch, HBASE-7649-v8.patch


 See HBASE-7520. When we go to server A, get a bunch of failures, then finally 
 learn the region is on B it doesn't make sense to wait for 30 seconds before 
 going to B.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4947) CopyTable warns about being unable to find table from wrong instance of HBase

2013-04-03 Thread Nick Dimiduk (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-4947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk updated HBASE-4947:


Component/s: mapreduce

 CopyTable warns about being unable to find table from wrong instance of HBase
 -

 Key: HBASE-4947
 URL: https://issues.apache.org/jira/browse/HBASE-4947
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 0.90.4, 0.92.0
Reporter: Jonathan Hsieh
Priority: Minor

 Using CopyTable with two clusters, test01 and test02.
 cluster cluster01 has a table called 'table' which has a little bit of data 
 in it.
 cluster cluster02 has a table called 'table2' which have the same schema as 
 cluster01's 'table'
 Here's the command line used from a machine in cluster01
 {code}
 $ hbase org.apache.hadoop.hbase.mapreduce.CopyTable 
 --peer.adr=cluster02:2181:/hbase --new.name=table2 table
 {code}
 Job succeeds, but gives a scary extraneous warning:
 {code}
 11/12/04 10:26:10 WARN client.HConnectionManager$HConnectionImplementation: 
 Encountered problems when prefetch META table:
 org.apache.hadoop.hbase.TableNotFoundException: Cannot find row in .META. for 
 table: table2, row=table2,,99
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:136)
 at 
 org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:95)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:649)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:703)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:594)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:559)
 at org.apache.hadoop.hbase.client.HTable.init(HTable.java:173)
 at org.apache.hadoop.hbase.client.HTable.init(HTable.java:147)
 at 
 org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:198)
 at 
 org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
 at 
 org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
 at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:869)
 at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Unknown Source)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
 at 
 org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
 at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
 at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
 at 
 org.apache.hadoop.hbase.mapreduce.CopyTable.main(CopyTable.java:201)
 11/12/04 10:26:10 ERROR mapreduce.TableOutputFormat: 
 org.apache.hadoop.hbase.TableNotFoundException: table2
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7692) [experiment] Adopt Orderly library

2013-04-03 Thread Nick Dimiduk (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk updated HBASE-7692:


Issue Type: New Feature  (was: Sub-task)
Parent: (was: HBASE-8089)

 [experiment] Adopt Orderly library
 --

 Key: HBASE-7692
 URL: https://issues.apache.org/jira/browse/HBASE-7692
 Project: HBase
  Issue Type: New Feature
  Components: util
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
 Attachments: HBASE-7692.v1.patch, HBASE-7692.v2.patch, 
 HBASE-7692.v3.patch, HBASE-7692.v4.patch, HBASE-7692.v5.patch


 The current Bytes utility class works, but produces output that does not 
 maintain the native sort ordering of the input value. This results in, for 
 example, a negative value that does not necessarily sort before a positive 
 value. HBase should provide a canonical implementation of such a 
 serialization format so that third-parties can reliably build on top of 
 HBase. This will allow an implementation for HIVE-3634, HIVE-2599, or 
 HIVE-2903 that is compatible with similar features in Pig.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7221) [experiment] RowKey utility class for rowkey construction

2013-04-03 Thread Nick Dimiduk (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk updated HBASE-7221:


Issue Type: New Feature  (was: Sub-task)
Parent: (was: HBASE-8089)

 [experiment] RowKey utility class for rowkey construction
 -

 Key: HBASE-7221
 URL: https://issues.apache.org/jira/browse/HBASE-7221
 Project: HBase
  Issue Type: New Feature
  Components: util
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
 Attachments: HBASE_7221.patch, hbase-common_hbase_7221_2.patch, 
 hbase-common_hbase_7221_v3.patch, hbase-common_hbase_7221_v4.patch, 
 hbase-server_hbase_7221_v5.patch, hbase-server_hbase_7221_v6.patch


 A common question in the dist-lists is how to construct rowkeys, particularly 
 composite keys.  Put/Get/Scan specifies byte[] as the rowkey, but it's up to 
 you to sensibly populate that byte-array, and that's where things tend to go 
 off the rails.
 The intent of this RowKey utility class isn't meant to add functionality into 
 Put/Get/Scan, but rather make it simpler for folks to construct said arrays.  
 Example:
 {code}
RowKey key = RowKey.create(RowKey.SIZEOF_MD5_HASH + RowKey.SIZEOF_LONG);
key.addHash(a);
key.add(b);
byte bytes[] = key.getBytes();
 {code} 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8240) CompoundConfiguration should implement Iterable

2013-04-03 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-8240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621295#comment-13621295
 ] 

Hadoop QA commented on HBASE-8240:
--

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12576819/8240-v6.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified tests.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5113//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5113//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5113//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5113//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5113//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5113//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5113//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5113//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5113//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5113//console

This message is automatically generated.

 CompoundConfiguration should implement Iterable
 ---

 Key: HBASE-8240
 URL: https://issues.apache.org/jira/browse/HBASE-8240
 Project: HBase
  Issue Type: Improvement
Reporter: Ted Yu
Assignee: Ted Yu
 Attachments: 8240-v1.txt, 8240-v2.txt, 8240-v3.txt, 8240-v4.txt, 
 8240-v6.txt


 Here is from hadoop Configuration class:
 {code}
 public class Configuration implements IterableMap.EntryString,String,
 {code}
 There're 3 addXX() methods for CompoundConfiguration:
 {code}
   public CompoundConfiguration add(final Configuration conf) {
   public CompoundConfiguration addWritableMap(
   final MapImmutableBytesWritable, ImmutableBytesWritable map) {
   public CompoundConfiguration addStringMap(final MapString, String map) {
 {code}
 Parameters to these methods all support iteration.
 We can enhance ImmutableConfigMap with the following new method:
 {code}
   public abstract java.util.Iterator iterator();
 {code}
 Then the following method of CompoundConfiguration can be implemented:
 {code}
   public IteratorMap.EntryString, String iterator() {
 {code}
 This enhancement would be useful in scenario where a mutable Configuration is 
 required.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8240) CompoundConfiguration should implement Iterable

2013-04-03 Thread Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-8240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-8240:
--

Fix Version/s: 0.98.0
   0.95.1

 CompoundConfiguration should implement Iterable
 ---

 Key: HBASE-8240
 URL: https://issues.apache.org/jira/browse/HBASE-8240
 Project: HBase
  Issue Type: Improvement
Reporter: Ted Yu
Assignee: Ted Yu
 Fix For: 0.95.1, 0.98.0

 Attachments: 8240-v1.txt, 8240-v2.txt, 8240-v3.txt, 8240-v4.txt, 
 8240-v6.txt


 Here is from hadoop Configuration class:
 {code}
 public class Configuration implements IterableMap.EntryString,String,
 {code}
 There're 3 addXX() methods for CompoundConfiguration:
 {code}
   public CompoundConfiguration add(final Configuration conf) {
   public CompoundConfiguration addWritableMap(
   final MapImmutableBytesWritable, ImmutableBytesWritable map) {
   public CompoundConfiguration addStringMap(final MapString, String map) {
 {code}
 Parameters to these methods all support iteration.
 We can enhance ImmutableConfigMap with the following new method:
 {code}
   public abstract java.util.Iterator iterator();
 {code}
 Then the following method of CompoundConfiguration can be implemented:
 {code}
   public IteratorMap.EntryString, String iterator() {
 {code}
 This enhancement would be useful in scenario where a mutable Configuration is 
 required.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7745) Import uses System.out.println instead of logging

2013-04-03 Thread Gustavo Anatoly (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621298#comment-13621298
 ] 

Gustavo Anatoly commented on HBASE-7745:


It's good to contribute a little bit, with hbase.

Thanks Nick, Thanks Stack.

 Import uses System.out.println instead of logging
 -

 Key: HBASE-7745
 URL: https://issues.apache.org/jira/browse/HBASE-7745
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Reporter: Nick Dimiduk
Assignee: Gustavo Anatoly
Priority: Trivial
 Fix For: 0.95.1, 0.98.0

 Attachments: HBASE-7745.patch


 Per the title.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7659) add an option for timeout, rather than retry limit, in HCM

2013-04-03 Thread Nicolas Liochon (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-7659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621307#comment-13621307
 ] 

Nicolas Liochon commented on HBASE-7659:


Each time I look at this JIRA, I think it's a good idea, but I can't see all 
the implications.

So here are some randoms thought:
1) I think we can have two settings: a retry.count.min and a timeout. This way:
 - people who wants the system to behave as it used to can set the retry min to 
the current retry limit and the timeout to +inf 
 - people who fancies new stuff can use the timeout
 - the recommended setting would be at least 2 or 3 retries

2) Such scenarios should still be supported imho:
 - client calls server S1. S1 is dead, so client waits until the socket timeout 
is raised (1 minute by default, sometimes more)
 - actually, while the client was waiting, the regions on S1 were moved to S2.
 - client goes to .meta., .meta. says go to S2. Client is happy (while a single 
1 minute timeout would have made it failed as it would not have done the second 
try)
 
My fear here is the subcases of this one, when there is a major crash: may be 
.meta. moved, so the second try may fail while the third would do it.

3) If we use a Future approach on top of the existing code, we may may leave 
running attempts in the system (i.e. we're still trying and holding resources, 
but nobody cares of the final result).

4) When the system go crazy, it takes ~15 minutes to recover (it could be even 
more). The client code may not like it, but there is nothing much we can do. 
Typically, I think we could have two settings:
 - insert it whatever the time it takes (critical data)
 - insert it if possible (lower value data like 'i like' increment).
For this kind of functional segregation, the retry count could be enough: with 
20 retries for the first category and 3 retries for the second one?

 add an option for timeout, rather than retry limit, in HCM
 --

 Key: HBASE-7659
 URL: https://issues.apache.org/jira/browse/HBASE-7659
 Project: HBase
  Issue Type: Bug
  Components: Client
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin

 Retry count limit is not the most useful user-facing measure for failing the 
 request, especially multi-requests that currently fail if any one sub-action 
 reaches the retry count. 
 Given the current deterministic implementation of retry count limit and 
 deterministic (+-jitter) sleep time between retries, the user is already 
 giving us an upper time bound for an operation to expire (with default 10 
 retries, around a minute).
 We can make this explicit.
 That will also make making retries smarter (e.g. retrying faster on certain 
 errors) more easy.
 In future these things can be set per request, which can be usable for people 
 using HBase directly from their code.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8165) Update our protobuf to 2.5 from 2.4.1

2013-04-03 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621317#comment-13621317
 ] 

Hadoop QA commented on HBASE-8165:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12576825/8165v2.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 8 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces lines longer than 
100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   org.apache.hadoop.hbase.rest.client.TestRemoteTable
  org.apache.hadoop.hbase.rest.TestVersionResource
  org.apache.hadoop.hbase.ipc.TestDelayedRpc
  org.apache.hadoop.hbase.rest.client.TestRemoteAdmin
  org.apache.hadoop.hbase.rest.TestRowResource
  org.apache.hadoop.hbase.coprocessor.TestRowProcessorEndpoint
  org.apache.hadoop.hbase.rest.TestSchemaResource
  org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol
  org.apache.hadoop.hbase.rest.TestScannerResource
  org.apache.hadoop.hbase.rest.TestTableResource
  org.apache.hadoop.hbase.rest.TestStatusResource
  org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint
  org.apache.hadoop.hbase.ipc.TestProtoBufRpc

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5114//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5114//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5114//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5114//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5114//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5114//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5114//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5114//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5114//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/5114//console

This message is automatically generated.

 Update our protobuf to 2.5 from 2.4.1
 -

 Key: HBASE-8165
 URL: https://issues.apache.org/jira/browse/HBASE-8165
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
 Fix For: 0.95.0

 Attachments: 8165.txt, 8165v2.txt


 Update to new 2.5 pb.  Some speedups and a new PARSER idiom that bypasses 
 making a builder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8256) Add category Flaky for tests which are flaky

2013-04-03 Thread Jimmy Xiang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-8256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621329#comment-13621329
 ] 

Jimmy Xiang commented on HBASE-8256:


I tried to use excludedGroups to exclude the Flaky methods.  However, it 
excludes the whole test class, not just those flaky methods.

Per https://github.com/junit-team/junit/wiki/Categories, we need to use test 
suite?

 Add category Flaky for tests which are flaky
 

 Key: HBASE-8256
 URL: https://issues.apache.org/jira/browse/HBASE-8256
 Project: HBase
  Issue Type: Bug
  Components: build, test
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang

 To make the Jenkin build more useful, it is good to keep it blue/green. We 
 can mark those flaky tests flaky, and don't run them by default.  However, 
 people can still run them.  We can also set up a Jekin build just for those 
 flaky tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

  1   2   3   4   >