[jira] [Created] (HBASE-8064) hbase connection could not reuse
Yuan Kang created HBASE-8064: Summary: hbase connection could not reuse Key: HBASE-8064 URL: https://issues.apache.org/jira/browse/HBASE-8064 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.94.0 Environment: hadoop-1.0.2 hbase-0.94.0 Reporter: Yuan Kang Assignee: Yuan Kang when hconnection is used by one matchine,the connection return to the pool. if anather matchine reget the connection,it can be resued. but in the code 0.94.0 the caching map don't be managered correctly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8064) hbase connection could not reuse
[ https://issues.apache.org/jira/browse/HBASE-8064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Kang updated HBASE-8064: - Labels: patch (was: ) Release Note: hbase connection manager can't resuse the connection for this code,the patch resolve it Status: Patch Available (was: Open) hbase connection could not reuse Key: HBASE-8064 URL: https://issues.apache.org/jira/browse/HBASE-8064 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.94.0 Environment: hadoop-1.0.2 hbase-0.94.0 Reporter: Yuan Kang Assignee: Yuan Kang Labels: patch when hconnection is used by one matchine,the connection return to the pool. if anather matchine reget the connection,it can be resued. but in the code 0.94.0 the caching map don't be managered correctly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8064) hbase connection could not reuse
[ https://issues.apache.org/jira/browse/HBASE-8064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Kang updated HBASE-8064: - Description: when hconnection is used by one matchine,the connection return to the pool. if anather matchine reget the connection,it can be resued. but in the code the caching map don't be managered correctly was: when hconnection is used by one matchine,the connection return to the pool. if anather matchine reget the connection,it can be resued. but in the code 0.94.0 the caching map don't be managered correctly hbase connection could not reuse Key: HBASE-8064 URL: https://issues.apache.org/jira/browse/HBASE-8064 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.94.0 Environment: hadoop-1.0.2 hbase-0.94.0 Reporter: Yuan Kang Assignee: Yuan Kang Labels: patch Attachments: HConnectionManager-connection-could-not-reuse.patch when hconnection is used by one matchine,the connection return to the pool. if anather matchine reget the connection,it can be resued. but in the code the caching map don't be managered correctly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8064) hbase connection could not reuse
[ https://issues.apache.org/jira/browse/HBASE-8064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Kang updated HBASE-8064: - Attachment: HConnectionManager-connection-could-not-reuse.patch hbase connection could not reuse Key: HBASE-8064 URL: https://issues.apache.org/jira/browse/HBASE-8064 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.94.0 Environment: hadoop-1.0.2 hbase-0.94.0 Reporter: Yuan Kang Assignee: Yuan Kang Labels: patch Attachments: HConnectionManager-connection-could-not-reuse.patch when hconnection is used by one matchine,the connection return to the pool. if anather matchine reget the connection,it can be resued. but in the code the caching map don't be managered correctly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8064) hbase connection could not reuse
[ https://issues.apache.org/jira/browse/HBASE-8064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598578#comment-13598578 ] Anoop Sam John commented on HBASE-8064: --- Have you attached any patch? I can not see any but you made status as Patch Available hbase connection could not reuse Key: HBASE-8064 URL: https://issues.apache.org/jira/browse/HBASE-8064 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.94.0 Environment: hadoop-1.0.2 hbase-0.94.0 Reporter: Yuan Kang Assignee: Yuan Kang Labels: patch Attachments: HConnectionManager-connection-could-not-reuse.patch when hconnection is used by one matchine,the connection return to the pool. if anather matchine reget the connection,it can be resued. but in the code the caching map don't be managered correctly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication
[ https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598579#comment-13598579 ] Jeffrey Zhong commented on HBASE-7709: -- Continue with more proposals... The disadvantages of option#2 is obvious as its advantages. Even in cases(maybe majority replication usage cases), there is no loop at all and just a long replication queue. The downstream RSs still need to replay and store a long list of clusterIds for each WALEdit. Encoding may help compress the clusterId list in sending part but not in storing. Let me firstly try to show you if we can do better than option#2 and then an alternative way which is good in most cases without more storage need. Both options are good IMHO. As we know loop is caused by back-edge in graph. We can roughly identify them by the fact if a region server sees there are more than one path from same source. If that's the case, loop situation is likely. Only by then, we need to append current cluster Id to the source cluster Id of a WAL edit for later loop detection. Therefore, in most cases, we don't need store long clusterId list if there is no loop or a simple master-master-master… cycle setup. I called the above updated option#2 as adaptive option#2 where it only need more storage when there is a need. We can implement it as following: 1) Maintain a hash string PathCheckum(= Hash(receivedPathChecksum + current clusterId)) of a WAL edit 2) Each replaying receiving region server maintains an internal memory ClusterDistanceMap clusterId, SetPathChecksums seen so far. 2.a Every time if it sees a new PathChecksum(which isn't in SetPathChecksums ), it add the new PathChecksum into SetPathChecksums or drop a stale one from SetPathChecksums when it's expired, i.e. after a configurable time period, a region server doesn't see any data coming in from the path. 3) When SetPathChecksums's size 1, append current cluster id into the WAL edit for later replication loop detection. We can use top 8 bytes of clusterId to store PathChecksum and the rest 8 bytes as the hash of the original cluserId value. After the update, we only need to pay cost when there is a need. While you can image in real life replication setup normally doesn't involve any complicated graph, the option#2 is using extra storage need to deal with situations most likely won't happen. Therefore, in the following, I want to propose a solution without changing current WAL format and is good for most cases including the situation triggering the JIRA. In extreme cases, it reports errors for infinite loop. The new proposal(option #6) is as following: 1) Maintain a hash string PathCheckum(= Hash(receivedPathChecksum + current clusterId)) of a WAL edit 2) Each replaying receiving region server maintains an internal memory ClusterDistanceMap clusterId, SetPathChecksums seen so far. 2.a Every time if it sees a new PathChecksum(which isn't in SetPathChecksums ), it add the new PathChecksum into SetPathChecksums or drop a stale one from SetPathChecksums when it's expired, i.e. after a configurable time period, a region server doesn't any data coming in from the path. 3) When SetPathChecksums's size 1, reset a WAL edit's clusterId to current clusterId and increment a counter(ResetCounter) to mark how many times current WAL edit's clusterId has been reset. 4) When ResetCounter 64, reports error( we could drop WAL edits as well because when ResetCounter 64, it means we have at least 64 back-edges or duplicated sources. I think it's way complicated to have such cases.) The advantage of the above option is possibly using existing HLog format to prevent possible loop situation in real life cases To implement, 1) we can introduce a new version(3) in HLogKey 2) use top 7 bytes of UUID to store PathChecksum, use the following 1 byte to store RD and the remaining 8 bytes as a hash value of the 16 bytes length of origin UUID value without compromising uniqueness because in most cases we have 10s clusters involved in replication and the collision probability is less than 10(-18) 3) we can introduce a configuration setting with default to false(suggested by Lars). After we rollout the feature, we can turn it on and turn if off in revert scenario. Thanks, -Jeffrey Infinite loop possible in Master/Master replication --- Key: HBASE-7709 URL: https://issues.apache.org/jira/browse/HBASE-7709 Project: HBase Issue Type: Bug Components: Replication Affects Versions: 0.95.0, 0.94.6 Reporter: Lars Hofhansl Fix For: 0.95.0, 0.94.7 We just discovered the following scenario: # Cluster A and B are setup in master/master replication # By accident we had Cluster C replicate to Cluster A. Now all edit
[jira] [Commented] (HBASE-8064) hbase connection could not reuse
[ https://issues.apache.org/jira/browse/HBASE-8064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598581#comment-13598581 ] Anoop Sam John commented on HBASE-8064: --- Ya it is there now :) hbase connection could not reuse Key: HBASE-8064 URL: https://issues.apache.org/jira/browse/HBASE-8064 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.94.0 Environment: hadoop-1.0.2 hbase-0.94.0 Reporter: Yuan Kang Assignee: Yuan Kang Labels: patch Attachments: HConnectionManager-connection-could-not-reuse.patch when hconnection is used by one matchine,the connection return to the pool. if anather matchine reget the connection,it can be resued. but in the code the caching map don't be managered correctly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7992) provide pre/post region offline hooks for HMaster.offlineRegion().
[ https://issues.apache.org/jira/browse/HBASE-7992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598582#comment-13598582 ] Anoop Sam John commented on HBASE-7992: --- Reviewing this. You can avoid postRegionOffline from AccessController. AccessController extending BaseRegionObserver Let me review more. provide pre/post region offline hooks for HMaster.offlineRegion(). --- Key: HBASE-7992 URL: https://issues.apache.org/jira/browse/HBASE-7992 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 0.95.0 Reporter: rajeshbabu Assignee: rajeshbabu Fix For: 0.98.0 Attachments: HBASE-7992_trunk.patch presently no hooks to provide access control to offline region in master. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8065) the bulkload can load the hfile into hbase table,but this mechanism can't remove the date before
Yuan Kang created HBASE-8065: Summary: the bulkload can load the hfile into hbase table,but this mechanism can't remove the date before Key: HBASE-8065 URL: https://issues.apache.org/jira/browse/HBASE-8065 Project: HBase Issue Type: Improvement Components: IPC/RPC, mapreduce, regionserver Affects Versions: 0.94.0 Environment: hadoop-1.0.2、hbase-0.94.0 Reporter: Yuan Kang Assignee: Yuan Kang Priority: Critical this patch can do bulkload for one more parameter ‘need to refresh’,when this parameter is true.bulkload can clean the old date in the hbase table ,then do the new date load -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8065) the bulkload can load the hfile into hbase table,but this mechanism can't remove the date before
[ https://issues.apache.org/jira/browse/HBASE-8065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Kang updated HBASE-8065: - Attachment: LoadIncrementalHFiles-bulkload-can-clean-olddata.patch this patch change the regionserver/ipc/mapreduce code to enable the bulkload remove the old date the bulkload can load the hfile into hbase table,but this mechanism can't remove the date before Key: HBASE-8065 URL: https://issues.apache.org/jira/browse/HBASE-8065 Project: HBase Issue Type: Improvement Components: IPC/RPC, mapreduce, regionserver Affects Versions: 0.94.0 Environment: hadoop-1.0.2、hbase-0.94.0 Reporter: Yuan Kang Assignee: Yuan Kang Priority: Critical Attachments: LoadIncrementalHFiles-bulkload-can-clean-olddata.patch this patch can do bulkload for one more parameter ‘need to refresh’,when this parameter is true.bulkload can clean the old date in the hbase table ,then do the new date load -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8064) hbase connection could not reuse
[ https://issues.apache.org/jira/browse/HBASE-8064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598585#comment-13598585 ] Hadoop QA commented on HBASE-8064: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12573031/HConnectionManager-connection-could-not-reuse.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4752//console This message is automatically generated. hbase connection could not reuse Key: HBASE-8064 URL: https://issues.apache.org/jira/browse/HBASE-8064 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.94.0 Environment: hadoop-1.0.2 hbase-0.94.0 Reporter: Yuan Kang Assignee: Yuan Kang Labels: patch Attachments: HConnectionManager-connection-could-not-reuse.patch when hconnection is used by one matchine,the connection return to the pool. if anather matchine reget the connection,it can be resued. but in the code the caching map don't be managered correctly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8065) the bulkload can load the hfile into hbase table,but this mechanism can't remove the date before
[ https://issues.apache.org/jira/browse/HBASE-8065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Kang updated HBASE-8065: - Status: Patch Available (was: Open) the bulkload can load the hfile into hbase table,but this mechanism can't remove the date before Key: HBASE-8065 URL: https://issues.apache.org/jira/browse/HBASE-8065 Project: HBase Issue Type: Improvement Components: IPC/RPC, mapreduce, regionserver Affects Versions: 0.94.0 Environment: hadoop-1.0.2、hbase-0.94.0 Reporter: Yuan Kang Assignee: Yuan Kang Priority: Critical Attachments: LoadIncrementalHFiles-bulkload-can-clean-olddata.patch this patch can do bulkload for one more parameter ‘need to refresh’,when this parameter is true.bulkload can clean the old date in the hbase table ,then do the new date load -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8064) hbase connection could not reuse
[ https://issues.apache.org/jira/browse/HBASE-8064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598586#comment-13598586 ] ramkrishna.s.vasudevan commented on HBASE-8064: --- Am not sure if this is a bug. The rsname and the {code} Addressing.createHostAndPortStr( address.getHostName(), address.getPort() {code} resolve to the same String right? So the lookup from map should be ok? hbase connection could not reuse Key: HBASE-8064 URL: https://issues.apache.org/jira/browse/HBASE-8064 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.94.0 Environment: hadoop-1.0.2 hbase-0.94.0 Reporter: Yuan Kang Assignee: Yuan Kang Labels: patch Attachments: HConnectionManager-connection-could-not-reuse.patch when hconnection is used by one matchine,the connection return to the pool. if anather matchine reget the connection,it can be resued. but in the code the caching map don't be managered correctly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7981) TestSplitTransactionOnCluster.testShutdownFixupWhenDaughterHasSplit failed in 0.95 build #11
[ https://issues.apache.org/jira/browse/HBASE-7981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598587#comment-13598587 ] ramkrishna.s.vasudevan commented on HBASE-7981: --- Tried to checkout the KeyComparator but it ends up trickier because we are not very sure if explicitly user has tried to set up the startkey and endkey. Will check on this more. TestSplitTransactionOnCluster.testShutdownFixupWhenDaughterHasSplit failed in 0.95 build #11 Key: HBASE-7981 URL: https://issues.apache.org/jira/browse/HBASE-7981 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Priority: Blocker Fix For: 0.95.0 Attachments: 7981.extra.logging.txt, 7981.extra.logging.txt, 7981v2.extra.logging.txt, HBASE-7981_1.patch, HBASE-7981.patch https://builds.apache.org/job/hbase-0.95/11/testReport/junit/org.apache.hadoop.hbase.regionserver/TestSplitTransactionOnCluster/testShutdownFixupWhenDaughterHasSplit/ Hard to tell which region is missing post crash. Not logged. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8064) hbase connection could not reuse
[ https://issues.apache.org/jira/browse/HBASE-8064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598588#comment-13598588 ] Anoop Sam John commented on HBASE-8064: --- Ram I was also wondering and was looking more into Trunk code also. @Yuan can you pls explain hbase connection could not reuse Key: HBASE-8064 URL: https://issues.apache.org/jira/browse/HBASE-8064 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.94.0 Environment: hadoop-1.0.2 hbase-0.94.0 Reporter: Yuan Kang Assignee: Yuan Kang Labels: patch Attachments: HConnectionManager-connection-could-not-reuse.patch when hconnection is used by one matchine,the connection return to the pool. if anather matchine reget the connection,it can be resued. but in the code the caching map don't be managered correctly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8064) hbase connection could not reuse
[ https://issues.apache.org/jira/browse/HBASE-8064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598589#comment-13598589 ] Anoop Sam John commented on HBASE-8064: --- And also see {code} InetSocketAddress address = isa != null? isa: new InetSocketAddress(hostname, port); {code} address creation is also in the same way as rsname hbase connection could not reuse Key: HBASE-8064 URL: https://issues.apache.org/jira/browse/HBASE-8064 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.94.0 Environment: hadoop-1.0.2 hbase-0.94.0 Reporter: Yuan Kang Assignee: Yuan Kang Labels: patch Attachments: HConnectionManager-connection-could-not-reuse.patch when hconnection is used by one matchine,the connection return to the pool. if anather matchine reget the connection,it can be resued. but in the code the caching map don't be managered correctly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8065) the bulkload can load the hfile into hbase table,but this mechanism can't remove the date before
[ https://issues.apache.org/jira/browse/HBASE-8065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598592#comment-13598592 ] Hadoop QA commented on HBASE-8065: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12573032/LoadIncrementalHFiles-bulkload-can-clean-olddata.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4753//console This message is automatically generated. the bulkload can load the hfile into hbase table,but this mechanism can't remove the date before Key: HBASE-8065 URL: https://issues.apache.org/jira/browse/HBASE-8065 Project: HBase Issue Type: Improvement Components: IPC/RPC, mapreduce, regionserver Affects Versions: 0.94.0 Environment: hadoop-1.0.2、hbase-0.94.0 Reporter: Yuan Kang Assignee: Yuan Kang Priority: Critical Attachments: LoadIncrementalHFiles-bulkload-can-clean-olddata.patch this patch can do bulkload for one more parameter ‘need to refresh’,when this parameter is true.bulkload can clean the old date in the hbase table ,then do the new date load -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7928) Scanning .META. with startRow and/or stopRow is not giving proper results
[ https://issues.apache.org/jira/browse/HBASE-7928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598591#comment-13598591 ] ramkrishna.s.vasudevan commented on HBASE-7928: --- Tried to checkout the KeyComparator but it ends up trickier because we are not very sure if explicitly user has tried to set up the startkey and endkey. Will check on this more. Scanning .META. with startRow and/or stopRow is not giving proper results - Key: HBASE-7928 URL: https://issues.apache.org/jira/browse/HBASE-7928 Project: HBase Issue Type: Bug Components: Usability Affects Versions: 0.94.5 Reporter: Jean-Marc Spaggiari Assignee: ramkrishna.s.vasudevan Attachments: HBASE-7928_0.94_1.patch, HBASE-7928_0.94_2.patch, HBASE-7928_0.94_3.patch, HBASE-7928_0.94_3.patch, HBASE-7928_0.94.patch, HBASE-7928_trunk_1.patch, HBASE-7928_trunk_2.patch, HBASE-7928_trunk_2.patch, HBASE-7928_trunk.patch {code} try { HTable metaTable = new HTable(config, Bytes.toBytes(.META.)); Scan scan = new Scan(); scan.setStartRow(Bytes.toBytes(e)); scan.setStopRow(Bytes.toBytes(z)); ResultScanner scanner = metaTable.getScanner(scan); Result[] results = scanner.next(100); while (results.length 0) { for (Result result : results) { System.out.println(Bytes.toString(result.getRow())); } results = scanner.next(100); } scanner.close(); metaTable.close(); } catch (Exception e) { e.printStackTrace(); } {code} This code will not return any result even if there is 10 tables with names starting with d to w, including one table called entry. If you comment the setStopRow you will get results, but will still get rows starting with d even if setStartRow is set to e. Same code using with a user table is working fine. Facing the same issue with the shell. scan '.META.' , {STARTROW = 'e', LIMIT = 10} is returning rows starting by d. scan '.META.' , {STARTROW = 'e', STOPROW = 'v', LIMIT = 10} is not returning anything. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8064) hbase connection could not reuse
[ https://issues.apache.org/jira/browse/HBASE-8064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598595#comment-13598595 ] Anoop Sam John commented on HBASE-8064: --- Will test with some test cases. hbase connection could not reuse Key: HBASE-8064 URL: https://issues.apache.org/jira/browse/HBASE-8064 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.94.0 Environment: hadoop-1.0.2 hbase-0.94.0 Reporter: Yuan Kang Assignee: Yuan Kang Labels: patch Attachments: HConnectionManager-connection-could-not-reuse.patch when hconnection is used by one matchine,the connection return to the pool. if anather matchine reget the connection,it can be resued. but in the code the caching map don't be managered correctly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8063) Filter HFiles based on first/last key
[ https://issues.apache.org/jira/browse/HBASE-8063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598599#comment-13598599 ] chunhui shen commented on HBASE-8063: - bq.could not work on non-get scan + bloomtype != rowcol you are right, so we can filter the hfile if its first key = scan's stop key or its lastkey = scan's start key Filter HFiles based on first/last key - Key: HBASE-8063 URL: https://issues.apache.org/jira/browse/HBASE-8063 Project: HBase Issue Type: Improvement Components: Scanners Affects Versions: 0.98.0 Reporter: Liang Xie Assignee: Liang Xie # hbase-5010 introduced a filter mechanism based on timerange hint # this jira is a placeholder just for keyrange hint: currently, low layer(hfile reader) has first/last key getter methods already. we should utilize this information to pre-filte hfiles while choosing scanners from storefiles. It's just a raw thought right now, no patch available yet. I'll summit a patch asap if it's reasonable, welcome to give some suggestions/comments:) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8063) Filter HFiles based on first/last key
[ https://issues.apache.org/jira/browse/HBASE-8063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598602#comment-13598602 ] Liang Xie commented on HBASE-8063: -- thanks for confirming:) Filter HFiles based on first/last key - Key: HBASE-8063 URL: https://issues.apache.org/jira/browse/HBASE-8063 Project: HBase Issue Type: Improvement Components: Scanners Affects Versions: 0.98.0 Reporter: Liang Xie Assignee: Liang Xie # hbase-5010 introduced a filter mechanism based on timerange hint # this jira is a placeholder just for keyrange hint: currently, low layer(hfile reader) has first/last key getter methods already. we should utilize this information to pre-filte hfiles while choosing scanners from storefiles. It's just a raw thought right now, no patch available yet. I'll summit a patch asap if it's reasonable, welcome to give some suggestions/comments:) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8062) Replace HBaseFsck.debugLsr() in TestFlushSnapshotFromClient with FSUtils.logFileSystemState()
[ https://issues.apache.org/jira/browse/HBASE-8062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598619#comment-13598619 ] Hadoop QA commented on HBASE-8062: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572973/HBASE-8062-v0.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified tests. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:red}-1 site{color}. The patch appears to cause mvn site goal to fail. {color:red}-1 core tests{color}. The patch failed these unit tests: {color:red}-1 core zombie tests{color}. There are 1 zombie test(s): at org.apache.hadoop.hbase.util.TestHBaseFsck.testFixByTable(TestHBaseFsck.java:1170) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4751//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4751//console This message is automatically generated. Replace HBaseFsck.debugLsr() in TestFlushSnapshotFromClient with FSUtils.logFileSystemState() - Key: HBASE-8062 URL: https://issues.apache.org/jira/browse/HBASE-8062 Project: HBase Issue Type: Bug Components: snapshots, test Affects Versions: 0.95.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Trivial Fix For: 0.95.0 Attachments: HBASE-8062-v0.patch Replace the HBaseFsck.debugLsr() calls in TestFlushSnapshotFromClient with FSUtils.logFileSystemState(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7949) Enable big content store in HBase
[ https://issues.apache.org/jira/browse/HBASE-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598642#comment-13598642 ] Maryann Xue commented on HBASE-7949: @chenning, as enis has clarified, the actual data move does not happen on the split point. instead, it happens in later compactions. and in the approach we proposed, the LOB family does not participate in split or minor compactions at all. @enis, the problem is not when the read and write happens, it is more of the unnecessary I/O overhead in splitting. and if the data is seldom updated, why compact them (for split) anyway? yes, utilizing level compactions could be a good approach. still, our approach can have three advantages over level compaction: 1. i/o overhead by split and minor compactions are completely eliminated; 2. clean-up is only done for those file that has reached a certain level of invalidation rate, during major compactions; 3. not every file reader is instantiated and kept in regionserver memory. instead, we'll have an LRU cache for frequently read LOB files. however, i suggest this issue not be committed into HBase trunk. instead we'd like to make the implementation a use case over HBase. and the only facility we need in HBase trunk is a pluggable flush process HBASE-8024. Enable big content store in HBase - Key: HBASE-7949 URL: https://issues.apache.org/jira/browse/HBASE-7949 Project: HBase Issue Type: Brainstorming Reporter: chenning Attachments: HBase_LOB.pdf Big content stored in hbase consumes a lot of system resource when region split or compaction operation happens. How HBase can be used to store big content along with some self descriptive meta-data. The general idea is to add a new type of column family, and the content of this kind of column family doesn't participate the region split and compaction. An index(rowkey-location) is introduced in this new column family and the split and compaction are only applied to this index. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8039) Make HDFS replication number configurable for a column family
[ https://issues.apache.org/jira/browse/HBASE-8039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598644#comment-13598644 ] Maryann Xue commented on HBASE-8039: Yes, Sergey, that would be a necessary part of the solution. but meanwhile the other part is to pass down the replication number into the HFile writer. Make HDFS replication number configurable for a column family - Key: HBASE-8039 URL: https://issues.apache.org/jira/browse/HBASE-8039 Project: HBase Issue Type: Improvement Components: HFile Affects Versions: 0.94.5 Reporter: Maryann Xue Priority: Minor To allow users to decide which column family's data is more important and which is less important by specifying a replica number instead of using the default replica number. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8024) Make Store flush algorithm pluggable
[ https://issues.apache.org/jira/browse/HBASE-8024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598651#comment-13598651 ] Andrew Purtell commented on HBASE-8024: --- Do you think we can get this into trunk/0.95 in time for 0.96 [~maryannxue]? Make Store flush algorithm pluggable Key: HBASE-8024 URL: https://issues.apache.org/jira/browse/HBASE-8024 Project: HBase Issue Type: Sub-task Components: regionserver Affects Versions: 0.95.0, 0.96.0, 0.94.5 Reporter: Maryann Xue The idea is to make StoreFlusher an interface instead of an implementation class, and have the original StoreFlusher as the default store flush impl. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8024) Make Store flush algorithm pluggable
[ https://issues.apache.org/jira/browse/HBASE-8024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-8024: -- Affects Version/s: 0.96.0 0.95.0 Make Store flush algorithm pluggable Key: HBASE-8024 URL: https://issues.apache.org/jira/browse/HBASE-8024 Project: HBase Issue Type: Sub-task Components: regionserver Affects Versions: 0.95.0, 0.96.0, 0.94.5 Reporter: Maryann Xue The idea is to make StoreFlusher an interface instead of an implementation class, and have the original StoreFlusher as the default store flush impl. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7949) Enable big content store in HBase
[ https://issues.apache.org/jira/browse/HBASE-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598652#comment-13598652 ] Andrew Purtell commented on HBASE-7949: --- bq. i suggest this issue not be committed into HBase trunk. instead we'd like to make the implementation a use case over HBase. and the only facility we need in HBase trunk is a pluggable flush process HBASE-8024 +1, agree with Maryann, let's consider making HBASE-8024 an Improvement issue and close this one, it lacks focus. Enable big content store in HBase - Key: HBASE-7949 URL: https://issues.apache.org/jira/browse/HBASE-7949 Project: HBase Issue Type: Brainstorming Reporter: chenning Attachments: HBase_LOB.pdf Big content stored in hbase consumes a lot of system resource when region split or compaction operation happens. How HBase can be used to store big content along with some self descriptive meta-data. The general idea is to add a new type of column family, and the content of this kind of column family doesn't participate the region split and compaction. An index(rowkey-location) is introduced in this new column family and the split and compaction are only applied to this index. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8066) Provide Admin.isTableAvailable() for a given table along with splitkeys
[ https://issues.apache.org/jira/browse/HBASE-8066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-8066: -- Status: Patch Available (was: Open) Provide Admin.isTableAvailable() for a given table along with splitkeys --- Key: HBASE-8066 URL: https://issues.apache.org/jira/browse/HBASE-8066 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.95.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Minor Fix For: 0.95.0, 0.98.0 Attachments: HBASE-8066.patch As part of HBASE-5583 if the master reboots during creation of table there is a chance that the table gets created with partial split keys. This api helps to check if the table was created with the required number of splitkeys. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8066) Provide Admin.isTableAvailable() for a given table along with splitkeys
[ https://issues.apache.org/jira/browse/HBASE-8066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-8066: -- Attachment: HBASE-8066.patch Provide Admin.isTableAvailable() for a given table along with splitkeys --- Key: HBASE-8066 URL: https://issues.apache.org/jira/browse/HBASE-8066 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.95.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Minor Fix For: 0.95.0, 0.98.0 Attachments: HBASE-8066.patch As part of HBASE-5583 if the master reboots during creation of table there is a chance that the table gets created with partial split keys. This api helps to check if the table was created with the required number of splitkeys. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8066) Provide Admin.isTableAvailable() for a given table along with splitkeys
ramkrishna.s.vasudevan created HBASE-8066: - Summary: Provide Admin.isTableAvailable() for a given table along with splitkeys Key: HBASE-8066 URL: https://issues.apache.org/jira/browse/HBASE-8066 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.95.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Minor Fix For: 0.95.0, 0.98.0 Attachments: HBASE-8066.patch As part of HBASE-5583 if the master reboots during creation of table there is a chance that the table gets created with partial split keys. This api helps to check if the table was created with the required number of splitkeys. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8066) Provide Admin.isTableAvailable() for a given table along with splitkeys
[ https://issues.apache.org/jira/browse/HBASE-8066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598683#comment-13598683 ] Hadoop QA commented on HBASE-8066: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12573055/HBASE-8066.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified tests. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 4 warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:red}-1 site{color}. The patch appears to cause mvn site goal to fail. {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.procedure.TestZKProcedureControllers Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4754//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4754//console This message is automatically generated. Provide Admin.isTableAvailable() for a given table along with splitkeys --- Key: HBASE-8066 URL: https://issues.apache.org/jira/browse/HBASE-8066 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.95.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Minor Fix For: 0.95.0, 0.98.0 Attachments: HBASE-8066.patch As part of HBASE-5583 if the master reboots during creation of table there is a chance that the table gets created with partial split keys. This api helps to check if the table was created with the required number of splitkeys. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8067) TestHFileArchiving.testArchiveOnTableDelete sometimes fails
Matteo Bertozzi created HBASE-8067: -- Summary: TestHFileArchiving.testArchiveOnTableDelete sometimes fails Key: HBASE-8067 URL: https://issues.apache.org/jira/browse/HBASE-8067 Project: HBase Issue Type: Bug Components: Admin, master, test Affects Versions: 0.96.0, 0.94.6 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi it seems that testArchiveOnTableDelete() fails because the archiving in DeleteTableHandler is still in progress when admin.deleteTable() returns. {code} Error Message Archived files are missing some of the store files! Stacktrace java.lang.AssertionError: Archived files are missing some of the store files! at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.apache.hadoop.hbase.backup.TestHFileArchiving.testArchiveOnTableDelete(TestHFileArchiving.java:262) {code} (Looking at the problem in a more generic way, we don't have any way to inform the client when an async operation is completed) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8067) TestHFileArchiving.testArchiveOnTableDelete sometimes fails
[ https://issues.apache.org/jira/browse/HBASE-8067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-8067: --- Attachment: HBASE-8067-debug.patch can I merge HBASE-8067-debug.patch to have more debug logs? TestHFileArchiving.testArchiveOnTableDelete sometimes fails --- Key: HBASE-8067 URL: https://issues.apache.org/jira/browse/HBASE-8067 Project: HBase Issue Type: Bug Components: Admin, master, test Affects Versions: 0.96.0, 0.94.6 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Attachments: HBASE-8067-debug.patch it seems that testArchiveOnTableDelete() fails because the archiving in DeleteTableHandler is still in progress when admin.deleteTable() returns. {code} Error Message Archived files are missing some of the store files! Stacktrace java.lang.AssertionError: Archived files are missing some of the store files! at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.apache.hadoop.hbase.backup.TestHFileArchiving.testArchiveOnTableDelete(TestHFileArchiving.java:262) {code} (Looking at the problem in a more generic way, we don't have any way to inform the client when an async operation is completed) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-7713) Maven build fails for hbase-common on windows environment
[ https://issues.apache.org/jira/browse/HBASE-7713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal resolved HBASE-7713. Resolution: Cannot Reproduce Maven build fails for hbase-common on windows environment - Key: HBASE-7713 URL: https://issues.apache.org/jira/browse/HBASE-7713 Project: HBase Issue Type: Bug Environment: Windows Environment Reporter: Raghu Doppalapudi Priority: Minor build fails with following error message org.codehaus.plexus.resource.loader.ResourceNotFoundException: Could not find resource 'dev-support/findbugs-exclude.xml' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7713) Maven build fails for hbase-common on windows environment
[ https://issues.apache.org/jira/browse/HBASE-7713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598736#comment-13598736 ] nkeywal commented on HBASE-7713: No feedback. Closing, @[~rdoppalapudi], please reopen if you think differently. Maven build fails for hbase-common on windows environment - Key: HBASE-7713 URL: https://issues.apache.org/jira/browse/HBASE-7713 Project: HBase Issue Type: Bug Environment: Windows Environment Reporter: Raghu Doppalapudi Priority: Minor build fails with following error message org.codehaus.plexus.resource.loader.ResourceNotFoundException: Could not find resource 'dev-support/findbugs-exclude.xml' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7713) Maven build fails for hbase-common on windows environment
[ https://issues.apache.org/jira/browse/HBASE-7713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598737#comment-13598737 ] ramkrishna.s.vasudevan commented on HBASE-7713: --- I have faced this once in my old windows machine which i don't have now :(. In the current machine it works fine. Maven build fails for hbase-common on windows environment - Key: HBASE-7713 URL: https://issues.apache.org/jira/browse/HBASE-7713 Project: HBase Issue Type: Bug Environment: Windows Environment Reporter: Raghu Doppalapudi Priority: Minor build fails with following error message org.codehaus.plexus.resource.loader.ResourceNotFoundException: Could not find resource 'dev-support/findbugs-exclude.xml' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7927) Two versions of netty with hadoop.profile=2.0: 3.5.9 and 3.2.4
[ https://issues.apache.org/jira/browse/HBASE-7927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-7927: --- Assignee: nkeywal Two versions of netty with hadoop.profile=2.0: 3.5.9 and 3.2.4 -- Key: HBASE-7927 URL: https://issues.apache.org/jira/browse/HBASE-7927 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal I don't know why, but when you do a mvn dependency:tree, everything looks fine. When you look at the generated target/cached_classpath.txt you see 2 versions of netty: netty-3.2.4.Final.jar and netty-3.5.9.Final.jar. This is bad and can lead to unpredictable behavior. I haven't looked at the other dependencies. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7327) Assignment Timeouts: Remove the code from the master
[ https://issues.apache.org/jira/browse/HBASE-7327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-7327: --- Resolution: Later Status: Resolved (was: Patch Available) Since the final decision was to make this optional, the code is still there, to be removed later. Assignment Timeouts: Remove the code from the master Key: HBASE-7327 URL: https://issues.apache.org/jira/browse/HBASE-7327 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Attachments: 7327.v1.uncomplete.patch, 7327.v2.patch As per HBASE-7247... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7247) Assignment performances decreased by 50% because of regionserver.OpenRegionHandler#tickleOpening
[ https://issues.apache.org/jira/browse/HBASE-7247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598749#comment-13598749 ] nkeywal commented on HBASE-7247: TimeOutManagement it now optional and deactivated by default. I will redo the measures. Assignment performances decreased by 50% because of regionserver.OpenRegionHandler#tickleOpening Key: HBASE-7247 URL: https://issues.apache.org/jira/browse/HBASE-7247 Project: HBase Issue Type: Improvement Components: master, Region Assignment, regionserver Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Fix For: 0.95.0 Attachments: 7247.v1.patch The regionserver.OpenRegionHandler#tickleOpening updates the region znode as Do this so master doesn't timeout this region-in-transition.. However, on the usual test, this makes the assignment time of 1500 regions goes from 70s to 100s, that is, we're 50% slower because of this. More generally, ZooKeper commits to disk all the data update, and this takes time. Using it to provide a keep alive seems overkill. At the very list, it could be made asynchronous. I'm not sure how necessary these updates are required (I need to go deeper in the internal, feedback welcome), but it seems very important to optimize this... The trival fix would be to make this optional. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8061) Missing test from TestFlushSnapshotFromClient in 0.94
[ https://issues.apache.org/jira/browse/HBASE-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598758#comment-13598758 ] Matteo Bertozzi commented on HBASE-8061: [~lhofhansl] opened HBASE-8067 for tracking the TestHFileArchiving failure Missing test from TestFlushSnapshotFromClient in 0.94 - Key: HBASE-8061 URL: https://issues.apache.org/jira/browse/HBASE-8061 Project: HBase Issue Type: Bug Components: snapshots Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.94.7 Attachments: HBASE-8061-v0.patch, HBASE-8061-v1.patch, HBASE-8061-v2.patch, long-lines-trunk.patch There's a missing test from TestFlushSnapshotFromClient (0.94) and a missing delete snapshot at the end of another test that cases jdk7 to fail, since the tests are executed in a different order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8058) Upgrade site plugin; fix assembly doc up on jenkins builds
[ https://issues.apache.org/jira/browse/HBASE-8058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598767#comment-13598767 ] Hudson commented on HBASE-8058: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #440 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/440/]) HBASE-8058 Upgrade site plugin; fix assembly doc up on jenkins builds (Revision 1454991) Result = FAILURE stack : Files : * /hbase/trunk/pom.xml Upgrade site plugin; fix assembly doc up on jenkins builds -- Key: HBASE-8058 URL: https://issues.apache.org/jira/browse/HBASE-8058 Project: HBase Issue Type: Bug Components: site Reporter: stack Assignee: stack Priority: Blocker Fix For: 0.95.0 Attachments: 8058.txt, 8058v2.txt Up on jenkins, we currently make assemblies but there no doc in them. The site goal runs last. You can't run it anywhere else else build fails. Upgrading the site plugin helps. Upgrading site plugin I notice that there are a bunch of extra reports generated that would be no harm showing on the web site; e.g. dependencies transitively included, what dependencies we have, etc. This issue is about upgrading site plugin to fix jenkins assemblies and to expose reports we are generating anyways (at least one report is new w/ the info-report upgrade from earlier today). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8063) Filter HFiles based on first/last key
[ https://issues.apache.org/jira/browse/HBASE-8063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HBASE-8063: - Status: Patch Available (was: Open) Filter HFiles based on first/last key - Key: HBASE-8063 URL: https://issues.apache.org/jira/browse/HBASE-8063 Project: HBase Issue Type: Improvement Components: Scanners Affects Versions: 0.98.0 Reporter: Liang Xie Assignee: Liang Xie Attachments: HBASE-8063.txt # hbase-5010 introduced a filter mechanism based on timerange hint # this jira is a placeholder just for keyrange hint: currently, low layer(hfile reader) has first/last key getter methods already. we should utilize this information to pre-filte hfiles while choosing scanners from storefiles. It's just a raw thought right now, no patch available yet. I'll summit a patch asap if it's reasonable, welcome to give some suggestions/comments:) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8063) Filter HFiles based on first/last key
[ https://issues.apache.org/jira/browse/HBASE-8063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HBASE-8063: - Attachment: HBASE-8063.txt let's see QA robot Filter HFiles based on first/last key - Key: HBASE-8063 URL: https://issues.apache.org/jira/browse/HBASE-8063 Project: HBase Issue Type: Improvement Components: Scanners Affects Versions: 0.98.0 Reporter: Liang Xie Assignee: Liang Xie Attachments: HBASE-8063.txt # hbase-5010 introduced a filter mechanism based on timerange hint # this jira is a placeholder just for keyrange hint: currently, low layer(hfile reader) has first/last key getter methods already. we should utilize this information to pre-filte hfiles while choosing scanners from storefiles. It's just a raw thought right now, no patch available yet. I'll summit a patch asap if it's reasonable, welcome to give some suggestions/comments:) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8063) Filter HFiles based on first/last key
[ https://issues.apache.org/jira/browse/HBASE-8063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598841#comment-13598841 ] Ted Yu commented on HBASE-8063: --- Looks like there is compilation error: https://builds.apache.org/job/PreCommit-HBASE-Build/4755/console Filter HFiles based on first/last key - Key: HBASE-8063 URL: https://issues.apache.org/jira/browse/HBASE-8063 Project: HBase Issue Type: Improvement Components: Scanners Affects Versions: 0.98.0 Reporter: Liang Xie Assignee: Liang Xie Attachments: HBASE-8063.txt # hbase-5010 introduced a filter mechanism based on timerange hint # this jira is a placeholder just for keyrange hint: currently, low layer(hfile reader) has first/last key getter methods already. we should utilize this information to pre-filte hfiles while choosing scanners from storefiles. It's just a raw thought right now, no patch available yet. I'll summit a patch asap if it's reasonable, welcome to give some suggestions/comments:) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8058) Upgrade site plugin; fix assembly doc up on jenkins builds
[ https://issues.apache.org/jira/browse/HBASE-8058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598864#comment-13598864 ] Hudson commented on HBASE-8058: --- Integrated in hbase-0.95-on-hadoop2 #20 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/20/]) HBASE-8058 Upgrade site plugin; fix assembly doc up on jenkins builds (Revision 1454992) Result = FAILURE stack : Files : * /hbase/branches/0.95/pom.xml Upgrade site plugin; fix assembly doc up on jenkins builds -- Key: HBASE-8058 URL: https://issues.apache.org/jira/browse/HBASE-8058 Project: HBase Issue Type: Bug Components: site Reporter: stack Assignee: stack Priority: Blocker Fix For: 0.95.0 Attachments: 8058.txt, 8058v2.txt Up on jenkins, we currently make assemblies but there no doc in them. The site goal runs last. You can't run it anywhere else else build fails. Upgrading the site plugin helps. Upgrading site plugin I notice that there are a bunch of extra reports generated that would be no harm showing on the web site; e.g. dependencies transitively included, what dependencies we have, etc. This issue is about upgrading site plugin to fix jenkins assemblies and to expose reports we are generating anyways (at least one report is new w/ the info-report upgrade from earlier today). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8063) Filter HFiles based on first/last key
[ https://issues.apache.org/jira/browse/HBASE-8063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598867#comment-13598867 ] Ted Yu commented on HBASE-8063: --- In passesKeyRangeFilter(): {code} + boolean nonOverLapping = (Bytes.compareTo(this.getFirstRowKey(), scan.getStopRow()) 0 {code} '' should be '=' because: {code} * @param stopRow row to stop scanner before (exclusive) {code} Filter HFiles based on first/last key - Key: HBASE-8063 URL: https://issues.apache.org/jira/browse/HBASE-8063 Project: HBase Issue Type: Improvement Components: Scanners Affects Versions: 0.98.0 Reporter: Liang Xie Assignee: Liang Xie Attachments: HBASE-8063.txt # hbase-5010 introduced a filter mechanism based on timerange hint # this jira is a placeholder just for keyrange hint: currently, low layer(hfile reader) has first/last key getter methods already. we should utilize this information to pre-filte hfiles while choosing scanners from storefiles. It's just a raw thought right now, no patch available yet. I'll summit a patch asap if it's reasonable, welcome to give some suggestions/comments:) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8058) Upgrade site plugin; fix assembly doc up on jenkins builds
[ https://issues.apache.org/jira/browse/HBASE-8058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598896#comment-13598896 ] stack commented on HBASE-8058: -- Undid the jenkins change. Works locally but not up on jenkins. So, still looking for how to make this work up on builds... Upgrade site plugin; fix assembly doc up on jenkins builds -- Key: HBASE-8058 URL: https://issues.apache.org/jira/browse/HBASE-8058 Project: HBase Issue Type: Bug Components: site Reporter: stack Assignee: stack Priority: Blocker Fix For: 0.95.0 Attachments: 8058.txt, 8058v2.txt Up on jenkins, we currently make assemblies but there no doc in them. The site goal runs last. You can't run it anywhere else else build fails. Upgrading the site plugin helps. Upgrading site plugin I notice that there are a bunch of extra reports generated that would be no harm showing on the web site; e.g. dependencies transitively included, what dependencies we have, etc. This issue is about upgrading site plugin to fix jenkins assemblies and to expose reports we are generating anyways (at least one report is new w/ the info-report upgrade from earlier today). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7979) Adding some utility methods in org.apache.hadoop.hbase.util.Bytes
[ https://issues.apache.org/jira/browse/HBASE-7979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598945#comment-13598945 ] Anoop Sam John commented on HBASE-7979: --- Committed to Trunk and 0.95 Thanks for the review [~apurtell] Adding some utility methods in org.apache.hadoop.hbase.util.Bytes - Key: HBASE-7979 URL: https://issues.apache.org/jira/browse/HBASE-7979 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.95.0, 0.98.0 Attachments: HBASE-7979_Trunk.patch, HBASE-7979_Trunk.patch, HBASE-7979_Trunk-V2.patch We can add some more utility methods in Bytes. Have seen some in Guava's Bytes class. What I am thinking is adding contains(byte[],byte[]) indexOf(byte[], byte[]) This came under the discussion in HBASE-7928 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8067) TestHFileArchiving.testArchiveOnTableDelete sometimes fails
[ https://issues.apache.org/jira/browse/HBASE-8067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598951#comment-13598951 ] Ted Yu commented on HBASE-8067: --- Looking at admin.deleteTable() in trunk, we use a loop to wait for table deletion to complete: {code} // Wait until all regions deleted ClientProtocol server = connection.getClient(firstMetaServer.getServerName()); for (int tries = 0; tries (this.numRetries * this.retryLongerMultiplier); tries++) { {code} If deletion took too long, I would expect the following: {code} if (tableExists) { throw new IOException(Retries exhausted, it took too long to wait+ for the table + Bytes.toString(tableName) + to be deleted.); {code} TestHFileArchiving.testArchiveOnTableDelete sometimes fails --- Key: HBASE-8067 URL: https://issues.apache.org/jira/browse/HBASE-8067 Project: HBase Issue Type: Bug Components: Admin, master, test Affects Versions: 0.96.0, 0.94.6 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Attachments: HBASE-8067-debug.patch it seems that testArchiveOnTableDelete() fails because the archiving in DeleteTableHandler is still in progress when admin.deleteTable() returns. {code} Error Message Archived files are missing some of the store files! Stacktrace java.lang.AssertionError: Archived files are missing some of the store files! at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.apache.hadoop.hbase.backup.TestHFileArchiving.testArchiveOnTableDelete(TestHFileArchiving.java:262) {code} (Looking at the problem in a more generic way, we don't have any way to inform the client when an async operation is completed) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8067) TestHFileArchiving.testArchiveOnTableDelete sometimes fails
[ https://issues.apache.org/jira/browse/HBASE-8067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598970#comment-13598970 ] stack commented on HBASE-8067: -- [~mbertozzi] +1 on adding debug; go for it. TestHFileArchiving.testArchiveOnTableDelete sometimes fails --- Key: HBASE-8067 URL: https://issues.apache.org/jira/browse/HBASE-8067 Project: HBase Issue Type: Bug Components: Admin, master, test Affects Versions: 0.96.0, 0.94.6 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Attachments: HBASE-8067-debug.patch it seems that testArchiveOnTableDelete() fails because the archiving in DeleteTableHandler is still in progress when admin.deleteTable() returns. {code} Error Message Archived files are missing some of the store files! Stacktrace java.lang.AssertionError: Archived files are missing some of the store files! at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.apache.hadoop.hbase.backup.TestHFileArchiving.testArchiveOnTableDelete(TestHFileArchiving.java:262) {code} (Looking at the problem in a more generic way, we don't have any way to inform the client when an async operation is completed) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7624) Backport HBASE-5359 and HBASE-7596 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599000#comment-13599000 ] Jeffrey Zhong commented on HBASE-7624: -- Sure. Let me to run the test suit for the issue because I also do the same for JIRA HBASE-7824. Backport HBASE-5359 and HBASE-7596 to 0.94 -- Key: HBASE-7624 URL: https://issues.apache.org/jira/browse/HBASE-7624 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Jeffrey Zhong Fix For: 0.94.7 Attachments: hbase-7624_0.patch Both HBASE-5359 and HBASE-7596 are useful and should be added to 0.94. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8068) TestMasterMetricsWrapper failed in 0.95 #60 jenkins
stack created HBASE-8068: Summary: TestMasterMetricsWrapper failed in 0.95 #60 jenkins Key: HBASE-8068 URL: https://issues.apache.org/jira/browse/HBASE-8068 Project: HBase Issue Type: Bug Reporter: stack https://builds.apache.org/job/hbase-0.95/60/testReport/junit/org.apache.hadoop.hbase.master/TestMasterMetricsWrapper/testInfo/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8068) TestMasterMetricsWrapper failed in 0.95 #60 jenkins
[ https://issues.apache.org/jira/browse/HBASE-8068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8068: - Attachment: 8068.txt Looks like master might take a while before it notices regionserver gone. Wait on master noticing its absence before proceeding to assert. TestMasterMetricsWrapper failed in 0.95 #60 jenkins --- Key: HBASE-8068 URL: https://issues.apache.org/jira/browse/HBASE-8068 Project: HBase Issue Type: Bug Reporter: stack Attachments: 8068.txt https://builds.apache.org/job/hbase-0.95/60/testReport/junit/org.apache.hadoop.hbase.master/TestMasterMetricsWrapper/testInfo/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8068) TestMasterMetricsWrapper failed in 0.95 #60 jenkins
[ https://issues.apache.org/jira/browse/HBASE-8068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8068: - Assignee: stack Affects Version/s: 0.95.0 Status: Patch Available (was: Open) TestMasterMetricsWrapper failed in 0.95 #60 jenkins --- Key: HBASE-8068 URL: https://issues.apache.org/jira/browse/HBASE-8068 Project: HBase Issue Type: Bug Affects Versions: 0.95.0 Reporter: stack Assignee: stack Attachments: 8068.txt https://builds.apache.org/job/hbase-0.95/60/testReport/junit/org.apache.hadoop.hbase.master/TestMasterMetricsWrapper/testInfo/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7327) Assignment Timeouts: Remove the code from the master
[ https://issues.apache.org/jira/browse/HBASE-7327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599036#comment-13599036 ] stack commented on HBASE-7327: -- [~nkeywal] It is off by default in 0.95/trunk? Assignment Timeouts: Remove the code from the master Key: HBASE-7327 URL: https://issues.apache.org/jira/browse/HBASE-7327 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Attachments: 7327.v1.uncomplete.patch, 7327.v2.patch As per HBASE-7247... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7827) Improve the speed of Hbase Thirft Batch mutation for deletes
[ https://issues.apache.org/jira/browse/HBASE-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599038#comment-13599038 ] Ted Yu commented on HBASE-7827: --- Integrated to 0.95 and trunk. Thanks for the patch, Shivendra. Thanks for the review, Anoop. Improve the speed of Hbase Thirft Batch mutation for deletes Key: HBASE-7827 URL: https://issues.apache.org/jira/browse/HBASE-7827 Project: HBase Issue Type: New Feature Components: Thrift Affects Versions: 0.94.0 Reporter: Shivendra Pratap Singh Priority: Minor Labels: Hbase, Thrift Fix For: 0.95.0, 0.98.0 Attachments: 7827-trunk.txt, hbase_7827.patch A batch mutate operation does both puts and deletes. Batch mutate for put uses table.put(puts) however batch mutate for delete loops over all deletes and calls table.delete for every single cell. This causes delete performance to degrade. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8040) Race condition in AM after HBASE-7521 (only 0.94)
[ https://issues.apache.org/jira/browse/HBASE-8040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599044#comment-13599044 ] Sergey Shelukhin commented on HBASE-8040: - Looks reasonable... +1 Race condition in AM after HBASE-7521 (only 0.94) - Key: HBASE-8040 URL: https://issues.apache.org/jira/browse/HBASE-8040 Project: HBase Issue Type: Bug Affects Versions: 0.94.6 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.94.7 Attachments: HBASE-8040_1.patch, HBASE-8040.patch This is a problem that introduced when we tried to solve HBASE-7521. https://issues.apache.org/jira/browse/HBASE-7521?focusedCommentId=13576083page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13576083 See the above comment and exactly the same has happened. Will come up with a solution for the same. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8011) Refactor ImportTsv
[ https://issues.apache.org/jira/browse/HBASE-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-8011: Status: Open (was: Patch Available) Refactor ImportTsv -- Key: HBASE-8011 URL: https://issues.apache.org/jira/browse/HBASE-8011 Project: HBase Issue Type: Improvement Components: mapreduce, Usability Reporter: Nick Dimiduk Assignee: Nick Dimiduk Priority: Minor Attachments: 0001-HBASE-8011-Refactor-ImportTsv.patch, 0001-HBASE-8011-Refactor-ImportTsv.patch, 0001-HBASE-8011-Refactor-ImportTsv.patch ImportTsv is a little goofy. - It doesn't use the Tool,Configured interfaces like a mapreduce job should. - It has a static HBaseAdmin field that must be initialized before the intended API of createSubmittableJob can be invoked. - TsvParser is critical to the default mapper implementation but is unavailable to user custom mapper implementations without forcing them into the o.a.h.h.mapreduce namespace. - The configuration key constants are not public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8011) Refactor ImportTsv
[ https://issues.apache.org/jira/browse/HBASE-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-8011: Attachment: 0001-HBASE-8011-Refactor-ImportTsv.patch Bump. Rebased onto trunk. Refactor ImportTsv -- Key: HBASE-8011 URL: https://issues.apache.org/jira/browse/HBASE-8011 Project: HBase Issue Type: Improvement Components: mapreduce, Usability Reporter: Nick Dimiduk Assignee: Nick Dimiduk Priority: Minor Attachments: 0001-HBASE-8011-Refactor-ImportTsv.patch, 0001-HBASE-8011-Refactor-ImportTsv.patch, 0001-HBASE-8011-Refactor-ImportTsv.patch ImportTsv is a little goofy. - It doesn't use the Tool,Configured interfaces like a mapreduce job should. - It has a static HBaseAdmin field that must be initialized before the intended API of createSubmittableJob can be invoked. - TsvParser is critical to the default mapper implementation but is unavailable to user custom mapper implementations without forcing them into the o.a.h.h.mapreduce namespace. - The configuration key constants are not public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8011) Refactor ImportTsv
[ https://issues.apache.org/jira/browse/HBASE-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-8011: Hadoop Flags: Reviewed Status: Patch Available (was: Open) Refactor ImportTsv -- Key: HBASE-8011 URL: https://issues.apache.org/jira/browse/HBASE-8011 Project: HBase Issue Type: Improvement Components: mapreduce, Usability Reporter: Nick Dimiduk Assignee: Nick Dimiduk Priority: Minor Attachments: 0001-HBASE-8011-Refactor-ImportTsv.patch, 0001-HBASE-8011-Refactor-ImportTsv.patch, 0001-HBASE-8011-Refactor-ImportTsv.patch ImportTsv is a little goofy. - It doesn't use the Tool,Configured interfaces like a mapreduce job should. - It has a static HBaseAdmin field that must be initialized before the intended API of createSubmittableJob can be invoked. - TsvParser is critical to the default mapper implementation but is unavailable to user custom mapper implementations without forcing them into the o.a.h.h.mapreduce namespace. - The configuration key constants are not public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8044) split/flush/compact/major_compact from hbase shell does not work for region key with \x format
[ https://issues.apache.org/jira/browse/HBASE-8044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8044: -- Attachment: 8044-trunk.txt split/flush/compact/major_compact from hbase shell does not work for region key with \x format -- Key: HBASE-8044 URL: https://issues.apache.org/jira/browse/HBASE-8044 Project: HBase Issue Type: Bug Components: Admin Affects Versions: 0.94.5 Reporter: Tianying Chang Assignee: Tianying Chang Fix For: 0.94.7 Attachments: 8044.patch, 8044-trunk.txt the conversion between bytes and string is incorrect -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8044) split/flush/compact/major_compact from hbase shell does not work for region key with \x format
[ https://issues.apache.org/jira/browse/HBASE-8044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8044: -- Fix Version/s: 0.98.0 0.95.0 split/flush/compact/major_compact from hbase shell does not work for region key with \x format -- Key: HBASE-8044 URL: https://issues.apache.org/jira/browse/HBASE-8044 Project: HBase Issue Type: Bug Components: Admin Affects Versions: 0.94.5 Reporter: Tianying Chang Assignee: Tianying Chang Fix For: 0.95.0, 0.98.0, 0.94.7 Attachments: 8044.patch, 8044-trunk.txt the conversion between bytes and string is incorrect -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8034) record on-disk data size for store file and make it available during writing
[ https://issues.apache.org/jira/browse/HBASE-8034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599072#comment-13599072 ] Sergey Shelukhin commented on HBASE-8034: - [~ted_yu] what do you mean? Seems complete to me. record on-disk data size for store file and make it available during writing Key: HBASE-8034 URL: https://issues.apache.org/jira/browse/HBASE-8034 Project: HBase Issue Type: Task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor Attachments: HBASE-8034-v0.patch, HBASE-8034-v1.patch To better estimate the size of data in the file, and to be able to split files intelligently during any multi-file compactor like stripe or level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7938) Add integration test for ImportTsv/LoadIncrementalHFiles workflow
[ https://issues.apache.org/jira/browse/HBASE-7938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-7938: Summary: Add integration test for ImportTsv/LoadIncrementalHFiles workflow (was: Add integration test for various MapReduce workflows) Add integration test for ImportTsv/LoadIncrementalHFiles workflow - Key: HBASE-7938 URL: https://issues.apache.org/jira/browse/HBASE-7938 Project: HBase Issue Type: Bug Components: mapreduce Reporter: Nick Dimiduk Fix For: 0.95.0, 0.98.0, 0.94.7 We have existing unit tests for smoke-testing the packaged MR jobs, however they do not create a runtime environment that is true to running on a real MR cluster. This is particularly true in regard to classpaths (HBASE-7934) but also other static state (HBASE-4802). An integration test that can be pointed to run on a pseudo-distributed Hadoop deployed on localhost would find these kinds of problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8034) record on-disk data size for store file and make it available during writing
[ https://issues.apache.org/jira/browse/HBASE-8034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599082#comment-13599082 ] Ted Yu commented on HBASE-8034: --- See the javadoc warning reported by QA. I think DATA_BLOCKS_FRACTION_ESTIMATE should have been DATA_SIZE_FRACTION_ESTIMATE. The constant is a percentage while estimate should be for size. record on-disk data size for store file and make it available during writing Key: HBASE-8034 URL: https://issues.apache.org/jira/browse/HBASE-8034 Project: HBase Issue Type: Task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor Attachments: HBASE-8034-v0.patch, HBASE-8034-v1.patch To better estimate the size of data in the file, and to be able to split files intelligently during any multi-file compactor like stripe or level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7938) Add integration test for ImportTsv/LoadIncrementalHFiles workflow
[ https://issues.apache.org/jira/browse/HBASE-7938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-7938: Attachment: 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch This patch depends on HBASE-8011. The new IntegrationTestImportTsv is written in the style of the recently added IntegrationTestLoadAndVerify. It is designed to be run from maven or directly against a distributed cluster. It shares core test logic with TestImportTsv, from which it inherits. To run it with maven, first refresh your install jars, and the run the test: $ mvn clean package install -DskipTests $ cd hbase-it $ mvn test-compile failsafe:integration-test -Dit.test=IntegrationTestImportTsv Or run it stand-alone: $ HBASE_CLASSPATH=hbase-it/target/hbase-it-version-tests.jar \ ./bin/hbase org.apache.hadoop.hbase.mapreduce.IntegrationTestImportTsv When run stand-alone, it supports all the usual Hadoop Tool arguments, particularly -conf. ImportTsv was refactored to make code sharing easier. It is upgraded from MediumTest to LargeTest because it take 180+ seconds to run on my machine. Tests for the TsvParser are separated into their own SmallTest. Add integration test for ImportTsv/LoadIncrementalHFiles workflow - Key: HBASE-7938 URL: https://issues.apache.org/jira/browse/HBASE-7938 Project: HBase Issue Type: Bug Components: mapreduce Reporter: Nick Dimiduk Fix For: 0.95.0, 0.98.0, 0.94.7 Attachments: 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch We have existing unit tests for smoke-testing the packaged MR jobs, however they do not create a runtime environment that is true to running on a real MR cluster. This is particularly true in regard to classpaths (HBASE-7934) but also other static state (HBASE-4802). An integration test that can be pointed to run on a pseudo-distributed Hadoop deployed on localhost would find these kinds of problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8065) the bulkload can load the hfile into hbase table,but this mechanism can't remove the date before
[ https://issues.apache.org/jira/browse/HBASE-8065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599092#comment-13599092 ] Ted Yu commented on HBASE-8065: --- Is your target release 0.94 ? {code} +return server.bulkLoadHFilesRefresh(famPaths, regionName); {code} How do we know that the server supports this new feature ? Can you provide patch for trunk ? It would be nice if you can upload the trunk patch onto review board. Thanks the bulkload can load the hfile into hbase table,but this mechanism can't remove the date before Key: HBASE-8065 URL: https://issues.apache.org/jira/browse/HBASE-8065 Project: HBase Issue Type: Improvement Components: IPC/RPC, mapreduce, regionserver Affects Versions: 0.94.0 Environment: hadoop-1.0.2、hbase-0.94.0 Reporter: Yuan Kang Assignee: Yuan Kang Priority: Critical Attachments: LoadIncrementalHFiles-bulkload-can-clean-olddata.patch this patch can do bulkload for one more parameter ‘need to refresh’,when this parameter is true.bulkload can clean the old date in the hbase table ,then do the new date load -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8035) Add site target check to precommit tests
[ https://issues.apache.org/jira/browse/HBASE-8035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599100#comment-13599100 ] Nick Dimiduk commented on HBASE-8035: - It looks like we could have the pre-patch script instead call xml:validate: http://mojo.codehaus.org/xml-maven-plugin/validate-mojo.html Add site target check to precommit tests Key: HBASE-8035 URL: https://issues.apache.org/jira/browse/HBASE-8035 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Nick Dimiduk Fix For: 0.98.0 Attachments: 0001-HBASE-8035-Add-site-generation-to-patch-validation.patch, 8035-addendum.txt We should check that the Maven 'site' target passes as part of precommit testing. See HBASE-8022. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8065) the bulkload can load the hfile into hbase table,but this mechanism can't remove the date before
[ https://issues.apache.org/jira/browse/HBASE-8065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599104#comment-13599104 ] Ted Yu commented on HBASE-8065: --- {code} + * @param familyPaths List of Pairbyte[] column family, String hfilePath {code} In javadoc, there is no need to mention parameter names. {code} +this.lock.writeLock().lock(); +this.lock.readLock().lock(); {code} Why are the write/read locks taken consecutively ? the bulkload can load the hfile into hbase table,but this mechanism can't remove the date before Key: HBASE-8065 URL: https://issues.apache.org/jira/browse/HBASE-8065 Project: HBase Issue Type: Improvement Components: IPC/RPC, mapreduce, regionserver Affects Versions: 0.94.0 Environment: hadoop-1.0.2、hbase-0.94.0 Reporter: Yuan Kang Assignee: Yuan Kang Priority: Critical Attachments: LoadIncrementalHFiles-bulkload-can-clean-olddata.patch this patch can do bulkload for one more parameter ‘need to refresh’,when this parameter is true.bulkload can clean the old date in the hbase table ,then do the new date load -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8055) Potentially missing null check in StoreFile.Reader.getMaxTimestamp()
[ https://issues.apache.org/jira/browse/HBASE-8055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599116#comment-13599116 ] Lars Hofhansl commented on HBASE-8055: -- Apparently on our side that happened when a generated HFile somehow bypassed our HFile validation. At the very least I would like to add the simple missing null-check here (as suggested in the description). I will do so, unless I hear objections. (In that case we might be able to undo the changes in HBASE-7581, but I won't do that as part of this jira) Potentially missing null check in StoreFile.Reader.getMaxTimestamp() Key: HBASE-8055 URL: https://issues.apache.org/jira/browse/HBASE-8055 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Fix For: 0.95.0, 0.98.0, 0.94.7 We just ran into a scenario where we got the following NPE: {code} 13/03/08 11:52:13 INFO regionserver.Store: Successfully loaded store file file:/tmp/hfile-import-00Dxx001lmJ-09CxxJm/COLFAM/file09CxxJm into store COLFAM (new location: file:/tmp/localhbase/data/SFDC.ENTITY_HISTORY_ARCHIVE/aeacee43aaf1748c6e60b9cc12bcac3d/COLFAM/120d683414e44478984b50ddd79b6826) 13/03/08 11:52:13 ERROR regionserver.HRegionServer: Failed openScanner java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.getMaxTimestamp(StoreFile.java:1702) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.requestSeek(StoreFileScanner.java:301) at org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:127) at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:2070) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3383) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1628) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1620) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1596) at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2342) at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400) 13/03/08 11:52:14 ERROR regionserver.HRegionServer: Failed openScanner {code} It's not clear, yet, how we got into this situation (we are generating HFiles via HFileOutputFormat and bulk load those). It seems that can only happen when the HFile itself is corrupted. Looking at the code, though, I see this is the only place where we access StoreFile.reader.timeRangeTracker without a null check. So it appears we are expecting scenarios in which it can be null. A simple fix would be: {code} public long getMaxTimestamp() { return timeRangeTracker == null ? Long.MAX_VALUE : timeRangeTracker.maximumTimestamp; } {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8035) Add site target check to precommit tests
[ https://issues.apache.org/jira/browse/HBASE-8035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599119#comment-13599119 ] stack commented on HBASE-8035: -- bq. I believe compile goal is necessary because site implies javadoc, and docbook depends on resources copied into target by compile... You are right. Site runs javadoc but it will be missing the above (unless we make sure they are included by they copy-resources plugin triggered by pre-site). Not important though. Can we close this issue then; the patch has been applied? Add site target check to precommit tests Key: HBASE-8035 URL: https://issues.apache.org/jira/browse/HBASE-8035 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Nick Dimiduk Fix For: 0.98.0 Attachments: 0001-HBASE-8035-Add-site-generation-to-patch-validation.patch, 8035-addendum.txt We should check that the Maven 'site' target passes as part of precommit testing. See HBASE-8022. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8055) Potentially missing null check in StoreFile.Reader.getMaxTimestamp()
[ https://issues.apache.org/jira/browse/HBASE-8055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599122#comment-13599122 ] stack commented on HBASE-8055: -- Lets see a patch. We have a null-check now (NPE -- smile). Interested to see what you will do when you encounter null down here. Potentially missing null check in StoreFile.Reader.getMaxTimestamp() Key: HBASE-8055 URL: https://issues.apache.org/jira/browse/HBASE-8055 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Fix For: 0.95.0, 0.98.0, 0.94.7 We just ran into a scenario where we got the following NPE: {code} 13/03/08 11:52:13 INFO regionserver.Store: Successfully loaded store file file:/tmp/hfile-import-00Dxx001lmJ-09CxxJm/COLFAM/file09CxxJm into store COLFAM (new location: file:/tmp/localhbase/data/SFDC.ENTITY_HISTORY_ARCHIVE/aeacee43aaf1748c6e60b9cc12bcac3d/COLFAM/120d683414e44478984b50ddd79b6826) 13/03/08 11:52:13 ERROR regionserver.HRegionServer: Failed openScanner java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.getMaxTimestamp(StoreFile.java:1702) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.requestSeek(StoreFileScanner.java:301) at org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:127) at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:2070) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3383) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1628) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1620) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1596) at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2342) at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400) 13/03/08 11:52:14 ERROR regionserver.HRegionServer: Failed openScanner {code} It's not clear, yet, how we got into this situation (we are generating HFiles via HFileOutputFormat and bulk load those). It seems that can only happen when the HFile itself is corrupted. Looking at the code, though, I see this is the only place where we access StoreFile.reader.timeRangeTracker without a null check. So it appears we are expecting scenarios in which it can be null. A simple fix would be: {code} public long getMaxTimestamp() { return timeRangeTracker == null ? Long.MAX_VALUE : timeRangeTracker.maximumTimestamp; } {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8044) split/flush/compact/major_compact from hbase shell does not work for region key with \x format
[ https://issues.apache.org/jira/browse/HBASE-8044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599124#comment-13599124 ] Jeffrey Zhong commented on HBASE-8044: -- [~tychang] I'm wondering in what scenarios you've seen the issue? Thanks. split/flush/compact/major_compact from hbase shell does not work for region key with \x format -- Key: HBASE-8044 URL: https://issues.apache.org/jira/browse/HBASE-8044 Project: HBase Issue Type: Bug Components: Admin Affects Versions: 0.94.5 Reporter: Tianying Chang Assignee: Tianying Chang Fix For: 0.95.0, 0.98.0, 0.94.7 Attachments: 8044.patch, 8044-trunk.txt the conversion between bytes and string is incorrect -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8028) Append, Increment don't handle wal-sync exceptions correctly
[ https://issues.apache.org/jira/browse/HBASE-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8028: --- Attachment: HBase-8028-v1.patch Patch to add rollback functionality for Appends. It maintains a map for older kvs that get modified by the append call. When CF#maxVersion = 1, it uses that map to replay the older KVs. And, removes the new KVs which share the same readpoint as the current Append mvcc transaction. It adds a unit test in the MemStore class, which checks whether the transaction rollback or not. Jenkins is green (one unrelated hbck failure was there) Append, Increment don't handle wal-sync exceptions correctly Key: HBASE-8028 URL: https://issues.apache.org/jira/browse/HBASE-8028 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.94.5 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Fix For: 0.95.0 Attachments: HBase-8028-v1.patch In case there is an exception while doing the log-sync, the memstore is not rollbacked, while the mvcc is _always_ forwarded to the writeentry created at the beginning of the operation. This may lead to scanners seeing results which are not synched to the fs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8035) Add site target check to precommit tests
[ https://issues.apache.org/jira/browse/HBASE-8035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599129#comment-13599129 ] Nick Dimiduk commented on HBASE-8035: - Looks like the addendum has not been applied. Add site target check to precommit tests Key: HBASE-8035 URL: https://issues.apache.org/jira/browse/HBASE-8035 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Nick Dimiduk Fix For: 0.98.0 Attachments: 0001-HBASE-8035-Add-site-generation-to-patch-validation.patch, 8035-addendum.txt We should check that the Maven 'site' target passes as part of precommit testing. See HBASE-8022. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8044) split/flush/compact/major_compact from hbase shell does not work for region key with \x format
[ https://issues.apache.org/jira/browse/HBASE-8044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599135#comment-13599135 ] Tianying Chang commented on HBASE-8044: --- for a region name with \x, for example, cmtsdbv2,\x00\x00\x04P]\xEE\x90\x00\x00\x01\x00\x15X\x00\x00\x02\x00\x00\x02\x00\x00\x03\x00\x15\xE3\x00\x00\x04\x00\x15Z\x00\x00\x05\x00\x15[\x00\x00\x06\x00\x15\xE4\x00\x00\x07\x00\x15]\x00\x00\x08\x00\x15\x93\x00\x00\x09\x00\x15\xE5\x00\x00\x0A\x00\x02\xD0,1358267261306.bcec1e1fbeb66de6d02b5ee0b671b1b1. split/flush/compact/major_compact from hbase shell does not work for region key with \x format -- Key: HBASE-8044 URL: https://issues.apache.org/jira/browse/HBASE-8044 Project: HBase Issue Type: Bug Components: Admin Affects Versions: 0.94.5 Reporter: Tianying Chang Assignee: Tianying Chang Fix For: 0.95.0, 0.98.0, 0.94.7 Attachments: 8044.patch, 8044-trunk.txt the conversion between bytes and string is incorrect -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8028) Append, Increment don't handle wal-sync exceptions correctly
[ https://issues.apache.org/jira/browse/HBASE-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8028: --- Status: Patch Available (was: Open) Append, Increment don't handle wal-sync exceptions correctly Key: HBASE-8028 URL: https://issues.apache.org/jira/browse/HBASE-8028 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.94.5 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Fix For: 0.95.0 Attachments: HBase-8028-v1.patch In case there is an exception while doing the log-sync, the memstore is not rollbacked, while the mvcc is _always_ forwarded to the writeentry created at the beginning of the operation. This may lead to scanners seeing results which are not synched to the fs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8068) TestMasterMetricsWrapper failed in 0.95 #60 jenkins
[ https://issues.apache.org/jira/browse/HBASE-8068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599140#comment-13599140 ] Elliott Clark commented on HBASE-8068: -- +1 lgtm TestMasterMetricsWrapper failed in 0.95 #60 jenkins --- Key: HBASE-8068 URL: https://issues.apache.org/jira/browse/HBASE-8068 Project: HBase Issue Type: Bug Affects Versions: 0.95.0 Reporter: stack Assignee: stack Attachments: 8068.txt https://builds.apache.org/job/hbase-0.95/60/testReport/junit/org.apache.hadoop.hbase.master/TestMasterMetricsWrapper/testInfo/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8068) TestMasterMetricsWrapper failed in 0.95 #60 jenkins
[ https://issues.apache.org/jira/browse/HBASE-8068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-8068: - Component/s: metrics TestMasterMetricsWrapper failed in 0.95 #60 jenkins --- Key: HBASE-8068 URL: https://issues.apache.org/jira/browse/HBASE-8068 Project: HBase Issue Type: Bug Components: metrics Affects Versions: 0.95.0 Reporter: stack Assignee: stack Attachments: 8068.txt https://builds.apache.org/job/hbase-0.95/60/testReport/junit/org.apache.hadoop.hbase.master/TestMasterMetricsWrapper/testInfo/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8011) Refactor ImportTsv
[ https://issues.apache.org/jira/browse/HBASE-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8011: - Resolution: Fixed Fix Version/s: 0.95.0 Status: Resolved (was: Patch Available) Committed to branch and trunk. Thanks Nick. Refactor ImportTsv -- Key: HBASE-8011 URL: https://issues.apache.org/jira/browse/HBASE-8011 Project: HBase Issue Type: Improvement Components: mapreduce, Usability Reporter: Nick Dimiduk Assignee: Nick Dimiduk Priority: Minor Fix For: 0.95.0 Attachments: 0001-HBASE-8011-Refactor-ImportTsv.patch, 0001-HBASE-8011-Refactor-ImportTsv.patch, 0001-HBASE-8011-Refactor-ImportTsv.patch ImportTsv is a little goofy. - It doesn't use the Tool,Configured interfaces like a mapreduce job should. - It has a static HBaseAdmin field that must be initialized before the intended API of createSubmittableJob can be invoked. - TsvParser is critical to the default mapper implementation but is unavailable to user custom mapper implementations without forcing them into the o.a.h.h.mapreduce namespace. - The configuration key constants are not public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8068) TestMasterMetricsWrapper failed in 0.95 #60 jenkins
[ https://issues.apache.org/jira/browse/HBASE-8068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599144#comment-13599144 ] Himanshu Vashishtha commented on HBASE-8068: +1; yes, the rs was processed by Master and added to the deadregionserver list too it just took some time (about 300ms). TestMasterMetricsWrapper failed in 0.95 #60 jenkins --- Key: HBASE-8068 URL: https://issues.apache.org/jira/browse/HBASE-8068 Project: HBase Issue Type: Bug Components: metrics Affects Versions: 0.95.0 Reporter: stack Assignee: stack Attachments: 8068.txt https://builds.apache.org/job/hbase-0.95/60/testReport/junit/org.apache.hadoop.hbase.master/TestMasterMetricsWrapper/testInfo/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8068) TestMasterMetricsWrapper failed in 0.95 #60 jenkins
[ https://issues.apache.org/jira/browse/HBASE-8068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599145#comment-13599145 ] Hadoop QA commented on HBASE-8068: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12573128/8068.txt against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified tests. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:red}-1 site{color}. The patch appears to cause mvn site goal to fail. {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestMasterMetricsWrapper org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4756//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4756//console This message is automatically generated. TestMasterMetricsWrapper failed in 0.95 #60 jenkins --- Key: HBASE-8068 URL: https://issues.apache.org/jira/browse/HBASE-8068 Project: HBase Issue Type: Bug Components: metrics Affects Versions: 0.95.0 Reporter: stack Assignee: stack Attachments: 8068.txt https://builds.apache.org/job/hbase-0.95/60/testReport/junit/org.apache.hadoop.hbase.master/TestMasterMetricsWrapper/testInfo/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8035) Add site target check to precommit tests
[ https://issues.apache.org/jira/browse/HBASE-8035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599146#comment-13599146 ] stack commented on HBASE-8035: -- I think the addendum is useless doing nothing but using up more space on jenkins. Add site target check to precommit tests Key: HBASE-8035 URL: https://issues.apache.org/jira/browse/HBASE-8035 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Nick Dimiduk Fix For: 0.98.0 Attachments: 0001-HBASE-8035-Add-site-generation-to-patch-validation.patch, 8035-addendum.txt We should check that the Maven 'site' target passes as part of precommit testing. See HBASE-8022. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8068) TestMasterMetricsWrapper failed in 0.95 #60 jenkins
[ https://issues.apache.org/jira/browse/HBASE-8068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599149#comment-13599149 ] stack commented on HBASE-8068: -- Wrong version of the patch. Had a fix for 'Use assertEquals(expected, actual, delta) to compare floating-point numbers' Let me redo hadoopqa. TestMasterMetricsWrapper failed in 0.95 #60 jenkins --- Key: HBASE-8068 URL: https://issues.apache.org/jira/browse/HBASE-8068 Project: HBase Issue Type: Bug Components: metrics Affects Versions: 0.95.0 Reporter: stack Assignee: stack Attachments: 8068.txt https://builds.apache.org/job/hbase-0.95/60/testReport/junit/org.apache.hadoop.hbase.master/TestMasterMetricsWrapper/testInfo/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8068) TestMasterMetricsWrapper failed in 0.95 #60 jenkins
[ https://issues.apache.org/jira/browse/HBASE-8068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8068: - Attachment: 8068v2.txt TestMasterMetricsWrapper failed in 0.95 #60 jenkins --- Key: HBASE-8068 URL: https://issues.apache.org/jira/browse/HBASE-8068 Project: HBase Issue Type: Bug Components: metrics Affects Versions: 0.95.0 Reporter: stack Assignee: stack Attachments: 8068.txt, 8068v2.txt https://builds.apache.org/job/hbase-0.95/60/testReport/junit/org.apache.hadoop.hbase.master/TestMasterMetricsWrapper/testInfo/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8044) split/flush/compact/major_compact from hbase shell does not work for region key with \x format
[ https://issues.apache.org/jira/browse/HBASE-8044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599152#comment-13599152 ] Jeffrey Zhong commented on HBASE-8044: -- I c. Thanks. Will closeRegion and getCompactionState have the same issue? split/flush/compact/major_compact from hbase shell does not work for region key with \x format -- Key: HBASE-8044 URL: https://issues.apache.org/jira/browse/HBASE-8044 Project: HBase Issue Type: Bug Components: Admin Affects Versions: 0.94.5 Reporter: Tianying Chang Assignee: Tianying Chang Fix For: 0.95.0, 0.98.0, 0.94.7 Attachments: 8044.patch, 8044-trunk.txt the conversion between bytes and string is incorrect -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8035) Add site target check to precommit tests
[ https://issues.apache.org/jira/browse/HBASE-8035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599158#comment-13599158 ] Ted Yu commented on HBASE-8035: --- Looking through recent QA reports, site check all failed. We should do something before closing this issue. Add site target check to precommit tests Key: HBASE-8035 URL: https://issues.apache.org/jira/browse/HBASE-8035 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Nick Dimiduk Fix For: 0.98.0 Attachments: 0001-HBASE-8035-Add-site-generation-to-patch-validation.patch, 8035-addendum.txt We should check that the Maven 'site' target passes as part of precommit testing. See HBASE-8022. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8044) split/flush/compact/major_compact from hbase shell does not work for region key with \x format
[ https://issues.apache.org/jira/browse/HBASE-8044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599174#comment-13599174 ] Hadoop QA commented on HBASE-8044: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12573133/8044-trunk.txt against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:red}-1 site{color}. The patch appears to cause mvn site goal to fail. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4757//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4757//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4757//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4757//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4757//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4757//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4757//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4757//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4757//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4757//console This message is automatically generated. split/flush/compact/major_compact from hbase shell does not work for region key with \x format -- Key: HBASE-8044 URL: https://issues.apache.org/jira/browse/HBASE-8044 Project: HBase Issue Type: Bug Components: Admin Affects Versions: 0.94.5 Reporter: Tianying Chang Assignee: Tianying Chang Fix For: 0.95.0, 0.98.0, 0.94.7 Attachments: 8044.patch, 8044-trunk.txt the conversion between bytes and string is incorrect -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8011) Refactor ImportTsv
[ https://issues.apache.org/jira/browse/HBASE-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599176#comment-13599176 ] Hadoop QA commented on HBASE-8011: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12573132/0001-HBASE-8011-Refactor-ImportTsv.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified tests. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 lineLengths{color}. The patch introduces lines longer than 100 {color:red}-1 site{color}. The patch appears to cause mvn site goal to fail. {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestAtomicOperation org.apache.hadoop.hbase.regionserver.wal.TestHLog Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4758//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4758//console This message is automatically generated. Refactor ImportTsv -- Key: HBASE-8011 URL: https://issues.apache.org/jira/browse/HBASE-8011 Project: HBase Issue Type: Improvement Components: mapreduce, Usability Reporter: Nick Dimiduk Assignee: Nick Dimiduk Priority: Minor Fix For: 0.95.0 Attachments: 0001-HBASE-8011-Refactor-ImportTsv.patch, 0001-HBASE-8011-Refactor-ImportTsv.patch, 0001-HBASE-8011-Refactor-ImportTsv.patch ImportTsv is a little goofy. - It doesn't use the Tool,Configured interfaces like a mapreduce job should. - It has a static HBaseAdmin field that must be initialized before the intended API of createSubmittableJob can be invoked. - TsvParser is critical to the default mapper implementation but is unavailable to user custom mapper implementations without forcing them into the o.a.h.h.mapreduce namespace. - The configuration key constants are not public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8035) Add site target check to precommit tests
[ https://issues.apache.org/jira/browse/HBASE-8035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599184#comment-13599184 ] stack commented on HBASE-8035: -- For sure would be good to know why site failed. Add site target check to precommit tests Key: HBASE-8035 URL: https://issues.apache.org/jira/browse/HBASE-8035 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Nick Dimiduk Fix For: 0.98.0 Attachments: 0001-HBASE-8035-Add-site-generation-to-patch-validation.patch, 8035-addendum.txt We should check that the Maven 'site' target passes as part of precommit testing. See HBASE-8022. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8035) Add site target check to precommit tests
[ https://issues.apache.org/jira/browse/HBASE-8035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599186#comment-13599186 ] Nick Dimiduk commented on HBASE-8035: - https://builds.apache.org/job/PreCommit-HBASE-Build/4758//artifact/trunk/patchprocess/patchSiteOutput.txt With the existing `clean compile site -DskipTests`, site didn't fail, the subsequent module builds do, which goes back to the compile vs package business. Add site target check to precommit tests Key: HBASE-8035 URL: https://issues.apache.org/jira/browse/HBASE-8035 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Nick Dimiduk Fix For: 0.98.0 Attachments: 0001-HBASE-8035-Add-site-generation-to-patch-validation.patch, 8035-addendum.txt We should check that the Maven 'site' target passes as part of precommit testing. See HBASE-8022. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8035) Add site target check to precommit tests
[ https://issues.apache.org/jira/browse/HBASE-8035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599190#comment-13599190 ] stack commented on HBASE-8035: -- I run it locally and the exit code is 0 Add site target check to precommit tests Key: HBASE-8035 URL: https://issues.apache.org/jira/browse/HBASE-8035 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Nick Dimiduk Fix For: 0.98.0 Attachments: 0001-HBASE-8035-Add-site-generation-to-patch-validation.patch, 8035-addendum.txt We should check that the Maven 'site' target passes as part of precommit testing. See HBASE-8022. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8035) Add site target check to precommit tests
[ https://issues.apache.org/jira/browse/HBASE-8035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599192#comment-13599192 ] stack commented on HBASE-8035: -- [~ndimiduk] Sorry, your script has `clean compile site -DskipTests` in it... how do we get to the 'compile vs package business'? (I missed that there is a siteOutput nice) Add site target check to precommit tests Key: HBASE-8035 URL: https://issues.apache.org/jira/browse/HBASE-8035 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Nick Dimiduk Fix For: 0.98.0 Attachments: 0001-HBASE-8035-Add-site-generation-to-patch-validation.patch, 8035-addendum.txt We should check that the Maven 'site' target passes as part of precommit testing. See HBASE-8022. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8035) Add site target check to precommit tests
[ https://issues.apache.org/jira/browse/HBASE-8035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599194#comment-13599194 ] Nick Dimiduk commented on HBASE-8035: - This appears to work. `mvn -N clean site -DskipTests` builds the docbook locally for me, and skips the walk through submodules. It complains loudly when I first `rm -r ~/.m2/repository/org/apache/hbase/hbase*/0.97-SNAPSHOT` but doesn't fail the build. The build succeeds and just creates docbook. {noformat} [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 43.186s {noformat} Add site target check to precommit tests Key: HBASE-8035 URL: https://issues.apache.org/jira/browse/HBASE-8035 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Nick Dimiduk Fix For: 0.98.0 Attachments: 0001-HBASE-8035-Add-site-generation-to-patch-validation.patch, 8035-addendum.txt We should check that the Maven 'site' target passes as part of precommit testing. See HBASE-8022. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8035) Add site target check to precommit tests
[ https://issues.apache.org/jira/browse/HBASE-8035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-8035: Attachment: 8035-addendum.txt Attaching new addendum patch. Add site target check to precommit tests Key: HBASE-8035 URL: https://issues.apache.org/jira/browse/HBASE-8035 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Nick Dimiduk Fix For: 0.98.0 Attachments: 0001-HBASE-8035-Add-site-generation-to-patch-validation.patch, 8035-addendum.txt, 8035-addendum.txt We should check that the Maven 'site' target passes as part of precommit testing. See HBASE-8022. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8069) TestHLog is dependent on the execution order
Matteo Bertozzi created HBASE-8069: -- Summary: TestHLog is dependent on the execution order Key: HBASE-8069 URL: https://issues.apache.org/jira/browse/HBASE-8069 Project: HBase Issue Type: Bug Components: test, wal Affects Versions: 0.94.5 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Changing the execution order of the tests, TestHLog show up as failing {code}runOrderalphabetical/runOrder{code} testAppendClose() changes the DFS cluster of the test (set globally) so the tests executed after has the new DFS. Trying to start a new mini cluster for every test @Before seems to solve the problem. {code} testSplit(org.apache.hadoop.hbase.regionserver.wal.TestHLog): 3 exceptions [org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/th30z/hbase/TestHLog/21985ea121a1d65fa82d90d521be7313/recovered.edits/001.temp File is not open for writing. Holder DFSClient_NONMAPREDUCE_1044150136_583 does not have any open files. {code} Looking at testSplit() the failure seems to be in logSplitter.splitLog(), OutputSink.finishWritingAndClose() is not able to close the files and rethrow the exception. (On trunk I'm not able to reproduce the problem) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7979) Adding some utility methods in org.apache.hadoop.hbase.util.Bytes
[ https://issues.apache.org/jira/browse/HBASE-7979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599200#comment-13599200 ] Hudson commented on HBASE-7979: --- Integrated in HBase-TRUNK #3945 (See [https://builds.apache.org/job/HBase-TRUNK/3945/]) HBASE-7979 Adding some utility methods in org.apache.hadoop.hbase.util.Bytes (Revision 1455213) Result = FAILURE anoopsamjohn : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestBytes.java Adding some utility methods in org.apache.hadoop.hbase.util.Bytes - Key: HBASE-7979 URL: https://issues.apache.org/jira/browse/HBASE-7979 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.95.0, 0.98.0 Attachments: HBASE-7979_Trunk.patch, HBASE-7979_Trunk.patch, HBASE-7979_Trunk-V2.patch We can add some more utility methods in Bytes. Have seen some in Guava's Bytes class. What I am thinking is adding contains(byte[],byte[]) indexOf(byte[], byte[]) This came under the discussion in HBASE-7928 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8067) TestHFileArchiving.testArchiveOnTableDelete sometimes fails
[ https://issues.apache.org/jira/browse/HBASE-8067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599201#comment-13599201 ] Hudson commented on HBASE-8067: --- Integrated in HBase-TRUNK #3945 (See [https://builds.apache.org/job/HBase-TRUNK/3945/]) HBASE-8067 TestHFileArchiving.testArchiveOnTableDelete sometimes fails -- more debug (Revision 1455234) Result = FAILURE mbertozzi : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java TestHFileArchiving.testArchiveOnTableDelete sometimes fails --- Key: HBASE-8067 URL: https://issues.apache.org/jira/browse/HBASE-8067 Project: HBase Issue Type: Bug Components: Admin, master, test Affects Versions: 0.96.0, 0.94.6 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Attachments: HBASE-8067-debug.patch it seems that testArchiveOnTableDelete() fails because the archiving in DeleteTableHandler is still in progress when admin.deleteTable() returns. {code} Error Message Archived files are missing some of the store files! Stacktrace java.lang.AssertionError: Archived files are missing some of the store files! at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.apache.hadoop.hbase.backup.TestHFileArchiving.testArchiveOnTableDelete(TestHFileArchiving.java:262) {code} (Looking at the problem in a more generic way, we don't have any way to inform the client when an async operation is completed) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7827) Improve the speed of Hbase Thirft Batch mutation for deletes
[ https://issues.apache.org/jira/browse/HBASE-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599202#comment-13599202 ] Hudson commented on HBASE-7827: --- Integrated in HBase-TRUNK #3945 (See [https://builds.apache.org/job/HBase-TRUNK/3945/]) HBASE-7827 Improve the speed of Hbase Thirft Batch mutation for deletes (Shivendra Pratap Singh) (Revision 1455248) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java Improve the speed of Hbase Thirft Batch mutation for deletes Key: HBASE-7827 URL: https://issues.apache.org/jira/browse/HBASE-7827 Project: HBase Issue Type: New Feature Components: Thrift Affects Versions: 0.94.0 Reporter: Shivendra Pratap Singh Priority: Minor Labels: Hbase, Thrift Fix For: 0.95.0, 0.98.0 Attachments: 7827-trunk.txt, hbase_7827.patch A batch mutate operation does both puts and deletes. Batch mutate for put uses table.put(puts) however batch mutate for delete loops over all deletes and calls table.delete for every single cell. This causes delete performance to degrade. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8028) Append, Increment don't handle wal-sync exceptions correctly
[ https://issues.apache.org/jira/browse/HBASE-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599203#comment-13599203 ] Hadoop QA commented on HBASE-8028: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12573143/HBase-8028-v1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified tests. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:red}-1 site{color}. The patch appears to cause mvn site goal to fail. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4759//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4759//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4759//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4759//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4759//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4759//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4759//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4759//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4759//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4759//console This message is automatically generated. Append, Increment don't handle wal-sync exceptions correctly Key: HBASE-8028 URL: https://issues.apache.org/jira/browse/HBASE-8028 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.94.5 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Fix For: 0.95.0 Attachments: HBase-8028-v1.patch In case there is an exception while doing the log-sync, the memstore is not rollbacked, while the mvcc is _always_ forwarded to the writeentry created at the beginning of the operation. This may lead to scanners seeing results which are not synched to the fs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira