[jira] [Commented] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally
[
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420114#comment-13420114
]
Hadoop QA commented on HBASE-5659:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12537477/5659.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 5 javac compiler warnings (more than
the trunk's current 4 warnings).
-1 findbugs. The patch appears to introduce 14 new Findbugs (version
1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestSplitLogManager
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2423//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2423//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2423//console
This message is automatically generated.
> TestAtomicOperation.testMultiRowMutationMultiThreads is still failing
> occasionally
> --
>
> Key: HBASE-5659
> URL: https://issues.apache.org/jira/browse/HBASE-5659
> Project: HBase
> Issue Type: Sub-task
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Minor
> Fix For: 0.96.0, 0.94.2
>
> Attachments: 5659.txt
>
>
> See run here:
> https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
> {quote}
> 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after =
> rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
> 2012-03-27 04:36:12,629 INFO [Thread-121] regionserver.HRegion(1558):
> Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region
> testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms,
> sequenceid=7927, compaction requested=true
> 2012-03-27 04:36:12,629 DEBUG [Thread-126]
> regionserver.TestAtomicOperation$2(362): flushing
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426):
> Started memstore flush for
> testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region
> memstore size 1.9k
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474):
> Finished snapshotting
> testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait
> for mvcc, flushsize=1968
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484):
> Finished snapshotting, commencing flushing stores
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating
> file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
> with permission=rwxrwxrwx
> 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143):
> Initialized with CacheConfig:enabled [cacheDataOnRead=true]
> [cacheDataOnWrite=false] [cacheIndexesOnWrite=false]
> [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
> 2012-03-27 04:36:12,631 INFO [Thread-126]
> regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/targ
[jira] [Updated] (HBASE-5547) Don't delete HFiles when in "backup mode"
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5547: --- Attachment: java_HBASE-5547.addendum Attaching addendum that updates the test (removes the TTL cleaner to avoid timing issues) and fixes a couple of comments. Looks like a flapper given that build #3159 worked. Couldn't get the NPE to reproduce (didn't seem fatal though and was from a log statement). Ran the updated test 20x without failure. > Don't delete HFiles when in "backup mode" > - > > Key: HBASE-5547 > URL: https://issues.apache.org/jira/browse/HBASE-5547 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Jesse Yates > Fix For: 0.96.0, 0.94.2 > > Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, > hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum, > java_HBASE-5547_v13.patch, java_HBASE-5547_v14.patch, > java_HBASE-5547_v15.patch, java_HBASE-5547_v4.patch, > java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, java_HBASE-5547_v7.patch > > > This came up in a discussion I had with Stack. > It would be nice if HBase could be notified that a backup is in progress (via > a znode for example) and in that case either: > 1. rename HFiles to be delete to .bck > 2. rename the HFiles into a special directory > 3. rename them to a general trash directory (which would not need to be tied > to backup mode). > That way it should be able to get a consistent backup based on HFiles (HDFS > snapshots or hard links would be better options here, but we do not have > those). > #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6433) Improve HBaseServer#getRemoteAddress by utilizing HBaseServer.Connection.hostAddress
[ https://issues.apache.org/jira/browse/HBASE-6433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6433: - Fix Version/s: 0.94.1 Let's get into 0.94 before the next RC. > Improve HBaseServer#getRemoteAddress by utilizing > HBaseServer.Connection.hostAddress > > > Key: HBASE-6433 > URL: https://issues.apache.org/jira/browse/HBASE-6433 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Fix For: 0.96.0, 0.94.1 > > Attachments: 6433-getRemoteAddress-trunk.txt, HBASE-6433-90.patch, > HBASE-6433-92.patch, HBASE-6433-94.patch, HBASE-6433-trunk.patch > > > Currently, HBaseServer#getRemoteAddress would call getRemoteIp(), leading to > call.connection.socket.getInetAddress(). > The host address is actually stored in HBaseServer.Connection.hostAddress > field. We don't need to go through Socket to get this information. > Without this patch it costs 4000ns, with this patch it costs 1600ns -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally
[
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-5659:
-
Status: Patch Available (was: Open)
> TestAtomicOperation.testMultiRowMutationMultiThreads is still failing
> occasionally
> --
>
> Key: HBASE-5659
> URL: https://issues.apache.org/jira/browse/HBASE-5659
> Project: HBase
> Issue Type: Sub-task
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Minor
> Fix For: 0.96.0, 0.94.2
>
> Attachments: 5659.txt
>
>
> See run here:
> https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
> {quote}
> 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after =
> rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
> 2012-03-27 04:36:12,629 INFO [Thread-121] regionserver.HRegion(1558):
> Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region
> testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms,
> sequenceid=7927, compaction requested=true
> 2012-03-27 04:36:12,629 DEBUG [Thread-126]
> regionserver.TestAtomicOperation$2(362): flushing
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426):
> Started memstore flush for
> testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region
> memstore size 1.9k
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474):
> Finished snapshotting
> testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait
> for mvcc, flushsize=1968
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484):
> Finished snapshotting, commencing flushing stores
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating
> file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
> with permission=rwxrwxrwx
> 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143):
> Initialized with CacheConfig:enabled [cacheDataOnRead=true]
> [cacheDataOnWrite=false] [cacheIndexesOnWrite=false]
> [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
> 2012-03-27 04:36:12,631 INFO [Thread-126]
> regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
> CompoundBloomFilterWriter
> 2012-03-27 04:36:12,632 INFO [Thread-126]
> regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was
> added to HFile
> (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
>
> 2012-03-27 04:36:12,632 INFO [Thread-126] regionserver.Store(770): Flushed ,
> sequenceid=7934, memsize=1.9k, into tmp file
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
> 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming
> flushed file at
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
> to
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57
> 2012-03-27 04:36:12,634 INFO [Thread-126] regionserver.Store(818): Added
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57,
> entries=12, sequenceid=7934, filesize=1.3k
> 2012-03-27 04:36:12,642 DEBUG [Thread-118]
> regionserver.TestAt
[jira] [Updated] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally
[
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-5659:
-
Attachment: 5659.txt
While this (probably) won't fix the problem I would like to propose this simple
change to the logic introduced in HBASE-5121.
The observation is that a change top KV is only a problem when the KV's row has
changed from the previous one.
> TestAtomicOperation.testMultiRowMutationMultiThreads is still failing
> occasionally
> --
>
> Key: HBASE-5659
> URL: https://issues.apache.org/jira/browse/HBASE-5659
> Project: HBase
> Issue Type: Sub-task
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Minor
> Fix For: 0.96.0, 0.94.2
>
> Attachments: 5659.txt
>
>
> See run here:
> https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
> {quote}
> 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after =
> rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
> 2012-03-27 04:36:12,629 INFO [Thread-121] regionserver.HRegion(1558):
> Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region
> testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms,
> sequenceid=7927, compaction requested=true
> 2012-03-27 04:36:12,629 DEBUG [Thread-126]
> regionserver.TestAtomicOperation$2(362): flushing
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426):
> Started memstore flush for
> testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region
> memstore size 1.9k
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474):
> Finished snapshotting
> testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait
> for mvcc, flushsize=1968
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484):
> Finished snapshotting, commencing flushing stores
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating
> file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
> with permission=rwxrwxrwx
> 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143):
> Initialized with CacheConfig:enabled [cacheDataOnRead=true]
> [cacheDataOnWrite=false] [cacheIndexesOnWrite=false]
> [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
> 2012-03-27 04:36:12,631 INFO [Thread-126]
> regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
> CompoundBloomFilterWriter
> 2012-03-27 04:36:12,632 INFO [Thread-126]
> regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was
> added to HFile
> (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
>
> 2012-03-27 04:36:12,632 INFO [Thread-126] regionserver.Store(770): Flushed ,
> sequenceid=7934, memsize=1.9k, into tmp file
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
> 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming
> flushed file at
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
> to
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57
> 2012-03-27 04:36:12,634 INFO [Thread-126] regionserver.Store(818): Added
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperati
[jira] [Assigned] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally
[
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl reassigned HBASE-5659:
Assignee: Lars Hofhansl
> TestAtomicOperation.testMultiRowMutationMultiThreads is still failing
> occasionally
> --
>
> Key: HBASE-5659
> URL: https://issues.apache.org/jira/browse/HBASE-5659
> Project: HBase
> Issue Type: Sub-task
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Minor
> Fix For: 0.96.0, 0.94.2
>
>
> See run here:
> https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
> {quote}
> 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after =
> rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
> 2012-03-27 04:36:12,629 INFO [Thread-121] regionserver.HRegion(1558):
> Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region
> testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms,
> sequenceid=7927, compaction requested=true
> 2012-03-27 04:36:12,629 DEBUG [Thread-126]
> regionserver.TestAtomicOperation$2(362): flushing
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426):
> Started memstore flush for
> testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region
> memstore size 1.9k
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474):
> Finished snapshotting
> testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait
> for mvcc, flushsize=1968
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484):
> Finished snapshotting, commencing flushing stores
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating
> file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
> with permission=rwxrwxrwx
> 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143):
> Initialized with CacheConfig:enabled [cacheDataOnRead=true]
> [cacheDataOnWrite=false] [cacheIndexesOnWrite=false]
> [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
> 2012-03-27 04:36:12,631 INFO [Thread-126]
> regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
> CompoundBloomFilterWriter
> 2012-03-27 04:36:12,632 INFO [Thread-126]
> regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was
> added to HFile
> (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
>
> 2012-03-27 04:36:12,632 INFO [Thread-126] regionserver.Store(770): Flushed ,
> sequenceid=7934, memsize=1.9k, into tmp file
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
> 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming
> flushed file at
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
> to
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57
> 2012-03-27 04:36:12,634 INFO [Thread-126] regionserver.Store(818): Added
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57,
> entries=12, sequenceid=7934, filesize=1.3k
> 2012-03-27 04:36:12,642 DEBUG [Thread-118]
> regionserver.TestAtomicOperation$2(392): []
> Exception in t
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in "backup mode"
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420103#comment-13420103 ] Jesse Yates commented on HBASE-5547: Hmmm, looks like there was a NPE generated during a compaction - probably what broke the test, but its pretty early on, so maybe not. Looking into it to see what is going on... > Don't delete HFiles when in "backup mode" > - > > Key: HBASE-5547 > URL: https://issues.apache.org/jira/browse/HBASE-5547 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Jesse Yates > Fix For: 0.96.0, 0.94.2 > > Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, > hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547_v13.patch, > java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, > java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, > java_HBASE-5547_v7.patch > > > This came up in a discussion I had with Stack. > It would be nice if HBase could be notified that a backup is in progress (via > a znode for example) and in that case either: > 1. rename HFiles to be delete to .bck > 2. rename the HFiles into a special directory > 3. rename them to a general trash directory (which would not need to be tied > to backup mode). > That way it should be able to get a consistent backup based on HFiles (HDFS > snapshots or hard links would be better options here, but we do not have > those). > #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally
[
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420099#comment-13420099
]
Lars Hofhansl commented on HBASE-5659:
--
I think there is a subtle bug lurking somewhere. This test just happens to
trigger that problem.
> TestAtomicOperation.testMultiRowMutationMultiThreads is still failing
> occasionally
> --
>
> Key: HBASE-5659
> URL: https://issues.apache.org/jira/browse/HBASE-5659
> Project: HBase
> Issue Type: Sub-task
>Reporter: Lars Hofhansl
>Priority: Minor
> Fix For: 0.96.0, 0.94.2
>
>
> See run here:
> https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
> {quote}
> 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after =
> rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
> 2012-03-27 04:36:12,629 INFO [Thread-121] regionserver.HRegion(1558):
> Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region
> testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms,
> sequenceid=7927, compaction requested=true
> 2012-03-27 04:36:12,629 DEBUG [Thread-126]
> regionserver.TestAtomicOperation$2(362): flushing
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426):
> Started memstore flush for
> testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region
> memstore size 1.9k
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474):
> Finished snapshotting
> testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait
> for mvcc, flushsize=1968
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484):
> Finished snapshotting, commencing flushing stores
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating
> file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
> with permission=rwxrwxrwx
> 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143):
> Initialized with CacheConfig:enabled [cacheDataOnRead=true]
> [cacheDataOnWrite=false] [cacheIndexesOnWrite=false]
> [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
> 2012-03-27 04:36:12,631 INFO [Thread-126]
> regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
> CompoundBloomFilterWriter
> 2012-03-27 04:36:12,632 INFO [Thread-126]
> regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was
> added to HFile
> (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
>
> 2012-03-27 04:36:12,632 INFO [Thread-126] regionserver.Store(770): Flushed ,
> sequenceid=7934, memsize=1.9k, into tmp file
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
> 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming
> flushed file at
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
> to
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57
> 2012-03-27 04:36:12,634 INFO [Thread-126] regionserver.Store(818): Added
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57,
> entries=12, sequenceid=7934, filesize=1.3k
> 2012-03-27
[jira] [Commented] (HBASE-5659) TestAtomicOperation.testMultiRowMutationMultiThreads is still failing occasionally
[
https://issues.apache.org/jira/browse/HBASE-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420098#comment-13420098
]
Lars Hofhansl commented on HBASE-5659:
--
This always comes after the change from HBASE-5121 is triggered.
Looking back at HBASE-5121 it should only be triggered during a major
compaction, but apparently it is not.
> TestAtomicOperation.testMultiRowMutationMultiThreads is still failing
> occasionally
> --
>
> Key: HBASE-5659
> URL: https://issues.apache.org/jira/browse/HBASE-5659
> Project: HBase
> Issue Type: Sub-task
>Reporter: Lars Hofhansl
>Priority: Minor
> Fix For: 0.96.0, 0.94.2
>
>
> See run here:
> https://builds.apache.org/job/PreCommit-HBASE-Build/1318//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/
> {quote}
> 2012-03-27 04:36:12,627 DEBUG [Thread-118] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/7202/Put/vlen=6/ts=7922,and after =
> rowB/colfamily11:qual1/7199/DeleteColumn/vlen=0/ts=0
> 2012-03-27 04:36:12,629 INFO [Thread-121] regionserver.HRegion(1558):
> Finished memstore flush of ~2.9k/2952, currentsize=1.6k/1640 for region
> testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81. in 14ms,
> sequenceid=7927, compaction requested=true
> 2012-03-27 04:36:12,629 DEBUG [Thread-126]
> regionserver.TestAtomicOperation$2(362): flushing
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1426):
> Started memstore flush for
> testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., current region
> memstore size 1.9k
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1474):
> Finished snapshotting
> testtable,,1332822963417.7cd30e219714cfc5e91f69def66e7f81., commencing wait
> for mvcc, flushsize=1968
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] regionserver.HRegion(1484):
> Finished snapshotting, commencing flushing stores
> 2012-03-27 04:36:12,630 DEBUG [Thread-126] util.FSUtils(153): Creating
> file=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
> with permission=rwxrwxrwx
> 2012-03-27 04:36:12,631 DEBUG [Thread-126] hfile.HFileWriterV2(143):
> Initialized with CacheConfig:enabled [cacheDataOnRead=true]
> [cacheDataOnWrite=false] [cacheIndexesOnWrite=false]
> [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
> 2012-03-27 04:36:12,631 INFO [Thread-126]
> regionserver.StoreFile$Writer(997): Delete Family Bloom filter type for
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57:
> CompoundBloomFilterWriter
> 2012-03-27 04:36:12,632 INFO [Thread-126]
> regionserver.StoreFile$Writer(1220): NO General Bloom and NO DeleteFamily was
> added to HFile
> (/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57)
>
> 2012-03-27 04:36:12,632 INFO [Thread-126] regionserver.Store(770): Flushed ,
> sequenceid=7934, memsize=1.9k, into tmp file
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
> 2012-03-27 04:36:12,632 DEBUG [Thread-126] regionserver.Store(795): Renaming
> flushed file at
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/.tmp/61954619003e469baf1a34be5ff2ec57
> to
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469baf1a34be5ff2ec57
> 2012-03-27 04:36:12,634 INFO [Thread-126] regionserver.Store(818): Added
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/b9091c3c-961e-4035-850a-83ad14d517cc/TestAtomicOperationtestMultiRowMutationMultiThreads/testtable/7cd30e219714cfc5e91f69def66e7f81/colfamily11/61954619003e469
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in "backup mode"
[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420097#comment-13420097
]
Lars Hofhansl commented on HBASE-5547:
--
Hmm... From the failed run:
{quote}
Results :
Failed tests:
testMultipleTables(org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient):
Archived HFiles should have gotten deleted, but didn't
testRowMutationMultiThreads(org.apache.hadoop.hbase.regionserver.TestAtomicOperation):
expected:<0> but was:<5>
Tests in error:
testMultiRowMutationMultiThreads(org.apache.hadoop.hbase.regionserver.TestAtomicOperation):
java.io.FileNotFoundException:
(Too many open files)
{quote}
The first one is suspicious. Could you have a look at it Jesse when you get a
chance.
> Don't delete HFiles when in "backup mode"
> -
>
> Key: HBASE-5547
> URL: https://issues.apache.org/jira/browse/HBASE-5547
> Project: HBase
> Issue Type: New Feature
>Reporter: Lars Hofhansl
>Assignee: Jesse Yates
> Fix For: 0.96.0, 0.94.2
>
> Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch,
> hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547_v13.patch,
> java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch,
> java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch,
> java_HBASE-5547_v7.patch
>
>
> This came up in a discussion I had with Stack.
> It would be nice if HBase could be notified that a backup is in progress (via
> a znode for example) and in that case either:
> 1. rename HFiles to be delete to .bck
> 2. rename the HFiles into a special directory
> 3. rename them to a general trash directory (which would not need to be tied
> to backup mode).
> That way it should be able to get a consistent backup based on HFiles (HDFS
> snapshots or hard links would be better options here, but we do not have
> those).
> #1 makes cleanup a bit harder.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in "backup mode"
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420095#comment-13420095 ] Hudson commented on HBASE-5547: --- Integrated in HBase-TRUNK #3158 (See [https://builds.apache.org/job/HBase-TRUNK/3158/]) HBASE-5547 Don't delete HFiles when in "backup mode" (Jesse Yates) (Revision 1364203) Result = FAILURE larsh : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/BaseConfigurable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/Chore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveTableMonitor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/LongTermArchivingHFileCleaner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/TableHFileArchiveTracker.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/ZKTableArchiveClient.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleaner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleanerDelegate.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/TimeToLiveLogCleaner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveHFileCleaner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveLogCleaner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java * /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestLogsCleaner.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/CheckedArchivingHFileCleaner.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHFileArchiveUtil.java
[jira] [Commented] (HBASE-6406) TestReplicationPeer.testResetZooKeeperSession and TestZooKeeper.testClientSessionExpired fail frequently
[ https://issues.apache.org/jira/browse/HBASE-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420094#comment-13420094 ] Hudson commented on HBASE-6406: --- Integrated in HBase-TRUNK #3158 (See [https://builds.apache.org/job/HBase-TRUNK/3158/]) HBASE-6406 Disable TestZooKeeper#testClientSessionExpired (Revision 1364204) Result = FAILURE larsh : Files : * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java > TestReplicationPeer.testResetZooKeeperSession and > TestZooKeeper.testClientSessionExpired fail frequently > > > Key: HBASE-6406 > URL: https://issues.apache.org/jira/browse/HBASE-6406 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.1 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 6406.txt, testReplication.jstack, testZooKeeper.jstack > > > Looking back through the 0.94 test runs these two tests accounted for 11 of > 34 failed tests. > They should be fixed or (temporarily) disabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6406) TestReplicationPeer.testResetZooKeeperSession and TestZooKeeper.testClientSessionExpired fail frequently
[ https://issues.apache.org/jira/browse/HBASE-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420090#comment-13420090 ] Hudson commented on HBASE-6406: --- Integrated in HBase-0.94 #351 (See [https://builds.apache.org/job/HBase-0.94/351/]) HBASE-6406 Disable TestZooKeeper#testClientSessionExpired (Revision 1364207) Result = SUCCESS larsh : Files : * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java > TestReplicationPeer.testResetZooKeeperSession and > TestZooKeeper.testClientSessionExpired fail frequently > > > Key: HBASE-6406 > URL: https://issues.apache.org/jira/browse/HBASE-6406 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.1 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 6406.txt, testReplication.jstack, testZooKeeper.jstack > > > Looking back through the 0.94 test runs these two tests accounted for 11 of > 34 failed tests. > They should be fixed or (temporarily) disabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6406) TestReplicationPeer.testResetZooKeeperSession and TestZooKeeper.testClientSessionExpired fail frequently
[ https://issues.apache.org/jira/browse/HBASE-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-6406. -- Resolution: Fixed Also disabled TestZooKeeper#testClientSessionExpired > TestReplicationPeer.testResetZooKeeperSession and > TestZooKeeper.testClientSessionExpired fail frequently > > > Key: HBASE-6406 > URL: https://issues.apache.org/jira/browse/HBASE-6406 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.1 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 6406.txt, testReplication.jstack, testZooKeeper.jstack > > > Looking back through the 0.94 test runs these two tests accounted for 11 of > 34 failed tests. > They should be fixed or (temporarily) disabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5547) Don't delete HFiles when in "backup mode"
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5547: - Fix Version/s: 0.96.0 Committed to 0.96. Phfewww. Thanks for the patch and the persistence Jesse! Leaving open for the 0.94 part. > Don't delete HFiles when in "backup mode" > - > > Key: HBASE-5547 > URL: https://issues.apache.org/jira/browse/HBASE-5547 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Jesse Yates > Fix For: 0.96.0, 0.94.2 > > Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, > hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547_v13.patch, > java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, > java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, > java_HBASE-5547_v7.patch > > > This came up in a discussion I had with Stack. > It would be nice if HBase could be notified that a backup is in progress (via > a znode for example) and in that case either: > 1. rename HFiles to be delete to .bck > 2. rename the HFiles into a special directory > 3. rename them to a general trash directory (which would not need to be tied > to backup mode). > That way it should be able to get a consistent backup based on HFiles (HDFS > snapshots or hard links would be better options here, but we do not have > those). > #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4255) Expose CatalogJanitor controls
[ https://issues.apache.org/jira/browse/HBASE-4255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419972#comment-13419972 ] Hudson commented on HBASE-4255: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #102 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/102/]) HBASE-4255 Expose CatalogJanitor controls (Revision 1364127) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/MasterAdminProtocol.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterStatusServlet.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterAdminProtos.java * /hbase/trunk/hbase-server/src/main/protobuf/MasterAdmin.proto * /hbase/trunk/hbase-server/src/main/ruby/hbase/admin.rb * /hbase/trunk/hbase-server/src/main/ruby/shell.rb * /hbase/trunk/hbase-server/src/main/ruby/shell/commands/catalogjanitor_enabled.rb * /hbase/trunk/hbase-server/src/main/ruby/shell/commands/catalogjanitor_run.rb * /hbase/trunk/hbase-server/src/main/ruby/shell/commands/catalogjanitor_switch.rb > Expose CatalogJanitor controls > -- > > Key: HBASE-4255 > URL: https://issues.apache.org/jira/browse/HBASE-4255 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.90.4 >Reporter: Jean-Daniel Cryans >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: 4255-4.2.patch, 4255-5.1.patch > > > When doing surgery or other operational tasks, it's nice to be able to have > the .META. table quickly cleaned of split parents. The CatalogJanitor already > has controls baked in (currently used in unit tests), I think we should > expose this the same way we do with the balancer, that is: > - start > - stop > - request a run > A client would need to go through HBaseAdmin, and shell commands need to be > created. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6363) HBaseConfiguration can carry a main method that dumps XML output for debug purposes
[ https://issues.apache.org/jira/browse/HBASE-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419973#comment-13419973 ] Hudson commented on HBASE-6363: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #102 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/102/]) HBASE-6363 HBaseConfiguration can carry a main method that dumps XML output for debug purposes (Revision 1364129) Result = FAILURE stack : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java > HBaseConfiguration can carry a main method that dumps XML output for debug > purposes > --- > > Key: HBASE-6363 > URL: https://issues.apache.org/jira/browse/HBASE-6363 > Project: HBase > Issue Type: Improvement > Components: util >Affects Versions: 0.94.0 >Reporter: Harsh J >Priority: Trivial > Labels: conf, newbie, noob > Fix For: 0.96.0, 0.94.1 > > Attachments: HBASE-6363.2.patch, HBASE-6363.patch > > > Just like the Configuration class carries a main() method in it, that simply > loads itself and writes XML out to System.out, HBaseConfiguration can use the > same kinda method. > That way we can do "hbase org.apache.hadoop.….HBaseConfiguration" to get an > Xml dump of things HBaseConfiguration has properly loaded. Nifty in checking > app classpaths sometimes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6381) AssignmentManager should use the same logic for clean startup and failover
[ https://issues.apache.org/jira/browse/HBASE-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419951#comment-13419951 ] Jimmy Xiang commented on HBASE-6381: @Ram, currently, I was thinking about three changes. I need to make sure they are good. 1. for the failover case, let SSH take case of those dead region servers so that we can share some code instead of doing some similar things in AM and SSH. 2. SSH is enabled so that we can handle meta/root region server failure before joinCluster is completed. However, we can hold SSH a little bit for the user region assignments. So in AM, we can avoid failoverProcessedRegions which is a little bit confusing. 3. those enablingTables and disablingTables, they should be some local variables. We can load them from ZKTable at the beginning instead of handling them per table. > AssignmentManager should use the same logic for clean startup and failover > -- > > Key: HBASE-6381 > URL: https://issues.apache.org/jira/browse/HBASE-6381 > Project: HBase > Issue Type: Bug > Components: master >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > > Currently AssignmentManager handles clean startup and failover very > differently. > Different logic is mingled together so it is hard to find out which is for > which. > We should clean it up and share the same logic so that AssignmentManager > handles > both cases the same way. This way, the code will much easier to understand > and > maintain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6363) HBaseConfiguration can carry a main method that dumps XML output for debug purposes
[ https://issues.apache.org/jira/browse/HBASE-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419905#comment-13419905 ] Hudson commented on HBASE-6363: --- Integrated in HBase-0.94 #350 (See [https://builds.apache.org/job/HBase-0.94/350/]) HBASE-6363 HBaseConfiguration can carry a main method that dumps XML output for debug purposes (Revision 1364131) Result = FAILURE stack : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java > HBaseConfiguration can carry a main method that dumps XML output for debug > purposes > --- > > Key: HBASE-6363 > URL: https://issues.apache.org/jira/browse/HBASE-6363 > Project: HBase > Issue Type: Improvement > Components: util >Affects Versions: 0.94.0 >Reporter: Harsh J >Priority: Trivial > Labels: conf, newbie, noob > Fix For: 0.96.0, 0.94.1 > > Attachments: HBASE-6363.2.patch, HBASE-6363.patch > > > Just like the Configuration class carries a main() method in it, that simply > loads itself and writes XML out to System.out, HBaseConfiguration can use the > same kinda method. > That way we can do "hbase org.apache.hadoop.….HBaseConfiguration" to get an > Xml dump of things HBaseConfiguration has properly loaded. Nifty in checking > app classpaths sometimes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4255) Expose CatalogJanitor controls
[ https://issues.apache.org/jira/browse/HBASE-4255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419903#comment-13419903 ] Hudson commented on HBASE-4255: --- Integrated in HBase-TRUNK #3157 (See [https://builds.apache.org/job/HBase-TRUNK/3157/]) HBASE-4255 Expose CatalogJanitor controls (Revision 1364127) Result = SUCCESS stack : Files : * /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/MasterAdminProtocol.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterStatusServlet.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterAdminProtos.java * /hbase/trunk/hbase-server/src/main/protobuf/MasterAdmin.proto * /hbase/trunk/hbase-server/src/main/ruby/hbase/admin.rb * /hbase/trunk/hbase-server/src/main/ruby/shell.rb * /hbase/trunk/hbase-server/src/main/ruby/shell/commands/catalogjanitor_enabled.rb * /hbase/trunk/hbase-server/src/main/ruby/shell/commands/catalogjanitor_run.rb * /hbase/trunk/hbase-server/src/main/ruby/shell/commands/catalogjanitor_switch.rb > Expose CatalogJanitor controls > -- > > Key: HBASE-4255 > URL: https://issues.apache.org/jira/browse/HBASE-4255 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.90.4 >Reporter: Jean-Daniel Cryans >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: 4255-4.2.patch, 4255-5.1.patch > > > When doing surgery or other operational tasks, it's nice to be able to have > the .META. table quickly cleaned of split parents. The CatalogJanitor already > has controls baked in (currently used in unit tests), I think we should > expose this the same way we do with the balancer, that is: > - start > - stop > - request a run > A client would need to go through HBaseAdmin, and shell commands need to be > created. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6363) HBaseConfiguration can carry a main method that dumps XML output for debug purposes
[ https://issues.apache.org/jira/browse/HBASE-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419904#comment-13419904 ] Hudson commented on HBASE-6363: --- Integrated in HBase-TRUNK #3157 (See [https://builds.apache.org/job/HBase-TRUNK/3157/]) HBASE-6363 HBaseConfiguration can carry a main method that dumps XML output for debug purposes (Revision 1364129) Result = SUCCESS stack : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java > HBaseConfiguration can carry a main method that dumps XML output for debug > purposes > --- > > Key: HBASE-6363 > URL: https://issues.apache.org/jira/browse/HBASE-6363 > Project: HBase > Issue Type: Improvement > Components: util >Affects Versions: 0.94.0 >Reporter: Harsh J >Priority: Trivial > Labels: conf, newbie, noob > Fix For: 0.96.0, 0.94.1 > > Attachments: HBASE-6363.2.patch, HBASE-6363.patch > > > Just like the Configuration class carries a main() method in it, that simply > loads itself and writes XML out to System.out, HBaseConfiguration can use the > same kinda method. > That way we can do "hbase org.apache.hadoop.….HBaseConfiguration" to get an > Xml dump of things HBaseConfiguration has properly loaded. Nifty in checking > app classpaths sometimes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6363) HBaseConfiguration can carry a main method that dumps XML output for debug purposes
[ https://issues.apache.org/jira/browse/HBASE-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-6363. -- Resolution: Fixed Fix Version/s: 0.94.1 0.96.0 Release Note: Adds a main to HBaseConfiguration to dump out configs. Committed to trunk and 0.94 branch. Thanks for the patch lads. > HBaseConfiguration can carry a main method that dumps XML output for debug > purposes > --- > > Key: HBASE-6363 > URL: https://issues.apache.org/jira/browse/HBASE-6363 > Project: HBase > Issue Type: Improvement > Components: util >Affects Versions: 0.94.0 >Reporter: Harsh J >Priority: Trivial > Labels: conf, newbie, noob > Fix For: 0.96.0, 0.94.1 > > Attachments: HBASE-6363.2.patch, HBASE-6363.patch > > > Just like the Configuration class carries a main() method in it, that simply > loads itself and writes XML out to System.out, HBaseConfiguration can use the > same kinda method. > That way we can do "hbase org.apache.hadoop.….HBaseConfiguration" to get an > Xml dump of things HBaseConfiguration has properly loaded. Nifty in checking > app classpaths sometimes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4255) Expose CatalogJanitor controls
[ https://issues.apache.org/jira/browse/HBASE-4255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4255: - Resolution: Fixed Release Note: Adds catalogjanitor_enabled, catalogjanitor_run, and catalogjanitor_switch to the shell. Status: Resolved (was: Patch Available) I tried the shell commands locally and seems to work. Committed to trunk. Thanks for the patch Devaraj. > Expose CatalogJanitor controls > -- > > Key: HBASE-4255 > URL: https://issues.apache.org/jira/browse/HBASE-4255 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.90.4 >Reporter: Jean-Daniel Cryans >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: 4255-4.2.patch, 4255-5.1.patch > > > When doing surgery or other operational tasks, it's nice to be able to have > the .META. table quickly cleaned of split parents. The CatalogJanitor already > has controls baked in (currently used in unit tests), I think we should > expose this the same way we do with the balancer, that is: > - start > - stop > - request a run > A client would need to go through HBaseAdmin, and shell commands need to be > created. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6435) Reading WAL files after a recovery leads to time lost in HDFS timeouts when using dead datanodes
[ https://issues.apache.org/jira/browse/HBASE-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419871#comment-13419871 ] stack commented on HBASE-6435: -- @Todd Given suggested Interface, how we map from an hbase session expiration to a Replica? What if the DN died but RS didn't? Won't the fact that DFSClient under the wraps is banging its head timingout against a dead DN -- once per DFSInputStream -- be hidden from the RS since its being handled down in DFSClient? Don't we need more knowledge on DFSClient workings than suggested API exposes if we are to avoid dead DNs? If we do figure we have a bad DN, do we then per open DFSInputStream iterate updating priorities? > Reading WAL files after a recovery leads to time lost in HDFS timeouts when > using dead datanodes > > > Key: HBASE-6435 > URL: https://issues.apache.org/jira/browse/HBASE-6435 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Affects Versions: 0.96.0 >Reporter: nkeywal >Assignee: nkeywal > Attachments: 6435.unfinished.patch > > > HBase writes a Write-Ahead-Log to revover from hardware failure. > This log is written with 'append' on hdfs. > Through ZooKeeper, HBase gets informed usually in 30s that it should start > the recovery process. > This means reading the Write-Ahead-Log to replay the edits on the other > servers. > In standards deployments, HBase process (regionserver) are deployed on the > same box as the datanodes. > It means that when the box stops, we've actually lost one of the edits, as we > lost both the regionserver and the datanode. > As HDFS marks a node as dead after ~10 minutes, it appears as available when > we try to read the blocks to recover. As such, we are delaying the recovery > process by 60 seconds as the read will usually fail with a socket timeout. If > the file is still opened for writing, it adds an extra 20s + a risk of losing > edits if we connect with ipc to the dead DN. > Possible solutions are: > - shorter dead datanodes detection by the NN. Requires a NN code change. > - better dead datanodes management in DFSClient. Requires a DFS code change. > - NN customisation to write the WAL files on another DN instead of the local > one. > - reordering the blocks returned by the NN on the client side to put the > blocks on the same DN as the dead RS at the end of the priority queue. > Requires a DFS code change or a kind of workaround. > The solution retained is the last one. Compared to what was discussed on the > mailing list, the proposed patch will not modify HDFS source code but adds a > proxy. This for two reasons: > - Some HDFS functions managing block orders are static > (MD5MD5CRC32FileChecksum). Implementing the hook in the DFSClient would > require to implement partially the fix, change the DFS interface to make this > function non static, or put the hook static. None of these solution is very > clean. > - Adding a proxy allows to put all the code in HBase, simplifying dependency > management. > Nevertheless, it would be better to have this in HDFS. But this solution > allows to target the last version only, and this could allow minimal > interface changes such as non static methods. > Moreover, writing the blocks to the non local DN would be an even better > solution long term. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6437) Avoid admin.balance during master initialize
[
https://issues.apache.org/jira/browse/HBASE-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419852#comment-13419852
]
Zhihong Ted Yu commented on HBASE-6437:
---
Balancer is started in finishInitialization() toward the end of initialization:
{code}
status.setStatus("Starting balancer and catalog janitor");
this.balancerChore = getAndStartBalancerChore(this);
this.catalogJanitorChore = new CatalogJanitor(this, this);
startCatalogJanitorChore();
registerMBean();
}
status.markComplete("Initialization successful");
{code}
> Avoid admin.balance during master initialize
>
>
> Key: HBASE-6437
> URL: https://issues.apache.org/jira/browse/HBASE-6437
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.2, 0.96.0, 0.94.2
>
>
> In HBASE-5850 many of the admin operations have been blocked till the master
> initializes. But the balancer is not. So this JIRA is to extend the
> PleaseHoldException in case of admin.balance() call before master is
> initialized.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit
[
https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419834#comment-13419834
]
Zhihong Ted Yu commented on HBASE-6429:
---
Year is not needed in license header of FilterWrapper.java and
TestFilterWithScanLimits.java
{code}
+ // Wrap the following function calls
+ @Override
+ public void filterRow(List kvs) {
{code}
Please change the above javadoc so that it explains the rationale behind this
JIRA.
{code}
+ if(scan.hasFilter()){
{code}
Space between if and (.
{code}
+ "Filter with filterRow(List) or filterRow()
incompatible with scan with limit!");
{code}
Line length limit is 100 characters.
In the test:
{code}
+} catch (IOException e) {
+ // TODO Auto-generated catch block
+ e.printStackTrace();
{code}
I think we should fail the test above.
There're other TODO's in test code which needs handling.
Please run your next patch through unit tests before attaching.
Thanks
> Filter with filterRow() returning true is incompatible with scan with limit
> ---
>
> Key: HBASE-6429
> URL: https://issues.apache.org/jira/browse/HBASE-6429
> Project: HBase
> Issue Type: Bug
> Components: filters
>Affects Versions: 0.96.0
>Reporter: Jason Dai
> Attachments: hbase-6429-trunk.patch, hbase-6429_0_94_0.patch
>
>
> Currently if we scan with bot limit and a Filter with
> filterRow(List) implemented, an IncompatibleFilterException will
> be thrown. The same exception should also be thrown if the filer has its
> filterRow() implemented.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit
[ https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6429: - Attachment: hbase-6429-trunk.patch Thank you very much, Ted. Here updates that trunk patch file. More comment, please let me know. > Filter with filterRow() returning true is incompatible with scan with limit > --- > > Key: HBASE-6429 > URL: https://issues.apache.org/jira/browse/HBASE-6429 > Project: HBase > Issue Type: Bug > Components: filters >Affects Versions: 0.96.0 >Reporter: Jason Dai > Attachments: hbase-6429-trunk.patch, hbase-6429_0_94_0.patch > > > Currently if we scan with bot limit and a Filter with > filterRow(List) implemented, an IncompatibleFilterException will > be thrown. The same exception should also be thrown if the filer has its > filterRow() implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit
[ https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6429: - Attachment: (was: hbase-6429-trunk.patch) > Filter with filterRow() returning true is incompatible with scan with limit > --- > > Key: HBASE-6429 > URL: https://issues.apache.org/jira/browse/HBASE-6429 > Project: HBase > Issue Type: Bug > Components: filters >Affects Versions: 0.96.0 >Reporter: Jason Dai > Attachments: hbase-6429_0_94_0.patch > > > Currently if we scan with bot limit and a Filter with > filterRow(List) implemented, an IncompatibleFilterException will > be thrown. The same exception should also be thrown if the filer has its > filterRow() implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6438) RegionAlreadyInTransitionException needs to give more info to avoid assignment inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-6438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan reassigned HBASE-6438: - Assignee: ramkrishna.s.vasudevan > RegionAlreadyInTransitionException needs to give more info to avoid > assignment inconsistencies > -- > > Key: HBASE-6438 > URL: https://issues.apache.org/jira/browse/HBASE-6438 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > > Seeing some of the recent issues in region assignment, > RegionAlreadyInTransitionException is one reason after which the region > assignment may or may not happen(in the sense we need to wait for the TM to > assign). > In HBASE-6317 we got one problem due to RegionAlreadyInTransitionException on > master restart. > Consider the following case, due to some reason like master restart or > external assign call, we try to assign a region that is already getting > opened in a RS. > Now the next call to assign has already changed the state of the znode and so > the current assign that is going on the RS is affected and it fails. The > second assignment that started also fails getting RAITE exception. Finally > both assignments not carrying on. Idea is to find whether any such RAITE > exception can be retried or not. > Here again we have following cases like where > -> The znode is yet to transitioned from OFFLINE to OPENING in RS > -> RS may be in the step of openRegion. > -> RS may be trying to transition OPENING to OPENED. > -> RS is yet to add to online regions in the RS side. > Here in openRegion() and updateMeta() any failures we are moving the znode to > FAILED_OPEN. So in these cases getting an RAITE should be ok. But in other > cases the assignment is stopped. > The idea is to just add the current state of the region assignment in the RIT > map in the RS side and using that info we can determine whether the > assignment can be retried or not on getting an RAITE. > Considering the current work going on in AM, pls do share if this is needed > atleast in the 0.92/0.94 versions? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6438) RegionAlreadyInTransitionException needs to give more info to avoid assignment inconsistencies
ramkrishna.s.vasudevan created HBASE-6438: - Summary: RegionAlreadyInTransitionException needs to give more info to avoid assignment inconsistencies Key: HBASE-6438 URL: https://issues.apache.org/jira/browse/HBASE-6438 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Seeing some of the recent issues in region assignment, RegionAlreadyInTransitionException is one reason after which the region assignment may or may not happen(in the sense we need to wait for the TM to assign). In HBASE-6317 we got one problem due to RegionAlreadyInTransitionException on master restart. Consider the following case, due to some reason like master restart or external assign call, we try to assign a region that is already getting opened in a RS. Now the next call to assign has already changed the state of the znode and so the current assign that is going on the RS is affected and it fails. The second assignment that started also fails getting RAITE exception. Finally both assignments not carrying on. Idea is to find whether any such RAITE exception can be retried or not. Here again we have following cases like where -> The znode is yet to transitioned from OFFLINE to OPENING in RS -> RS may be in the step of openRegion. -> RS may be trying to transition OPENING to OPENED. -> RS is yet to add to online regions in the RS side. Here in openRegion() and updateMeta() any failures we are moving the znode to FAILED_OPEN. So in these cases getting an RAITE should be ok. But in other cases the assignment is stopped. The idea is to just add the current state of the region assignment in the RIT map in the RS side and using that info we can determine whether the assignment can be retried or not on getting an RAITE. Considering the current work going on in AM, pls do share if this is needed atleast in the 0.92/0.94 versions? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6437) Avoid admin.balance during master initialize
ramkrishna.s.vasudevan created HBASE-6437: - Summary: Avoid admin.balance during master initialize Key: HBASE-6437 URL: https://issues.apache.org/jira/browse/HBASE-6437 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.92.2, 0.96.0, 0.94.2 In HBASE-5850 many of the admin operations have been blocked till the master initializes. But the balancer is not. So this JIRA is to extend the PleaseHoldException in case of admin.balance() call before master is initialized. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6381) AssignmentManager should use the same logic for clean startup and failover
[ https://issues.apache.org/jira/browse/HBASE-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419799#comment-13419799 ] ramkrishna.s.vasudevan commented on HBASE-6381: --- @Jimmy Just curious to know.. Can you tel what idea you have to solve this? > AssignmentManager should use the same logic for clean startup and failover > -- > > Key: HBASE-6381 > URL: https://issues.apache.org/jira/browse/HBASE-6381 > Project: HBase > Issue Type: Bug > Components: master >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > > Currently AssignmentManager handles clean startup and failover very > differently. > Different logic is mingled together so it is hard to find out which is for > which. > We should clean it up and share the same logic so that AssignmentManager > handles > both cases the same way. This way, the code will much easier to understand > and > maintain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6272) In-memory region state is inconsistent
[ https://issues.apache.org/jira/browse/HBASE-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419797#comment-13419797 ] ramkrishna.s.vasudevan commented on HBASE-6272: --- Some comments put up on RB. > In-memory region state is inconsistent > -- > > Key: HBASE-6272 > URL: https://issues.apache.org/jira/browse/HBASE-6272 > Project: HBase > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > > AssignmentManger stores region state related information in several places: > regionsInTransition, regions (region info to server name map), and servers > (server name to region info set map). However the access to these places is > not coordinated properly. It leads to inconsistent in-memory region state > information. Sometimes, some region could even be offline, and not in > transition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6401) HBase may lose edits after a crash if used with HDFS 1.0.3 or older
[ https://issues.apache.org/jira/browse/HBASE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419787#comment-13419787 ] stack commented on HBASE-6401: -- @Nkeywal Sounds great. A patch/JIRA will give us something to rally around. > HBase may lose edits after a crash if used with HDFS 1.0.3 or older > --- > > Key: HBASE-6401 > URL: https://issues.apache.org/jira/browse/HBASE-6401 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.96.0 > Environment: all >Reporter: nkeywal >Priority: Critical > Attachments: TestReadAppendWithDeadDN.java > > > This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the > hdfs jira for this. > Context: HBase Write Ahead Log features. This is using hdfs append. If the > node crashes, the file that was written is read by other processes to replay > the action. > - So we have in hdfs one (dead) process writing with another process reading. > - But, despite the call to syncFs, we don't always see the data when we have > a dead node. It seems to be because the call in DFSClient#updateBlockInfo > ignores the ipc errors and set the length to 0. > - So we may miss all the writes to the last block if we try to connect to the > dead DN. > hdfs 1.0.3, branch-1 or branch-1-win: we have the issue > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java?revision=1359853&view=markup > hdfs branch-2 or trunk: we should not have the issue (but not tested) > http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java?view=markup > The attached test will fail ~50 of the time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6406) TestReplicationPeer.testResetZooKeeperSession and TestZooKeeper.testClientSessionExpired fail frequently
[ https://issues.apache.org/jira/browse/HBASE-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419778#comment-13419778 ] stack commented on HBASE-6406: -- Disable the test for now I'd say Lars. Make an issue to examine whats up but get it out of the way of 0.94.1 I'd say. > TestReplicationPeer.testResetZooKeeperSession and > TestZooKeeper.testClientSessionExpired fail frequently > > > Key: HBASE-6406 > URL: https://issues.apache.org/jira/browse/HBASE-6406 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.1 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 6406.txt, testReplication.jstack, testZooKeeper.jstack > > > Looking back through the 0.94 test runs these two tests accounted for 11 of > 34 failed tests. > They should be fixed or (temporarily) disabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in "backup mode"
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419756#comment-13419756 ] Jesse Yates commented on HBASE-5547: bq. I find that deleteRegion() was addressed, but not deleteTable(). In the deletetablehandler, it first goes and deletes each region (which gets replaced with archiveRegion) and then once all the regions have been archived, we call deleteTable, which basically just "rm -rf"'s the entire directory. TL;DR - hfiles on table delete are archived, but not any more information about the state of the table. Wrote a unit test to check that too, so should be good. > Don't delete HFiles when in "backup mode" > - > > Key: HBASE-5547 > URL: https://issues.apache.org/jira/browse/HBASE-5547 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Jesse Yates > Fix For: 0.94.2 > > Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, > hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547_v13.patch, > java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, > java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, > java_HBASE-5547_v7.patch > > > This came up in a discussion I had with Stack. > It would be nice if HBase could be notified that a backup is in progress (via > a znode for example) and in that case either: > 1. rename HFiles to be delete to .bck > 2. rename the HFiles into a special directory > 3. rename them to a general trash directory (which would not need to be tied > to backup mode). > That way it should be able to get a consistent backup based on HFiles (HDFS > snapshots or hard links would be better options here, but we do not have > those). > #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
