[jira] [Created] (HBASE-19563) A few hbase-procedure classes missing @InterfaceAudience annotation
Guanghao Zhang created HBASE-19563: -- Summary: A few hbase-procedure classes missing @InterfaceAudience annotation Key: HBASE-19563 URL: https://issues.apache.org/jira/browse/HBASE-19563 Project: HBase Issue Type: Bug Components: proc-v2 Reporter: Guanghao Zhang Priority: Minor NoopProcedureStore.java ProcedureStoreBase.java ProcedureMetrics.java LockStatus.java LockAndQueue.java ProcedureStateSerializer.java -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19562) Purge mirror writing of region and table info into fs at .tableinfo and .regioninfo
[ https://issues.apache.org/jira/browse/HBASE-19562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297967#comment-16297967 ] Zach York commented on HBASE-19562: --- [~stack] I know you have some misgivings on the approach for HBASE-18840, but the current approach relies on reading regioninfo's from the filesystem to construct an initial meta table. If we are removing these files, is it possible to recover a meta table using just the storefiles (no meta table though)? I'd be happy to get rid of these files if there is a better way to do this :) I'll also use this opportunity to give that JIRA a ping :) > Purge mirror writing of region and table info into fs at .tableinfo and > .regioninfo > --- > > Key: HBASE-19562 > URL: https://issues.apache.org/jira/browse/HBASE-19562 > Project: HBase > Issue Type: Bug > Components: fs >Reporter: stack >Assignee: stack > Fix For: 2.0.0-beta-2 > > Attachments: > 0002-HBASE-19562-Purge-mirror-writing-of-region-and-table.patch > > > We don't use these files in hbase2 yet we keep writing them when we create a > table or region. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19553) Old replica regions should be cleared from AM memory after primary region split or merge
[ https://issues.apache.org/jira/browse/HBASE-19553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297964#comment-16297964 ] Pankaj Kumar commented on HBASE-19553: -- Ok [~huaxiang], we can wait until HBASE-18625 committed, will upload the modified patch after that. > Old replica regions should be cleared from AM memory after primary region > split or merge > > > Key: HBASE-19553 > URL: https://issues.apache.org/jira/browse/HBASE-19553 > Project: HBase > Issue Type: Bug > Components: master >Reporter: huaxiang sun >Assignee: Pankaj Kumar >Priority: Minor > Fix For: 1.5.0 > > Attachments: HBASE-19553-branch-1-v2.patch, HBASE-19553-branch-1.patch > > > Similar to HBASE-18025, the replica parent's info is not removed from master. > Actually I think it can be removed after replica region is split or merged, I > will check the logic and apply one patch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-13428) Migration to hbase-2.0.0
[ https://issues.apache.org/jira/browse/HBASE-13428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297963#comment-16297963 ] stack commented on HBASE-13428: --- Wrote data w/ 0.98 w/ ACL enabled. Added some grants. Stopped and started branch-2 over the 0.98 data. Seems to be the tail of hfile only. Running again w/ a closer eye on it > Migration to hbase-2.0.0 > > > Key: HBASE-13428 > URL: https://issues.apache.org/jira/browse/HBASE-13428 > Project: HBase > Issue Type: Umbrella > Components: migration >Reporter: stack >Priority: Blocker > Fix For: 2.0.0-beta-1 > > > Opening a 2.0 umbrella migration issue. Lets hang off this one any tools and > expectations migrating from 1.0 (or earlier) to 2.0. So far there are none > that I know of though there is an expectation in HBASE-13373 that hfiles are > at least major version 2 and minor version 3. Lets list all such > expectations, etc., here. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19562) Purge mirror writing of region and table info into fs at .tableinfo and .regioninfo
[ https://issues.apache.org/jira/browse/HBASE-19562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297959#comment-16297959 ] stack commented on HBASE-19562: --- Should probably up the fs version number when we do this. > Purge mirror writing of region and table info into fs at .tableinfo and > .regioninfo > --- > > Key: HBASE-19562 > URL: https://issues.apache.org/jira/browse/HBASE-19562 > Project: HBase > Issue Type: Bug > Components: fs >Reporter: stack >Assignee: stack > Fix For: 2.0.0-beta-2 > > Attachments: > 0002-HBASE-19562-Purge-mirror-writing-of-region-and-table.patch > > > We don't use these files in hbase2 yet we keep writing them when we create a > table or region. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19562) Purge mirror writing of region and table info into fs at .tableinfo and .regioninfo
[ https://issues.apache.org/jira/browse/HBASE-19562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19562: -- Attachment: 0002-HBASE-19562-Purge-mirror-writing-of-region-and-table.patch Can remove a bunch of crud. Patch is not finished. Need to come back to this. > Purge mirror writing of region and table info into fs at .tableinfo and > .regioninfo > --- > > Key: HBASE-19562 > URL: https://issues.apache.org/jira/browse/HBASE-19562 > Project: HBase > Issue Type: Bug > Components: fs >Reporter: stack >Assignee: stack > Fix For: 2.0.0-beta-2 > > Attachments: > 0002-HBASE-19562-Purge-mirror-writing-of-region-and-table.patch > > > We don't use these files in hbase2 yet we keep writing them when we create a > table or region. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19491) Exclude flaky tests from nightly master run
[ https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297946#comment-16297946 ] Appy commented on HBASE-19491: -- For some reason, findbugs on root is not working, but running on individual modules is working. For time being, going with latter approach. {noformat} # When using MODULES=(.) findbugs detection: HBASE-19491 [Wed Dec 20 06:05:19 UTC 2017 INFO]: Personality: branch findbugs {noformat} {noformat} # When using CHANGED_MODULES findbugs detection: HBASE-19491 [Wed Dec 20 02:48:26 UTC 2017 INFO]: Personality: branch findbugs cd /testptch/hbase/hbase-build-support/hbase-error-prone /usr/share/maven/bin/mvn test-compile findbugs:findbugs -DskipTests=true > /testptch/patchprocess/branch-findbugs-hbase-build-support_hbase-error-prone.txt 2>&1 Elapsed: 0m 14s cd /testptch/hbase/hbase-protocol-shaded /usr/share/maven/bin/mvn test-compile findbugs:findbugs -DskipTests=true > /testptch/patchprocess/branch-findbugs-hbase-protocol-shaded.txt 2>&1 Elapsed: 1m 54s cd /testptch/hbase/hbase-common /usr/share/maven/bin/mvn test-compile findbugs:findbugs -DskipTests=true > /testptch/patchprocess/branch-findbugs-hbase-common.txt 2>&1 Elapsed: 0m 32s cd /testptch/hbase/hbase-metrics-api /usr/share/maven/bin/mvn test-compile findbugs:findbugs -DskipTests=true > /testptch/patchprocess/branch-findbugs-hbase-metrics-api.txt 2>&1 Elapsed: 0m 15s cd /testptch/hbase/hbase-hadoop-compat /usr/share/maven/bin/mvn test-compile findbugs:findbugs -DskipTests=true > /testptch/patchprocess/branch-findbugs-hbase-hadoop-compat.txt 2>&1 Elapsed: 0m 15s cd /testptch/hbase/hbase-metrics /usr/share/maven/bin/mvn test-compile findbugs:findbugs -DskipTests=true > /testptch/patchprocess/branch-findbugs-hbase-metrics.txt 2>&1 Elapsed: 0m 16s cd /testptch/hbase/hbase-hadoop2-compat /usr/share/maven/bin/mvn test-compile findbugs:findbugs -DskipTests=true > /testptch/patchprocess/branch-findbugs-hbase-hadoop2-compat.txt 2>&1 Elapsed: 0m 19s cd /testptch/hbase/hbase-protocol /usr/share/maven/bin/mvn test-compile findbugs:findbugs -DskipTests=true > /testptch/patchprocess/branch-findbugs-hbase-protocol.txt 2>&1 Elapsed: 1m 7s cd /testptch/hbase/hbase-client /usr/share/maven/bin/mvn test-compile findbugs:findbugs -DskipTests=true > /testptch/patchprocess/branch-findbugs-hbase-client.txt 2>&1 Elapsed: 0m 49s cd /testptch/hbase/hbase-zookeeper /usr/share/maven/bin/mvn test-compile findbugs:findbugs -DskipTests=true > /testptch/patchprocess/branch-findbugs-hbase-zookeeper.txt 2>&1 Elapsed: 0m 23s cd /testptch/hbase/hbase-replication /usr/share/maven/bin/mvn test-compile findbugs:findbugs -DskipTests=true > /testptch/patchprocess/branch-findbugs-hbase-replication.txt 2>&1 Elapsed: 0m 19s cd /testptch/hbase/hbase-http /usr/share/maven/bin/mvn test-compile findbugs:findbugs -DskipTests=true > /testptch/patchprocess/branch-findbugs-hbase-http.txt 2>&1 Elapsed: 0m 23s cd /testptch/hbase/hbase-procedure /usr/share/maven/bin/mvn test-compile findbugs:findbugs -DskipTests=true > /testptch/patchprocess/branch-findbugs-hbase-procedure.txt 2>&1 Elapsed: 0m 21s cd /testptch/hbase/hbase-server /usr/share/maven/bin/mvn test-compile findbugs:findbugs -DskipTests=true > /testptch/patchprocess/branch-findbugs-hbase-server.txt 2>&1 Elapsed: 1m 58s cd /testptch/hbase/hbase-mapreduce /usr/share/maven/bin/mvn test-compile findbugs:findbugs -DskipTests=true > /testptch/patchprocess/branch-findbugs-hbase-mapreduce.txt 2>&1 Elapsed: 0m 33s cd /testptch/hbase/hbase-thrift /usr/share/maven/bin/mvn test-compile findbugs:findbugs -DskipTests=true > /testptch/patchprocess/branch-findbugs-hbase-thrift.txt 2>&1 Elapsed: 0m 58s cd /testptch/hbase/hbase-rsgroup /usr/share/maven/bin/mvn test-compile findbugs:findbugs -DskipTests=true > /testptch/patchprocess/branch-findbugs-hbase-rsgroup.txt 2>&1 Elapsed: 0m 33s cd /testptch/hbase/hbase-endpoint /usr/share/maven/bin/mvn test-compile findbugs:findbugs -DskipTests=true > /testptch/patchprocess/branch-findbugs-hbase-endpoint.txt 2>&1 Elapsed: 0m 32s cd /testptch/hbase/hbase-backup /usr/share/maven/bin/mvn test-compile findbugs:f
[jira] [Created] (HBASE-19562) Purge mirror writing of region and table info into fs at .tableinfo and .regioninfo
stack created HBASE-19562: - Summary: Purge mirror writing of region and table info into fs at .tableinfo and .regioninfo Key: HBASE-19562 URL: https://issues.apache.org/jira/browse/HBASE-19562 Project: HBase Issue Type: Bug Components: fs Reporter: stack Assignee: stack Fix For: 2.0.0-beta-2 We don't use these files in hbase2 yet we keep writing them when we create a table or region. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19561) maxCacheSize in CacheEvictionStats can't be accumulated repeatedly When dealing with each region
[ https://issues.apache.org/jira/browse/HBASE-19561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated HBASE-19561: -- Status: Patch Available (was: Open) > maxCacheSize in CacheEvictionStats can't be accumulated repeatedly When > dealing with each region > > > Key: HBASE-19561 > URL: https://issues.apache.org/jira/browse/HBASE-19561 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Attachments: HBASE-19561.master.001.patch > > > {code:title=HRegionServer.java|borderStyle=solid} > public CacheEvictionStats clearRegionBlockCache(Region region) { > BlockCache blockCache = this.getCacheConfig().getBlockCache(); > long evictedBlocks = 0; > for(Store store : region.getStores()) { > for(StoreFile hFile : store.getStorefiles()) { > evictedBlocks += > blockCache.evictBlocksByHfileName(hFile.getPath().getName()); > } > } > return CacheEvictionStats.builder() > .withEvictedBlocks(evictedBlocks) > .withMaxCacheSize(blockCache.getMaxSize()) > .build(); > } > {code} > BlockCache is the regionserver level object.If we accumulate this value for > each region, we will get a wrong result. > The value of maxCacheSize should be set to BlockCache.getMaxSize() after > processing all the regions on this regionserver. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19364) Truncate_preserve fails with table when replica region > 1
[ https://issues.apache.org/jira/browse/HBASE-19364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297936#comment-16297936 ] Pankaj Kumar commented on HBASE-19364: -- Hi [~huaxiang], HBASE-17319 applicable for other branches (brnach-1.3/1.2) as well. Any plan to commit HBASE-17319 in other branches, this Jira is still open. > Truncate_preserve fails with table when replica region > 1 > -- > > Key: HBASE-19364 > URL: https://issues.apache.org/jira/browse/HBASE-19364 > Project: HBase > Issue Type: Bug > Components: master >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 1.5.0 > > Attachments: HBASE-19364-branch-1.patch > > > Root cause is same as HBASE-17319, here we need to exclude secondary regions > while reading meta. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19468) FNFE during scans and flushes
[ https://issues.apache.org/jira/browse/HBASE-19468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-19468: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to branch-1, 1.4, branch-2 and master. For branch-1 and branch-1.4 ensured the patch applies cleanly and there are no compile issues. Thanks for reporting this issue and appreciate all the careful reviews here. > FNFE during scans and flushes > - > > Key: HBASE-19468 > URL: https://issues.apache.org/jira/browse/HBASE-19468 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 1.3.1 >Reporter: Thiruvel Thirumoolan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0, 1.4.1, 1.5.0, 1.3.3 > > Attachments: HBASE-19468-poc.patch, HBASE-19468_1.4.patch, > HBASE-19468_master.patch > > > We see FNFE exceptions on our 1.3 clusters when scans and flushes happen at > the same time. This causes regionserver to throw a UnknownScannerException > and client retries. > This happens during the following sequence: > 1. Scanner open, client fetched some rows from regionserver and working on it > 2. Flush happens and storeScanner is updated with flushed files > (StoreScanner.updateReaders()) > 3. Compaction happens on the region while scanner is still open > 4. compaction discharger runs and cleans up the newly flushed file as we > don't have new scanners on it yet. > 5. Client issues scan.next and during StoreScanner.resetScannerStack(), we > get a FNFE. RegionServer throws a UnknownScannerThe client retries in 1.3. > With branch-1.4, the scan fails with a DoNotRetryIOException. > [~ram_krish], My proposal is to increment the reader count during > updateReaders() and decrement it during resetScannerStack(), so discharger > doesn't clean it up. Scan lease expiries also have to be taken care of. Am I > missing anything? Is there a better approach? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HBASE-19468) FNFE during scans and flushes
[ https://issues.apache.org/jira/browse/HBASE-19468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan reassigned HBASE-19468: -- Assignee: ramkrishna.s.vasudevan > FNFE during scans and flushes > - > > Key: HBASE-19468 > URL: https://issues.apache.org/jira/browse/HBASE-19468 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 1.3.1 >Reporter: Thiruvel Thirumoolan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0, 1.4.1, 1.5.0, 1.3.3 > > Attachments: HBASE-19468-poc.patch, HBASE-19468_1.4.patch, > HBASE-19468_master.patch > > > We see FNFE exceptions on our 1.3 clusters when scans and flushes happen at > the same time. This causes regionserver to throw a UnknownScannerException > and client retries. > This happens during the following sequence: > 1. Scanner open, client fetched some rows from regionserver and working on it > 2. Flush happens and storeScanner is updated with flushed files > (StoreScanner.updateReaders()) > 3. Compaction happens on the region while scanner is still open > 4. compaction discharger runs and cleans up the newly flushed file as we > don't have new scanners on it yet. > 5. Client issues scan.next and during StoreScanner.resetScannerStack(), we > get a FNFE. RegionServer throws a UnknownScannerThe client retries in 1.3. > With branch-1.4, the scan fails with a DoNotRetryIOException. > [~ram_krish], My proposal is to increment the reader count during > updateReaders() and decrement it during resetScannerStack(), so discharger > doesn't clean it up. Scan lease expiries also have to be taken care of. Am I > missing anything? Is there a better approach? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19116) Currently the tail of hfiles with CellComparator* classname makes it so hbase1 can't open hbase2 written hfiles; fix
[ https://issues.apache.org/jira/browse/HBASE-19116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297916#comment-16297916 ] ramkrishna.s.vasudevan commented on HBASE-19116: That's silly. Problem is here {code} else if (comparatorClassName.equals("org.apache.hadoop.hbase.KeyValue.RawBytesComparator") {code} It should have been {code} else if (comparatorClassName.equals("org.apache.hadoop.hbase.KeyValue$RawBytesComparator") {code} The '.' should be '$'. :( > Currently the tail of hfiles with CellComparator* classname makes it so > hbase1 can't open hbase2 written hfiles; fix > > > Key: HBASE-19116 > URL: https://issues.apache.org/jira/browse/HBASE-19116 > Project: HBase > Issue Type: Sub-task > Components: HFile, migration >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-2 > > > See tail of HBASE-19052 for discussion which concludes we should try and make > it so operators do not have to go to latest hbase version before they > upgrade, at least if we can avoid it. > The necessary change of our default comparator from KV to Cell naming has > hfiles with tails that have the classname CellComparator in them in place of > KeyValueComparator. If an hbase1 tries to open them, it will fail not having > a CellComparator in its classpath (We have name of comparator in tail because > different files require different comparators... perhaps we write an alias > instead of a class one day... TODO). HBASE-16189 and HBASE-19052 are about > trying to carry knowledge of hbase2 back to hbase1, a brittle approach making > it so operators will have to upgrade to the latest branch-1 before they can > go to hbase2. > This issue is about undoing our writing of an incompatible (to hbase1) tail, > not unless we really have to (and it sounds like we could do without writing > an incompatible tail) to see if we can avoid requiring operators go to > lastest branch-1 (we may end up needing this but lets a have a really good > reason for it if we do). > Oh, let this filing be an answer to our [~anoop.hbase]'s old high-level > question over in HBASE-16189: > bq. ...means when rolling upgrade done to 2.0, first users have to upgrade to > some 1.x versions which is having this fix and then to 2.0.. What do you guys > think Whether we should avoid this kind of indirection? cc Enis Soztutar, > Stack, Ted Yu, Matteo Bertozzi > Yeah, lets try to avoid this if we can... -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19483) Add proper privilege check for rsgroup commands
[ https://issues.apache.org/jira/browse/HBASE-19483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297917#comment-16297917 ] Hadoop QA commented on HBASE-19483: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 51s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 43s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 43s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 8m 22s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 46s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 6s{color} | {color:green} hbase-server: The patch generated 0 new + 81 unchanged - 11 fixed = 81 total (was 92) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} The patch hbase-rsgroup passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 21s{color} | {color:green} root: The patch generated 0 new + 100 unchanged - 11 fixed = 100 total (was 111) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 44s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 19m 41s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 45s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}140m 38s{color} | {color:red} root in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 21s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}202m 37s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.regionserver.TestEncryptionKeyRotation | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-19483 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12902961/HBASE-19483.master.005.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 924a24f46550 3.13.0-13
[jira] [Updated] (HBASE-19148) Reevaluate default values of configurations
[ https://issues.apache.org/jira/browse/HBASE-19148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19148: -- Attachment: HBASE-19148.master.003.patch Retry > Reevaluate default values of configurations > --- > > Key: HBASE-19148 > URL: https://issues.apache.org/jira/browse/HBASE-19148 > Project: HBase > Issue Type: Bug > Components: defaults >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0-beta-1 > > Attachments: > 0002-HBASE-19148-Reevaluate-default-values-of-configurati.patch, > HBASE-19148.master.001.patch, HBASE-19148.master.002.patch, > HBASE-19148.master.003.patch, HBASE-19148.master.003.patch > > > Remove cruft and mythologies. Make descriptions more digestible. Change > defaults given experience. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19148) Reevaluate default values of configurations
[ https://issues.apache.org/jira/browse/HBASE-19148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297912#comment-16297912 ] stack commented on HBASE-19148: --- Go these timeouts: {code} [ERROR] Please refer to /testptch/hbase/hbase-server/target/surefire-reports for the individual test results. [ERROR] Please refer to dump files (if any exist) [date]-jvmRun[N].dump, [date].dumpstream and [date]-jvmRun[N].dumpstream. [ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called? [ERROR] Command was /bin/sh -c cd /testptch/hbase/hbase-server && /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -enableassertions -Dhbase.build.id=2017-12-20T00:56:00Z -Xmx2800m -Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -jar /testptch/hbase/hbase-server/target/surefire/surefirebooter7794492890783134322.jar /testptch/hbase/hbase-server/target/surefire 2017-12-20T00-56-37_947-jvmRun2 surefire8203219361689656729tmp surefire_8856831918730071656587tmp [ERROR] Error occurred in starting fork, check output in log [ERROR] Process Exit Code: 1 [ERROR] Crashed tests: [ERROR] org.apache.hadoop.hbase.TestAcidGuaranteesWithNoInMemCompaction [ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called? [ERROR] Command was /bin/sh -c cd /testptch/hbase/hbase-server && /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -enableassertions -Dhbase.build.id=2017-12-20T00:56:00Z -Xmx2800m -Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -jar /testptch/hbase/hbase-server/target/surefire/surefirebooter1002995583245243176.jar /testptch/hbase/hbase-server/target/surefire 2017-12-20T00-56-37_947-jvmRun4 surefire4934381225274999644tmp surefire_8896607335703885400109tmp [ERROR] Error occurred in starting fork, check output in log [ERROR] Process Exit Code: 1 [ERROR] Crashed tests: [ERROR] org.apache.hadoop.hbase.TestAcidGuaranteesWithAdaptivePolicy [ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called? [ERROR] Command was /bin/sh -c cd /testptch/hbase/hbase-server && /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -enableassertions -Dhbase.build.id=2017-12-20T00:56:00Z -Xmx2800m -Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -jar /testptch/hbase/hbase-server/target/surefire/surefirebooter7893607961903870294.jar /testptch/hbase/hbase-server/target/surefire 2017-12-20T00-56-37_947-jvmRun1 surefire928706360139890990tmp surefire_9064920373703367929366tmp [ERROR] Error occurred in starting fork, check output in log [ERROR] Process Exit Code: 1 [ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called? [ERROR] Command was /bin/sh -c cd /testptch/hbase/hbase-server && /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -enableassertions -Dhbase.build.id=2017-12-20T00:56:00Z -Xmx2800m -Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -jar /testptch/hbase/hbase-server/target/surefire/surefirebooter8109115696455571252.jar /testptch/hbase/hbase-server/target/surefire 2017-12-20T00-56-37_947-jvmRun5 surefire531917999007925890tmp surefire_9073220887283439304467tmp [ERROR] Error occurred in starting fork, check output in log [ERROR] Process Exit Code: 1 [ERROR] Crashed tests: [ERROR] org.apache.hadoop.hbase.tool.TestCanaryTool [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called? [ERROR] Command was /bin/sh -c cd /testptch/hbase/hbase-server && /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -enableassertions -Dhbase.build.id=2017-12-20T00:56:00Z -Xmx2800m -Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -jar /testptch/hbase/hbase-server/target/surefire/surefirebooter7794492890783134322.jar /testptch/hbase/hbase-server/target/surefire 2017-12-20T00-56-37_947-jvmRun2 surefire8203219361689656729tmp surefire_8856831918730071656587tmp [ERROR] Error occurred in starting fork, check output in log {code} I tried the above locally w/ the patch in place and they seem fine. Let me retry. > Reevaluate default values of configurations > --- > > Key: HBASE-19148 > URL: https://issues.apache.org/jira/browse/HBASE-19148 > Project: HBase > Issue Type: Bug > Components: defaults >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0-beta-1 > > Attachments: > 0002-HBASE-19148-Reevaluate-default-values-of-configurat
[jira] [Assigned] (HBASE-19520) Add UTs for the new lock type PEER
[ https://issues.apache.org/jira/browse/HBASE-19520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang reassigned HBASE-19520: -- Assignee: Guanghao Zhang > Add UTs for the new lock type PEER > -- > > Key: HBASE-19520 > URL: https://issues.apache.org/jira/browse/HBASE-19520 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Guanghao Zhang > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18625) Splitting of region with replica, doesn't update region list in serverHolding. A server crash leads to overlap.
[ https://issues.apache.org/jira/browse/HBASE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297907#comment-16297907 ] ramkrishna.s.vasudevan commented on HBASE-18625: I thought after RS restart is also needed. But since this RegionState is in Master I think clearing there is good enough. Sorry for the noise. Patch LGTM. +1 > Splitting of region with replica, doesn't update region list in > serverHolding. A server crash leads to overlap. > --- > > Key: HBASE-18625 > URL: https://issues.apache.org/jira/browse/HBASE-18625 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.6 >Reporter: Igloo >Assignee: huaxiang sun > Fix For: 1.2.8 > > Attachments: HBASE-18625-branch-1-v001.patch, > HBASE-18625-branch-1-v002.patch > > > The situation can appear in following steps in release hbase1.2.6 > 1. create 'testtable', 'info', {REGION_REPLICATION=>2} > 2. write somerecords into 'testtable' > 3. split the table 'testtable' > 4. after the spliting, the serverHoldings in RegionStates still holds the > regioninfo for the replica of parent region > 5. restart the regionserver where the parent replica-region located > 6. the offlined replica of parent region will be assigned in > ServerCrashProcedure. > hbase hbck 'testtable‘ > ERROR: Region { meta => null, hdfs => null, deployed => > qabb-qa-hdp-hbase1,16020,1503022958093;testtable,,1503022907686_0001.42d11cfe195b3cc4d08b2c078a687f6d > ., replicaId => 1 } not in META, but deployed on > qabb-qa-hdp-hbase1,16020,1503022958093 > 18 ERROR: No regioninfo in Meta or HDFS. { meta => null, hdfs => null, > deployed => > qabb-qa-hdp-hbase1,16020,1503022958093;testtable,,1503022907686_0001.42d11cfe >195b3cc4d08b2c078a687f6d., replicaId => 1 } -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19116) Currently the tail of hfiles with CellComparator* classname makes it so hbase1 can't open hbase2 written hfiles; fix
[ https://issues.apache.org/jira/browse/HBASE-19116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297906#comment-16297906 ] stack commented on HBASE-19116: --- Just tried a branch-2 starting over data written by an 0.98. This is the exception to fix: {code} 382603 2017-12-19 21:25:08,333 DEBUG [StoreOpener-8e6e2b71d8a413561d9103b0cb6e4686-1] regionserver.HStore: loaded hdfs://ve0524.halxg.cloudera.com:8020/hbase/data/default/ycsb/8e6e2b71d8a413561d9103b0cb6e4686/family/3633fe2e03ad4cceb0d2018f1f0d76da, isReference=false, isBulkLoadResult=false, seqid=341101 6, majorCompaction=false 382604 2017-12-19 21:25:08,339 DEBUG [StoreFileOpenerThread-family-1] hfile.HFile: Opening HFile v2 with v3 reader 382605 2017-12-19 21:25:08,342 ERROR [StoreFileOpenerThread-family-1] regionserver.StoreFileReader: Error reading bloom filter meta for GENERAL_BLOOM_META -- proceeding without 382606 java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.KeyValue$RawBytesComparator 382607 at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.getComparatorClass(FixedFileTrailer.java:583) 382608 at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:592) 382609 at org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.(CompoundBloomFilter.java:79) 382610 at org.apache.hadoop.hbase.util.BloomFilterFactory.createFromMeta(BloomFilterFactory.java:104) 382611 at org.apache.hadoop.hbase.regionserver.StoreFileReader.loadBloomfilter(StoreFileReader.java:480) 382612 at org.apache.hadoop.hbase.regionserver.HStoreFile.open(HStoreFile.java:426) 382613 at org.apache.hadoop.hbase.regionserver.HStoreFile.initReader(HStoreFile.java:461) 382614 at org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:665) 382615 at org.apache.hadoop.hbase.regionserver.HStore.lambda$openStoreFiles$0(HStore.java:533) 382616 at java.util.concurrent.FutureTask.run(FutureTask.java:266) 382617 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 382618 at java.util.concurrent.FutureTask.run(FutureTask.java:266) 382619 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 382620 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 382621 at java.lang.Thread.run(Thread.java:745) 382622 Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.KeyValue$RawBytesComparator 382623 at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 382624 at java.lang.ClassLoader.loadClass(ClassLoader.java:424) 382625 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) 382626 at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 382627 at java.lang.Class.forName0(Native Method) 382628 at java.lang.Class.forName(Class.java:264) 382629 at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.getComparatorClass(FixedFileTrailer.java:581) 382630 ... 14 more {code} > Currently the tail of hfiles with CellComparator* classname makes it so > hbase1 can't open hbase2 written hfiles; fix > > > Key: HBASE-19116 > URL: https://issues.apache.org/jira/browse/HBASE-19116 > Project: HBase > Issue Type: Sub-task > Components: HFile, migration >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-2 > > > See tail of HBASE-19052 for discussion which concludes we should try and make > it so operators do not have to go to latest hbase version before they > upgrade, at least if we can avoid it. > The necessary change of our default comparator from KV to Cell naming has > hfiles with tails that have the classname CellComparator in them in place of > KeyValueComparator. If an hbase1 tries to open them, it will fail not having > a CellComparator in its classpath (We have name of comparator in tail because > different files require different comparators... perhaps we write an alias > instead of a class one day... TODO). HBASE-16189 and HBASE-19052 are about > trying to carry knowledge of hbase2 back to hbase1, a brittle approach making > it so operators will have to upgrade to the latest branch-1 before they can > go to hbase2. > This issue is about undoing our writing of an incompatible (to hbase1) tail, > not unless we really have to (and it sounds like we could do without writing > an incompatible tail) to see if we can avoid requiring operators go to > lastest branch-1 (we may end up needing this but lets a have a really good > reason for it if we do). > Oh, let this filing be an answer to our [~anoop.hbase]'s old high-level > question over in HBASE-16189: > bq. ...means when rolling upgrade done to 2.0, f
[jira] [Commented] (HBASE-19558) TestRegionsOnMasterOptions hack so it works reliablly
[ https://issues.apache.org/jira/browse/HBASE-19558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297874#comment-16297874 ] Hudson commented on HBASE-19558: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4257 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4257/]) HBASE-19558 TestRegionsOnMasterOptions hack so it works reliably (stack: rev 3c78ad52157576641ee6b42e874b5015b792f34d) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRegionsOnMasterOptions.java > TestRegionsOnMasterOptions hack so it works reliablly > - > > Key: HBASE-19558 > URL: https://issues.apache.org/jira/browse/HBASE-19558 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0-beta-1 > > > See tail of HBASE-19533 for description of problem; our shutdown of a cluster > needs work. Meantime hack the test so it doesn't hang on close so it passes > reliably again. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19556) Remove TestAssignmentManager#testGoodSplit, which no longer make sense
[ https://issues.apache.org/jira/browse/HBASE-19556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297872#comment-16297872 ] Hudson commented on HBASE-19556: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4257 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4257/]) HBASE-19556: Remove TestAssignmentManager#testGoodSplit, which no longer (stack: rev 70608acf289f6de4aeb8f159667a1f21397ec433) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestAssignmentManager.java > Remove TestAssignmentManager#testGoodSplit, which no longer make sense > -- > > Key: HBASE-19556 > URL: https://issues.apache.org/jira/browse/HBASE-19556 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Yi Liang >Assignee: Yi Liang >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19556-master-v1.patch > > > {quote} > [ERROR] > testGoodSplit(org.apache.hadoop.hbase.master.assignment.TestAssignmentManager) > Time elapsed: 0.478 s <<< ERROR! > java.io.IOException: 5a50732f7cb3a05dd3a297bacbc34943 NOT splittable > {quote} > GoodSplitExecutor can only mock some functions in RSProcedureDispatcher, > however, to test split, we need to create(or mock) a real table on region > server side. And GoodSplitExecutor can not mock those kinds of function. > And, similar test has already been covered in TestSplitTableRegionProcedure, > we no longer need to test it again in TestAssignmentManager -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15482) Provide an option to skip calculating block locations for SnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297873#comment-16297873 ] Hudson commented on HBASE-15482: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4257 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4257/]) HBASE-15482 Provide an option to skip calculating block locations for (tedyu: rev 5e7d16a3ceaeec5057474f9bae2d40d306f6dd8e) * (edit) hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatTestBase.java * (edit) hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapred/TestTableSnapshotInputFormat.java * (edit) hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSnapshotInputFormat.java * (edit) hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatImpl.java > Provide an option to skip calculating block locations for SnapshotInputFormat > - > > Key: HBASE-15482 > URL: https://issues.apache.org/jira/browse/HBASE-15482 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Liyin Tang >Assignee: Xiang Li >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: 15482.v3.txt, HBASE-15482.master.000.patch, > HBASE-15482.master.001.patch, HBASE-15482.master.002.patch, > HBASE-15482.master.003.patch > > > When a MR job is reading from SnapshotInputFormat, it needs to calculate the > splits based on the block locations in order to get best locality. However, > this process may take a long time for large snapshots. > In some setup, the computing layer, Spark, Hive or Presto could run out side > of HBase cluster. In these scenarios, the block locality doesn't matter. > Therefore, it will be great to have an option to skip calculating the block > locations for every job. That will super useful for the Hive/Presto/Spark > connectors. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19494) Create simple WALKey filter that can be plugged in on the Replication Sink
[ https://issues.apache.org/jira/browse/HBASE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297871#comment-16297871 ] Hudson commented on HBASE-19494: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4257 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4257/]) HBASE-19494 Create simple WALKey filter that can be plugged in on the (stack: rev 32f6fd41c274a955338b7c43ab80309b9adbba0d) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestWALEntrySinkFilter.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALEntrySinkFilter.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/WALEntryFilter.java > Create simple WALKey filter that can be plugged in on the Replication Sink > -- > > Key: HBASE-19494 > URL: https://issues.apache.org/jira/browse/HBASE-19494 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: stack >Assignee: stack > Fix For: 2.0.0-beta-1 > > Attachments: > 0003-HBASE-19494-Create-simple-WALKey-filter-that-can-be-.patch, > HBASE-19494.master.001.patch > > > hbase-indexer used to look at WALKeys on the sink to see if their time of > creation was before the time at which the replication stream was enabled. > In the parent redo, there is no means for doing this anymore (because WALKey > used to be Private and because to get at the WALKey in the Sink, you had to > override all of the Replication which meant importing a million Private > objects...). > This issue is about adding a simple filter to Replication on the sink-side > that just takes a WALKey (now InterfaceAudience LimitedPrivate and recently > made read-only). > Assigned myself. Need to do this so hbase-indexer can move to hbase2. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19561) maxCacheSize in CacheEvictionStats can't be accumulated repeatedly When dealing with each region
[ https://issues.apache.org/jira/browse/HBASE-19561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297865#comment-16297865 ] Ted Yu commented on HBASE-19561: lgtm > maxCacheSize in CacheEvictionStats can't be accumulated repeatedly When > dealing with each region > > > Key: HBASE-19561 > URL: https://issues.apache.org/jira/browse/HBASE-19561 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Attachments: HBASE-19561.master.001.patch > > > {code:title=HRegionServer.java|borderStyle=solid} > public CacheEvictionStats clearRegionBlockCache(Region region) { > BlockCache blockCache = this.getCacheConfig().getBlockCache(); > long evictedBlocks = 0; > for(Store store : region.getStores()) { > for(StoreFile hFile : store.getStorefiles()) { > evictedBlocks += > blockCache.evictBlocksByHfileName(hFile.getPath().getName()); > } > } > return CacheEvictionStats.builder() > .withEvictedBlocks(evictedBlocks) > .withMaxCacheSize(blockCache.getMaxSize()) > .build(); > } > {code} > BlockCache is the regionserver level object.If we accumulate this value for > each region, we will get a wrong result. > The value of maxCacheSize should be set to BlockCache.getMaxSize() after > processing all the regions on this regionserver. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19561) maxCacheSize in CacheEvictionStats can't be accumulated repeatedly When dealing with each region
[ https://issues.apache.org/jira/browse/HBASE-19561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297864#comment-16297864 ] Zach York commented on HBASE-19561: --- [~andrewcheng] Good catch! LGTM, +1 > maxCacheSize in CacheEvictionStats can't be accumulated repeatedly When > dealing with each region > > > Key: HBASE-19561 > URL: https://issues.apache.org/jira/browse/HBASE-19561 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Attachments: HBASE-19561.master.001.patch > > > {code:title=HRegionServer.java|borderStyle=solid} > public CacheEvictionStats clearRegionBlockCache(Region region) { > BlockCache blockCache = this.getCacheConfig().getBlockCache(); > long evictedBlocks = 0; > for(Store store : region.getStores()) { > for(StoreFile hFile : store.getStorefiles()) { > evictedBlocks += > blockCache.evictBlocksByHfileName(hFile.getPath().getName()); > } > } > return CacheEvictionStats.builder() > .withEvictedBlocks(evictedBlocks) > .withMaxCacheSize(blockCache.getMaxSize()) > .build(); > } > {code} > BlockCache is the regionserver level object.If we accumulate this value for > each region, we will get a wrong result. > The value of maxCacheSize should be set to BlockCache.getMaxSize() after > processing all the regions on this regionserver. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19561) maxCacheSize in CacheEvictionStats can't be accumulated repeatedly When dealing with each region
[ https://issues.apache.org/jira/browse/HBASE-19561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated HBASE-19561: -- Attachment: HBASE-19561.master.001.patch [~yuzhih...@gmail.com] [~zyork] Can you take a look?Thanks > maxCacheSize in CacheEvictionStats can't be accumulated repeatedly When > dealing with each region > > > Key: HBASE-19561 > URL: https://issues.apache.org/jira/browse/HBASE-19561 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Attachments: HBASE-19561.master.001.patch > > > {code:title=HRegionServer.java|borderStyle=solid} > public CacheEvictionStats clearRegionBlockCache(Region region) { > BlockCache blockCache = this.getCacheConfig().getBlockCache(); > long evictedBlocks = 0; > for(Store store : region.getStores()) { > for(StoreFile hFile : store.getStorefiles()) { > evictedBlocks += > blockCache.evictBlocksByHfileName(hFile.getPath().getName()); > } > } > return CacheEvictionStats.builder() > .withEvictedBlocks(evictedBlocks) > .withMaxCacheSize(blockCache.getMaxSize()) > .build(); > } > {code} > BlockCache is the regionserver level object.If we accumulate this value for > each region, we will get a wrong result. > The value of maxCacheSize should be set to BlockCache.getMaxSize() after > processing all the regions on this regionserver. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19561) maxCacheSize in CacheEvictionStats can't be accumulated repeatedly When dealing with each region
Guangxu Cheng created HBASE-19561: - Summary: maxCacheSize in CacheEvictionStats can't be accumulated repeatedly When dealing with each region Key: HBASE-19561 URL: https://issues.apache.org/jira/browse/HBASE-19561 Project: HBase Issue Type: Bug Affects Versions: 2.0.0 Reporter: Guangxu Cheng Assignee: Guangxu Cheng {code:title=HRegionServer.java|borderStyle=solid} public CacheEvictionStats clearRegionBlockCache(Region region) { BlockCache blockCache = this.getCacheConfig().getBlockCache(); long evictedBlocks = 0; for(Store store : region.getStores()) { for(StoreFile hFile : store.getStorefiles()) { evictedBlocks += blockCache.evictBlocksByHfileName(hFile.getPath().getName()); } } return CacheEvictionStats.builder() .withEvictedBlocks(evictedBlocks) .withMaxCacheSize(blockCache.getMaxSize()) .build(); } {code} BlockCache is the regionserver level object.If we accumulate this value for each region, we will get a wrong result. The value of maxCacheSize should be set to BlockCache.getMaxSize() after processing all the regions on this regionserver. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19547) HBase fails building on AArch64 due to asciidoctor-maven-plugin.
[ https://issues.apache.org/jira/browse/HBASE-19547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297854#comment-16297854 ] Ted Yu commented on HBASE-19547: Several occurrences of the following in output: {code} [ERROR] Command was /bin/sh -c cd /testptch/hbase/hbase-server && /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -enableassertions -Dhbase.build.id=2017-12-19T04:24:11Z -Xmx2800m -Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -jar /testptch/hbase/hbase-server/target/surefire/surefirebooter8863999073719792036.jar /testptch/hbase/hbase-server/target/surefire 2017-12-19T04-24-17_238-jvmRun3 surefire1877980246042989754tmp surefire_11936792857066479530434tmp [ERROR] Error occurred in starting fork, check output in log [ERROR] Process Exit Code: 1 [ERROR] Crashed tests: {code} Please run test suite again. > HBase fails building on AArch64 due to asciidoctor-maven-plugin. > > > Key: HBASE-19547 > URL: https://issues.apache.org/jira/browse/HBASE-19547 > Project: HBase > Issue Type: Bug > Components: build, hbase, jruby >Affects Versions: 3.0.0 >Reporter: Yuqi Gu >Assignee: Yuqi Gu > Attachments: HBASE-19547.patch > > > HBase fails building on AArch64 due to asciidoctor-maven-plugin. > {code:java} > mvn clean site package install -DskipTests > [ERROR] Failed to execute > goal org.asciidoctor:asciidoctor-maven-plugin:1.5.5: > process-asciidoc (output-pdf) on project hbase: > Execution output-pdf of goal > org.asciidoctor:asciidoctor-maven-plugin:1.5.5: > process-asciidoc failed: (NotImplementedError) > fstat unimplemented unsupported or native support failed to load -> > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19560) create make-rc.sh for hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-19560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-19560: -- Fix Version/s: thirdparty-2.0.1 > create make-rc.sh for hbase-thirdparty > -- > > Key: HBASE-19560 > URL: https://issues.apache.org/jira/browse/HBASE-19560 > Project: HBase > Issue Type: Task > Components: hbase-thirdparty >Reporter: Mike Drob > Fix For: thirdparty-2.0.1 > > > Make it easier to automate the process... > Some notes, may not correspond to exact commands I ran... > {noformat} > mvn clean deploy -Papache-release > git tag -s HEAD > git archive HEAD -o release.tar.gz > gpg --sign release.tar.gz > gpg --print-mds > gpg --print-md MD5 > gpg --print-md SHA512 > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19560) create make-rc.sh for hbase-thirdparty
Mike Drob created HBASE-19560: - Summary: create make-rc.sh for hbase-thirdparty Key: HBASE-19560 URL: https://issues.apache.org/jira/browse/HBASE-19560 Project: HBase Issue Type: Task Components: hbase-thirdparty Reporter: Mike Drob Make it easier to automate the process... Some notes, may not correspond to exact commands I ran... {noformat} mvn clean deploy -Papache-release git tag -s HEAD git archive HEAD -o release.tar.gz gpg --sign release.tar.gz gpg --print-mds gpg --print-md MD5 gpg --print-md SHA512 {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19557) Build and release source jars for hbase-shaded-client and others
[ https://issues.apache.org/jira/browse/HBASE-19557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-19557: - Issue Type: Sub-task (was: Improvement) Parent: HBASE-18800 > Build and release source jars for hbase-shaded-client and others > > > Key: HBASE-19557 > URL: https://issues.apache.org/jira/browse/HBASE-19557 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.3.1, 1.2.6 >Reporter: Jerry He > > It seems that currently we don't build and release source jars for > hbase-shaded-client (and server or mapreduce). IDEs will complain from the > dependent users. We should provide them. > http://central.maven.org/maven2/org/apache/hbase/hbase-shaded-client/1.3.1/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-15482) Provide an option to skip calculating block locations for SnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297060#comment-16297060 ] Xiang Li edited comment on HBASE-15482 at 12/20/17 2:56 AM: Hi [~tedyu], [~jerryhe], thanks for your comments and guide! Patch 003 is uploaded to address the following changes mainly: * Simplify the logic in light of 15482.v3.txt. Besides, add the logic to ** Check if numTopsAtMost < 1 (which is invalid) ** Check if top is 1. When it is 1, return top host directly. * Adjust testGetBestLocations() * Change the conf key string from {{hbase.TableSnapshotInputFormat.locality.enable}} into {{hbase.TableSnapshotInputFormat.locality.enabled}}, by using "enabled" instead of "enable", as I see most of the conf key strings are using "enabled" was (Author: water): Hi [~tedyu], [~jerryhe], thanks for your comments and guide! Patch 003 is uploaded to address the following changes mainly: * Simply the logic in light of 15482.v3.txt. Besides, add the logic to ** Check if numTopsAtMost < 1 (which is invalid) ** Check if top is 1. When it is 1, return top host directly. * Adjust testGetBestLocations() * Change the conf key string from {{hbase.TableSnapshotInputFormat.locality.enable}} into {{hbase.TableSnapshotInputFormat.locality.enabled}}, by using "enabled" instead of "enable", as I see most of the conf key strings are using "enabled" > Provide an option to skip calculating block locations for SnapshotInputFormat > - > > Key: HBASE-15482 > URL: https://issues.apache.org/jira/browse/HBASE-15482 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Liyin Tang >Assignee: Xiang Li >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: 15482.v3.txt, HBASE-15482.master.000.patch, > HBASE-15482.master.001.patch, HBASE-15482.master.002.patch, > HBASE-15482.master.003.patch > > > When a MR job is reading from SnapshotInputFormat, it needs to calculate the > splits based on the block locations in order to get best locality. However, > this process may take a long time for large snapshots. > In some setup, the computing layer, Spark, Hive or Presto could run out side > of HBase cluster. In these scenarios, the block locality doesn't matter. > Therefore, it will be great to have an option to skip calculating the block > locations for every job. That will super useful for the Hive/Presto/Spark > connectors. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-15482) Provide an option to skip calculating block locations for SnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297084#comment-16297084 ] Xiang Li edited comment on HBASE-15482 at 12/20/17 2:56 AM: Regarding the UT: * For mapred, the number of splits generated(=10) is exactly the same as numRegions specified(=10), while only 8 of them have locations not being an empty array. * For mapreduce, only 8 splits are generated when there are 10 regions. was (Author: water): Regarding the UT: * For mapred, the number of splits generated(=10) is exactly the same as numRegions specified(=10), while only 8 of them have locations not being an empty array. * For mapreduce, only 8 splits are generated when there is 10 regions. > Provide an option to skip calculating block locations for SnapshotInputFormat > - > > Key: HBASE-15482 > URL: https://issues.apache.org/jira/browse/HBASE-15482 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Liyin Tang >Assignee: Xiang Li >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: 15482.v3.txt, HBASE-15482.master.000.patch, > HBASE-15482.master.001.patch, HBASE-15482.master.002.patch, > HBASE-15482.master.003.patch > > > When a MR job is reading from SnapshotInputFormat, it needs to calculate the > splits based on the block locations in order to get best locality. However, > this process may take a long time for large snapshots. > In some setup, the computing layer, Spark, Hive or Presto could run out side > of HBase cluster. In these scenarios, the block locality doesn't matter. > Therefore, it will be great to have an option to skip calculating the block > locations for every job. That will super useful for the Hive/Presto/Spark > connectors. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19218) Master stuck thinking hbase:namespace is assigned after restart preventing intialization
[ https://issues.apache.org/jira/browse/HBASE-19218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297826#comment-16297826 ] Mike Drob commented on HBASE-19218: --- I was chatting with our [~esteban] about this earlier and it sounded like he's seen this outside of RSGroups, possibly related to regions on master? > Master stuck thinking hbase:namespace is assigned after restart preventing > intialization > > > Key: HBASE-19218 > URL: https://issues.apache.org/jira/browse/HBASE-19218 > Project: HBase > Issue Type: Bug >Reporter: Josh Elser >Assignee: Umesh Agashe >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: > hbase-hbase-master-ctr-e134-1499953498516-282290-01-03.hwx.site.log.zip, > hbase-site.xml > > > Our [~romil.choksi] brought this one to my attention after trying to get some > cluster tests running. > The Master seems to have gotten stuck never initializing after it thinks that > hbase:namespace was already deployed on the cluster when it actually was not. > On a Master restart, it reads the location out of meta and assumes that it's > there (I assume this invalid entry is the issue): > {noformat} > 2017-11-08 00:29:17,556 INFO > [ctr-e134-1499953498516-282290-01-03:2.masterManager] > assignment.RegionStateStore: Load hbase:meta entry region={ENCODED => > f147f204a579b885c351bdc0a7ebbf94, NAME => > 'hbase:namespace,,1510084256045.f147f204a579b885c351bdc0a7ebbf94.', STARTKEY > => '', ENDKEY => ''} regionState=OPENING > lastHost=ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728 > regionLocation=ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510100695534 > {noformat} > Prior to this, the RS5 went through the ServerCrashProcedure, but it looks > like this bailed out unexpectedly: > {noformat} > 2017-11-08 00:25:25,187 WARN > [ctr-e134-1499953498516-282290-01-03:2.masterManager] > master.ServerManager: Expiration of > ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728 but > server not online > 2017-11-08 00:25:25,187 INFO [ProcExecWrkr-5] > procedure.ServerCrashProcedure: Start pid=36, > state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure > server=ctr-e134-1499953498516-282290-01-03.hwx.site,16020,1510084580111, > splitWal=t > rue, meta=false > 2017-11-08 00:25:25,188 INFO > [ctr-e134-1499953498516-282290-01-03:2.masterManager] > master.ServerManager: Processing expiration of > ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728 on > ctr-e134-1499953498516-28 > 2290-01-03.hwx.site,2,1510100690324 > ... > 2017-11-08 00:25:27,211 ERROR [ProcExecWrkr-22] procedure2.ProcedureExecutor: > CODE-BUG: Uncaught runtime exception: pid=40, ppid=37, > state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure > table=hbase:namespace, region=f147f204a579b885c351bdc0a7ebbf94 > java.lang.NullPointerException > at > java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) > at > org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher.addOperationToNode(RemoteProcedureDispatcher.java:171) > at > org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.addToRemoteDispatcher(RegionTransitionProcedure.java:223) > at > org.apache.hadoop.hbase.master.assignment.AssignProcedure.updateTransition(AssignProcedure.java:252) > at > org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:309) > at > org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:82) > at > org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1452) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1221) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:77) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1731) > 2017-11-08 00:25:27,239 FATAL [ProcExecWrkr-22] procedure2.ProcedureExecutor: > CODE-BUG: Uncaught runtime exception for pid=37, > state=FAILED:SERVER_CRASH_FINISH, exception=java.lang.NullPointerException > via CODE-BUG: Uncaught runtime exception: pid=40, ppid=37, > state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure > table=hbase:namespace, > region=f147f204a579b885c351bdc0a7ebbf94:java.lang.NullPointerException; > ServerCrashProcedure > server=ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728, > splitWal=
[jira] [Commented] (HBASE-19148) Reevaluate default values of configurations
[ https://issues.apache.org/jira/browse/HBASE-19148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297825#comment-16297825 ] Hadoop QA commented on HBASE-19148: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 24s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 5s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 35s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m 19s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 10s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 9s{color} | {color:red} hbase-server: The patch generated 1 new + 262 unchanged - 0 fixed = 263 total (was 262) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 45s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 20m 5s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 11s{color} | {color:green} hbase-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}118m 53s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 35s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}164m 25s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-19148 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12902950/HBASE-19148.master.003.patch | | Optional Tests | asflicense javac javadoc unit xml findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 9c9330bfb1a3 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-
[jira] [Comment Edited] (HBASE-15482) Provide an option to skip calculating block locations for SnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297060#comment-16297060 ] Xiang Li edited comment on HBASE-15482 at 12/20/17 2:50 AM: Hi [~tedyu], [~jerryhe], thanks for your comments and guide! Patch 003 is uploaded to address the following changes mainly: * Simply the logic in light of 15482.v3.txt. Besides, add the logic to ** Check if numTopsAtMost < 1 (which is invalid) ** Check if top is 1. When it is 1, return top host directly. * Adjust testGetBestLocations() * Change the conf key string from {{hbase.TableSnapshotInputFormat.locality.enable}} into {{hbase.TableSnapshotInputFormat.locality.enabled}}, by using "enabled" instead of "enable", as I see most of the conf key strings are using "enabled" was (Author: water): Hi [~tedyu], [~jerryhe], thanks for your comments and guide! Patch 003 is uploaded to address the following changes mainly: * Simple the logic in light of 15482.v3.txt. Besides, add the logic to ** Check if numTopsAtMost < 1 (which is invalid) ** Check if top is 1. When it is 1, return top host directly. * Adjust testGetBestLocations() * Change the conf key string from {{hbase.TableSnapshotInputFormat.locality.enable}} into {{hbase.TableSnapshotInputFormat.locality.enabled}}, by using "enabled" instead of "enable", as I see most of the conf key strings are using "enabled" > Provide an option to skip calculating block locations for SnapshotInputFormat > - > > Key: HBASE-15482 > URL: https://issues.apache.org/jira/browse/HBASE-15482 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Liyin Tang >Assignee: Xiang Li >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: 15482.v3.txt, HBASE-15482.master.000.patch, > HBASE-15482.master.001.patch, HBASE-15482.master.002.patch, > HBASE-15482.master.003.patch > > > When a MR job is reading from SnapshotInputFormat, it needs to calculate the > splits based on the block locations in order to get best locality. However, > this process may take a long time for large snapshots. > In some setup, the computing layer, Spark, Hive or Presto could run out side > of HBase cluster. In these scenarios, the block locality doesn't matter. > Therefore, it will be great to have an option to skip calculating the block > locations for every job. That will super useful for the Hive/Presto/Spark > connectors. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18970) The version of jruby we use now can't get interactive input from prompt
[ https://issues.apache.org/jira/browse/HBASE-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-18970: -- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed, thanks for the patch, [~avirmani]! > The version of jruby we use now can't get interactive input from prompt > --- > > Key: HBASE-18970 > URL: https://issues.apache.org/jira/browse/HBASE-18970 > Project: HBase > Issue Type: Task > Components: shell >Reporter: Chia-Ping Tsai >Assignee: AMIT VIRMANI >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBase-18970.branch-2.001.patch, samplecmd.txt, > test_shell-Output.txt, test_shell.sh > > > case 1: press enter > {code} > hbase(main):002:0> disable_all 'chia(.*)' > chia_1 > > > Disable the above 1 tables (y/n)? > y^M > {code} > case 2: press ctrl-j > {code} > hbase(main):001:0> disable_all 'chia(.*)' > chia_1 > > > Disable the above 1 tables (y/n)? > y^J1 tables successfully disabled > Took 5.0059 seconds > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19559) Fix TestLogRolling.testLogRollOnDatanodeDeath
[ https://issues.apache.org/jira/browse/HBASE-19559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19559: -- Component/s: wal test > Fix TestLogRolling.testLogRollOnDatanodeDeath > - > > Key: HBASE-19559 > URL: https://issues.apache.org/jira/browse/HBASE-19559 > Project: HBase > Issue Type: Bug > Components: test, wal >Reporter: Duo Zhang > > It appears several times on the failing list of pre commit result. > The test code is a bit confusing to me. Need to dig more. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19483) Add proper privilege check for rsgroup commands
[ https://issues.apache.org/jira/browse/HBASE-19483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297805#comment-16297805 ] Ted Yu commented on HBASE-19483: [~appy]: Can you take a look ? > Add proper privilege check for rsgroup commands > --- > > Key: HBASE-19483 > URL: https://issues.apache.org/jira/browse/HBASE-19483 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Guangxu Cheng > Fix For: 1.4.1, 1.5.0, 2.0.0-beta-2 > > Attachments: HBASE-19483.master.001.patch, > HBASE-19483.master.002.patch, HBASE-19483.master.003.patch, > HBASE-19483.master.004.patch, HBASE-19483.master.005.patch > > > Currently list_rsgroups command can be executed by any user. > This is inconsistent with other list commands such as list_peers and > list_peer_configs. > We should add proper privilege check for list_rsgroups command. > privilege check should be added for get_table_rsgroup / get_server_rsgroup / > get_rsgroup commands. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19559) Fix TestLogRolling.testLogRollOnDatanodeDeath
Duo Zhang created HBASE-19559: - Summary: Fix TestLogRolling.testLogRollOnDatanodeDeath Key: HBASE-19559 URL: https://issues.apache.org/jira/browse/HBASE-19559 Project: HBase Issue Type: Bug Reporter: Duo Zhang It appears several times on the failing list of pre commit result. The test code is a bit confusing to me. Need to dig more. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17513) Thrift Server 1 uses different QOP settings than RPC and Thrift Server 2 and can easily be misconfigured so there is no encryption when the operator expects it.
[ https://issues.apache.org/jira/browse/HBASE-17513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297803#comment-16297803 ] Reid Chan commented on HBASE-17513: --- Thank you [~tedyu]. Hi [~busbey], would you mind taking some time to have a look. > Thrift Server 1 uses different QOP settings than RPC and Thrift Server 2 and > can easily be misconfigured so there is no encryption when the operator > expects it. > > > Key: HBASE-17513 > URL: https://issues.apache.org/jira/browse/HBASE-17513 > Project: HBase > Issue Type: Bug > Components: documentation, security, Thrift, Usability >Affects Versions: 2.0.0, 1.2.0, 1.3.0, 0.98.15, 1.0.3, 1.1.3 >Reporter: Sean Busbey >Assignee: Reid Chan >Priority: Critical > Fix For: 2.0.0, 1.3.2, 1.4.1, 1.2.8 > > Attachments: HBASE-17513.branch-1.001.patch, > HBASE-17513.master.001.patch, HBASE-17513.master.002.patch > > > As of HBASE-14400 the setting {{hbase.thrift.security.qop}} was unified to > behave the same as the general HBase RPC protection. However, this only > happened for the Thrift2 server. The Thrift server found in the thrift > package (aka Thrift Server 1) still hard codes the old configs of 'auth', > 'auth-int', and 'auth-conf'. > Additionally, these Quality of Protection (qop) settings are used only by the > SASL transport. If a user configures the HBase Thrift Server to make use of > the HTTP transport (to enable doAs proxying e.g. for Hue) then a QOP setting > of 'privacy' or 'auth-conf' won't get them encryption as expected. > We should > 1) update {{hbase-thrift/src/main/.../thrift/ThriftServerRunner}} to rely on > {{SaslUtil}} to use the same 'authentication', 'integrity', 'privacy' configs > in a backward compatible way > 2) also have ThriftServerRunner warn when both {{hbase.thrift.security.qop}} > and {{hbase.regionserver.thrift.http}} are set, since the latter will cause > the former to be ignored. (users should be directed to > {{hbase.thrift.ssl.enabled}} and related configs to ensure their transport is > encrypted when using the HTTP transport.) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18440) ITs and Actions modify immutable TableDescriptors
[ https://issues.apache.org/jira/browse/HBASE-18440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297799#comment-16297799 ] Guanghao Zhang commented on HBASE-18440: bq. Do you have a JIRA filed for the failed verification? Can you post error there? Thanks for the remind. Let me try it again on master branch. Will file a jira after get the failed verification again. > ITs and Actions modify immutable TableDescriptors > - > > Key: HBASE-18440 > URL: https://issues.apache.org/jira/browse/HBASE-18440 > Project: HBase > Issue Type: Sub-task > Components: integration tests >Reporter: Mike Drob >Assignee: Guanghao Zhang > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-18440.master.004.patch, > HBASE-18440.master.005.patch, HBASE-18440.patch, HBASE-18440.v2.patch, > HBASE-18440.v3.patch, HBASE-18440.v3.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19483) Add proper privilege check for rsgroup commands
[ https://issues.apache.org/jira/browse/HBASE-19483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated HBASE-19483: -- Attachment: HBASE-19483.master.005.patch Update patch as [~yuzhih...@gmail.com] suggestions and fix checkstyle warnings. Thanks > Add proper privilege check for rsgroup commands > --- > > Key: HBASE-19483 > URL: https://issues.apache.org/jira/browse/HBASE-19483 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Guangxu Cheng > Fix For: 1.4.1, 1.5.0, 2.0.0-beta-2 > > Attachments: HBASE-19483.master.001.patch, > HBASE-19483.master.002.patch, HBASE-19483.master.003.patch, > HBASE-19483.master.004.patch, HBASE-19483.master.005.patch > > > Currently list_rsgroups command can be executed by any user. > This is inconsistent with other list commands such as list_peers and > list_peer_configs. > We should add proper privilege check for list_rsgroups command. > privilege check should be added for get_table_rsgroup / get_server_rsgroup / > get_rsgroup commands. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17436) Add facility to provide more information for Other Regions seen on Master UI
[ https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297791#comment-16297791 ] Appy commented on HBASE-17436: -- Looking at 17436.v11.txt. Is that the right patch? (It's the most recent one.) I ask because i don't see regionstate.jsp which is there in earlier patch. Otherwise looks good. > Add facility to provide more information for Other Regions seen on Master UI > > > Key: HBASE-17436 > URL: https://issues.apache.org/jira/browse/HBASE-17436 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Janos Gub > Labels: ui, usability > Fix For: 2.0.0 > > Attachments: 17436.v10.txt, 17436.v11.txt, 17436.v9.txt, > HBASE-17436-v2.patch, HBASE-17436-v5.patch, HBASE-17436-v6.patch, > HBASE-17436-v6.patch, HBASE-17436-v7.patch, HBASE-17436-v8.patch, > HBASE-17436.patch, HBASE-17779-v3.patch, HBASE-17779-v4.patch, Screen Shot > 2017-04-24 at 8.43.31 PM.png, Screen Shot 2017-04-26 at 4.39.49 PM.png, > Screen Shot 2017-11-08 at 10.29.49 AM.png, Screen Shot 2017-11-08 at 10.30.09 > AM.png, Screen Shot 2017-11-22 at 4.17.35 PM.png, initial.patch, > regionstates.png, table-selected.png, tableregions.png > > > [~rpednekar] and I were looking at a case where the count displayed under > Other Regions was high (~1200). > Since the table page just maintains a count instead of List of region names, > it is very difficult for user to determine which regions belong to this > category. > We should provide facility to provide more details for this category (LOG, > JMX, etc). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18440) ITs and Actions modify immutable TableDescriptors
[ https://issues.apache.org/jira/browse/HBASE-18440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297780#comment-16297780 ] Mike Drob commented on HBASE-18440: --- Thanks for following up on this [~zghaobac]! Do you have a JIRA filed for the failed verification? Can you post error there? > ITs and Actions modify immutable TableDescriptors > - > > Key: HBASE-18440 > URL: https://issues.apache.org/jira/browse/HBASE-18440 > Project: HBase > Issue Type: Sub-task > Components: integration tests >Reporter: Mike Drob >Assignee: Guanghao Zhang > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-18440.master.004.patch, > HBASE-18440.master.005.patch, HBASE-18440.patch, HBASE-18440.v2.patch, > HBASE-18440.v3.patch, HBASE-18440.v3.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-18429) ITs attempt to modify immutable table/column descriptors
[ https://issues.apache.org/jira/browse/HBASE-18429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-18429. Resolution: Fixed Assignee: Mike Drob Resolve this as all sub-tasks done. > ITs attempt to modify immutable table/column descriptors > > > Key: HBASE-18429 > URL: https://issues.apache.org/jira/browse/HBASE-18429 > Project: HBase > Issue Type: Umbrella > Components: integration tests >Affects Versions: 2.0.0-alpha-1 >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Critical > Fix For: 2.0.0-beta-1 > > > ITs: > * IntegrationTestIngestWithMOB (HBASE-18419) > * IntegrationTestDDLMasterFailover (HBASE-18428) > * IntegrationTestIngestWithEncryption::setUp (HBASE-18440) > * IntegrationTestBulkLoad::installSlowingCoproc (HBASE-18440) > Other Related: > * ChangeBloomFilterAction (HBASE-18419) > * ChangeCompressionAction (HBASE-18419) > * ChangeEncodingAction (HBASE-18419) > * ChangeVersionsAction (HBASE-18419) > * RemoveColumnAction (HBASE-18419) > * AddColumnAction::perform (HBASE-18440) > * ChangeSplitPolicyAction::perform (HBASE-18440) > * DecreaseMaxHFileSizeAction::perform (HBASE-18440) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18429) ITs attempt to modify immutable table/column descriptors
[ https://issues.apache.org/jira/browse/HBASE-18429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-18429: --- Fix Version/s: 2.0.0-beta-1 > ITs attempt to modify immutable table/column descriptors > > > Key: HBASE-18429 > URL: https://issues.apache.org/jira/browse/HBASE-18429 > Project: HBase > Issue Type: Umbrella > Components: integration tests >Affects Versions: 2.0.0-alpha-1 >Reporter: Mike Drob >Priority: Critical > Fix For: 2.0.0-beta-1 > > > ITs: > * IntegrationTestIngestWithMOB (HBASE-18419) > * IntegrationTestDDLMasterFailover (HBASE-18428) > * IntegrationTestIngestWithEncryption::setUp (HBASE-18440) > * IntegrationTestBulkLoad::installSlowingCoproc (HBASE-18440) > Other Related: > * ChangeBloomFilterAction (HBASE-18419) > * ChangeCompressionAction (HBASE-18419) > * ChangeEncodingAction (HBASE-18419) > * ChangeVersionsAction (HBASE-18419) > * RemoveColumnAction (HBASE-18419) > * AddColumnAction::perform (HBASE-18440) > * ChangeSplitPolicyAction::perform (HBASE-18440) > * DecreaseMaxHFileSizeAction::perform (HBASE-18440) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-10092) Move to slf4j
[ https://issues.apache.org/jira/browse/HBASE-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297772#comment-16297772 ] Appy commented on HBASE-10092: -- Facepalm. I had a review on RB which i forgot to publish yesterday. Ignore it now. > Move to slf4j > - > > Key: HBASE-10092 > URL: https://issues.apache.org/jira/browse/HBASE-10092 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Balazs Meszaros >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: 10092.txt, 10092v2.txt, HBASE-10092-preview-v0.patch, > HBASE-10092.master.001.patch, HBASE-10092.master.002.patch, > HBASE-10092.master.003.patch, HBASE-10092.patch > > > Allows logging with less friction. See http://logging.apache.org/log4j/2.x/ > This rather radical transition can be done w/ minor change given they have an > adapter for apache's logging, the one we use. They also have and adapter for > slf4j so we likely can remove at least some of the 4 versions of this module > our dependencies make use of. > I made a start in attached patch but am currently stuck in maven dependency > resolve hell courtesy of our slf4j. Fixing will take some concentration and > a good net connection, an item I currently lack. Other TODOs are that will > need to fix our little log level setting jsp page -- will likely have to undo > our use of hadoop's tool here -- and the config system changes a little. > I will return to this project soon. Will bring numbers. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19525) RS side changes for moving peer modification from zk watcher to procedure
[ https://issues.apache.org/jira/browse/HBASE-19525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297771#comment-16297771 ] Zheng Hu commented on HBASE-19525: -- [~Apache9], Let me rebase & fix the failed UTs, Will put the patch.v2 to RB. Thanks . > RS side changes for moving peer modification from zk watcher to procedure > - > > Key: HBASE-19525 > URL: https://issues.apache.org/jira/browse/HBASE-19525 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Replication >Reporter: Duo Zhang >Assignee: Zheng Hu > Fix For: HBASE-19397 > > Attachments: HBASE-19525.HBASE-19397.v1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19468) FNFE during scans and flushes
[ https://issues.apache.org/jira/browse/HBASE-19468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297769#comment-16297769 ] Thiruvel Thirumoolan commented on HBASE-19468: -- [~ram_krish] +1 I also verified with a unit test that files are cleaned up even when scanner expires. I will run ITBLL with this patch sometime in next few days. I guess this patch can get in and not have to wait for that. Thanks Ram! > FNFE during scans and flushes > - > > Key: HBASE-19468 > URL: https://issues.apache.org/jira/browse/HBASE-19468 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 1.3.1 >Reporter: Thiruvel Thirumoolan >Priority: Critical > Fix For: 2.0.0, 1.4.1, 1.5.0, 1.3.3 > > Attachments: HBASE-19468-poc.patch, HBASE-19468_1.4.patch, > HBASE-19468_master.patch > > > We see FNFE exceptions on our 1.3 clusters when scans and flushes happen at > the same time. This causes regionserver to throw a UnknownScannerException > and client retries. > This happens during the following sequence: > 1. Scanner open, client fetched some rows from regionserver and working on it > 2. Flush happens and storeScanner is updated with flushed files > (StoreScanner.updateReaders()) > 3. Compaction happens on the region while scanner is still open > 4. compaction discharger runs and cleans up the newly flushed file as we > don't have new scanners on it yet. > 5. Client issues scan.next and during StoreScanner.resetScannerStack(), we > get a FNFE. RegionServer throws a UnknownScannerThe client retries in 1.3. > With branch-1.4, the scan fails with a DoNotRetryIOException. > [~ram_krish], My proposal is to increment the reader count during > updateReaders() and decrement it during resetScannerStack(), so discharger > doesn't clean it up. Scan lease expiries also have to be taken care of. Am I > missing anything? Is there a better approach? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19525) RS side changes for moving peer modification from zk watcher to procedure
[ https://issues.apache.org/jira/browse/HBASE-19525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297766#comment-16297766 ] Duo Zhang commented on HBASE-19525: --- [~openinx] The DummyModifyPeerProcedure and its UT can be removed now. And since HBASE-19536 has been committed, you can rebase your patch and then the old UTs will work properly? And please upload the patch to review board. Thanks. > RS side changes for moving peer modification from zk watcher to procedure > - > > Key: HBASE-19525 > URL: https://issues.apache.org/jira/browse/HBASE-19525 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Replication >Reporter: Duo Zhang >Assignee: Zheng Hu > Fix For: HBASE-19397 > > Attachments: HBASE-19525.HBASE-19397.v1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19148) Reevaluate default values of configurations
[ https://issues.apache.org/jira/browse/HBASE-19148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297760#comment-16297760 ] stack commented on HBASE-19148: --- Played with setting SNAPPY on by default but was reminded that it could wreck havoc if the native lib was not available... Caused by: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support. ... so will skip on this one. > Reevaluate default values of configurations > --- > > Key: HBASE-19148 > URL: https://issues.apache.org/jira/browse/HBASE-19148 > Project: HBase > Issue Type: Bug > Components: defaults >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0-beta-1 > > Attachments: > 0002-HBASE-19148-Reevaluate-default-values-of-configurati.patch, > HBASE-19148.master.001.patch, HBASE-19148.master.002.patch, > HBASE-19148.master.003.patch > > > Remove cruft and mythologies. Make descriptions more digestible. Change > defaults given experience. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19536) Client side changes for moving peer modification from zk watcher to procedure
[ https://issues.apache.org/jira/browse/HBASE-19536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19536: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HBASE-19397 Status: Resolved (was: Patch Available) Rebased to the newest master and pushed to branch HBASE-19397. Thanks [~zghaobac] for the contributing. > Client side changes for moving peer modification from zk watcher to procedure > - > > Key: HBASE-19536 > URL: https://issues.apache.org/jira/browse/HBASE-19536 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Duo Zhang >Assignee: Guanghao Zhang > Fix For: HBASE-19397 > > Attachments: HBASE-19536.HBASE-19397.001.patch, > HBASE-19536.HBASE-19397.002.patch, HBASE-19536.HBASE-19397.003.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19536) Client side changes for moving peer modification from zk watcher to procedure
[ https://issues.apache.org/jira/browse/HBASE-19536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297744#comment-16297744 ] Duo Zhang commented on HBASE-19536: --- Seems not, HBASE-19558 is opened for it. Let me do a rebase and then commit. > Client side changes for moving peer modification from zk watcher to procedure > - > > Key: HBASE-19536 > URL: https://issues.apache.org/jira/browse/HBASE-19536 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Duo Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-19536.HBASE-19397.001.patch, > HBASE-19536.HBASE-19397.002.patch, HBASE-19536.HBASE-19397.003.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-10092) Move to slf4j
[ https://issues.apache.org/jira/browse/HBASE-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-10092: - Summary: Move to slf4j (was: Move up on to log4j2) > Move to slf4j > - > > Key: HBASE-10092 > URL: https://issues.apache.org/jira/browse/HBASE-10092 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Balazs Meszaros >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: 10092.txt, 10092v2.txt, HBASE-10092-preview-v0.patch, > HBASE-10092.master.001.patch, HBASE-10092.master.002.patch, > HBASE-10092.master.003.patch, HBASE-10092.patch > > > Allows logging with less friction. See http://logging.apache.org/log4j/2.x/ > This rather radical transition can be done w/ minor change given they have an > adapter for apache's logging, the one we use. They also have and adapter for > slf4j so we likely can remove at least some of the 4 versions of this module > our dependencies make use of. > I made a start in attached patch but am currently stuck in maven dependency > resolve hell courtesy of our slf4j. Fixing will take some concentration and > a good net connection, an item I currently lack. Other TODOs are that will > need to fix our little log level setting jsp page -- will likely have to undo > our use of hadoop's tool here -- and the config system changes a little. > I will return to this project soon. Will bring numbers. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-10092) Move up on to log4j2
[ https://issues.apache.org/jira/browse/HBASE-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297716#comment-16297716 ] Appy commented on HBASE-10092: -- +1 This has tackled many issues already. Let's get it in and work incrementally. Great stuff [~balazs.meszaros]. Impressed. :) On some of the previous things: - Yeah i meant removing of the guards. That's one thing slf4j was promising. But ignore it, too onerous and mechanical of a task to be done by single person - bq.I did some verification, setting log levels still works through web ui. Nice! forgot that one. Just listing down remaining stuff for clarity (so doesn't get lost in extensive discussion above): - Update to log4j 2 backend - Fix transitive include from hadoop > Move up on to log4j2 > > > Key: HBASE-10092 > URL: https://issues.apache.org/jira/browse/HBASE-10092 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Balazs Meszaros >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: 10092.txt, 10092v2.txt, HBASE-10092-preview-v0.patch, > HBASE-10092.master.001.patch, HBASE-10092.master.002.patch, > HBASE-10092.master.003.patch, HBASE-10092.patch > > > Allows logging with less friction. See http://logging.apache.org/log4j/2.x/ > This rather radical transition can be done w/ minor change given they have an > adapter for apache's logging, the one we use. They also have and adapter for > slf4j so we likely can remove at least some of the 4 versions of this module > our dependencies make use of. > I made a start in attached patch but am currently stuck in maven dependency > resolve hell courtesy of our slf4j. Fixing will take some concentration and > a good net connection, an item I currently lack. Other TODOs are that will > need to fix our little log level setting jsp page -- will likely have to undo > our use of hadoop's tool here -- and the config system changes a little. > I will return to this project soon. Will bring numbers. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19540) Reduce number of unnecessary semicolons
[ https://issues.apache.org/jira/browse/HBASE-19540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297707#comment-16297707 ] Hudson commented on HBASE-19540: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4256 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4256/]) HBASE-19540 Reduced number of unnecessary semicolons (jan.hentschel: rev f46a6d1637bc2c87001f08472e7ae6e39c8caeba) * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestWithCellVisibilityLoadAndVerify.java * (edit) hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/TestProcedureToString.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/io/MetricsIOSourceImpl.java * (edit) hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/TestStateMachineProcedure.java * (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestIngestWithMOB.java * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestBackupRestore.java * (edit) hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java * (edit) hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/model/ScannerModel.java * (edit) hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/SyncTable.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java * (edit) hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/StripeCompactionsPerformanceEvaluation.java * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/RollingBatchRestartRsExceptMetaAction.java * (edit) hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java * (edit) hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestCopyTable.java > Reduce number of unnecessary semicolons > --- > > Key: HBASE-19540 > URL: https://issues.apache.org/jira/browse/HBASE-19540 > Project: HBase > Issue Type: Sub-task >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-19540.master.001.patch, > HBASE-19540.master.002.patch > > > Some modules contain unnecessary semicolons, which should be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19480) Enable Checkstyle in hbase-annotations
[ https://issues.apache.org/jira/browse/HBASE-19480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297712#comment-16297712 ] Hudson commented on HBASE-19480: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4256 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4256/]) HBASE-19480 Enabled Checkstyle to fail on violations in (jan.hentschel: rev df351e4035df0f8db56245ea9cf54952874294ca) * (edit) hbase-annotations/pom.xml > Enable Checkstyle in hbase-annotations > -- > > Key: HBASE-19480 > URL: https://issues.apache.org/jira/browse/HBASE-19480 > Project: HBase > Issue Type: Sub-task >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Trivial > Fix For: 3.0.0, 2.0.0-beta-1 > > Attachments: HBASE-19480.master.001.patch > > > After all Checkstyle errors are now resolved in the *hbase-annotations* > module, Checkstyle can now be configured to fail on violations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19537) Remove unnecessary semicolons in hbase-backup
[ https://issues.apache.org/jira/browse/HBASE-19537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297710#comment-16297710 ] Hudson commented on HBASE-19537: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4256 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4256/]) HBASE-19537 Removed unnecessary semicolons from hbase-backup (jan.hentschel: rev d50ae037160cee874017280147263ce0ab07101a) * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalBackupManager.java * (edit) hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestIncrementalBackupMergeWithFailures.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupSystemTable.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/TableBackupClient.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/RestoreDriver.java * (edit) hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupBase.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/mapreduce/MapReduceBackupCopyJob.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/RestoreTool.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupCommands.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupAdminImpl.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/mapreduce/MapReduceBackupMergeJob.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/RestoreTablesClient.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/FullTableBackupClient.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/BackupInfo.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupManifest.java * (edit) hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupDeleteRestore.java > Remove unnecessary semicolons in hbase-backup > - > > Key: HBASE-19537 > URL: https://issues.apache.org/jira/browse/HBASE-19537 > Project: HBase > Issue Type: Sub-task >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-19537.master.001.patch > > > Currently *hbase-backup* has some places with unnecessary semicolons, which > should be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19538) Remove unnecessary semicolons in hbase-client
[ https://issues.apache.org/jira/browse/HBASE-19538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297709#comment-16297709 ] Hudson commented on HBASE-19538: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4256 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4256/]) HBASE-19538 Removed unnecessary semicolons in hbase-client (jan.hentschel: rev dd00081c26fb4222e223b510e58923118cf9df43) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/SnapshotType.java * (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/CompactionState.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/ThrottlingException.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/SpaceViolationPolicy.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ClusterStatus.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/security/SecurityCapability.java > Remove unnecessary semicolons in hbase-client > - > > Key: HBASE-19538 > URL: https://issues.apache.org/jira/browse/HBASE-19538 > Project: HBase > Issue Type: Sub-task >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Minor > Fix For: 3.0.0, 2.0.0-beta-1 > > Attachments: HBASE-19538.master.001.patch > > > Currently *hbase-client* has some places with unnecessary semicolons, which > should be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19481) Enable Checkstyle in hbase-error-prone
[ https://issues.apache.org/jira/browse/HBASE-19481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297711#comment-16297711 ] Hudson commented on HBASE-19481: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4256 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4256/]) HBASE-19481 Enabled Checkstyle to fail on violations in (jan.hentschel: rev ec7bf57390d8302598901a6e80ca710121cb5f48) * (edit) hbase-build-support/hbase-error-prone/pom.xml > Enable Checkstyle in hbase-error-prone > -- > > Key: HBASE-19481 > URL: https://issues.apache.org/jira/browse/HBASE-19481 > Project: HBase > Issue Type: Sub-task >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Trivial > Fix For: 3.0.0, 1.4.1, 1.5.0, 2.0.0-beta-1 > > Attachments: BREAKING-HBASE-19481.master.001.patch, > HBASE-19481.master.001.patch > > > *hbase-error-prone* doesn't contain any Checkstyle errors. With that > Checkstyle can now be configured to fail on violations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19539) Remove unnecessary semicolons in hbase-common
[ https://issues.apache.org/jira/browse/HBASE-19539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297708#comment-16297708 ] Hudson commented on HBASE-19539: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4256 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4256/]) HBASE-19539 Removed unnecessary semicolons in hbase-common (jan.hentschel: rev 0f8ea39a5b88b830ca6280e3262e7aa5fcb6280a) * (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/ClassTestFinder.java * (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/ClassFinder.java * (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/io/crypto/KeyProviderForTesting.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/io/TagCompressionContext.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/ProcedureState.java * (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/TestClassFinder.java * (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/TestCellUtil.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/codec/Codec.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/aes/CommonsCryptoAESDecryptor.java > Remove unnecessary semicolons in hbase-common > - > > Key: HBASE-19539 > URL: https://issues.apache.org/jira/browse/HBASE-19539 > Project: HBase > Issue Type: Sub-task >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Minor > Fix For: 3.0.0, 2.0.0-beta-1 > > Attachments: HBASE-19539.master.001.patch > > > Currently *hbase-common* has some places with unnecessary semicolons, which > should be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19555) TestSplitTransactionOnCluster is flaky
[ https://issues.apache.org/jira/browse/HBASE-19555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297706#comment-16297706 ] Hudson commented on HBASE-19555: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4256 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4256/]) HBASE-19555 TestSplitTransactionOnCluster is flaky (tedyu: rev dbe409e2c7af5312ab370165b89b57d176486e27) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java > TestSplitTransactionOnCluster is flaky > -- > > Key: HBASE-19555 > URL: https://issues.apache.org/jira/browse/HBASE-19555 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0-alpha-4 >Reporter: Peter Somogyi >Assignee: Peter Somogyi > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19555.master.001.patch > > > TestSplitTransactionOnCluster is #3 on flaky list. > Regions are checked 2 different ways: {{cluster.getRegions(hri.getTable())} > vs {{ProtobufUtil.getOnlineRegions(server.getRSRpcServices())}}. The later > one can also include {{hbase:namespace}} table as well causing the test to > fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19148) Reevaluate default values of configurations
[ https://issues.apache.org/jira/browse/HBASE-19148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297673#comment-16297673 ] stack commented on HBASE-19148: --- .003 Set balance by table as per Dave's suggestion. Fix checkstyle and whitespace. Nothing on thread asking for new suggestion so let this be what we roll with in beta-1. > Reevaluate default values of configurations > --- > > Key: HBASE-19148 > URL: https://issues.apache.org/jira/browse/HBASE-19148 > Project: HBase > Issue Type: Bug > Components: defaults >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0-beta-1 > > Attachments: > 0002-HBASE-19148-Reevaluate-default-values-of-configurati.patch, > HBASE-19148.master.001.patch, HBASE-19148.master.002.patch, > HBASE-19148.master.003.patch > > > Remove cruft and mythologies. Make descriptions more digestible. Change > defaults given experience. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19148) Reevaluate default values of configurations
[ https://issues.apache.org/jira/browse/HBASE-19148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19148: -- Attachment: HBASE-19148.master.003.patch > Reevaluate default values of configurations > --- > > Key: HBASE-19148 > URL: https://issues.apache.org/jira/browse/HBASE-19148 > Project: HBase > Issue Type: Bug > Components: defaults >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0-beta-1 > > Attachments: > 0002-HBASE-19148-Reevaluate-default-values-of-configurati.patch, > HBASE-19148.master.001.patch, HBASE-19148.master.002.patch, > HBASE-19148.master.003.patch > > > Remove cruft and mythologies. Make descriptions more digestible. Change > defaults given experience. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-19558) TestRegionsOnMasterOptions hack so it works reliablly
[ https://issues.apache.org/jira/browse/HBASE-19558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-19558. --- Resolution: Fixed I pushed the below which just waits until there are not RIT before it calls cluster shutdown commit f80d2e9c8e88d3eaeed9a174c770699737f8c574 Author: Michael Stack Date: Tue Dec 19 15:57:11 2017 -0800 HBASE-19558 TestRegionsOnMasterOptions hack so it works reliably diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRegionsOnMasterOptions.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRegionsOnMasterOptions.java index 7319820d39..58c1db 100644 --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRegionsOnMasterOptions.java +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRegionsOnMasterOptions.java @@ -193,6 +193,15 @@ public class TestRegionsOnMasterOptions { // still only carry system regions post crash. assertEquals(masterCount, mNewActualCount); } + // Disable balancer and wait till RIT done else cluster won't go down. + TEST_UTIL.getAdmin().balancerSwitch(false, true); + while (true) { +if (!TEST_UTIL.getHBaseCluster().getMaster().getAssignmentManager(). +isMetaRegionInTransition()) { + break; +} +Threads.sleep(10); + } } finally { LOG.info("Running shutdown of cluster"); TEST_UTIL.shutdownMiniCluster(); > TestRegionsOnMasterOptions hack so it works reliablly > - > > Key: HBASE-19558 > URL: https://issues.apache.org/jira/browse/HBASE-19558 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0-beta-1 > > > See tail of HBASE-19533 for description of problem; our shutdown of a cluster > needs work. Meantime hack the test so it doesn't hang on close so it passes > reliably again. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19558) TestRegionsOnMasterOptions hack so it works reliablly
stack created HBASE-19558: - Summary: TestRegionsOnMasterOptions hack so it works reliablly Key: HBASE-19558 URL: https://issues.apache.org/jira/browse/HBASE-19558 Project: HBase Issue Type: Bug Components: test Reporter: stack Assignee: stack Fix For: 2.0.0-beta-1 See tail of HBASE-19533 for description of problem; our shutdown of a cluster needs work. Meantime hack the test so it doesn't hang on close so it passes reliably again. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19533) How to do controlled shutdown in branch-2?
[ https://issues.apache.org/jira/browse/HBASE-19533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297657#comment-16297657 ] stack commented on HBASE-19533: --- Let me schedule this one. There's a little prob. in shutdown we'll have to fix. I noticed it over in TestRegionsOnMasterOptions. It fails occasionally. All tests pass but for the last one in the list, testRegionsOnAllServers, concurrent w/ the shutdown of the cluster, the balancer happens to run and schedules a region move. In the unassign of move, all is fine until the update of hbase:meta w/ changed state. hbase:meta has just closed so we are stuck trying to reach a server that won't come back. We'll retry up to the max. Meantime we block the shutdown of the Master. The running thread in the hbase client is actually a daemon thread. Its the retrying thread holding a lock that prevents the shutdown. Will be back here. To reliably reproduce, remove this bit I just added to TestRegionsOnMasterOptions: // Disable balancer and wait till RIT done else cluster won't go down. TEST_UTIL.getAdmin().balancerSwitch(false, true); while (true) { if (!TEST_UTIL.getHBaseCluster().getMaster().getAssignmentManager(). isMetaRegionInTransition()) { break; } Threads.sleep(10); } > How to do controlled shutdown in branch-2? > -- > > Key: HBASE-19533 > URL: https://issues.apache.org/jira/browse/HBASE-19533 > Project: HBase > Issue Type: Task >Reporter: stack > Fix For: 2.0.0-beta-2 > > > Before HBASE-18946, setting shutdown of a cluster, the Master would exit > immediately. RegionServers would run region closes and then try and notify > the Master of the close and would spew exceptions that the Master was > unreachable. > This is different to how branch-1 used to do it. It used to keep Master up > and it would be like the captain of the ship, the last to go down. As of > HBASE-18946, this is again the case but there are still open issues. > # Usually Master does all open and close of regions. On cluster shutdown, it > is the one time where the Regions run the region close. Currently, the > regions report the close to the Master which disregards the message since it > did not start the region closes. Should we do different? Try and update state > in hbase:meta setting it to CLOSE? We might not be able to write CLOSE for > all regions since hbase:meta will be closing too (the RS that is hosting > hbase:meta will close it last but that may not be enough). > # Should the Master run the cluster shutdown sending out close for all > regions? What if cluster of 1M regions? Untenable? Send a message per server? > That might be better. > Anyways, this needs attention. Filing issue in meantime. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-15482) Provide an option to skip calculating block locations for SnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15482: --- Hadoop Flags: Reviewed Fix Version/s: (was: 2.1.0) 2.0.0-beta-1 Integrated to branch-2 and master. If you want to backport to branch-1, please attach patch. Thanks for the patch, Xiang. Thanks for the review, Jerry. > Provide an option to skip calculating block locations for SnapshotInputFormat > - > > Key: HBASE-15482 > URL: https://issues.apache.org/jira/browse/HBASE-15482 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Liyin Tang >Assignee: Xiang Li >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: 15482.v3.txt, HBASE-15482.master.000.patch, > HBASE-15482.master.001.patch, HBASE-15482.master.002.patch, > HBASE-15482.master.003.patch > > > When a MR job is reading from SnapshotInputFormat, it needs to calculate the > splits based on the block locations in order to get best locality. However, > this process may take a long time for large snapshots. > In some setup, the computing layer, Spark, Hive or Presto could run out side > of HBase cluster. In these scenarios, the block locality doesn't matter. > Therefore, it will be great to have an option to skip calculating the block > locations for every job. That will super useful for the Hive/Presto/Spark > connectors. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19533) How to do controlled shutdown in branch-2?
[ https://issues.apache.org/jira/browse/HBASE-19533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19533: -- Fix Version/s: 2.0.0-beta-2 > How to do controlled shutdown in branch-2? > -- > > Key: HBASE-19533 > URL: https://issues.apache.org/jira/browse/HBASE-19533 > Project: HBase > Issue Type: Task >Reporter: stack > Fix For: 2.0.0-beta-2 > > > Before HBASE-18946, setting shutdown of a cluster, the Master would exit > immediately. RegionServers would run region closes and then try and notify > the Master of the close and would spew exceptions that the Master was > unreachable. > This is different to how branch-1 used to do it. It used to keep Master up > and it would be like the captain of the ship, the last to go down. As of > HBASE-18946, this is again the case but there are still open issues. > # Usually Master does all open and close of regions. On cluster shutdown, it > is the one time where the Regions run the region close. Currently, the > regions report the close to the Master which disregards the message since it > did not start the region closes. Should we do different? Try and update state > in hbase:meta setting it to CLOSE? We might not be able to write CLOSE for > all regions since hbase:meta will be closing too (the RS that is hosting > hbase:meta will close it last but that may not be enough). > # Should the Master run the cluster shutdown sending out close for all > regions? What if cluster of 1M regions? Untenable? Send a message per server? > That might be better. > Anyways, this needs attention. Filing issue in meantime. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19557) Build and release source jars for hbase-shaded-client and others
Jerry He created HBASE-19557: Summary: Build and release source jars for hbase-shaded-client and others Key: HBASE-19557 URL: https://issues.apache.org/jira/browse/HBASE-19557 Project: HBase Issue Type: Improvement Affects Versions: 1.2.6, 1.3.1 Reporter: Jerry He It seems that currently we don't build and release source jars for hbase-shaded-client (and server or mapreduce). IDEs will complain from the dependent users. We should provide them. http://central.maven.org/maven2/org/apache/hbase/hbase-shaded-client/1.3.1/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19491) Exclude flaky tests from nightly master run
[ https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19491: - Description: Testing infra improvements - Exclude flaky tests from nightly master run - Fixes findbugs check (seems like wasn't working earlier : "0 findbugs 0m 0s Findbugs executables are not available.") - Add nightly testing against hadoop3 was: I was of the opinion that nightly master runs were excluding flaky tests after seeing https://github.com/apache/hbase/blob/856ee283faf003404e8925006ce0e591c4eba600/dev-support/Jenkinsfile#L54 few days ago. After looking at our new set of scripts again and better understanding them, looks like that's not enough. We need to set {code}DOCKER_EXTRAARGS=--env=EXCLUDE_TESTS_URL=${EXCLUDE_TESTS_URL}{code} like in https://builds.apache.org/job/PreCommit-HBASE-Build/configure > Exclude flaky tests from nightly master run > --- > > Key: HBASE-19491 > URL: https://issues.apache.org/jira/browse/HBASE-19491 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > > Testing infra improvements > - Exclude flaky tests from nightly master run > - Fixes findbugs check (seems like wasn't working earlier : "0 > findbugs0m 0s Findbugs executables are not available.") > - Add nightly testing against hadoop3 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17436) Add facility to provide more information for Other Regions seen on Master UI
[ https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297546#comment-16297546 ] Ted Yu commented on HBASE-17436: [~stack]: Do you have other comment ? > Add facility to provide more information for Other Regions seen on Master UI > > > Key: HBASE-17436 > URL: https://issues.apache.org/jira/browse/HBASE-17436 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Janos Gub > Labels: ui, usability > Fix For: 2.0.0 > > Attachments: 17436.v10.txt, 17436.v11.txt, 17436.v9.txt, > HBASE-17436-v2.patch, HBASE-17436-v5.patch, HBASE-17436-v6.patch, > HBASE-17436-v6.patch, HBASE-17436-v7.patch, HBASE-17436-v8.patch, > HBASE-17436.patch, HBASE-17779-v3.patch, HBASE-17779-v4.patch, Screen Shot > 2017-04-24 at 8.43.31 PM.png, Screen Shot 2017-04-26 at 4.39.49 PM.png, > Screen Shot 2017-11-08 at 10.29.49 AM.png, Screen Shot 2017-11-08 at 10.30.09 > AM.png, Screen Shot 2017-11-22 at 4.17.35 PM.png, initial.patch, > regionstates.png, table-selected.png, tableregions.png > > > [~rpednekar] and I were looking at a case where the count displayed under > Other Regions was high (~1200). > Since the table page just maintains a count instead of List of region names, > it is very difficult for user to determine which regions belong to this > category. > We should provide facility to provide more details for this category (LOG, > JMX, etc). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-10092) Move up on to log4j2
[ https://issues.apache.org/jira/browse/HBASE-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297519#comment-16297519 ] stack commented on HBASE-10092: --- I tried it. We'll have to fix transitive include from hadoop... Can do in follow-on stack@ve0524:~$ ./hbase/bin/start-hbase.sh --config ~/conf_hbase SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/stack/hbase-3.0.0-SNAPSHOT/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/stack/hadoop-3.0.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/stack/hbase-3.0.0-SNAPSHOT/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/stack/hadoop-3.0.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] ve0524.halxg.cloudera.com: running zookeeper, logging to /home/stack/logs/hbase-stack-zookeeper-ve0524.out running master, logging to /home/stack/logs/hbase-stack-master-ve0524.out Log format is the same (duh) Setting log level works for me (hbase master branch on hadoop3) This move to slf4j should be a subissue of this one? We are still on log4j backend. Related, useful blog benchmarking logging frameworks https://www.loggly.com/blog/benchmarking-java-logging-frameworks/ > Move up on to log4j2 > > > Key: HBASE-10092 > URL: https://issues.apache.org/jira/browse/HBASE-10092 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Balazs Meszaros >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: 10092.txt, 10092v2.txt, HBASE-10092-preview-v0.patch, > HBASE-10092.master.001.patch, HBASE-10092.master.002.patch, > HBASE-10092.master.003.patch, HBASE-10092.patch > > > Allows logging with less friction. See http://logging.apache.org/log4j/2.x/ > This rather radical transition can be done w/ minor change given they have an > adapter for apache's logging, the one we use. They also have and adapter for > slf4j so we likely can remove at least some of the 4 versions of this module > our dependencies make use of. > I made a start in attached patch but am currently stuck in maven dependency > resolve hell courtesy of our slf4j. Fixing will take some concentration and > a good net connection, an item I currently lack. Other TODOs are that will > need to fix our little log level setting jsp page -- will likely have to undo > our use of hadoop's tool here -- and the config system changes a little. > I will return to this project soon. Will bring numbers. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19171) Update package references to match new shaded offset in hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-19171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297517#comment-16297517 ] Mike Drob commented on HBASE-19171: --- Didn't realize this issue existed, thanks [~jinghe]! > Update package references to match new shaded offset in hbase-thirdparty > > > Key: HBASE-19171 > URL: https://issues.apache.org/jira/browse/HBASE-19171 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Priority: Critical > > This has dependency on the parent task, and can only be done after a new > hbase-thirdparty release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19552) update hbase to use new thirdparty libs
[ https://issues.apache.org/jira/browse/HBASE-19552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-19552: -- Fix Version/s: 2.0.0-beta-1 > update hbase to use new thirdparty libs > --- > > Key: HBASE-19552 > URL: https://issues.apache.org/jira/browse/HBASE-19552 > Project: HBase > Issue Type: Task > Components: dependencies, thirdparty >Reporter: Mike Drob >Assignee: Mike Drob > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19552.patch, HBASE-19552.v2.patch > > > When we release hbase-thirdparty next release, we will need to update our > code for the packaging change. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19556) Remove TestAssignmentManager#testGoodSplit, which no longer make sense
[ https://issues.apache.org/jira/browse/HBASE-19556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297494#comment-16297494 ] Hadoop QA commented on HBASE-19556: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 15s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 53s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m 3s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 13s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 17m 25s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}100m 26s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 20s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}135m 18s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-19556 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12902919/HBASE-19556-master-v1.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 306c9ee1b190 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 0f8ea39a5b | | maven | version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) | | Default Java | 1.8.0_151 | | unit | https://builds.apache.org/job/PreCommit-HBASE-Build/10569/artifact/patchprocess/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/10569/testReport/ | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/10569/console | | Powered by | Apache Yetus 0.6.0 http://yetus.apache.org | This message was automatically generated. > Remove TestAssignmentManager#testGoodSplit, which no longer make sense > -
[jira] [Updated] (HBASE-19556) Remove TestAssignmentManager#testGoodSplit, which no longer make sense
[ https://issues.apache.org/jira/browse/HBASE-19556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19556: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master and branch-2. Thanks for patch [~easyliangjob] > Remove TestAssignmentManager#testGoodSplit, which no longer make sense > -- > > Key: HBASE-19556 > URL: https://issues.apache.org/jira/browse/HBASE-19556 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Yi Liang >Assignee: Yi Liang >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19556-master-v1.patch > > > {quote} > [ERROR] > testGoodSplit(org.apache.hadoop.hbase.master.assignment.TestAssignmentManager) > Time elapsed: 0.478 s <<< ERROR! > java.io.IOException: 5a50732f7cb3a05dd3a297bacbc34943 NOT splittable > {quote} > GoodSplitExecutor can only mock some functions in RSProcedureDispatcher, > however, to test split, we need to create(or mock) a real table on region > server side. And GoodSplitExecutor can not mock those kinds of function. > And, similar test has already been covered in TestSplitTableRegionProcedure, > we no longer need to test it again in TestAssignmentManager -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19494) Create simple WALKey filter that can be plugged in on the Replication Sink
[ https://issues.apache.org/jira/browse/HBASE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19494: -- Release Note: Adds means of adding very basic filter on the sink side of replication. We already have a means of installing filter source-side, which is better place to filter edits before they are shipped over the network, but this facility is needed by hbase-indexer. Set hbase.replication.sink.walentrysinkfilter with a no-param Constructor implementation. See test in patch for example. > Create simple WALKey filter that can be plugged in on the Replication Sink > -- > > Key: HBASE-19494 > URL: https://issues.apache.org/jira/browse/HBASE-19494 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: stack >Assignee: stack > Fix For: 2.0.0-beta-1 > > Attachments: > 0003-HBASE-19494-Create-simple-WALKey-filter-that-can-be-.patch, > HBASE-19494.master.001.patch > > > hbase-indexer used to look at WALKeys on the sink to see if their time of > creation was before the time at which the replication stream was enabled. > In the parent redo, there is no means for doing this anymore (because WALKey > used to be Private and because to get at the WALKey in the Sink, you had to > override all of the Replication which meant importing a million Private > objects...). > This issue is about adding a simple filter to Replication on the sink-side > that just takes a WALKey (now InterfaceAudience LimitedPrivate and recently > made read-only). > Assigned myself. Need to do this so hbase-indexer can move to hbase2. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19552) update hbase to use new thirdparty libs
[ https://issues.apache.org/jira/browse/HBASE-19552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297487#comment-16297487 ] Jerry He commented on HBASE-19552: -- Glanced through. Looks good. > update hbase to use new thirdparty libs > --- > > Key: HBASE-19552 > URL: https://issues.apache.org/jira/browse/HBASE-19552 > Project: HBase > Issue Type: Task > Components: dependencies, thirdparty >Reporter: Mike Drob >Assignee: Mike Drob > Attachments: HBASE-19552.patch, HBASE-19552.v2.patch > > > When we release hbase-thirdparty next release, we will need to update our > code for the packaging change. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19494) Create simple WALKey filter that can be plugged in on the Replication Sink
[ https://issues.apache.org/jira/browse/HBASE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19494: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to branch-2 and to master. Thanks for reviews [~whoschek] > Create simple WALKey filter that can be plugged in on the Replication Sink > -- > > Key: HBASE-19494 > URL: https://issues.apache.org/jira/browse/HBASE-19494 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: stack >Assignee: stack > Fix For: 2.0.0-beta-1 > > Attachments: > 0003-HBASE-19494-Create-simple-WALKey-filter-that-can-be-.patch, > HBASE-19494.master.001.patch > > > hbase-indexer used to look at WALKeys on the sink to see if their time of > creation was before the time at which the replication stream was enabled. > In the parent redo, there is no means for doing this anymore (because WALKey > used to be Private and because to get at the WALKey in the Sink, you had to > override all of the Replication which meant importing a million Private > objects...). > This issue is about adding a simple filter to Replication on the sink-side > that just takes a WALKey (now InterfaceAudience LimitedPrivate and recently > made read-only). > Assigned myself. Need to do this so hbase-indexer can move to hbase2. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-19171) Update package references to match new shaded offset in hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-19171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He resolved HBASE-19171. -- Resolution: Duplicate Fix Version/s: (was: 2.0.0-beta-1) Resolve as dup of the other task [~mdrob] is working on. > Update package references to match new shaded offset in hbase-thirdparty > > > Key: HBASE-19171 > URL: https://issues.apache.org/jira/browse/HBASE-19171 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Priority: Critical > > This has dependency on the parent task, and can only be done after a new > hbase-thirdparty release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19494) Create simple WALKey filter that can be plugged in on the Replication Sink
[ https://issues.apache.org/jira/browse/HBASE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297481#comment-16297481 ] stack commented on HBASE-19494: --- We have a few timeouts going on. This time it was this: [INFO] Results: [INFO] [WARNING] Flakes: [WARNING] org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient.testSkipFlushTableSnapshot(org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient) [ERROR] Run 1: TestFlushSnapshotFromClient.testSkipFlushTableSnapshot:201 Expected snapshot to contain family 'fam', but it does not. [INFO] Run 2: PASS [INFO] [INFO] [WARNING] Tests run: 3402, Failures: 0, Errors: 0, Skipped: 33, Flakes: 1 Need to fix. Unrelated. Let me push this change. > Create simple WALKey filter that can be plugged in on the Replication Sink > -- > > Key: HBASE-19494 > URL: https://issues.apache.org/jira/browse/HBASE-19494 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: stack >Assignee: stack > Fix For: 2.0.0-beta-1 > > Attachments: > 0003-HBASE-19494-Create-simple-WALKey-filter-that-can-be-.patch, > HBASE-19494.master.001.patch > > > hbase-indexer used to look at WALKeys on the sink to see if their time of > creation was before the time at which the replication stream was enabled. > In the parent redo, there is no means for doing this anymore (because WALKey > used to be Private and because to get at the WALKey in the Sink, you had to > override all of the Replication which meant importing a million Private > objects...). > This issue is about adding a simple filter to Replication on the sink-side > that just takes a WALKey (now InterfaceAudience LimitedPrivate and recently > made read-only). > Assigned myself. Need to do this so hbase-indexer can move to hbase2. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15482) Provide an option to skip calculating block locations for SnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297478#comment-16297478 ] Jerry He commented on HBASE-15482: -- +1 on the latest patch. > Provide an option to skip calculating block locations for SnapshotInputFormat > - > > Key: HBASE-15482 > URL: https://issues.apache.org/jira/browse/HBASE-15482 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Liyin Tang >Assignee: Xiang Li >Priority: Minor > Fix For: 2.1.0 > > Attachments: 15482.v3.txt, HBASE-15482.master.000.patch, > HBASE-15482.master.001.patch, HBASE-15482.master.002.patch, > HBASE-15482.master.003.patch > > > When a MR job is reading from SnapshotInputFormat, it needs to calculate the > splits based on the block locations in order to get best locality. However, > this process may take a long time for large snapshots. > In some setup, the computing layer, Spark, Hive or Presto could run out side > of HBase cluster. In these scenarios, the block locality doesn't matter. > Therefore, it will be great to have an option to skip calculating the block > locations for every job. That will super useful for the Hive/Presto/Spark > connectors. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15482) Provide an option to skip calculating block locations for SnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297466#comment-16297466 ] Ted Yu commented on HBASE-15482: lgtm > Provide an option to skip calculating block locations for SnapshotInputFormat > - > > Key: HBASE-15482 > URL: https://issues.apache.org/jira/browse/HBASE-15482 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Liyin Tang >Assignee: Xiang Li >Priority: Minor > Fix For: 2.1.0 > > Attachments: 15482.v3.txt, HBASE-15482.master.000.patch, > HBASE-15482.master.001.patch, HBASE-15482.master.002.patch, > HBASE-15482.master.003.patch > > > When a MR job is reading from SnapshotInputFormat, it needs to calculate the > splits based on the block locations in order to get best locality. However, > this process may take a long time for large snapshots. > In some setup, the computing layer, Spark, Hive or Presto could run out side > of HBase cluster. In these scenarios, the block locality doesn't matter. > Therefore, it will be great to have an option to skip calculating the block > locations for every job. That will super useful for the Hive/Presto/Spark > connectors. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19494) Create simple WALKey filter that can be plugged in on the Replication Sink
[ https://issues.apache.org/jira/browse/HBASE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297459#comment-16297459 ] Hadoop QA commented on HBASE-19494: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 6s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 29s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 3s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m 36s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 3s{color} | {color:red} hbase-server: The patch generated 22 new + 17 unchanged - 3 fixed = 39 total (was 20) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 34s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 18m 49s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}113m 41s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}151m 5s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-19494 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12902907/HBASE-19494.master.001.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 9e6be9603a28 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / dbe409e2c7 | | maven | version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) | | Default Java | 1.8.0_151 | | checkstyle | https://builds.apache.org/job/PreCommit-HBASE-Build/10567/artifact/patchprocess/diff-checkstyle-hbase-server.txt | | unit | https://builds.apache.org/job/PreCommit-HBASE-Build/10567/artifact/patchprocess/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/10567/testReport/ | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/10567/console | | Powered b
[jira] [Commented] (HBASE-10092) Move up on to log4j2
[ https://issues.apache.org/jira/browse/HBASE-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297431#comment-16297431 ] stack commented on HBASE-10092: --- Thanks [~mikael.stal...@home.se] Lets commit this. Going to log4j2 we can do in a follow-on, later. > Move up on to log4j2 > > > Key: HBASE-10092 > URL: https://issues.apache.org/jira/browse/HBASE-10092 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Balazs Meszaros >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: 10092.txt, 10092v2.txt, HBASE-10092-preview-v0.patch, > HBASE-10092.master.001.patch, HBASE-10092.master.002.patch, > HBASE-10092.master.003.patch, HBASE-10092.patch > > > Allows logging with less friction. See http://logging.apache.org/log4j/2.x/ > This rather radical transition can be done w/ minor change given they have an > adapter for apache's logging, the one we use. They also have and adapter for > slf4j so we likely can remove at least some of the 4 versions of this module > our dependencies make use of. > I made a start in attached patch but am currently stuck in maven dependency > resolve hell courtesy of our slf4j. Fixing will take some concentration and > a good net connection, an item I currently lack. Other TODOs are that will > need to fix our little log level setting jsp page -- will likely have to undo > our use of hadoop's tool here -- and the config system changes a little. > I will return to this project soon. Will bring numbers. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-10092) Move up on to log4j2
[ https://issues.apache.org/jira/browse/HBASE-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297413#comment-16297413 ] Mikael Ståldal commented on HBASE-10092: https://www.slf4j.org/faq.html#fatal is not correct, you can do that with Log4j 2 as well. You can use SLF4J as frontend with Log4j 2 as backend. However, then you cannot use Log4j 1 configuration files. As far as I know, only Log4j 1 can read those configuration files. > Move up on to log4j2 > > > Key: HBASE-10092 > URL: https://issues.apache.org/jira/browse/HBASE-10092 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Balazs Meszaros >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: 10092.txt, 10092v2.txt, HBASE-10092-preview-v0.patch, > HBASE-10092.master.001.patch, HBASE-10092.master.002.patch, > HBASE-10092.master.003.patch, HBASE-10092.patch > > > Allows logging with less friction. See http://logging.apache.org/log4j/2.x/ > This rather radical transition can be done w/ minor change given they have an > adapter for apache's logging, the one we use. They also have and adapter for > slf4j so we likely can remove at least some of the 4 versions of this module > our dependencies make use of. > I made a start in attached patch but am currently stuck in maven dependency > resolve hell courtesy of our slf4j. Fixing will take some concentration and > a good net connection, an item I currently lack. Other TODOs are that will > need to fix our little log level setting jsp page -- will likely have to undo > our use of hadoop's tool here -- and the config system changes a little. > I will return to this project soon. Will bring numbers. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19148) Reevaluate default values of configurations
[ https://issues.apache.org/jira/browse/HBASE-19148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297411#comment-16297411 ] stack commented on HBASE-19148: --- Thanks [~anastas] Added link here. > Reevaluate default values of configurations > --- > > Key: HBASE-19148 > URL: https://issues.apache.org/jira/browse/HBASE-19148 > Project: HBase > Issue Type: Bug > Components: defaults >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0-beta-1 > > Attachments: > 0002-HBASE-19148-Reevaluate-default-values-of-configurati.patch, > HBASE-19148.master.001.patch, HBASE-19148.master.002.patch > > > Remove cruft and mythologies. Make descriptions more digestible. Change > defaults given experience. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19480) Enable Checkstyle in hbase-annotations
[ https://issues.apache.org/jira/browse/HBASE-19480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Hentschel updated HBASE-19480: -- Resolution: Fixed Fix Version/s: 2.0.0-beta-1 3.0.0 Status: Resolved (was: Patch Available) > Enable Checkstyle in hbase-annotations > -- > > Key: HBASE-19480 > URL: https://issues.apache.org/jira/browse/HBASE-19480 > Project: HBase > Issue Type: Sub-task >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Trivial > Fix For: 3.0.0, 2.0.0-beta-1 > > Attachments: HBASE-19480.master.001.patch > > > After all Checkstyle errors are now resolved in the *hbase-annotations* > module, Checkstyle can now be configured to fail on violations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-10092) Move up on to log4j2
[ https://issues.apache.org/jira/browse/HBASE-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297407#comment-16297407 ] stack commented on HBASE-10092: --- The former. [~mikael.stal...@home.se] SLF4J as frontend reading our log4j1 property files with log4j2 as our backend? > Move up on to log4j2 > > > Key: HBASE-10092 > URL: https://issues.apache.org/jira/browse/HBASE-10092 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Balazs Meszaros >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: 10092.txt, 10092v2.txt, HBASE-10092-preview-v0.patch, > HBASE-10092.master.001.patch, HBASE-10092.master.002.patch, > HBASE-10092.master.003.patch, HBASE-10092.patch > > > Allows logging with less friction. See http://logging.apache.org/log4j/2.x/ > This rather radical transition can be done w/ minor change given they have an > adapter for apache's logging, the one we use. They also have and adapter for > slf4j so we likely can remove at least some of the 4 versions of this module > our dependencies make use of. > I made a start in attached patch but am currently stuck in maven dependency > resolve hell courtesy of our slf4j. Fixing will take some concentration and > a good net connection, an item I currently lack. Other TODOs are that will > need to fix our little log level setting jsp page -- will likely have to undo > our use of hadoop's tool here -- and the config system changes a little. > I will return to this project soon. Will bring numbers. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19494) Create simple WALKey filter that can be plugged in on the Replication Sink
[ https://issues.apache.org/jira/browse/HBASE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297392#comment-16297392 ] stack commented on HBASE-19494: --- Thanks for the feedback [~whoschek] Will commit when hadoopqa comes back. > Create simple WALKey filter that can be plugged in on the Replication Sink > -- > > Key: HBASE-19494 > URL: https://issues.apache.org/jira/browse/HBASE-19494 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: stack >Assignee: stack > Fix For: 2.0.0-beta-1 > > Attachments: > 0003-HBASE-19494-Create-simple-WALKey-filter-that-can-be-.patch, > HBASE-19494.master.001.patch > > > hbase-indexer used to look at WALKeys on the sink to see if their time of > creation was before the time at which the replication stream was enabled. > In the parent redo, there is no means for doing this anymore (because WALKey > used to be Private and because to get at the WALKey in the Sink, you had to > override all of the Replication which meant importing a million Private > objects...). > This issue is about adding a simple filter to Replication on the sink-side > that just takes a WALKey (now InterfaceAudience LimitedPrivate and recently > made read-only). > Assigned myself. Need to do this so hbase-indexer can move to hbase2. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19481) Enable Checkstyle in hbase-error-prone
[ https://issues.apache.org/jira/browse/HBASE-19481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Hentschel updated HBASE-19481: -- Resolution: Fixed Fix Version/s: 2.0.0-beta-1 1.5.0 1.4.1 3.0.0 Status: Resolved (was: Patch Available) > Enable Checkstyle in hbase-error-prone > -- > > Key: HBASE-19481 > URL: https://issues.apache.org/jira/browse/HBASE-19481 > Project: HBase > Issue Type: Sub-task >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Trivial > Fix For: 3.0.0, 1.4.1, 1.5.0, 2.0.0-beta-1 > > Attachments: BREAKING-HBASE-19481.master.001.patch, > HBASE-19481.master.001.patch > > > *hbase-error-prone* doesn't contain any Checkstyle errors. With that > Checkstyle can now be configured to fail on violations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-18684) [hbase-thirdparty] Remove CHANGES, update NOTICES
[ https://issues.apache.org/jira/browse/HBASE-18684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob resolved HBASE-18684. --- Resolution: Fixed > [hbase-thirdparty] Remove CHANGES, update NOTICES > - > > Key: HBASE-18684 > URL: https://issues.apache.org/jira/browse/HBASE-18684 > Project: HBase > Issue Type: Task > Components: hbase-thirdparty >Reporter: stack >Assignee: Mike Drob > Fix For: thirdparty-2.0.0 > > Attachments: HBASE-18684.patch > > > From [~mdrob] on vote thread for hbase-thirdparty 1.0.1RC0: > "LICENSE question... do we actually bundle jquery, asciidoctor, and the orca > logo here? I didn't see them anywhere. > CHANGES is mostly blank... If we don't use it, get rid of it in the next > version." > Let me fix in next release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19494) Create simple WALKey filter that can be plugged in on the Replication Sink
[ https://issues.apache.org/jira/browse/HBASE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297391#comment-16297391 ] wolfgang hoschek commented on HBASE-19494: -- Great. I think this works. Thanks! > Create simple WALKey filter that can be plugged in on the Replication Sink > -- > > Key: HBASE-19494 > URL: https://issues.apache.org/jira/browse/HBASE-19494 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: stack >Assignee: stack > Fix For: 2.0.0-beta-1 > > Attachments: > 0003-HBASE-19494-Create-simple-WALKey-filter-that-can-be-.patch, > HBASE-19494.master.001.patch > > > hbase-indexer used to look at WALKeys on the sink to see if their time of > creation was before the time at which the replication stream was enabled. > In the parent redo, there is no means for doing this anymore (because WALKey > used to be Private and because to get at the WALKey in the Sink, you had to > override all of the Replication which meant importing a million Private > objects...). > This issue is about adding a simple filter to Replication on the sink-side > that just takes a WALKey (now InterfaceAudience LimitedPrivate and recently > made read-only). > Assigned myself. Need to do this so hbase-indexer can move to hbase2. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-10092) Move up on to log4j2
[ https://issues.apache.org/jira/browse/HBASE-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297390#comment-16297390 ] Mikael Ståldal commented on HBASE-10092: {quote}SLF4J doc says it will only show if backend is logback. Otherwise, dropped. Good to have it though. We'll have to call out in release notes that FATAL no longer shows in our logs (usually){quote} Are you referring to https://www.slf4j.org/faq.html#fatal or https://www.slf4j.org/faq.html#marker_interface ? That's not correct, Log4j 2 also support markers. > Move up on to log4j2 > > > Key: HBASE-10092 > URL: https://issues.apache.org/jira/browse/HBASE-10092 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Balazs Meszaros >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: 10092.txt, 10092v2.txt, HBASE-10092-preview-v0.patch, > HBASE-10092.master.001.patch, HBASE-10092.master.002.patch, > HBASE-10092.master.003.patch, HBASE-10092.patch > > > Allows logging with less friction. See http://logging.apache.org/log4j/2.x/ > This rather radical transition can be done w/ minor change given they have an > adapter for apache's logging, the one we use. They also have and adapter for > slf4j so we likely can remove at least some of the 4 versions of this module > our dependencies make use of. > I made a start in attached patch but am currently stuck in maven dependency > resolve hell courtesy of our slf4j. Fixing will take some concentration and > a good net connection, an item I currently lack. Other TODOs are that will > need to fix our little log level setting jsp page -- will likely have to undo > our use of hadoop's tool here -- and the config system changes a little. > I will return to this project soon. Will bring numbers. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19552) update hbase to use new thirdparty libs
[ https://issues.apache.org/jira/browse/HBASE-19552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-19552: -- Attachment: HBASE-19552.v2.patch v2: rebased, also dropping the ugly system property from tests for netty relocation > update hbase to use new thirdparty libs > --- > > Key: HBASE-19552 > URL: https://issues.apache.org/jira/browse/HBASE-19552 > Project: HBase > Issue Type: Task > Components: dependencies, thirdparty >Reporter: Mike Drob >Assignee: Mike Drob > Attachments: HBASE-19552.patch, HBASE-19552.v2.patch > > > When we release hbase-thirdparty next release, we will need to update our > code for the packaging change. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18970) The version of jruby we use now can't get interactive input from prompt
[ https://issues.apache.org/jira/browse/HBASE-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297379#comment-16297379 ] Mike Drob commented on HBASE-18970: --- Neat catch on the stderr redirection, patch LGTM, will try to commit later. > The version of jruby we use now can't get interactive input from prompt > --- > > Key: HBASE-18970 > URL: https://issues.apache.org/jira/browse/HBASE-18970 > Project: HBase > Issue Type: Task > Components: shell >Reporter: Chia-Ping Tsai >Assignee: AMIT VIRMANI >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBase-18970.branch-2.001.patch, samplecmd.txt, > test_shell-Output.txt, test_shell.sh > > > case 1: press enter > {code} > hbase(main):002:0> disable_all 'chia(.*)' > chia_1 > > > Disable the above 1 tables (y/n)? > y^M > {code} > case 2: press ctrl-j > {code} > hbase(main):001:0> disable_all 'chia(.*)' > chia_1 > > > Disable the above 1 tables (y/n)? > y^J1 tables successfully disabled > Took 5.0059 seconds > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HBASE-18970) The version of jruby we use now can't get interactive input from prompt
[ https://issues.apache.org/jira/browse/HBASE-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob reassigned HBASE-18970: - Assignee: AMIT VIRMANI (was: Mike Drob) > The version of jruby we use now can't get interactive input from prompt > --- > > Key: HBASE-18970 > URL: https://issues.apache.org/jira/browse/HBASE-18970 > Project: HBase > Issue Type: Task > Components: shell >Reporter: Chia-Ping Tsai >Assignee: AMIT VIRMANI >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBase-18970.branch-2.001.patch, samplecmd.txt, > test_shell-Output.txt, test_shell.sh > > > case 1: press enter > {code} > hbase(main):002:0> disable_all 'chia(.*)' > chia_1 > > > Disable the above 1 tables (y/n)? > y^M > {code} > case 2: press ctrl-j > {code} > hbase(main):001:0> disable_all 'chia(.*)' > chia_1 > > > Disable the above 1 tables (y/n)? > y^J1 tables successfully disabled > Took 5.0059 seconds > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19556) Remove TestAssignmentManager#testGoodSplit, which no longer make sense
[ https://issues.apache.org/jira/browse/HBASE-19556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297376#comment-16297376 ] stack commented on HBASE-19556: --- Thanks [~easyliangjob] +1 > Remove TestAssignmentManager#testGoodSplit, which no longer make sense > -- > > Key: HBASE-19556 > URL: https://issues.apache.org/jira/browse/HBASE-19556 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Yi Liang >Assignee: Yi Liang >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19556-master-v1.patch > > > {quote} > [ERROR] > testGoodSplit(org.apache.hadoop.hbase.master.assignment.TestAssignmentManager) > Time elapsed: 0.478 s <<< ERROR! > java.io.IOException: 5a50732f7cb3a05dd3a297bacbc34943 NOT splittable > {quote} > GoodSplitExecutor can only mock some functions in RSProcedureDispatcher, > however, to test split, we need to create(or mock) a real table on region > server side. And GoodSplitExecutor can not mock those kinds of function. > And, similar test has already been covered in TestSplitTableRegionProcedure, > we no longer need to test it again in TestAssignmentManager -- This message was sent by Atlassian JIRA (v6.4.14#64029)