[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993313#comment-14993313
]
Heng Chen commented on HBASE-14279:
---
{quote}
I think ConcurrentIndex is never appropriate for the package hbase.util.
{quote}
I don't think so. We really need one DataStructure 'multiMapSet'. And
concurrentIndex can do it, maybe we could add more interface for it. wdyt? :)
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14769) Remove unused functions and duplicate javadocs from HBaseAdmin
[
https://issues.apache.org/jira/browse/HBASE-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993309#comment-14993309
]
Hadoop QA commented on HBASE-14769:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770954/HBASE-14769-master-v3.patch
against master branch at commit bfa36891901b96b95d82f5307642c35fd2b9f534.
ATTACHMENT ID: 12770954
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 33 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1731 checkstyle errors (more than the master's current 1726 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ * Create a timestamp consistent snapshot for the given table. Snapshots
are considered unique based
+Pair pair = new Pair<>(ret.getYetToUpdateRegions(),
ret.getTotalRegions());
+ raise ArgumentError, "Table #{table_name_str} is not enabled. Enable it
first." unless enabled?(table_name_str)
+ raise(ArgumentError, "Table name must be of type String") unless
table_name_str.kind_of?(String)
+ raise(ArgumentError, "Snapshot name must be of type String") unless
snapshot_name.kind_of?(String)
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16423//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16423//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16423//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16423//console
This message is automatically generated.
> Remove unused functions and duplicate javadocs from HBaseAdmin
> ---
>
> Key: HBASE-14769
> URL: https://issues.apache.org/jira/browse/HBASE-14769
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-14769-master-v2.patch,
> HBASE-14769-master-v3.patch, HBASE-14769-master.patch
>
>
> HBaseAdmin is marked private, so removing the functions not being used
> anywhere.
> Also, the javadocs of overridden functions are same as corresponding ones in
> Admin.java. Since javadocs are automatically inherited from the interface
> class, we can remove these redundant 100s of lines.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14463) Severe performance downgrade when parallel reading a single key from BucketCache
[
https://issues.apache.org/jira/browse/HBASE-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993307#comment-14993307
]
Hadoop QA commented on HBASE-14463:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770952/HBASE-14463_v12.patch
against master branch at commit bfa36891901b96b95d82f5307642c35fd2b9f534.
ATTACHMENT ID: 12770952
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16422//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16422//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16422//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16422//console
This message is automatically generated.
> Severe performance downgrade when parallel reading a single key from
> BucketCache
>
>
> Key: HBASE-14463
> URL: https://issues.apache.org/jira/browse/HBASE-14463
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.14, 1.1.2
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.17
>
> Attachments: GC_with_WeakObjectPool.png, HBASE-14463.patch,
> HBASE-14463_v11.patch, HBASE-14463_v12.patch, HBASE-14463_v12.patch,
> HBASE-14463_v2.patch, HBASE-14463_v3.patch, HBASE-14463_v4.patch,
> HBASE-14463_v5.patch, TestBucketCache-new_with_IdLock.png,
> TestBucketCache-new_with_IdReadWriteLock.png,
> TestBucketCache_with_IdLock-latest.png, TestBucketCache_with_IdLock.png,
> TestBucketCache_with_IdReadWriteLock-latest.png,
> TestBucketCache_with_IdReadWriteLock-resolveLockLeak.png,
> TestBucketCache_with_IdReadWriteLock.png, pe_use_same_keys.patch,
> test-results.tar.gz
>
>
> We store feature data of online items in HBase, do machine learning on these
> features, and supply the outputs to our online search engine. In such
> scenario we will launch hundreds of yarn workers and each worker will read
> all features of one item(i.e. single rowkey in HBase), so there'll be heavy
> parallel reading on a single rowkey.
> We were using LruCache but start to try BucketCache recently to resolve gc
> issue, and just as titled we have observed severe performance downgrade.
> After some analytics we found the root cause is the lock in
> BucketCache#getBlock, as shown below
> {code}
> try {
> lockEntry = offsetLock.getLockEntry(bucketEntry.offset());
> // ...
> if (bucketEntry.equals(backingMap.get(key))) {
> // ...
> int len = bucketEntry.getLength();
> Cacheable cachedBlock = ioEngine.read(bucketEntry.offset(), len,
> bucketEntry.deserializerReference(this.deserialiserMap));
> {code}
> Since ioEnging.read involves array copy, it's much more time-costed than the
> operation in LruCache. And since we're using synchronized in
> IdLock#getLockEntry, parallel read dropping on the same bucket would be
> executed in serial, which causes a really bad performance.
> To resolve the problem, we propose to use ReentranceReadWriteLock in
> BucketCache, and introduce a new class called IdReadWriteLock to implement it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993302#comment-14993302
]
Hiroshi Ikeda commented on HBASE-14279:
---
{quote}
Currently there is only one place to call this method, it is is BucketCache.
{quote}
I think ConcurrentIndex is never appropriate for the package hbase.util. This
is a part of BucketCahce and should be placed in the same package of
BucketCahce, and it is better to be a package private class. (Fortunately this
is still declared as InterfaceAudience.Private.)
Anyone would not so much complain encapsulated classes' specification as long
as they well work together.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14473) Compute region locality in parallel
[ https://issues.apache.org/jira/browse/HBASE-14473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993293#comment-14993293 ] Guanghao Zhang commented on HBASE-14473: [~eclark] What do you think about that make region location cache time configurable again? Large cluster may need increase cache time to reduce the region locality computation. Small cluster may need config the cache time same with balancer period to response the locality change. > Compute region locality in parallel > --- > > Key: HBASE-14473 > URL: https://issues.apache.org/jira/browse/HBASE-14473 > Project: HBase > Issue Type: Bug > Components: Balancer >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14473-v1.patch, HBASE-14473-v2.patch, > HBASE-14473-v3.patch, HBASE-14473-v4.patch, HBASE-14473.patch > > > Right now on large clusters it's necessary to turn off the locality balance > cost as it takes too long to compute the region locality. This is because > it's computed when need in serial. > We should compute this in parallel before it's needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993268#comment-14993268
]
Heng Chen commented on HBASE-14279:
---
{quote}
Is it important to be able to select between hash-set and sorted-set?
{quote}
Yeah, i think we need to do it as original logic.
{quote}
I think the caller would expect the returned object is created by the given
value factory, and some methods might have different behaviors from the
immutable set, especially if the caller gives a value factory which creates a
sorted set (or a value comparator).
{quote}
Currently there is only one place to call this method, it is is BucketCache.
After call it, it use ImmutableSet.copyOf(set) to make another set to read.
IMO we can return this immutable set directly by ConcurrentIndex.values.
wdyt? :)
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14773) Fix HBase shell tests are skipped when skipping server tests.
[ https://issues.apache.org/jira/browse/HBASE-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993266#comment-14993266 ] Hudson commented on HBASE-14773: SUCCESS: Integrated in HBase-1.2 #351 (See [https://builds.apache.org/job/HBase-1.2/351/]) HBASE-14773 Fix HBase shell tests are skipped when skipping server (eclark: rev d1bdd68a813ea5c56eeb2576839177eb7b83726d) * hbase-shell/pom.xml > Fix HBase shell tests are skipped when skipping server tests. > - > > Key: HBASE-14773 > URL: https://issues.apache.org/jira/browse/HBASE-14773 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14773.patch > > > Looks like a copy pasta error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14772) Improve zombie detector; be more discerning
[
https://issues.apache.org/jira/browse/HBASE-14772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14772:
--
Assignee: stack
Status: Patch Available (was: Open)
Or, first, let me make sure the pom change goes over ok and doesn't break
anything.
> Improve zombie detector; be more discerning
> ---
>
> Key: HBASE-14772
> URL: https://issues.apache.org/jira/browse/HBASE-14772
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>Assignee: stack
> Attachments: zombie.patch, zombiev2.patch
>
>
> Currently, any surefire process with the hbase flag is a potential zombie.
> Our zombie check currently takes a reading and if it finds candidate zombies,
> it waits 30 seconds and then does another reading. If a concurrent build
> going on, in both cases the zombie detector will come up positive though the
> adjacent test run may be making progress; i.e. the cast of surefire processes
> may have changed between readings but our detector just sees presence of
> hbase surefire processes.
> Here is example:
> {code}
> Suspicious java process found - waiting 30s to see if there are just slow to
> stop
> There appear to be 5 zombie tests, they should have been killed by surefire
> but survived
> 12823 surefirebooter852180186418035480.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 7653 surefirebooter8579074445899448699.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 12614 surefirebooter136529596936417090.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 7836 surefirebooter3217047564606450448.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 13566 surefirebooter2084039411151963494.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> BEGIN zombies jstack extract
> END zombies jstack extract
> {code}
> 5 is the number of forked processes we allow when doing medium and large
> tests so an adjacent build will always show as '5 zombies'.
> Need to add discerning if list of processes changes between readings.
> Can I also add a tag per build run that all forked processes pick up so I can
> look at the current builds progeny only?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14772) Improve zombie detector; be more discerning
[
https://issues.apache.org/jira/browse/HBASE-14772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14772:
--
Attachment: zombiev2.patch
Some small improvement... more dogged at figuring what test we are running when
zombie found. Let me add this and then massage in place.
> Improve zombie detector; be more discerning
> ---
>
> Key: HBASE-14772
> URL: https://issues.apache.org/jira/browse/HBASE-14772
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
> Attachments: zombie.patch, zombiev2.patch
>
>
> Currently, any surefire process with the hbase flag is a potential zombie.
> Our zombie check currently takes a reading and if it finds candidate zombies,
> it waits 30 seconds and then does another reading. If a concurrent build
> going on, in both cases the zombie detector will come up positive though the
> adjacent test run may be making progress; i.e. the cast of surefire processes
> may have changed between readings but our detector just sees presence of
> hbase surefire processes.
> Here is example:
> {code}
> Suspicious java process found - waiting 30s to see if there are just slow to
> stop
> There appear to be 5 zombie tests, they should have been killed by surefire
> but survived
> 12823 surefirebooter852180186418035480.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 7653 surefirebooter8579074445899448699.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 12614 surefirebooter136529596936417090.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 7836 surefirebooter3217047564606450448.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 13566 surefirebooter2084039411151963494.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> BEGIN zombies jstack extract
> END zombies jstack extract
> {code}
> 5 is the number of forked processes we allow when doing medium and large
> tests so an adjacent build will always show as '5 zombies'.
> Need to add discerning if list of processes changes between readings.
> Can I also add a tag per build run that all forked processes pick up so I can
> look at the current builds progeny only?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14769) Remove unused functions and duplicate javadocs from HBaseAdmin
[
https://issues.apache.org/jira/browse/HBASE-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993249#comment-14993249
]
Hadoop QA commented on HBASE-14769:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770944/HBASE-14769-master-v2.patch
against master branch at commit bfa36891901b96b95d82f5307642c35fd2b9f534.
ATTACHMENT ID: 12770944
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 33 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1731 checkstyle errors (more than the master's current 1726 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ * Create a timestamp consistent snapshot for the given table. Snapshots
are considered unique based
+Pair pair = new Pair<>(ret.getYetToUpdateRegions(),
ret.getTotalRegions());
+ raise ArgumentError, "Table #{table_name_str} is not enabled. Enable it
first." unless enabled?(table_name_str)
+ raise(ArgumentError, "Table name must be of type String") unless
table_name_str.kind_of?(String)
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16421//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16421//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16421//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16421//console
This message is automatically generated.
> Remove unused functions and duplicate javadocs from HBaseAdmin
> ---
>
> Key: HBASE-14769
> URL: https://issues.apache.org/jira/browse/HBASE-14769
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-14769-master-v2.patch,
> HBASE-14769-master-v3.patch, HBASE-14769-master.patch
>
>
> HBaseAdmin is marked private, so removing the functions not being used
> anywhere.
> Also, the javadocs of overridden functions are same as corresponding ones in
> Admin.java. Since javadocs are automatically inherited from the interface
> class, we can remove these redundant 100s of lines.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14773) Fix HBase shell tests are skipped when skipping server tests.
[ https://issues.apache.org/jira/browse/HBASE-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993247#comment-14993247 ] Hudson commented on HBASE-14773: SUCCESS: Integrated in HBase-1.3 #351 (See [https://builds.apache.org/job/HBase-1.3/351/]) HBASE-14773 Fix HBase shell tests are skipped when skipping server (eclark: rev 7c9e1738306264def7f19ed1607974acb98cecfb) * hbase-shell/pom.xml > Fix HBase shell tests are skipped when skipping server tests. > - > > Key: HBASE-14773 > URL: https://issues.apache.org/jira/browse/HBASE-14773 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14773.patch > > > Looks like a copy pasta error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993245#comment-14993245 ] ramkrishna.s.vasudevan commented on HBASE-13082: bq.The locks are not the problem, it's the memory barriers on every call to next and pretty that cause the performance issue. The locks are (almost) never contended Yes Lars. I too agree with this. I am trying to profile before I could come to any conclusion on the memory barrier that this volatile introduces. Ideally for the compaction cases we don't even need the notifying mechansim and can always be going on without any locks or resets of the heap but for flushes we may need it because if we don't reset then the flushed snapshot cannot be GCed and if this is going to be bigger scans then may lead to GC issues. Other than that as you always say Scanner is >99% single threaded. > Coarsen StoreScanner locks to RegionScanner > --- > > Key: HBASE-13082 > URL: https://issues.apache.org/jira/browse/HBASE-13082 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: ramkrishna.s.vasudevan > Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, > 13082-v4.txt, 13082.txt, 13082.txt, HBASE-13082.pdf, HBASE-13082_1.pdf, > HBASE-13082_1_WIP.patch, HBASE-13082_2_WIP.patch, HBASE-13082_3.patch, > HBASE-13082_4.patch, HBASE-13082_9.patch, HBASE-13082_9.patch, gc.png, > gc.png, gc.png, hits.png, next.png, next.png > > > Continuing where HBASE-10015 left of. > We can avoid locking (and memory fencing) inside StoreScanner by deferring to > the lock already held by the RegionScanner. > In tests this shows quite a scan improvement and reduced CPU (the fences make > the cores wait for memory fetches). > There are some drawbacks too: > * All calls to RegionScanner need to be remain synchronized > * Implementors of coprocessors need to be diligent in following the locking > contract. For example Phoenix does not lock RegionScanner.nextRaw() and > required in the documentation (not picking on Phoenix, this one is my fault > as I told them it's OK) > * possible starving of flushes and compaction with heavy read load. > RegionScanner operations would keep getting the locks and the > flushes/compactions would not be able finalize the set of files. > I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12790) Support fairness across parallelized scans
[ https://issues.apache.org/jira/browse/HBASE-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993243#comment-14993243 ] ramkrishna.s.vasudevan commented on HBASE-12790: bq.I've asked if this groupid specialization is a deal breaker or will pulling on exisiting attributes whether connection identifier or scan attributes will do. I may have been answered above or over in rb but have not seen it if I have. Scan attributes alone will not do IMHO because the queues cannot do this round robin for now. It can handle only priority but cannot round robin within the same priority. Let me see if the connection identifier can be of any use. Then use this connection identifier to go thro a queue that can do the round robin with the connection identifier. Let me check that more closely in terms of phoenix code also. > Support fairness across parallelized scans > -- > > Key: HBASE-12790 > URL: https://issues.apache.org/jira/browse/HBASE-12790 > Project: HBase > Issue Type: New Feature >Reporter: James Taylor >Assignee: ramkrishna.s.vasudevan > Labels: Phoenix > Attachments: AbstractRoundRobinQueue.java, HBASE-12790.patch, > HBASE-12790_1.patch, HBASE-12790_5.patch, HBASE-12790_callwrapper.patch, > HBASE-12790_trunk_1.patch, PHOENIX_4.5.3-HBase-0.98-2317-SNAPSHOT.zip > > > Some HBase clients parallelize the execution of a scan to reduce latency in > getting back results. This can lead to starvation with a loaded cluster and > interleaved scans, since the RPC queue will be ordered and processed on a > FIFO basis. For example, if there are two clients, A & B that submit largish > scans at the same time. Say each scan is broken down into 100 scans by the > client (broken down into equal depth chunks along the row key), and the 100 > scans of client A are queued first, followed immediately by the 100 scans of > client B. In this case, client B will be starved out of getting any results > back until the scans for client A complete. > One solution to this is to use the attached AbstractRoundRobinQueue instead > of the standard FIFO queue. The queue to be used could be (maybe it already > is) configurable based on a new config parameter. Using this queue would > require the client to have the same identifier for all of the 100 parallel > scans that represent a single logical scan from the clients point of view. > With this information, the round robin queue would pick off a task from the > queue in a round robin fashion (instead of a strictly FIFO manner) to prevent > starvation over interleaved parallelized scans. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993241#comment-14993241
]
Hiroshi Ikeda commented on HBASE-14279:
---
{quote}
BucketCache will define sortedSet by itself, so we should pass the comparator
into it.
{quote}
Oops, I overlooked... Is it important to be able to select between hash-set and
sorted-set? Anyway concurrency/synchronized one just gives nothing but
overhead, and at least it is required to rewrite the default factory.
{code}
public Set values(K key) {
...skip...
+ return ImmutableSet.copyOf(set);
+}
}
{code}
I think the caller would expect the returned object is created by the given
value factory, and some methods might have different behaviors from the
immutable set, especially if the caller gives a value factory which creates a
sorted set (or a value comparator).
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993240#comment-14993240
]
Hadoop QA commented on HBASE-14279:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770950/HBASE-14279_v4.patch
against master branch at commit bfa36891901b96b95d82f5307642c35fd2b9f534.
ATTACHMENT ID: 12770950
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1728 checkstyle errors (more than the master's current 1726 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16420//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16420//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16420//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16420//console
This message is automatically generated.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14767) Remove deprecated functions from HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-14767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993216#comment-14993216 ] Appy commented on HBASE-14767: -- working on changes in shell code > Remove deprecated functions from HBaseAdmin > --- > > Key: HBASE-14767 > URL: https://issues.apache.org/jira/browse/HBASE-14767 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-14767-master-v2.patch, > HBASE-14767-master-v3.patch, HBASE-14767-master.patch > > > Many functions in HBaseAdmin are marked deprecated. Removing them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14767) Remove deprecated functions from HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-14767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14767: - Attachment: HBASE-14767-master-v3.patch v3: fixing checkstyle error > Remove deprecated functions from HBaseAdmin > --- > > Key: HBASE-14767 > URL: https://issues.apache.org/jira/browse/HBASE-14767 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-14767-master-v2.patch, > HBASE-14767-master-v3.patch, HBASE-14767-master.patch > > > Many functions in HBaseAdmin are marked deprecated. Removing them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14708) Use copy on write Map for region location cache
[ https://issues.apache.org/jira/browse/HBASE-14708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993168#comment-14993168 ] Hiroshi Ikeda commented on HBASE-14708: --- What I want to say is, it is better to expose the internal data in all the methods rather than paying cost of copying partially, because you cannot guard its consistency after all. In for a penny, in for a pound :P > Use copy on write Map for region location cache > --- > > Key: HBASE-14708 > URL: https://issues.apache.org/jira/browse/HBASE-14708 > Project: HBase > Issue Type: Improvement > Components: Client >Affects Versions: 1.1.2 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14708-v10.patch, HBASE-14708-v11.patch, > HBASE-14708-v12.patch, HBASE-14708-v13.patch, HBASE-14708-v2.patch, > HBASE-14708-v3.patch, HBASE-14708-v4.patch, HBASE-14708-v5.patch, > HBASE-14708-v6.patch, HBASE-14708-v7.patch, HBASE-14708-v8.patch, > HBASE-14708-v9.patch, HBASE-14708.patch, anotherbench.zip, > location_cache_times.pdf, result.csv > > > Internally a co-worker profiled their application that was talking to HBase. > > 60% of the time was spent in locating a region. This was while the cluster > was stable and no regions were moving. > To figure out if there was a faster way to cache region location I wrote up a > benchmark here: https://github.com/elliottneilclark/benchmark-hbase-cache > This tries to simulate a heavy load on the location cache. > * 24 different threads. > * 2 Deleting location data > * 2 Adding location data > * Using floor to get the result. > To repeat my work just run ./run.sh and it should produce a result.csv > Results: > ConcurrentSkiplistMap is a good middle ground. It's got equal speed for > reading and writing. > However most operations will not need to remove or add a region location. > There will be potentially several orders of magnitude more reads for cached > locations than there will be on clearing the cache. > So I propose a copy on write tree map. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14355) Scan different TimeRange for each column family
[ https://issues.apache.org/jira/browse/HBASE-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993167#comment-14993167 ] churro morales commented on HBASE-14355: [~anoop] sorry been sick for the past week, will get you a patch tomorrow and I will fix my oversight. > Scan different TimeRange for each column family > --- > > Key: HBASE-14355 > URL: https://issues.apache.org/jira/browse/HBASE-14355 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver, Scanners >Reporter: Dave Latham >Assignee: churro morales > Fix For: 2.0.0, 1.3.0, 0.98.17 > > Attachments: HBASE-14355-v1.patch, HBASE-14355-v2.patch, > HBASE-14355-v3.patch, HBASE-14355-v4.patch, HBASE-14355-v5.patch, > HBASE-14355-v6.patch, HBASE-14355-v7.patch, HBASE-14355-v8.patch, > HBASE-14355.patch > > > At present the Scan API supports only table level time range. We have > specific use cases that will benefit from per column family time range. (See > background discussion at > https://mail-archives.apache.org/mod_mbox/hbase-user/201508.mbox/%3ccaa4mzom00ef5eoxstk0hetxeby8mqss61gbvgttgpaspmhq...@mail.gmail.com%3E) > There are a couple of choices that would be good to validate. First - how to > update the Scan API to support family and table level updates. One proposal > would be to add Scan.setTimeRange(byte family, long minTime, long maxTime), > then store it in a Map. When executing the scan, if a > family has a specified TimeRange, then use it, otherwise fall back to using > the table level TimeRange. Clients using the new API against old region > servers would not get the families correctly filterd. Old clients sending > scans to new region servers would work correctly. > The other question is how to get StoreFileScanner.shouldUseScanner to match > up the proper family and time range. It has the Scan available but doesn't > currently have available which family it is a part of. One option would be > to try to pass down the column family in each constructor path. Another > would be to instead alter shouldUseScanner to pass down the specific > TimeRange to use (similar to how it currently passes down the columns to use > which also appears to be a workaround for not having the family available). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14771) RpcServer.getRemoteAddress always returns null.
[
https://issues.apache.org/jira/browse/HBASE-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993165#comment-14993165
]
Abhishek Kumar commented on HBASE-14771:
it was mainly for logging purpose such as the use of
HMaster.getClientIdAuditPrefix() in create/modify etc operations and we have
some custom co-processor also making use of this getRemoteAddress method which
was causing some issues for us.
> RpcServer.getRemoteAddress always returns null.
> ---
>
> Key: HBASE-14771
> URL: https://issues.apache.org/jira/browse/HBASE-14771
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
>Affects Versions: 1.2.0
>Reporter: Abhishek Kumar
>Assignee: Abhishek Kumar
>Priority: Minor
> Attachments: HBASE-14771.patch
>
>
> RpcServer.getRemoteAddress always returns null, because Call object is
> getting initialized with null.This seems to be happening because of using
> RpcServer.getRemoteIp() in Call object constructor before RpcServer thread
> local 'CurCall' being set in CallRunner.run method:
> {noformat}
> // --- RpcServer.java ---
> protected void processRequest(byte[] buf) throws IOException,
> InterruptedException {
> .
> // Call object getting initialized here with address
> // obtained from RpcServer.getRemoteIp()
> Call call = new Call(id, this.service, md, header, param, cellScanner, this,
> responder,
> totalRequestSize, traceInfo, RpcServer.getRemoteIp());
> scheduler.dispatch(new CallRunner(RpcServer.this, call));
> }
> // getRemoteIp method gets address from threadlocal 'CurCall' which
> // gets set in CallRunner.run and calling it before this as in above case,
> will return null
> // --- CallRunner.java ---
> public void run() {
> .
> Pair resultPair = null;
> RpcServer.CurCall.set(call);
> ..
> }
> // Using 'this.addr' in place of getRemoteIp method in RpcServer.java seems
> to be fixing this issue
> Call call = new Call(id, this.service, md, header, param, cellScanner, this,
> responder,
> totalRequestSize, traceInfo, this.addr);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14769) Remove unused functions and duplicate javadocs from HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14769: - Attachment: HBASE-14769-master-v3.patch v3: Fixes some failing shell tests (manually copied TestShell from history to ran the tests) > Remove unused functions and duplicate javadocs from HBaseAdmin > --- > > Key: HBASE-14769 > URL: https://issues.apache.org/jira/browse/HBASE-14769 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-14769-master-v2.patch, > HBASE-14769-master-v3.patch, HBASE-14769-master.patch > > > HBaseAdmin is marked private, so removing the functions not being used > anywhere. > Also, the javadocs of overridden functions are same as corresponding ones in > Admin.java. Since javadocs are automatically inherited from the interface > class, we can remove these redundant 100s of lines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14773) Fix HBase shell tests are skipped when skipping server tests.
[ https://issues.apache.org/jira/browse/HBASE-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993108#comment-14993108 ] Hudson commented on HBASE-14773: SUCCESS: Integrated in HBase-1.2-IT #267 (See [https://builds.apache.org/job/HBase-1.2-IT/267/]) HBASE-14773 Fix HBase shell tests are skipped when skipping server (eclark: rev d1bdd68a813ea5c56eeb2576839177eb7b83726d) * hbase-shell/pom.xml > Fix HBase shell tests are skipped when skipping server tests. > - > > Key: HBASE-14773 > URL: https://issues.apache.org/jira/browse/HBASE-14773 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14773.patch > > > Looks like a copy pasta error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14463) Severe performance downgrade when parallel reading a single key from BucketCache
[
https://issues.apache.org/jira/browse/HBASE-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-14463:
--
Attachment: HBASE-14463_v12.patch
Thanks [~anoop.hbase] for the double check and confirmation!
Checked and confirmed v12 patch could apply in the latest code base, re-attach
to see what HadoopQA will say (with recent commits/code changes)
> Severe performance downgrade when parallel reading a single key from
> BucketCache
>
>
> Key: HBASE-14463
> URL: https://issues.apache.org/jira/browse/HBASE-14463
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.14, 1.1.2
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.17
>
> Attachments: GC_with_WeakObjectPool.png, HBASE-14463.patch,
> HBASE-14463_v11.patch, HBASE-14463_v12.patch, HBASE-14463_v12.patch,
> HBASE-14463_v2.patch, HBASE-14463_v3.patch, HBASE-14463_v4.patch,
> HBASE-14463_v5.patch, TestBucketCache-new_with_IdLock.png,
> TestBucketCache-new_with_IdReadWriteLock.png,
> TestBucketCache_with_IdLock-latest.png, TestBucketCache_with_IdLock.png,
> TestBucketCache_with_IdReadWriteLock-latest.png,
> TestBucketCache_with_IdReadWriteLock-resolveLockLeak.png,
> TestBucketCache_with_IdReadWriteLock.png, pe_use_same_keys.patch,
> test-results.tar.gz
>
>
> We store feature data of online items in HBase, do machine learning on these
> features, and supply the outputs to our online search engine. In such
> scenario we will launch hundreds of yarn workers and each worker will read
> all features of one item(i.e. single rowkey in HBase), so there'll be heavy
> parallel reading on a single rowkey.
> We were using LruCache but start to try BucketCache recently to resolve gc
> issue, and just as titled we have observed severe performance downgrade.
> After some analytics we found the root cause is the lock in
> BucketCache#getBlock, as shown below
> {code}
> try {
> lockEntry = offsetLock.getLockEntry(bucketEntry.offset());
> // ...
> if (bucketEntry.equals(backingMap.get(key))) {
> // ...
> int len = bucketEntry.getLength();
> Cacheable cachedBlock = ioEngine.read(bucketEntry.offset(), len,
> bucketEntry.deserializerReference(this.deserialiserMap));
> {code}
> Since ioEnging.read involves array copy, it's much more time-costed than the
> operation in LruCache. And since we're using synchronized in
> IdLock#getLockEntry, parallel read dropping on the same bucket would be
> executed in serial, which causes a really bad performance.
> To resolve the problem, we propose to use ReentranceReadWriteLock in
> BucketCache, and introduce a new class called IdReadWriteLock to implement it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14776) Rewrite smart-apply-patch.sh to use 'git am' or 'git apply' rather than 'patch'
[
https://issues.apache.org/jira/browse/HBASE-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993084#comment-14993084
]
Hadoop QA commented on HBASE-14776:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770923/HBASE-14776.patch
against master branch at commit bfa36891901b96b95d82f5307642c35fd2b9f534.
ATTACHMENT ID: 12770923
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation, build,
or dev-support patch that doesn't require tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16419//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16419//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16419//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16419//console
This message is automatically generated.
> Rewrite smart-apply-patch.sh to use 'git am' or 'git apply' rather than
> 'patch'
> ---
>
> Key: HBASE-14776
> URL: https://issues.apache.org/jira/browse/HBASE-14776
> Project: HBase
> Issue Type: Bug
> Components: scripts
>Affects Versions: 2.0.0
>Reporter: Misty Stanley-Jones
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: HBASE-14776.patch
>
>
> We require patches to be created using 'git format-patch' or 'git diff', so
> patches should be tested using 'git am' or 'git apply', not 'patch -pX'. This
> causes false errors in the Jenkins patch tester.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14776) Rewrite smart-apply-patch.sh to use 'git am' or 'git apply' rather than 'patch'
[ https://issues.apache.org/jira/browse/HBASE-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993067#comment-14993067 ] Misty Stanley-Jones commented on HBASE-14776: - For the passing - for stdin, I am not sure. I didn't think so. Maybe you can tweak it. I don't understand enough about how the script works. It didn't look like the script actually applied the patch anyway. Did it? If so, that's easy enough to accomodate. In the case of 'git apply, we won't be able to commit the change, and in the case of 'git am' we will have an extra commit in the stack. It's a bit messy. > Rewrite smart-apply-patch.sh to use 'git am' or 'git apply' rather than > 'patch' > --- > > Key: HBASE-14776 > URL: https://issues.apache.org/jira/browse/HBASE-14776 > Project: HBase > Issue Type: Bug > Components: scripts >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14776.patch > > > We require patches to be created using 'git format-patch' or 'git diff', so > patches should be tested using 'git am' or 'git apply', not 'patch -pX'. This > causes false errors in the Jenkins patch tester. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993063#comment-14993063
]
Heng Chen commented on HBASE-14279:
---
{quote}
Defensively copy under the lock of the segment
{quote}
Make sense, fix it.
{quote}
valueSetFactory is almost meaningless
{quote}
I don't think so, because BucketCache will define sortedSet by itself, so we
should pass the comparator into it.
I add some tests to verify this class.
Thanks [~ikeda] a lot. update patch
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14279:
--
Attachment: HBASE-14279_v4.patch
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, HBASE-14279_v4.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14776) Rewrite smart-apply-patch.sh to use 'git am' or 'git apply' rather than 'patch'
[
https://issues.apache.org/jira/browse/HBASE-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993054#comment-14993054
]
Sean Busbey commented on HBASE-14776:
-
no worries. I checked and it looks like it does.
{code}
PATCH_FILE=$1
-if [ -z "$PATCH_FILE" ]; then
+if [ -z "$PATCH_FILE" -o ! -f "$PATCH_FILE" ]; then
echo usage: $0 patch-file
exit 1
fi
{code}
won't this break the "pass '-' for stdin" bit?
{code}
-echo Going to apply patch with: $PATCH -p$PLEVEL
-$PATCH -p$PLEVEL -E < $PATCH_FILE
-
+if [ $AM_WORKS -eq 0 ]; then
+ # try git apply
+ git apply $PATCH_FILE
+ status=$?
+ if [ $status -eq 0 ]; then
+APPLY_WORKS=1
+echo "Patch applies with git apply. Consider using git format-patch to
create patches."
+# Apply does not commit, just reset
+git reset -q --hard
+ else
+echo "Patch does not apply with git apply."
+ fi
+fi
+if [ $AM_WORKS -eq 0 -a $APPLY_WORKS -eq 0 ]; then
+ echo "Could not apply the patch."
+ cleanup 1
+fi
cleanup $?
{code}
It looks like after this change we don't actually apply the patch?
> Rewrite smart-apply-patch.sh to use 'git am' or 'git apply' rather than
> 'patch'
> ---
>
> Key: HBASE-14776
> URL: https://issues.apache.org/jira/browse/HBASE-14776
> Project: HBase
> Issue Type: Bug
> Components: scripts
>Affects Versions: 2.0.0
>Reporter: Misty Stanley-Jones
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: HBASE-14776.patch
>
>
> We require patches to be created using 'git format-patch' or 'git diff', so
> patches should be tested using 'git am' or 'git apply', not 'patch -pX'. This
> causes false errors in the Jenkins patch tester.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14774) Raise the font size on high-DPI small-screen devices like iphone 6+
[
https://issues.apache.org/jira/browse/HBASE-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993051#comment-14993051
]
Hadoop QA commented on HBASE-14774:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770910/HBASE-14774.patch
against master branch at commit bfa36891901b96b95d82f5307642c35fd2b9f534.
ATTACHMENT ID: 12770910
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation, build,
or dev-support patch that doesn't require tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16418//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16418//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16418//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16418//console
This message is automatically generated.
> Raise the font size on high-DPI small-screen devices like iphone 6+
> ---
>
> Key: HBASE-14774
> URL: https://issues.apache.org/jira/browse/HBASE-14774
> Project: HBase
> Issue Type: Bug
> Components: website
>Affects Versions: 2.0.0
>Reporter: Misty Stanley-Jones
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: HBASE-14774.patch, image.jpg
>
>
> On iPads and things like that, the website looks fine. But the fonts are too
> small on high-DPI small screens. It's tiny on my iPhone 6+.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14769) Remove unused functions and duplicate javadocs from HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14769: - Attachment: HBASE-14769-master-v2.patch v2 patch: fixing shell usages and docs. > Remove unused functions and duplicate javadocs from HBaseAdmin > --- > > Key: HBASE-14769 > URL: https://issues.apache.org/jira/browse/HBASE-14769 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-14769-master-v2.patch, HBASE-14769-master.patch > > > HBaseAdmin is marked private, so removing the functions not being used > anywhere. > Also, the javadocs of overridden functions are same as corresponding ones in > Admin.java. Since javadocs are automatically inherited from the interface > class, we can remove these redundant 100s of lines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14755) Fix some broken links and HTML problems
[ https://issues.apache.org/jira/browse/HBASE-14755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993028#comment-14993028 ] Hudson commented on HBASE-14755: FAILURE: Integrated in HBase-Trunk_matrix #438 (See [https://builds.apache.org/job/HBase-Trunk_matrix/438/]) HBASE-14755 Fix some broken links and HTML problems (mstanleyjones: rev bfa36891901b96b95d82f5307642c35fd2b9f534) * src/main/asciidoc/_chapters/security.adoc * src/main/asciidoc/_chapters/performance.adoc * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java * src/main/asciidoc/_chapters/appendix_hfile_format.adoc * src/main/site/xdoc/poweredbyhbase.xml * src/main/asciidoc/_chapters/cp.adoc * src/main/asciidoc/_chapters/datamodel.adoc * pom.xml * src/main/asciidoc/_chapters/architecture.adoc * src/main/asciidoc/_chapters/ops_mgt.adoc > Fix some broken links and HTML problems > --- > > Key: HBASE-14755 > URL: https://issues.apache.org/jira/browse/HBASE-14755 > Project: HBase > Issue Type: Bug > Components: documentation >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14755-v1.patch, HBASE-14755-v1.patch, > HBASE-14755.patch > > > Problems seen in > https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Website%20Link%20Ckecker/3/artifact/link_report/index.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14773) Fix HBase shell tests are skipped when skipping server tests.
[ https://issues.apache.org/jira/browse/HBASE-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993027#comment-14993027 ] Hudson commented on HBASE-14773: FAILURE: Integrated in HBase-Trunk_matrix #438 (See [https://builds.apache.org/job/HBase-Trunk_matrix/438/]) HBASE-14773 Fix HBase shell tests are skipped when skipping server (eclark: rev 68b94886a5cc0d8db26c49616669c244f9cae916) * hbase-shell/pom.xml > Fix HBase shell tests are skipped when skipping server tests. > - > > Key: HBASE-14773 > URL: https://issues.apache.org/jira/browse/HBASE-14773 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14773.patch > > > Looks like a copy pasta error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14773) Fix HBase shell tests are skipped when skipping server tests.
[ https://issues.apache.org/jira/browse/HBASE-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993026#comment-14993026 ] Hudson commented on HBASE-14773: FAILURE: Integrated in HBase-1.3-IT #297 (See [https://builds.apache.org/job/HBase-1.3-IT/297/]) HBASE-14773 Fix HBase shell tests are skipped when skipping server (eclark: rev 7c9e1738306264def7f19ed1607974acb98cecfb) * hbase-shell/pom.xml > Fix HBase shell tests are skipped when skipping server tests. > - > > Key: HBASE-14773 > URL: https://issues.apache.org/jira/browse/HBASE-14773 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14773.patch > > > Looks like a copy pasta error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14763) Remove usages of deprecated HConnection
[
https://issues.apache.org/jira/browse/HBASE-14763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993008#comment-14993008
]
Hadoop QA commented on HBASE-14763:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770907/hbase-14763.v3.patch
against master branch at commit 68b94886a5cc0d8db26c49616669c244f9cae916.
ATTACHMENT ID: 12770907
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 75 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:red}-1 javac{color}. The applied patch generated 36 javac compiler
warnings (more than the master's current 35 warnings).
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16417//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16417//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16417//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16417//console
This message is automatically generated.
> Remove usages of deprecated HConnection
>
>
> Key: HBASE-14763
> URL: https://issues.apache.org/jira/browse/HBASE-14763
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-14763.patch, hbase-14763.v2.patch,
> hbase-14763.v3.patch, hbase-14763.v3.patch
>
>
> HConnection was deprecated in 1.0.0. There are two interfaces that are
> supposed to be used instead -- Connection for client programs and
> ClusterConnection for internal hbaes and special tools (LoadIncremental,
> HBCK, etc).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14759) Avoid using Math.abs when selecting SyncRunner in FSHLog
[
https://issues.apache.org/jira/browse/HBASE-14759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-14759:
--
Affects Version/s: 1.3.0
Fix Version/s: 1.3.0
> Avoid using Math.abs when selecting SyncRunner in FSHLog
>
>
> Key: HBASE-14759
> URL: https://issues.apache.org/jira/browse/HBASE-14759
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.0.2, 1.2.0, 1.1.2, 1.3.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3
>
> Attachments: HBASE-14759.patch
>
>
> {code:title=FSHLog.java}
> int index = Math.abs(this.syncRunnerIndex++) % this.syncRunners.length;
> try {
> this.syncRunners[index].offer(sequence, this.syncFutures,
> this.syncFuturesCount);
> } catch (Exception e) {
> // Should NEVER get here.
> requestLogRoll();
> this.exception = new DamagedWALException("Failed offering sync",
> e);
> }
> {code}
> Math.abs will return Integer.MIN_VALUE if you pass Integer.MIN_VALUE in since
> the actual absolute value of Integer.MIN_VALUE is out of range.
> I think {{this.syncRunnerIndex++}} will overflow eventually if we keep the
> regionserver running for enough time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12794) Guidelines for filing JIRA issues
[ https://issues.apache.org/jira/browse/HBASE-12794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992982#comment-14992982 ] Misty Stanley-Jones commented on HBASE-12794: - Ping, want me to take the Google Doc and turn it into something in the book? > Guidelines for filing JIRA issues > - > > Key: HBASE-12794 > URL: https://issues.apache.org/jira/browse/HBASE-12794 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: stack >Assignee: stack > > Following on from Andrew's JIRA year-end cleaning spree, lets get some > guidelines on filing issues e.g. fill out all pertinent fields, add context > and provenance, add value (i.e. triage), don't file issues that are nought > but repeat of info available elsewhere (build box or mailing list), be > reluctant filing issues that don't have a resource behind them, don't file > issues on behalf of others, don't split fixes across multiple issues (because > there are poor folks coming behind us trying to backport our mess and > piecemeal makes their jobs harder), and so on. > The guidelines are not meant to put a chill on the opening of issues when > problems are found, especially not for new contributors. They are more meant > for quoting to veteran contributors who continue to file issues in violation > of what was thought a common understanding; rather than explain each time why > an issue has been marked invalid, it would be better if we can quote chapter > and verse from the refguide. > Dump any suggestion in here and I'll wind them up as a patch that I'll run by > dev mailing list to get consensus before committing. > Here is a running google doc if you'd like to add comment: > https://docs.google.com/document/d/1p3ArVLcnQnifk6ZsF635qWBhMmfTUJsISyK15DXnam0/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14776) Rewrite smart-apply-patch.sh to use 'git am' or 'git apply' rather than 'patch'
[ https://issues.apache.org/jira/browse/HBASE-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992947#comment-14992947 ] Misty Stanley-Jones commented on HBASE-14776: - I ... don't know? Not familiar with Yetus. > Rewrite smart-apply-patch.sh to use 'git am' or 'git apply' rather than > 'patch' > --- > > Key: HBASE-14776 > URL: https://issues.apache.org/jira/browse/HBASE-14776 > Project: HBase > Issue Type: Bug > Components: scripts >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14776.patch > > > We require patches to be created using 'git format-patch' or 'git diff', so > patches should be tested using 'git am' or 'git apply', not 'patch -pX'. This > causes false errors in the Jenkins patch tester. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14776) Rewrite smart-apply-patch.sh to use 'git am' or 'git apply' rather than 'patch'
[ https://issues.apache.org/jira/browse/HBASE-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992934#comment-14992934 ] Sean Busbey commented on HBASE-14776: - Can we make sure this is also solved in Yetus so that when there's a release we can move over to using it? > Rewrite smart-apply-patch.sh to use 'git am' or 'git apply' rather than > 'patch' > --- > > Key: HBASE-14776 > URL: https://issues.apache.org/jira/browse/HBASE-14776 > Project: HBase > Issue Type: Bug > Components: scripts >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14776.patch > > > We require patches to be created using 'git format-patch' or 'git diff', so > patches should be tested using 'git am' or 'git apply', not 'patch -pX'. This > causes false errors in the Jenkins patch tester. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14479) Apply the Leader/Followers pattern to RpcServer's Reader
[
https://issues.apache.org/jira/browse/HBASE-14479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992886#comment-14992886
]
Hiroshi Ikeda commented on HBASE-14479:
---
I didn't realize that a wrapped data by SASL might create a several tasks. When
we do a task within a received thread, we would make followers process the rest
of the tasks.
> Apply the Leader/Followers pattern to RpcServer's Reader
>
>
> Key: HBASE-14479
> URL: https://issues.apache.org/jira/browse/HBASE-14479
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: HBASE-14479-V2 (1).patch, HBASE-14479-V2.patch,
> HBASE-14479-V2.patch, HBASE-14479.patch, flamegraph-19152.svg,
> flamegraph-32667.svg, gc.png, gets.png, io.png, median.png
>
>
> {{RpcServer}} uses multiple selectors to read data for load distribution, but
> the distribution is just done by round-robin. It is uncertain, especially for
> long run, whether load is equally divided and resources are used without
> being wasted.
> Moreover, multiple selectors may cause excessive context switches which give
> priority to low latency (while we just add the requests to queues), and it is
> possible to reduce throughput of the whole server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992856#comment-14992856
]
Hiroshi Ikeda commented on HBASE-14279:
---
{code}
public Set values(K key) {
-return container.get(key);
+Map> seg = segments(key);
+return seg.get(key);
}
{code}
Defensively copy under the lock of the segment (the reason was already
commented a few month ago), and consequently using valueSetFactory is almost
meaningless:
{code}
+ Set set = seg.get(key);
+ if (set == null) {
+set = valueSetFactory.get();
+seg.put(key, set);
}
{code}
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch,
> HBASE-14279_v3.patch, LockStripedBag.java
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14774) Raise the font size on high-DPI small-screen devices like iphone 6+
[ https://issues.apache.org/jira/browse/HBASE-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992852#comment-14992852 ] Misty Stanley-Jones commented on HBASE-14774: - Cool. Do you think that looks too big? > Raise the font size on high-DPI small-screen devices like iphone 6+ > --- > > Key: HBASE-14774 > URL: https://issues.apache.org/jira/browse/HBASE-14774 > Project: HBase > Issue Type: Bug > Components: website >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14774.patch, image.jpg > > > On iPads and things like that, the website looks fine. But the fonts are too > small on high-DPI small screens. It's tiny on my iPhone 6+. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14755) Fix some broken links and HTML problems
[ https://issues.apache.org/jira/browse/HBASE-14755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-14755: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master. > Fix some broken links and HTML problems > --- > > Key: HBASE-14755 > URL: https://issues.apache.org/jira/browse/HBASE-14755 > Project: HBase > Issue Type: Bug > Components: documentation >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14755-v1.patch, HBASE-14755-v1.patch, > HBASE-14755.patch > > > Problems seen in > https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Website%20Link%20Ckecker/3/artifact/link_report/index.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14776) Rewrite smart-apply-patch.sh to use 'git am' or 'git apply' rather than 'patch'
[ https://issues.apache.org/jira/browse/HBASE-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-14776: Status: Patch Available (was: Open) > Rewrite smart-apply-patch.sh to use 'git am' or 'git apply' rather than > 'patch' > --- > > Key: HBASE-14776 > URL: https://issues.apache.org/jira/browse/HBASE-14776 > Project: HBase > Issue Type: Bug > Components: scripts >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14776.patch > > > We require patches to be created using 'git format-patch' or 'git diff', so > patches should be tested using 'git am' or 'git apply', not 'patch -pX'. This > causes false errors in the Jenkins patch tester. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14776) Rewrite smart-apply-patch.sh to use 'git am' or 'git apply' rather than 'patch'
[ https://issues.apache.org/jira/browse/HBASE-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-14776: Attachment: HBASE-14776.patch Tested it locally by running smart-apply-patch against a patch created with git format-patch and one created using git diff. Not sure how to test it in the context of the larger test-patch.sh script. > Rewrite smart-apply-patch.sh to use 'git am' or 'git apply' rather than > 'patch' > --- > > Key: HBASE-14776 > URL: https://issues.apache.org/jira/browse/HBASE-14776 > Project: HBase > Issue Type: Bug > Components: scripts >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14776.patch > > > We require patches to be created using 'git format-patch' or 'git diff', so > patches should be tested using 'git am' or 'git apply', not 'patch -pX'. This > causes false errors in the Jenkins patch tester. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14776) Rewrite smart-apply-patch.sh to use 'git am' or 'git apply' rather than 'patch'
[ https://issues.apache.org/jira/browse/HBASE-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-14776: Summary: Rewrite smart-apply-patch.sh to use 'git am' or 'git apply' rather than 'patch' (was: Rewrite smart-patch.sh to use 'git am' or 'git apply' rather than 'patch') > Rewrite smart-apply-patch.sh to use 'git am' or 'git apply' rather than > 'patch' > --- > > Key: HBASE-14776 > URL: https://issues.apache.org/jira/browse/HBASE-14776 > Project: HBase > Issue Type: Bug > Components: scripts >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > > We require patches to be created using 'git format-patch' or 'git diff', so > patches should be tested using 'git am' or 'git apply', not 'patch -pX'. This > causes false errors in the Jenkins patch tester. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14772) Improve zombie detector; be more discerning
[
https://issues.apache.org/jira/browse/HBASE-14772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14772:
--
Attachment: zombie.patch
Here's a start. It changes our tagging of surefire forked processes so instead
of hbase.test, its now is hbase.build.id=BUILD_ID where BUILD_ID is whatever we
pass maven on command line with -Dbuild.id. Was planning to pass in the jenkins
BUILD_ID. If no BUILD_ID, it goes with the maven build time as ISO8601.
The zombie detector script in this patch takes recording of processes
outstanding that have the current build id attached. It then waits 30 seconds.
It then checks for presence of these process ids ... if they are still around
(before, any surefire with the hbase.test flag would do... whether ours or not,
whether it was a new test or not). So, more discerning.
> Improve zombie detector; be more discerning
> ---
>
> Key: HBASE-14772
> URL: https://issues.apache.org/jira/browse/HBASE-14772
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
> Attachments: zombie.patch
>
>
> Currently, any surefire process with the hbase flag is a potential zombie.
> Our zombie check currently takes a reading and if it finds candidate zombies,
> it waits 30 seconds and then does another reading. If a concurrent build
> going on, in both cases the zombie detector will come up positive though the
> adjacent test run may be making progress; i.e. the cast of surefire processes
> may have changed between readings but our detector just sees presence of
> hbase surefire processes.
> Here is example:
> {code}
> Suspicious java process found - waiting 30s to see if there are just slow to
> stop
> There appear to be 5 zombie tests, they should have been killed by surefire
> but survived
> 12823 surefirebooter852180186418035480.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 7653 surefirebooter8579074445899448699.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 12614 surefirebooter136529596936417090.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 7836 surefirebooter3217047564606450448.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 13566 surefirebooter2084039411151963494.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> BEGIN zombies jstack extract
> END zombies jstack extract
> {code}
> 5 is the number of forked processes we allow when doing medium and large
> tests so an adjacent build will always show as '5 zombies'.
> Need to add discerning if list of processes changes between readings.
> Can I also add a tag per build run that all forked processes pick up so I can
> look at the current builds progeny only?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14774) Raise the font size on high-DPI small-screen devices like iphone 6+
[ https://issues.apache.org/jira/browse/HBASE-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-14774: --- Attachment: image.jpg here's on an iphone5 > Raise the font size on high-DPI small-screen devices like iphone 6+ > --- > > Key: HBASE-14774 > URL: https://issues.apache.org/jira/browse/HBASE-14774 > Project: HBase > Issue Type: Bug > Components: website >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14774.patch, image.jpg > > > On iPads and things like that, the website looks fine. But the fonts are too > small on high-DPI small screens. It's tiny on my iPhone 6+. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14773) Fix HBase shell tests are skipped when skipping server tests.
[ https://issues.apache.org/jira/browse/HBASE-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14773: - Description: Looks like a copy pasta error. (was: Looks like a copy paste error.) > Fix HBase shell tests are skipped when skipping server tests. > - > > Key: HBASE-14773 > URL: https://issues.apache.org/jira/browse/HBASE-14773 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14773.patch > > > Looks like a copy pasta error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14773) Fix HBase shell tests are skipped when skipping server tests.
[ https://issues.apache.org/jira/browse/HBASE-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992820#comment-14992820 ] Appy commented on HBASE-14773: -- Mmmm, delicious bug. > Fix HBase shell tests are skipped when skipping server tests. > - > > Key: HBASE-14773 > URL: https://issues.apache.org/jira/browse/HBASE-14773 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14773.patch > > > Looks like a copy paste error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14773) Fix HBase shell tests are skipped when skipping server tests.
[ https://issues.apache.org/jira/browse/HBASE-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark resolved HBASE-14773. --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.3.0 1.2.0 2.0.0 > Fix HBase shell tests are skipped when skipping server tests. > - > > Key: HBASE-14773 > URL: https://issues.apache.org/jira/browse/HBASE-14773 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14773.patch > > > Looks like a copy paste error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14773) Fix HBase shell tests are skipped when skipping server tests.
[ https://issues.apache.org/jira/browse/HBASE-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992788#comment-14992788 ] Elliott Clark commented on HBASE-14773: --- [~appy] http://knowyourmeme.com/memes/copypasta :-) > Fix HBase shell tests are skipped when skipping server tests. > - > > Key: HBASE-14773 > URL: https://issues.apache.org/jira/browse/HBASE-14773 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14773.patch > > > Looks like a copy paste error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14765) Remove snappy profile
[ https://issues.apache.org/jira/browse/HBASE-14765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992783#comment-14992783 ] Hudson commented on HBASE-14765: FAILURE: Integrated in HBase-Trunk_matrix #437 (See [https://builds.apache.org/job/HBase-Trunk_matrix/437/]) HBASE-14765 Remove snappy profile (eclark: rev 86da57f498c68703979dcd7ecc4b0ef2ce03723d) * src/main/asciidoc/_chapters/developer.adoc * pom.xml * hbase-shell/pom.xml * hbase-server/pom.xml * hbase-common/pom.xml > Remove snappy profile > - > > Key: HBASE-14765 > URL: https://issues.apache.org/jira/browse/HBASE-14765 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14765.patch > > > Snappy is provided by hadoop and has been for, a long while. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14773) Fix HBase shell tests are skipped when skipping server tests.
[ https://issues.apache.org/jira/browse/HBASE-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14773: - Description: Looks like a copy paste error. (was: Looks like a copy pasta error.) > Fix HBase shell tests are skipped when skipping server tests. > - > > Key: HBASE-14773 > URL: https://issues.apache.org/jira/browse/HBASE-14773 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-14773.patch > > > Looks like a copy paste error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14766) WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with the new low-cost implementation.
[
https://issues.apache.org/jira/browse/HBASE-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992774#comment-14992774
]
Hadoop QA commented on HBASE-14766:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770874/HBASE-14766-v003.patch
against master branch at commit 86da57f498c68703979dcd7ecc4b0ef2ce03723d.
ATTACHMENT ID: 12770874
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are possible 1 zombie
test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16416//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16416//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16416//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16416//console
This message is automatically generated.
> WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with
> the new low-cost implementation.
> --
>
> Key: HBASE-14766
> URL: https://issues.apache.org/jira/browse/HBASE-14766
> Project: HBase
> Issue Type: Improvement
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: HBASE-14766-v001.patch, HBASE-14766-v002.patch,
> HBASE-14766-v003.patch
>
>
> Cell's getFamily() gets an array copy of the cell's family, while in the
> filter function, it just needs to peek into the family and do a compare.
> Replace
> Bytes.toString(cell.getFamily())
> with
> Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength())
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14776) Rewrite smart-patch.sh to use 'git am' or 'git apply' rather than 'patch'
Misty Stanley-Jones created HBASE-14776: --- Summary: Rewrite smart-patch.sh to use 'git am' or 'git apply' rather than 'patch' Key: HBASE-14776 URL: https://issues.apache.org/jira/browse/HBASE-14776 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 2.0.0 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Fix For: 2.0.0 We require patches to be created using 'git format-patch' or 'git diff', so patches should be tested using 'git am' or 'git apply', not 'patch -pX'. This causes false errors in the Jenkins patch tester. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14775) Replication can't authenticate with peer Zookeeper with different server principal
Gary Helmling created HBASE-14775:
-
Summary: Replication can't authenticate with peer Zookeeper with
different server principal
Key: HBASE-14775
URL: https://issues.apache.org/jira/browse/HBASE-14775
Project: HBase
Issue Type: Bug
Reporter: Gary Helmling
Assignee: Gary Helmling
When replication is setup with security, where the local ZK cluster and peer ZK
cluster use different server principals, the source HBase cluster is unable to
authenticate with the peer ZK cluster.
When ZK is configured for SASL authentication and a server principal other than
the default ("zookeeper") is used, the correct server principal must be
specified on the client as a system property -- the confusingly named
{{zookeeper.sasl.client.username}}. However, since this is given as a system
property, authentication with the peer cluster breaks when it uses a different
ZK server principal than the local cluster.
We need a way of tying this setting to the replication peer config and then
setting the property when the peer's ZooKeeperWatcher is created.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14775) Replication can't authenticate with peer Zookeeper with different server principal
[
https://issues.apache.org/jira/browse/HBASE-14775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Helmling updated HBASE-14775:
--
Component/s: security
Replication
> Replication can't authenticate with peer Zookeeper with different server
> principal
> --
>
> Key: HBASE-14775
> URL: https://issues.apache.org/jira/browse/HBASE-14775
> Project: HBase
> Issue Type: Bug
> Components: Replication, security
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>
> When replication is setup with security, where the local ZK cluster and peer
> ZK cluster use different server principals, the source HBase cluster is
> unable to authenticate with the peer ZK cluster.
> When ZK is configured for SASL authentication and a server principal other
> than the default ("zookeeper") is used, the correct server principal must be
> specified on the client as a system property -- the confusingly named
> {{zookeeper.sasl.client.username}}. However, since this is given as a system
> property, authentication with the peer cluster breaks when it uses a
> different ZK server principal than the local cluster.
> We need a way of tying this setting to the replication peer config and then
> setting the property when the peer's ZooKeeperWatcher is created.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14774) Raise the font size on high-DPI small-screen devices like iphone 6+
[ https://issues.apache.org/jira/browse/HBASE-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-14774: Status: Patch Available (was: Open) > Raise the font size on high-DPI small-screen devices like iphone 6+ > --- > > Key: HBASE-14774 > URL: https://issues.apache.org/jira/browse/HBASE-14774 > Project: HBase > Issue Type: Bug > Components: website >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14774.patch > > > On iPads and things like that, the website looks fine. But the fonts are too > small on high-DPI small screens. It's tiny on my iPhone 6+. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14774) Raise the font size on high-DPI small-screen devices like iphone 6+
[ https://issues.apache.org/jira/browse/HBASE-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-14774: Attachment: HBASE-14774.patch This patch raises the font size a bit if the screen size is less than 768px and the density is more than 1.5. > Raise the font size on high-DPI small-screen devices like iphone 6+ > --- > > Key: HBASE-14774 > URL: https://issues.apache.org/jira/browse/HBASE-14774 > Project: HBase > Issue Type: Bug > Components: website >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14774.patch > > > On iPads and things like that, the website looks fine. But the fonts are too > small on high-DPI small screens. It's tiny on my iPhone 6+. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14774) Raise the font size on high-DPI small-screen devices like iphone 6+
Misty Stanley-Jones created HBASE-14774: --- Summary: Raise the font size on high-DPI small-screen devices like iphone 6+ Key: HBASE-14774 URL: https://issues.apache.org/jira/browse/HBASE-14774 Project: HBase Issue Type: Bug Components: website Affects Versions: 2.0.0 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Fix For: 2.0.0 On iPads and things like that, the website looks fine. But the fonts are too small on high-DPI small screens. It's tiny on my iPhone 6+. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14765) Remove snappy profile
[ https://issues.apache.org/jira/browse/HBASE-14765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992736#comment-14992736 ] Hudson commented on HBASE-14765: SUCCESS: Integrated in HBase-1.3-IT #296 (See [https://builds.apache.org/job/HBase-1.3-IT/296/]) HBASE-14765 Remove snappy profile (eclark: rev 2b84c40f014bc57808f726412b976bb67e337f0b) * hbase-common/pom.xml * hbase-shell/pom.xml * pom.xml * hbase-server/pom.xml * src/main/asciidoc/_chapters/developer.adoc > Remove snappy profile > - > > Key: HBASE-14765 > URL: https://issues.apache.org/jira/browse/HBASE-14765 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14765.patch > > > Snappy is provided by hadoop and has been for, a long while. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14427) Fix 'should' assertions in TestFastFail
[ https://issues.apache.org/jira/browse/HBASE-14427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992733#comment-14992733 ] stack commented on HBASE-14427: --- [~abhishek.chouhan] Want me to apply both patches then sir? The first for fix and the second to reenable the test? Let me put them together and get some test runs in here. > Fix 'should' assertions in TestFastFail > --- > > Key: HBASE-14427 > URL: https://issues.apache.org/jira/browse/HBASE-14427 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: Abhishek Singh Chouhan >Priority: Minor > Labels: beginner > Fix For: 2.0.0 > > Attachments: HBASE-14427.addendum.patch, HBASE-14427.patch > > > Over in HBASE-14421, TestFastFail has been failing assertions that talk of > events that 'should' be happening. Fix. For now HBASE-14421 has disabled the > 'should' assertions. They seem fine on apache jenkins build but fail fairly > reliably for me on alternate HW. > To address, get familiar with the test. Change the commented out asserts to > be yes/no instead of a 'likely' (On a cursory scan, it is possible that a > test run may not involve preemption and it is these runs that are throwing > asserts). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14755) Fix some broken links and HTML problems
[ https://issues.apache.org/jira/browse/HBASE-14755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992728#comment-14992728 ] stack commented on HBASE-14755: --- +1 > Fix some broken links and HTML problems > --- > > Key: HBASE-14755 > URL: https://issues.apache.org/jira/browse/HBASE-14755 > Project: HBase > Issue Type: Bug > Components: documentation >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14755-v1.patch, HBASE-14755-v1.patch, > HBASE-14755.patch > > > Problems seen in > https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Website%20Link%20Ckecker/3/artifact/link_report/index.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14763) Remove usages of deprecated HConnection
[ https://issues.apache.org/jira/browse/HBASE-14763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14763: -- Attachment: hbase-14763.v3.patch > Remove usages of deprecated HConnection > > > Key: HBASE-14763 > URL: https://issues.apache.org/jira/browse/HBASE-14763 > Project: HBase > Issue Type: Bug > Components: hbase >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-14763.patch, hbase-14763.v2.patch, > hbase-14763.v3.patch, hbase-14763.v3.patch > > > HConnection was deprecated in 1.0.0. There are two interfaces that are > supposed to be used instead -- Connection for client programs and > ClusterConnection for internal hbaes and special tools (LoadIncremental, > HBCK, etc). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14773) Fix HBase shell tests are skipped when skipping server tests.
[ https://issues.apache.org/jira/browse/HBASE-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992723#comment-14992723 ] stack commented on HBASE-14773: --- +1 > Fix HBase shell tests are skipped when skipping server tests. > - > > Key: HBASE-14773 > URL: https://issues.apache.org/jira/browse/HBASE-14773 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-14773.patch > > > Looks like a copy pasta error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14763) Remove usages of deprecated HConnection
[ https://issues.apache.org/jira/browse/HBASE-14763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992721#comment-14992721 ] stack commented on HBASE-14763: --- Failed here: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test (secondPartTestsExecution) on project hbase-server: ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: org.apache.maven.surefire.report.ReporterException: When writing xml report stdout/stderr: /tmp/stderr2779164428906562928deferred (No such file or directory) -> [Help 1] I've not seen this one before. Retrying. > Remove usages of deprecated HConnection > > > Key: HBASE-14763 > URL: https://issues.apache.org/jira/browse/HBASE-14763 > Project: HBase > Issue Type: Bug > Components: hbase >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-14763.patch, hbase-14763.v2.patch, > hbase-14763.v3.patch > > > HConnection was deprecated in 1.0.0. There are two interfaces that are > supposed to be used instead -- Connection for client programs and > ClusterConnection for internal hbaes and special tools (LoadIncremental, > HBCK, etc). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14771) RpcServer.getRemoteAddress always returns null.
[
https://issues.apache.org/jira/browse/HBASE-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992714#comment-14992714
]
Appy commented on HBASE-14771:
--
getRemoteAddress() always returns null because CurCall is thread local variable
and is only set in threads responsible for executing CallRunners (executor in
FifoRpcScheduler and those being created in RpcExecutor.startHandlers).
RpcServer.getRemoteIp also uses CurCall, so it'll always return null here.
Using {code}this.addr{code} makes sense.
It'll definitely fix 2 of the 3 uses of getRemoteAddress(). I am not sure about
the use in AccessController since I don't know it's internals.
[~a72877] what do you mean by "... fixing this issue"? Where are you seeing the
null values which change to legitimate address after this change?
> RpcServer.getRemoteAddress always returns null.
> ---
>
> Key: HBASE-14771
> URL: https://issues.apache.org/jira/browse/HBASE-14771
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
>Affects Versions: 1.2.0
>Reporter: Abhishek Kumar
>Assignee: Abhishek Kumar
>Priority: Minor
> Attachments: HBASE-14771.patch
>
>
> RpcServer.getRemoteAddress always returns null, because Call object is
> getting initialized with null.This seems to be happening because of using
> RpcServer.getRemoteIp() in Call object constructor before RpcServer thread
> local 'CurCall' being set in CallRunner.run method:
> {noformat}
> // --- RpcServer.java ---
> protected void processRequest(byte[] buf) throws IOException,
> InterruptedException {
> .
> // Call object getting initialized here with address
> // obtained from RpcServer.getRemoteIp()
> Call call = new Call(id, this.service, md, header, param, cellScanner, this,
> responder,
> totalRequestSize, traceInfo, RpcServer.getRemoteIp());
> scheduler.dispatch(new CallRunner(RpcServer.this, call));
> }
> // getRemoteIp method gets address from threadlocal 'CurCall' which
> // gets set in CallRunner.run and calling it before this as in above case,
> will return null
> // --- CallRunner.java ---
> public void run() {
> .
> Pair resultPair = null;
> RpcServer.CurCall.set(call);
> ..
> }
> // Using 'this.addr' in place of getRemoteIp method in RpcServer.java seems
> to be fixing this issue
> Call call = new Call(id, this.service, md, header, param, cellScanner, this,
> responder,
> totalRequestSize, traceInfo, this.addr);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14765) Remove snappy profile
[ https://issues.apache.org/jira/browse/HBASE-14765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992712#comment-14992712 ] Hudson commented on HBASE-14765: SUCCESS: Integrated in HBase-1.3 #350 (See [https://builds.apache.org/job/HBase-1.3/350/]) HBASE-14765 Remove snappy profile (eclark: rev 2b84c40f014bc57808f726412b976bb67e337f0b) * hbase-server/pom.xml * src/main/asciidoc/_chapters/developer.adoc * pom.xml * hbase-shell/pom.xml * hbase-common/pom.xml > Remove snappy profile > - > > Key: HBASE-14765 > URL: https://issues.apache.org/jira/browse/HBASE-14765 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14765.patch > > > Snappy is provided by hadoop and has been for, a long while. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14773) Fix HBase shell tests are skipped when skipping server tests.
[ https://issues.apache.org/jira/browse/HBASE-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14773: -- Attachment: HBASE-14773.patch > Fix HBase shell tests are skipped when skipping server tests. > - > > Key: HBASE-14773 > URL: https://issues.apache.org/jira/browse/HBASE-14773 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-14773.patch > > > Looks like a copy pasta error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14773) Fix HBase shell tests are skipped when skipping server tests.
Elliott Clark created HBASE-14773: - Summary: Fix HBase shell tests are skipped when skipping server tests. Key: HBASE-14773 URL: https://issues.apache.org/jira/browse/HBASE-14773 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Looks like a copy pasta error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14765) Remove snappy profile
[ https://issues.apache.org/jira/browse/HBASE-14765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992670#comment-14992670 ] Hudson commented on HBASE-14765: FAILURE: Integrated in HBase-1.2 #350 (See [https://builds.apache.org/job/HBase-1.2/350/]) HBASE-14765 Remove snappy profile (eclark: rev b378b3459da34a64633fb41aeac157c006756267) * hbase-server/pom.xml * src/main/asciidoc/_chapters/developer.adoc * pom.xml * hbase-shell/pom.xml * hbase-common/pom.xml > Remove snappy profile > - > > Key: HBASE-14765 > URL: https://issues.apache.org/jira/browse/HBASE-14765 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14765.patch > > > Snappy is provided by hadoop and has been for, a long while. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14763) Remove usages of deprecated HConnection
[
https://issues.apache.org/jira/browse/HBASE-14763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992654#comment-14992654
]
Hadoop QA commented on HBASE-14763:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770855/hbase-14763.v3.patch
against master branch at commit 86da57f498c68703979dcd7ecc4b0ef2ce03723d.
ATTACHMENT ID: 12770855
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 75 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:red}-1 javac{color}. The applied patch generated 36 javac compiler
warnings (more than the master's current 35 warnings).
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16415//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16415//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16415//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16415//console
This message is automatically generated.
> Remove usages of deprecated HConnection
>
>
> Key: HBASE-14763
> URL: https://issues.apache.org/jira/browse/HBASE-14763
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-14763.patch, hbase-14763.v2.patch,
> hbase-14763.v3.patch
>
>
> HConnection was deprecated in 1.0.0. There are two interfaces that are
> supposed to be used instead -- Connection for client programs and
> ClusterConnection for internal hbaes and special tools (LoadIncremental,
> HBCK, etc).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14708) Use copy on write Map for region location cache
[ https://issues.apache.org/jira/browse/HBASE-14708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992568#comment-14992568 ] Sean Busbey commented on HBASE-14708: - the plan sounds reasonable for 1.2. > Use copy on write Map for region location cache > --- > > Key: HBASE-14708 > URL: https://issues.apache.org/jira/browse/HBASE-14708 > Project: HBase > Issue Type: Improvement > Components: Client >Affects Versions: 1.1.2 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14708-v10.patch, HBASE-14708-v11.patch, > HBASE-14708-v12.patch, HBASE-14708-v13.patch, HBASE-14708-v2.patch, > HBASE-14708-v3.patch, HBASE-14708-v4.patch, HBASE-14708-v5.patch, > HBASE-14708-v6.patch, HBASE-14708-v7.patch, HBASE-14708-v8.patch, > HBASE-14708-v9.patch, HBASE-14708.patch, anotherbench.zip, > location_cache_times.pdf, result.csv > > > Internally a co-worker profiled their application that was talking to HBase. > > 60% of the time was spent in locating a region. This was while the cluster > was stable and no regions were moving. > To figure out if there was a faster way to cache region location I wrote up a > benchmark here: https://github.com/elliottneilclark/benchmark-hbase-cache > This tries to simulate a heavy load on the location cache. > * 24 different threads. > * 2 Deleting location data > * 2 Adding location data > * Using floor to get the result. > To repeat my work just run ./run.sh and it should produce a result.csv > Results: > ConcurrentSkiplistMap is a good middle ground. It's got equal speed for > reading and writing. > However most operations will not need to remove or add a region location. > There will be potentially several orders of magnitude more reads for cached > locations than there will be on clearing the cache. > So I propose a copy on write tree map. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14708) Use copy on write Map for region location cache
[ https://issues.apache.org/jira/browse/HBASE-14708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992554#comment-14992554 ] Elliott Clark commented on HBASE-14708: --- [~enis] Thoughts on this? You seem to be the last person to touch MetaCache. [~busbey] Thoughts on the plan for branch-1.2 ? > Use copy on write Map for region location cache > --- > > Key: HBASE-14708 > URL: https://issues.apache.org/jira/browse/HBASE-14708 > Project: HBase > Issue Type: Improvement > Components: Client >Affects Versions: 1.1.2 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14708-v10.patch, HBASE-14708-v11.patch, > HBASE-14708-v12.patch, HBASE-14708-v13.patch, HBASE-14708-v2.patch, > HBASE-14708-v3.patch, HBASE-14708-v4.patch, HBASE-14708-v5.patch, > HBASE-14708-v6.patch, HBASE-14708-v7.patch, HBASE-14708-v8.patch, > HBASE-14708-v9.patch, HBASE-14708.patch, anotherbench.zip, > location_cache_times.pdf, result.csv > > > Internally a co-worker profiled their application that was talking to HBase. > > 60% of the time was spent in locating a region. This was while the cluster > was stable and no regions were moving. > To figure out if there was a faster way to cache region location I wrote up a > benchmark here: https://github.com/elliottneilclark/benchmark-hbase-cache > This tries to simulate a heavy load on the location cache. > * 24 different threads. > * 2 Deleting location data > * 2 Adding location data > * Using floor to get the result. > To repeat my work just run ./run.sh and it should produce a result.csv > Results: > ConcurrentSkiplistMap is a good middle ground. It's got equal speed for > reading and writing. > However most operations will not need to remove or add a region location. > There will be potentially several orders of magnitude more reads for cached > locations than there will be on clearing the cache. > So I propose a copy on write tree map. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14708) Use copy on write Map for region location cache
[
https://issues.apache.org/jira/browse/HBASE-14708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992489#comment-14992489
]
Hadoop QA commented on HBASE-14708:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770848/HBASE-14708-v13.patch
against master branch at commit 050ebe850b32057860fb94b46f955352db139db1.
ATTACHMENT ID: 12770848
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1727 checkstyle errors (more than the master's current 1726 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16413//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16413//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16413//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16413//console
This message is automatically generated.
> Use copy on write Map for region location cache
> ---
>
> Key: HBASE-14708
> URL: https://issues.apache.org/jira/browse/HBASE-14708
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Affects Versions: 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14708-v10.patch, HBASE-14708-v11.patch,
> HBASE-14708-v12.patch, HBASE-14708-v13.patch, HBASE-14708-v2.patch,
> HBASE-14708-v3.patch, HBASE-14708-v4.patch, HBASE-14708-v5.patch,
> HBASE-14708-v6.patch, HBASE-14708-v7.patch, HBASE-14708-v8.patch,
> HBASE-14708-v9.patch, HBASE-14708.patch, anotherbench.zip,
> location_cache_times.pdf, result.csv
>
>
> Internally a co-worker profiled their application that was talking to HBase.
> > 60% of the time was spent in locating a region. This was while the cluster
> was stable and no regions were moving.
> To figure out if there was a faster way to cache region location I wrote up a
> benchmark here: https://github.com/elliottneilclark/benchmark-hbase-cache
> This tries to simulate a heavy load on the location cache.
> * 24 different threads.
> * 2 Deleting location data
> * 2 Adding location data
> * Using floor to get the result.
> To repeat my work just run ./run.sh and it should produce a result.csv
> Results:
> ConcurrentSkiplistMap is a good middle ground. It's got equal speed for
> reading and writing.
> However most operations will not need to remove or add a region location.
> There will be potentially several orders of magnitude more reads for cached
> locations than there will be on clearing the cache.
> So I propose a copy on write tree map.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14765) Remove snappy profile
[ https://issues.apache.org/jira/browse/HBASE-14765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992481#comment-14992481 ] Hudson commented on HBASE-14765: SUCCESS: Integrated in HBase-1.2-IT #266 (See [https://builds.apache.org/job/HBase-1.2-IT/266/]) HBASE-14765 Remove snappy profile (eclark: rev b378b3459da34a64633fb41aeac157c006756267) * hbase-shell/pom.xml * hbase-server/pom.xml * src/main/asciidoc/_chapters/developer.adoc * pom.xml * hbase-common/pom.xml > Remove snappy profile > - > > Key: HBASE-14765 > URL: https://issues.apache.org/jira/browse/HBASE-14765 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14765.patch > > > Snappy is provided by hadoop and has been for, a long while. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14766) WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with the new low-cost implementation.
[ https://issues.apache.org/jira/browse/HBASE-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-14766: - Attachment: HBASE-14766-v003.patch Address checkstyle errors. > WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with > the new low-cost implementation. > -- > > Key: HBASE-14766 > URL: https://issues.apache.org/jira/browse/HBASE-14766 > Project: HBase > Issue Type: Improvement >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-14766-v001.patch, HBASE-14766-v002.patch, > HBASE-14766-v003.patch > > > Cell's getFamily() gets an array copy of the cell's family, while in the > filter function, it just needs to peek into the family and do a compare. > Replace > Bytes.toString(cell.getFamily()) > with > Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(), > cell.getFamilyLength()) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14755) Fix some broken links and HTML problems
[
https://issues.apache.org/jira/browse/HBASE-14755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992458#comment-14992458
]
Hadoop QA commented on HBASE-14755:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770846/HBASE-14755-v1.patch
against master branch at commit 050ebe850b32057860fb94b46f955352db139db1.
ATTACHMENT ID: 12770846
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ The full HBase API test code, including private
and unstable APIs
+
org.apache.hadoop.hbase.backup*:org.apache.hadoop.hbase.catalog:org.apache.hadoop.hbase.client.coprocessor:org.apache.hadoop.hbase.client.metrics:org.apache.hadoop.hbase.codec*:org.apache.hadoop.hbase.constraint:org.apache.hadoop.hbase.coprocessor.*:org.apache.hadoop.hbase.executor:org.apache.hadoop.hbase.fs:*.generated.*:org.apache.hadoop.hbase.io.hfile.*:org.apache.hadoop.hbase.mapreduce.hadoopbackport:org.apache.hadoop.hbase.mapreduce.replication:org.apache.hadoop.hbase.master.*:org.apache.hadoop.hbase.metrics*:org.apache.hadoop.hbase.migration:org.apache.hadoop.hbase.monitoring:org.apache.hadoop.hbase.p*:org.apache.hadoop.hbase.regionserver.compactions:org.apache.hadoop.hbase.regionserver.handler:org.apache.hadoop.hbase.regionserver.snapshot:org.apache.hadoop.hbase.replication.*:org.apache.hadoop.hbase.rest.filter:org.apache.hadoop.hbase.rest.model:org.apache.hadoop.hbase.rest.p*:org.apache.hadoop.hbase.security.*:org.apache.hadoop.hbase.thrift*:org.apache.hadoop.hbase.tmpl.*:org.apache.hadoop.hbase.tool:org.apache.hadoop.hbase.trace:org.apache.hadoop.hbase.util.byterange*:org.apache.hadoop.hbase.util.test:org.apache.hadoop.hbase.util.vint:org.apache.hadoop.hbase.zookeeper.lock:org.apache.hadoop.metrics2*:org.apache.hadoop.hbase.io.compress*
+
org.apache.hadoop.hbase.backup*:org.apache.hadoop.hbase.catalog:org.apache.hadoop.hbase.client.coprocessor:org.apache.hadoop.hbase.client.metrics:org.apache.hadoop.hbase.codec*:org.apache.hadoop.hbase.constraint:org.apache.hadoop.hbase.coprocessor.*:org.apache.hadoop.hbase.executor:org.apache.hadoop.hbase.fs:*.generated.*:org.apache.hadoop.hbase.io.hfile.*:org.apache.hadoop.hbase.mapreduce.hadoopbackport:org.apache.hadoop.hbase.mapreduce.replication:org.apache.hadoop.hbase.master.*:org.apache.hadoop.hbase.metrics*:org.apache.hadoop.hbase.migration:org.apache.hadoop.hbase.monitoring:org.apache.hadoop.hbase.p*:org.apache.hadoop.hbase.regionserver.compactions:org.apache.hadoop.hbase.regionserver.handler:org.apache.hadoop.hbase.regionserver.snapshot:org.apache.hadoop.hbase.replication.*:org.apache.hadoop.hbase.rest.filter:org.apache.hadoop.hbase.rest.model:org.apache.hadoop.hbase.rest.p*:org.apache.hadoop.hbase.security.*:org.apache.hadoop.hbase.thrift*:org.apache.hadoop.hbase.tmpl.*:org.apache.hadoop.hbase.tool:org.apache.hadoop.hbase.trace:org.apache.hadoop.hbase.util.byterange*:org.apache.hadoop.hbase.util.test:org.apache.hadoop.hbase.util.vint:org.apache.hadoop.hbase.zookeeper.lock:org.apache.hadoop.metrics2*:org.apache.hadoop.hbase.io.compress*
+. 8 bytes: Block type, a sequence of bytes equivalent to version 1's "magic
records". Supported block types are:
+ As opposed to the above, this is not an HFile v2 block but a fixed>size (for
each HFile version) data structure
+We are supporting "meta" blocks in version 2 the same way they were supported
in version 1, even though we do not store Bloom filter data in these blocks
anymore.
+There are three types of block indexes in HFile version 2, stored in two
different formats (root and non-root):
+.. Optionally, version 2 intermediate levels, stored in the non%root format in
the data index section of the file. Intermediate levels can only be present
if leaf level blocks are pres
[jira] [Commented] (HBASE-14463) Severe performance downgrade when parallel reading a single key from BucketCache
[
https://issues.apache.org/jira/browse/HBASE-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992448#comment-14992448
]
Anoop Sam John commented on HBASE-14463:
Sorry for the delay.
I did the tests again by fixing the RKs what we fetch (via multi get) every
time. ( Not exactly this way as in patchpe_use_same_keys.patch. instead of
doing a random RK generation, I can use the incoming int 'i' to testRow(final
int i) to make RKs )
Both with and with out patch gives almost similar avg total run time for
threads. So in my test, I have ensured that every thread doing distinct RK
fetch only. And there also we don't degrade perf.
So here is my +1
Thanks for the perseverance [~carp84]
> Severe performance downgrade when parallel reading a single key from
> BucketCache
>
>
> Key: HBASE-14463
> URL: https://issues.apache.org/jira/browse/HBASE-14463
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.14, 1.1.2
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.17
>
> Attachments: GC_with_WeakObjectPool.png, HBASE-14463.patch,
> HBASE-14463_v11.patch, HBASE-14463_v12.patch, HBASE-14463_v2.patch,
> HBASE-14463_v3.patch, HBASE-14463_v4.patch, HBASE-14463_v5.patch,
> TestBucketCache-new_with_IdLock.png,
> TestBucketCache-new_with_IdReadWriteLock.png,
> TestBucketCache_with_IdLock-latest.png, TestBucketCache_with_IdLock.png,
> TestBucketCache_with_IdReadWriteLock-latest.png,
> TestBucketCache_with_IdReadWriteLock-resolveLockLeak.png,
> TestBucketCache_with_IdReadWriteLock.png, pe_use_same_keys.patch,
> test-results.tar.gz
>
>
> We store feature data of online items in HBase, do machine learning on these
> features, and supply the outputs to our online search engine. In such
> scenario we will launch hundreds of yarn workers and each worker will read
> all features of one item(i.e. single rowkey in HBase), so there'll be heavy
> parallel reading on a single rowkey.
> We were using LruCache but start to try BucketCache recently to resolve gc
> issue, and just as titled we have observed severe performance downgrade.
> After some analytics we found the root cause is the lock in
> BucketCache#getBlock, as shown below
> {code}
> try {
> lockEntry = offsetLock.getLockEntry(bucketEntry.offset());
> // ...
> if (bucketEntry.equals(backingMap.get(key))) {
> // ...
> int len = bucketEntry.getLength();
> Cacheable cachedBlock = ioEngine.read(bucketEntry.offset(), len,
> bucketEntry.deserializerReference(this.deserialiserMap));
> {code}
> Since ioEnging.read involves array copy, it's much more time-costed than the
> operation in LruCache. And since we're using synchronized in
> IdLock#getLockEntry, parallel read dropping on the same bucket would be
> executed in serial, which causes a really bad performance.
> To resolve the problem, we propose to use ReentranceReadWriteLock in
> BucketCache, and introduce a new class called IdReadWriteLock to implement it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14766) WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with the new low-cost implementation.
[
https://issues.apache.org/jira/browse/HBASE-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992424#comment-14992424
]
Hadoop QA commented on HBASE-14766:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770845/HBASE-14766-v002.patch
against master branch at commit 050ebe850b32057860fb94b46f955352db139db1.
ATTACHMENT ID: 12770845
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1728 checkstyle errors (more than the master's current 1726 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16412//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16412//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16412//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16412//console
This message is automatically generated.
> WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with
> the new low-cost implementation.
> --
>
> Key: HBASE-14766
> URL: https://issues.apache.org/jira/browse/HBASE-14766
> Project: HBase
> Issue Type: Improvement
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: HBASE-14766-v001.patch, HBASE-14766-v002.patch
>
>
> Cell's getFamily() gets an array copy of the cell's family, while in the
> filter function, it just needs to peek into the family and do a compare.
> Replace
> Bytes.toString(cell.getFamily())
> with
> Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength())
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13982) Add info for visibility labels/cell TTLs to ImportTsv
[
https://issues.apache.org/jira/browse/HBASE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992423#comment-14992423
]
Hadoop QA commented on HBASE-13982:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12770860/HBASE_13982_1.patch
against master branch at commit 86da57f498c68703979dcd7ecc4b0ef2ce03723d.
ATTACHMENT ID: 12770860
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16414//console
This message is automatically generated.
> Add info for visibility labels/cell TTLs to ImportTsv
> -
>
> Key: HBASE-13982
> URL: https://issues.apache.org/jira/browse/HBASE-13982
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 1.1.0.1
>Reporter: Lars George
>Assignee: NIDHI GAMBHIR
> Labels: beginner
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13982.patch, HBASE_13982_1.patch
>
>
> HBASE-9832 added support for two more optional, special TSV columns, but no
> usage info was added. Add.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12072) Standardize retry handling for master operations
[
https://issues.apache.org/jira/browse/HBASE-12072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992342#comment-14992342
]
Apekshit Sharma commented on HBASE-12072:
-
You're right, i should have seen the pattern and corrected myself on seeing the
same thing in 3 jiras.
To confirm, we add unreleased version corresponding to every brach the change
was pushed, right?
If yes, seems like 0.99.2 was made from 1.0 branch.
Also let me know if I should revert the changes i made.
> Standardize retry handling for master operations
>
>
> Key: HBASE-12072
> URL: https://issues.apache.org/jira/browse/HBASE-12072
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.6
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 1.0.0, 2.0.0, 0.99.2
>
> Attachments: 12072-v1.txt, 12072-v2.txt, hbase-12072_v1.patch,
> hbase-12072_v2.patch, hbase-12072_v2.patch, hbase-12072_v3.patch
>
>
> For master requests, there are two retry mechanisms in effect. The first one
> is from HBaseAdmin.executeCallable()
> {code}
> private V executeCallable(MasterCallable callable) throws
> IOException {
> RpcRetryingCaller caller = rpcCallerFactory.newCaller();
> try {
> return caller.callWithRetries(callable);
> } finally {
> callable.close();
> }
> }
> {code}
> And inside, the other one is from StubMaker.makeStub():
> {code}
> /**
>* Create a stub against the master. Retry if necessary.
>* @return A stub to do intf against the master
>* @throws MasterNotRunningException
>*/
> @edu.umd.cs.findbugs.annotations.SuppressWarnings
> (value="SWL_SLEEP_WITH_LOCK_HELD")
> Object makeStub() throws MasterNotRunningException {
> {code}
> The tests will just hang for 10 min * 35 ~= 6hours.
> {code}
> 2014-09-23 16:19:05,151 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 1 of 35
> failed; retrying after sleep of 100, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:05,253 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 2 of 35
> failed; retrying after sleep of 200, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:05,456 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 3 of 35
> failed; retrying after sleep of 300, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:05,759 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 4 of 35
> failed; retrying after sleep of 500, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:06,262 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 5 of 35
> failed; retrying after sleep of 1008, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:07,273 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 6 of 35
> failed; retrying after sleep of 2011, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:09,286 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 7 of 35
> failed; retrying after sleep of 4012, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:13,303 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 8 of 35
> failed; retrying after sleep of 10033, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:23,343 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 9 of 35
> failed; retrying after sleep of 10089, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:33,439 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 10 of
> 35 failed; retrying after sleep of 10027, exception=java.io.IOException:
> Can't get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:43,473 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 11 of
> 35 failed; retrying after sleep of 10004, exception=java.io.IOException:
> Can't get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:53,485 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 12 of
> 35 failed; retrying after sleep of 20160, exception=ja
[jira] [Updated] (HBASE-13982) Add info for visibility labels/cell TTLs to ImportTsv
[ https://issues.apache.org/jira/browse/HBASE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] NIDHI GAMBHIR updated HBASE-13982: -- Attachment: HBASE_13982_1.patch Fixed the line length issue. > Add info for visibility labels/cell TTLs to ImportTsv > - > > Key: HBASE-13982 > URL: https://issues.apache.org/jira/browse/HBASE-13982 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 1.1.0.1 >Reporter: Lars George >Assignee: NIDHI GAMBHIR > Labels: beginner > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13982.patch, HBASE_13982_1.patch > > > HBASE-9832 added support for two more optional, special TSV columns, but no > usage info was added. Add. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12072) Standardize retry handling for master operations
[
https://issues.apache.org/jira/browse/HBASE-12072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992306#comment-14992306
]
Enis Soztutar commented on HBASE-12072:
---
[~appy] can you please not add the fixVersions=1.0.0 to jiras with fix versions
= 0.99.x.
The fixVersions tracks the first release that the issue appeared in. Since we
have done 0.99.x releases before 1.0.0, some of the jiras that you recently
modified appeared on earlier 0.99.x releases.
Let me know whether this helps:
http://markmail.org/message/u43qluenc7soxloe
> Standardize retry handling for master operations
>
>
> Key: HBASE-12072
> URL: https://issues.apache.org/jira/browse/HBASE-12072
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.6
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 1.0.0, 2.0.0, 0.99.2
>
> Attachments: 12072-v1.txt, 12072-v2.txt, hbase-12072_v1.patch,
> hbase-12072_v2.patch, hbase-12072_v2.patch, hbase-12072_v3.patch
>
>
> For master requests, there are two retry mechanisms in effect. The first one
> is from HBaseAdmin.executeCallable()
> {code}
> private V executeCallable(MasterCallable callable) throws
> IOException {
> RpcRetryingCaller caller = rpcCallerFactory.newCaller();
> try {
> return caller.callWithRetries(callable);
> } finally {
> callable.close();
> }
> }
> {code}
> And inside, the other one is from StubMaker.makeStub():
> {code}
> /**
>* Create a stub against the master. Retry if necessary.
>* @return A stub to do intf against the master
>* @throws MasterNotRunningException
>*/
> @edu.umd.cs.findbugs.annotations.SuppressWarnings
> (value="SWL_SLEEP_WITH_LOCK_HELD")
> Object makeStub() throws MasterNotRunningException {
> {code}
> The tests will just hang for 10 min * 35 ~= 6hours.
> {code}
> 2014-09-23 16:19:05,151 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 1 of 35
> failed; retrying after sleep of 100, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:05,253 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 2 of 35
> failed; retrying after sleep of 200, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:05,456 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 3 of 35
> failed; retrying after sleep of 300, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:05,759 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 4 of 35
> failed; retrying after sleep of 500, exception=java.io.IOException: Can't get
> master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:06,262 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 5 of 35
> failed; retrying after sleep of 1008, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:07,273 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 6 of 35
> failed; retrying after sleep of 2011, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:09,286 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 7 of 35
> failed; retrying after sleep of 4012, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:13,303 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 8 of 35
> failed; retrying after sleep of 10033, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:23,343 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 9 of 35
> failed; retrying after sleep of 10089, exception=java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:33,439 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 10 of
> 35 failed; retrying after sleep of 10027, exception=java.io.IOException:
> Can't get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:43,473 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 11 of
> 35 failed; retrying after sleep of 10004, exception=java.io.IOException:
> Can't get master address from ZooKeeper; znode data == null
> 2014-09-23 16:19:53,485 INFO [main]
> client.ConnectionManager$HConnectionImplementation: getMaster attempt 1
[jira] [Commented] (HBASE-14223) Meta WALs are not cleared if meta region was closed and RS aborts
[
https://issues.apache.org/jira/browse/HBASE-14223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992301#comment-14992301
]
Enis Soztutar commented on HBASE-14223:
---
The logic is something like this in server failover:
- check whether the failed region server was carrying meta, if so split the
meta wals and assign meta. Split the other wals after that.
- else, split only the files that matches the regular log file pattern and
assign the regions. If there is a .meta WAL, the file is left there in the
splitting directory.
So, if a region server dies while hosting meta, the regular recovery process
recovers the meta wal files and cleans them. Only in cases where a meta region
was opened in some region server, then the meta was moved to someplace else
when the region server failed, we are left with a meta WAL file in the WALs
directory.
> Meta WALs are not cleared if meta region was closed and RS aborts
> -
>
> Key: HBASE-14223
> URL: https://issues.apache.org/jira/browse/HBASE-14223
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 1.0.4
>
> Attachments: HBASE-14223logs, hbase-14223_v0.patch
>
>
> When an RS opens meta, and later closes it, the WAL(FSHlog) is not closed.
> The last WAL file just sits there in the RS WAL directory. If RS stops
> gracefully, the WAL file for meta is deleted. Otherwise if RS aborts, WAL for
> meta is not cleaned. It is also not split (which is correct) since master
> determines that the RS no longer hosts meta at the time of RS abort.
> From a cluster after running ITBLL with CM, I see a lot of {{-splitting}}
> directories left uncleaned:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs
> Found 31 items
> drwxr-xr-x - hbase hadoop 0 2015-06-05 01:14
> /apps/hbase/data/WALs/hregion-58203265
> drwxr-xr-x - hbase hadoop 0 2015-06-05 07:54
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433489308745-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 09:28
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433494382959-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 10:01
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433498252205-splitting
> ...
> {code}
> The directories contain WALs from meta:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting
> Found 2 items
> -rw-r--r-- 3 hbase hadoop 201608 2015-06-05 03:15
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> -rw-r--r-- 3 hbase hadoop 44420 2015-06-05 04:36
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> The RS hosted the meta region for some time:
> {code}
> 2015-06-05 03:14:28,692 INFO [PostOpenDeployTasks:1588230740]
> zookeeper.MetaTableLocator: Setting hbase:meta region location in ZooKeeper
> as os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285
> ...
> 2015-06-05 03:15:17,302 INFO
> [RS_CLOSE_META-os-enis-dal-test-jun-4-5:16020-0] regionserver.HRegion: Closed
> hbase:meta,,1.1588230740
> {code}
> In between, a WAL is created:
> {code}
> 2015-06-05 03:15:11,707 INFO
> [RS_OPEN_META-os-enis-dal-test-jun-4-5:16020-0-MetaLogRoller] wal.FSHLog:
> Rolled WAL
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> with entries=385, filesize=196.88 KB; new WAL
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> When CM killed the region server later master did not see these WAL files:
> {code}
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:46,075
> INFO [MASTER_SERVER_OPERATIONS-os-enis-dal-test-jun-4-3:16000-0]
> master.SplitLogManager: started splitting 2 logs in
> [hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting]
> for [os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285]
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:47,300
> INFO [main-EventThread] wal.WALSplitter: Archived processed log
> hdfs://os-enis-dal-test
[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992285#comment-14992285 ] Enis Soztutar commented on HBASE-13082: --- bq. In one of the test case - so may this is what I was observing. Anyway will verify once in linux box also. Yeah, when FS is local filesystem on windows, the rename while having readers does not work. If fs is hdfs on top of windows, it works. This particular unit test is using the local file system, hence the workaround needed, but in production, we are not using ntfs as a local fs, we are using hdfs on top of ntfs. > Coarsen StoreScanner locks to RegionScanner > --- > > Key: HBASE-13082 > URL: https://issues.apache.org/jira/browse/HBASE-13082 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: ramkrishna.s.vasudevan > Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, > 13082-v4.txt, 13082.txt, 13082.txt, HBASE-13082.pdf, HBASE-13082_1.pdf, > HBASE-13082_1_WIP.patch, HBASE-13082_2_WIP.patch, HBASE-13082_3.patch, > HBASE-13082_4.patch, HBASE-13082_9.patch, HBASE-13082_9.patch, gc.png, > gc.png, gc.png, hits.png, next.png, next.png > > > Continuing where HBASE-10015 left of. > We can avoid locking (and memory fencing) inside StoreScanner by deferring to > the lock already held by the RegionScanner. > In tests this shows quite a scan improvement and reduced CPU (the fences make > the cores wait for memory fetches). > There are some drawbacks too: > * All calls to RegionScanner need to be remain synchronized > * Implementors of coprocessors need to be diligent in following the locking > contract. For example Phoenix does not lock RegionScanner.nextRaw() and > required in the documentation (not picking on Phoenix, this one is my fault > as I told them it's OK) > * possible starving of flushes and compaction with heavy read load. > RegionScanner operations would keep getting the locks and the > flushes/compactions would not be able finalize the set of files. > I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14763) Remove usages of deprecated HConnection
[ https://issues.apache.org/jira/browse/HBASE-14763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-14763: --- Attachment: hbase-14763.v3.patch v3. cleaned up some checkstyle. Looks like the javac warns are actually in the spark code, which I didn't touch. > Remove usages of deprecated HConnection > > > Key: HBASE-14763 > URL: https://issues.apache.org/jira/browse/HBASE-14763 > Project: HBase > Issue Type: Bug > Components: hbase >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-14763.patch, hbase-14763.v2.patch, > hbase-14763.v3.patch > > > HConnection was deprecated in 1.0.0. There are two interfaces that are > supposed to be used instead -- Connection for client programs and > ClusterConnection for internal hbaes and special tools (LoadIncremental, > HBCK, etc). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14765) Remove snappy profile
[ https://issues.apache.org/jira/browse/HBASE-14765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992246#comment-14992246 ] Elliott Clark commented on HBASE-14765: --- bq.Thats some old stuff you are removing there Mr Clark Yeah I barely remember ever using it. > Remove snappy profile > - > > Key: HBASE-14765 > URL: https://issues.apache.org/jira/browse/HBASE-14765 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14765.patch > > > Snappy is provided by hadoop and has been for, a long while. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14765) Remove snappy profile
[ https://issues.apache.org/jira/browse/HBASE-14765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14765: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.2.0 Status: Resolved (was: Patch Available) > Remove snappy profile > - > > Key: HBASE-14765 > URL: https://issues.apache.org/jira/browse/HBASE-14765 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14765.patch > > > Snappy is provided by hadoop and has been for, a long while. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14427) Fix 'should' assertions in TestFastFail
[ https://issues.apache.org/jira/browse/HBASE-14427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992237#comment-14992237 ] Abhishek Singh Chouhan commented on HBASE-14427: [~stack] in case you missed this. This test is not same as TestFailFastWithoutTestUtil which was removed. This one is still there on master but is currently ignored. > Fix 'should' assertions in TestFastFail > --- > > Key: HBASE-14427 > URL: https://issues.apache.org/jira/browse/HBASE-14427 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: Abhishek Singh Chouhan >Priority: Minor > Labels: beginner > Fix For: 2.0.0 > > Attachments: HBASE-14427.addendum.patch, HBASE-14427.patch > > > Over in HBASE-14421, TestFastFail has been failing assertions that talk of > events that 'should' be happening. Fix. For now HBASE-14421 has disabled the > 'should' assertions. They seem fine on apache jenkins build but fail fairly > reliably for me on alternate HW. > To address, get familiar with the test. Change the commented out asserts to > be yes/no instead of a 'likely' (On a cursory scan, it is possible that a > test run may not involve preemption and it is these runs that are throwing > asserts). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14772) Improve zombie detector; be more discerning
[
https://issues.apache.org/jira/browse/HBASE-14772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992223#comment-14992223
]
Sean Busbey edited comment on HBASE-14772 at 11/5/15 6:46 PM:
--
yeah, a UUID or some such specific to the current build would be perfect. on
Jenkins at least, there's $\{BUILD_ID} that we can use.
was (Author: busbey):
yeah, a UUID or some such specific to the current build would be perfect. on
Jenkins at least, there's ${BUILD_ID} that we can use.
> Improve zombie detector; be more discerning
> ---
>
> Key: HBASE-14772
> URL: https://issues.apache.org/jira/browse/HBASE-14772
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>
> Currently, any surefire process with the hbase flag is a potential zombie.
> Our zombie check currently takes a reading and if it finds candidate zombies,
> it waits 30 seconds and then does another reading. If a concurrent build
> going on, in both cases the zombie detector will come up positive though the
> adjacent test run may be making progress; i.e. the cast of surefire processes
> may have changed between readings but our detector just sees presence of
> hbase surefire processes.
> Here is example:
> {code}
> Suspicious java process found - waiting 30s to see if there are just slow to
> stop
> There appear to be 5 zombie tests, they should have been killed by surefire
> but survived
> 12823 surefirebooter852180186418035480.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 7653 surefirebooter8579074445899448699.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 12614 surefirebooter136529596936417090.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 7836 surefirebooter3217047564606450448.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 13566 surefirebooter2084039411151963494.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> BEGIN zombies jstack extract
> END zombies jstack extract
> {code}
> 5 is the number of forked processes we allow when doing medium and large
> tests so an adjacent build will always show as '5 zombies'.
> Need to add discerning if list of processes changes between readings.
> Can I also add a tag per build run that all forked processes pick up so I can
> look at the current builds progeny only?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14772) Improve zombie detector; be more discerning
[
https://issues.apache.org/jira/browse/HBASE-14772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992223#comment-14992223
]
Sean Busbey commented on HBASE-14772:
-
yeah, a UUID or some such specific to the current build would be perfect. on
Jenkins at least, there's ${BUILD_ID} that we can use.
> Improve zombie detector; be more discerning
> ---
>
> Key: HBASE-14772
> URL: https://issues.apache.org/jira/browse/HBASE-14772
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>
> Currently, any surefire process with the hbase flag is a potential zombie.
> Our zombie check currently takes a reading and if it finds candidate zombies,
> it waits 30 seconds and then does another reading. If a concurrent build
> going on, in both cases the zombie detector will come up positive though the
> adjacent test run may be making progress; i.e. the cast of surefire processes
> may have changed between readings but our detector just sees presence of
> hbase surefire processes.
> Here is example:
> {code}
> Suspicious java process found - waiting 30s to see if there are just slow to
> stop
> There appear to be 5 zombie tests, they should have been killed by surefire
> but survived
> 12823 surefirebooter852180186418035480.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 7653 surefirebooter8579074445899448699.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 12614 surefirebooter136529596936417090.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 7836 surefirebooter3217047564606450448.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> 13566 surefirebooter2084039411151963494.jar -enableassertions -Dhbase.test
> -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
> BEGIN zombies jstack extract
> END zombies jstack extract
> {code}
> 5 is the number of forked processes we allow when doing medium and large
> tests so an adjacent build will always show as '5 zombies'.
> Need to add discerning if list of processes changes between readings.
> Can I also add a tag per build run that all forked processes pick up so I can
> look at the current builds progeny only?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14708) Use copy on write Map for region location cache
[ https://issues.apache.org/jira/browse/HBASE-14708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14708: -- Attachment: HBASE-14708-v13.patch > Use copy on write Map for region location cache > --- > > Key: HBASE-14708 > URL: https://issues.apache.org/jira/browse/HBASE-14708 > Project: HBase > Issue Type: Improvement > Components: Client >Affects Versions: 1.1.2 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14708-v10.patch, HBASE-14708-v11.patch, > HBASE-14708-v12.patch, HBASE-14708-v13.patch, HBASE-14708-v2.patch, > HBASE-14708-v3.patch, HBASE-14708-v4.patch, HBASE-14708-v5.patch, > HBASE-14708-v6.patch, HBASE-14708-v7.patch, HBASE-14708-v8.patch, > HBASE-14708-v9.patch, HBASE-14708.patch, anotherbench.zip, > location_cache_times.pdf, result.csv > > > Internally a co-worker profiled their application that was talking to HBase. > > 60% of the time was spent in locating a region. This was while the cluster > was stable and no regions were moving. > To figure out if there was a faster way to cache region location I wrote up a > benchmark here: https://github.com/elliottneilclark/benchmark-hbase-cache > This tries to simulate a heavy load on the location cache. > * 24 different threads. > * 2 Deleting location data > * 2 Adding location data > * Using floor to get the result. > To repeat my work just run ./run.sh and it should produce a result.csv > Results: > ConcurrentSkiplistMap is a good middle ground. It's got equal speed for > reading and writing. > However most operations will not need to remove or add a region location. > There will be potentially several orders of magnitude more reads for cached > locations than there will be on clearing the cache. > So I propose a copy on write tree map. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14708) Use copy on write Map for region location cache
[ https://issues.apache.org/jira/browse/HBASE-14708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992171#comment-14992171 ] Elliott Clark commented on HBASE-14708: --- bq.It is quite confusing that some of methods provide copies, and some of methods expose the internal data that the client should not modify. That's only for tree map. The array based version has non-modifiable versions of everything. The tree map is less risky as it relies on a know and tested version of map. However it's pretty specialized and should only really ever be used in meta cache. The array based map is faster and more complete. However it is also more risky. My thought was that in branch-1.2 we would use the tree map and in branch-1 + we would use the array map. > Use copy on write Map for region location cache > --- > > Key: HBASE-14708 > URL: https://issues.apache.org/jira/browse/HBASE-14708 > Project: HBase > Issue Type: Improvement > Components: Client >Affects Versions: 1.1.2 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14708-v10.patch, HBASE-14708-v11.patch, > HBASE-14708-v12.patch, HBASE-14708-v2.patch, HBASE-14708-v3.patch, > HBASE-14708-v4.patch, HBASE-14708-v5.patch, HBASE-14708-v6.patch, > HBASE-14708-v7.patch, HBASE-14708-v8.patch, HBASE-14708-v9.patch, > HBASE-14708.patch, anotherbench.zip, location_cache_times.pdf, result.csv > > > Internally a co-worker profiled their application that was talking to HBase. > > 60% of the time was spent in locating a region. This was while the cluster > was stable and no regions were moving. > To figure out if there was a faster way to cache region location I wrote up a > benchmark here: https://github.com/elliottneilclark/benchmark-hbase-cache > This tries to simulate a heavy load on the location cache. > * 24 different threads. > * 2 Deleting location data > * 2 Adding location data > * Using floor to get the result. > To repeat my work just run ./run.sh and it should produce a result.csv > Results: > ConcurrentSkiplistMap is a good middle ground. It's got equal speed for > reading and writing. > However most operations will not need to remove or add a region location. > There will be potentially several orders of magnitude more reads for cached > locations than there will be on clearing the cache. > So I propose a copy on write tree map. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14771) RpcServer.getRemoteAddress always returns null.
[
https://issues.apache.org/jira/browse/HBASE-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992161#comment-14992161
]
stack commented on HBASE-14771:
---
Good one. Hmmm. This used to work. Stuff got moved around? Do you know [~appy]
? Can you see? Is this the only place a Call is made? There are a few others
if only on error. We should set address there too? Nice one.
> RpcServer.getRemoteAddress always returns null.
> ---
>
> Key: HBASE-14771
> URL: https://issues.apache.org/jira/browse/HBASE-14771
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
>Affects Versions: 1.2.0
>Reporter: Abhishek Kumar
>Assignee: Abhishek Kumar
>Priority: Minor
> Attachments: HBASE-14771.patch
>
>
> RpcServer.getRemoteAddress always returns null, because Call object is
> getting initialized with null.This seems to be happening because of using
> RpcServer.getRemoteIp() in Call object constructor before RpcServer thread
> local 'CurCall' being set in CallRunner.run method:
> {noformat}
> // --- RpcServer.java ---
> protected void processRequest(byte[] buf) throws IOException,
> InterruptedException {
> .
> // Call object getting initialized here with address
> // obtained from RpcServer.getRemoteIp()
> Call call = new Call(id, this.service, md, header, param, cellScanner, this,
> responder,
> totalRequestSize, traceInfo, RpcServer.getRemoteIp());
> scheduler.dispatch(new CallRunner(RpcServer.this, call));
> }
> // getRemoteIp method gets address from threadlocal 'CurCall' which
> // gets set in CallRunner.run and calling it before this as in above case,
> will return null
> // --- CallRunner.java ---
> public void run() {
> .
> Pair resultPair = null;
> RpcServer.CurCall.set(call);
> ..
> }
> // Using 'this.addr' in place of getRemoteIp method in RpcServer.java seems
> to be fixing this issue
> Call call = new Call(id, this.service, md, header, param, cellScanner, this,
> responder,
> totalRequestSize, traceInfo, this.addr);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14766) WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with the new low-cost implementation.
[ https://issues.apache.org/jira/browse/HBASE-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992149#comment-14992149 ] huaxiang sun commented on HBASE-14766: -- s/patch/path in the previous comment. > WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with > the new low-cost implementation. > -- > > Key: HBASE-14766 > URL: https://issues.apache.org/jira/browse/HBASE-14766 > Project: HBase > Issue Type: Improvement >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-14766-v001.patch, HBASE-14766-v002.patch > > > Cell's getFamily() gets an array copy of the cell's family, while in the > filter function, it just needs to peek into the family and do a compare. > Replace > Bytes.toString(cell.getFamily()) > with > Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(), > cell.getFamilyLength()) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14755) Fix some broken links and HTML problems
[ https://issues.apache.org/jira/browse/HBASE-14755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14755: -- Attachment: HBASE-14755-v1.patch Retry > Fix some broken links and HTML problems > --- > > Key: HBASE-14755 > URL: https://issues.apache.org/jira/browse/HBASE-14755 > Project: HBase > Issue Type: Bug > Components: documentation >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14755-v1.patch, HBASE-14755-v1.patch, > HBASE-14755.patch > > > Problems seen in > https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Website%20Link%20Ckecker/3/artifact/link_report/index.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14772) Improve zombie detector; be more discerning
stack created HBASE-14772:
-
Summary: Improve zombie detector; be more discerning
Key: HBASE-14772
URL: https://issues.apache.org/jira/browse/HBASE-14772
Project: HBase
Issue Type: Sub-task
Reporter: stack
Currently, any surefire process with the hbase flag is a potential zombie. Our
zombie check currently takes a reading and if it finds candidate zombies, it
waits 30 seconds and then does another reading. If a concurrent build going on,
in both cases the zombie detector will come up positive though the adjacent
test run may be making progress; i.e. the cast of surefire processes may have
changed between readings but our detector just sees presence of hbase surefire
processes.
Here is example:
{code}
Suspicious java process found - waiting 30s to see if there are just slow to
stop
There appear to be 5 zombie tests, they should have been killed by surefire but
survived
12823 surefirebooter852180186418035480.jar -enableassertions -Dhbase.test
-Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
7653 surefirebooter8579074445899448699.jar -enableassertions -Dhbase.test
-Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
12614 surefirebooter136529596936417090.jar -enableassertions -Dhbase.test
-Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
7836 surefirebooter3217047564606450448.jar -enableassertions -Dhbase.test
-Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
13566 surefirebooter2084039411151963494.jar -enableassertions -Dhbase.test
-Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
BEGIN zombies jstack extract
END zombies jstack extract
{code}
5 is the number of forked processes we allow when doing medium and large
tests so an adjacent build will always show as '5 zombies'.
Need to add discerning if list of processes changes between readings.
Can I also add a tag per build run that all forked processes pick up so I can
look at the current builds progeny only?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14766) WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with the new low-cost implementation.
[ https://issues.apache.org/jira/browse/HBASE-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992125#comment-14992125 ] huaxiang sun commented on HBASE-14766: -- [~ted_yu] Thanks! Updated a new patch. The change in ScopeWALEntryFilter.filter() has been done. The new patch avoids a clone of cell in the replication patch. > WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with > the new low-cost implementation. > -- > > Key: HBASE-14766 > URL: https://issues.apache.org/jira/browse/HBASE-14766 > Project: HBase > Issue Type: Improvement >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-14766-v001.patch, HBASE-14766-v002.patch > > > Cell's getFamily() gets an array copy of the cell's family, while in the > filter function, it just needs to peek into the family and do a compare. > Replace > Bytes.toString(cell.getFamily()) > with > Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(), > cell.getFamilyLength()) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14766) WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with the new low-cost implementation.
[ https://issues.apache.org/jira/browse/HBASE-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-14766: - Attachment: HBASE-14766-v002.patch The change in ScopeWALEntryFilter.java has been done in master branch. InTableCfWALEntryFilter's filter(), no clone is needed. > WALEntryFilter's filter implement, cell.getFamily() needs to be replaced with > the new low-cost implementation. > -- > > Key: HBASE-14766 > URL: https://issues.apache.org/jira/browse/HBASE-14766 > Project: HBase > Issue Type: Improvement >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-14766-v001.patch, HBASE-14766-v002.patch > > > Cell's getFamily() gets an array copy of the cell's family, while in the > filter function, it just needs to peek into the family and do a compare. > Replace > Bytes.toString(cell.getFamily()) > with > Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(), > cell.getFamilyLength()) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
