[jira] [Created] (HBASE-17676) Get class name once for all in AbstractFSWAL
Yu Li created HBASE-17676:
-
Summary: Get class name once for all in AbstractFSWAL
Key: HBASE-17676
URL: https://issues.apache.org/jira/browse/HBASE-17676
Project: HBase
Issue Type: Bug
Components: Performance
Reporter: Yu Li
Assignee: Yu Li
While verifying HBASE-17471 with high write workload, observed several handler
thread at getting class name in jstack, as shown below:
{noformat}
"B.defaultRpcServer.handler=60,queue=3,port=16020" daemon prio=10
tid=0x7f0673835800 nid=0x4dec runnable [0x7f06721b5000]
java.lang.Thread.State: RUNNABLE
at java.lang.Class.getEnclosingMethod0(Native Method)
at java.lang.Class.getEnclosingMethodInfo(Class.java:964)
at java.lang.Class.getEnclosingClass(Class.java:1137)
at java.lang.Class.getSimpleBinaryName(Class.java:1282)
at java.lang.Class.getSimpleName(Class.java:1174)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog.stampSequenceIdAndPublishToRingBuffer(FSHLog.java:1251)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog.append(FSHLog.java:1238)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3173)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2874)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2814)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:823)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:785)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2259)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:848)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:102)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:756)
{noformat}
We could get the class name in constructor and use it for all places needed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (HBASE-13718) Add a pretty printed table description to the table detail page of HBase's master
[ https://issues.apache.org/jira/browse/HBASE-13718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-13718. - Resolution: Fixed Release Note: The table information page in the Master UI now includes a schema section that describes the column families defined for that table as well as any column family specific properties that are set. > Add a pretty printed table description to the table detail page of HBase's > master > - > > Key: HBASE-13718 > URL: https://issues.apache.org/jira/browse/HBASE-13718 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 2.0.0 >Reporter: Joao Girao >Assignee: Joao Girao >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: D38649.diff, D38649.diff.txt, Screen Shot 2015-05-18 at > 1.57.50 PM.png > > > HBase's master has an info server that's useful for debugging and getting a > general overview of what's in the cluster. It has a page dedicated to > describing a cluster. You can reach it by going to something like: > http://localhost:54677/table.jsp?name=cluster_test > That page currently doesn't have anything about the current table schema. It > would be nice to have a table that lists the different column families and > how they are set up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Reopened] (HBASE-13718) Add a pretty printed table description to the table detail page of HBase's master
[ https://issues.apache.org/jira/browse/HBASE-13718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey reopened HBASE-13718: - > Add a pretty printed table description to the table detail page of HBase's > master > - > > Key: HBASE-13718 > URL: https://issues.apache.org/jira/browse/HBASE-13718 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 2.0.0 >Reporter: Joao Girao >Assignee: Joao Girao >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: D38649.diff, D38649.diff.txt, Screen Shot 2015-05-18 at > 1.57.50 PM.png > > > HBase's master has an info server that's useful for debugging and getting a > general overview of what's in the cluster. It has a page dedicated to > describing a cluster. You can reach it by going to something like: > http://localhost:54677/table.jsp?name=cluster_test > That page currently doesn't have anything about the current table schema. It > would be nice to have a table that lists the different column families and > how they are set up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Re: Testing and CI -- Apache Jenkins Builds (WAS -> Re: Testing)
FYI, I updated the 1.2-IT and 1.3-IT jobs today to use Appy's
suggested "custom child workspace" of "${SHORT_COMBINATION}", since
spaces in paths had caused them to fail for a v long time.
On Fri, Oct 14, 2016 at 4:44 PM, Andrew Purtell wrote:
> Thanks Ted, that would be a nice contribution, thank you.
>
>
> On Fri, Oct 14, 2016 at 12:07 PM, Apekshit Sharma wrote:
>
>> @Ted, here's the old jira, HBASE-14167. Use that.
>>
>> On Fri, Oct 14, 2016 at 12:02 PM, Ted Yu wrote:
>>
>> > I just ran the tests in hbase-spark module using 'mvn verify'.
>> >
>> > All passed.
>> >
>> > I am testing a patch locally where hbase-spark tests are run in test
>> phase.
>> >
>> > If the tests pass, I will log a JIRA.
>> >
>> > Thanks
>> >
>> > > On Oct 14, 2016, at 11:41 AM, Andrew Purtell
>> > wrote:
>> > >
>> > > The hbase-spark integration tests run (and fail) for me locally
>> whenever
>> > I
>> > > build master with 'mvn clean install -DskipITs' .
>> > >
>> > > HBaseConnectionCacheSuite:
>> > > - all test cases *** FAILED ***
>> > > 2 did not equal 1 (HBaseConnectionCacheSuite.scala:92)
>> > >
>> > > Saw it but had to ignore/triage to get something else done.
>> > >
>> > > We have a weird situation where integration tests run when they
>> shouldn't
>> > > locally yet no tests run at all for patch process?
>> > >
>> > > I would like to see Spark behave like the other modules. I remember
>> > filing
>> > > a JIRA asking that hbase-spark honor -DskipITs. It still doesn't.
>> > > Meanwhile, it does its own thing with '-DskipSparkTests', which is not
>> > > appropriate given that none of the other modules have their own
>> distinct
>> > > control parameters. There also doesn't seem to be a distinction between
>> > > unit tests and integration tests. The 'test' target does nothing.
>> > > Everything happens during the 'integration-test' phase. Is this a Spark
>> > > limitation?
>> > >
>> > >
>> > >> On Fri, Oct 14, 2016 at 11:27 AM, Sean Busbey
>> > wrote:
>> > >>
>> > >> Do the HBase Spark tests only run during the maven verify command?
>> > >> We'll need to update our personality to say that that command should
>> > >> be used for unit tests when in the hbase spark module. ugh.
>> > >>
>> > >> On Thu, Oct 13, 2016 at 7:42 PM, Apekshit Sharma
>> > >> wrote:
>> > >>> Our patch process isn't running hbase-spark tests. See this for
>> > example:
>> > >>>
>> > >>> https://builds.apache.org/job/PreCommit-HBASE-Build/3842/
>> > >>> https://builds.apache.org/job/PreCommit-HBASE-Build/3842/
>> > >> artifact/patchprocess/patch-unit-hbase-spark.txt/*view*/
>> > >>>
>> > >>> Found it when trying to debug cause of trunk failures. Part of the
>> > cause
>> > >> is
>> > >>> hbase-spark's HBaseConnectionCacheSuite test failure (
>> > >>> https://builds.apache.org/view/All/job/HBase-Trunk_
>> > >> matrix/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/1776/consoleFull
>> > >> > > matrix/jdk=JDK%201.8%20%28latest%29,label=yahoo-not-h2/1776/consoleFull>
>> > >> )
>> > >>> which was added in HBASE-16638. However, to be fair, QA was green and
>> > >>> reported passing hbase-spark tests for that jira.
>> > >>>
>> > On Mon, Sep 19, 2016 at 12:57 PM, Stack wrote:
>> >
>> > childCustomWorkspace seems to be just the ticket. Nice find Appy.
>> > St.Ack
>> >
>> > On Mon, Sep 19, 2016 at 10:03 AM, Sean Busbey
>> > >> wrote:
>> >
>> > > Option 2c looks to be working really well. Thanks for tackling this
>> > >> Appy!
>> > >
>> > > We still have some failures on the master build, but it looks like
>> > > actual problems (or perhaps a flakey). There are several passing
>> > > builds.
>> > >
>> > > This should be pretty easy to replicate on the other jobs. I don't
>> > see
>> > > a downside. Anyone else have concerns?
>> > >
>> > >
>> > > On Fri, Sep 16, 2016 at 6:15 PM, Apekshit Sharma <
>> a...@cloudera.com>
>> > > wrote:
>> > >> So this all started with spaces-in-path issue, right? I think it
>> > >> has
>> > >> gobbled up a lot of time of a lot of people.
>> > >> Let's discuss our options and try to fix it for good. Here are
>> what
>> > >> i
>> > can
>> > >> think of, and my opinion about them.
>> > >>
>> > >> 1. Not use matrix build
>> > >> Temporary fix. Not preferred since not applicable to
>> other
>> > >> branches' builds.
>> > >>
>> > >> 2. Use matrix build
>> > >>
>> > >> a. Use tool environment trick
>> > >> I applied this few days ago. Seemed to work until we
>> > > discovered
>> > >> scalatest issue. While the solution looks legitimate, we can't
>> trust
>> > that
>> > >> all tools will use JAVA_HOME instead of directly using java
>> command.
>> > >>
>> > >> b. Use JDK axix
>> > >> Doesn't work right now. I don't have much idea of what's
>> > >> the
>> > > cost
>> > >> for fixing
[jira] [Reopened] (HBASE-17069) RegionServer writes invalid META entries for split daughters in some circumstances
[
https://issues.apache.org/jira/browse/HBASE-17069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell reopened HBASE-17069:
> RegionServer writes invalid META entries for split daughters in some
> circumstances
> --
>
> Key: HBASE-17069
> URL: https://issues.apache.org/jira/browse/HBASE-17069
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.2.4
>Reporter: Andrew Purtell
>Assignee: Abhishek Singh Chouhan
>Priority: Blocker
> Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5
>
> Attachments: daughter_1_d55ef81c2f8299abbddfce0445067830.log,
> daughter_2_08629d59564726da2497f70451aafcdb.log,
> HBASE-17069.branch-1.3.001.patch, HBASE-17069.branch-1.3.002.patch,
> HBASE-17069.master.001.patch, logs.tar.gz,
> parent-393d2bfd8b1c52ce08540306659624f2.log
>
>
> I have been seeing frequent ITBLL failures testing various versions of 1.2.x.
> Over the lifetime of 1.2.x the following issues have been fixed:
> - HBASE-15315 (Remove always set super user call as high priority)
> - HBASE-16093 (Fix splits failed before creating daughter regions leave meta
> inconsistent)
> And this one is pending:
> - HBASE-17044 (Fix merge failed before creating merged region leaves meta
> inconsistent)
> I can apply all of the above to branch-1.2 and still see this failure:
> *The life of stillborn region d55ef81c2f8299abbddfce0445067830*
> *Master sees SPLITTING_NEW*
> {noformat}
> 2016-11-08 04:23:21,186 INFO [AM.ZK.Worker-pool2-t82] master.RegionStates:
> Transition null to {d55ef81c2f8299abbddfce0445067830 state=SPLITTING_NEW,
> ts=1478579001186, server=node-3.cluster,16020,1478578389506}
> {noformat}
> *The RegionServer creates it*
> {noformat}
> 2016-11-08 04:23:26,035 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for GomnU: blockCache=LruBlockCache{blockCount=34,
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320,
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,038 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for big: blockCache=LruBlockCache{blockCount=34,
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320,
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,442 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for meta: blockCache=LruBlockCache{blockCount=63,
> currentSize=17187656, freeSize=12821524664, maxSize=12838712320,
> heapSize=17187656, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,713 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for nwmrW: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,715 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for piwbr: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,717 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for tiny: blockCache=LruBlockCache{blockCount=96,
> c
[jira] [Created] (HBASE-17675) ReplicationEndpoint should choose new sinks if a SaslException occurs
churro morales created HBASE-17675: -- Summary: ReplicationEndpoint should choose new sinks if a SaslException occurs Key: HBASE-17675 URL: https://issues.apache.org/jira/browse/HBASE-17675 Project: HBase Issue Type: Bug Reporter: churro morales We had an issue where a regionserver on our destination side failed to refresh the keytabs. The source side's replication got stuck because the HBaseInterClusterReplicationEndpoint only chooses new sinks if there happens to be a ConnectException but the SaslException is an IOException, which does not choose new sinks. I'll put up a patch to check this exception and choose new sinks. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Successful: HBase Generate Website
Build status: Successful
If successful, the website and docs have been generated. To update the live
site, follow the instructions below. If failed, skip to the bottom of this
email.
Use the following commands to download the patch and apply it to a clean branch
based on origin/asf-site. If you prefer to keep the hbase-site repo around
permanently, you can skip the clone step.
git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git
cd hbase-site
wget -O-
https://builds.apache.org/job/hbase_generate_website/495/artifact/website.patch.zip
| funzip > d7ffa0013bde592bb035ce5306c09883a192989f.patch
git fetch
git checkout -b asf-site-d7ffa0013bde592bb035ce5306c09883a192989f
origin/asf-site
git am --whitespace=fix d7ffa0013bde592bb035ce5306c09883a192989f.patch
At this point, you can preview the changes by opening index.html or any of the
other HTML pages in your local
asf-site-d7ffa0013bde592bb035ce5306c09883a192989f branch.
There are lots of spurious changes, such as timestamps and CSS styles in
tables, so a generic git diff is not very useful. To see a list of files that
have been added, deleted, renamed, changed type, or are otherwise interesting,
use the following command:
git diff --name-status --diff-filter=ADCRTXUB origin/asf-site
To see only files that had 100 or more lines changed:
git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'
When you are satisfied, publish your changes to origin/asf-site using these
commands:
git commit --allow-empty -m "Empty commit" # to work around a current ASF
INFRA bug
git push origin asf-site-d7ffa0013bde592bb035ce5306c09883a192989f:asf-site
git checkout asf-site
git branch -D asf-site-d7ffa0013bde592bb035ce5306c09883a192989f
Changes take a couple of minutes to be propagated. You can verify whether they
have been propagated by looking at the Last Published date at the bottom of
http://hbase.apache.org/. It should match the date in the index.html on the
asf-site branch in Git.
As a courtesy- reply-all to this email to let other committers know you pushed
the site.
If failed, see https://builds.apache.org/job/hbase_generate_website/495/console
[jira] [Created] (HBASE-17674) Major compaction may be canceled in CompactionChecker
Guangxu Cheng created HBASE-17674:
-
Summary: Major compaction may be canceled in CompactionChecker
Key: HBASE-17674
URL: https://issues.apache.org/jira/browse/HBASE-17674
Project: HBase
Issue Type: Bug
Components: Compaction
Affects Versions: 0.98.24, 1.2.4, 1.3.0, 2.0.0
Reporter: Guangxu Cheng
CompactionChecker will periodically check whether a region should perform a
major compaction.
If a region should perform a major compact, a request is submitted. But before
the request is submitted, the variable forceMajor is not set to true by calling
triggerMajorCompaction.
When filtering a storefile, the large storefile may cause the request to be
canceled or became minor compact.
{code:title=HRegionServer.java|borderStyle=solid}
@Override
protected void chore() {
for (Region r : this.instance.onlineRegions.values()) {
if (r == null)
continue;
for (Store s : r.getStores()) {
try {
long multiplier = s.getCompactionCheckMultiplier();
assert multiplier > 0;
if (iteration % multiplier != 0) continue;
if (s.needsCompaction()) {
// Queue a compaction. Will recognize if major is needed.
this.instance.compactSplitThread.requestSystemCompaction(r, s,
getName()
+ " requests compaction");
} else if (s.isMajorCompaction()) {
if (majorCompactPriority == DEFAULT_PRIORITY
|| majorCompactPriority > ((HRegion)r).getCompactPriority()) {
this.instance.compactSplitThread.requestCompaction(r, s,
getName()
+ " requests major compaction; use default priority", null);
} else {
this.instance.compactSplitThread.requestCompaction(r, s,
getName()
+ " requests major compaction; use configured priority",
this.majorCompactPriority, null, null);
}
}
} catch (IOException e) {
LOG.warn("Failed major compaction check on " + r, e);
}
}
}
iteration = (iteration == Long.MAX_VALUE) ? 0 : (iteration + 1);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (HBASE-17518) HBase Reference Guide has a syntax error
[ https://issues.apache.org/jira/browse/HBASE-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhang Ningbo resolved HBASE-17518. -- Resolution: Fixed > HBase Reference Guide has a syntax error > > > Key: HBASE-17518 > URL: https://issues.apache.org/jira/browse/HBASE-17518 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Zhang Ningbo >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17518.patch > > > The image of "HFile Version 2 Structure" in Appendix F of HBase Reference > Guide (pdf) is missing because of a wrong asciidoc syntax: > image:hfilev2.png[HFile Version 2] > modified as: > image::hfilev2.png[HFile Version 2] > it should be a double colon instead of single one -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Reopened] (HBASE-17518) HBase Reference Guide has a syntax error
[ https://issues.apache.org/jira/browse/HBASE-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhang Ningbo reopened HBASE-17518: -- > HBase Reference Guide has a syntax error > > > Key: HBASE-17518 > URL: https://issues.apache.org/jira/browse/HBASE-17518 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Zhang Ningbo >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17518.patch > > > The image of "HFile Version 2 Structure" in Appendix F of HBase Reference > Guide (pdf) is missing because of a wrong asciidoc syntax: > image:hfilev2.png[HFile Version 2] > modified as: > image::hfilev2.png[HFile Version 2] > it should be a double colon instead of single one -- This message was sent by Atlassian JIRA (v6.3.15#6346)
