[jira] [Commented] (HBASE-6418) Minor bug in delete flow.
[
https://issues.apache.org/jira/browse/HBASE-6418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416893#comment-13416893
]
Lars Hofhansl commented on HBASE-6418:
--
Are you sure Laxman?
This logic only needs to be done for version delete markers (not column or
family markers).
kv.isDeleteType() is only true to version delete markers.
KeyValue.isDelete(type) would be true for all delete markers.
I think it is correct the way it is.
> Minor bug in delete flow.
> -
>
> Key: HBASE-6418
> URL: https://issues.apache.org/jira/browse/HBASE-6418
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.0, 0.96.0, 0.94.1, 0.94.2
>Reporter: Laxman
>Assignee: Laxman
>
> Timestamp updation in Delete flow is not considering all flavors (Delete
> record, Delete Family, Delete Column) of Delete API. Currently its
> considering Delete Record only.
>
> org.apache.hadoop.hbase.regionserver.HRegion.prepareDeleteTimestamps(Delete,
> byte[])
> {code}
> for (KeyValue kv: kvs) {
> // Check if time is LATEST, change to time of most recent addition
> if so
> // This is expensive.
> if (kv.isLatestTimestamp() && kv.isDeleteType()) {
> {code}
> Basically used a wrong API.
> kv.isDeleteType() should be KeyValue.isDelete(type);
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6381) AssignmentManager should use the same logic for clean startup and failover
[ https://issues.apache.org/jira/browse/HBASE-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416861#comment-13416861 ] ramkrishna.s.vasudevan commented on HBASE-6381: --- @Jimmy Correct me if am wrong. While doing rebuildUserRegions if the RS that was online dies, any way the SSH will be able to assign it again by going thro the META. If the RS went down before starting with rebuildUserRegions it will not consider the RS as an alive RS. > AssignmentManager should use the same logic for clean startup and failover > -- > > Key: HBASE-6381 > URL: https://issues.apache.org/jira/browse/HBASE-6381 > Project: HBase > Issue Type: Bug > Components: master >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > > Currently AssignmentManager handles clean startup and failover very > differently. > Different logic is mingled together so it is hard to find out which is for > which. > We should clean it up and share the same logic so that AssignmentManager > handles > both cases the same way. This way, the code will much easier to understand > and > maintain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6418) Minor bug in delete flow.
Laxman created HBASE-6418:
-
Summary: Minor bug in delete flow.
Key: HBASE-6418
URL: https://issues.apache.org/jira/browse/HBASE-6418
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0, 0.94.1, 0.94.2
Reporter: Laxman
Assignee: Laxman
Timestamp updation in Delete flow is not considering all flavors (Delete
record, Delete Family, Delete Column) of Delete API. Currently its considering
Delete Record only.
org.apache.hadoop.hbase.regionserver.HRegion.prepareDeleteTimestamps(Delete,
byte[])
{code}
for (KeyValue kv: kvs) {
// Check if time is LATEST, change to time of most recent addition if
so
// This is expensive.
if (kv.isLatestTimestamp() && kv.isDeleteType()) {
{code}
Basically used a wrong API.
kv.isDeleteType() should be KeyValue.isDelete(type);
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6220) PersistentMetricsTimeVaryingRate gets used for non-time-based metrics
[ https://issues.apache.org/jira/browse/HBASE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416842#comment-13416842 ] Hadoop QA commented on HBASE-6220: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536928/ServerMetrics_HBASE_6220_Flush_Metrics.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 10 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2396//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2396//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2396//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2396//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2396//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2396//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2396//console This message is automatically generated. > PersistentMetricsTimeVaryingRate gets used for non-time-based metrics > - > > Key: HBASE-6220 > URL: https://issues.apache.org/jira/browse/HBASE-6220 > Project: HBase > Issue Type: Bug > Components: metrics >Affects Versions: 0.96.0 >Reporter: David S. Wang >Assignee: Paul Cavallaro >Priority: Minor > Labels: noob > Attachments: ServerMetrics_HBASE_6220.patch, > ServerMetrics_HBASE_6220_Flush_Metrics.patch > > > PersistentMetricsTimeVaryingRate gets used for metrics that are not > time-based, leading to confusing names such as "avg_time" for compaction > size, etc. You hav to read the code in order to understand that this is > actually referring to bytes, not seconds. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6399) MetricsContext should be different between RegionServerMetrics and RegionServerDynamicMetrics
[ https://issues.apache.org/jira/browse/HBASE-6399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu updated HBASE-6399: -- Hadoop Flags: Reviewed Summary: MetricsContext should be different between RegionServerMetrics and RegionServerDynamicMetrics (was: MetricsContext be different between RegionServerMetrics and RegionServerDynamicMetrics) > MetricsContext should be different between RegionServerMetrics and > RegionServerDynamicMetrics > - > > Key: HBASE-6399 > URL: https://issues.apache.org/jira/browse/HBASE-6399 > Project: HBase > Issue Type: Bug > Components: metrics >Affects Versions: 0.94.0 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.96.0, 0.94.2 > > Attachments: HBASE-6399.patch, HBASE-6399v2.patch > > > In hadoop-metrics.properties, GangliaContext is optional metrics context, I > think we will use ganglia to monitor hbase cluster generally. > However, I find a serious problem: > RegionServerDynamicMetrics will generate lots of rrd file because we would > move region or create/delete table. > Especially if table is created everyday in some applications, there are much > more and more rrd files in Gmetad Server. It will make Gmetad Server > corrupted. > IMO, MetricsContext should be different between RegionServerMetrics and > RegionServerDynamicMetrics -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6399) MetricsContext be different between RegionServerMetrics and RegionServerDynamicMetrics
[ https://issues.apache.org/jira/browse/HBASE-6399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416839#comment-13416839 ] Zhihong Ted Yu commented on HBASE-6399: --- Patch v2 looks good to me. > MetricsContext be different between RegionServerMetrics and > RegionServerDynamicMetrics > -- > > Key: HBASE-6399 > URL: https://issues.apache.org/jira/browse/HBASE-6399 > Project: HBase > Issue Type: Bug > Components: metrics >Affects Versions: 0.94.0 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.96.0, 0.94.2 > > Attachments: HBASE-6399.patch, HBASE-6399v2.patch > > > In hadoop-metrics.properties, GangliaContext is optional metrics context, I > think we will use ganglia to monitor hbase cluster generally. > However, I find a serious problem: > RegionServerDynamicMetrics will generate lots of rrd file because we would > move region or create/delete table. > Especially if table is created everyday in some applications, there are much > more and more rrd files in Gmetad Server. It will make Gmetad Server > corrupted. > IMO, MetricsContext should be different between RegionServerMetrics and > RegionServerDynamicMetrics -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6399) MetricsContext be different between RegionServerMetrics and RegionServerDynamicMetrics
[ https://issues.apache.org/jira/browse/HBASE-6399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416838#comment-13416838 ] chunhui shen commented on HBASE-6399: - We find this problem when using 0.94 version, It is serious to us. If not go into 0.94.X, I could accpet. > MetricsContext be different between RegionServerMetrics and > RegionServerDynamicMetrics > -- > > Key: HBASE-6399 > URL: https://issues.apache.org/jira/browse/HBASE-6399 > Project: HBase > Issue Type: Bug > Components: metrics >Affects Versions: 0.94.0 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.96.0, 0.94.2 > > Attachments: HBASE-6399.patch, HBASE-6399v2.patch > > > In hadoop-metrics.properties, GangliaContext is optional metrics context, I > think we will use ganglia to monitor hbase cluster generally. > However, I find a serious problem: > RegionServerDynamicMetrics will generate lots of rrd file because we would > move region or create/delete table. > Especially if table is created everyday in some applications, there are much > more and more rrd files in Gmetad Server. It will make Gmetad Server > corrupted. > IMO, MetricsContext should be different between RegionServerMetrics and > RegionServerDynamicMetrics -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6399) MetricsContext be different between RegionServerMetrics and RegionServerDynamicMetrics
[ https://issues.apache.org/jira/browse/HBASE-6399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-6399: Attachment: HBASE-6399v2.patch Updating patch as the suggestion. > MetricsContext be different between RegionServerMetrics and > RegionServerDynamicMetrics > -- > > Key: HBASE-6399 > URL: https://issues.apache.org/jira/browse/HBASE-6399 > Project: HBase > Issue Type: Bug > Components: metrics >Affects Versions: 0.94.0 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.96.0, 0.94.2 > > Attachments: HBASE-6399.patch, HBASE-6399v2.patch > > > In hadoop-metrics.properties, GangliaContext is optional metrics context, I > think we will use ganglia to monitor hbase cluster generally. > However, I find a serious problem: > RegionServerDynamicMetrics will generate lots of rrd file because we would > move region or create/delete table. > Especially if table is created everyday in some applications, there are much > more and more rrd files in Gmetad Server. It will make Gmetad Server > corrupted. > IMO, MetricsContext should be different between RegionServerMetrics and > RegionServerDynamicMetrics -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6406) TestReplicationPeer.testResetZooKeeperSession and TestZooKeeper.testClientSessionExpired fail frequently
[ https://issues.apache.org/jira/browse/HBASE-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu updated HBASE-6406: -- Attachment: testZooKeeper.jstack testReplication.jstack I ran trunk test suite and there were two surefire JVMs hanging. Here're their jstack's > TestReplicationPeer.testResetZooKeeperSession and > TestZooKeeper.testClientSessionExpired fail frequently > > > Key: HBASE-6406 > URL: https://issues.apache.org/jira/browse/HBASE-6406 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.1 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.2 > > Attachments: testReplication.jstack, testZooKeeper.jstack > > > Looking back through the 0.94 test runs these two tests accounted for 11 of > 34 failed tests. > They should be fixed or (temporarily) disabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6220) PersistentMetricsTimeVaryingRate gets used for non-time-based metrics
[ https://issues.apache.org/jira/browse/HBASE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416765#comment-13416765 ] Paul Cavallaro commented on HBASE-6220: --- Hey all, back from the offline world, replies below: @stack Yep, everything seems to show up in /jmx just fine. @David Wang No reason that it wasn't included, I've actually attached another patch to switch those over the Metrics Histogram as well if desired we can include it. Thanks all, and sorry for the slow responses. > PersistentMetricsTimeVaryingRate gets used for non-time-based metrics > - > > Key: HBASE-6220 > URL: https://issues.apache.org/jira/browse/HBASE-6220 > Project: HBase > Issue Type: Bug > Components: metrics >Affects Versions: 0.96.0 >Reporter: David S. Wang >Assignee: Paul Cavallaro >Priority: Minor > Labels: noob > Attachments: ServerMetrics_HBASE_6220.patch, > ServerMetrics_HBASE_6220_Flush_Metrics.patch > > > PersistentMetricsTimeVaryingRate gets used for metrics that are not > time-based, leading to confusing names such as "avg_time" for compaction > size, etc. You hav to read the code in order to understand that this is > actually referring to bytes, not seconds. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6220) PersistentMetricsTimeVaryingRate gets used for non-time-based metrics
[ https://issues.apache.org/jira/browse/HBASE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Cavallaro updated HBASE-6220: -- Attachment: ServerMetrics_HBASE_6220_Flush_Metrics.patch > PersistentMetricsTimeVaryingRate gets used for non-time-based metrics > - > > Key: HBASE-6220 > URL: https://issues.apache.org/jira/browse/HBASE-6220 > Project: HBase > Issue Type: Bug > Components: metrics >Affects Versions: 0.96.0 >Reporter: David S. Wang >Assignee: Paul Cavallaro >Priority: Minor > Labels: noob > Attachments: ServerMetrics_HBASE_6220.patch, > ServerMetrics_HBASE_6220_Flush_Metrics.patch > > > PersistentMetricsTimeVaryingRate gets used for metrics that are not > time-based, leading to confusing names such as "avg_time" for compaction > size, etc. You hav to read the code in order to understand that this is > actually referring to bytes, not seconds. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6381) AssignmentManager should use the same logic for clean startup and failover
[ https://issues.apache.org/jira/browse/HBASE-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416763#comment-13416763 ] Jimmy Xiang commented on HBASE-6381: Looked AM.rebuildUserRegions(), it thinks a region is online if the region server is online. This is not reliable. For example, a master dies, then a region server dies. At this time SSH is not running. Now, master fails over. Now all the regions in the dead server won't be assigned. Is that right? > AssignmentManager should use the same logic for clean startup and failover > -- > > Key: HBASE-6381 > URL: https://issues.apache.org/jira/browse/HBASE-6381 > Project: HBase > Issue Type: Bug > Components: master >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > > Currently AssignmentManager handles clean startup and failover very > differently. > Different logic is mingled together so it is hard to find out which is for > which. > We should clean it up and share the same logic so that AssignmentManager > handles > both cases the same way. This way, the code will much easier to understand > and > maintain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6310) -ROOT- corruption when .META. is using the old encoding scheme
[
https://issues.apache.org/jira/browse/HBASE-6310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416760#comment-13416760
]
Jean-Daniel Cryans commented on HBASE-6310:
---
It could be an issue caused by hbck, I tried running it a few months ago and it
didn't work for us and I'm now thinking that it actually put that invalid
.META. row in ROOT. My first try on recreating the problem ended up with
HBASE-6417 so it might no be -repair that I ran.
> -ROOT- corruption when .META. is using the old encoding scheme
> --
>
> Key: HBASE-6310
> URL: https://issues.apache.org/jira/browse/HBASE-6310
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.94.0
>Reporter: Jean-Daniel Cryans
>Priority: Blocker
> Fix For: 0.96.0, 0.94.2
>
>
> We're still working the on the root cause here, but after the leap second
> armageddon we had a hard time getting our 0.94 cluster back up. This is what
> we saw in the logs until the master died by itself:
> {noformat}
> 2012-07-01 23:01:52,149 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> locateRegionInMeta parentTable=-ROOT-,
> metaLocation={region=-ROOT-,,0.70236052, hostname=sfor3s28,
> port=10304}, attempt=16 of 100 failed; retrying after sleep of 32000
> because: HRegionInfo was null or empty in -ROOT-,
> row=keyvalues={.META.,,1259448304806/info:server/1341124914705/Put/vlen=14/ts=0,
> .META.,,1259448304806/info:serverstartcode/1341124914705/Put/vlen=8/ts=0}
> {noformat}
> (it's strage that we retry this)
> This was really misleading because I could see the regioninfo in a scan:
> {noformat}
> hbase(main):002:0> scan '-ROOT-'
> ROW COLUMN+CELL
> .META.,,1column=info:regioninfo,
> timestamp=1331755381142, value={NAME => '.META.,,1', STARTKEY => '',
> ENDKEY => '', ENCODED => 1028785192,}
> .META.,,1column=info:server,
> timestamp=1341183448693, value=sfor3s40:10304
> .META.,,1
> column=info:serverstartcode, timestamp=1341183448693,
> value=1341183444689
> .META.,,1column=info:v,
> timestamp=1331755419291, value=\x00\x00
> .META.,,1259448304806column=info:server,
> timestamp=1341124914705, value=sfor3s24:10304
> .META.,,1259448304806
> column=info:serverstartcode, timestamp=1341124914705,
> value=1341124455863
> {noformat}
> Except that the devil is in the details, ".META.,,1" is not
> ".META.,,1259448304806". Basically something writes to .META. by directly
> creating the row key without caring if the row is in the old format. I did a
> deleteall in the shell and it fixed the issue... until some time later it was
> stuck again because the edits reappeared (still not sure why). This time the
> PostOpenDeployTasksThread were stuck in the RS trying to update .META. but
> there was no logging (saw it with a jstack). I deleted the row again to make
> it work.
> I'm marking this as a blocker against 0.94.2 since we're trying to get 0.94.1
> out, but I wouldn't recommend upgrading to 0.94 if your cluster was created
> before 0.89
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6417) hbck merges .META. regions if there's an old leftover
[ https://issues.apache.org/jira/browse/HBASE-6417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Daniel Cryans updated HBASE-6417: -- Attachment: hbck.log The full log on what happened. I was able to fix it by recreating the old region's folder, moving the right store file back and moving a .regioninfo from somewhere else. Good thing it was a dev cluster. > hbck merges .META. regions if there's an old leftover > - > > Key: HBASE-6417 > URL: https://issues.apache.org/jira/browse/HBASE-6417 > Project: HBase > Issue Type: Bug >Reporter: Jean-Daniel Cryans > Fix For: 0.96.0, 0.94.2 > > Attachments: hbck.log > > > Trying to see what caused HBASE-6310, one of the things I figured is that the > bad .META. row is actually one from the time that we were permitting meta > splitting and that folder had just been staying there for a while. > So I tried to recreate the issue with -repair and it merged my good .META. > region with the one that's 3 years old that also has the same start key. I > ended up with a brand new .META. region! > I'll be attaching the full log in a separate file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6417) hbck merges .META. regions if there's an old leftover
Jean-Daniel Cryans created HBASE-6417: - Summary: hbck merges .META. regions if there's an old leftover Key: HBASE-6417 URL: https://issues.apache.org/jira/browse/HBASE-6417 Project: HBase Issue Type: Bug Reporter: Jean-Daniel Cryans Fix For: 0.96.0, 0.94.2 Trying to see what caused HBASE-6310, one of the things I figured is that the bad .META. row is actually one from the time that we were permitting meta splitting and that folder had just been staying there for a while. So I tried to recreate the issue with -repair and it merged my good .META. region with the one that's 3 years old that also has the same start key. I ended up with a brand new .META. region! I'll be attaching the full log in a separate file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6405) Create Hadoop compatibilty modules and Metrics2 implementation of replication metrics
[ https://issues.apache.org/jira/browse/HBASE-6405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416740#comment-13416740 ] Hudson commented on HBASE-6405: --- Integrated in HBase-TRUNK #3142 (See [https://builds.apache.org/job/HBase-TRUNK/3142/]) HBASE-6405 Addendum fixes RAT checking (Elliot Clark) (Revision 1362696) Result = FAILURE tedyu : Files : * /hbase/trunk/pom.xml > Create Hadoop compatibilty modules and Metrics2 implementation of replication > metrics > - > > Key: HBASE-6405 > URL: https://issues.apache.org/jira/browse/HBASE-6405 > Project: HBase > Issue Type: Sub-task >Reporter: Zhihong Ted Yu >Assignee: Elliott Clark > Fix For: 0.96.0 > > Attachments: 6405.txt, HBASE-6405-ADD.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in "backup mode"
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416737#comment-13416737 ] Hadoop QA commented on HBASE-5547: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536901/java_HBASE-5547_v13.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 22 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2395//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2395//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2395//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2395//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2395//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2395//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2395//console This message is automatically generated. > Don't delete HFiles when in "backup mode" > - > > Key: HBASE-5547 > URL: https://issues.apache.org/jira/browse/HBASE-5547 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Jesse Yates > Fix For: 0.94.2 > > Attachments: 5547-v12.txt, hbase-5447-v8.patch, hbase-5447-v8.patch, > hbase-5547-v9.patch, java_HBASE-5547_v13.patch, java_HBASE-5547_v4.patch, > java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, java_HBASE-5547_v7.patch > > > This came up in a discussion I had with Stack. > It would be nice if HBase could be notified that a backup is in progress (via > a znode for example) and in that case either: > 1. rename HFiles to be delete to .bck > 2. rename the HFiles into a special directory > 3. rename them to a general trash directory (which would not need to be tied > to backup mode). > That way it should be able to get a consistent backup based on HFiles (HDFS > snapshots or hard links would be better options here, but we do not have > those). > #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4050) Update HBase metrics framework to metrics2 framework
[
https://issues.apache.org/jira/browse/HBASE-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416736#comment-13416736
]
Elliott Clark commented on HBASE-4050:
--
{code}
Collection metrics()
{code}
Removing from a linked list (the collection backing the map in question) is
icky. I filed a jira a while ago. HADOOP-8313
yes this is not great. As soon as possible I would love to remove the metrics
factory and all related maps. That would greatly simplify stuff. I think we
should get things working now and refactor away as soon as possible.
> Update HBase metrics framework to metrics2 framework
>
>
> Key: HBASE-4050
> URL: https://issues.apache.org/jira/browse/HBASE-4050
> Project: HBase
> Issue Type: New Feature
> Components: metrics
>Affects Versions: 0.90.4
> Environment: Java 6
>Reporter: Eric Yang
>Assignee: Elliott Clark
>Priority: Critical
> Fix For: 0.96.0
>
> Attachments: 4050-metrics-v2.patch, 4050-metrics-v3.patch,
> HBASE-4050-0.patch, HBASE-4050-1.patch, HBASE-4050-2.patch,
> HBASE-4050-3.patch, HBASE-4050-5.patch, HBASE-4050-6.patch,
> HBASE-4050-7.patch, HBASE-4050-8.patch, HBASE-4050.patch
>
>
> Metrics Framework has been marked deprecated in Hadoop 0.20.203+ and 0.22+,
> and it might get removed in future Hadoop release. Hence, HBase needs to
> revise the dependency of MetricsContext to use Metrics2 framework.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6405) Create Hadoop compatibilty modules and Metrics2 implementation of replication metrics
[ https://issues.apache.org/jira/browse/HBASE-6405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416734#comment-13416734 ] Hudson commented on HBASE-6405: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #97 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/97/]) HBASE-6405 Addendum fixes RAT checking (Elliot Clark) (Revision 1362696) HBASE-6405 Create Hadoop compatibilty modules and Metrics2 implementation of replication metrics (Elliot Clark) (Revision 1362681) Result = FAILURE tedyu : Files : * /hbase/trunk/pom.xml tedyu : Files : * /hbase/trunk/hbase-hadoop-compat * /hbase/trunk/hbase-hadoop-compat/pom.xml * /hbase/trunk/hbase-hadoop-compat/src * /hbase/trunk/hbase-hadoop-compat/src/main * /hbase/trunk/hbase-hadoop-compat/src/main/java * /hbase/trunk/hbase-hadoop-compat/src/main/java/org * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/metrics * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/metrics/BaseMetricsSource.java * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/metrics * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/metrics/ReplicationMetricsSource.java * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/metrics/ReplicationMetricsSourceFactory.java * /hbase/trunk/hbase-hadoop-compat/src/test * /hbase/trunk/hbase-hadoop-compat/src/test/java * /hbase/trunk/hbase-hadoop-compat/src/test/java/org * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/replication * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/replication/regionserver * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/replication/regionserver/metrics * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/replication/regionserver/metrics/ReplicationMetricsSourceFactoryTest.java * /hbase/trunk/hbase-hadoop1-compat * /hbase/trunk/hbase-hadoop1-compat/pom.xml * /hbase/trunk/hbase-hadoop1-compat/src * /hbase/trunk/hbase-hadoop1-compat/src/main * /hbase/trunk/hbase-hadoop1-compat/src/main/java * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/metrics * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/metrics/BaseMetricsSourceImpl.java * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/metrics * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/metrics/ReplicationMetricsSourceImpl.java * /hbase/trunk/hbase-hadoop1-compat/src/main/resources * /hbase/trunk/hbase-hadoop1-compat/src/main/resources/META-INF * /hbase/trunk/hbase-hadoop1-compat/src/main/resources/META-INF/services * /hbase/trunk/hbase-hadoop1-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.replication.regionserver.metrics.ReplicationMetricsSource * /hbase/trunk/hbase-hadoop1-compat/src/test * /hbase/trunk/hbase-hadoop1-compat/src/test/java * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org/apache * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org/apache/hadoop * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org/apache/hadoop/hbase * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org/apache/hadoop/hbase/metrics * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org/apache/hadoop/hbase/metrics/BaseMetricsSourceImplTest.java * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org/apache/hadoop/hbase/replication * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org/apache/hadoop/hbase/replication/regionserver * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org/apache/hadoop/hbase/replication/regionserver/metrics * /hbase/trunk/hbase-hadoop1-compat/src/test/ja
[jira] [Updated] (HBASE-4470) ServerNotRunningException coming out of assignRootAndMeta kills the Master
[
https://issues.apache.org/jira/browse/HBASE-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Chanan updated HBASE-4470:
--
Attachment: HBASE-4470-90.patch
Here's a patch against 90. This adds a test (which I'll forward port to
92/94/96 if this gets +1'ed) and small fixups that are 90 specific. The fixups
cause the test to patch, when it failed previously.
I originally had a more complex test that actually threw the exception out of
getHConnection, but this was invasive (I had to restructure the code for
Mockito purposes) and it throwing out of get may be more resilient (i.e. we'll
probably call "get" when obtaining the meta location forever, but may not
always call getHConection in the future).
> ServerNotRunningException coming out of assignRootAndMeta kills the Master
> --
>
> Key: HBASE-4470
> URL: https://issues.apache.org/jira/browse/HBASE-4470
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
>Assignee: Gregory Chanan
>Priority: Critical
> Fix For: 0.90.7
>
> Attachments: HBASE-4470-90.patch
>
>
> I'm surprised we still have issues like that and I didn't get a hit while
> googling so forgive me if there's already a jira about it.
> When the master starts it verifies the locations of root and meta before
> assigning them, if the server is started but not running you'll get this:
> {quote}
> 2011-09-23 04:47:44,859 WARN
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> RemoteException connecting to RS
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hbase.ipc.ServerNotRunningException: Server is not running
> yet
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1038)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> at $Proxy6.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:969)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.getCachedConnection(CatalogTracker.java:388)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:287)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.verifyMetaRegionLocation(CatalogTracker.java:484)
> at
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:441)
> at
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:388)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:282)
> {quote}
> I hit that 3-4 times this week while debugging something else. The worst is
> that when you restart the master it sees that as a failover, but none of the
> regions are assigned so it takes an eternity to get back fully online.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4050) Update HBase metrics framework to metrics2 framework
[ https://issues.apache.org/jira/browse/HBASE-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416709#comment-13416709 ] Zhihong Ted Yu commented on HBASE-4050: --- bq. filing JIRA to allow metrics removals +1 on above. > Update HBase metrics framework to metrics2 framework > > > Key: HBASE-4050 > URL: https://issues.apache.org/jira/browse/HBASE-4050 > Project: HBase > Issue Type: New Feature > Components: metrics >Affects Versions: 0.90.4 > Environment: Java 6 >Reporter: Eric Yang >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.96.0 > > Attachments: 4050-metrics-v2.patch, 4050-metrics-v3.patch, > HBASE-4050-0.patch, HBASE-4050-1.patch, HBASE-4050-2.patch, > HBASE-4050-3.patch, HBASE-4050-5.patch, HBASE-4050-6.patch, > HBASE-4050-7.patch, HBASE-4050-8.patch, HBASE-4050.patch > > > Metrics Framework has been marked deprecated in Hadoop 0.20.203+ and 0.22+, > and it might get removed in future Hadoop release. Hence, HBase needs to > revise the dependency of MetricsContext to use Metrics2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6261) Better approximate high-percentile percentile latency metrics
[ https://issues.apache.org/jira/browse/HBASE-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416706#comment-13416706 ] Zhihong Ted Yu commented on HBASE-6261: --- bq. copy the code over and then refactor it away +1 on above. > Better approximate high-percentile percentile latency metrics > - > > Key: HBASE-6261 > URL: https://issues.apache.org/jira/browse/HBASE-6261 > Project: HBase > Issue Type: New Feature >Reporter: Andrew Wang >Assignee: Andrew Wang > Labels: metrics > Attachments: Latencyestimation.pdf > > > The existing reservoir-sampling based latency metrics in HBase are not > well-suited for providing accurate estimates of high-percentile (e.g. 90th, > 95th, or 99th) latency. This is a well-studied problem in the literature (see > [1] and [2]), the question is determining which methods best suit our needs > and then implementing it. > Ideally, we should be able to estimate these high percentiles with minimal > memory and CPU usage as well as minimal error (e.g. 1% error on 90th, or .1% > on 99th). It's also desirable to provide this over different time-based > sliding windows, e.g. last 1 min, 5 mins, 15 mins, and 1 hour. > I'll note that this would also be useful in HDFS, or really anywhere latency > metrics are kept. > [1] http://www.cs.rutgers.edu/~muthu/bquant.pdf > [2] http://infolab.stanford.edu/~manku/papers/04pods-sliding.pdf -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6416) hbck dies on NPE when a region folder exists but the table does not
Jean-Daniel Cryans created HBASE-6416:
-
Summary: hbck dies on NPE when a region folder exists but the
table does not
Key: HBASE-6416
URL: https://issues.apache.org/jira/browse/HBASE-6416
Project: HBase
Issue Type: Bug
Reporter: Jean-Daniel Cryans
Fix For: 0.96.0, 0.94.2
This is what I'm getting for leftover data that has no .regioninfo
First:
{quote}
12/07/17 23:13:37 WARN util.HBaseFsck: Failed to read .regioninfo file for
region null
java.io.FileNotFoundException: File does not exist:
/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46/.regioninfo
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.(DFSClient.java:1813)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:611)
at
org.apache.hadoop.hbase.util.HBaseFsck.access$2200(HBaseFsck.java:140)
at
org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2882)
at
org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2866)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
Then it hangs on:
{quote}
12/07/17 23:13:39 INFO util.HBaseFsck: Attempting to handle orphan hdfs dir:
hdfs://sfor3s24:10101/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46
12/07/17 23:13:39 INFO util.HBaseFsck: checking orphan for table null
Exception in thread "main" java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$100(HBaseFsck.java:1634)
at
org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:435)
at
org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:408)
at
org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:529)
at
org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:313)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:386)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3227)
{quote}
The NPE is sent by:
{code}
Preconditions.checkNotNull("Table " + tableName + "' not present!", tableInfo);
{code}
I wonder why the condition checking was added if we don't handle it... In any
case hbck dies but it hangs because there are some non-daemon hanging around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6228) Fixup daughters twice cause daughter region assigned twice
[ https://issues.apache.org/jira/browse/HBASE-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416697#comment-13416697 ] Jimmy Xiang commented on HBASE-6228: @Chunhui, this is a very rare case. Is it better to synchronize SSH.isDaughterMissing()? This method is not in critical path, so there should be no performance issue. > Fixup daughters twice cause daughter region assigned twice > --- > > Key: HBASE-6228 > URL: https://issues.apache.org/jira/browse/HBASE-6228 > Project: HBase > Issue Type: Bug > Components: master >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0 > > Attachments: HBASE-6228.patch, HBASE-6228v2.patch, > HBASE-6228v2.patch, HBASE-6228v3.patch, HBASE-6228v4.patch > > > First, how fixup daughters twice happen? > 1.we will fixupDaughters at the last of HMaster#finishInitialization > 2.ServerShutdownHandler will fixupDaughters when reassigning region through > ServerShutdownHandler#processDeadRegion > When fixupDaughters, we will added daughters to .META., but it coudn't > prevent the above case, because FindDaughterVisitor. > The detail is as the following: > Suppose region A is a splitted parent region, and its daughter region B is > missing > 1.First, ServerShutdownHander thread fixup daughter, so add daughter region B > to .META. with serverName=null, and assign the daughter. > 2.Then, Master's initialization thread will also find the daughter region B > is missing and assign it. It is because FindDaughterVisitor consider daughter > is missing if its serverName=null -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5547) Don't delete HFiles when in "backup mode"
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5547: --- Attachment: java_HBASE-5547_v13.patch Attaching updated version based on trunk and addressing comments on RB. Also, had some replies to outstanding issues on RB. > Don't delete HFiles when in "backup mode" > - > > Key: HBASE-5547 > URL: https://issues.apache.org/jira/browse/HBASE-5547 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Jesse Yates > Fix For: 0.94.2 > > Attachments: 5547-v12.txt, hbase-5447-v8.patch, hbase-5447-v8.patch, > hbase-5547-v9.patch, java_HBASE-5547_v13.patch, java_HBASE-5547_v4.patch, > java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, java_HBASE-5547_v7.patch > > > This came up in a discussion I had with Stack. > It would be nice if HBase could be notified that a backup is in progress (via > a znode for example) and in that case either: > 1. rename HFiles to be delete to .bck > 2. rename the HFiles into a special directory > 3. rename them to a general trash directory (which would not need to be tied > to backup mode). > That way it should be able to get a consistent backup based on HFiles (HDFS > snapshots or hard links would be better options here, but we do not have > those). > #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4050) Update HBase metrics framework to metrics2 framework
[ https://issues.apache.org/jira/browse/HBASE-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416681#comment-13416681 ] Alex Baranau commented on HBASE-4050: - Looks to me that we are going to copy-paste MetricsRegistry with one addition: allow remove metrics. Not nice. Btw, it seems to me (if I'm not mistaken) that there's a way to remove metric from MetricRegistry. Though it is far from nice, may be it is OK to depend on it and file a JIRA issue to allow removals? There's a method metrics() that returns map entries (in hadoop1) or map values (in hadoop2) which are backing internal map of metrics and thus can be used to remove items. However: 1. searching in list for item to remove is not performance efficient. Do we want to do it frequently? I suppose we may, but not sure 2. in general it's very bad to rely on implementation details (in future they can return copy of values collection) Having said that, is it better to duplicate MetricsRegistry class in our code or use/rely on that metrics() method for now + filing JIRA to allow metrics removals? > Update HBase metrics framework to metrics2 framework > > > Key: HBASE-4050 > URL: https://issues.apache.org/jira/browse/HBASE-4050 > Project: HBase > Issue Type: New Feature > Components: metrics >Affects Versions: 0.90.4 > Environment: Java 6 >Reporter: Eric Yang >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.96.0 > > Attachments: 4050-metrics-v2.patch, 4050-metrics-v3.patch, > HBASE-4050-0.patch, HBASE-4050-1.patch, HBASE-4050-2.patch, > HBASE-4050-3.patch, HBASE-4050-5.patch, HBASE-4050-6.patch, > HBASE-4050-7.patch, HBASE-4050-8.patch, HBASE-4050.patch > > > Metrics Framework has been marked deprecated in Hadoop 0.20.203+ and 0.22+, > and it might get removed in future Hadoop release. Hence, HBase needs to > revise the dependency of MetricsContext to use Metrics2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6261) Better approximate high-percentile percentile latency metrics
[ https://issues.apache.org/jira/browse/HBASE-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416680#comment-13416680 ] Andrew Wang commented on HBASE-6261: Sorry I haven't had time to push on this more. I talked with Jon Hsieh last week about doing a more convincing analysis of the performance of the new MutableQuantiles class from HADOOP-8541 vs the existing reservoir-sampling histogram method. I'll try to get that done within a week. I'm also not sure about the right course of action at getting it used in HBase. Stack indicated way back on the mailing list that he was okay waiting for a hadoop-common version bump, which is kind of a long timescale. If people really urgently want this, we could just copy the code over and then refactor it away when it's released in hadoop-common. > Better approximate high-percentile percentile latency metrics > - > > Key: HBASE-6261 > URL: https://issues.apache.org/jira/browse/HBASE-6261 > Project: HBase > Issue Type: New Feature >Reporter: Andrew Wang >Assignee: Andrew Wang > Labels: metrics > Attachments: Latencyestimation.pdf > > > The existing reservoir-sampling based latency metrics in HBase are not > well-suited for providing accurate estimates of high-percentile (e.g. 90th, > 95th, or 99th) latency. This is a well-studied problem in the literature (see > [1] and [2]), the question is determining which methods best suit our needs > and then implementing it. > Ideally, we should be able to estimate these high percentiles with minimal > memory and CPU usage as well as minimal error (e.g. 1% error on 90th, or .1% > on 99th). It's also desirable to provide this over different time-based > sliding windows, e.g. last 1 min, 5 mins, 15 mins, and 1 hour. > I'll note that this would also be useful in HDFS, or really anywhere latency > metrics are kept. > [1] http://www.cs.rutgers.edu/~muthu/bquant.pdf > [2] http://infolab.stanford.edu/~manku/papers/04pods-sliding.pdf -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6405) Create Hadoop compatibilty modules and Metrics2 implementation of replication metrics
[ https://issues.apache.org/jira/browse/HBASE-6405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416679#comment-13416679 ] Zhihong Ted Yu commented on HBASE-6405: --- Addendum checked in. Thanks for the quick turn-around, Elliot. > Create Hadoop compatibilty modules and Metrics2 implementation of replication > metrics > - > > Key: HBASE-6405 > URL: https://issues.apache.org/jira/browse/HBASE-6405 > Project: HBase > Issue Type: Sub-task >Reporter: Zhihong Ted Yu >Assignee: Elliott Clark > Fix For: 0.96.0 > > Attachments: 6405.txt, HBASE-6405-ADD.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6409) Create histogram class for metrics 2
[ https://issues.apache.org/jira/browse/HBASE-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416676#comment-13416676 ] Elliott Clark commented on HBASE-6409: -- Until they are in a version that is released and we've rev'd to that we can't use the hadoop implementation. > Create histogram class for metrics 2 > > > Key: HBASE-6409 > URL: https://issues.apache.org/jira/browse/HBASE-6409 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > > Create the replacement for MetricsHistogram and PersistantTimeVaryingRate > classes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6405) Create Hadoop compatibilty modules and Metrics2 implementation of replication metrics
[ https://issues.apache.org/jira/browse/HBASE-6405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-6405: - Attachment: HBASE-6405-ADD.patch Fix rat. I also filed: HBASE-6415 as it looks like trunk has been broken on rat for a while and the pre-commit hooks didn't alert us when the broken checkin was committed. > Create Hadoop compatibilty modules and Metrics2 implementation of replication > metrics > - > > Key: HBASE-6405 > URL: https://issues.apache.org/jira/browse/HBASE-6405 > Project: HBase > Issue Type: Sub-task >Reporter: Zhihong Ted Yu >Assignee: Elliott Clark > Fix For: 0.96.0 > > Attachments: 6405.txt, HBASE-6405-ADD.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6415) Add Rat checking to the pre-checkin script.
Elliott Clark created HBASE-6415: Summary: Add Rat checking to the pre-checkin script. Key: HBASE-6415 URL: https://issues.apache.org/jira/browse/HBASE-6415 Project: HBase Issue Type: Task Reporter: Elliott Clark Assignee: Elliott Clark -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6406) TestReplicationPeer.testResetZooKeeperSession and TestZooKeeper.testClientSessionExpired fail frequently
[
https://issues.apache.org/jira/browse/HBASE-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416673#comment-13416673
]
Lars Hofhansl commented on HBASE-6406:
--
The first one fails due to a race condition:
{quote}
2012-07-16 03:36:04,631 INFO [pool-1-thread-1]
zookeeper.MiniZooKeeperCluster(193): Started MiniZK Cluster and connect 1 ZK
server on client port: 61529
2012-07-16 03:36:04,760 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(100):
connection to cluster: clusterId opening connection to ZooKeeper with ensemble
(localhost:61529)
2012-07-16 03:36:04,831 INFO [pool-1-thread-1]
zookeeper.RecoverableZooKeeper(97): The identifier of this process is
22...@vesta.apache.org
2012-07-16 03:36:04,927 INFO [pool-1-thread-1] hbase.ResourceChecker(145):
before replication.TestReplicationPeer#testResetZooKeeperSession: 11 threads,
86 file descriptors 0 connections,
2012-07-16 03:36:07,918 DEBUG [pool-1-thread-1-EventThread]
zookeeper.ZooKeeperWatcher(262): connection to cluster: clusterId Received
ZooKeeper Event, type=None, state=SyncConnected, path=null
2012-07-16 03:36:07,926 INFO [Thread-2] replication.TestReplicationPeer(54):
Expiring ReplicationPeer ZooKeeper session.
2012-07-16 03:36:07,950 DEBUG [pool-1-thread-1-EventThread]
zookeeper.ZooKeeperWatcher(339): connection to cluster:
clusterId-0x1388ddb6141 connected
2012-07-16 03:36:08,091 INFO [Thread-2] hbase.HBaseTestingUtility(1344): ZK
Closed Session 0x1388ddb6141; sleeping=7000
2012-07-16 03:36:15,092 INFO [Thread-2] replication.TestReplicationPeer(58):
Attempting to use expired ReplicationPeer ZooKeeper session.
2012-07-16 03:36:15,095 INFO [pool-1-thread-1] hbase.ResourceChecker(145):
after replication.TestReplicationPeer#testResetZooKeeperSession: 11 threads
(was 11), 89 file descriptors (was 89). 0 connections,
{quote}
A successful run looks like this:
{quote}
2012-07-17 15:20:35,285 INFO [main] zookeeper.MiniZooKeeperCluster(193):
Started MiniZK Cluster and connect 1 ZK server on client port: 49834
2012-07-17 15:20:35,298 DEBUG [main] zookeeper.ZKUtil(100): connection to
cluster: clusterId opening connection to ZooKeeper with ensemble
(localhost:49834)
2012-07-17 15:20:35,312 INFO [main] zookeeper.RecoverableZooKeeper(97): The
identifier of this process is 26186@
2012-07-17 15:20:35,336 DEBUG [main-EventThread]
zookeeper.ZooKeeperWatcher(262): connection to cluster: clusterId Received
ZooKeeper Event, type=None, state=SyncConnected, path=null
2012-07-17 15:20:35,338 DEBUG [main-EventThread]
zookeeper.ZooKeeperWatcher(339): connection to cluster:
clusterId-0x1389707502a connected
2012-07-17 15:20:35,348 INFO [main] hbase.ResourceChecker(145): before
replication.TestReplicationPeer#testResetZooKeeperSession: 10 threads, 87 file
descriptors 0 connections,
2012-07-17 15:20:35,356 INFO [Thread-2] replication.TestReplicationPeer(56):
Expiring ReplicationPeer ZooKeeper session.
2012-07-17 15:20:35,360 INFO [Thread-2] hbase.HBaseTestingUtility(1344): ZK
Closed Session 0x1389707502a; sleeping=7000
2012-07-17 15:20:35,459 DEBUG [main-EventThread]
zookeeper.ZooKeeperWatcher(262): connection to cluster:
clusterId-0x1389707502a Received ZooKeeper Event, type=None,
state=Disconnected, path=null
2012-07-17 15:20:35,459 DEBUG [main-EventThread]
zookeeper.ZooKeeperWatcher(360): connection to cluster:
clusterId-0x1389707502a Received Disconnected from ZooKeeper, ignoring
2012-07-17 15:20:37,267 DEBUG [main-EventThread]
zookeeper.ZooKeeperWatcher(262): connection to cluster:
clusterId-0x1389707502a Received ZooKeeper Event, type=None, state=Expired,
path=null
2012-07-17 15:20:37,269 WARN [main-EventThread]
replication.ReplicationPeer(157): The ReplicationPeer coresponding to peer
clusterKey was aborted for the following reason(s):connection to cluster:
clusterId-0x1389707502a connection to cluster: clusterId-0x1389707502a
received expired from ZooKeeper, aborting
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode =
Session expired
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
2012-07-17 15:20:42,360 INFO [Thread-2] replication.TestReplicationPeer(60):
Attempting to use expired ReplicationPeer ZooKeeper session.
2012-07-17 15:20:42,362 DEBUG [Thread-2] zookeeper.ZKUtil(100): connection to
cluster: clusterId opening connection to ZooKeeper with ensemble
(localhost:49834)
2012-07-17 15:20:42,363 INFO [Thread-2] zookeeper.RecoverableZooKeeper(97):
The identifier of this process is 26186@
2012-07-17 15:20:42,364 INFO [Thread-2] rep
[jira] [Commented] (HBASE-6409) Create histogram class for metrics 2
[ https://issues.apache.org/jira/browse/HBASE-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416672#comment-13416672 ] Andrew Wang commented on HBASE-6409: This seems related to HBASE-6261 and HADOOP-8541, which introduced a new MutableQuantiles class into hadoop-common metrics2. You could consider using that in lieu of rolling your own histogram. > Create histogram class for metrics 2 > > > Key: HBASE-6409 > URL: https://issues.apache.org/jira/browse/HBASE-6409 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > > Create the replacement for MetricsHistogram and PersistantTimeVaryingRate > classes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4050) Update HBase metrics framework to metrics2 framework
[ https://issues.apache.org/jira/browse/HBASE-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416667#comment-13416667 ] Elliott Clark commented on HBASE-4050: -- s/pre region/per region/g > Update HBase metrics framework to metrics2 framework > > > Key: HBASE-4050 > URL: https://issues.apache.org/jira/browse/HBASE-4050 > Project: HBase > Issue Type: New Feature > Components: metrics >Affects Versions: 0.90.4 > Environment: Java 6 >Reporter: Eric Yang >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.96.0 > > Attachments: 4050-metrics-v2.patch, 4050-metrics-v3.patch, > HBASE-4050-0.patch, HBASE-4050-1.patch, HBASE-4050-2.patch, > HBASE-4050-3.patch, HBASE-4050-5.patch, HBASE-4050-6.patch, > HBASE-4050-7.patch, HBASE-4050-8.patch, HBASE-4050.patch > > > Metrics Framework has been marked deprecated in Hadoop 0.20.203+ and 0.22+, > and it might get removed in future Hadoop release. Hence, HBase needs to > revise the dependency of MetricsContext to use Metrics2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4050) Update HBase metrics framework to metrics2 framework
[ https://issues.apache.org/jira/browse/HBASE-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416665#comment-13416665 ] Elliott Clark commented on HBASE-4050: -- # Yes that does for sure. # When replication sources are turned off we need to remove them. Also pre region metrics and per table metrics need to be removed when they are no longer on a rs. > Update HBase metrics framework to metrics2 framework > > > Key: HBASE-4050 > URL: https://issues.apache.org/jira/browse/HBASE-4050 > Project: HBase > Issue Type: New Feature > Components: metrics >Affects Versions: 0.90.4 > Environment: Java 6 >Reporter: Eric Yang >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.96.0 > > Attachments: 4050-metrics-v2.patch, 4050-metrics-v3.patch, > HBASE-4050-0.patch, HBASE-4050-1.patch, HBASE-4050-2.patch, > HBASE-4050-3.patch, HBASE-4050-5.patch, HBASE-4050-6.patch, > HBASE-4050-7.patch, HBASE-4050-8.patch, HBASE-4050.patch > > > Metrics Framework has been marked deprecated in Hadoop 0.20.203+ and 0.22+, > and it might get removed in future Hadoop release. Hence, HBase needs to > revise the dependency of MetricsContext to use Metrics2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4050) Update HBase metrics framework to metrics2 framework
[ https://issues.apache.org/jira/browse/HBASE-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416664#comment-13416664 ] Alex Baranau commented on HBASE-4050: - bq. After I saw Ted create the first sub task I created all the rest. Alex I assigned some to you. Great, thanx! > Update HBase metrics framework to metrics2 framework > > > Key: HBASE-4050 > URL: https://issues.apache.org/jira/browse/HBASE-4050 > Project: HBase > Issue Type: New Feature > Components: metrics >Affects Versions: 0.90.4 > Environment: Java 6 >Reporter: Eric Yang >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.96.0 > > Attachments: 4050-metrics-v2.patch, 4050-metrics-v3.patch, > HBASE-4050-0.patch, HBASE-4050-1.patch, HBASE-4050-2.patch, > HBASE-4050-3.patch, HBASE-4050-5.patch, HBASE-4050-6.patch, > HBASE-4050-7.patch, HBASE-4050-8.patch, HBASE-4050.patch > > > Metrics Framework has been marked deprecated in Hadoop 0.20.203+ and 0.22+, > and it might get removed in future Hadoop release. Hence, HBase needs to > revise the dependency of MetricsContext to use Metrics2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4050) Update HBase metrics framework to metrics2 framework
[ https://issues.apache.org/jira/browse/HBASE-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416661#comment-13416661 ] Alex Baranau commented on HBASE-4050: - 2Elliott, just to understand: bq. MetricsRegistry does not allow metrics to be removed. 1. As we don't use MetricsRegistry class and basically reimplementing it inside BaseMetricsSourceImpl, should we also allow adding tags (MetricsTag) there? Sounds like a useful feature. 2. Why do we need to remove metrics? > Update HBase metrics framework to metrics2 framework > > > Key: HBASE-4050 > URL: https://issues.apache.org/jira/browse/HBASE-4050 > Project: HBase > Issue Type: New Feature > Components: metrics >Affects Versions: 0.90.4 > Environment: Java 6 >Reporter: Eric Yang >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.96.0 > > Attachments: 4050-metrics-v2.patch, 4050-metrics-v3.patch, > HBASE-4050-0.patch, HBASE-4050-1.patch, HBASE-4050-2.patch, > HBASE-4050-3.patch, HBASE-4050-5.patch, HBASE-4050-6.patch, > HBASE-4050-7.patch, HBASE-4050-8.patch, HBASE-4050.patch > > > Metrics Framework has been marked deprecated in Hadoop 0.20.203+ and 0.22+, > and it might get removed in future Hadoop release. Hence, HBase needs to > revise the dependency of MetricsContext to use Metrics2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6405) Create Hadoop compatibilty modules and Metrics2 implementation of replication metrics
[ https://issues.apache.org/jira/browse/HBASE-6405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416659#comment-13416659 ] Hudson commented on HBASE-6405: --- Integrated in HBase-TRUNK #3141 (See [https://builds.apache.org/job/HBase-TRUNK/3141/]) HBASE-6405 Create Hadoop compatibilty modules and Metrics2 implementation of replication metrics (Elliot Clark) (Revision 1362681) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-hadoop-compat * /hbase/trunk/hbase-hadoop-compat/pom.xml * /hbase/trunk/hbase-hadoop-compat/src * /hbase/trunk/hbase-hadoop-compat/src/main * /hbase/trunk/hbase-hadoop-compat/src/main/java * /hbase/trunk/hbase-hadoop-compat/src/main/java/org * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/metrics * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/metrics/BaseMetricsSource.java * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/metrics * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/metrics/ReplicationMetricsSource.java * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/metrics/ReplicationMetricsSourceFactory.java * /hbase/trunk/hbase-hadoop-compat/src/test * /hbase/trunk/hbase-hadoop-compat/src/test/java * /hbase/trunk/hbase-hadoop-compat/src/test/java/org * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/replication * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/replication/regionserver * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/replication/regionserver/metrics * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/replication/regionserver/metrics/ReplicationMetricsSourceFactoryTest.java * /hbase/trunk/hbase-hadoop1-compat * /hbase/trunk/hbase-hadoop1-compat/pom.xml * /hbase/trunk/hbase-hadoop1-compat/src * /hbase/trunk/hbase-hadoop1-compat/src/main * /hbase/trunk/hbase-hadoop1-compat/src/main/java * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/metrics * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/metrics/BaseMetricsSourceImpl.java * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/metrics * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/metrics/ReplicationMetricsSourceImpl.java * /hbase/trunk/hbase-hadoop1-compat/src/main/resources * /hbase/trunk/hbase-hadoop1-compat/src/main/resources/META-INF * /hbase/trunk/hbase-hadoop1-compat/src/main/resources/META-INF/services * /hbase/trunk/hbase-hadoop1-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.replication.regionserver.metrics.ReplicationMetricsSource * /hbase/trunk/hbase-hadoop1-compat/src/test * /hbase/trunk/hbase-hadoop1-compat/src/test/java * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org/apache * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org/apache/hadoop * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org/apache/hadoop/hbase * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org/apache/hadoop/hbase/metrics * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org/apache/hadoop/hbase/metrics/BaseMetricsSourceImplTest.java * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org/apache/hadoop/hbase/replication * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org/apache/hadoop/hbase/replication/regionserver * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org/apache/hadoop/hbase/replication/regionserver/metrics * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org/apache/hadoop/hbase/replication/regionserver/metrics/ReplicationMetricsSourceImplTest.java * /hbase/trunk/hbase-hadoop2-compat * /hbase/
[jira] [Created] (HBASE-6414) Remove the WritableRpcEngine & associated Invocation classes
Devaraj Das created HBASE-6414: -- Summary: Remove the WritableRpcEngine & associated Invocation classes Key: HBASE-6414 URL: https://issues.apache.org/jira/browse/HBASE-6414 Project: HBase Issue Type: Improvement Affects Versions: 0.96.0 Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.96.0 Remove the WritableRpcEngine & Invocation classes once HBASE-5705 gets committed and all the protocols are rebased to use PB. Raising this jira in advance.. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6405) Create Hadoop compatibilty modules and Metrics2 implementation of replication metrics
[ https://issues.apache.org/jira/browse/HBASE-6405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416654#comment-13416654 ] Elliott Clark commented on HBASE-6405: -- Thanks > Create Hadoop compatibilty modules and Metrics2 implementation of replication > metrics > - > > Key: HBASE-6405 > URL: https://issues.apache.org/jira/browse/HBASE-6405 > Project: HBase > Issue Type: Sub-task >Reporter: Zhihong Ted Yu >Assignee: Elliott Clark > Fix For: 0.96.0 > > Attachments: 6405.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4050) Update HBase metrics framework to metrics2 framework
[ https://issues.apache.org/jira/browse/HBASE-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416653#comment-13416653 ] Elliott Clark commented on HBASE-4050: -- There. After I saw Ted create the first sub task I created all the rest. Alex I assigned some to you. Feel free to assign them as you see fit. > Update HBase metrics framework to metrics2 framework > > > Key: HBASE-4050 > URL: https://issues.apache.org/jira/browse/HBASE-4050 > Project: HBase > Issue Type: New Feature > Components: metrics >Affects Versions: 0.90.4 > Environment: Java 6 >Reporter: Eric Yang >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.96.0 > > Attachments: 4050-metrics-v2.patch, 4050-metrics-v3.patch, > HBASE-4050-0.patch, HBASE-4050-1.patch, HBASE-4050-2.patch, > HBASE-4050-3.patch, HBASE-4050-5.patch, HBASE-4050-6.patch, > HBASE-4050-7.patch, HBASE-4050-8.patch, HBASE-4050.patch > > > Metrics Framework has been marked deprecated in Hadoop 0.20.203+ and 0.22+, > and it might get removed in future Hadoop release. Hence, HBase needs to > revise the dependency of MetricsContext to use Metrics2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6405) Create Hadoop compatibilty modules and Metrics2 implementation of replication metrics
[ https://issues.apache.org/jira/browse/HBASE-6405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416651#comment-13416651 ] Zhihong Ted Yu commented on HBASE-6405: --- Integrated to trunk. Thanks for the patch, Elliot. Thanks for the review, Stack. > Create Hadoop compatibilty modules and Metrics2 implementation of replication > metrics > - > > Key: HBASE-6405 > URL: https://issues.apache.org/jira/browse/HBASE-6405 > Project: HBase > Issue Type: Sub-task >Reporter: Zhihong Ted Yu >Assignee: Elliott Clark > Fix For: 0.96.0 > > Attachments: 6405.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6413) Investigate having hbase-env.sh decide which hadoop-compat to include
Elliott Clark created HBASE-6413: Summary: Investigate having hbase-env.sh decide which hadoop-compat to include Key: HBASE-6413 URL: https://issues.apache.org/jira/browse/HBASE-6413 Project: HBase Issue Type: Sub-task Reporter: Elliott Clark Allow for one package to be created with both compat jars in and have hbase-env load the correct one. This would simplify shipping tar.gz's -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6412) Move external servers to metrics2 (thrift,thrift2,rest)
Elliott Clark created HBASE-6412: Summary: Move external servers to metrics2 (thrift,thrift2,rest) Key: HBASE-6412 URL: https://issues.apache.org/jira/browse/HBASE-6412 Project: HBase Issue Type: Sub-task Reporter: Elliott Clark Implement metrics2 for all the external servers: * Thrift * Thrift2 * Rest * Avro ? (Not sure if we should do this as it's deprecated.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6410) Move RegionServer Metrics to metrics2
Elliott Clark created HBASE-6410: Summary: Move RegionServer Metrics to metrics2 Key: HBASE-6410 URL: https://issues.apache.org/jira/browse/HBASE-6410 Project: HBase Issue Type: Sub-task Reporter: Elliott Clark Assignee: Alex Baranau Move RegionServer Metrics to metrics2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4470) ServerNotRunningException coming out of assignRootAndMeta kills the Master
[
https://issues.apache.org/jira/browse/HBASE-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Chanan updated HBASE-4470:
--
Assignee: Gregory Chanan
> ServerNotRunningException coming out of assignRootAndMeta kills the Master
> --
>
> Key: HBASE-4470
> URL: https://issues.apache.org/jira/browse/HBASE-4470
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
>Assignee: Gregory Chanan
>Priority: Critical
> Fix For: 0.90.7
>
>
> I'm surprised we still have issues like that and I didn't get a hit while
> googling so forgive me if there's already a jira about it.
> When the master starts it verifies the locations of root and meta before
> assigning them, if the server is started but not running you'll get this:
> {quote}
> 2011-09-23 04:47:44,859 WARN
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> RemoteException connecting to RS
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hbase.ipc.ServerNotRunningException: Server is not running
> yet
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1038)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> at $Proxy6.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:969)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.getCachedConnection(CatalogTracker.java:388)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:287)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.verifyMetaRegionLocation(CatalogTracker.java:484)
> at
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:441)
> at
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:388)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:282)
> {quote}
> I hit that 3-4 times this week while debugging something else. The worst is
> that when you restart the master it sees that as a failover, but none of the
> regions are assigned so it takes an eternity to get back fully online.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4470) ServerNotRunningException coming out of assignRootAndMeta kills the Master
[
https://issues.apache.org/jira/browse/HBASE-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416646#comment-13416646
]
Gregory Chanan commented on HBASE-4470:
---
@Jon I'm working on a test, should be done soon.
> ServerNotRunningException coming out of assignRootAndMeta kills the Master
> --
>
> Key: HBASE-4470
> URL: https://issues.apache.org/jira/browse/HBASE-4470
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
>Priority: Critical
> Fix For: 0.90.7
>
>
> I'm surprised we still have issues like that and I didn't get a hit while
> googling so forgive me if there's already a jira about it.
> When the master starts it verifies the locations of root and meta before
> assigning them, if the server is started but not running you'll get this:
> {quote}
> 2011-09-23 04:47:44,859 WARN
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> RemoteException connecting to RS
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hbase.ipc.ServerNotRunningException: Server is not running
> yet
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1038)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> at $Proxy6.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:969)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.getCachedConnection(CatalogTracker.java:388)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:287)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.verifyMetaRegionLocation(CatalogTracker.java:484)
> at
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:441)
> at
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:388)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:282)
> {quote}
> I hit that 3-4 times this week while debugging something else. The worst is
> that when you restart the master it sees that as a failover, but none of the
> regions are assigned so it takes an eternity to get back fully online.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6411) Move Master Metrics to metrics 2
Elliott Clark created HBASE-6411: Summary: Move Master Metrics to metrics 2 Key: HBASE-6411 URL: https://issues.apache.org/jira/browse/HBASE-6411 Project: HBase Issue Type: Sub-task Reporter: Elliott Clark Assignee: Alex Baranau Move Master Metrics to metrics 2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6409) Create histogram class for metrics 2
Elliott Clark created HBASE-6409: Summary: Create histogram class for metrics 2 Key: HBASE-6409 URL: https://issues.apache.org/jira/browse/HBASE-6409 Project: HBase Issue Type: Sub-task Reporter: Elliott Clark Assignee: Elliott Clark Create the replacement for MetricsHistogram and PersistantTimeVaryingRate classes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6408) Naming and documenting of the hadoop-metrics2.properties file
Elliott Clark created HBASE-6408: Summary: Naming and documenting of the hadoop-metrics2.properties file Key: HBASE-6408 URL: https://issues.apache.org/jira/browse/HBASE-6408 Project: HBase Issue Type: Sub-task Reporter: Elliott Clark hadoop-metrics2.properties is currently where metrics2 loads it's sinks. This file could be better named, hadoop-hbase-metrics2.properties In addition it needs examples like the current hadoop-metrics.properties has. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6407) Investigate moving to DI (guice) framework for plugin arch.
Elliott Clark created HBASE-6407: Summary: Investigate moving to DI (guice) framework for plugin arch. Key: HBASE-6407 URL: https://issues.apache.org/jira/browse/HBASE-6407 Project: HBase Issue Type: Sub-task Reporter: Elliott Clark Assignee: Elliott Clark Investigate using Guice to inject the correct compat object provided by compat plugins -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4470) ServerNotRunningException coming out of assignRootAndMeta kills the Master
[
https://issues.apache.org/jira/browse/HBASE-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416640#comment-13416640
]
Jonathan Hsieh commented on HBASE-4470:
---
[~gchanan] So if this was fixed in HBASE-5883 do you think we can legitimately
close this out?
> ServerNotRunningException coming out of assignRootAndMeta kills the Master
> --
>
> Key: HBASE-4470
> URL: https://issues.apache.org/jira/browse/HBASE-4470
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
>Priority: Critical
> Fix For: 0.90.7
>
>
> I'm surprised we still have issues like that and I didn't get a hit while
> googling so forgive me if there's already a jira about it.
> When the master starts it verifies the locations of root and meta before
> assigning them, if the server is started but not running you'll get this:
> {quote}
> 2011-09-23 04:47:44,859 WARN
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> RemoteException connecting to RS
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hbase.ipc.ServerNotRunningException: Server is not running
> yet
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1038)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> at $Proxy6.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:969)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.getCachedConnection(CatalogTracker.java:388)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:287)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.verifyMetaRegionLocation(CatalogTracker.java:484)
> at
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:441)
> at
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:388)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:282)
> {quote}
> I hit that 3-4 times this week while debugging something else. The worst is
> that when you restart the master it sees that as a failover, but none of the
> regions are assigned so it takes an eternity to get back fully online.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6406) TestReplicationPeer.testResetZooKeeperSession and TestZooKeeper.testClientSessionExpired fail frequently
Lars Hofhansl created HBASE-6406: Summary: TestReplicationPeer.testResetZooKeeperSession and TestZooKeeper.testClientSessionExpired fail frequently Key: HBASE-6406 URL: https://issues.apache.org/jira/browse/HBASE-6406 Project: HBase Issue Type: Bug Affects Versions: 0.94.1 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.2 Looking back through the 0.94 test runs these two tests accounted for 11 of 34 failed tests. They should be fixed or (temporarily) disabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6405) Create Hadoop compatibilty modules and Metrics2 implementation of replication metrics
[ https://issues.apache.org/jira/browse/HBASE-6405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu updated HBASE-6405: -- Attachment: 6405.txt Same patch as 4050-8.patch from Elliot. > Create Hadoop compatibilty modules and Metrics2 implementation of replication > metrics > - > > Key: HBASE-6405 > URL: https://issues.apache.org/jira/browse/HBASE-6405 > Project: HBase > Issue Type: Sub-task >Reporter: Zhihong Ted Yu >Assignee: Elliott Clark > Fix For: 0.96.0 > > Attachments: 6405.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4050) Update HBase metrics framework to metrics2 framework
[ https://issues.apache.org/jira/browse/HBASE-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu updated HBASE-4050: -- Status: Open (was: Patch Available) > Update HBase metrics framework to metrics2 framework > > > Key: HBASE-4050 > URL: https://issues.apache.org/jira/browse/HBASE-4050 > Project: HBase > Issue Type: New Feature > Components: metrics >Affects Versions: 0.90.4 > Environment: Java 6 >Reporter: Eric Yang >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.96.0 > > Attachments: 4050-metrics-v2.patch, 4050-metrics-v3.patch, > HBASE-4050-0.patch, HBASE-4050-1.patch, HBASE-4050-2.patch, > HBASE-4050-3.patch, HBASE-4050-5.patch, HBASE-4050-6.patch, > HBASE-4050-7.patch, HBASE-4050-8.patch, HBASE-4050.patch > > > Metrics Framework has been marked deprecated in Hadoop 0.20.203+ and 0.22+, > and it might get removed in future Hadoop release. Hence, HBase needs to > revise the dependency of MetricsContext to use Metrics2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6404) Collect p50, p75 and p95 stats
[ https://issues.apache.org/jira/browse/HBASE-6404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416635#comment-13416635 ] Elliott Clark commented on HBASE-6404: -- Moving things to MetricsHistogram (or whatever the new metrics2 class name will be) will do this. However it comes with a cost so we should be judicious about it's usage. > Collect p50, p75 and p95 stats > -- > > Key: HBASE-6404 > URL: https://issues.apache.org/jira/browse/HBASE-6404 > Project: HBase > Issue Type: Improvement > Components: monitoring >Reporter: Arjen Roodselaar >Assignee: Liyin Tang > > Stats in current versions of HBase are currently exposed as avg, min and max. > This gives a skewed view of performance as the outliers are usually the > indicators of problems. Please revise the stats collection framework to use > true buckets and expose the p50, p75 and p95 values of these buckets through > JMX. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6405) Create Hadoop compatibilty modules and Metrics2 implementation of replication metrics
Zhihong Ted Yu created HBASE-6405: - Summary: Create Hadoop compatibilty modules and Metrics2 implementation of replication metrics Key: HBASE-6405 URL: https://issues.apache.org/jira/browse/HBASE-6405 Project: HBase Issue Type: Sub-task Reporter: Zhihong Ted Yu Assignee: Elliott Clark Fix For: 0.96.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6404) Collect p50, p75 and p95 stats
[ https://issues.apache.org/jira/browse/HBASE-6404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjen Roodselaar reassigned HBASE-6404: --- Assignee: Liyin Tang Feel free to re-assign to someone in the eng team and/or adjust the priority. This feature will really help us move forward performance wise, so it would be nice if it would be available in our next major release in the not too distant future. > Collect p50, p75 and p95 stats > -- > > Key: HBASE-6404 > URL: https://issues.apache.org/jira/browse/HBASE-6404 > Project: HBase > Issue Type: Improvement > Components: monitoring >Reporter: Arjen Roodselaar >Assignee: Liyin Tang > > Stats in current versions of HBase are currently exposed as avg, min and max. > This gives a skewed view of performance as the outliers are usually the > indicators of problems. Please revise the stats collection framework to use > true buckets and expose the p50, p75 and p95 values of these buckets through > JMX. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5883) Backup master is going down due to connection refused exception
[
https://issues.apache.org/jira/browse/HBASE-5883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-5883:
--
Resolution: Fixed
Fix Version/s: (was: 0.94.2)
0.94.1
Status: Resolved (was: Patch Available)
I'm resolving this. I believe Greg tested this and the addendum version and
found that the pre-addendum version fixed the problem more effectively than
afterwards.
The patch has been committed already on 0.94.1 (it is in the 0.94.1rc0) an in
the other branches.
Please file a new issue if to address the addendum.
> Backup master is going down due to connection refused exception
> ---
>
> Key: HBASE-5883
> URL: https://issues.apache.org/jira/browse/HBASE-5883
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.90.6, 0.92.1, 0.94.0
>Reporter: Gopinathan A
>Assignee: Jieshan Bean
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: 90-addendum.patch, 92-addendum.patch, 94-addendum.patch,
> HBASE-5883-90.patch, HBASE-5883-92.patch, HBASE-5883-94.patch,
> HBASE-5883-trunk.patch, trunk-addendum.patch
>
>
> The active master node network was down for some time (This node contains
> Master,DN,ZK,RS). Here backup node got
> notification, and started to became active. Immedietly backup node got
> aborted with the below exception.
> {noformat}
> 2012-04-09 10:42:24,270 INFO org.apache.hadoop.hbase.master.SplitLogManager:
> finished splitting (more than or equal to) 861248320 bytes in 4 log files in
> [hdfs://192.168.47.205:9000/hbase/.logs/HOST-192-168-47-202,60020,1333715537172-splitting]
> in 26374ms
> 2012-04-09 10:42:24,316 FATAL org.apache.hadoop.hbase.master.HMaster: Master
> server abort: loaded coprocessors are: []
> 2012-04-09 10:42:24,333 FATAL org.apache.hadoop.hbase.master.HMaster:
> Unhandled exception. Starting shutdown.
> java.io.IOException: java.net.ConnectException: Connection refused
> at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:375)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1045)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:897)
> at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
> at $Proxy13.getProtocolVersion(Unknown Source)
> at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:183)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:303)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:280)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:332)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:236)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1276)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1233)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1220)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.getCachedConnection(CatalogTracker.java:569)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.getRootServerConnection(CatalogTracker.java:369)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRootServerConnection(CatalogTracker.java:353)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRootRegionLocation(CatalogTracker.java:660)
> at
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:616)
> at
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:540)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:363)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:488)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupConnection(HBaseClient.java:328)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:362)
> ... 20 more
> 2012-04-09 10:42:24,336 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
> 2012-04-09 10:42:24,336 DEBUG org.apache.hadoop.hbase.master.HMaster:
> Stopping service threads
> {noformat}
--
This m
[jira] [Commented] (HBASE-4050) Update HBase metrics framework to metrics2 framework
[ https://issues.apache.org/jira/browse/HBASE-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416628#comment-13416628 ] Elliott Clark commented on HBASE-4050: -- Doesn't matter to me too much > Update HBase metrics framework to metrics2 framework > > > Key: HBASE-4050 > URL: https://issues.apache.org/jira/browse/HBASE-4050 > Project: HBase > Issue Type: New Feature > Components: metrics >Affects Versions: 0.90.4 > Environment: Java 6 >Reporter: Eric Yang >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.96.0 > > Attachments: 4050-metrics-v2.patch, 4050-metrics-v3.patch, > HBASE-4050-0.patch, HBASE-4050-1.patch, HBASE-4050-2.patch, > HBASE-4050-3.patch, HBASE-4050-5.patch, HBASE-4050-6.patch, > HBASE-4050-7.patch, HBASE-4050-8.patch, HBASE-4050.patch > > > Metrics Framework has been marked deprecated in Hadoop 0.20.203+ and 0.22+, > and it might get removed in future Hadoop release. Hence, HBase needs to > revise the dependency of MetricsContext to use Metrics2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6404) Collect p50, p75 and p95 stats
Arjen Roodselaar created HBASE-6404: --- Summary: Collect p50, p75 and p95 stats Key: HBASE-6404 URL: https://issues.apache.org/jira/browse/HBASE-6404 Project: HBase Issue Type: Improvement Components: monitoring Reporter: Arjen Roodselaar Stats in current versions of HBase are currently exposed as avg, min and max. This gives a skewed view of performance as the outliers are usually the indicators of problems. Please revise the stats collection framework to use true buckets and expose the p50, p75 and p95 values of these buckets through JMX. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6379) [0.90 branch] Backport HBASE-6334 to 0.90
[ https://issues.apache.org/jira/browse/HBASE-6379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-6379: -- Tags: test Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Greg! Committed for 0.90.7 > [0.90 branch] Backport HBASE-6334 to 0.90 > - > > Key: HBASE-6379 > URL: https://issues.apache.org/jira/browse/HBASE-6379 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 0.90.7 > > Attachments: HBASE-6379.patch > > > See HBASE-6334 for details. > The issue is that HBASE-6334 detects both HBASE-4195 (which should be > backported to 0.90 -- I'll file another JIRA for that) and HBASE-2856 (which > is a known issue in 0.90 that won't be fixed because it requires a change to > the HFile format). So in 0.90, we need a way to only catch HBASE-4195 > failures and ignore HBASE-2856 failures. > Luckily, HBASE-4195 only occurs *within* a column family, while HBASE-2856 > occurs *between* column families, so we just need to add a little to the > backport to differentiate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6379) [0.90 branch] Backport HBASE-6334 to 0.90
[ https://issues.apache.org/jira/browse/HBASE-6379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416621#comment-13416621 ] Jonathan Hsieh commented on HBASE-6379: --- Thanks for the test greg. +1, lgtm. I found a few nits in spacing/spelling and will fix on commit. > [0.90 branch] Backport HBASE-6334 to 0.90 > - > > Key: HBASE-6379 > URL: https://issues.apache.org/jira/browse/HBASE-6379 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 0.90.7 > > Attachments: HBASE-6379.patch > > > See HBASE-6334 for details. > The issue is that HBASE-6334 detects both HBASE-4195 (which should be > backported to 0.90 -- I'll file another JIRA for that) and HBASE-2856 (which > is a known issue in 0.90 that won't be fixed because it requires a change to > the HFile format). So in 0.90, we need a way to only catch HBASE-4195 > failures and ignore HBASE-2856 failures. > Luckily, HBASE-4195 only occurs *within* a column family, while HBASE-2856 > occurs *between* column families, so we just need to add a little to the > backport to differentiate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6385) [0.90 branch] Backport HBASE-4195 to 0.90
[ https://issues.apache.org/jira/browse/HBASE-6385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-6385: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Greg! I've committed for 0.90.7 > [0.90 branch] Backport HBASE-4195 to 0.90 > - > > Key: HBASE-6385 > URL: https://issues.apache.org/jira/browse/HBASE-6385 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 0.90.7 > > Attachments: HBASE-6385.patch > > > We are seeing some HBASE-4195 failures on 0.90. Backport this fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4050) Update HBase metrics framework to metrics2 framework
[ https://issues.apache.org/jira/browse/HBASE-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416608#comment-13416608 ] Zhihong Ted Yu commented on HBASE-4050: --- The above is a nice list. The title of this JIRA is general sounding. Does it make sense to create the above sub-tasks (including Elliot's latest patch) under this JIRA ? > Update HBase metrics framework to metrics2 framework > > > Key: HBASE-4050 > URL: https://issues.apache.org/jira/browse/HBASE-4050 > Project: HBase > Issue Type: New Feature > Components: metrics >Affects Versions: 0.90.4 > Environment: Java 6 >Reporter: Eric Yang >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.96.0 > > Attachments: 4050-metrics-v2.patch, 4050-metrics-v3.patch, > HBASE-4050-0.patch, HBASE-4050-1.patch, HBASE-4050-2.patch, > HBASE-4050-3.patch, HBASE-4050-5.patch, HBASE-4050-6.patch, > HBASE-4050-7.patch, HBASE-4050-8.patch, HBASE-4050.patch > > > Metrics Framework has been marked deprecated in Hadoop 0.20.203+ and 0.22+, > and it might get removed in future Hadoop release. Hence, HBase needs to > revise the dependency of MetricsContext to use Metrics2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4050) Update HBase metrics framework to metrics2 framework
[ https://issues.apache.org/jira/browse/HBASE-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416606#comment-13416606 ] Alex Baranau commented on HBASE-4050: - This works too. I was going to add "almost empty" placeholders for these points: bq. Move RegionServer Metrics to metrics2 bq. Move Master Metrics to metrics 2 With one metric in each. So that actual moving of metrics can be done separately by placing them in these classes, as was discussed in the first part of comments for the issue. Can do that as a part of other issue (I'd say the code + some unit-test is there, in that early patch) > Update HBase metrics framework to metrics2 framework > > > Key: HBASE-4050 > URL: https://issues.apache.org/jira/browse/HBASE-4050 > Project: HBase > Issue Type: New Feature > Components: metrics >Affects Versions: 0.90.4 > Environment: Java 6 >Reporter: Eric Yang >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.96.0 > > Attachments: 4050-metrics-v2.patch, 4050-metrics-v3.patch, > HBASE-4050-0.patch, HBASE-4050-1.patch, HBASE-4050-2.patch, > HBASE-4050-3.patch, HBASE-4050-5.patch, HBASE-4050-6.patch, > HBASE-4050-7.patch, HBASE-4050-8.patch, HBASE-4050.patch > > > Metrics Framework has been marked deprecated in Hadoop 0.20.203+ and 0.22+, > and it might get removed in future Hadoop release. Hence, HBase needs to > revise the dependency of MetricsContext to use Metrics2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6379) [0.90 branch] Backport HBASE-6334 to 0.90
[ https://issues.apache.org/jira/browse/HBASE-6379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-6379: -- Issue Type: Improvement (was: Bug) > [0.90 branch] Backport HBASE-6334 to 0.90 > - > > Key: HBASE-6379 > URL: https://issues.apache.org/jira/browse/HBASE-6379 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 0.90.7 > > Attachments: HBASE-6379.patch > > > See HBASE-6334 for details. > The issue is that HBASE-6334 detects both HBASE-4195 (which should be > backported to 0.90 -- I'll file another JIRA for that) and HBASE-2856 (which > is a known issue in 0.90 that won't be fixed because it requires a change to > the HFile format). So in 0.90, we need a way to only catch HBASE-4195 > failures and ignore HBASE-2856 failures. > Luckily, HBASE-4195 only occurs *within* a column family, while HBASE-2856 > occurs *between* column families, so we just need to add a little to the > backport to differentiate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6385) [0.90 branch] Backport HBASE-4195 to 0.90
[ https://issues.apache.org/jira/browse/HBASE-6385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-6385: -- Issue Type: Bug (was: Task) > [0.90 branch] Backport HBASE-4195 to 0.90 > - > > Key: HBASE-6385 > URL: https://issues.apache.org/jira/browse/HBASE-6385 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 0.90.7 > > Attachments: HBASE-6385.patch > > > We are seeing some HBASE-4195 failures on 0.90. Backport this fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6379) [0.90 branch] Backport HBASE-6334 to 0.90
[ https://issues.apache.org/jira/browse/HBASE-6379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-6379: -- Issue Type: Bug (was: Task) > [0.90 branch] Backport HBASE-6334 to 0.90 > - > > Key: HBASE-6379 > URL: https://issues.apache.org/jira/browse/HBASE-6379 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 0.90.7 > > Attachments: HBASE-6379.patch > > > See HBASE-6334 for details. > The issue is that HBASE-6334 detects both HBASE-4195 (which should be > backported to 0.90 -- I'll file another JIRA for that) and HBASE-2856 (which > is a known issue in 0.90 that won't be fixed because it requires a change to > the HFile format). So in 0.90, we need a way to only catch HBASE-4195 > failures and ignore HBASE-2856 failures. > Luckily, HBASE-4195 only occurs *within* a column family, while HBASE-2856 > occurs *between* column families, so we just need to add a little to the > backport to differentiate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4050) Update HBase metrics framework to metrics2 framework
[ https://issues.apache.org/jira/browse/HBASE-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416602#comment-13416602 ] Elliott Clark commented on HBASE-4050: -- Lets do those in another issue. I'm planning on creating an umbrella jira for the move to metrics2. This would probably also be where most of the metrics clean up work, that lars was asking for, would be listed under. So Far I have: * Investigate moving to DI (guice) framework for plugin arch. * Naming and documenting of the hadoop-metrics2.properties file * Create histogram class for metrics 2 * Move RegionServer Metrics to metrics2 * Move Master Metrics to metrics 2 * Move external servers to metrics2 (thrift,thrift2,rest) * Investigate having hbase-env.sh decide which hadoop-compat to include > Update HBase metrics framework to metrics2 framework > > > Key: HBASE-4050 > URL: https://issues.apache.org/jira/browse/HBASE-4050 > Project: HBase > Issue Type: New Feature > Components: metrics >Affects Versions: 0.90.4 > Environment: Java 6 >Reporter: Eric Yang >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.96.0 > > Attachments: 4050-metrics-v2.patch, 4050-metrics-v3.patch, > HBASE-4050-0.patch, HBASE-4050-1.patch, HBASE-4050-2.patch, > HBASE-4050-3.patch, HBASE-4050-5.patch, HBASE-4050-6.patch, > HBASE-4050-7.patch, HBASE-4050-8.patch, HBASE-4050.patch > > > Metrics Framework has been marked deprecated in Hadoop 0.20.203+ and 0.22+, > and it might get removed in future Hadoop release. Hence, HBase needs to > revise the dependency of MetricsContext to use Metrics2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4050) Update HBase metrics framework to metrics2 framework
[ https://issues.apache.org/jira/browse/HBASE-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416601#comment-13416601 ] Alex Baranau commented on HBASE-4050: - Let me try to do that in coming hour or two. If not - you can proceed without it. I guess. > Update HBase metrics framework to metrics2 framework > > > Key: HBASE-4050 > URL: https://issues.apache.org/jira/browse/HBASE-4050 > Project: HBase > Issue Type: New Feature > Components: metrics >Affects Versions: 0.90.4 > Environment: Java 6 >Reporter: Eric Yang >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.96.0 > > Attachments: 4050-metrics-v2.patch, 4050-metrics-v3.patch, > HBASE-4050-0.patch, HBASE-4050-1.patch, HBASE-4050-2.patch, > HBASE-4050-3.patch, HBASE-4050-5.patch, HBASE-4050-6.patch, > HBASE-4050-7.patch, HBASE-4050-8.patch, HBASE-4050.patch > > > Metrics Framework has been marked deprecated in Hadoop 0.20.203+ and 0.22+, > and it might get removed in future Hadoop release. Hence, HBase needs to > revise the dependency of MetricsContext to use Metrics2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6383) Investigate using 2Q for block cache
[ https://issues.apache.org/jira/browse/HBASE-6383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416597#comment-13416597 ] Jean-Daniel Cryans commented on HBASE-6383: --- bq. I don't know how much I would expect that - in theory under 'regular' workload (depending on the number of scans) I would expect the times to be increasingly separate as the queue becomes more efficient (scan resistant) compared to the regular block cache, which would have to go to disk. I meant the read time for a single access to the block cache is in the nanoseconds, the total runtime for the tests doesn't make much sense since it's mostly skewed by the time it takes to generate a block (or read from disk) in order to cache it. bq. Sounds like its not worth it overall, but maybe for some workloads? I'd like to do some testing on our cluster too, but probably don't have time for a bit. Sorry I mislead you, there is a difference and the best way to measure it is the hit ratio (unless it takes too long to read from the cache, to a point where it would affect latency). In some cases a saw a difference of a few percent which in the caching domain is a big deal. > Investigate using 2Q for block cache > > > Key: HBASE-6383 > URL: https://issues.apache.org/jira/browse/HBASE-6383 > Project: HBase > Issue Type: New Feature > Components: performance, regionserver >Affects Versions: 0.96.0 >Reporter: Jesse Yates >Priority: Minor > > Currently we use a basic version of LRU to handle block caching. LRU is know > to be very susceptible to scan thrashing (not scan resistant), which is a > common operation in HBase. 2Q is an efficient caching algorithm that emulates > the effectivness of LRU/2 (eviction based not on the last access, but rather > the access before the last), but is O(1), rather than O(lg\(n)) in complexity. > JD has long been talking about investigating 2Q as it may be far better for > HBase than LRU and has been shown to be incredibly useful for traditional > database caching on production systems. > One would need to implement 2Q (though the pseudocode in the paper is quite > explicit) and then test against the existing cache implementation. > The link to the original paper is here: www.vldb.org/conf/1994/P439.PDF > A short overview of 2Q: > 2Q uses two queues (hence the name) and a list of pointers to keep track of > cached blocks. The first queue is for new, hot items (Ain). If an item is > accessed that isn't in Ain, the coldest block is evicted from Ain and the new > item replaces it. Anything accessed in Ain is already stored in memory and > kept in Ain. > When a block is evicted from Ain, it is moved to Aout _as a pointer_. If Aout > is full, the oldest element is evicted and replaced with the new pointer. > The key to 2Q comes in that when you access something in Aout, it is reloaded > into memory and stored in queue B. If B becomes full, then the coldest block > is evicted. > This essentially makes Aout a filter for long-term hot items, based on the > size of Aout. The original authors found that while you can tune Aout, it > generally performs very well at at "50% of the number of pages as would fit > into the buffer", but can be tuned as low as 5% at only a slight cost to > responsiveness to changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4050) Update HBase metrics framework to metrics2 framework
[ https://issues.apache.org/jira/browse/HBASE-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416599#comment-13416599 ] Zhihong Ted Yu commented on HBASE-4050: --- How long would it take to align them with the work from Elliot ? > Update HBase metrics framework to metrics2 framework > > > Key: HBASE-4050 > URL: https://issues.apache.org/jira/browse/HBASE-4050 > Project: HBase > Issue Type: New Feature > Components: metrics >Affects Versions: 0.90.4 > Environment: Java 6 >Reporter: Eric Yang >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.96.0 > > Attachments: 4050-metrics-v2.patch, 4050-metrics-v3.patch, > HBASE-4050-0.patch, HBASE-4050-1.patch, HBASE-4050-2.patch, > HBASE-4050-3.patch, HBASE-4050-5.patch, HBASE-4050-6.patch, > HBASE-4050-7.patch, HBASE-4050-8.patch, HBASE-4050.patch > > > Metrics Framework has been marked deprecated in Hadoop 0.20.203+ and 0.22+, > and it might get removed in future Hadoop release. Hence, HBase needs to > revise the dependency of MetricsContext to use Metrics2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6382) Upgrade Jersey to 1.8 to match Hadoop 1 and 2
[ https://issues.apache.org/jira/browse/HBASE-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-6382: -- Affects Version/s: 0.90.7 Fix Version/s: 0.90.7 Included in 0.90.7 as well. Thanks all! > Upgrade Jersey to 1.8 to match Hadoop 1 and 2 > - > > Key: HBASE-6382 > URL: https://issues.apache.org/jira/browse/HBASE-6382 > Project: HBase > Issue Type: Improvement > Components: rest >Affects Versions: 0.90.7, 0.92.2, 0.96.0, 0.94.2 >Reporter: David S. Wang >Assignee: David S. Wang > Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1 > > Attachments: HBASE-6382-trunk.patch > > > Upgrade Jersey dependency from 1.4 to 1.8 to match Hadoop dependencies. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4050) Update HBase metrics framework to metrics2 framework
[ https://issues.apache.org/jira/browse/HBASE-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416592#comment-13416592 ] Alex Baranau commented on HBASE-4050: - Should I add metric sources for RegionServer and Master in a patch? (those from 4050-metrics-v3.patch) > Update HBase metrics framework to metrics2 framework > > > Key: HBASE-4050 > URL: https://issues.apache.org/jira/browse/HBASE-4050 > Project: HBase > Issue Type: New Feature > Components: metrics >Affects Versions: 0.90.4 > Environment: Java 6 >Reporter: Eric Yang >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.96.0 > > Attachments: 4050-metrics-v2.patch, 4050-metrics-v3.patch, > HBASE-4050-0.patch, HBASE-4050-1.patch, HBASE-4050-2.patch, > HBASE-4050-3.patch, HBASE-4050-5.patch, HBASE-4050-6.patch, > HBASE-4050-7.patch, HBASE-4050-8.patch, HBASE-4050.patch > > > Metrics Framework has been marked deprecated in Hadoop 0.20.203+ and 0.22+, > and it might get removed in future Hadoop release. Hence, HBase needs to > revise the dependency of MetricsContext to use Metrics2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6383) Investigate using 2Q for block cache
[ https://issues.apache.org/jira/browse/HBASE-6383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416584#comment-13416584 ] Jesse Yates commented on HBASE-6383: bq. the difference is in nanoseconds (as you would expect from reading from memory). I don't know how much I would expect that - in theory under 'regular' workload (depending on the number of scans) I would expect the times to be increasingly separate as the queue becomes more efficient (scan resistant) compared to the regular block cache, which would have to go to disk. Sounds like its not worth it overall, but maybe for some workloads? I'd like to do some testing on our cluster too, but probably don't have time for a bit. > Investigate using 2Q for block cache > > > Key: HBASE-6383 > URL: https://issues.apache.org/jira/browse/HBASE-6383 > Project: HBase > Issue Type: New Feature > Components: performance, regionserver >Affects Versions: 0.96.0 >Reporter: Jesse Yates >Priority: Minor > > Currently we use a basic version of LRU to handle block caching. LRU is know > to be very susceptible to scan thrashing (not scan resistant), which is a > common operation in HBase. 2Q is an efficient caching algorithm that emulates > the effectivness of LRU/2 (eviction based not on the last access, but rather > the access before the last), but is O(1), rather than O(lg\(n)) in complexity. > JD has long been talking about investigating 2Q as it may be far better for > HBase than LRU and has been shown to be incredibly useful for traditional > database caching on production systems. > One would need to implement 2Q (though the pseudocode in the paper is quite > explicit) and then test against the existing cache implementation. > The link to the original paper is here: www.vldb.org/conf/1994/P439.PDF > A short overview of 2Q: > 2Q uses two queues (hence the name) and a list of pointers to keep track of > cached blocks. The first queue is for new, hot items (Ain). If an item is > accessed that isn't in Ain, the coldest block is evicted from Ain and the new > item replaces it. Anything accessed in Ain is already stored in memory and > kept in Ain. > When a block is evicted from Ain, it is moved to Aout _as a pointer_. If Aout > is full, the oldest element is evicted and replaced with the new pointer. > The key to 2Q comes in that when you access something in Aout, it is reloaded > into memory and stored in queue B. If B becomes full, then the coldest block > is evicted. > This essentially makes Aout a filter for long-term hot items, based on the > size of Aout. The original authors found that while you can tune Aout, it > generally performs very well at at "50% of the number of pages as would fit > into the buffer", but can be tuned as low as 5% at only a slight cost to > responsiveness to changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6383) Investigate using 2Q for block cache
[ https://issues.apache.org/jira/browse/HBASE-6383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416556#comment-13416556 ] Jean-Daniel Cryans commented on HBASE-6383: --- bq. Which it? 2Q shouldn't be double caching (and from the quick read through of your impl its not). Ah sorry I forgot that I'm using Todd's HBASE-5898 and it happens in that case because not all the reads are protected with the lock. In the current code base it's fine. bq. Do you have any stats as to how much better it performs, e.g. mean, percentile, etc read times? I do, except that I have a ton of raw data to analyze (it's for my Masters). FWIW the timings are about the same, the difference is in nanoseconds (as you would expect from reading from memory). > Investigate using 2Q for block cache > > > Key: HBASE-6383 > URL: https://issues.apache.org/jira/browse/HBASE-6383 > Project: HBase > Issue Type: New Feature > Components: performance, regionserver >Affects Versions: 0.96.0 >Reporter: Jesse Yates >Priority: Minor > > Currently we use a basic version of LRU to handle block caching. LRU is know > to be very susceptible to scan thrashing (not scan resistant), which is a > common operation in HBase. 2Q is an efficient caching algorithm that emulates > the effectivness of LRU/2 (eviction based not on the last access, but rather > the access before the last), but is O(1), rather than O(lg\(n)) in complexity. > JD has long been talking about investigating 2Q as it may be far better for > HBase than LRU and has been shown to be incredibly useful for traditional > database caching on production systems. > One would need to implement 2Q (though the pseudocode in the paper is quite > explicit) and then test against the existing cache implementation. > The link to the original paper is here: www.vldb.org/conf/1994/P439.PDF > A short overview of 2Q: > 2Q uses two queues (hence the name) and a list of pointers to keep track of > cached blocks. The first queue is for new, hot items (Ain). If an item is > accessed that isn't in Ain, the coldest block is evicted from Ain and the new > item replaces it. Anything accessed in Ain is already stored in memory and > kept in Ain. > When a block is evicted from Ain, it is moved to Aout _as a pointer_. If Aout > is full, the oldest element is evicted and replaced with the new pointer. > The key to 2Q comes in that when you access something in Aout, it is reloaded > into memory and stored in queue B. If B becomes full, then the coldest block > is evicted. > This essentially makes Aout a filter for long-term hot items, based on the > size of Aout. The original authors found that while you can tune Aout, it > generally performs very well at at "50% of the number of pages as would fit > into the buffer", but can be tuned as low as 5% at only a slight cost to > responsiveness to changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6400) Add getMasterAdmin() and getMasterMonitor() to HConnection
[ https://issues.apache.org/jira/browse/HBASE-6400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416553#comment-13416553 ] Hadoop QA commented on HBASE-6400: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536866/6400-v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2394//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2394//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2394//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2394//console This message is automatically generated. > Add getMasterAdmin() and getMasterMonitor() to HConnection > -- > > Key: HBASE-6400 > URL: https://issues.apache.org/jira/browse/HBASE-6400 > Project: HBase > Issue Type: Improvement >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 0.96.0 > > Attachments: 6400-v2.patch, HBASE-6400_v1.patch > > > HConnection used to have getMaster() which returns HMasterInterface, but > after HBASE-6039 it has been removed. I think we need to expose > HConnection.getMasterAdmin() and getMasterMonitor() a la > HConnection.getAdmin(), and getClient(). > HConnectionImplementation has getKeepAliveMasterAdmin() but, I see no reason > to leak keep alive classes to upper layers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4050) Update HBase metrics framework to metrics2 framework
[ https://issues.apache.org/jira/browse/HBASE-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416543#comment-13416543 ] Zhihong Ted Yu commented on HBASE-4050: --- Would integrate the patch later this afternoon if there is no further review comment. > Update HBase metrics framework to metrics2 framework > > > Key: HBASE-4050 > URL: https://issues.apache.org/jira/browse/HBASE-4050 > Project: HBase > Issue Type: New Feature > Components: metrics >Affects Versions: 0.90.4 > Environment: Java 6 >Reporter: Eric Yang >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.96.0 > > Attachments: 4050-metrics-v2.patch, 4050-metrics-v3.patch, > HBASE-4050-0.patch, HBASE-4050-1.patch, HBASE-4050-2.patch, > HBASE-4050-3.patch, HBASE-4050-5.patch, HBASE-4050-6.patch, > HBASE-4050-7.patch, HBASE-4050-8.patch, HBASE-4050.patch > > > Metrics Framework has been marked deprecated in Hadoop 0.20.203+ and 0.22+, > and it might get removed in future Hadoop release. Hence, HBase needs to > revise the dependency of MetricsContext to use Metrics2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6382) Upgrade Jersey to 1.8 to match Hadoop 1 and 2
[ https://issues.apache.org/jira/browse/HBASE-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416512#comment-13416512 ] David S. Wang commented on HBASE-6382: -- @Jon: I confirmed that Hadoop 1.0 is using Jersey 1.8. See: http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/ivy/libraries.properties?view=log and search for jersey. > Upgrade Jersey to 1.8 to match Hadoop 1 and 2 > - > > Key: HBASE-6382 > URL: https://issues.apache.org/jira/browse/HBASE-6382 > Project: HBase > Issue Type: Improvement > Components: rest >Affects Versions: 0.92.2, 0.96.0, 0.94.2 >Reporter: David S. Wang >Assignee: David S. Wang > Fix For: 0.92.2, 0.96.0, 0.94.1 > > Attachments: HBASE-6382-trunk.patch > > > Upgrade Jersey dependency from 1.4 to 1.8 to match Hadoop dependencies. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6400) Add getMasterAdmin() and getMasterMonitor() to HConnection
[ https://issues.apache.org/jira/browse/HBASE-6400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu updated HBASE-6400: -- Attachment: 6400-v2.patch Patch v2 adds @Override. > Add getMasterAdmin() and getMasterMonitor() to HConnection > -- > > Key: HBASE-6400 > URL: https://issues.apache.org/jira/browse/HBASE-6400 > Project: HBase > Issue Type: Improvement >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 0.96.0 > > Attachments: 6400-v2.patch, HBASE-6400_v1.patch > > > HConnection used to have getMaster() which returns HMasterInterface, but > after HBASE-6039 it has been removed. I think we need to expose > HConnection.getMasterAdmin() and getMasterMonitor() a la > HConnection.getAdmin(), and getClient(). > HConnectionImplementation has getKeepAliveMasterAdmin() but, I see no reason > to leak keep alive classes to upper layers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6400) Add getMasterAdmin() and getMasterMonitor() to HConnection
[ https://issues.apache.org/jira/browse/HBASE-6400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu updated HBASE-6400: -- Hadoop Flags: Reviewed Status: Patch Available (was: Open) > Add getMasterAdmin() and getMasterMonitor() to HConnection > -- > > Key: HBASE-6400 > URL: https://issues.apache.org/jira/browse/HBASE-6400 > Project: HBase > Issue Type: Improvement >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 0.96.0 > > Attachments: 6400-v2.patch, HBASE-6400_v1.patch > > > HConnection used to have getMaster() which returns HMasterInterface, but > after HBASE-6039 it has been removed. I think we need to expose > HConnection.getMasterAdmin() and getMasterMonitor() a la > HConnection.getAdmin(), and getClient(). > HConnectionImplementation has getKeepAliveMasterAdmin() but, I see no reason > to leak keep alive classes to upper layers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6383) Investigate using 2Q for block cache
[
https://issues.apache.org/jira/browse/HBASE-6383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416444#comment-13416444
]
Jesse Yates commented on HBASE-6383:
{quote}
It's main deficiency is that it can double cache a block (just like the
DoubleBlockCache).
{quote}
Which it? 2Q shouldn't be double caching (and from the quick read through of
your impl its not).
Do you have any stats as to how much better it performs, e.g. mean, percentile,
etc read times?
good stuff J.D.
> Investigate using 2Q for block cache
>
>
> Key: HBASE-6383
> URL: https://issues.apache.org/jira/browse/HBASE-6383
> Project: HBase
> Issue Type: New Feature
> Components: performance, regionserver
>Affects Versions: 0.96.0
>Reporter: Jesse Yates
>Priority: Minor
>
> Currently we use a basic version of LRU to handle block caching. LRU is know
> to be very susceptible to scan thrashing (not scan resistant), which is a
> common operation in HBase. 2Q is an efficient caching algorithm that emulates
> the effectivness of LRU/2 (eviction based not on the last access, but rather
> the access before the last), but is O(1), rather than O(lg\(n)) in complexity.
> JD has long been talking about investigating 2Q as it may be far better for
> HBase than LRU and has been shown to be incredibly useful for traditional
> database caching on production systems.
> One would need to implement 2Q (though the pseudocode in the paper is quite
> explicit) and then test against the existing cache implementation.
> The link to the original paper is here: www.vldb.org/conf/1994/P439.PDF
> A short overview of 2Q:
> 2Q uses two queues (hence the name) and a list of pointers to keep track of
> cached blocks. The first queue is for new, hot items (Ain). If an item is
> accessed that isn't in Ain, the coldest block is evicted from Ain and the new
> item replaces it. Anything accessed in Ain is already stored in memory and
> kept in Ain.
> When a block is evicted from Ain, it is moved to Aout _as a pointer_. If Aout
> is full, the oldest element is evicted and replaced with the new pointer.
> The key to 2Q comes in that when you access something in Aout, it is reloaded
> into memory and stored in queue B. If B becomes full, then the coldest block
> is evicted.
> This essentially makes Aout a filter for long-term hot items, based on the
> size of Aout. The original authors found that while you can tune Aout, it
> generally performs very well at at "50% of the number of pages as would fit
> into the buffer", but can be tuned as low as 5% at only a slight cost to
> responsiveness to changes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6400) Add getMasterAdmin() and getMasterMonitor() to HConnection
[ https://issues.apache.org/jira/browse/HBASE-6400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu updated HBASE-6400: -- Description: HConnection used to have getMaster() which returns HMasterInterface, but after HBASE-6039 it has been removed. I think we need to expose HConnection.getMasterAdmin() and getMasterMonitor() a la HConnection.getAdmin(), and getClient(). HConnectionImplementation has getKeepAliveMasterAdmin() but, I see no reason to leak keep alive classes to upper layers. was: HConnection used to have getMasterInterface(), but after HBASE-6039 it has been removed. I think we need to expose HConnection.getMasterAdmin() and getMasterMonitor() a la HConnection.getAdmin(), and getClient(). HConnectionImplementation has getKeepAliveMasterAdmin() but, I see no reason to leak keep alive classes to upper layers. Fix Version/s: 0.96.0 > Add getMasterAdmin() and getMasterMonitor() to HConnection > -- > > Key: HBASE-6400 > URL: https://issues.apache.org/jira/browse/HBASE-6400 > Project: HBase > Issue Type: Improvement >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 0.96.0 > > Attachments: HBASE-6400_v1.patch > > > HConnection used to have getMaster() which returns HMasterInterface, but > after HBASE-6039 it has been removed. I think we need to expose > HConnection.getMasterAdmin() and getMasterMonitor() a la > HConnection.getAdmin(), and getClient(). > HConnectionImplementation has getKeepAliveMasterAdmin() but, I see no reason > to leak keep alive classes to upper layers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6403) RegionCoprocessorHost provides empty config when loading a coprocessor
[ https://issues.apache.org/jira/browse/HBASE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416421#comment-13416421 ] Zhihong Ted Yu commented on HBASE-6403: --- Nice finding, Eric. Looks like HBaseConfiguration.create() should be aware of the type of Configuration object passed to it. Meaning, CompoundConfiguration should be pulled into hbase-common module. > RegionCoprocessorHost provides empty config when loading a coprocessor > -- > > Key: HBASE-6403 > URL: https://issues.apache.org/jira/browse/HBASE-6403 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Eric Newton >Priority: Minor > > I started playing with Giraffa. I am running it against Hadoop 2.0.0-alpha, > and current HBase trunk. On line 159 of RegionCoprocessorHost, the server's > configuration is copied... or at least an attempt is made to copy it. > However, the server's configuration object, a CompoundConfiguration, does not > store the data in the same way as the base Configuration object, and so > nothing is copied. This leaves the coprocessor without access to > configuration values, like the fs.defaultFS, which Giraffa is looking for. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-5449) Support for wire-compatible security functionality
[ https://issues.apache.org/jira/browse/HBASE-5449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates reassigned HBASE-5449: -- Assignee: (was: Jesse Yates) Don't have time to work on this at the moment - someone else can pick it up. > Support for wire-compatible security functionality > -- > > Key: HBASE-5449 > URL: https://issues.apache.org/jira/browse/HBASE-5449 > Project: HBase > Issue Type: Sub-task > Components: ipc, master, migration, regionserver >Reporter: Todd Lipcon > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5449) Support for wire-compatible security functionality
[ https://issues.apache.org/jira/browse/HBASE-5449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416416#comment-13416416 ] Gary Helmling commented on HBASE-5449: -- I see two parts of the current security controls that would need better cross version support: # AccessControllerProtocol implementation for handling grant(), revoke(), etc. calls as a coprocessor endpoint # ACL serialization into ZooKeeper znodes for synchronization across the cluster -- I think these are still serialized just as Writables I'm tackling the AccessControllerProtocol conversion as part of HBASE-5448 (convert coprocessor endpoints to PB-based RPC). It's making a handy proof-of-concept/test case. I believe the znode serialization still needs to be handled, unless that's been picked up elsewhere. > Support for wire-compatible security functionality > -- > > Key: HBASE-5449 > URL: https://issues.apache.org/jira/browse/HBASE-5449 > Project: HBase > Issue Type: Sub-task > Components: ipc, master, migration, regionserver >Reporter: Todd Lipcon >Assignee: Jesse Yates > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6403) RegionCoprocessorHost provides empty config when loading a coprocessor
Eric Newton created HBASE-6403: -- Summary: RegionCoprocessorHost provides empty config when loading a coprocessor Key: HBASE-6403 URL: https://issues.apache.org/jira/browse/HBASE-6403 Project: HBase Issue Type: Bug Components: regionserver Reporter: Eric Newton Priority: Minor I started playing with Giraffa. I am running it against Hadoop 2.0.0-alpha, and current HBase trunk. On line 159 of RegionCoprocessorHost, the server's configuration is copied... or at least an attempt is made to copy it. However, the server's configuration object, a CompoundConfiguration, does not store the data in the same way as the base Configuration object, and so nothing is copied. This leaves the coprocessor without access to configuration values, like the fs.defaultFS, which Giraffa is looking for. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6400) Add getMasterAdmin() and getMasterMonitor() to HConnection
[ https://issues.apache.org/jira/browse/HBASE-6400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416414#comment-13416414 ] Gregory Chanan commented on HBASE-6400: --- I think you meant getMaster() not getMasterInterface() in your description. Otherwise, looks good with Ted's comment. > Add getMasterAdmin() and getMasterMonitor() to HConnection > -- > > Key: HBASE-6400 > URL: https://issues.apache.org/jira/browse/HBASE-6400 > Project: HBase > Issue Type: Improvement >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Attachments: HBASE-6400_v1.patch > > > HConnection used to have getMasterInterface(), but after HBASE-6039 it has > been removed. I think we need to expose HConnection.getMasterAdmin() and > getMasterMonitor() a la HConnection.getAdmin(), and getClient(). > HConnectionImplementation has getKeepAliveMasterAdmin() but, I see no reason > to leak keep alive classes to upper layers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6400) Add getMasterAdmin() and getMasterMonitor() to HConnection
[
https://issues.apache.org/jira/browse/HBASE-6400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416358#comment-13416358
]
Zhihong Ted Yu commented on HBASE-6400:
---
{code}
+public MasterAdminProtocol getMasterAdmin() throws
MasterNotRunningException {
{code}
@Override seems to be missing.
> Add getMasterAdmin() and getMasterMonitor() to HConnection
> --
>
> Key: HBASE-6400
> URL: https://issues.apache.org/jira/browse/HBASE-6400
> Project: HBase
> Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: HBASE-6400_v1.patch
>
>
> HConnection used to have getMasterInterface(), but after HBASE-6039 it has
> been removed. I think we need to expose HConnection.getMasterAdmin() and
> getMasterMonitor() a la HConnection.getAdmin(), and getClient().
> HConnectionImplementation has getKeepAliveMasterAdmin() but, I see no reason
> to leak keep alive classes to upper layers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6399) MetricsContext be different between RegionServerMetrics and RegionServerDynamicMetrics
[ https://issues.apache.org/jira/browse/HBASE-6399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416350#comment-13416350 ] Elliott Clark commented on HBASE-6399: -- I agree giving users the choice is the correct thing to do. However i'm pretty sure there is already a way that these metrics can be turned off. Adding a different context would break users who expect dynamic metrics to be in a certain location. So I don't think this should go into 0.94.X as we shouldn't break people in a point release. In addition I think we would need a better name for the context. "dynamic" does not give enough context to what this will hold. "hbase-dynamic" or something like that seems better. > MetricsContext be different between RegionServerMetrics and > RegionServerDynamicMetrics > -- > > Key: HBASE-6399 > URL: https://issues.apache.org/jira/browse/HBASE-6399 > Project: HBase > Issue Type: Bug > Components: metrics >Affects Versions: 0.94.0 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.96.0, 0.94.2 > > Attachments: HBASE-6399.patch > > > In hadoop-metrics.properties, GangliaContext is optional metrics context, I > think we will use ganglia to monitor hbase cluster generally. > However, I find a serious problem: > RegionServerDynamicMetrics will generate lots of rrd file because we would > move region or create/delete table. > Especially if table is created everyday in some applications, there are much > more and more rrd files in Gmetad Server. It will make Gmetad Server > corrupted. > IMO, MetricsContext should be different between RegionServerMetrics and > RegionServerDynamicMetrics -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6272) In-memory region state is inconsistent
[ https://issues.apache.org/jira/browse/HBASE-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416347#comment-13416347 ] Jimmy Xiang commented on HBASE-6272: Thanks a lot, Ram! > In-memory region state is inconsistent > -- > > Key: HBASE-6272 > URL: https://issues.apache.org/jira/browse/HBASE-6272 > Project: HBase > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > > AssignmentManger stores region state related information in several places: > regionsInTransition, regions (region info to server name map), and servers > (server name to region info set map). However the access to these places is > not coordinated properly. It leads to inconsistent in-memory region state > information. Sometimes, some region could even be offline, and not in > transition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6317) Master clean start up and Partially enabled tables make region assignment inconsistent.
[
https://issues.apache.org/jira/browse/HBASE-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416326#comment-13416326
]
rajeshbabu commented on HBASE-6317:
---
Patch was uploaded to RB: https://reviews.apache.org/r/6011/.
> Master clean start up and Partially enabled tables make region assignment
> inconsistent.
> ---
>
> Key: HBASE-6317
> URL: https://issues.apache.org/jira/browse/HBASE-6317
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: rajeshbabu
> Fix For: 0.92.2, 0.96.0, 0.94.2
>
> Attachments: HBASE-6317_94.patch, HBASE-6317_94_3.patch
>
>
> If we have a table in partially enabled state (ENABLING) then on HMaster
> restart we treat it as a clean cluster start up and do a bulk assign.
> Currently in 0.94 bulk assign will not handle ALREADY_OPENED scenarios and it
> leads to region assignment problems. Analysing more on this we found that we
> have better way to handle these scenarios.
> {code}
> if (false == checkIfRegionBelongsToDisabled(regionInfo)
> && false == checkIfRegionsBelongsToEnabling(regionInfo)) {
> synchronized (this.regions) {
> regions.put(regionInfo, regionLocation);
> addToServers(regionLocation, regionInfo);
> }
> {code}
> We dont add to regions map so that enable table handler can handle it. But
> as nothing is added to regions map we think it as a clean cluster start up.
> Will come up with a patch tomorrow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6402) Ability to explicitly include and exclude hosts from cluster
Philip Zeyliger created HBASE-6402: -- Summary: Ability to explicitly include and exclude hosts from cluster Key: HBASE-6402 URL: https://issues.apache.org/jira/browse/HBASE-6402 Project: HBase Issue Type: New Feature Components: master Affects Versions: 0.94.0 Reporter: Philip Zeyliger Both MR and HDFS (see http://hadoop.apache.org/common/docs/r0.20.0/cluster_setup.html; look for "dfs.hosts", "dfs.hosts.exclude", "mapred.hosts", "mapred.hosts.exclude") provide the user a way to deny certain slave daemons from joining the cluster. The use for this is two-fold: it prevents developers with a client configuration from joining the cluster from their laptop on accident, and it provides a mechanism to explicitly decommission a node while it's in repair or what-not. A similar explicit cluster membership would be useful for HBase, for the same reasons. Just yesterday a user found out that his organization was running regionservers on machines that weren't supposed to be running regionservers. It wasn't a huge deal, but this feature would make it easier for this user to prevent the co-admins from making this mistake. I'd note that HDFS and MR are inconsistent at the moment as to whether their exclude files should have IPs, hostnames, or both. Clearly defining and documenting that is useful. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6385) [0.90 branch] Backport HBASE-4195 to 0.90
[ https://issues.apache.org/jira/browse/HBASE-6385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416247#comment-13416247 ] Jonathan Hsieh commented on HBASE-6385: --- +1 pending tests. This brings 0.90's path in line with 0.92. > [0.90 branch] Backport HBASE-4195 to 0.90 > - > > Key: HBASE-6385 > URL: https://issues.apache.org/jira/browse/HBASE-6385 > Project: HBase > Issue Type: Task > Components: regionserver >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 0.90.7 > > Attachments: HBASE-6385.patch > > > We are seeing some HBASE-4195 failures on 0.90. Backport this fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5754) data lost with gora continuous ingest test (goraci)
[ https://issues.apache.org/jira/browse/HBASE-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416246#comment-13416246 ] nkeywal commented on HBASE-5754: @keith: thanks for the info. It will be interesting to see the results. fyi, I've just created HBASE-6401 on an hdfs linked issue. > data lost with gora continuous ingest test (goraci) > --- > > Key: HBASE-5754 > URL: https://issues.apache.org/jira/browse/HBASE-5754 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.1 > Environment: 10 node test cluster >Reporter: Eric Newton >Assignee: stack > > Keith Turner re-wrote the accumulo continuous ingest test using gora, which > has both hbase and accumulo back-ends. > I put a billion entries into HBase, and ran the Verify map/reduce job. The > verification failed because about 21K entries were missing. The goraci > [README|https://github.com/keith-turner/goraci] explains the test, and how it > detects missing data. > I re-ran the test with 100 million entries, and it verified successfully. > Both of the times I tested using a billion entries, the verification failed. > If I run the verification step twice, the results are consistent, so the > problem is > probably not on the verify step. > Here's the versions of the various packages: > ||package||version|| > |hadoop|0.20.205.0| > |hbase|0.92.1| > |gora|http://svn.apache.org/repos/asf/gora/trunk r1311277| > |goraci|https://github.com/ericnewton/goraci tagged 2012-04-08| > The change I made to goraci was to configure it for hbase and to allow it to > build properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5754) data lost with gora continuous ingest test (goraci)
[ https://issues.apache.org/jira/browse/HBASE-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416243#comment-13416243 ] Enis Soztutar commented on HBASE-5754: -- Just FYI in case, I've been working on adding "long running ingestion tests while randomly killing of servers", and other types of integration tests over at HBASE-6241, HBASE-6201. Feel free to chime in. > data lost with gora continuous ingest test (goraci) > --- > > Key: HBASE-5754 > URL: https://issues.apache.org/jira/browse/HBASE-5754 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.1 > Environment: 10 node test cluster >Reporter: Eric Newton >Assignee: stack > > Keith Turner re-wrote the accumulo continuous ingest test using gora, which > has both hbase and accumulo back-ends. > I put a billion entries into HBase, and ran the Verify map/reduce job. The > verification failed because about 21K entries were missing. The goraci > [README|https://github.com/keith-turner/goraci] explains the test, and how it > detects missing data. > I re-ran the test with 100 million entries, and it verified successfully. > Both of the times I tested using a billion entries, the verification failed. > If I run the verification step twice, the results are consistent, so the > problem is > probably not on the verify step. > Here's the versions of the various packages: > ||package||version|| > |hadoop|0.20.205.0| > |hbase|0.92.1| > |gora|http://svn.apache.org/repos/asf/gora/trunk r1311277| > |goraci|https://github.com/ericnewton/goraci tagged 2012-04-08| > The change I made to goraci was to configure it for hbase and to allow it to > build properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6401) HBase may lose edits after a crash if used with HDFS 1.0.3 or older
[ https://issues.apache.org/jira/browse/HBASE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416241#comment-13416241 ] nkeywal commented on HBASE-6401: Yeah, I wrote it for hadoop, but then saw it was fixed on their trunk, so I didn't created it. But I haven't found a related jira. We could want to fix it on 1.0.3, adding the ordering I was mentionning on the dev list (put the DN on the same box as the RS as the last locations to make sure we don't use a dead DN). > HBase may lose edits after a crash if used with HDFS 1.0.3 or older > --- > > Key: HBASE-6401 > URL: https://issues.apache.org/jira/browse/HBASE-6401 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.96.0 > Environment: all >Reporter: nkeywal >Priority: Critical > Attachments: TestReadAppendWithDeadDN.java > > > This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the > hdfs jira for this. > Context: HBase Write Ahead Log features. This is using hdfs append. If the > node crashes, the file that was written is read by other processes to replay > the action. > - So we have in hdfs one (dead) process writing with another process reading. > - But, despite the call to syncFs, we don't always see the data when we have > a dead node. It seems to be because the call in DFSClient#updateBlockInfo > ignores the ipc errors and set the length to 0. > - So we may miss all the writes to the last block if we try to connect to the > dead DN. > hdfs 1.0.3, branch-1 or branch-1-win: we have the issue > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java?revision=1359853&view=markup > hdfs branch-2 or trunk: we should not have the issue (but not tested) > http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java?view=markup > The attached test will fail ~50 of the time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6401) HBase may lose edits after a crash if used with HDFS 1.0.3 or older
[ https://issues.apache.org/jira/browse/HBASE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416223#comment-13416223 ] Zhihong Ted Yu commented on HBASE-6401: --- The test is for hadoop. Is there HADOOP- JIRA for this bug ? > HBase may lose edits after a crash if used with HDFS 1.0.3 or older > --- > > Key: HBASE-6401 > URL: https://issues.apache.org/jira/browse/HBASE-6401 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.96.0 > Environment: all >Reporter: nkeywal >Priority: Critical > Attachments: TestReadAppendWithDeadDN.java > > > This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the > hdfs jira for this. > Context: HBase Write Ahead Log features. This is using hdfs append. If the > node crashes, the file that was written is read by other processes to replay > the action. > - So we have in hdfs one (dead) process writing with another process reading. > - But, despite the call to syncFs, we don't always see the data when we have > a dead node. It seems to be because the call in DFSClient#updateBlockInfo > ignores the ipc errors and set the length to 0. > - So we may miss all the writes to the last block if we try to connect to the > dead DN. > hdfs 1.0.3, branch-1 or branch-1-win: we have the issue > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java?revision=1359853&view=markup > hdfs branch-2 or trunk: we should not have the issue (but not tested) > http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java?view=markup > The attached test will fail ~50 of the time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5754) data lost with gora continuous ingest test (goraci)
[ https://issues.apache.org/jira/browse/HBASE-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416207#comment-13416207 ] Keith Turner commented on HBASE-5754: - In the past we just killed Accumulo processes, the tablet servers, loggers, and master process(es). We were not looking to test HDFS. For the next release, Accumulo has moved its write ahead logging to HDFS. So for the next release we plan to start killing HDFS processes as part of our testing. See ACCUMULO-578, ACCUMULO-623, and ACCUMULO-636. > data lost with gora continuous ingest test (goraci) > --- > > Key: HBASE-5754 > URL: https://issues.apache.org/jira/browse/HBASE-5754 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.1 > Environment: 10 node test cluster >Reporter: Eric Newton >Assignee: stack > > Keith Turner re-wrote the accumulo continuous ingest test using gora, which > has both hbase and accumulo back-ends. > I put a billion entries into HBase, and ran the Verify map/reduce job. The > verification failed because about 21K entries were missing. The goraci > [README|https://github.com/keith-turner/goraci] explains the test, and how it > detects missing data. > I re-ran the test with 100 million entries, and it verified successfully. > Both of the times I tested using a billion entries, the verification failed. > If I run the verification step twice, the results are consistent, so the > problem is > probably not on the verify step. > Here's the versions of the various packages: > ||package||version|| > |hadoop|0.20.205.0| > |hbase|0.92.1| > |gora|http://svn.apache.org/repos/asf/gora/trunk r1311277| > |goraci|https://github.com/ericnewton/goraci tagged 2012-04-08| > The change I made to goraci was to configure it for hbase and to allow it to > build properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
