[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310324#comment-14310324 ] Hudson commented on HBASE-10569: FAILURE: Integrated in HBase-TRUNK #6100 (See [https://builds.apache.org/job/HBase-TRUNK/6100/]) HBASE-12956 Binding to 0.0.0.0 is broken after HBASE-10569 (enis: rev 3b56d2a0bc36f9dcb901bb709b8d9ae58df955ff) * hbase-server/src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: Co-locateMetaAndMasterHBASE-10569.pdf, hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch, master_rs.pdf I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310351#comment-14310351 ] Hudson commented on HBASE-10569: SUCCESS: Integrated in HBase-1.0 #718 (See [https://builds.apache.org/job/HBase-1.0/718/]) HBASE-12956 Binding to 0.0.0.0 is broken after HBASE-10569 (enis: rev 15140bf48491d92dae2d514f2cc84c09205d87b7) * hbase-server/src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: Co-locateMetaAndMasterHBASE-10569.pdf, hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch, master_rs.pdf I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310366#comment-14310366 ] Hudson commented on HBASE-10569: FAILURE: Integrated in HBase-1.1 #152 (See [https://builds.apache.org/job/HBase-1.1/152/]) HBASE-12956 Binding to 0.0.0.0 is broken after HBASE-10569 (enis: rev fc7f53f240b3a5a8a1819b0ec2f0f7d8e3aa) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: Co-locateMetaAndMasterHBASE-10569.pdf, hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch, master_rs.pdf I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152590#comment-14152590 ] Jimmy Xiang commented on HBASE-10569: - Updated the release notes because HBASE-12034 changed the behavior a little bit. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: Co-locateMetaAndMasterHBASE-10569.pdf, hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch, master_rs.pdf I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998795#comment-13998795 ] Jonathan Hsieh commented on HBASE-10569: bq. Point of order: If I read it right, you can only do 'agreement' on mailing list (Community decisions must be reached on the mailing list. [1]). Thanks for the reminder. Duly noted. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: Co-locateMetaAndMasterHBASE-10569.pdf, hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch, master_rs.pdf I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14000339#comment-14000339 ] Mikhail Antonov commented on HBASE-10569: - Regarding some discussion about hosting META on master - if we want to run cluster with 1M or 50M regions (HBASE-11165), we may (will?) have to host META regions on multiple servers - on the zk-less assignments (HBASE-11059), short-circuit atomic updates of meta were mentioned (making use of the fact that meta is local to master). That said, how do we now define that - is hosting meta on master the very-nice-to-have optimization, or is that pre-requisite for cluster to function? Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: Co-locateMetaAndMasterHBASE-10569.pdf, hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch, master_rs.pdf I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[
https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998559#comment-13998559
]
Francis Liu commented on HBASE-10569:
-
{quote}
I think the main motivation is the colocation of all the components that are
usually involved in a single transaction. The
main example of this is the Assignment, which involve: Master, META, ZooKeeper.
{quote}
I see but the notion of consolidation is counter-intuitive if we want hbase to
be horizontally scalable. See the-node-that-must-not-be-named as an example.
{quote}
#1 and #2 are part of a generic notification system, which will be used to
propagate ACLs, Visibility, Quotas. (In theory the base of this system is also
the one behind the ZK-less Assignment)
{quote}
We should be able to implement a notification system without #1 and #2? Would
be good to have a doc describing and motivatiing why we need the dependency.
And why it is better than a generic version of the current approach.
{quote}
For the Horizontal scalability, I think that we are going to have Multiple
Master each one operating on its subsection of meta (and the notification
system). This means that you will have concurrent assignments on different
masters.
The best case is where you can fit a full table (regions metadata) on a single
master, the other case is where your table is split on multiple master which
means that operation that requires to work on the full set of regions e.g.
delete, disable, enable need some sort of coordination to provide the full
consistency that you'll get with a full table that fits on a single master.
{quote}
Yep, instead avoiding distributed coordination tasks we should make distributed
operations a first class operation.
Co-locate meta and master
-
Key: HBASE-10569
URL: https://issues.apache.org/jira/browse/HBASE-10569
Project: HBase
Issue Type: Improvement
Components: master, Region Assignment
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: 0.99.0
Attachments: Co-locateMetaAndMasterHBASE-10569.pdf,
hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch,
hbase-10569_v3.patch, master_rs.pdf
I was thinking simplifying/improving the region assignments. The first step
is to co-locate the meta and the master as many people agreed on HBASE-5487.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998302#comment-13998302 ] Jimmy Xiang commented on HBASE-10569: - The default is on now. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: Co-locateMetaAndMasterHBASE-10569.pdf, hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch, master_rs.pdf I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998368#comment-13998368 ] Enis Soztutar commented on HBASE-10569: --- bq. I think having a brand new deployment style with gotcha's we haven't figured out at scale with yet is risky for what we want to be a super stable release. +1 for that. It was also one of my concerns for 1.0. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: Co-locateMetaAndMasterHBASE-10569.pdf, hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch, master_rs.pdf I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997854#comment-13997854 ] Jonathan Hsieh commented on HBASE-10569: I thought that is what we agreed upon old topology default for 1.0 and new topology default for trunk/2.0 during the pow-wow. My vote is to keep the old topology for hbase 1.0, but have the option to go new in 1.0. For 2.0/trunk, we default to the new topology. The old topology is well understood operationally for deploys. Though straightforward for us devs, I think having a brand new deployment style with gotcha's we haven't figured out at scale with yet is risky for what we want to be a super stable release. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: Co-locateMetaAndMasterHBASE-10569.pdf, hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch, master_rs.pdf I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998479#comment-13998479 ] stack commented on HBASE-10569: --- bq. I thought that is what we agreed upon old topology default for 1.0 and new topology default for trunk/2.0 during the pow-wow. Point of order: If I read it right, you can only do 'agreement' on mailing list (Community decisions must be reached on the mailing list. [1]). At the powwow we raised this issue, yeah. We said stuff like it probably more palatable, as noted above, that ...for 1.0 we should just go w/ the old topology and do clean switch to new layout in 2.0... and that we needed to bring this issue to the 1.0 RM's attention. I'd be in favor of keeping our current default for 1.0 apache hbase but would be fine if folks want to flip the option so master carrying regions is not on by default. 1. https://blogs.apache.org/comdev/entry/how_apache_projects_use_consensus Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: Co-locateMetaAndMasterHBASE-10569.pdf, hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch, master_rs.pdf I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997130#comment-13997130 ] stack commented on HBASE-10569: --- [~enis] bq. Will this break rolling upgrades from 0.98 - 0.99? We want to keep 0.98 to 1.0 rolling restart support. It is required we can rolling upgrade from 0.98 to 1.0 (do we have that in our 1.0 scope doc? If not, lets do so). This feature should not preclude. At hackathon discussion was a means of maintaining the current topology -- i.e. Masters do mastering and nothing else, BU Masters sit idle -- because at least for vendors who are not at a major version juncture when hbase 1.0 ships, they'll probably want to keep the old layout. I think that as long as we can do a rolling upgrade and we broadcast it loud enough in the release notes, a master's role changing going from 0.98 to 0.99 such that it hosts regions would be fine (in apache hbase). bq. if we are also allowing to completely disable this feature (as in the other jira), will there still be benefit for this? I think it on by default in 1.0 apache hbase. If not then, if others think differently, on by default in apache hbase 2.0. I updated the attached doc. to address some of the comments. I also put there as something to keep in mind Francis's requirement that hbase master be able to do 1M regions (host and assign fast) very soon and 50M regions not too long after that. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: stack Fix For: 0.99.0 Attachments: Co-locateMetaAndMasterHBASE-10569.pdf, hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch, master_rs.pdf I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997248#comment-13997248 ] stack commented on HBASE-10569: --- bq. ...what is the benefit of having this complexity. Complex because we need to be able to support both topologies? (The patch is taking us toward a simpler deploy and a master state that is easier to reason about; these is also some nice refactoring of the megalithic master and regionserver classes). I could buy the argument that for 1.0 we should just go w/ the old topology and do clean switch to new layout in 2.0. I would like it to be on sooner than this but we are short on testing as it is so one less option, is likely the way to go. bq. We should not do this for saving on RPC's, but actually getting rid of master in-memory state. This is not about saving RPCs. It is not about getting rid of master-in-memory state either. If anything, it is about more of the cluster state being owned by master. bq. If we have the state in meta and only in meta, then we won't need colocation requirements or meta-as-a-single-region. Not sure how this would work. meta is dumb. You need an agent of some kind. I'd like to hear more E. This issue has gotten a bit messy ([~toffer] called it first in attached doc saying we seem to be talking about what this issue is not about). It is raising loads of good stuff that we need to break out and make sure it all gets covered (I opened HBASE-11165 at Francis prompting). I could give this a pass and add links to our 1.0 issue. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: Co-locateMetaAndMasterHBASE-10569.pdf, hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch, master_rs.pdf I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997647#comment-13997647 ] Jimmy Xiang commented on HBASE-10569: - Totally agree with what Stack and Matteo said. bq. 2) if we are also allowing to completely disable this feature (as in the other jira), will there still be benefit for this? Big features usually come in with an option to disable it at first. This is a basic idea to introduce great features with smooth migration paths for users at the beginning, right? Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: Co-locateMetaAndMasterHBASE-10569.pdf, hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch, master_rs.pdf I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997755#comment-13997755 ] stack commented on HBASE-10569: --- [~jxiang] Should we default on co-located master and meta for apache hbase 1.0 (nice new 'feature'?) If defaulted on, it'll be tested. We (and vendors) can also test old topology still works. We should probably ask out on list. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: Co-locateMetaAndMasterHBASE-10569.pdf, hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch, master_rs.pdf I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996691#comment-13996691 ] Matteo Bertozzi commented on HBASE-10569: - I think the main motivation is the colocation of all the components that are usually involved in a single transaction. The main example of this is the Assignment, which involve: Master, META, ZooKeeper. #1 and #2 are part of a generic notification system, which will be used to propagate ACLs, Visibility, Quotas. (In theory the base of this system is also the one behind the ZK-less Assignment) For the Horizontal scalability, I think that we are going to have Multiple Master each one operating on its subsection of meta (and the notification system). This means that you will have concurrent assignments on different masters. The best case is where you can fit a full table (regions metadata) on a single master, the other case is where your table is split on multiple master which means that operation that requires to work on the full set of regions e.g. delete, disable, enable need some sort of coordination to provide the full consistency that you'll get with a full table that fits on a single master. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: Co-locateMetaAndMasterHBASE-10569.pdf, hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch, master_rs.pdf I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996641#comment-13996641 ] Francis Liu commented on HBASE-10569: - Thanks for the doc. Itd be great if we could have the list of use cases we are trying to solve. So we have that motivate the design decisions. During the discussion and my chat with Matteo and Jimmy here is what I got: 1. A method to guarantee security acl changes are fully propagated when a acl change is requested 2. Same as #1 but for quota 3. Remove master daemon to simplify deployment/ops 4. Have a designated set of servers system tables will be hosted on. To isolate it from user region workloads. Feel free to add if I missed anything. #3 and #4 directly motivates this patch. Tho it seeems there was an agreement to still hqve designated hosts as masters? It seems to me #1 and #2 are use cases for a synchronous coordination frameowrk (consensus discussion). Which may or may not require system table colocation. Having fault tolerant coordination as a first class primitive is sorely missing. And I believe enable us avoid design choices which would impede horizontal scalability. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: Co-locateMetaAndMasterHBASE-10569.pdf, hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch, master_rs.pdf I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997217#comment-13997217 ] Enis Soztutar commented on HBASE-10569: --- bq. It is required we can rolling upgrade from 0.98 to 1.0 (do we have that in our 1.0 scope doc? If not, lets do so). I though unless we break it, it should be de facto from release to release. Opened HBASE-11164. bq. At hackathon discussion was a means of maintaining the current topology – i.e. Masters do mastering and nothing else, BU Masters sit idle This is why I was arguing that backup masters should default to hosting no regions unless explicitly specified. bq. because at least for vendors who are not at a major version juncture when hbase 1.0 ships, they'll probably want to keep the old layout. Agreed, that is what we are going to do. bq. I think it on by default in 1.0 apache hbase. If not then, if others think differently, on by default in apache hbase 2.0. My concern is that if we cannot immediately get the benefits of co-locating, and the vendors or some big deployments won't turn this on, what is the benefit of having this complexity. We should not do this for saving on RPC's, but actually getting rid of master in-memory state. If we have the state in meta and only in meta, then we won't need colocation requirements or meta-as-a-single-region. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: Co-locateMetaAndMasterHBASE-10569.pdf, hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch, master_rs.pdf I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969283#comment-13969283 ] Francis Liu commented on HBASE-10569: - Thanks for the writeup. Left some comments, mainly concerned about splittability, as it seems the main draw of this approach is local writes to the meta which seems to be at odds with meta splittability? Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969780#comment-13969780 ] Andrew Purtell commented on HBASE-10569: Please capture the Google document and its comments on this JIRA, all discussions should eventually appear on ASF resources. Thanks! Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969871#comment-13969871 ] stack commented on HBASE-10569: --- I attached the file. Comments are missing. Need to go to Google Doc for that or I can copy them here. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: Co-locateMetaAndMasterHBASE-10569.pdf, hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970083#comment-13970083 ] Andrew Purtell commented on HBASE-10569: Thanks. I can export the doc + comments as DOCX and attach when the discussion there is done. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: Co-locateMetaAndMasterHBASE-10569.pdf, hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13964443#comment-13964443 ] Lars Hofhansl commented on HBASE-10569: --- Thanks for writing this up. If the active HMaster only holds META that should be fine, it'd be moved/recovered quickly. Local writes are great, no cross machine coordination needed. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963672#comment-13963672 ] stack commented on HBASE-10569: --- So, chatted with Jimmy and Matteo and intend doubling-down on this issues' original description; i.e. colocating meta and master; no meta if no master and vice versa so ignore my suggestions we change the subject of this issue. See one-pager here for more argument why we think this the way to go. Includes answers for above concerns: https://docs.google.com/document/d/1xC-bCzAAKO59Xo3XN-Cl6p-5CM_4DMoR-WpnkmYZgpw/edit#heading=h.99evfnn62059 Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961979#comment-13961979 ] Jimmy Xiang commented on HBASE-10569: - Meta regions of course can be assigned to other region servers too. As to Lars' concern, I was thinking to make it a load balancer decision about where to put meta regions. So it can be changed easily. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962019#comment-13962019 ] Jimmy Xiang commented on HBASE-10569: - I filed HBASE-10923 to make it configurable as to where to assign the meta region. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962169#comment-13962169 ] stack commented on HBASE-10569: --- Is this issue misnamed? Should it be One process to do both regionserver and master duties or some such? Colocating master and meta has gotten push back both by Francis and Lars just recently but also earlier up in rb (https://reviews.apache.org/r/19198/ -- see tail of first review comment). Yes, we need to be able to split meta Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962185#comment-13962185 ] Sergey Shelukhin commented on HBASE-10569: -- We can have split meta in this case with distributed master? :) Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961582#comment-13961582 ] Francis Liu commented on HBASE-10569: - Sorry late seeing this patch. Would this patch prevent us from ever splitting the meta in the future? HDFS is moving towards separating namenode responsibilities to have better scalability, just concerned we are doing the reverse (consolidation) and will make scaling harder to do down the road? Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961590#comment-13961590 ] Jimmy Xiang commented on HBASE-10569: - We can still split the meta. A master can host several regions, right? Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961596#comment-13961596 ] Francis Liu commented on HBASE-10569: - Sorry just to clarify, the regions also have to be servable by different servers this way we have a much stronger case for horizontal scalability. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961628#comment-13961628 ] Lars Hofhansl commented on HBASE-10569: --- I share the concern. Part of the benefit of splitting META is that we server portions of it in different RegionServers. Also, seeing the other fallouts from this (eager assignment of Region to the master, moving META during a HMaster failover - which was lightweight before)... I am no longer sure that this is a good idea at all. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13949426#comment-13949426 ] Hudson commented on HBASE-10569: FAILURE: Integrated in HBase-TRUNK #5044 (See [https://builds.apache.org/job/HBase-TRUNK/5044/]) HBASE-10840 Fix findbug warn induced by HBASE-10569.(Anoop) (anoopsamjohn: rev 1582242) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13949612#comment-13949612 ] Lars Hofhansl commented on HBASE-10569: --- Just noticed this issue now. bq. Active master serves table META, namespace, and ACL in a secure installation. So when we failover the master we'll force a move of meta. Won't that lead to bigger blib than before. bq. Backup masters are regionservers too. They serve regions, while trying to be the next active master. Once a backup master becomes the active one, it will serve META and namespace table. Load balancer will move user regions to other regionservers. Seems like this will wreak havoc to data locality. Master failover used to be a light weight event. I fear this makes it much more heavy weight. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13949729#comment-13949729 ] Jimmy Xiang commented on HBASE-10569: - bq. So when we failover the master we'll force a move of meta. Won't that lead to bigger blib than before. Namespace and ACL are small tables cached in ZK/RS. For META, yes, it's a bigger blib. If we remove ZK from the assignment in the future, when the master is failing over, nobody can update the meta. In this case, the meta info cached in the client side won't change. bq. Seems like this will wreak havoc to data locality. In HBASE-10815, we will introduce a configuration so that we can exclude backup masters from serving regions. That should help. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13948729#comment-13948729 ] Enis Soztutar commented on HBASE-10569: --- Did minor edit to release notes. Nice work Jimmy! Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947496#comment-13947496 ] Hudson commented on HBASE-10569: SUCCESS: Integrated in HBase-TRUNK #5040 (See [https://builds.apache.org/job/HBase-TRUNK/5040/]) HBASE-10569 Co-locate meta and master - ADDENDUM (jxiang: rev 1581513) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMasterCommandLine.java HBASE-10569 Co-locate meta and master (jxiang: rev 1581479) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionAdapter.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/DistributedHBaseCluster.java * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/Action.java * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/RestartRsHoldingMetaAction.java * /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/BackupMasterStatusTmpl.jamon * /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon * /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/RegionServerListTmpl.jamon * /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/CoprocessorHConnection.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServerInterface.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterStatusServlet.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterWrapperImpl.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/SimpleLoadBalancer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DisableTableHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/EnableTableHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/TableAddFamilyHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/TableDeleteFamilyHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/TableModifyFamilyHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/StateDumpServlet.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AnnotationReadingPriorityFunction.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServerCommandLine.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java *
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13945807#comment-13945807 ] stack commented on HBASE-10569: --- I'm +1 on this going into trunk. Jimmy is working on ensuring rolling restart works in other issues. Any objections? This thing needs a fat release note Jimmy. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[
https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944490#comment-13944490
]
Hadoop QA commented on HBASE-10569:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636254/hbase-10569_v3.1.patch
against trunk revision .
ATTACHMENT ID: 12636254
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 183
new or modified tests.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 6
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+
t.getMessage().contains(org/apache/hadoop/hdfs/protocol/FSConstants$SafeModeAction))
{
+ for(HTableDescriptor htd:
master.listTableDescriptorsByNamespace(request.getNamespaceName())) {
+ Threads.setDaemonThreadRunning(this.healthCheckChore.getThread(),
getName() + .healthChecker,
+ Threads.setDaemonThreadRunning(this.nonceManagerChore.getThread(),
getName() + .nonceCleaner,
+ final RegionActionResult.Builder builder, ListCellScannable
cellsToReturn, long nonceGroup) {
+ boolean closed = regionServer.closeRegion(encodedRegionName, false, zk,
versionOfClosingNode, sn);
+Boolean closing =
regionServer.regionsInTransitionInRS.get(region.getEncodedNameAsBytes());
+ regionServer.nonceManager.reportOperationFromWal(nonceGroup, nonce,
entry.getKey().getWriteTime());
+MasterCoprocessorHost cpHost =
util.getMiniHBaseCluster().getMaster().getMasterCoprocessorHost();
+for (HRegionInfo r :
ProtobufUtil.getOnlineRegions(t.getRegionServer().getRSRpcServices())) {
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9077//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9077//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9077//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9077//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9077//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9077//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9077//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9077//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9077//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9077//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9077//console
This message is automatically generated.
Co-locate meta and master
-
Key: HBASE-10569
URL: https://issues.apache.org/jira/browse/HBASE-10569
Project: HBase
Issue Type: Improvement
Components: master, Region Assignment
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Attachments: hbase-10569_v1.patch, hbase-10569_v2.patch,
hbase-10569_v3.1.patch, hbase-10569_v3.patch
I was thinking simplifying/improving the region assignments. The first step
is to co-locate the meta and the master as many people agreed on HBASE-5487.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13943687#comment-13943687 ] Jimmy Xiang commented on HBASE-10569: - In patch v3, I reverted the change to MasterServices. ActiveMasterManager doesn't have a thread now. The thread creation is moved to HMaster. Reverted the change in master web UI about the master info port so that it will be easier to support rolling restart. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[
https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13943785#comment-13943785
]
Hadoop QA commented on HBASE-10569:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636132/hbase-10569_v3.patch
against trunk revision .
ATTACHMENT ID: 12636132
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 183
new or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9073//console
This message is automatically generated.
Co-locate meta and master
-
Key: HBASE-10569
URL: https://issues.apache.org/jira/browse/HBASE-10569
Project: HBase
Issue Type: Improvement
Components: master, Region Assignment
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Attachments: hbase-10569_v1.patch, hbase-10569_v2.patch,
hbase-10569_v3.patch
I was thinking simplifying/improving the region assignments. The first step
is to co-locate the meta and the master as many people agreed on HBASE-5487.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[
https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13942686#comment-13942686
]
Hadoop QA commented on HBASE-10569:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12635924/hbase-10569_v2.patch
against trunk revision .
ATTACHMENT ID: 12635924
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 183
new or modified tests.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 8
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+
t.getMessage().contains(org/apache/hadoop/hdfs/protocol/FSConstants$SafeModeAction))
{
+ for(HTableDescriptor htd:
master.listTableDescriptorsByNamespace(request.getNamespaceName())) {
+ Threads.setDaemonThreadRunning(this.healthCheckChore.getThread(),
getName() + .healthChecker,
+ Threads.setDaemonThreadRunning(this.nonceManagerChore.getThread(),
getName() + .nonceCleaner,
+ final RegionActionResult.Builder builder, ListCellScannable
cellsToReturn, long nonceGroup) {
+ boolean closed = regionServer.closeRegion(encodedRegionName, false, zk,
versionOfClosingNode, sn);
+Boolean closing =
regionServer.regionsInTransitionInRS.get(region.getEncodedNameAsBytes());
+ regionServer.nonceManager.reportOperationFromWal(nonceGroup, nonce,
entry.getKey().getWriteTime());
+MasterCoprocessorHost cpHost =
util.getMiniHBaseCluster().getMaster().getMasterCoprocessorHost();
+for (HRegionInfo r :
ProtobufUtil.getOnlineRegions(t.getRegionServer().getRSRpcServices())) {
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnDatanodeDeath(TestLogRolling.java:368)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9056//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9056//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9056//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9056//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9056//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9056//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9056//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9056//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9056//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9056//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9056//console
This message is automatically generated.
Co-locate meta and master
-
Key: HBASE-10569
URL: https://issues.apache.org/jira/browse/HBASE-10569
Project: HBase
Issue Type: Improvement
Components: master, Region Assignment
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Attachments: hbase-10569_v1.patch, hbase-10569_v2.patch
I was thinking simplifying/improving the region assignments. The first step
is to co-locate the meta and the master as many people agreed on HBASE-5487.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13933578#comment-13933578 ] Jimmy Xiang commented on HBASE-10569: - Attached a patch that passed unit tests, integration tests (including ITBLL), and some live cluster tests. Will put it on RB soon. Here is what I have done in this patch: * Moved RPC related code out of HRegionServer and HMaster so that they are smaller for easier change/maintenance. * Make HMaster extends HRegionServer so that HMaster is also a HRegionServer, removed duplicate code/parameters. * Due to B, HMaster#getMetrics is renamed to getMasterMetrics to avoid naming conflict with HRegionServer#getMetrics. The same has been done to HMaster#getCoprocessors, #getCoprocessorHost. * Added HRegionServer#getRpcServices and HMaster#getMasterRpcServices to expose the RPC functionalities. * Changed references related to C and D (a lot, especially in tests). * HMaster and HRegionServer share one RPC server and one InfoServer. * RpcServiceInterface is changed a little. Method #startThreads and #openServer are removed since backup master doesn’t hold the RPC server any more. A parameter HMaster#serviceStarted is introduced to indicate if a master is active so as ServerNotRunningYetException can be thrown before a master is active. * Master recovery in case of ZK connection loss is removed since it doesn’t recover listeners added in HRegionServer. We can get this feature back if needed. The other reason I didn’t try to get it back is because we are going to use raft to choose active master instead of relying on ZK. * HRegionServer on the active HMaster communicates with the active HMaster directly instead of going through the RPC. Shortcut helps. * Master(active/backup) web UI contains info about the corresponding region server. * Backup master moves users regions away (and meta/namespace region to the master if already assigned somewhere else) after becoming active. * Integration testing doesn’t restart the master as a region server, or restart the region server that holds the meta. One reason is because the startup script can’t tell if a region server should be master. Here is a list of things to be done (in separate issues): * Need to make sure the master listens to the old ports (RPC + webUI) too, so as to support rolling upgrade from old versions (0.96+), and be backward compatible. * Need to consolidate(?) chores/threads/handlers in master/regionserver, so that the active master manager in the backup master has a high priority so that it can grab the ZK node faster, before we move to raft. * Clean up MetaServerShutdownHandler and HMaster#assignMeta in next major release when rolling upgrade is not an issue any more. This should be done much later. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-10569_v1.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13933581#comment-13933581 ] Jimmy Xiang commented on HBASE-10569: - The patch contains several bug fixes. I will create separate issues (already created some actually) so that I can push the fixes to 0.96 and 0.98. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-10569_v1.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13933596#comment-13933596 ] Jimmy Xiang commented on HBASE-10569: - RB is down now. Will put the patch on it later. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-10569_v1.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13933621#comment-13933621 ] Ted Yu commented on HBASE-10569: bq. Backup master moves users regions away (and meta/namespace region to the master if already assigned somewhere else) after becoming active. Can you elaborate a bit more about what 'moves users regions away' means ? nit: items are under bullets but referenced by B, C, etc It would be easier to read if items are labeled alphabetically. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-10569_v1.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13933649#comment-13933649 ] Jimmy Xiang commented on HBASE-10569: - Fixed. Thanks. bq. Can you elaborate a bit more about what 'moves users regions away' means ? Since a backup master is also a region server, it could hold many user regions. After it becomes the active master, we try to move these user regions to other region servers so that the active master holds just the meta and the namespace regions. The purpose is to reduce the load on the active master. We could add other regions to the active master later on. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-10569_v1.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13933659#comment-13933659 ] Ted Yu commented on HBASE-10569: bq. the active master holds just the meta and the namespace regions What about ACL and visibility regions ? Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-10569_v1.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13933679#comment-13933679 ] Jimmy Xiang commented on HBASE-10569: - Yes, we can put them on master too. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-10569_v1.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13933881#comment-13933881 ] Jimmy Xiang commented on HBASE-10569: - Patch v1 is on RB now: https://reviews.apache.org/r/19198/ Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-10569_v1.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[
https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13933897#comment-13933897
]
Hadoop QA commented on HBASE-10569:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634484/hbase-10569_v1.patch
against trunk revision .
ATTACHMENT ID: 12634484
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 176
new or modified tests.
{color:red}-1 hadoop1.0{color}. The patch failed to compile against the
hadoop 1.0 profile.
Here is snippet of errors:
{code}[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile
(default-testCompile) on project hbase-server: Compilation failure
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java:[87,24]
cannot find symbol
[ERROR] symbol : method setMiniClusterMode(boolean)
[ERROR] location: class org.apache.hadoop.metrics2.lib.DefaultMetricsSystem
[ERROR] - [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile
(default-testCompile) on project hbase-server: Compilation failure
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java:[87,24]
cannot find symbol
symbol : method setMiniClusterMode(boolean)
location: class org.apache.hadoop.metrics2.lib.DefaultMetricsSystem
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
--
Caused by: org.apache.maven.plugin.CompilationFailureException: Compilation
failure
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java:[87,24]
cannot find symbol
symbol : method setMiniClusterMode(boolean)
location: class org.apache.hadoop.metrics2.lib.DefaultMetricsSystem
at
org.apache.maven.plugin.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:729){code}
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8974//console
This message is automatically generated.
Co-locate meta and master
-
Key: HBASE-10569
URL: https://issues.apache.org/jira/browse/HBASE-10569
Project: HBase
Issue Type: Improvement
Components: master, Region Assignment
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Attachments: hbase-10569_v1.patch
I was thinking simplifying/improving the region assignments. The first step
is to co-locate the meta and the master as many people agreed on HBASE-5487.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13933913#comment-13933913 ] Jimmy Xiang commented on HBASE-10569: - We are not going to support hadoop 1.0 any more, right? http://search-hadoop.com/m/DHED4OxD4C/Next+releases+of+HBase+will+drop+Hadoop-1.x+supportsubj=+ANNOUNCE+Next+releases+of+HBase+will+drop+Hadoop+1+x+support Can we fix the pre-commit test? Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-10569_v1.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934025#comment-13934025 ] Enis Soztutar commented on HBASE-10569: --- Ted has a fix for it in HBASE-10691, but I think it is good to keep the hadoop1 compile around for some more time, since we are still backporting bug fixes etc to 0.98 and 0.96. Maybe we can make it so that the pre-commit test will continue, and just report that the compilation failed as an FYI. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-10569_v1.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934032#comment-13934032 ] Jimmy Xiang commented on HBASE-10569: - I see. That's a good idea. Thanks. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-10569_v1.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934194#comment-13934194 ] stack commented on HBASE-10569: --- bq. # Due to 2, HMaster#getMetrics is renamed to getMasterMetrics to avoid naming conflict with HRegionServer#getMetrics. The same has been done to HMaster#getCoprocessors, #getCoprocessorHost. Make it symmetrical? s/getMetrics/getRegionServerMetrics/ bq. Added HRegionServer#getRpcServices and HMaster#getMasterRpcServices to expose the RPC functionalities. Why expose these? Were they exposed before? When I ask the master for its rpc engine, will it be the same as the regionservers? (It sounds like they will be the same going by #6). If so, should the method be getRpcServices whether on Master or on HRegionServrer (add it to the Server Interface?). Do you mean RpcServiceInterface or RpcServerInterface? If the latter, change it all you want. What is there currently is a bit of a mess. bq. Master recovery in case of ZK connection loss is removed since it doesn’t recover listeners added in HRegionServer. This was a bad idea in the first place? Or rather, it improved our 'usability' when a single Master only in that Master would 'come back to life'but if backup Masters, it was racing the backup Master? (IIRC). Let me look at the patch. This is great Jimmy. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-10569_v1.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934521#comment-13934521 ] Jimmy Xiang commented on HBASE-10569: - Thanks for reviewing it. bq. Make it symmetrical? s/getMetrics/getRegionServerMetrics/ Sure, will do it. bq. Why expose these? Were they exposed before? The reason is that HRegionServer doesn't implement the regionserver RPC interface any more. To access the RPC functionalities directly, we need to expose it. Before that, HRegionServer implements the RPC interface, so there is no such issue. bq. When I ask the master for its rpc engine, will it be the same as the regionservers? Yes, it is the same. Actually, here we are not trying to get the RPC engine. We are just trying to get the RPC functions bypass the RPC. bq. add it to the Server Interface? Probably not apply. Perhaps the method name is confusing. I was meant to get the RPC interface. bq. Do you mean RpcServiceInterface or RpcServerInterface? You are right, the RpcServerInterface. bq. This was a bad idea in the first place? Or rather, it improved our 'usability' when a single Master only in that Master would 'come back to life'but if backup Masters, it was racing the backup Master? (IIRC). I think it is a good idea. But we are going to move away from ZK for leader selection. So I didn't fix this part. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-10569_v1.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934548#comment-13934548 ] stack commented on HBASE-10569: --- [~jxiang] ACK'ing our comments. Sounds good to me. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-10569_v1.patch I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13905749#comment-13905749 ] Elliott Clark commented on HBASE-10569: --- +1 Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13905851#comment-13905851 ] Enis Soztutar commented on HBASE-10569: --- I think this would be a good incremental step towards a more sane master implementation. We can alternatively decide to not co-locate meta, but make meta read only for region servers and clients, only to be written by master. RS operations will be carried through master (RS sends RPC to master, and master does the meta update). If co-locating, are you thinking of opening the region (and rest of RS machinery in master), or move some of the assignment logic to the RS who hosts meta? Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13905883#comment-13905883 ] Jimmy Xiang commented on HBASE-10569: - bq. If co-locating, are you thinking of opening the region (and rest of RS machinery in master), or move some of the assignment logic to the RS who hosts meta? The first one. I was thinking to make a master also a region server. So that all are region servers. By configuration, some region servers can host the master processes. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
