[jira] [Commented] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487431#comment-13487431 ] Subbu M Iyer commented on HBASE-3909: - Ted, Addressed all your comments except the support for having more than one pair of values for the update request. My thought is we shall definitely add that as a additional feature once we commit this one. Also updated the latest patch to review board. thanks a lot for your cooperation on this. Subbu Add dynamic config -- Key: HBASE-3909 URL: https://issues.apache.org/jira/browse/HBASE-3909 Project: HBase Issue Type: Bug Reporter: stack Assignee: Subbu M Iyer Fix For: 0.96.0 Attachments: 3909_090712-2.patch, 3909-102812.patch, 3909-102912.patch, 3909.v1, 3909-v1.patch, HBase Cluster Config Details.xlsx, patch-v2.patch, testMasterNoCluster.stack I'm sure this issue exists already, at least as part of the discussion around making online schema edits possible, but no hard this having its own issue. Ted started a conversation on this topic up on dev and Todd suggested we lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487432#comment-13487432 ] Subbu M Iyer commented on HBASE-3909: - call to containsKey(). If null is returned from get(), we delegate to super.get(). This applies to getLong(), etc. This is tricky. Basically my thought is it's quite possible that some one has overridden a configuration key with null value explicitly and in such cases we do not want to override the same with default value from config. So, if the key is present in CHM, then we must use it no matter what the value is. thoughts ? On Mon, Oct 29, 2012 at 10:12 AM, Ted Yu (JIRA) j...@apache.org wrote: Add dynamic config -- Key: HBASE-3909 URL: https://issues.apache.org/jira/browse/HBASE-3909 Project: HBase Issue Type: Bug Reporter: stack Assignee: Subbu M Iyer Fix For: 0.96.0 Attachments: 3909_090712-2.patch, 3909-102812.patch, 3909-102912.patch, 3909.v1, 3909-v1.patch, HBase Cluster Config Details.xlsx, patch-v2.patch, testMasterNoCluster.stack I'm sure this issue exists already, at least as part of the discussion around making online schema edits possible, but no hard this having its own issue. Ted started a conversation on this topic up on dev and Todd suggested we lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-3909: Attachment: 3909-102912.patch Add dynamic config -- Key: HBASE-3909 URL: https://issues.apache.org/jira/browse/HBASE-3909 Project: HBase Issue Type: Bug Reporter: stack Fix For: 0.96.0 Attachments: 3909_090712-2.patch, 3909-102812.patch, 3909-102912.patch, 3909.v1, 3909-v1.patch, HBase Cluster Config Details.xlsx, patch-v2.patch I'm sure this issue exists already, at least as part of the discussion around making online schema edits possible, but no hard this having its own issue. Ted started a conversation on this topic up on dev and Todd suggested we lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486146#comment-13486146 ] Subbu M Iyer commented on HBASE-3909: - Added new patch to address the missing scripts issue. Thanks Ted for pointing it out. Add dynamic config -- Key: HBASE-3909 URL: https://issues.apache.org/jira/browse/HBASE-3909 Project: HBase Issue Type: Bug Reporter: stack Fix For: 0.96.0 Attachments: 3909_090712-2.patch, 3909-102812.patch, 3909-102912.patch, 3909.v1, 3909-v1.patch, HBase Cluster Config Details.xlsx, patch-v2.patch I'm sure this issue exists already, at least as part of the discussion around making online schema edits possible, but no hard this having its own issue. Ted started a conversation on this topic up on dev and Todd suggested we lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13485805#comment-13485805 ] Subbu M Iyer commented on HBASE-3909: - Attaching latest patch that addresses the review comments. All tests pass except the following and I am reviewing the failures. Tests in error: testGetRowVersions(org.apache.hadoop.hbase.TestMultiVersions): Shutting down testScanMultipleVersions(org.apache.hadoop.hbase.TestMultiVersions): org.apache.hadoop.hbase.MasterNotRunningException: Can create a proxy to master, but it is not running Add dynamic config -- Key: HBASE-3909 URL: https://issues.apache.org/jira/browse/HBASE-3909 Project: HBase Issue Type: Bug Reporter: stack Fix For: 0.96.0 Attachments: 3909_090712-2.patch, 3909-102812.patch, 3909.v1, 3909-v1.patch, HBase Cluster Config Details.xlsx, patch-v2.patch I'm sure this issue exists already, at least as part of the discussion around making online schema edits possible, but no hard this having its own issue. Ted started a conversation on this topic up on dev and Todd suggested we lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-3909: Attachment: 3909-102812.patch Add dynamic config -- Key: HBASE-3909 URL: https://issues.apache.org/jira/browse/HBASE-3909 Project: HBase Issue Type: Bug Reporter: stack Fix For: 0.96.0 Attachments: 3909_090712-2.patch, 3909-102812.patch, 3909.v1, 3909-v1.patch, HBase Cluster Config Details.xlsx, patch-v2.patch I'm sure this issue exists already, at least as part of the discussion around making online schema edits possible, but no hard this having its own issue. Ted started a conversation on this topic up on dev and Todd suggested we lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461560#comment-13461560 ] Subbu M Iyer commented on HBASE-3909: - Stack: Which RB version should l target my patch on ? Please let me know. thanks Subbu Add dynamic config -- Key: HBASE-3909 URL: https://issues.apache.org/jira/browse/HBASE-3909 Project: HBase Issue Type: Bug Reporter: stack Fix For: 0.96.0 Attachments: 3909_090712-2.patch, 3909.v1, 3909-v1.patch, HBase Cluster Config Details.xlsx, patch-v2.patch I'm sure this issue exists already, at least as part of the discussion around making online schema edits possible, but no hard this having its own issue. Ted started a conversation on this topic up on dev and Todd suggested we lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13451167#comment-13451167 ] Subbu M Iyer commented on HBASE-3909: - Added latest patch with protobuf support and addressed the review comments. All 7 tests pass. Here is a sample of how things work from shell: hbase(main):007:0 get_config 'hbase.balancer.period' config_value = : 2 0 row(s) in 0.0170 seconds hbase(main):008:0 update_config 'hbase.balancer.period', '35000' updated config key with new value 0 row(s) in 0.0200 seconds hbase(main):009:0 get_config 'hbase.balancer.period' config_value = : 35000 0 row(s) in 0.0180 seconds Add dynamic config -- Key: HBASE-3909 URL: https://issues.apache.org/jira/browse/HBASE-3909 Project: HBase Issue Type: Bug Reporter: stack Fix For: 0.96.0 Attachments: 3909.v1, 3909-v1.patch, HBase Cluster Config Details.xlsx, patch-v2.patch I'm sure this issue exists already, at least as part of the discussion around making online schema edits possible, but no hard this having its own issue. Ted started a conversation on this topic up on dev and Todd suggested we lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-3909: Attachment: 3909_090712-2.patch Add dynamic config -- Key: HBASE-3909 URL: https://issues.apache.org/jira/browse/HBASE-3909 Project: HBase Issue Type: Bug Reporter: stack Fix For: 0.96.0 Attachments: 3909_090712-2.patch, 3909.v1, 3909-v1.patch, HBase Cluster Config Details.xlsx, patch-v2.patch I'm sure this issue exists already, at least as part of the discussion around making online schema edits possible, but no hard this having its own issue. Ted started a conversation on this topic up on dev and Todd suggested we lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13295400#comment-13295400 ] Subbu M Iyer commented on HBASE-3909: - Thanks Stack/Ted. The latest version of patch basically enables any and all of the configuration properties (that I have captured in the excel. Roughly 300 config properties) to be changed through the shell. Some config changes will take immediate effect, while others will have no effect mid-flight, but will take effect when a Master/RS newly joins the cluster or fails over. So, basically what we have now is all you can change option with no exception. Only caveat is that the user may not know which one will be immediate and which is not. Add dynamic config -- Key: HBASE-3909 URL: https://issues.apache.org/jira/browse/HBASE-3909 Project: HBase Issue Type: Bug Reporter: stack Fix For: 0.96.0 Attachments: 3909-v1.patch, 3909.v1, HBase Cluster Config Details.xlsx, patch-v2.patch I'm sure this issue exists already, at least as part of the discussion around making online schema edits possible, but no hard this having its own issue. Ted started a conversation on this topic up on dev and Todd suggested we lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5713) Introduce throttling during Instant schema change process to throttle opening/closing regions.
[ https://issues.apache.org/jira/browse/HBASE-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294717#comment-13294717 ] Subbu M Iyer commented on HBASE-5713: - Ted: Attached a latest patch. 1. Reduced the throttle time to 100ms. 2. Addressed your comments. All unit tests passed. As a next step, we need to get this tested in a production grade cluster and hope Lars can help us with that. thanks Introduce throttling during Instant schema change process to throttle opening/closing regions. --- Key: HBASE-5713 URL: https://issues.apache.org/jira/browse/HBASE-5713 Project: HBase Issue Type: Bug Components: client, master, regionserver, shell Reporter: Subbu M Iyer Assignee: Subbu M Iyer Priority: Minor Attachments: 5713.txt, patch-v4.patch There is a potential for region open/close stampede during instant schema change process as the process attempts to close/open impacted regions in rapid succession. We need to introduce some kind of throttling to eliminate the race condition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5713) Introduce throttling during Instant schema change process to throttle opening/closing regions.
[ https://issues.apache.org/jira/browse/HBASE-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-5713: Attachment: patch-v4.patch Introduce throttling during Instant schema change process to throttle opening/closing regions. --- Key: HBASE-5713 URL: https://issues.apache.org/jira/browse/HBASE-5713 Project: HBase Issue Type: Bug Components: client, master, regionserver, shell Reporter: Subbu M Iyer Assignee: Subbu M Iyer Priority: Minor Attachments: 5713.txt, patch-v4.patch There is a potential for region open/close stampede during instant schema change process as the process attempts to close/open impacted regions in rapid succession. We need to introduce some kind of throttling to eliminate the race condition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294735#comment-13294735 ] Subbu M Iyer commented on HBASE-3909: - I have tried to capture all (most of) of configuration properties that are currently being used in an excel. The objective here is to figure out what properties may be effectively changed using the dynamic config utility and what is not. While doing this exercise I figured that almost close to 200 properties are missing from hbase-default.xml but is getting used in the system. So, here are my thoughts on this and would like to hear from you guys your thoughts as well: 1. My understanding is that all configuration properties will be/must be defined in hbase-default.xml with a default value. Some exceptions are possible in some corner cases but majority of properties must be defined here. 2. There is no way for a user to know that a property is configurable/changeable unless 1) the property is defined in hbase-default 2) Or communicated through some documentation or API doc. Add dynamic config -- Key: HBASE-3909 URL: https://issues.apache.org/jira/browse/HBASE-3909 Project: HBase Issue Type: Bug Reporter: stack Fix For: 0.96.0 Attachments: 3909-v1.patch, 3909.v1 I'm sure this issue exists already, at least as part of the discussion around making online schema edits possible, but no hard this having its own issue. Ted started a conversation on this topic up on dev and Todd suggested we lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294736#comment-13294736 ] Subbu M Iyer commented on HBASE-3909: - Ted/Uma: Addressed all of your comments in this patch. Add dynamic config -- Key: HBASE-3909 URL: https://issues.apache.org/jira/browse/HBASE-3909 Project: HBase Issue Type: Bug Reporter: stack Fix For: 0.96.0 Attachments: 3909-v1.patch, 3909.v1, HBase Cluster Config Details.xlsx, patch-v2.patch I'm sure this issue exists already, at least as part of the discussion around making online schema edits possible, but no hard this having its own issue. Ted started a conversation on this topic up on dev and Todd suggested we lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-3909: Attachment: HBase Cluster Config Details.xlsx Add dynamic config -- Key: HBASE-3909 URL: https://issues.apache.org/jira/browse/HBASE-3909 Project: HBase Issue Type: Bug Reporter: stack Fix For: 0.96.0 Attachments: 3909-v1.patch, 3909.v1, HBase Cluster Config Details.xlsx, patch-v2.patch I'm sure this issue exists already, at least as part of the discussion around making online schema edits possible, but no hard this having its own issue. Ted started a conversation on this topic up on dev and Todd suggested we lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-3909: Attachment: patch-v2.patch Add dynamic config -- Key: HBASE-3909 URL: https://issues.apache.org/jira/browse/HBASE-3909 Project: HBase Issue Type: Bug Reporter: stack Fix For: 0.96.0 Attachments: 3909-v1.patch, 3909.v1, HBase Cluster Config Details.xlsx, patch-v2.patch I'm sure this issue exists already, at least as part of the discussion around making online schema edits possible, but no hard this having its own issue. Ted started a conversation on this topic up on dev and Todd suggested we lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294740#comment-13294740 ] Subbu M Iyer commented on HBASE-3909: - I am seeing that this is a complicated feature with lots of open questions. 1. There are lot of properties/configuration that may have no effect (even with dynamic config tool) once the server is up. How do we communicate such cases? Do we maintain a smaller subset of configurations that can be changed dynamically and only allow those subsets for dynamic changes? 2. There are lot of properties that might take effect during cases of server fail over/back up server taking over etc. Basically every dynamic configuration change that we apply to the cluster will take effect during server failover/ or back server taking over. How do we communicate such cases? Do we even want this? 3. Do we really think that this feature will be a valuable addition? Add dynamic config -- Key: HBASE-3909 URL: https://issues.apache.org/jira/browse/HBASE-3909 Project: HBase Issue Type: Bug Reporter: stack Fix For: 0.96.0 Attachments: 3909-v1.patch, 3909.v1, HBase Cluster Config Details.xlsx, patch-v2.patch I'm sure this issue exists already, at least as part of the discussion around making online schema edits possible, but no hard this having its own issue. Ted started a conversation on this topic up on dev and Todd suggested we lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5715) Revert 'Instant schema alter' for now, HBASE-4213
[ https://issues.apache.org/jira/browse/HBASE-5715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-5715: Attachment: patch1.patch 1. Completely removed the monitoring code. The thought is since schema change is supposed to happen rather quickly we dont get much from the monitoring capability. We can add it later if there is a real need for same. 2. Added a rudimentary throttling when the RS closes/opens the regions. 3. Increased the sleep time during concurrent split plus schema change executions. Revert 'Instant schema alter' for now, HBASE-4213 - Key: HBASE-5715 URL: https://issues.apache.org/jira/browse/HBASE-5715 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Fix For: 0.94.0 Attachments: patch1.patch, revert.txt, revert.v2.txt, revert.v3.txt, revert.v4.txt, revert094.v4.txt See this discussion: http://search-hadoop.com/m/NxCQh1KlSxR1/Pull+instant+schema+updating+out%253Fsubj=Pull+instant+schema+updating+out+ Pull out hbase-4213 for now. Can add it back later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-4213: Attachment: 4213-V10-Support_instant_schema_changes_through_ZK.patch Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: 4213-Instant_Schema_change_through_ZK.patch, 4213-V10-Support_instant_schema_changes_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-v9.txt, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-4213: Attachment: 4213-V9-Support_instant_schema_changes_through_ZK.patch Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: 4213-Instant_Schema_change_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106776#comment-13106776 ] Subbu M Iyer commented on HBASE-4213: - In addition to most of the review comments, V9 adds: 1. Handle cases when the table is not in enabled state. 2. Ensures that concurrent alter requests are not missed. 3. Validate that alter can survive cases when one of the RS goes down during alter, or multiple RS going down during alter or adding new RS servers to the cloud during an alter. 4. Handle cases when primary master is brought down during alter and secondary takes over. 5. Added new test cases for all of the above mentioned cases. To-Do: 1. Revisit the load balancer toggling. It doesn't seem right. 2. Add monitoring support for alter process. 3. Gracefully handle cases when ZooKeeper throws exceptions during an alter. 4. Add fine grained tests to validate master failover cases. 5. Address any other comments/suggestions. Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: 4213-Instant_Schema_change_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13103925#comment-13103925 ] Subbu M Iyer commented on HBASE-4213: - Based on further discussions, here are some call outs: 1. Provide separate public API's for alter instant operations so as to not break existing public API's with out deprecation or prior notice. 2. Provide a config level setting to enable instant schema update feature. (defaults to false). This will also enable us to release this feature in a more controlled and transparent manner. 3. We don't want to intimidate developers with scary boolean flags that does things in a such a way that they may not completely understand or care about. 4. Providing a developer level API to use instant-alter is good in the sense they can fully capitalize a scalable/fault tolerant variants. At the same time it might be confusing to some in the sense that why we are even providing a not scalable/fault tolerant variant in the first place. 5. We don't want to expose implementation details such as this flag uses ZK to track schema changes and so on So, long story short: In addition to review comments, V7 will include the following: 1. Add new config parameter hbase.instant.schema.change.enabled (exact name of flag is open) and default to false so that all existing API's will go over the current path untouched. 2. Separate public API's for all alter operations which supports the new pattern in addition to existing public API's. New API's will take in a boolean parameter that overrides the config setting on a per request basis. 3. Internally both the API's will go through the same pipeline to promote reuse as well as maintainability. Please let me know your thoughts/comments. Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: 4213-Instant_Schema_change_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-4213: Attachment: 4213-V7-Support_instant_schema_changes_through_ZK.patch Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: 4213-Instant_Schema_change_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13104010#comment-13104010 ] Subbu M Iyer commented on HBASE-4213: - Attached V7 with all the review comments + above mentioned additions. Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: 4213-Instant_Schema_change_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13104255#comment-13104255 ] Subbu M Iyer commented on HBASE-4213: - Andrew, thanks for your comments. In V8. 1. Axed all boolean based public API's from HBaseAdmin and HMAster. 2. Schema changes will now drive off the config param. 3. alter_instant removed. 4. We have new JIRA 4370 where we plan to address the Master/RS failover scenarios in depth. So, we will probably address your new test case scenario there. The current JIRA is to put a basic framework to support schema changes through ZK in place. Currently, when a RS dies halfway through the change the ZK node might stay on for ever, although the schema changes would have been completed when the RS comes back alive. This will be addressed intelligently in 4370 by stamping the RS hostname in ZK and so on. (more details in 4370) Hope this helps and let me know for any thing else. thanks Subbu Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: 4213-Instant_Schema_change_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-4213: Attachment: 4213-V8-Support_instant_schema_changes_through_ZK.patch Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: 4213-Instant_Schema_change_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-4213: Summary: Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. (was: Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: 4213-Instant_Schema_change_through_ZK.patch, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-4213: Attachment: 4213-V5-Support_instant_schema_changes_through_ZK.patch Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: 4213-Instant_Schema_change_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13102335#comment-13102335 ] Subbu M Iyer commented on HBASE-4213: - Ted, I addressed all the issues + a merge conflict and also ran all the tests on my local. All tests pass except the following which has nothing to do with this effort. Results : Failed tests: testWorkerAbort(org.apache.hadoop.hbase.master.TestDistributedLogSplitting): expected:1 but was:0 testMultipleResubmits(org.apache.hadoop.hbase.master.TestSplitLogManager): expected:2 but was:3 testMergeTool(org.apache.hadoop.hbase.util.TestMergeTool): 'merging regions 0 and 1' failed with errCode -1 Tests in error: testScanMultipleVersions(org.apache.hadoop.hbase.TestScanMultipleVersions): Cannot lock storage /work/HBase0909/hbase/build/hbase/test/dfs/name1. The directory is already locked. testAll(org.apache.hadoop.hbase.thrift.TestThriftServer): Cannot lock storage /work/HBase0909/hbase/build/hbase/test/dfs/name2. The directory is already locked. Tests run: 970, Failures: 3, Errors: 2, Skipped: 9 Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: 4213-Instant_Schema_change_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13102336#comment-13102336 ] Subbu M Iyer commented on HBASE-4213: - Yes, I agree with you that we shall address 1) progress reporting and 2) secondary master take over through separate JIRA's. Let me know if you have any comments. The changes are same except new additions to address the 1730 merge. Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: 4213-Instant_Schema_change_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13102344#comment-13102344 ] Subbu M Iyer commented on HBASE-4213: - Ted, Can you help me post it to RB? I have problems posting the same. thanks Subbu Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: 4213-Instant_Schema_change_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4213) Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-4213: Attachment: 4213-Instant_Schema_change_through_ZK.patch Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: 4213-Instant_Schema_change_through_ZK.patch, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13102229#comment-13102229 ] Subbu M Iyer commented on HBASE-4213: - Ted/Stack, As we discussed here is the latest patch that support alter_instant, an instant schema change command that supports (Add, Modify, Delete column and Modify table) actions through ZK. 1. This pattern capitalizes on the fact that HRI's are now in HDFS and need not be sent over again from Master to RS cloud on every schema change event. 2. Offers real time instant schema change as we bypass the explicit bulk reassign (unassign + assign) of regions from master to RS. 3. Offers fault tolerant schema change support as schema changes now go through ZK. Secondary master taking over a failed schema change will be addressed through a separate JIRA. Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: 4213-Instant_Schema_change_through_ZK.patch, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails
[ https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100914#comment-13100914 ] Subbu M Iyer commented on HBASE-4301: - Updated with latest changes. Cleaned up some dead code and refactored a bit to eliminate redundancy. Ted, accept it, it's a team work from both you and me. META migration from 0.90 to trunk fails --- Key: HBASE-4301 URL: https://issues.apache.org/jira/browse/HBASE-4301 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Subbu M Iyer Priority: Blocker Fix For: 0.92.0 Attachments: 4301-1-Fixed_Root_migration_to_newer_HRI_format_.patch, 4301-2-Fixed_Root_migration_to_newer_HRI_format_.patch, 4301-Fixed_Root_migration_to_newer_HRI_format_.patch, 4301-v3.txt, 4301-v4.txt, 4301.txt, 4301_v2.txt, logs.tar.gz, master-log.txt, meta_migrate, meta_trunk, root_migrate, root_trunk I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table is screwed. I can't scan it, etc, and other operations fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4301) META migration from 0.90 to trunk fails
[ https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-4301: Attachment: 4301-2-Fixed_Root_migration_to_newer_HRI_format_.patch META migration from 0.90 to trunk fails --- Key: HBASE-4301 URL: https://issues.apache.org/jira/browse/HBASE-4301 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Subbu M Iyer Priority: Blocker Fix For: 0.92.0 Attachments: 4301-1-Fixed_Root_migration_to_newer_HRI_format_.patch, 4301-2-Fixed_Root_migration_to_newer_HRI_format_.patch, 4301-Fixed_Root_migration_to_newer_HRI_format_.patch, 4301-v3.txt, 4301-v4.txt, 4301.txt, 4301_v2.txt, logs.tar.gz, master-log.txt, meta_migrate, meta_trunk, root_migrate, root_trunk I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table is screwed. I can't scan it, etc, and other operations fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails
[ https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13099720#comment-13099720 ] Subbu M Iyer commented on HBASE-4301: - This patch addresses ROOT migration from older versions of HBASE to trunk, in addition to what is already done by Ted. I removed the table name derivation logic in HConnectionManager as we no longer need it. META migration from 0.90 to trunk fails --- Key: HBASE-4301 URL: https://issues.apache.org/jira/browse/HBASE-4301 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Ted Yu Priority: Blocker Fix For: 0.92.0 Attachments: 4301-Fixed_Root_migration_to_newer_HRI_format_.patch, 4301-v3.txt, 4301-v4.txt, 4301.txt, 4301_v2.txt, logs.tar.gz, master-log.txt, meta_migrate, meta_trunk, root_migrate, root_trunk I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table is screwed. I can't scan it, etc, and other operations fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4301) META migration from 0.90 to trunk fails
[ https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-4301: Attachment: 4301-Fixed_Root_migration_to_newer_HRI_format_.patch META migration from 0.90 to trunk fails --- Key: HBASE-4301 URL: https://issues.apache.org/jira/browse/HBASE-4301 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Ted Yu Priority: Blocker Fix For: 0.92.0 Attachments: 4301-Fixed_Root_migration_to_newer_HRI_format_.patch, 4301-v3.txt, 4301-v4.txt, 4301.txt, 4301_v2.txt, logs.tar.gz, master-log.txt, meta_migrate, meta_trunk, root_migrate, root_trunk I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table is screwed. I can't scan it, etc, and other operations fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4301) META migration from 0.90 to trunk fails
[ https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-4301: Attachment: 4301-1-Fixed_Root_migration_to_newer_HRI_format_.patch META migration from 0.90 to trunk fails --- Key: HBASE-4301 URL: https://issues.apache.org/jira/browse/HBASE-4301 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Ted Yu Priority: Blocker Fix For: 0.92.0 Attachments: 4301-1-Fixed_Root_migration_to_newer_HRI_format_.patch, 4301-Fixed_Root_migration_to_newer_HRI_format_.patch, 4301-v3.txt, 4301-v4.txt, 4301.txt, 4301_v2.txt, logs.tar.gz, master-log.txt, meta_migrate, meta_trunk, root_migrate, root_trunk I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table is screwed. I can't scan it, etc, and other operations fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails
[ https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096898#comment-13096898 ] Subbu M Iyer commented on HBASE-4301: - How come -ROOT- tableinfo has it's name showing as -META- ? That doesn't look correct to me. META migration from 0.90 to trunk fails --- Key: HBASE-4301 URL: https://issues.apache.org/jira/browse/HBASE-4301 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Ted Yu Priority: Blocker Fix For: 0.92.0 Attachments: 4301.txt, 4301_v2.txt, logs.tar.gz, master-log.txt I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table is screwed. I can't scan it, etc, and other operations fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails
[ https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13094549#comment-13094549 ] Subbu M Iyer commented on HBASE-4301: - Todd, From startup log it seems that ROOT and META were assigned fine, followed by meta migration and user region assignment. I do see that all 101 regions were assigned, but noticing a disable table request after that. 11/08/30 12:44:05 INFO master.AssignmentManager: Bulk assigning done 11/08/30 12:44:05 INFO master.HMaster: Master has completed initialization 11/08/30 12:44:06 WARN master.AssignmentManager: Received OPENED for region 08027caad32ade9887e6da234ba4d2a2 from server c0336.x,60020,1314733427177 but region was in the state usertable,user807964132,1314670560271.08027caad32ade9887e6da234ba4d2a2. state=OPEN, ts=1314733445675, server=c0336.x,60020,1314733427177 and not in expected PENDING_OPEN or OPENING states 11/08/30 12:45:38 INFO handler.DisableTableHandler: Attemping to disable table usertable 11/08/30 12:45:38 INFO handler.DisableTableHandler: Offlining 101 regions. Similar patter for table t1: 11/08/30 12:53:09 INFO master.AssignmentManager: Bulk assigning done 11/08/30 12:53:09 INFO master.AssignmentManager: c0317.x,60020,1314733427170 unassigned znodes=1 of total=1 11/08/30 12:53:09 WARN master.AssignmentManager: Node deleted but still in RIT: t1,,1314733988388.8e368edb82720ee77312988c9d4cf496. state=OPEN, ts=1314733989131, server=c0317.x,60020,1314733427170 11/08/30 12:53:19 INFO handler.DisableTableHandler: Attemping to disable table t1 11/08/30 12:53:19 INFO handler.DisableTableHandler: Offlining 1 regions. were you manually trying to 'disable' these tables? META migration from 0.90 to trunk fails --- Key: HBASE-4301 URL: https://issues.apache.org/jira/browse/HBASE-4301 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: Todd Lipcon Priority: Blocker Fix For: 0.92.0 Attachments: master-log.txt I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table is screwed. I can't scan it, etc, and other operations fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails
[ https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13094717#comment-13094717 ] Subbu M Iyer commented on HBASE-4301: - Todd, You mentioned you are seeing the same problem in a fresh cluster as well. Could you post me that log? thanks Subbu META migration from 0.90 to trunk fails --- Key: HBASE-4301 URL: https://issues.apache.org/jira/browse/HBASE-4301 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: Todd Lipcon Priority: Blocker Fix For: 0.92.0 Attachments: master-log.txt I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table is screwed. I can't scan it, etc, and other operations fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails
[ https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13094735#comment-13094735 ] Subbu M Iyer commented on HBASE-4301: - Are you also able to create new table, get, put, disable, truncate etc on fresh cluster? If so, I think the problem can be isolated to a migration from prior version to latest trunk. META migration from 0.90 to trunk fails --- Key: HBASE-4301 URL: https://issues.apache.org/jira/browse/HBASE-4301 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: Todd Lipcon Priority: Blocker Fix For: 0.92.0 Attachments: master-log.txt I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table is screwed. I can't scan it, etc, and other operations fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails
[
https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13094821#comment-13094821
]
Subbu M Iyer commented on HBASE-4301:
-
I think I know what the root cause of this problem might be:
Todd, you mentioned that you are not seeing the .tableinfo for -ROOT-
and .META. for both fresh startup cluster as well as migrated one
which is a problem. We must have .tableinfo for all the tables
including ROOT and META.
In the case of the migration from older version to trunk, the
tableinfo data for .META. and -ROOT- are not getting created. We
originally had this taken care by creating the .tableinfo at create
region time which would address both older migration as well as fresh
start cases.
Long story short, we need to patch this by making master create the
ROOT and META tableinfo at startup (if required, only once during
first time start up).
In MasterFileSystem we might do some thing like:
324 if (!FSUtils.rootRegionExists(fs, rd)) {
325 bootstrap(rd, c);
326 }
// Create META and ROOT tableInfo if required.
if (!FSUtils.rootTableInfoExists(fs, rd)) {
FSUtils.createTableDescriptor(HTableDescriptor.ROOT_TABLEDESC,
this.conf);
FSUtils.createTableDescriptor(HTableDescriptor.META_TABLEDESC,
this.conf);
}
Stack: Can you take this since I am out?
thanks
Subbu
META migration from 0.90 to trunk fails
---
Key: HBASE-4301
URL: https://issues.apache.org/jira/browse/HBASE-4301
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Priority: Blocker
Fix For: 0.92.0
Attachments: master-log.txt
I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table
is screwed. I can't scan it, etc, and other operations fail.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails
[ https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095059#comment-13095059 ] Subbu M Iyer commented on HBASE-4301: - Todd, Can you validate if this addresses your issue? Thanks Sent from my iPhone META migration from 0.90 to trunk fails --- Key: HBASE-4301 URL: https://issues.apache.org/jira/browse/HBASE-4301 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Ted Yu Priority: Blocker Fix For: 0.92.0 Attachments: 4301.txt, master-log.txt I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table is screwed. I can't scan it, etc, and other operations fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails
[ https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13094286#comment-13094286 ] Subbu M Iyer commented on HBASE-4301: - Hi Todd, Can you check your master log and let me know if you see Meta has HRI with HTDs. Updating meta now. and Meta updated with new HRI. and Updated -ROOT- row with metaMigrated status = true ? So, just FYI: We migrate old HRI's (i.e HRI with HTD information) to new HRI (with out HTD, and creating the HTD's in HDFS) at server startup for the first time and if the process is successful we also update -ROOT- with a flag metaMigrated=true. Based on your symptoms it seems to me that either the migration did not go through properly, or some thing else is missing, and we need to troubleshoot further to determine the root cause and the resolution. Please let me know what you see in your logs and also if possible post your logs. thanks Subbu META migration from 0.90 to trunk fails --- Key: HBASE-4301 URL: https://issues.apache.org/jira/browse/HBASE-4301 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: Todd Lipcon Priority: Blocker Fix For: 0.92.0 Attachments: master-log.txt I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table is screwed. I can't scan it, etc, and other operations fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails
[ https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13094290#comment-13094290 ] Subbu M Iyer commented on HBASE-4301: - Sorry missed your log attachment. From the log it seems that the meta migration went through fine and I do see that all tables have .tableinfo created in HDFS. Can you validate in your HDFS whether you see .tableinfo for all your tables? Let me go over the log to see if there is some thing else that related to meta migration. Subbu META migration from 0.90 to trunk fails --- Key: HBASE-4301 URL: https://issues.apache.org/jira/browse/HBASE-4301 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: Todd Lipcon Priority: Blocker Fix For: 0.92.0 Attachments: master-log.txt I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table is screwed. I can't scan it, etc, and other operations fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-1730) Near-instantaneous online schema and table state updates
[
https://issues.apache.org/jira/browse/HBASE-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13092168#comment-13092168
]
Subbu M Iyer commented on HBASE-1730:
-
Nilemma/Nicholas,
Are you (or some one) already working on incorporating the required changes
from 4213 to trunk after your patch is applied?
I would like to work on it as I already have most of the required changes, but
would not want to step into your shoes if some one on your side is already
working on it.
Please let me know.
Near-instantaneous online schema and table state updates
Key: HBASE-1730
URL: https://issues.apache.org/jira/browse/HBASE-1730
Project: HBase
Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: stack
Priority: Critical
Fix For: 0.92.0
Attachments: 1730-v2.patch, 1730-v3.patch, 1730.patch,
HBASE-1730.patch
We should not need to take a table offline to update HCD or HTD.
One option for that is putting HTDs and HCDs up into ZK, with mirror on disk
catalog tables to be used only for cold init scenarios, as discussed on IRC.
In this scheme, regionservers hosting regions of a table would watch
permanent nodes in ZK associated with that table for schema updates and take
appropriate actions out of the watcher. In effect, schema updates become
another item in the ToDo list.
{{/hbase/tables/table-name/schema}}
Must be associated with a write locking scheme also handled with ZK
primitives to avoid situations where one concurrent update clobbers another.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-1730) Near-instantaneous online schema and table state updates
[
https://issues.apache.org/jira/browse/HBASE-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13092170#comment-13092170
]
Subbu M Iyer commented on HBASE-1730:
-
Thanks Nileema. I shall take it from here after your patch is commited to trunk.
Near-instantaneous online schema and table state updates
Key: HBASE-1730
URL: https://issues.apache.org/jira/browse/HBASE-1730
Project: HBase
Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: stack
Priority: Critical
Fix For: 0.92.0
Attachments: 1730-v2.patch, 1730-v3.patch, 1730.patch,
HBASE-1730.patch
We should not need to take a table offline to update HCD or HTD.
One option for that is putting HTDs and HCDs up into ZK, with mirror on disk
catalog tables to be used only for cold init scenarios, as discussed on IRC.
In this scheme, regionservers hosting regions of a table would watch
permanent nodes in ZK associated with that table for schema updates and take
appropriate actions out of the watcher. In effect, schema updates become
another item in the ToDo list.
{{/hbase/tables/table-name/schema}}
Must be associated with a write locking scheme also handled with ZK
primitives to avoid situations where one concurrent update clobbers another.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-1730) Near-instantaneous online schema and table state updates
[
https://issues.apache.org/jira/browse/HBASE-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089862#comment-13089862
]
Subbu M Iyer commented on HBASE-1730:
-
Hi Nicolas,
Couple of call outs:
1. 4213, just like 1730, also implements the online schema change as
well. (but the pattern is slightly different, in that 4213 relies on
the RS cloud to refresh the online regions applicable for the table
with latest schema changes rather than master explicitly unassigning
and reassigning all the regions of the table). When we alter a table
with add/modify/delete CF the only thing that changes is the HTD for
the table, and since the RS has access to this new version of HTD
(after master completes the schema change) it can simply close/open a
region for the changes to take effect. This was the fundamental
assumption behind 4213's implementation.
2. The testing and maturity for 4213 is TBD/Unknown, as I don't have
the required resources (a real cluster setup) to perform the necessary
acid tests to comment on that.
3. Â Karthik other developers had a handful of design concerns about 4123:
Could you please iterate those concerns so we can address them when we
add the persistent capability to 1730?
thanks
Subbu
On Tue, Aug 23, 2011 at 3:35 PM, Nicolas Spiegelberg (JIRA)
Near-instantaneous online schema and table state updates
Key: HBASE-1730
URL: https://issues.apache.org/jira/browse/HBASE-1730
Project: HBase
Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: stack
Priority: Critical
Fix For: 0.92.0
Attachments: 1730-v2.patch, 1730-v3.patch, 1730.patch,
HBASE-1730.patch
We should not need to take a table offline to update HCD or HTD.
One option for that is putting HTDs and HCDs up into ZK, with mirror on disk
catalog tables to be used only for cold init scenarios, as discussed on IRC.
In this scheme, regionservers hosting regions of a table would watch
permanent nodes in ZK associated with that table for schema updates and take
appropriate actions out of the watcher. In effect, schema updates become
another item in the ToDo list.
{{/hbase/tables/table-name/schema}}
Must be associated with a write locking scheme also handled with ZK
primitives to avoid situations where one concurrent update clobbers another.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4213) Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-4213: Attachment: HBASE_Instant_schema_change-version_3_.patch Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13088462#comment-13088462 ] Subbu M Iyer commented on HBASE-4213: - Latest patch includes the following changes: 1. Load balancer will be turned off (if already enabled) before an instant schema change operation. 2. Load balancer will be turned on (if was on before the instant schema change). 3. Added a ZK based check instead of sleep timer for couple of tests. 4. Master stores the number of online RS while creating the schema change node. On completion the master compares against this count (rather than current number of active RS which might have changed during the schema change operation) with number of active RS to determine if all the RS have completed the schema change operation. Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089188#comment-13089188 ] Subbu M Iyer commented on HBASE-4213: - Sure. We have two approaches. 1730 + master failover Or 4213 + throttling + progress status. Let me know which one we want and I can address it accordingly. One question I have is: Do we really need to go through bulk unassign/assign(through ZK) for schema changes ? What is the real drawback of simply signaling the RS cloud to refresh their regions on schema changes? Sent from my iPhone Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13088224#comment-13088224 ] Subbu M Iyer commented on HBASE-4213: - Yes. I will.. I am on it. Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13087110#comment-13087110 ] Subbu M Iyer commented on HBASE-4213: - They both are checking different counts. One counts the number of active RS at this moment (based on number of childrens of /hbase/rs) and other counts the number of RS successfully processed the schema change (based on number of childrens of /hbase/schema/table name ). If number of RS who have processed the schema change is = number of active RS at this moment then master presumes that all RS have acknowledged the schema change and hence goes ahead with delete operation. On Thu, Aug 18, 2011 at 6:45 AM, Ted Yu (JIRA) j...@apache.org wrote: Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: HBASE-4213-Instant_schema_change.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13087383#comment-13087383 ] Subbu M Iyer commented on HBASE-4213: - Ted, Yes, you are absolutely right and very good catch. The second check should be like: int rsCount = ZKUtil.getNumberOfChildren(this.watcher, watcher.rsZNode); instead of int rsCount = ZKUtil.getNumberOfChildren(this.watcher, path); I have taken care of this. thanks Subbu Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: HBASE-4213-Instant_schema_change.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13087503#comment-13087503 ] Subbu M Iyer commented on HBASE-4213: - Attached version 1 with most of call outs addressed. Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4213) Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-4213: Attachment: HBASE-4213_Instant_schema_change_-Version_2_.patch Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13087507#comment-13087507 ] Subbu M Iyer commented on HBASE-4213: - If a new region for this table comes in meantime, it'll be ok? It'll find new schema? Yes. New regions are perfectly fine as they will see new HTD. What happens if master decides to balance a region at this time? Or disable it? While reopenRegions is running? Hmm. This one is interesting. Let me take a closer look at how to address it. Any suggestions? What Ted says on delete cf. TBD. I am on it. Can you look in zk or something rather than wait on a timer before moving on? Yup. Will do. Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13087514#comment-13087514 ] Subbu M Iyer commented on HBASE-4213: - I did test delete family from shell and it worked with out any additional support. I just found that TableDeleteFamilyHandler is using event type EventType.C_M_ADD_FAMILY instead of EventType.C_M_DELETE_FAMILY. So, delete family is already taken care but i will fix the DeleteHandler to use the correct event type as well as well as incorporate the required changes to reflect the same during instant schema change. Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13087523#comment-13087523 ] Subbu M Iyer commented on HBASE-4213: - Yup. sure that makes perfect sense. Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4213) Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)
Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4213) Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-4213: Attachment: HBASE-4213-Instant_schema_change.patch Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Attachments: HBASE-4213-Instant_schema_change.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13086328#comment-13086328 ] Subbu M Iyer commented on HBASE-4213: - As earlier, for some reasons I am not able to attach my patch to the review board. Ted/Stack: Can one of you help me with this? thanks Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Attachments: HBASE-4213-Instant_schema_change.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-1730) Near-instantaneous online schema and table state updates
[
https://issues.apache.org/jira/browse/HBASE-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13086333#comment-13086333
]
Subbu M Iyer commented on HBASE-1730:
-
In continuation of HBASE-451, I was working on a patch for this issue and just
realized that there is already a patch submitted for this Jira. I created a
related patch HBASE-4213 that follows a slightly different approach to the same
problem and thought will any way submit my patch as well.
Near-instantaneous online schema and table state updates
Key: HBASE-1730
URL: https://issues.apache.org/jira/browse/HBASE-1730
Project: HBase
Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: stack
Priority: Critical
Fix For: 0.92.0
Attachments: 1730-v2.patch, 1730-v3.patch, 1730.patch,
HBASE-1730.patch
We should not need to take a table offline to update HCD or HTD.
One option for that is putting HTDs and HCDs up into ZK, with mirror on disk
catalog tables to be used only for cold init scenarios, as discussed on IRC.
In this scheme, regionservers hosting regions of a table would watch
permanent nodes in ZK associated with that table for schema updates and take
appropriate actions out of the watcher. In effect, schema updates become
another item in the ToDo list.
{{/hbase/tables/table-name/schema}}
Must be associated with a write locking scheme also handled with ZK
primitives to avoid situations where one concurrent update clobbers another.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13086745#comment-13086745 ] Subbu M Iyer commented on HBASE-4213: - Ted, Yes. DELETE_FAMILY also needs to be added. Will take care of that. You meant recursively deleting the children of node /hbase/schema/table name, right ? Yes. For region server which joins the cluster after the creation of /hbase/schema/table name, it should be able to find out that it already reads the most recent HTD. Does it create a child node under /hbase/schema/table name ? The newly joined RS will not create a child under /hbase/schema/table name as it will not be processing the schema change event. Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: HBASE-4213-Instant_schema_change.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)
[
https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13086751#comment-13086751
]
Subbu M Iyer commented on HBASE-4213:
-
Regarding:
public boolean refreshSchema(byte[] tableName) throws IOException {
I agree that it could be a void instead of a boolean. Will change that as well.
Support instant schema updates with out master's intervention (i.e with out
enable/disable and bulk assign/unassign)
Key: HBASE-4213
URL: https://issues.apache.org/jira/browse/HBASE-4213
Project: HBase
Issue Type: Improvement
Reporter: Subbu M Iyer
Assignee: Subbu M Iyer
Fix For: 0.92.0
Attachments: HBASE-4213-Instant_schema_change.patch
This Jira is a slight variation in approach to what is being done as part of
https://issues.apache.org/jira/browse/HBASE-1730
Support instant schema updates such as Modify Table, Add Column, Modify
Column operations:
1. With out enable/disabling the table.
2. With out bulk unassign/assign of regions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4025) Server startup fails during startup due to failure in loading all table descriptors. We should ignore .logs,.oldlogs,.corrupt,.META.,-ROOT- folders while reading descript
[ https://issues.apache.org/jira/browse/HBASE-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-4025: Attachment: HBASE-4025_-_Server_startup_fails_while_reading_table_descriptor_from__corrupt_folder_1.patch Server startup fails during startup due to failure in loading all table descriptors. We should ignore .logs,.oldlogs,.corrupt,.META.,-ROOT- folders while reading descriptors -- Key: HBASE-4025 URL: https://issues.apache.org/jira/browse/HBASE-4025 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Subbu M Iyer Attachments: HBASE-4025_-_Server_startup_fails_while_reading_table_descriptor_from__corrupt_folder_1.patch Original Estimate: 2h Remaining Estimate: 2h 2011-06-23 21:39:52,524 WARN org.apache.hadoop.hbase.monitoring.TaskMonitor: Status org.apache.hadoop.hbase.monitoring.MonitoredTaskImpl@2f56f920 appears to have been leaked 2011-06-23 21:40:06,465 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors java.io.FileNotFoundException: No status for hdfs://ciq.com:9000/hbase/.corrupt at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122) at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149) at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1442) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:340) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1138) 2011-06-23 21:40:26,790 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors java.io.FileNotFoundException: No status for hdfs://ciq.com:9000/hbase/.corrupt at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122) at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149) at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1442) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:340) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1138) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4025) Server startup fails during startup due to failure in loading all table descriptors. We should ignore .logs,.oldlogs,.corrupt,.META.,-ROOT- folders while reading descri
[ https://issues.apache.org/jira/browse/HBASE-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13054590#comment-13054590 ] Subbu M Iyer commented on HBASE-4025: - May be we should create all tables under /hbase/tables/table name instead of /hbase/table name so that we can avoid future cases where we create other folder under /hbase such as .logs,.corrupt et al that does not contain table descriptors? So, this pattern will have some thing like: /hbase/.logs /hbase/.corrupt /hbase/.oldlogs /hbase/.META. /hbase/-ROOT- /hbase/future non user system folders /hbase/UserTables/user table folder/.tableinfo and when we need to retrieve all the table descriptors we simply iterate over the /hbase/UserTables folder rather than the /hbase and ignore all system folders. Other option would be: /hbase/System/.logs, .oldlogs, .corrupt et al. /hbase/UserTables/user tables This way we can avoid adding a band-aid fix to this read table descriptor logic every time we have a new system folder. thoughts? Server startup fails during startup due to failure in loading all table descriptors. We should ignore .logs,.oldlogs,.corrupt,.META.,-ROOT- folders while reading descriptors -- Key: HBASE-4025 URL: https://issues.apache.org/jira/browse/HBASE-4025 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Subbu M Iyer Attachments: HBASE-4025_-_Server_startup_fails_while_reading_table_descriptor_from__corrupt_folder_1.patch Original Estimate: 2h Remaining Estimate: 2h 2011-06-23 21:39:52,524 WARN org.apache.hadoop.hbase.monitoring.TaskMonitor: Status org.apache.hadoop.hbase.monitoring.MonitoredTaskImpl@2f56f920 appears to have been leaked 2011-06-23 21:40:06,465 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors java.io.FileNotFoundException: No status for hdfs://ciq.com:9000/hbase/.corrupt at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122) at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149) at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1442) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:340) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1138) 2011-06-23 21:40:26,790 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors java.io.FileNotFoundException: No status for hdfs://ciq.com:9000/hbase/.corrupt at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122) at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149) at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1442) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:340) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1138) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4025) Server startup fails during startup due to failure in loading all table descriptors. We should ignore .logs,.oldlogs,.corrupt,.META.,-ROOT- folders while reading descri
[ https://issues.apache.org/jira/browse/HBASE-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13054685#comment-13054685 ] Subbu M Iyer commented on HBASE-4025: - Agree regarding the naming of variable to HBASE_NON_USER_TABLE_DIRS and as far as where it should go, I don't have a strong opinion either. As far as other issue is concerned, we can go either way as long as we have unique way of identifying all the user tables. Server startup fails during startup due to failure in loading all table descriptors. We should ignore .logs,.oldlogs,.corrupt,.META.,-ROOT- folders while reading descriptors -- Key: HBASE-4025 URL: https://issues.apache.org/jira/browse/HBASE-4025 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Subbu M Iyer Assignee: Subbu M Iyer Attachments: HBASE-4025_-_Server_startup_fails_while_reading_table_descriptor_from__corrupt_folder_1.patch Original Estimate: 2h Remaining Estimate: 2h 2011-06-23 21:39:52,524 WARN org.apache.hadoop.hbase.monitoring.TaskMonitor: Status org.apache.hadoop.hbase.monitoring.MonitoredTaskImpl@2f56f920 appears to have been leaked 2011-06-23 21:40:06,465 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors java.io.FileNotFoundException: No status for hdfs://ciq.com:9000/hbase/.corrupt at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122) at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149) at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1442) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:340) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1138) 2011-06-23 21:40:26,790 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors java.io.FileNotFoundException: No status for hdfs://ciq.com:9000/hbase/.corrupt at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122) at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149) at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1442) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:340) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1138) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4025) Server startup fails during startup due to failure in loading all table descriptors. We should ignore .logs,.oldlogs,.corrupt,.META.,-ROOT- folders while reading descript
Server startup fails during startup due to failure in loading all table descriptors. We should ignore .logs,.oldlogs,.corrupt,.META.,-ROOT- folders while reading descriptors -- Key: HBASE-4025 URL: https://issues.apache.org/jira/browse/HBASE-4025 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Subbu M Iyer 2011-06-23 21:39:52,524 WARN org.apache.hadoop.hbase.monitoring.TaskMonitor: Status org.apache.hadoop.hbase.monitoring.MonitoredTaskImpl@2f56f920 appears to have been leaked 2011-06-23 21:40:06,465 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors java.io.FileNotFoundException: No status for hdfs://sjc1-hadoop0.sjc1.carrieriq.com:9000/hbase/.corrupt at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122) at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149) at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1442) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:340) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1138) 2011-06-23 21:40:26,790 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors java.io.FileNotFoundException: No status for hdfs://sjc1-hadoop0.sjc1.carrieriq.com:9000/hbase/.corrupt at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122) at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149) at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1442) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:340) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1138) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-451) Remove HTableDescriptor from HRegionInfo
[ https://issues.apache.org/jira/browse/HBASE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13051197#comment-13051197 ] Subbu M Iyer commented on HBASE-451: Stack: Checked your patch and this is exactly what I was trying to fight with. Hopefully it works this time. A simple test would be to run the TestColumnPrefixFilter and see if it succeeds. It used to hang around the PUT and it is a positive sign if this one succeeds. Let me know your findings. My MVN version is 3.0.2 and as I mentioned my problem occurs only with newer version of pom. Remove HTableDescriptor from HRegionInfo Key: HBASE-451 URL: https://issues.apache.org/jira/browse/HBASE-451 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.2.0 Reporter: Jim Kellerman Assignee: Subbu M Iyer Priority: Critical Fix For: 0.92.0 Attachments: 451_support_for_removing_HTD_from_HRI_trunk.txt, HBASE-451-Fixed_broken_TestAdmin.patch, HBASE-451-Fixed_broken_TestAdmin1.patch, HBASE-451_-_First_draft_support_for_removing_HTD_from_HRI1.patch, HBASE-451_-_Fourth_draft_support_for_removing_HTD_from_HRI.patch, HBASE-451_-_Second_draft_-_Remove_HTD_from_HRI.patch, fixtestadmin.txt, pass_htd_on_region_construction.txt There is an HRegionInfo for every region in HBase. Currently HRegionInfo also contains the HTableDescriptor (the schema). That means we store the schema n times where n is the number of regions in the table. Additionally, for every region of the same table that the region server has open, there is a copy of the schema. Thus it is stored in memory once for each open region. If HRegionInfo merely contained the table name the HTableDescriptor could be stored in a separate file and easily found. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3970) Address HMaster crash/failure half way through meta migration
[ https://issues.apache.org/jira/browse/HBASE-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050587#comment-13050587 ] Subbu M Iyer commented on HBASE-3970: - Here is how this patch addresses this issue: When a master dies during meta migration, we will have some rows of META.CatalogFamily updated with new HRI, (i.e HRI with out HTD) and some still hanging with legacy HRI. (i.e HRI with HTD). When the backup master/ or fresh start of master attempts the migration again, it will encounter some rows of META already updated with new HRI and some still legacy. This patch adds additional logic during HRegionInfo deserialization to ignore the cases where HRI is missing a HTD (i.e already migrated to new HRI) and attempts to migrate rest of the rows. Address HMaster crash/failure half way through meta migration - Key: HBASE-3970 URL: https://issues.apache.org/jira/browse/HBASE-3970 Project: HBase Issue Type: Bug Components: client, master, regionserver Affects Versions: 0.92.0 Reporter: Subbu M Iyer Assignee: Subbu M Iyer Priority: Blocker Attachments: HBASE-3970-First_Draft_-_Address_HMaster_crash_failure_half_way_through_meta_migration1.patch Original Estimate: 168h Remaining Estimate: 168h When HMaster tries to migrate (after HBASE-451 goes live) the old HRI (with HTD) to new HRI (with out HTD) and if the Master or the migration process crashes/fails midway, it will leave the .META. in a corrupt state and may not allow successful cluster startup. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3970) Address HMaster crash/failure half way through meta migration
[ https://issues.apache.org/jira/browse/HBASE-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-3970: Attachment: HBASE-3970-First_Draft_-_Address_HMaster_crash_failure_half_way_through_meta_migration2.patch Address HMaster crash/failure half way through meta migration - Key: HBASE-3970 URL: https://issues.apache.org/jira/browse/HBASE-3970 Project: HBase Issue Type: Bug Components: client, master, regionserver Affects Versions: 0.92.0 Reporter: Subbu M Iyer Assignee: Subbu M Iyer Priority: Blocker Attachments: HBASE-3970-First_Draft_-_Address_HMaster_crash_failure_half_way_through_meta_migration1.patch, HBASE-3970-First_Draft_-_Address_HMaster_crash_failure_half_way_through_meta_migration2.patch Original Estimate: 168h Remaining Estimate: 168h When HMaster tries to migrate (after HBASE-451 goes live) the old HRI (with HTD) to new HRI (with out HTD) and if the Master or the migration process crashes/fails midway, it will leave the .META. in a corrupt state and may not allow successful cluster startup. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-451) Remove HTableDescriptor from HRegionInfo
[ https://issues.apache.org/jira/browse/HBASE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049849#comment-13049849 ] Subbu M Iyer commented on HBASE-451: Hi Stack: I did see this problem with concurrent create and fixed it before, but looks like it slipped through the cracks. Here is a simple fix that addresses this and I have attached the patch. I validated that the TestAdmin passes on my latest trunk snapshot. Please let me know if any other tests are failing like TestMergeTool and TestSCannerTimeOut. I am kind of confused on why it failed it consistently before for you and it passed suddenly. May be some other networking blip/timeout or some thing? Regarding passing HTD as a variable in constructor: I did have one version of patch which had both the constructors and for some mysterious reasons (which I could never figure out), whenever I created a HREgion with HTD in the constructor, the PUTS will hang while trying to compute the MemStore size. (@ MemStore.internalAdd - this.size.addAndGet(s);). I have spent like days on this before giving it up and sticking to only doing so at create time. By all means please feel free to make changes as you see fit and I am all open for it. I definitely want to take another pass at this once trunk is stabilized. thanks for your help/support on this. Remove HTableDescriptor from HRegionInfo Key: HBASE-451 URL: https://issues.apache.org/jira/browse/HBASE-451 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.2.0 Reporter: Jim Kellerman Assignee: Subbu M Iyer Priority: Critical Fix For: 0.92.0 Attachments: 451_support_for_removing_HTD_from_HRI_trunk.txt, HBASE-451_-_First_draft_support_for_removing_HTD_from_HRI1.patch, HBASE-451_-_Fourth_draft_support_for_removing_HTD_from_HRI.patch, HBASE-451_-_Second_draft_-_Remove_HTD_from_HRI.patch, fixtestadmin.txt There is an HRegionInfo for every region in HBase. Currently HRegionInfo also contains the HTableDescriptor (the schema). That means we store the schema n times where n is the number of regions in the table. Additionally, for every region of the same table that the region server has open, there is a copy of the schema. Thus it is stored in memory once for each open region. If HRegionInfo merely contained the table name the HTableDescriptor could be stored in a separate file and easily found. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-451) Remove HTableDescriptor from HRegionInfo
[ https://issues.apache.org/jira/browse/HBASE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-451: --- Attachment: HBASE-451-Fixed_broken_TestAdmin.patch Remove HTableDescriptor from HRegionInfo Key: HBASE-451 URL: https://issues.apache.org/jira/browse/HBASE-451 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.2.0 Reporter: Jim Kellerman Assignee: Subbu M Iyer Priority: Critical Fix For: 0.92.0 Attachments: 451_support_for_removing_HTD_from_HRI_trunk.txt, HBASE-451-Fixed_broken_TestAdmin.patch, HBASE-451_-_First_draft_support_for_removing_HTD_from_HRI1.patch, HBASE-451_-_Fourth_draft_support_for_removing_HTD_from_HRI.patch, HBASE-451_-_Second_draft_-_Remove_HTD_from_HRI.patch, fixtestadmin.txt There is an HRegionInfo for every region in HBase. Currently HRegionInfo also contains the HTableDescriptor (the schema). That means we store the schema n times where n is the number of regions in the table. Additionally, for every region of the same table that the region server has open, there is a copy of the schema. Thus it is stored in memory once for each open region. If HRegionInfo merely contained the table name the HTableDescriptor could be stored in a separate file and easily found. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-451) Remove HTableDescriptor from HRegionInfo
[ https://issues.apache.org/jira/browse/HBASE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-451: --- Attachment: HBASE-451-Fixed_broken_TestAdmin1.patch Remove HTableDescriptor from HRegionInfo Key: HBASE-451 URL: https://issues.apache.org/jira/browse/HBASE-451 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.2.0 Reporter: Jim Kellerman Assignee: Subbu M Iyer Priority: Critical Fix For: 0.92.0 Attachments: 451_support_for_removing_HTD_from_HRI_trunk.txt, HBASE-451-Fixed_broken_TestAdmin.patch, HBASE-451-Fixed_broken_TestAdmin1.patch, HBASE-451_-_First_draft_support_for_removing_HTD_from_HRI1.patch, HBASE-451_-_Fourth_draft_support_for_removing_HTD_from_HRI.patch, HBASE-451_-_Second_draft_-_Remove_HTD_from_HRI.patch, fixtestadmin.txt There is an HRegionInfo for every region in HBase. Currently HRegionInfo also contains the HTableDescriptor (the schema). That means we store the schema n times where n is the number of regions in the table. Additionally, for every region of the same table that the region server has open, there is a copy of the schema. Thus it is stored in memory once for each open region. If HRegionInfo merely contained the table name the HTableDescriptor could be stored in a separate file and easily found. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-451) Remove HTableDescriptor from HRegionInfo
[ https://issues.apache.org/jira/browse/HBASE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049857#comment-13049857 ] Subbu M Iyer commented on HBASE-451: Please use TestAdmin1.patch as I accidentally included my pom changes. For some reason the latest trunk is failing for me when I do a compile. I am still using old POM to make thins work. Any ideas why? Remove HTableDescriptor from HRegionInfo Key: HBASE-451 URL: https://issues.apache.org/jira/browse/HBASE-451 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.2.0 Reporter: Jim Kellerman Assignee: Subbu M Iyer Priority: Critical Fix For: 0.92.0 Attachments: 451_support_for_removing_HTD_from_HRI_trunk.txt, HBASE-451-Fixed_broken_TestAdmin.patch, HBASE-451-Fixed_broken_TestAdmin1.patch, HBASE-451_-_First_draft_support_for_removing_HTD_from_HRI1.patch, HBASE-451_-_Fourth_draft_support_for_removing_HTD_from_HRI.patch, HBASE-451_-_Second_draft_-_Remove_HTD_from_HRI.patch, fixtestadmin.txt There is an HRegionInfo for every region in HBase. Currently HRegionInfo also contains the HTableDescriptor (the schema). That means we store the schema n times where n is the number of regions in the table. Additionally, for every region of the same table that the region server has open, there is a copy of the schema. Thus it is stored in memory once for each open region. If HRegionInfo merely contained the table name the HTableDescriptor could be stored in a separate file and easily found. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-451) Remove HTableDescriptor from HRegionInfo
[ https://issues.apache.org/jira/browse/HBASE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049965#comment-13049965 ] Subbu M Iyer commented on HBASE-451: Stack: can you please confirm that all tests are now passing? Also regarding my build, this is what I get when I compile goal from Idea. I simply rolled back my pom to a previous version (Revision 1099198) to make things work. Any ideas how I can fix this? = /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -Dclassworlds.conf=/work/maven/apache-maven-3.0.2/bin/m2.conf -Dmaven.home=/work/maven/apache-maven-3.0.2 -Dfile.encoding=MacRoman -classpath /work/maven/apache-maven-3.0.2/boot/plexus-classworlds-2.4.jar org.codehaus.classworlds.Launcher --no-plugin-registry --fail-fast --no-plugin-updates --strict-checksums --update-snapshots -f /work/HBaseTrunk0615/hbase/pom.xml compile [WARNING] Command line option -npu is deprecated and will be removed in future Maven versions. [WARNING] Command line option -npr is deprecated and will be removed in future Maven versions. [INFO] Scanning for projects... [WARNING] [WARNING] Some problems were encountered while building the effective model for org.apache.hbase:hbase:jar:0.91.0-SNAPSHOT [WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-eclipse-plugin is missing. @ line 574, column 15 [WARNING] [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build. [WARNING] [WARNING] For this reason, future Maven versions might no longer support building such malformed projects. [WARNING] [INFO] [INFO] [INFO] Building HBase 0.91.0-SNAPSHOT [INFO] Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-eclipse-plugin/maven-metadata.xml [WARNING] Could not transfer metadata org.apache.maven.plugins:maven-eclipse-plugin/maven-metadata.xml from/to central (http://repo1.maven.org/maven2): Error transferring file: repo1.maven.org [INFO] [INFO] --- maven-antrun-plugin:1.6:run (generate) @ hbase --- Downloading: http://repo1.maven.org/maven2/org/apache/maven/maven-plugin-api/2.0.11/maven-plugin-api-2.0.11.pom Downloading: http://repo1.maven.org/maven2/org/apache/maven/maven-project/2.0.11/maven-project-2.0.11.pom Downloading: http://repo1.maven.org/maven2/org/apache/maven/maven-artifact/2.0.11/maven-artifact-2.0.11.pom Downloading: http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-utils/2.0.5/plexus-utils-2.0.5.pom Downloading: http://repo1.maven.org/maven2/org/apache/ant/ant-nodeps/1.8.1/ant-nodeps-1.8.1.pom [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 2.576s [INFO] Finished at: Wed Jun 15 12:03:30 PDT 2011 [INFO] Final Memory: 5M/81M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (generate) on project hbase: Execution generate of goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run failed: Plugin org.apache.maven.plugins:maven-antrun-plugin:1.6 or one of its dependencies could not be resolved: Failed to collect dependencies for org.apache.maven.plugins:maven-antrun-plugin:jar:1.6 (): Failed to read artifact descriptor for org.apache.maven:maven-plugin-api:jar:2.0.11: Could not transfer artifact org.apache.maven:maven-plugin-api:pom:2.0.11 from/to central (http://repo1.maven.org/maven2): Error transferring file: repo1.maven.org: Unknown host repo1.maven.org - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException Process finished with exit code 1 Remove HTableDescriptor from HRegionInfo Key: HBASE-451 URL: https://issues.apache.org/jira/browse/HBASE-451 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.2.0 Reporter: Jim Kellerman Assignee: Subbu M Iyer Priority: Critical Fix For: 0.92.0 Attachments: 451_support_for_removing_HTD_from_HRI_trunk.txt,
[jira] [Commented] (HBASE-451) Remove HTableDescriptor from HRegionInfo
[ https://issues.apache.org/jira/browse/HBASE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13047816#comment-13047816 ] Subbu M Iyer commented on HBASE-451: Stack: I ran both these tests on my local version and both are passing. Please attach your output/error log so I can see what's happening. Also would help if u can attach your version of both the test files so I can compare if there is missing there. thanks --- T E S T S --- Running org.apache.hadoop.hbase.client.TestScannerTimeout Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 58.164 sec Results : Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 1:06.905s [INFO] Finished at: Fri Jun 10 18:30:43 PDT 2011 [INFO] Final Memory: 18M/81M [INFO] --- T E S T S --- Running org.apache.hadoop.hbase.util.TestMergeTool Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 120.018 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 2:21.650s [INFO] Finished at: Fri Jun 10 16:47:04 PDT 2011 [INFO] Final Memory: 18M/81M [INFO] Remove HTableDescriptor from HRegionInfo Key: HBASE-451 URL: https://issues.apache.org/jira/browse/HBASE-451 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.2.0 Reporter: Jim Kellerman Assignee: Subbu M Iyer Priority: Critical Fix For: 0.92.0 Attachments: 451_support_for_removing_HTD_from_HRI_trunk.txt, HBASE-451_-_First_draft_support_for_removing_HTD_from_HRI1.patch, HBASE-451_-_Fourth_draft_support_for_removing_HTD_from_HRI.patch, HBASE-451_-_Second_draft_-_Remove_HTD_from_HRI.patch There is an HRegionInfo for every region in HBase. Currently HRegionInfo also contains the HTableDescriptor (the schema). That means we store the schema n times where n is the number of regions in the table. Additionally, for every region of the same table that the region server has open, there is a copy of the schema. Thus it is stored in memory once for each open region. If HRegionInfo merely contained the table name the HTableDescriptor could be stored in a separate file and easily found. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-451) Remove HTableDescriptor from HRegionInfo
[ https://issues.apache.org/jira/browse/HBASE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-451: --- Attachment: HBASE-451_-_Fourth_draft_support_for_removing_HTD_from_HRI.patch Remove HTableDescriptor from HRegionInfo Key: HBASE-451 URL: https://issues.apache.org/jira/browse/HBASE-451 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.2.0 Reporter: Jim Kellerman Assignee: Subbu M Iyer Priority: Critical Fix For: 0.92.0 Attachments: 451_support_for_removing_HTD_from_HRI_trunk.txt, HBASE-451_-_First_draft_support_for_removing_HTD_from_HRI1.patch, HBASE-451_-_Fourth_draft_support_for_removing_HTD_from_HRI.patch, HBASE-451_-_Second_draft_-_Remove_HTD_from_HRI.patch There is an HRegionInfo for every region in HBase. Currently HRegionInfo also contains the HTableDescriptor (the schema). That means we store the schema n times where n is the number of regions in the table. Additionally, for every region of the same table that the region server has open, there is a copy of the schema. Thus it is stored in memory once for each open region. If HRegionInfo merely contained the table name the HTableDescriptor could be stored in a separate file and easily found. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-451) Remove HTableDescriptor from HRegionInfo
[ https://issues.apache.org/jira/browse/HBASE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13046681#comment-13046681 ] Subbu M Iyer commented on HBASE-451: I have created a new trunk snapshot last evening and applied my patch as well as resolved the conflicts. All tests have passed except the two below. 1. TestReplication.queueFailover - I see a very informative note on this failure in the test as shown below, so I would think that this is not due to HBASE-451 changes. -- WARNING this test sometimes fails because of HBASE-3515 2. TestMasterFailover - Need to research more. Both these tests passed in my older snapshot build (which is one week+ old) and I will review it closer again. Also, a closer look at the failure log shows that these failures may not be due to HBASE-451 changes. Here are my priorities now: 1. Address failures from your test run. 2. Work on the issue of master crash during meta migration at startup. Remove HTableDescriptor from HRegionInfo Key: HBASE-451 URL: https://issues.apache.org/jira/browse/HBASE-451 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.2.0 Reporter: Jim Kellerman Assignee: Subbu M Iyer Priority: Critical Fix For: 0.92.0 Attachments: 451_support_for_removing_HTD_from_HRI_trunk.txt, HBASE-451_-_First_draft_support_for_removing_HTD_from_HRI1.patch, HBASE-451_-_Fourth_draft_support_for_removing_HTD_from_HRI.patch, HBASE-451_-_Second_draft_-_Remove_HTD_from_HRI.patch There is an HRegionInfo for every region in HBase. Currently HRegionInfo also contains the HTableDescriptor (the schema). That means we store the schema n times where n is the number of regions in the table. Additionally, for every region of the same table that the region server has open, there is a copy of the schema. Thus it is stored in memory once for each open region. If HRegionInfo merely contained the table name the HTableDescriptor could be stored in a separate file and easily found. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-3970) Address HMaster crash/failure half way through meta migration
Address HMaster crash/failure half way through meta migration - Key: HBASE-3970 URL: https://issues.apache.org/jira/browse/HBASE-3970 Project: HBase Issue Type: Bug Components: client, master, regionserver Affects Versions: 0.92.0 Reporter: Subbu M Iyer Assignee: Subbu M Iyer Priority: Blocker When HMaster tries to migrate (after HBASE-451 goes live) the old HRI (with HTD) to new HRI (with out HTD) and if the Master or the migration process crashes/fails midway, it will leave the .META. in a corrupt state and may not allow successful cluster startup. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-451) Remove HTableDescriptor from HRegionInfo
[ https://issues.apache.org/jira/browse/HBASE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13043345#comment-13043345 ] Subbu M Iyer commented on HBASE-451: Hi Ted, Let me take a look at it. For reference I have attached my test results as well. All tests were passing in my local. == /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -Dclassworlds.conf=/work/maven/apache-maven-3.0.2/bin/m2.conf -Dmaven.home=/work/maven/apache-maven-3.0.2 -Dfile.encoding=MacRoman -classpath /work/maven/apache-maven-3.0.2/boot/plexus-classworlds-2.4.jar org.codehaus.classworlds.Launcher --no-plugin-registry --fail-fast --no-plugin-updates --strict-checksums --update-snapshots -f /work/HBase451/hbase/pom.xml test [WARNING] Command line option -npu is deprecated and will be removed in future Maven versions. [WARNING] Command line option -npr is deprecated and will be removed in future Maven versions. [INFO] Scanning for projects... [WARNING] [WARNING] Some problems were encountered while building the effective model for org.apache.hbase:hbase:jar:0.91.0-SNAPSHOT [WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-eclipse-plugin is missing. @ line 531, column 15 [WARNING] [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build. [WARNING] [WARNING] For this reason, future Maven versions might no longer support building such malformed projects. [WARNING] [INFO] [INFO] [INFO] Building HBase 0.91.0-SNAPSHOT [INFO] Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-eclipse-plugin/maven-metadata.xml 680 B Downloaded: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-eclipse-plugin/maven-metadata.xml (680 B at 1.4 KB/sec) [INFO] [INFO] --- maven-antrun-plugin:1.3:run (generate) @ hbase --- [INFO] Executing tasks SLF4J: Failed to load class org.slf4j.impl.StaticLoggerBinder. SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. 2011-05-31 18:57:10.472:INFO::Logging to STDERR via org.mortbay.log.StdErrLog [exec] /work/HBase451/hbase/src/saveVersion.sh: line 32: git: command not found [INFO] Executed tasks [INFO] [INFO] --- build-helper-maven-plugin:1.5:add-source (add-jspc-source) @ hbase --- [INFO] Source directory: /work/HBase451/hbase/target/jspc added. [INFO] [INFO] --- build-helper-maven-plugin:1.5:add-source (add-package-info) @ hbase --- [INFO] Source directory: /work/HBase451/hbase/target/generated-sources added. [INFO] [INFO] --- jamon-maven-plugin:2.3.4:translate (default) @ hbase --- [INFO] [INFO] --- maven-remote-resources-plugin:1.1:process (default) @ hbase --- [INFO] Setting property: classpath.resource.loader.class = 'org.codehaus.plexus.velocity.ContextClassLoaderResourceLoader'. [INFO] Setting property: velocimacro.messages.on = 'false'. [INFO] Setting property: resource.loader = 'classpath'. [INFO] Setting property: resource.manager.logwhenfound = 'false'. [INFO] [INFO] --- maven-resources-plugin:2.4:resources (default-resources) @ hbase --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 1 resource [INFO] Copying 6 resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-antrun-plugin:1.3:run (default) @ hbase --- [INFO] Executing tasks [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:2.0.2:compile (default-compile) @ hbase --- [INFO] Compiling 8 source files to /work/HBase451/hbase/target/classes [INFO] [INFO] --- maven-resources-plugin:2.4:testResources (default-testResources) @ hbase --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 4 resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.0.2:testCompile (default-testCompile) @ hbase --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-surefire-plugin:2.4.3:test (default-test) @ hbase --- [INFO] Surefire report directory: /work/HBase451/hbase/target/surefire-reports 2011-05-31 18:57:18,255 INFO [main] hbase.HRegionInfo(260): *** New HRegionInfo testscanner region id = 1306893438255 --- T E S T S --- Running org.apache.hadoop.hbase.master.TestHMasterRPCException Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.726 sec Running org.apache.hadoop.hbase.regionserver.TestColumnSeeking Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.908 sec Running org.apache.hadoop.hbase.client.TestMultipleTimestamps Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.488
[jira] [Commented] (HBASE-451) Remove HTableDescriptor from HRegionInfo
[ https://issues.apache.org/jira/browse/HBASE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13043346#comment-13043346 ] Subbu M Iyer commented on HBASE-451: Hi Ted, Can you send me your complete failure list? Remove HTableDescriptor from HRegionInfo Key: HBASE-451 URL: https://issues.apache.org/jira/browse/HBASE-451 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.2.0 Reporter: Jim Kellerman Assignee: Subbu M Iyer Priority: Critical Fix For: 0.92.0 Attachments: 451_support_for_removing_HTD_from_HRI_trunk.txt, HBASE-451_-_First_draft_support_for_removing_HTD_from_HRI1.patch There is an HRegionInfo for every region in HBase. Currently HRegionInfo also contains the HTableDescriptor (the schema). That means we store the schema n times where n is the number of regions in the table. Additionally, for every region of the same table that the region server has open, there is a copy of the schema. Thus it is stored in memory once for each open region. If HRegionInfo merely contained the table name the HTableDescriptor could be stored in a separate file and easily found. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-451) Remove HTableDescriptor from HRegionInfo
[ https://issues.apache.org/jira/browse/HBASE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-451: --- Assignee: Subbu M Iyer Status: Patch Available (was: Open) Here is a brief summary of changes for this JIRA: 1. HTD will be stored in HDFS in .tableinfo file under the table folder. 2. HTD removed from HRI 3. Bootstrap creates ROOT and META with out HTD in HRI. (for all fresh setup/startup. This process will also set a flag in -ROOT- called metamigrated and set it to true to indicate that the system supports new HRI's with out HTD.) 4. HRegion.createHRegion stores HTD in HDFS. (Master also stores HTD in HDFS during table creation, but it seems redundant and will be removed). 5. New HRegion instances will load the HTD of the table from HDFS. 6. AssignmentManager maintains an in-memory map of tableName, HTD that gets modified during table manipulations (such as modifyTable, AddFamily, DeleteFamily etc) as well as during user region assignments. 7. All schema operations such as modifyTable, AddFamily, DeleteFamily et al will result in appropriate changes to HTD on HDFS (.tableinfo changes). 8. For existing installations: If the system is not updated (metamigrated is False or NULL in -ROOT-) to support HRI with out HTD, then HMaster at startup (before assigning ROOT and META) does 1-3 else jumps to 4. 1. Creates new .tableInfo files for all tables in the system (based on -ROOT- and .META. scan) 2. Updates existing META.HRI (with HTD) with the new version of HRI (with out HTD). 3. Sets a flag in -ROOT- (metamigrated) to record this process and to avoid duplicate processing. 4. Continue with regular startup. 9. Appropriate changes to source and test files to work with new HRI structure with out HTD. Remove HTableDescriptor from HRegionInfo Key: HBASE-451 URL: https://issues.apache.org/jira/browse/HBASE-451 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.2.0 Reporter: Jim Kellerman Assignee: Subbu M Iyer Priority: Critical Fix For: 0.92.0 Attachments: HBASE-451_-_First_draft_support_for_removing_HTD_from_HRI1.patch There is an HRegionInfo for every region in HBase. Currently HRegionInfo also contains the HTableDescriptor (the schema). That means we store the schema n times where n is the number of regions in the table. Additionally, for every region of the same table that the region server has open, there is a copy of the schema. Thus it is stored in memory once for each open region. If HRegionInfo merely contained the table name the HTableDescriptor could be stored in a separate file and easily found. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-451) Remove HTableDescriptor from HRegionInfo
[ https://issues.apache.org/jira/browse/HBASE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-451: --- Attachment: HBASE-451_-_First_draft_support_for_removing_HTD_from_HRI1.patch Remove HTableDescriptor from HRegionInfo Key: HBASE-451 URL: https://issues.apache.org/jira/browse/HBASE-451 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.2.0 Reporter: Jim Kellerman Assignee: Subbu M Iyer Priority: Critical Fix For: 0.92.0 Attachments: HBASE-451_-_First_draft_support_for_removing_HTD_from_HRI1.patch There is an HRegionInfo for every region in HBase. Currently HRegionInfo also contains the HTableDescriptor (the schema). That means we store the schema n times where n is the number of regions in the table. Additionally, for every region of the same table that the region server has open, there is a copy of the schema. Thus it is stored in memory once for each open region. If HRegionInfo merely contained the table name the HTableDescriptor could be stored in a separate file and easily found. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-451) Remove HTableDescriptor from HRegionInfo
[ https://issues.apache.org/jira/browse/HBASE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13042955#comment-13042955 ] Subbu M Iyer commented on HBASE-451: Hi Ted, Could you please help me with posting a new review request? I am getting the following error when I try to upload my patch. Unable to parse diff revision header '(revision a13cc6087b9b0d6924c32d100b612eedf01fd3c4)' I am using Git for my version control. Do I need to do it through the RBTools? Please let me know. thanks Subbu Remove HTableDescriptor from HRegionInfo Key: HBASE-451 URL: https://issues.apache.org/jira/browse/HBASE-451 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.2.0 Reporter: Jim Kellerman Assignee: Subbu M Iyer Priority: Critical Fix For: 0.92.0 Attachments: 451_support_for_removing_HTD_from_HRI_trunk.txt, HBASE-451_-_First_draft_support_for_removing_HTD_from_HRI1.patch There is an HRegionInfo for every region in HBase. Currently HRegionInfo also contains the HTableDescriptor (the schema). That means we store the schema n times where n is the number of regions in the table. Additionally, for every region of the same table that the region server has open, there is a copy of the schema. Thus it is stored in memory once for each open region. If HRegionInfo merely contained the table name the HTableDescriptor could be stored in a separate file and easily found. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-451) Remove HTableDescriptor from HRegionInfo
[ https://issues.apache.org/jira/browse/HBASE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13042978#comment-13042978 ] Subbu M Iyer commented on HBASE-451: Yes, because its a new addition. This patch has added two new files. 1. '/src/test/java/org/apache/hadoop/hbase/client/TestMetaMigration.java 2. /src/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java Please let me know how to address this. thanks Subbu Remove HTableDescriptor from HRegionInfo Key: HBASE-451 URL: https://issues.apache.org/jira/browse/HBASE-451 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.2.0 Reporter: Jim Kellerman Assignee: Subbu M Iyer Priority: Critical Fix For: 0.92.0 Attachments: 451_support_for_removing_HTD_from_HRI_trunk.txt, HBASE-451_-_First_draft_support_for_removing_HTD_from_HRI1.patch There is an HRegionInfo for every region in HBase. Currently HRegionInfo also contains the HTableDescriptor (the schema). That means we store the schema n times where n is the number of regions in the table. Additionally, for every region of the same table that the region server has open, there is a copy of the schema. Thus it is stored in memory once for each open region. If HRegionInfo merely contained the table name the HTableDescriptor could be stored in a separate file and easily found. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-451) Remove HTableDescriptor from HRegionInfo
[ https://issues.apache.org/jira/browse/HBASE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13043109#comment-13043109 ] Subbu M Iyer commented on HBASE-451: Thanks Stack for your help. I wouldn't have gotten through this blocker myself. thanks again. Remove HTableDescriptor from HRegionInfo Key: HBASE-451 URL: https://issues.apache.org/jira/browse/HBASE-451 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.2.0 Reporter: Jim Kellerman Assignee: Subbu M Iyer Priority: Critical Fix For: 0.92.0 Attachments: 451_support_for_removing_HTD_from_HRI_trunk.txt, HBASE-451_-_First_draft_support_for_removing_HTD_from_HRI1.patch There is an HRegionInfo for every region in HBase. Currently HRegionInfo also contains the HTableDescriptor (the schema). That means we store the schema n times where n is the number of regions in the table. Additionally, for every region of the same table that the region server has open, there is a copy of the schema. Thus it is stored in memory once for each open region. If HRegionInfo merely contained the table name the HTableDescriptor could be stored in a separate file and easily found. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3210) HBASE-1921 for the new master
[ https://issues.apache.org/jira/browse/HBASE-3210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022235#comment-13022235 ] Subbu M Iyer commented on HBASE-3210: - attaching the patch with corrected formatting and line breaks. HBASE-1921 for the new master - Key: HBASE-3210 URL: https://issues.apache.org/jira/browse/HBASE-3210 Project: HBase Issue Type: Improvement Reporter: Jean-Daniel Cryans Priority: Critical Fix For: 0.92.0 Attachments: HBASE-3210-When_the_Master_s_session_times_out_and_there_s_only_one,_cluster_is_wedged.patch, HBASE-3210-When_the_Master_s_session_times_out_and_there_s_only_one_cluster_is_wedged-2.patch HBASE-1921 was lost when writing the new master code. I guess it's going to be much harder to implement now, but I think it's a critical feature to have considering the reasons that brought me do it in the old master. There's already a test in TestZooKeeper which has been disabled a while ago. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3580) Remove RS from DeadServer when new instance checks in
[
https://issues.apache.org/jira/browse/HBASE-3580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022326#comment-13022326
]
Subbu M Iyer commented on HBASE-3580:
-
thanks JD for taking a closer look. I reviewed your patch and it looks good to
me.
+1 for a commit.
Remove RS from DeadServer when new instance checks in
-
Key: HBASE-3580
URL: https://issues.apache.org/jira/browse/HBASE-3580
Project: HBase
Issue Type: Improvement
Affects Versions: 0.90.0
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
Fix For: 0.90.3
Attachments:
HBASE-3580-Remove-RS-from-DeadServer-when-new-instance-checks-in.patch,
HBASE-3580-v3.patch,
HBASE-3580_-_Remove_RS_from_dead_server_when_the_RS_when_new_instance_checks_in3.patch
Keeping the servers in DeadServer until it reaches some maximum isn't super
friendly, it confuses even the best of our users:
{quote}
09:27 gbowyer Hi all, I have apparently three dead RS in my cluster, I
cannot find references to them in HDFS or in ZK, how do I still report dead RS
09:27 gbowyer also the same nodes are reported as live region servers
{quote}
The subtil startcode difference can be hard to catch, also this behavior
differs from 0.20 (so old users get confused, like I did when debugging this
problem) and it also differs from Hadoop's handling of dead DataNodes. It was
introduced in HBASE-3282.
I think this should be improved by doing like Hadoop does, removing the RS
from DeadServers when a new instance with the same hostname+port checks in.
Stack says we should do it in ServerManager.checkIsDead
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3210) HBASE-1921 for the new master
[ https://issues.apache.org/jira/browse/HBASE-3210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022329#comment-13022329 ] Subbu M Iyer commented on HBASE-3210: - JD: thanks for your comments. Will address all your call outs. HBASE-1921 for the new master - Key: HBASE-3210 URL: https://issues.apache.org/jira/browse/HBASE-3210 Project: HBase Issue Type: Improvement Reporter: Jean-Daniel Cryans Priority: Critical Fix For: 0.92.0 Attachments: HBASE-3210-When_the_Master_s_session_times_out_and_there_s_only_one,_cluster_is_wedged.patch, HBASE-3210-When_the_Master_s_session_times_out_and_there_s_only_one_cluster_is_wedged-2.patch HBASE-1921 was lost when writing the new master code. I guess it's going to be much harder to implement now, but I think it's a critical feature to have considering the reasons that brought me do it in the old master. There's already a test in TestZooKeeper which has been disabled a while ago. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3210) HBASE-1921 for the new master
[ https://issues.apache.org/jira/browse/HBASE-3210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022605#comment-13022605 ] Subbu M Iyer commented on HBASE-3210: - Attached latest patch that addresses JD's comments/suggestions. Thanks Stack for running the tests and I did it too. thanks again. HBASE-1921 for the new master - Key: HBASE-3210 URL: https://issues.apache.org/jira/browse/HBASE-3210 Project: HBase Issue Type: Improvement Reporter: Jean-Daniel Cryans Priority: Critical Fix For: 0.92.0 Attachments: HBASE-3210-When_the_Master_s_session_times_out_and_there_s_only_one,_cluster_is_wedged.patch, HBASE-3210-When_the_Master_s_session_times_out_and_there_s_only_one_cluster_is_wedged-2.patch, HBASE-3210-When_the_Master_s_session_times_out_and_there_s_only_one_cluster_is_wedged-3.patch HBASE-1921 was lost when writing the new master code. I guess it's going to be much harder to implement now, but I think it's a critical feature to have considering the reasons that brought me do it in the old master. There's already a test in TestZooKeeper which has been disabled a while ago. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3210) HBASE-1921 for the new master
[ https://issues.apache.org/jira/browse/HBASE-3210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-3210: Attachment: HBASE-3210-When_the_Master_s_session_times_out_and_there_s_only_one_cluster_is_wedged-3.patch HBASE-1921 for the new master - Key: HBASE-3210 URL: https://issues.apache.org/jira/browse/HBASE-3210 Project: HBase Issue Type: Improvement Reporter: Jean-Daniel Cryans Priority: Critical Fix For: 0.92.0 Attachments: HBASE-3210-When_the_Master_s_session_times_out_and_there_s_only_one,_cluster_is_wedged.patch, HBASE-3210-When_the_Master_s_session_times_out_and_there_s_only_one_cluster_is_wedged-2.patch, HBASE-3210-When_the_Master_s_session_times_out_and_there_s_only_one_cluster_is_wedged-3.patch HBASE-1921 was lost when writing the new master code. I guess it's going to be much harder to implement now, but I think it's a critical feature to have considering the reasons that brought me do it in the old master. There's already a test in TestZooKeeper which has been disabled a while ago. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3210) HBASE-1921 for the new master
[ https://issues.apache.org/jira/browse/HBASE-3210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13021921#comment-13021921 ] Subbu M Iyer commented on HBASE-3210: - Sure deal. will do it ASAP. Thanks Stack for the review. HBASE-1921 for the new master - Key: HBASE-3210 URL: https://issues.apache.org/jira/browse/HBASE-3210 Project: HBase Issue Type: Improvement Reporter: Jean-Daniel Cryans Priority: Critical Fix For: 0.92.0 Attachments: HBASE-3210-When_the_Master_s_session_times_out_and_there_s_only_one,_cluster_is_wedged.patch HBASE-1921 was lost when writing the new master code. I guess it's going to be much harder to implement now, but I think it's a critical feature to have considering the reasons that brought me do it in the old master. There's already a test in TestZooKeeper which has been disabled a while ago. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-451) Remove HTableDescriptor from HRegionInfo
[ https://issues.apache.org/jira/browse/HBASE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13016077#comment-13016077 ] Subbu M Iyer commented on HBASE-451: So i think the consensus is to 1) start with writing the descriptor to HDFS rather than ZK? 2) Also, I believe that the data format should be in JSON in the file or ZK which ever it may be? Any preference on the JSON library (Jetty util/Jersey/Jettison/jackson) to use? Remove HTableDescriptor from HRegionInfo Key: HBASE-451 URL: https://issues.apache.org/jira/browse/HBASE-451 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.2.0 Reporter: Jim Kellerman Priority: Critical Fix For: 0.92.0 There is an HRegionInfo for every region in HBase. Currently HRegionInfo also contains the HTableDescriptor (the schema). That means we store the schema n times where n is the number of regions in the table. Additionally, for every region of the same table that the region server has open, there is a copy of the schema. Thus it is stored in memory once for each open region. If HRegionInfo merely contained the table name the HTableDescriptor could be stored in a separate file and easily found. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3488) Add CellCounter to count multiple versions of rows
[
https://issues.apache.org/jira/browse/HBASE-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13016212#comment-13016212
]
Subbu M Iyer commented on HBASE-3488:
-
Ted/Stack:
thanks a lot for your valuable review comments.
I don't think I understand when you say you want to port it to 0.90.3? What
does that require exactly? will be more than happy to address that.
Add CellCounter to count multiple versions of rows
--
Key: HBASE-3488
URL: https://issues.apache.org/jira/browse/HBASE-3488
Project: HBase
Issue Type: Bug
Components: util
Affects Versions: 0.90.0
Reporter: Ted Yu
Assignee: Subbu M Iyer
Fix For: 0.92.0
Attachments:
3488-Allow_RowCounter_to_retrieve_multiple_versions_of_rows-version2.patch,
3488-Allow_RowCounter_to_retrieve_multiple_versions_of_rows.patch
Currently RowCounter only retrieves latest version for each row.
Some applications would store multiple versions for the same row.
RowCounter should accept a new parameter for the number of versions to return.
Scan object would be configured with version parameter (for scan.maxVersions).
Then the following API should be called:
{code}
public KeyValue[] raw() {
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3488) Allow RowCounter to retrieve multiple versions of rows
[
https://issues.apache.org/jira/browse/HBASE-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subbu M Iyer updated HBASE-3488:
Attachment:
3488-Allow_RowCounter_to_retrieve_multiple_versions_of_rows-version2.patch
Allow RowCounter to retrieve multiple versions of rows
--
Key: HBASE-3488
URL: https://issues.apache.org/jira/browse/HBASE-3488
Project: HBase
Issue Type: Bug
Components: util
Affects Versions: 0.90.0
Reporter: Ted Yu
Fix For: 0.92.0
Attachments:
3488-Allow_RowCounter_to_retrieve_multiple_versions_of_rows-version2.patch,
3488-Allow_RowCounter_to_retrieve_multiple_versions_of_rows.patch
Currently RowCounter only retrieves latest version for each row.
Some applications would store multiple versions for the same row.
RowCounter should accept a new parameter for the number of versions to return.
Scan object would be configured with version parameter (for scan.maxVersions).
Then the following API should be called:
{code}
public KeyValue[] raw() {
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3488) Allow RowCounter to retrieve multiple versions of rows
[
https://issues.apache.org/jira/browse/HBASE-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13015491#comment-13015491
]
Subbu M Iyer commented on HBASE-3488:
-
Submitted version 2 of the patch.
1. Fixed the comments per Stack's suggestion.
2. Added a Regex/Prefix based row filter to limit the counter to work on a
smaller subset of rows
3. Changed the Bytes.toString to Bytes.toStringBinary.
4. Parameterized the row/cf/qualifier separator string.
Allow RowCounter to retrieve multiple versions of rows
--
Key: HBASE-3488
URL: https://issues.apache.org/jira/browse/HBASE-3488
Project: HBase
Issue Type: Bug
Components: util
Affects Versions: 0.90.0
Reporter: Ted Yu
Fix For: 0.92.0
Attachments:
3488-Allow_RowCounter_to_retrieve_multiple_versions_of_rows-version2.patch,
3488-Allow_RowCounter_to_retrieve_multiple_versions_of_rows.patch
Currently RowCounter only retrieves latest version for each row.
Some applications would store multiple versions for the same row.
RowCounter should accept a new parameter for the number of versions to return.
Scan object would be configured with version parameter (for scan.maxVersions).
Then the following API should be called:
{code}
public KeyValue[] raw() {
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3488) Allow RowCounter to retrieve multiple versions of rows
[
https://issues.apache.org/jira/browse/HBASE-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13015525#comment-13015525
]
Subbu M Iyer commented on HBASE-3488:
-
1. To run the cellcounter with default row/cf/qualifier separator ':' for table
'planet'
java -cp
./conf:./hbase-0.91.0-SNAPSHOT.jar:./lib/hadoop-0.20.1-core.jar:./lib/commons-logging-1.1.1.jar:./lib/commons-cli-1.2.jar:./lib/zookeeper-3.3.2.jar:./lib/log4j-1.2.16.jar:./lib/commons-httpclient-3.1.jar
org.apache.hadoop.hbase.mapreduce.CellCounter planet
/work/HBaseExport/cellcounter30
2. To run the cellcounter with row/cf/qualifier separator '%' for table 'planet'
java -cp
./conf:./hbase-0.91.0-SNAPSHOT.jar:./lib/hadoop-0.20.1-core.jar:./lib/commons-logging-1.1.1.jar:./lib/commons-cli-1.2.jar:./lib/zookeeper-3.3.2.jar:./lib/log4j-1.2.16.jar:./lib/commons-httpclient-3.1.jar
org.apache.hadoop.hbase.mapreduce.CellCounter planet
/work/HBaseExport/cellcounter31 %
3. To run the cellcounter with default row/cf/qualifier separator '%' for table
'planet' with a prefix filter row_55
java -cp
./conf:./hbase-0.91.0-SNAPSHOT.jar:./lib/hadoop-0.20.1-core.jar:./lib/commons-logging-1.1.1.jar:./lib/commons-cli-1.2.jar:./lib/zookeeper-3.3.2.jar:./lib/log4j-1.2.16.jar:./lib/commons-httpclient-3.1.jar
org.apache.hadoop.hbase.mapreduce.CellCounter planet
/work/HBaseExport/cellcounter31 % row_55
4. To run the cellcounter with row/cf/qualifier separator '%' for table
'planet' with a regex filter ^11
java -cp
./conf:./hbase-0.91.0-SNAPSHOT.jar:./lib/hadoop-0.20.1-core.jar:./lib/commons-logging-1.1.1.jar:./lib/commons-cli-1.2.jar:./lib/zookeeper-3.3.2.jar:./lib/log4j-1.2.16.jar:./lib/commons-httpclient-3.1.jar
org.apache.hadoop.hbase.mapreduce.CellCounter planet
/work/HBaseExport/cellcounter31 % ^11
Allow RowCounter to retrieve multiple versions of rows
--
Key: HBASE-3488
URL: https://issues.apache.org/jira/browse/HBASE-3488
Project: HBase
Issue Type: Bug
Components: util
Affects Versions: 0.90.0
Reporter: Ted Yu
Fix For: 0.92.0
Attachments:
3488-Allow_RowCounter_to_retrieve_multiple_versions_of_rows-version2.patch,
3488-Allow_RowCounter_to_retrieve_multiple_versions_of_rows.patch
Currently RowCounter only retrieves latest version for each row.
Some applications would store multiple versions for the same row.
RowCounter should accept a new parameter for the number of versions to return.
Scan object would be configured with version parameter (for scan.maxVersions).
Then the following API should be called:
{code}
public KeyValue[] raw() {
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
