[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml
[ https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118275#comment-13118275 ] Jonathan Gray commented on HBASE-4522: -- Can't hbase-site import hbase-site-custom? > Make hbase-site-custom.xml override the hbase-site.xml > -- > > Key: HBASE-4522 > URL: https://issues.apache.org/jira/browse/HBASE-4522 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Liyin Tang >Priority: Minor > Fix For: 0.94.0 > > > The motivation for diff is that we want to override some config change for > any specific cluster easily by just adding the config entries in the > hbase-site-custom.xml for that cluster. This change adds the > hbase-site-custom.xml configuration file into HBaseConfiguration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml
[ https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118284#comment-13118284 ] Liyin Tang commented on HBASE-4522: --- @Jonathon: It can:) That's why I am wondering should I open source this change:) For us, hbase-site.xml works as hbase-default.xml and hbase-site-custom.xml works as hbase-site xml. That's why we need to make hbase-site-custom.xml overrides to hbase-site.xml. But in the open source trunk, we don't even have hbase-site-custom.xml at all. > Make hbase-site-custom.xml override the hbase-site.xml > -- > > Key: HBASE-4522 > URL: https://issues.apache.org/jira/browse/HBASE-4522 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Liyin Tang >Priority: Minor > Fix For: 0.94.0 > > > The motivation for diff is that we want to override some config change for > any specific cluster easily by just adding the config entries in the > hbase-site-custom.xml for that cluster. This change adds the > hbase-site-custom.xml configuration file into HBaseConfiguration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml
[ https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118300#comment-13118300 ] Mikhail Bautin commented on HBASE-4522: --- I discussed this with Kannan, and we thought that this might be good for people operating multiple HBase clusters for the same application. In that case hbase-site.xml could hold configuration for the entire application and hbase-site-custom.xml would have cluster-specific settings, e.g. the ZK quorum. > Make hbase-site-custom.xml override the hbase-site.xml > -- > > Key: HBASE-4522 > URL: https://issues.apache.org/jira/browse/HBASE-4522 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Liyin Tang >Priority: Minor > Fix For: 0.94.0 > > > The motivation for diff is that we want to override some config change for > any specific cluster easily by just adding the config entries in the > hbase-site-custom.xml for that cluster. This change adds the > hbase-site-custom.xml configuration file into HBaseConfiguration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml
[ https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118312#comment-13118312 ] Mikhail Bautin commented on HBASE-4522: --- Discussed this with Liyin. To clarify Liyin's point above: in our setup, we have multiple clusters per application, and hbase-site.xml works as the application-specific configuration, while hbase-site-custom.xml is the cluster-specific configuration within a particular application. So in our case hbase-site.xml does not hard-code any host names. The reason we cannot use hbase-default.xml for application-specific configuration is that there is already an hbase-default.xml file in the HBase jar, and overriding it would make us dependent on the relative ordering of HBase configuration directory and the HBase jar in the classpath, which is an unnecessarily brittle approach. With the hbase-site-custom.xml file, however, we get additional flexibility in configuring our HBase clusters without breaking backwards-compatibility. If the hbase-site-custom.xml file is not there, everything will function exactly as before. > Make hbase-site-custom.xml override the hbase-site.xml > -- > > Key: HBASE-4522 > URL: https://issues.apache.org/jira/browse/HBASE-4522 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Liyin Tang >Priority: Minor > Fix For: 0.94.0 > > > The motivation for diff is that we want to override some config change for > any specific cluster easily by just adding the config entries in the > hbase-site-custom.xml for that cluster. This change adds the > hbase-site-custom.xml configuration file into HBaseConfiguration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml
[ https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118329#comment-13118329 ] Todd Lipcon commented on HBASE-4522: Why not just use template-based configuration management here? This seems like an HBase feature papering over poor config management, no? > Make hbase-site-custom.xml override the hbase-site.xml > -- > > Key: HBASE-4522 > URL: https://issues.apache.org/jira/browse/HBASE-4522 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Liyin Tang >Priority: Minor > Fix For: 0.94.0 > > > The motivation for diff is that we want to override some config change for > any specific cluster easily by just adding the config entries in the > hbase-site-custom.xml for that cluster. This change adds the > hbase-site-custom.xml configuration file into HBaseConfiguration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml
[ https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118334#comment-13118334 ] Mikhail Bautin commented on HBASE-4522: --- @Todd: if I understand correctly, template-based configuration management means you generate configuration files using an external site-specific tool. As far as I see, there is no standardized tool like that so far in the HBase user community (could you please point it out to me if I'm wrong?) Our approach does not require an external template system (it only relies on copying/symlinking the right files into the conf directory) and provides a clearer separation of between application-specific and site-specific configuration. The "feature" in this JIRA is around two lines of code and does not break existing functionality while providing additional flexibility. That said, it would be very interesting to hear about alternative approaches to Hadoop/HBase configuration management for large multi-cluster deployments -- please feel free to share. > Make hbase-site-custom.xml override the hbase-site.xml > -- > > Key: HBASE-4522 > URL: https://issues.apache.org/jira/browse/HBASE-4522 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Liyin Tang >Priority: Minor > Fix For: 0.94.0 > > > The motivation for diff is that we want to override some config change for > any specific cluster easily by just adding the config entries in the > hbase-site-custom.xml for that cluster. This change adds the > hbase-site-custom.xml configuration file into HBaseConfiguration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml
[ https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118341#comment-13118341 ] Todd Lipcon commented on HBASE-4522: We do configuration management here using puppet. Other deploys I"m aware of use Chef or bcfg2. It's not that I think the feature is risky, really... it just feels like bloat and yet another thing to document. Do any other organizations find this feature useful? A more general solution that I'd support would be to support something like "hbase/conf.d" where any XML files in that directory would be sorted by name, and loaded in that order. So you could have 01-site.xml, 02-myapplication.xml, 03-my-machine-overrides.xml, etc. This is a more familiar construct for most sysadmins. > Make hbase-site-custom.xml override the hbase-site.xml > -- > > Key: HBASE-4522 > URL: https://issues.apache.org/jira/browse/HBASE-4522 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Liyin Tang >Priority: Minor > Fix For: 0.94.0 > > > The motivation for diff is that we want to override some config change for > any specific cluster easily by just adding the config entries in the > hbase-site-custom.xml for that cluster. This change adds the > hbase-site-custom.xml configuration file into HBaseConfiguration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml
[ https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118380#comment-13118380 ] Mikhail Bautin commented on HBASE-4522: --- In our case what you call 01-site.xml and 02-myapplication.xml correspond to hbase-site.xml, and 03-my-cluster-overrides.xml is what we call hbase-site-custom.xml. Right now we only need another level of indirection, so implementing the conf.d approach would probably be an overkill for now. We definitely need to keep hbase-site.xml, and I am open for suggestions for a better name for the next level of customization above hbase-site.xml than hbase-site-custom.xml. We used to include the per-cluster configuration into hbase-site.xml previously, but that created problems when we tried to override some properties in hbase-site-custom.xml that were already present in hbase-site.xml, since if there are multiple occurrences of the same key in a single configuration file, currently there is no guarantee that the earliest or the latest one will take precedence, as far as I know. This is why we moved to a multi-file approach. On the feature bloat and documentation side, we are probably looking at two lines of code and a few lines of documentation here... > Make hbase-site-custom.xml override the hbase-site.xml > -- > > Key: HBASE-4522 > URL: https://issues.apache.org/jira/browse/HBASE-4522 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Liyin Tang >Priority: Minor > Fix For: 0.94.0 > > > The motivation for diff is that we want to override some config change for > any specific cluster easily by just adding the config entries in the > hbase-site-custom.xml for that cluster. This change adds the > hbase-site-custom.xml configuration file into HBaseConfiguration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml
[ https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118401#comment-13118401 ] jirapos...@reviews.apache.org commented on HBASE-4522: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2135/ --- Review request for Paul Lindner. Summary --- The motivation for diff is that we want to override some config change for any specific cluster easily by just adding the config entries in the hbase-site-custom.xml for that cluster. This change adds the hbase-site-custom.xml configuration file into HBaseConfiguration. This addresses bug HBASE-4522. https://issues.apache.org/jira/browse/HBASE-4522 Diffs - src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java 06fd29c Diff: https://reviews.apache.org/r/2135/diff Testing --- This has been in production on multiple clusters for a few weeks. Thanks, Mikhail > Make hbase-site-custom.xml override the hbase-site.xml > -- > > Key: HBASE-4522 > URL: https://issues.apache.org/jira/browse/HBASE-4522 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Liyin Tang >Priority: Minor > Fix For: 0.94.0 > > > The motivation for diff is that we want to override some config change for > any specific cluster easily by just adding the config entries in the > hbase-site-custom.xml for that cluster. This change adds the > hbase-site-custom.xml configuration file into HBaseConfiguration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml
[ https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118405#comment-13118405 ] jirapos...@reviews.apache.org commented on HBASE-4522: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2135/ --- (Updated 2011-09-30 20:30:01.515458) Review request for hbase, Todd Lipcon and Jonathan Gray. Changes --- Changing reviewers -- sorry for spam. Summary --- The motivation for diff is that we want to override some config change for any specific cluster easily by just adding the config entries in the hbase-site-custom.xml for that cluster. This change adds the hbase-site-custom.xml configuration file into HBaseConfiguration. This addresses bug HBASE-4522. https://issues.apache.org/jira/browse/HBASE-4522 Diffs - src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java 06fd29c Diff: https://reviews.apache.org/r/2135/diff Testing --- This has been in production on multiple clusters for a few weeks. Thanks, Mikhail > Make hbase-site-custom.xml override the hbase-site.xml > -- > > Key: HBASE-4522 > URL: https://issues.apache.org/jira/browse/HBASE-4522 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Liyin Tang >Priority: Minor > Fix For: 0.94.0 > > > The motivation for diff is that we want to override some config change for > any specific cluster easily by just adding the config entries in the > hbase-site-custom.xml for that cluster. This change adds the > hbase-site-custom.xml configuration file into HBaseConfiguration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml
[ https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119048#comment-13119048 ] Mikhail Bautin commented on HBASE-4522: --- @Michael, J-D, anyone else at StumbleUpon: how do you guys do configuration management? Would there be any benefit in having a way to override the settings in hbase-site.xml with another configuration file? > Make hbase-site-custom.xml override the hbase-site.xml > -- > > Key: HBASE-4522 > URL: https://issues.apache.org/jira/browse/HBASE-4522 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Liyin Tang >Priority: Minor > Fix For: 0.94.0 > > > The motivation for diff is that we want to override some config change for > any specific cluster easily by just adding the config entries in the > hbase-site-custom.xml for that cluster. This change adds the > hbase-site-custom.xml configuration file into HBaseConfiguration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml
[ https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119063#comment-13119063 ] Jean-Daniel Cryans commented on HBASE-4522: --- Our applications use thrift and define a topology (currently just a dumb list of servers and ports) per cluster. So the dev basically does "I want to access that $table on that $cluster". It seems that even if you used the java API directly, you could have the same topology configuration file (or it could live in ZK, or whatever) and then on the fly do the proper configurations before creating the HTable. > Make hbase-site-custom.xml override the hbase-site.xml > -- > > Key: HBASE-4522 > URL: https://issues.apache.org/jira/browse/HBASE-4522 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Liyin Tang >Priority: Minor > Fix For: 0.94.0 > > > The motivation for diff is that we want to override some config change for > any specific cluster easily by just adding the config entries in the > hbase-site-custom.xml for that cluster. This change adds the > hbase-site-custom.xml configuration file into HBaseConfiguration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml
[ https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119397#comment-13119397 ] stack commented on HBASE-4522: -- @Mikhail We do Puppet at SU with per-cluster configs burned into the checked-in puppet scripts. > Make hbase-site-custom.xml override the hbase-site.xml > -- > > Key: HBASE-4522 > URL: https://issues.apache.org/jira/browse/HBASE-4522 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Liyin Tang >Priority: Minor > Fix For: 0.94.0 > > > The motivation for diff is that we want to override some config change for > any specific cluster easily by just adding the config entries in the > hbase-site-custom.xml for that cluster. This change adds the > hbase-site-custom.xml configuration file into HBaseConfiguration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml
[ https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120522#comment-13120522 ] Mikhail Bautin commented on HBASE-4522: --- OK, discarding this JIRA. Implemented as an internal patch instead. > Make hbase-site-custom.xml override the hbase-site.xml > -- > > Key: HBASE-4522 > URL: https://issues.apache.org/jira/browse/HBASE-4522 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Liyin Tang >Priority: Minor > Fix For: 0.94.0 > > > The motivation for diff is that we want to override some config change for > any specific cluster easily by just adding the config entries in the > hbase-site-custom.xml for that cluster. This change adds the > hbase-site-custom.xml configuration file into HBaseConfiguration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
