You can do this with zk_load_configs and Ambari’s blueprint api, so we kinda already do.
Simon On Tue, 27 Aug 2019 at 20:19, Otto Fowler <ottobackwa...@gmail.com> wrote: > Maybe we need some automated method to backup configurations and restore > them. > > > > > On August 27, 2019 at 14:46:58, Michael Miklavcic ( > michael.miklav...@gmail.com) wrote: > > > Once you back up your metron configs, the same configs that worked on > the previous version will continue to work on the version running on HDP > 3.x. If there is any discrepancy between the two or additional settings > will be required, those will be documented in the release notes. From the > Metron perspective, this upgrade would be no different than simply > upgrading to the new Metron version. > > This upgrade cannot be performed the same way we've done it in the past. A > number of platform upgrades, including OS, are required: > > 1. Requires the OS to be updated on all nodes because there are no > Centos6 RPMs provided in HDP 3.1. Must bump to Centos7. > 2. The final new HBase code will not run on HDP 2.6 > 3. The MPack changes for new Ambari are not backwards compatible > 4. YARN and HDFS/MR are also at risk to be backwards incompatible > > > On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> Adding the dev list back into the thread (a reply-all was missed). >> >> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <jsir...@apache.org> wrote: >> >>> I agree with Simon. HDP 2.x platform is rapidly approaching EOL and >>> everyone will likely need to migrate by end of year. Doing this platform >>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x >>> looks like so they have time to plan and upgrade at their own pace. >>> Feature-wise, the Metron application itself will be unchanged. It is >>> merely the platform underneath that is changing. HDP itself can be >>> upgraded per instructions here: >>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html >>> >>> Once you back up your metron configs, the same configs that worked on >>> the previous version will continue to work on the version running on HDP >>> 3.x. If there is any discrepancy between the two or additional settings >>> will be required, those will be documented in the release notes. From the >>> Metron perspective, this upgrade would be no different than simply >>> upgrading to the new Metron version. >>> >>> James >>> >>> >>> 27.08.2019, 07:09, "Simon Elliston Ball" <si...@simonellistonball.com>: >>> >>> Something worth noting here is that HDP 2.6.5 is quite old and >>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a >>> large number of users who require this upgrade ASAP, and in fact an aware >>> of zero users who wish to remain on HDP 2. >>> >>> Perhaps those users who want to stay on the old platform can stick their >>> hands up and raise concerns, but this move will likely have to happen very >>> soon. >>> >>> Simon >>> >>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ottobackwa...@gmail.com> >>> wrote: >>> >>> Although we had the discussion, and some great ideas where passed >>> around, I do not believe we came to some kind of consensus on what 1.0 >>> should look like. So that discussion would have to be picked up again so >>> that we could know where we are at, and make it an actual thing if we were >>> going to make this a 1.0 release. >>> >>> I believe that the issues raised in that discussion gating 1.0 are still >>> largely applicable, including upgrade. >>> >>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only* >>> supporting 3.1 work and releases. So every user and deployment we currently >>> have will feel real pain, have to slay real dragons to move forward with >>> metron. >>> >>> With regards to support for older versions, the “backward capability” >>> that has been mentioned, I would not say that I have any specific plan for >>> that in mind. What I would say rather, is that I believe that we must be >>> explicit, setting expectations correctly and clearly with regards to our >>> intent while demonstrating that we have thought through the situation. That >>> discussion has not happened, at least I do not believe that the prior dev >>> thread really handles it in context. >>> >>> Depending on the upgrade situation for going to 3.1, it may be that a >>> dual stream of releases or fixes or new features to the extent that we can >>> do it will greatly reduce the pain for folks, or make it viable to stick >>> with metron until they can upgrade. >>> >>> The issue of what metron *is* features wise may be another one we want >>> to take up at some point. The idea being can we separate the metron >>> _integration parts from the metron core functionality such that we can work >>> on them separately and thus support multiple platforms through >>> integrations/applications. Of course definition of metron’s value beyond >>> integration, and what those features and application boundaries are would >>> be necessary. >>> >>> >>> >>> On August 26, 2019 at 18:52:57, Michael Miklavcic ( >>> michael.miklav...@gmail.com) wrote: >>> >>> Hi devs and users, >>> >>> Some questions were asked in the Slack channel about our ongoing >>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original >>> Hadoop upgrade discuss thread can be found here >>> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E >>> >>> The major issue and impact with upgrading the Hadoop platform is that >>> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x. >>> Here is a sampling of core components we depend on that we know of so >>> far that are not backwards compatible: >>> >>> 1. The Core OS - we currently base our builds and test deployment >>> off of artifacts pulled from HDP. The latest rev of HDP no longer ships >>> RPMs for Centos 6, which means we need to upgrade to Centos 7 >>> 2. Ambari >>> 3. HBase >>> >>> This differs from individual components we've upgraded in the past in >>> that our code could still be deployed on the old as well as new version of >>> the component in a backwards compatible way. Based on semantic versioning, >>> I don't know if we can introduce this level of change in a point release, >>> which is the reason for kicking off this discussion. In the past, users and >>> developers in the community have suggested that they are -1 on a 1.x >>> release that does not provide an upgrade >>> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E >>> . >>> >>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we >>> still see upgrades as a gating function? The main issue is that this has >>> the potential to drag out the upgrade and further couple it with other >>> features. And with Storm 1.x being eol'ed, I'm not sure this is something >>> we can wait much longer for. I'll think on this and send out my own >>> thoughts once folks have had a chance to review. >>> >>> Best, >>> Mike Miklavcic >>> Apache Metron, PMC, committer >>> >>> >>> -- >>> -- >>> simon elliston ball >>> @sireb >>> >>> >>> >>> ------------------- >>> Thank you, >>> >>> James Sirota >>> PMC- Apache Metron >>> jsirota AT apache DOT org >>> >>> -- -- simon elliston ball @sireb
