Not sure it’s in the scope of the project to handle the HDP upgrade as well, I would scope it to metron config only, and point at the extensive upgrade capability of Ambari, rather than us trying to recreate the way the distribution works.
Simon On Tue, 27 Aug 2019 at 22:23, Otto Fowler <ottobackwa...@gmail.com> wrote: > If anyone can think of the things that need to be backed up, please > comment the jira. > > > > > On August 27, 2019 at 17:07:20, Otto Fowler (ottobackwa...@gmail.com) > wrote: > > Good idea METRON–2239 [blocker]. > > > > On August 27, 2019 at 16:30:13, Simon Elliston Ball ( > si...@simonellistonball.com) wrote: > > You could always submit a Jira :) > > On Tue, 27 Aug 2019 at 21:27, Otto Fowler <ottobackwa...@gmail.com> wrote: > >> You are right, that is much better than backup_metron_configs.sh. >> >> >> >> On August 27, 2019 at 16:05:38, Simon Elliston Ball ( >> si...@simonellistonball.com) wrote: >> >> You can do this with zk_load_configs and Ambari’s blueprint api, so we >> kinda already do. >> >> Simon >> >> On Tue, 27 Aug 2019 at 20:19, Otto Fowler <ottobackwa...@gmail.com> >> wrote: >> >>> Maybe we need some automated method to backup configurations and restore >>> them. >>> >>> >>> >>> On August 27, 2019 at 14:46:58, Michael Miklavcic ( >>> michael.miklav...@gmail.com) wrote: >>> >>> > Once you back up your metron configs, the same configs that worked on >>> the previous version will continue to work on the version running on HDP >>> 3.x. If there is any discrepancy between the two or additional settings >>> will be required, those will be documented in the release notes. From the >>> Metron perspective, this upgrade would be no different than simply >>> upgrading to the new Metron version. >>> >>> This upgrade cannot be performed the same way we've done it in the past. >>> A number of platform upgrades, including OS, are required: >>> >>> 1. Requires the OS to be updated on all nodes because there are no >>> Centos6 RPMs provided in HDP 3.1. Must bump to Centos7. >>> 2. The final new HBase code will not run on HDP 2.6 >>> 3. The MPack changes for new Ambari are not backwards compatible >>> 4. YARN and HDFS/MR are also at risk to be backwards incompatible >>> >>> >>> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic < >>> michael.miklav...@gmail.com> wrote: >>> >>>> Adding the dev list back into the thread (a reply-all was missed). >>>> >>>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <jsir...@apache.org> >>>> wrote: >>>> >>>>> I agree with Simon. HDP 2.x platform is rapidly approaching EOL and >>>>> everyone will likely need to migrate by end of year. Doing this platform >>>>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x >>>>> looks like so they have time to plan and upgrade at their own pace. >>>>> Feature-wise, the Metron application itself will be unchanged. It is >>>>> merely the platform underneath that is changing. HDP itself can be >>>>> upgraded per instructions here: >>>>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html >>>>> >>>>> Once you back up your metron configs, the same configs that worked on >>>>> the previous version will continue to work on the version running on HDP >>>>> 3.x. If there is any discrepancy between the two or additional settings >>>>> will be required, those will be documented in the release notes. From the >>>>> Metron perspective, this upgrade would be no different than simply >>>>> upgrading to the new Metron version. >>>>> >>>>> James >>>>> >>>>> >>>>> 27.08.2019, 07:09, "Simon Elliston Ball" <si...@simonellistonball.com >>>>> >: >>>>> >>>>> Something worth noting here is that HDP 2.6.5 is quite old and >>>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of >>>>> a >>>>> large number of users who require this upgrade ASAP, and in fact an aware >>>>> of zero users who wish to remain on HDP 2. >>>>> >>>>> Perhaps those users who want to stay on the old platform can stick >>>>> their hands up and raise concerns, but this move will likely have to >>>>> happen >>>>> very soon. >>>>> >>>>> Simon >>>>> >>>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ottobackwa...@gmail.com> >>>>> wrote: >>>>> >>>>> Although we had the discussion, and some great ideas where passed >>>>> around, I do not believe we came to some kind of consensus on what 1.0 >>>>> should look like. So that discussion would have to be picked up again so >>>>> that we could know where we are at, and make it an actual thing if we were >>>>> going to make this a 1.0 release. >>>>> >>>>> I believe that the issues raised in that discussion gating 1.0 are >>>>> still largely applicable, including upgrade. >>>>> >>>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only* >>>>> supporting 3.1 work and releases. So every user and deployment we >>>>> currently >>>>> have will feel real pain, have to slay real dragons to move forward with >>>>> metron. >>>>> >>>>> With regards to support for older versions, the “backward capability” >>>>> that has been mentioned, I would not say that I have any specific plan for >>>>> that in mind. What I would say rather, is that I believe that we must be >>>>> explicit, setting expectations correctly and clearly with regards to our >>>>> intent while demonstrating that we have thought through the situation. >>>>> That >>>>> discussion has not happened, at least I do not believe that the prior dev >>>>> thread really handles it in context. >>>>> >>>>> Depending on the upgrade situation for going to 3.1, it may be that a >>>>> dual stream of releases or fixes or new features to the extent that we can >>>>> do it will greatly reduce the pain for folks, or make it viable to stick >>>>> with metron until they can upgrade. >>>>> >>>>> The issue of what metron *is* features wise may be another one we >>>>> want to take up at some point. The idea being can we separate the metron >>>>> _integration parts from the metron core functionality such that we can >>>>> work >>>>> on them separately and thus support multiple platforms through >>>>> integrations/applications. Of course definition of metron’s value beyond >>>>> integration, and what those features and application boundaries are would >>>>> be necessary. >>>>> >>>>> >>>>> >>>>> On August 26, 2019 at 18:52:57, Michael Miklavcic ( >>>>> michael.miklav...@gmail.com) wrote: >>>>> >>>>> Hi devs and users, >>>>> >>>>> Some questions were asked in the Slack channel about our ongoing >>>>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original >>>>> Hadoop upgrade discuss thread can be found here >>>>> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E >>>>> >>>>> The major issue and impact with upgrading the Hadoop platform is that >>>>> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x. >>>>> Here is a sampling of core components we depend on that we know of so >>>>> far that are not backwards compatible: >>>>> >>>>> 1. The Core OS - we currently base our builds and test deployment >>>>> off of artifacts pulled from HDP. The latest rev of HDP no longer ships >>>>> RPMs for Centos 6, which means we need to upgrade to Centos 7 >>>>> 2. Ambari >>>>> 3. HBase >>>>> >>>>> This differs from individual components we've upgraded in the past in >>>>> that our code could still be deployed on the old as well as new version of >>>>> the component in a backwards compatible way. Based on semantic versioning, >>>>> I don't know if we can introduce this level of change in a point release, >>>>> which is the reason for kicking off this discussion. In the past, users >>>>> and >>>>> developers in the community have suggested that they are -1 on a 1.x >>>>> release that does not provide an upgrade >>>>> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E >>>>> . >>>>> >>>>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we >>>>> still see upgrades as a gating function? The main issue is that this has >>>>> the potential to drag out the upgrade and further couple it with other >>>>> features. And with Storm 1.x being eol'ed, I'm not sure this is something >>>>> we can wait much longer for. I'll think on this and send out my own >>>>> thoughts once folks have had a chance to review. >>>>> >>>>> Best, >>>>> Mike Miklavcic >>>>> Apache Metron, PMC, committer >>>>> >>>>> >>>>> -- >>>>> -- >>>>> simon elliston ball >>>>> @sireb >>>>> >>>>> >>>>> >>>>> ------------------- >>>>> Thank you, >>>>> >>>>> James Sirota >>>>> PMC- Apache Metron >>>>> jsirota AT apache DOT org >>>>> >>>>> -- >> -- >> simon elliston ball >> @sireb >> >> -- > -- > simon elliston ball > @sireb > > -- -- simon elliston ball @sireb
