I agree that having a scripted approach for backup and restore of Metron configs should be necessary for such a large change/upgrade process. Having been through this many times in the past I can tell you that the difficulty of upgrading (whether perceived or actual) holds back adoption of the platform.
Jon Zeolla On Tue, Aug 27, 2019, 5:25 PM Simon Elliston Ball < si...@simonellistonball.com> wrote: > Not sure it’s in the scope of the project to handle the HDP upgrade as > well, I would scope it to metron config only, and point at the extensive > upgrade capability of Ambari, rather than us trying to recreate the way the > distribution works. > > Simon > > On Tue, 27 Aug 2019 at 22:23, Otto Fowler <ottobackwa...@gmail.com> wrote: > > > If anyone can think of the things that need to be backed up, please > > comment the jira. > > > > > > > > > > On August 27, 2019 at 17:07:20, Otto Fowler (ottobackwa...@gmail.com) > > wrote: > > > > Good idea METRON–2239 [blocker]. > > > > > > > > On August 27, 2019 at 16:30:13, Simon Elliston Ball ( > > si...@simonellistonball.com) wrote: > > > > You could always submit a Jira :) > > > > On Tue, 27 Aug 2019 at 21:27, Otto Fowler <ottobackwa...@gmail.com> > wrote: > > > >> You are right, that is much better than backup_metron_configs.sh. > >> > >> > >> > >> On August 27, 2019 at 16:05:38, Simon Elliston Ball ( > >> si...@simonellistonball.com) wrote: > >> > >> You can do this with zk_load_configs and Ambari’s blueprint api, so we > >> kinda already do. > >> > >> Simon > >> > >> On Tue, 27 Aug 2019 at 20:19, Otto Fowler <ottobackwa...@gmail.com> > >> wrote: > >> > >>> Maybe we need some automated method to backup configurations and > restore > >>> them. > >>> > >>> > >>> > >>> On August 27, 2019 at 14:46:58, Michael Miklavcic ( > >>> michael.miklav...@gmail.com) wrote: > >>> > >>> > Once you back up your metron configs, the same configs that worked on > >>> the previous version will continue to work on the version running on > HDP > >>> 3.x. If there is any discrepancy between the two or additional > settings > >>> will be required, those will be documented in the release notes. From > the > >>> Metron perspective, this upgrade would be no different than simply > >>> upgrading to the new Metron version. > >>> > >>> This upgrade cannot be performed the same way we've done it in the > past. > >>> A number of platform upgrades, including OS, are required: > >>> > >>> 1. Requires the OS to be updated on all nodes because there are no > >>> Centos6 RPMs provided in HDP 3.1. Must bump to Centos7. > >>> 2. The final new HBase code will not run on HDP 2.6 > >>> 3. The MPack changes for new Ambari are not backwards compatible > >>> 4. YARN and HDFS/MR are also at risk to be backwards incompatible > >>> > >>> > >>> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic < > >>> michael.miklav...@gmail.com> wrote: > >>> > >>>> Adding the dev list back into the thread (a reply-all was missed). > >>>> > >>>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <jsir...@apache.org> > >>>> wrote: > >>>> > >>>>> I agree with Simon. HDP 2.x platform is rapidly approaching EOL and > >>>>> everyone will likely need to migrate by end of year. Doing this > platform > >>>>> upgrade sooner will give everyone visibility into what Metron on HDP > 3.x > >>>>> looks like so they have time to plan and upgrade at their own pace. > >>>>> Feature-wise, the Metron application itself will be unchanged. It is > >>>>> merely the platform underneath that is changing. HDP itself can be > >>>>> upgraded per instructions here: > >>>>> > https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html > >>>>> > >>>>> Once you back up your metron configs, the same configs that worked on > >>>>> the previous version will continue to work on the version running on > HDP > >>>>> 3.x. If there is any discrepancy between the two or additional > settings > >>>>> will be required, those will be documented in the release notes. > From the > >>>>> Metron perspective, this upgrade would be no different than simply > >>>>> upgrading to the new Metron version. > >>>>> > >>>>> James > >>>>> > >>>>> > >>>>> 27.08.2019, 07:09, "Simon Elliston Ball" < > si...@simonellistonball.com > >>>>> >: > >>>>> > >>>>> Something worth noting here is that HDP 2.6.5 is quite old and > >>>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am > aware of a > >>>>> large number of users who require this upgrade ASAP, and in fact an > aware > >>>>> of zero users who wish to remain on HDP 2. > >>>>> > >>>>> Perhaps those users who want to stay on the old platform can stick > >>>>> their hands up and raise concerns, but this move will likely have to > happen > >>>>> very soon. > >>>>> > >>>>> Simon > >>>>> > >>>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ottobackwa...@gmail.com> > >>>>> wrote: > >>>>> > >>>>> Although we had the discussion, and some great ideas where passed > >>>>> around, I do not believe we came to some kind of consensus on what > 1.0 > >>>>> should look like. So that discussion would have to be picked up > again so > >>>>> that we could know where we are at, and make it an actual thing if > we were > >>>>> going to make this a 1.0 release. > >>>>> > >>>>> I believe that the issues raised in that discussion gating 1.0 are > >>>>> still largely applicable, including upgrade. > >>>>> > >>>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to > *only* > >>>>> supporting 3.1 work and releases. So every user and deployment we > currently > >>>>> have will feel real pain, have to slay real dragons to move forward > with > >>>>> metron. > >>>>> > >>>>> With regards to support for older versions, the “backward capability” > >>>>> that has been mentioned, I would not say that I have any specific > plan for > >>>>> that in mind. What I would say rather, is that I believe that we > must be > >>>>> explicit, setting expectations correctly and clearly with regards to > our > >>>>> intent while demonstrating that we have thought through the > situation. That > >>>>> discussion has not happened, at least I do not believe that the > prior dev > >>>>> thread really handles it in context. > >>>>> > >>>>> Depending on the upgrade situation for going to 3.1, it may be that a > >>>>> dual stream of releases or fixes or new features to the extent that > we can > >>>>> do it will greatly reduce the pain for folks, or make it viable to > stick > >>>>> with metron until they can upgrade. > >>>>> > >>>>> The issue of what metron *is* features wise may be another one we > >>>>> want to take up at some point. The idea being can we separate the > metron > >>>>> _integration parts from the metron core functionality such that we > can work > >>>>> on them separately and thus support multiple platforms through > >>>>> integrations/applications. Of course definition of metron’s value > beyond > >>>>> integration, and what those features and application boundaries are > would > >>>>> be necessary. > >>>>> > >>>>> > >>>>> > >>>>> On August 26, 2019 at 18:52:57, Michael Miklavcic ( > >>>>> michael.miklav...@gmail.com) wrote: > >>>>> > >>>>> Hi devs and users, > >>>>> > >>>>> Some questions were asked in the Slack channel about our ongoing > >>>>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The > original > >>>>> Hadoop upgrade discuss thread can be found here > >>>>> > https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E > >>>>> > >>>>> The major issue and impact with upgrading the Hadoop platform is that > >>>>> there are breaking changes. Code that runs on HDP 3.1 will not run > on 2.x. > >>>>> Here is a sampling of core components we depend on that we know of so > >>>>> far that are not backwards compatible: > >>>>> > >>>>> 1. The Core OS - we currently base our builds and test deployment > >>>>> off of artifacts pulled from HDP. The latest rev of HDP no longer > ships > >>>>> RPMs for Centos 6, which means we need to upgrade to Centos 7 > >>>>> 2. Ambari > >>>>> 3. HBase > >>>>> > >>>>> This differs from individual components we've upgraded in the past in > >>>>> that our code could still be deployed on the old as well as new > version of > >>>>> the component in a backwards compatible way. Based on semantic > versioning, > >>>>> I don't know if we can introduce this level of change in a point > release, > >>>>> which is the reason for kicking off this discussion. In the past, > users and > >>>>> developers in the community have suggested that they are -1 on a 1.x > >>>>> release that does not provide an upgrade > >>>>> > https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E > >>>>> . > >>>>> > >>>>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we > >>>>> still see upgrades as a gating function? The main issue is that this > has > >>>>> the potential to drag out the upgrade and further couple it with > other > >>>>> features. And with Storm 1.x being eol'ed, I'm not sure this is > something > >>>>> we can wait much longer for. I'll think on this and send out my own > >>>>> thoughts once folks have had a chance to review. > >>>>> > >>>>> Best, > >>>>> Mike Miklavcic > >>>>> Apache Metron, PMC, committer > >>>>> > >>>>> > >>>>> -- > >>>>> -- > >>>>> simon elliston ball > >>>>> @sireb > >>>>> > >>>>> > >>>>> > >>>>> ------------------- > >>>>> Thank you, > >>>>> > >>>>> James Sirota > >>>>> PMC- Apache Metron > >>>>> jsirota AT apache DOT org > >>>>> > >>>>> -- > >> -- > >> simon elliston ball > >> @sireb > >> > >> -- > > -- > > simon elliston ball > > @sireb > > > > -- > -- > simon elliston ball > @sireb >
