Hi devs and users, Some questions were asked in the Slack channel about our ongoing HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original Hadoop upgrade discuss thread can be found here https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
The major issue and impact with upgrading the Hadoop platform is that there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x. Here is a sampling of core components we depend on that we know of so far that are not backwards compatible: 1. The Core OS - we currently base our builds and test deployment off of artifacts pulled from HDP. The latest rev of HDP no longer ships RPMs for Centos 6, which means we need to upgrade to Centos 7 2. Ambari 3. HBase This differs from individual components we've upgraded in the past in that our code could still be deployed on the old as well as new version of the component in a backwards compatible way. Based on semantic versioning, I don't know if we can introduce this level of change in a point release, which is the reason for kicking off this discussion. In the past, users and developers in the community have suggested that they are -1 on a 1.x release that does not provide an upgrade https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E . Is there a way we can avoid a 1.x release? If we do need 1.x, do we still see upgrades as a gating function? The main issue is that this has the potential to drag out the upgrade and further couple it with other features. And with Storm 1.x being eol'ed, I'm not sure this is something we can wait much longer for. I'll think on this and send out my own thoughts once folks have had a chance to review. Best, Mike Miklavcic Apache Metron, PMC, committer
