I'd like to discuss Metron's installation and management. We have used Ambari for some time now, with and without an MPack for Metron (and Elasticsearch). While this mechanism has proved useful to the project, it is not without cost. This makes us an outlier among Apache projects in terms of what we support as part of our installation and management mechanism. It greatly expands the scope of responsibility for the Metron community in terms of feature coverage that is not core to the cybersecurity platform.
We are currently undertaking an upgrade ( https://issues.apache.org/jira/browse/METRON-2088) to beat some product EOL deadlines for Storm and the HDP binaries we depend on for deployment with Ambari. This upgrade is critical for the Metron community. Ambari does not prepackage any Hadoop binaries (see for example https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide), so for convenience we had been leveraging a commercial vendor's (Hortonworks) binaries for HDP to fill this gap. There are a couple issues with this. First, and most dire, is that in the recent Cloudera earnings call (Hortonworks and Cloudera have merged, if you're unaware), they have publicly stated that *binaries will no longer be freely available to the public *(see footnote 1). This means that our development environment is tightly coupled to deployment artifacts that may cease to exist, and subsequently break over night. We need to act on this immediately. Second, there is currently a bug in the latest versions of Ambari that supports HDP 3.1 and our MPack and we are still working out if there is a fix we can implement on our end to get around it - see https://issues.apache.org/jira/browse/AMBARI-25375. This is very likely a blocker for an upgrade if we cannot find a workaround (there has been one failed attempt, another is under way so fingers crossed). Many other projects simply provide binaries and basic instructions for deploying and using their products. See NiFi, for example - https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#how-to-install-and-start-nifi. The product is developed in an a flexible and extensible way, and individuals and vendors are free to expand on that basic functionality. I propose we take a similar approach and move to deprecate and replace Ambari in one fell swoop as part of our 1.0 Metron release initiative. In addition to the reasons listed above, this would also simplify our development process, ease onboarding of new contributors and committers in the community, and decrease the overall burden to the community for new features and releases. The replacement for the development environment could be a combination of Ansible and shell/Python scripts for managing laying down the Metron bits. One option that has been discussed in the past is to leverage open source versions of Docker containers for our service dependencies. Recent conversations I've had with community devs suggests this is still a popular solution. I agree. Quite a bit of Metron system management is currently duplicated between Ambari and the management UI, and we could maintain or expand on the current feature set in our UI as desired. For general Hadoop/cluster management, we would no longer provide a prepackaged option - BYOD as it were. There are numerous OSS and commercial solutions available today, and users and vendors alike would be free to choose one that works best for their specific use case(s) and customer(s). Thanks everyone. This is not a vote, but opening the floor for discussion. Best, Mike Miklavcic Metron PMC *Notes:* (1) From the Cloudera earnings call - https://www.fool.com/earnings/call-transcripts/2019/09/04/cloudera-inc-cldr-q2-2020-earnings-call-transcript.aspx "Secondly, the distribution of our compiled software, the binaries, will be > limited. Specifically, access to these binaries, as well as support > services and technical expertise will be available only with the current > subscription agreement. The binaries contain Cloudera-specific intellectual > property, the testing, securing, and integration of various Open Source > projects into an enterprise grade system that meets the requirements of our > customers. > Consistent with the Red Hat model, our binaries will no longer be freely > available to non-paying customers. These changes to licensing and > distribution will affect all subsequent versions of our current products as > well as new products, ensuring their future development by Cloudera and the > community is better protected."
