I agree with all of that, my only thinking on contrib is, if that component is not tested and has no coverage, and needs manual steps, then we may want to separate it. We would have to have some handle on full dev as well right?
On January 13, 2020 at 16:57:13, Michael Miklavcic ( michael.miklav...@gmail.com) wrote: Hey Otto, As I mentioned above, we have had this issue with other components before, e.g. mysql. I don't see a compelling reason to discontinue or push this component to contrib just yet - it's a type of enrichment that happens to require an additional manual step. Per the article ( https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/), Maxmind is not requiring purchase, merely a user to register due to GDPR and CCPA requirements. That being said, it does cause issues for the integration tests. I think we should minimally add an @Ignore annotation to the testLoadGeoIpDatabase test in MaxmindDbEnrichmentLoaderTest and provide manual instructions for running the integration test if/when someone submits a PR affecting this code. Users can still use the geolite DB by uploading it to HDFS. (Incidentally, this even provides a crude mechanism for versioning, should a user choose to use the config path that way). We separated the deployment side of the geolite DB from the consumption side https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/metron-enrichment-common/src/main/java/org/apache/metron/enrichment/adapters/maxmind/geo/GeoLiteCityDatabase.java#L135. All that needs to happen is the user would drop the file in HDFS. This had previously happened via Ambari - see here - https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py#L95. We should probably do a couple things: 1. Minimally, remove the DB download and install from Ambari - we might choose to keep around the HDFS path creation so that we have a reasonable default prepped and ready. 2. Add documentation to manually install the geolite DB (register with maxmind) OR remove it from our default demo enrichments shipped with the platform. Manual DB file loading would be managed using the CLI tool. You should be able to load it via local file URL ( https://en.wikipedia.org/wiki/File_URI_scheme) or via a custom hosted solution via the --geo_url option. See more details here - https://github.com/apache/metron/tree/master/metron-platform/metron-data-management#geolite2-loader Anything else I'm missing? Probably worth some feedback from Justin Leet on this as well. Thanks, Mike On Mon, Jan 13, 2020 at 1:59 PM Otto Fowler <ottobackwa...@gmail.com> wrote: > Hi Tom, that is true, and I think that is the only viable approach. We do > however use the database for testing during build, and we do however setup > the components that use the data base in the ’sample’ flow with the > simulated sensors for our vagrant deploy… and our contrib/docker deploy > etc. > > So, having the user download their own properly licensed version ( and > having the user responsible for the privacy law issues ) is fine, but I > think we need to talk through all the ways we are going to change the > build, what it means for testing that component ( does it move to contrib ? > ), and the default deployment to vagrant/topology. > > > > On January 13, 2020 at 13:41:01, Yerex, Tom (tom.ye...@ubc.ca) wrote: > > Hi Otto, > > Thank you for raising this in the discussion. > > It seems to me that Maxmind is proactive about providing instructions and > code to deliver updates to the local system. I can recall being surprised > that the current Metron solution seemed to do more than I expected, i.e., I > thought I would need to get Maxmind files into the local file system where > Metron would pick those up and load them into HDFS and instead Metron did > it all. > > Perhaps the approach to simplify Metron and have it load files from the > local file system into HDFS, how you get the files to the local file system > is up to you? > > > On 2020-01-13, 3:52 AM, "Otto Fowler" <ottobackwa...@gmail.com> wrote: > > https://issues.apache.org/jira/browse/METRON–2340 > > > https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/ > > Maxmind has changed the way the distribute and license the geolite2 > database that we use in our builds and distribution. > > Master build is broken, and users are having issues setting up metron ( > https://the-asf.slack.com/archives/CB7Q6AN3T/p1578556024012200) > > > We need to fix the build and figure out how we are going to move on from > this. >
