Re: Kafka error when upgrading to Metron 0.3.0
Hi Tyler, You shouldn't have to skip the unit tests as we ensure every commit runs with 100% of tests passing. There are rare instances that this is not true, but in general all tests should pass. I just pulled down latest master and re-ran the following successfully. mvn clean install -PHDP-2.5.0.0 Which branch or commit are you working from, and what command did you run for the build? Be aware that Maven will ignore misspelled profile names and instead use the default profile without warning. On Mon, Jan 9, 2017 at 10:32 AM, Tyler Moore wrote: > Seems like it's just the unit tests that are failing for pcap backend. Was > able to build successfully using new came with -DskipTests option along > with the profile argument. > > Regards, > > Tyler Moore > IT Specialist > tyler.math...@yahoo.com > 248-909-2769 > > > On Jan 9, 2017, at 02:29, Michael Miklavcic > wrote: > > > > Hi Tyler, I don't recall seeing any failures the last time I ran this. I > > will take a look. > > > > On Jan 8, 2017 9:56 PM, "Tyler Moore" wrote: > > > > Michael, > > > > I am receiving error when trying to build with HPD profile, have you had > > any problems with pcap-backend tests failing when building with hdp > profile? > > Errors and log files provided below: > > > > Results : > > > > Failed tests: > > PcapTopologyIntegrationTest.testTimestampInKey:152->testTopology:388-> > assertInOrder:542 > > null > > PcapTopologyIntegrationTest.testTimestampInPacket:135-> > testTopology:388->assertInOrder:542 > > null > > > > > > > > Tests run: 2, Failures: 2, Errors: 0, Skipped: 0 > > > > [INFO] > > > > [INFO] Reactor Summary: > > [INFO] > > [INFO] Metron . SUCCESS [ > > 0.639 s] > > [INFO] metron-analytics ... SUCCESS [ > > 0.029 s] > > [INFO] metron-maas-common . SUCCESS [ > > 8.771 s] > > [INFO] metron-platform SUCCESS [ > > 0.057 s] > > [INFO] metron-test-utilities .. SUCCESS [ > > 0.976 s] > > [INFO] metron-integration-test SUCCESS [ > > 3.888 s] > > [INFO] metron-maas-service SUCCESS > [01:19 > > min] > > [INFO] metron-common .. SUCCESS [ > > 41.718 s] > > [INFO] metron-statistics .. SUCCESS [ > > 30.713 s] > > [INFO] metron-hbase ... SUCCESS [ > > 37.286 s] > > [INFO] metron-profiler-common . SUCCESS [ > > 7.967 s] > > [INFO] metron-profiler-client . SUCCESS [ > > 54.088 s] > > [INFO] metron-profiler SUCCESS > [02:28 > > min] > > [INFO] metron-writer .. SUCCESS [ > > 7.151 s] > > [INFO] metron-enrichment .. SUCCESS [ > > 57.076 s] > > [INFO] metron-indexing SUCCESS [ > > 7.028 s] > > [INFO] metron-solr SUCCESS [ > > 36.695 s] > > [INFO] metron-pcap SUCCESS [ > > 1.123 s] > > [INFO] metron-parsers . SUCCESS > [01:58 > > min] > > [INFO] metron-pcap-backend FAILURE [ > > 49.928 s] > > [INFO] metron-data-management . SKIPPED > > [INFO] metron-api . SKIPPED > > [INFO] metron-management .. SKIPPED > > [INFO] elasticsearch-shaded ... SKIPPED > > [INFO] metron-elasticsearch ... SKIPPED > > [INFO] metron-deployment .. SKIPPED > > [INFO] Metron Ambari Management Pack .. SKIPPED > > [INFO] > > > > [INFO] BUILD FAILURE > > [INFO] > > > > [INFO] Total time: 11:31 min > > [INFO] Finished at: 2017-01-08T22:23:59-05:00 > > [INFO] Final Memory: 287M/6202M > > [INFO] > > > > [ERROR] Failed to execute goal org.apache.maven.plugins: > > maven-surefire-plugin:2.18:test (integration-tests) on project > > metron-pcap-backend: There are test failures. > > > > > > Regards, > > > > Tyler Moore > > Software Engineer > > Phone: 248-909-2769 <(248)%20909-2769> > > Email: moore.ty...@goflyball.com > > > > > > On Thu, Jan 5, 2017 at 5:21 PM, Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > >> Hey Tyler, > >> > >> Build Metron with the HDP profile to get the proper deps for this > >> -PHDP-2.5.0.0 > >> > >> Hopefully that works for you. > >> > >> Best, > >> Mike > >> > >>
[GitHub] incubator-metron pull request #414: Metron-532 Define Profile Period When Ca...
GitHub user mattf-horton opened a pull request: https://github.com/apache/incubator-metron/pull/414 Metron-532 Define Profile Period When Calling PROFILE_GET This PR builds on top of @nickwallen 's work in METRON-529, which has already been committed to master. It adds an optional argument to PROFILE_GET, allowing run-time overrides of the profiler client global configuration parameters. The primary use case is when historical profiles have been created with a different profile configuration than is currently configured, and the analyst needing to access them does not want to change the global client configuration so as not to disrupt the work of other analysts working with current profiles. This PR is essentially similar to the former proposed PR-345, except that the signature of the PROFILE_GET function has been changed to make `groups` be a List instead of a "varargs"-style sequence, so the new `config_overrides` Map argument can come after it. This avoids a really hacky over-indulgence in polymorphism, and is consistent with discussion in PR-395. I'm opening a new PR, rather than reopening the old one, because I also had to rebase to changes in master, and the forced push destroyed the history that PR-345 pointed at. At suggestion of @cestella , scope of this work item was expanded to include checking for config changes at run time, in the PROFILE_GET Stellar function. This is a much needed enhancement. I also considerably improved the documentation of PROFILE_GET as regards the `groups` argument, with input from @nickwallen . Thanks, Nick! Testing: This build has passed Travis in my fork, and should shortly be reported passed here. Since the only code changes were to the Stellar function, the corresponding junit tests are actually sufficient testing. However, I will go ahead and test them on a live single-host test platform, taking the opportunity to test the new Profiler RPMs from PR-413! I'll report the results here tomorrow. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattf-horton/incubator-metron METRON-532 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/414.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #414 commit 837b7fa3656ddc6a408d69e6438d5270d7fa6eec Author: mattf-horton Date: 2017-01-04T20:09:22Z METRON-532 Define Profile Period When Calling PROFILE_GET Rebase original PR-345 to current master. commit 55a7c7aa0757bf55cfdd709063b11d7f4770525c Author: mattf-horton Date: 2017-01-06T20:33:38Z METRON-532 continued. Changed PROFILE_GET signature to make 'groups' argument a list; deprecate but still allow the former varargs-style usage. Improved documentation with input from Nick. commit f7ef2c2fd438933677b661a20ec35f8cd98cba3f Author: mattf-horton Date: 2017-01-06T23:40:12Z METRON-532 fix unit tests for new PROFILE_GET signature, and explicitly test the deprecated older form. commit 40ecab2798cca6ef96b73e543c2161a64a943f2f Author: mattf-horton Date: 2017-01-09T19:31:32Z Update metron-platform/metron-common/README.md to match new Stellar doc standard, and tweak the description a little more. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: [DISCUSS] Hosting Kraken maven artifacts in incubator-metron git repo
Just a heads up and hopefully not a distraction: Per https://issues.apache.org/jira/browse/METRON-583, if we ever update to a never version of GROK, many of our existing rules are going to break. I have a pull request out for this issue ( https://github.com/thekrakken/java-grok/pull/60 ) with them since November, but there has been no comment or movement on it. On January 9, 2017 at 16:01:00, Michael Miklavcic (michael.miklav...@gmail.com) wrote: This is a question primarily for the mentors. *Background* metron-common is currently depending on the openSOC github repo for hosting kraken artifacts. The original reason for this was that these jars are not hosted in Maven Central, and they were not reliably available in the Kraken repo. https://issues.apache.org/jira/browse/METRON-650 is tracking work around copying these artifacts to the Metron repo. Kraken source on openSOC - https://github.com/OpenSOC/kraken Krake maven repo on openSOC - https://github.com/OpenSOC/kraken/tree/mvn-repo *Ask* Create a new branch in incubator-metron to host any necessary maven artifacts. This branch would simply be incubator-metron/mvn-repo. This is similar to how we've hosted the asf-site. *Concerns/Questions* 1. Can we host these jars/artifacts in this manner? 2. Concerns regarding licensing? 3. Do we need to also grab and host the source code?
[DISCUSS] Hosting Kraken maven artifacts in incubator-metron git repo
This is a question primarily for the mentors. *Background* metron-common is currently depending on the openSOC github repo for hosting kraken artifacts. The original reason for this was that these jars are not hosted in Maven Central, and they were not reliably available in the Kraken repo. https://issues.apache.org/jira/browse/METRON-650 is tracking work around copying these artifacts to the Metron repo. Kraken source on openSOC - https://github.com/OpenSOC/kraken Krake maven repo on openSOC - https://github.com/OpenSOC/kraken/tree/mvn-repo *Ask* Create a new branch in incubator-metron to host any necessary maven artifacts. This branch would simply be incubator-metron/mvn-repo. This is similar to how we've hosted the asf-site. *Concerns/Questions* 1. Can we host these jars/artifacts in this manner? 2. Concerns regarding licensing? 3. Do we need to also grab and host the source code?
Re: Enrich enrichment
I think one of the big improvements vs. opensoc was the splitter joiner parallel execution model for enrichments though, switching them all to stellar as you state would be a regression. On January 9, 2017 at 14:27:29, Ryan Merriman (merrim...@gmail.com) wrote: We can already do what Carolyn suggests with Stellar enrichments, assuming we get a Stellar function created for Geo enrichments (all enrichments ideally). Would be configured like this under enrichment section in enrichment config: "stellar" : { "config" : { "geo_enriched_field" : "GEO_ENRICHMENT(field)" "enriched_geo_enriched_field": "ENRICHMENT_GET('some_other_enrichment', geo_enriched_field, 'enrichments', 'cf')" } } These statements are executed in order and can be grouped together. As Casey pointed out the limitation is that both would run in a single Storm worker. Would that be acceptable tradeoff? Sure it would be ideal if we could execute each one in separate workers but then we would have to re-architect our topologies and our system would become much more complex. Ryan On Mon, Jan 9, 2017 at 12:59 PM, Carolyn Duby wrote: > Adding new topologies adds more processing requirements to the system. It > adds more topics (storage) and more producers and consumers to kafka > (processing). > > I think what we need is a dependency of enrichments. Maybe we need to > either derive the dependencies using the Stellar (potentially not that > easy) or allow the enrichment to specify the order or enrichment > calculations. > This will allow users to calculate more enrichment in the same topology. > > Thanks > Carolyn > > > > > Sent from my Verizon, Samsung Galaxy smartphone > > > Original message > From: Nick Allen > Date: 1/9/17 8:49 AM (GMT-08:00) > To: dev@metron.incubator.apache.org > Subject: Re: Enrich enrichment > > I agree that making it easy for the user to "enrich enrichments", as Dima > put it, to an arbitrary depth, would be extremely useful for a lot of use > cases. We've discussed the use case a little in the past in this thread > [1]. > > Re-purposing the "threat intel" phase gives us something that is feasible > today, but only to a "depth" of 2. We would also need to rename and > redocument it so that users understand how they can leverage the two > phases. This seems like a minimally viable option if we want to head down > this road. > > The other extreme might involve inferring the topology needed based on the > user's configuration. If the user needs 3 phases, then we build a topology > that supports 3 phases. Under the covers instead of using Flux, we would > use Storm's topology builder Java API to grok the configuration and build > the topology(ies) that the user needs. > > I am not sure if we can infer this from the configuration as it exists > today or if we would need to redefine the configuration somehow. Like I > said this is "extreme", but could give the user more expressive and > intuitive options. > > > > > --- > [1] > http://mail-archives.apache.org/mod_mbox/incubator-metron- > dev/201610.mbox/%3CCAHSJ8NwJUiyp3YO6NVE4tfLoSSk > Oc6QG%2BMsAJSSDu%2B-wfct_vw%40mail.gmail.com%3E > > > > On Mon, Jan 9, 2017 at 10:56 AM, Casey Stella wrote: > > > I think that would be a good feature to add to have arbitrary number of > > phases, though it might be tricky to code (the way I envisioned it would > > involve a loop in storm, which is possible[1]), might have unintended > > consequences to guarantees (e.g. updating enrichments might not be able > to > > be applied in realtime) and could be tricky to reason about > > performance-wise. > > > > As it stands, the number of phases is a consequence of the topology > > itself. We do not currently have an architecture which would allow an > > arbitrary number of phases without changing the flux file itself. What > you > > can do, though, in a stellar enrichment is stack enrichments (e.g. depend > > on previous enrichments) because it's just a list of stellar statements. > > The consequence, of course, is that these statements get run within the > > same worker, which is unfortunate, but may be a stopgap workaround. > > > > *1. https://groups.google.com/forum/#!topic/storm-user/EjN1hU58Q_8 > > > > On Mon, Jan 9, 2017 at 10:48 AM, Otto Fowler > > wrote: > > > > > Maybe the naming of the phases is misleading? What if you could set up > > an > > > arbitrary number of stages, with defaults? > > > > > > > > > On January 8, 2017 at 16:25:01, Casey Stella (ceste...@gmail.com) > wrote: > > > > > > You could do the geo enrichment normally and do a stellar hbase > > enrichment > > > in the threat Intel phase. > > > > > > On Sun, Jan 8, 2017 at 16:22 Ryan Merriman > wrote: > > > > > > > Hbase enrichments and geo enrichments are done in parallel so I would > > > not > > > > expect this to work. You could do the Hbase enrichment as a threat > > Intel > > > > enrichment and that should work because enrichments and threat Intel > > are > > > > done in series. > > > > > > > > >
RE: Enrich enrichment
Dima - does this solve the enrichment you are trying to add? Thanks Carolyn Sent from my Verizon, Samsung Galaxy smartphone Original message From: Ryan Merriman Date: 1/9/17 11:27 AM (GMT-08:00) To: dev@metron.incubator.apache.org Subject: Re: Enrich enrichment We can already do what Carolyn suggests with Stellar enrichments, assuming we get a Stellar function created for Geo enrichments (all enrichments ideally). Would be configured like this under enrichment section in enrichment config: "stellar" : { "config" : { "geo_enriched_field" : "GEO_ENRICHMENT(field)" "enriched_geo_enriched_field": "ENRICHMENT_GET('some_other_enrichment', geo_enriched_field, 'enrichments', 'cf')" } } These statements are executed in order and can be grouped together. As Casey pointed out the limitation is that both would run in a single Storm worker. Would that be acceptable tradeoff? Sure it would be ideal if we could execute each one in separate workers but then we would have to re-architect our topologies and our system would become much more complex. Ryan On Mon, Jan 9, 2017 at 12:59 PM, Carolyn Duby wrote: > Adding new topologies adds more processing requirements to the system. It > adds more topics (storage) and more producers and consumers to kafka > (processing). > > I think what we need is a dependency of enrichments. Maybe we need to > either derive the dependencies using the Stellar (potentially not that > easy) or allow the enrichment to specify the order or enrichment > calculations. > This will allow users to calculate more enrichment in the same topology. > > Thanks > Carolyn > > > > > Sent from my Verizon, Samsung Galaxy smartphone > > > Original message > From: Nick Allen > Date: 1/9/17 8:49 AM (GMT-08:00) > To: dev@metron.incubator.apache.org > Subject: Re: Enrich enrichment > > I agree that making it easy for the user to "enrich enrichments", as Dima > put it, to an arbitrary depth, would be extremely useful for a lot of use > cases. We've discussed the use case a little in the past in this thread > [1]. > > Re-purposing the "threat intel" phase gives us something that is feasible > today, but only to a "depth" of 2. We would also need to rename and > redocument it so that users understand how they can leverage the two > phases. This seems like a minimally viable option if we want to head down > this road. > > The other extreme might involve inferring the topology needed based on the > user's configuration. If the user needs 3 phases, then we build a topology > that supports 3 phases. Under the covers instead of using Flux, we would > use Storm's topology builder Java API to grok the configuration and build > the topology(ies) that the user needs. > > I am not sure if we can infer this from the configuration as it exists > today or if we would need to redefine the configuration somehow. Like I > said this is "extreme", but could give the user more expressive and > intuitive options. > > > > > --- > [1] > http://mail-archives.apache.org/mod_mbox/incubator-metron- > dev/201610.mbox/%3CCAHSJ8NwJUiyp3YO6NVE4tfLoSSk > Oc6QG%2BMsAJSSDu%2B-wfct_vw%40mail.gmail.com%3E > > > > On Mon, Jan 9, 2017 at 10:56 AM, Casey Stella wrote: > > > I think that would be a good feature to add to have arbitrary number of > > phases, though it might be tricky to code (the way I envisioned it would > > involve a loop in storm, which is possible[1]), might have unintended > > consequences to guarantees (e.g. updating enrichments might not be able > to > > be applied in realtime) and could be tricky to reason about > > performance-wise. > > > > As it stands, the number of phases is a consequence of the topology > > itself. We do not currently have an architecture which would allow an > > arbitrary number of phases without changing the flux file itself. What > you > > can do, though, in a stellar enrichment is stack enrichments (e.g. depend > > on previous enrichments) because it's just a list of stellar statements. > > The consequence, of course, is that these statements get run within the > > same worker, which is unfortunate, but may be a stopgap workaround. > > > > *1. https://groups.google.com/forum/#!topic/storm-user/EjN1hU58Q_8 > > > > On Mon, Jan 9, 2017 at 10:48 AM, Otto Fowler > > wrote: > > > > > Maybe the naming of the phases is misleading? What if you could set up > > an > > > arbitrary number of stages, with defaults? > > > > > > > > > On January 8, 2017 at 16:25:01, Casey Stella (ceste...@gmail.com) > wrote: > > > > > > You could do the geo enrichment normally and do a stellar hbase > > enrichment > > > in the threat Intel phase. > > > > > > On Sun, Jan 8, 2017 at 16:22 Ryan Merriman > wrote: > > > > > > > Hbase enrichments and geo enrichments are done in parallel so I would > > > not > > > > expect this to work. You could do the Hbase enrichment as a threat > > Intel > > > > enrichment and that should work because enrichments and threat Intel >
Re: Enrich enrichment
We can already do what Carolyn suggests with Stellar enrichments, assuming we get a Stellar function created for Geo enrichments (all enrichments ideally). Would be configured like this under enrichment section in enrichment config: "stellar" : { "config" : { "geo_enriched_field" : "GEO_ENRICHMENT(field)" "enriched_geo_enriched_field": "ENRICHMENT_GET('some_other_enrichment', geo_enriched_field, 'enrichments', 'cf')" } } These statements are executed in order and can be grouped together. As Casey pointed out the limitation is that both would run in a single Storm worker. Would that be acceptable tradeoff? Sure it would be ideal if we could execute each one in separate workers but then we would have to re-architect our topologies and our system would become much more complex. Ryan On Mon, Jan 9, 2017 at 12:59 PM, Carolyn Duby wrote: > Adding new topologies adds more processing requirements to the system. It > adds more topics (storage) and more producers and consumers to kafka > (processing). > > I think what we need is a dependency of enrichments. Maybe we need to > either derive the dependencies using the Stellar (potentially not that > easy) or allow the enrichment to specify the order or enrichment > calculations. > This will allow users to calculate more enrichment in the same topology. > > Thanks > Carolyn > > > > > Sent from my Verizon, Samsung Galaxy smartphone > > > Original message > From: Nick Allen > Date: 1/9/17 8:49 AM (GMT-08:00) > To: dev@metron.incubator.apache.org > Subject: Re: Enrich enrichment > > I agree that making it easy for the user to "enrich enrichments", as Dima > put it, to an arbitrary depth, would be extremely useful for a lot of use > cases. We've discussed the use case a little in the past in this thread > [1]. > > Re-purposing the "threat intel" phase gives us something that is feasible > today, but only to a "depth" of 2. We would also need to rename and > redocument it so that users understand how they can leverage the two > phases. This seems like a minimally viable option if we want to head down > this road. > > The other extreme might involve inferring the topology needed based on the > user's configuration. If the user needs 3 phases, then we build a topology > that supports 3 phases. Under the covers instead of using Flux, we would > use Storm's topology builder Java API to grok the configuration and build > the topology(ies) that the user needs. > > I am not sure if we can infer this from the configuration as it exists > today or if we would need to redefine the configuration somehow. Like I > said this is "extreme", but could give the user more expressive and > intuitive options. > > > > > --- > [1] > http://mail-archives.apache.org/mod_mbox/incubator-metron- > dev/201610.mbox/%3CCAHSJ8NwJUiyp3YO6NVE4tfLoSSk > Oc6QG%2BMsAJSSDu%2B-wfct_vw%40mail.gmail.com%3E > > > > On Mon, Jan 9, 2017 at 10:56 AM, Casey Stella wrote: > > > I think that would be a good feature to add to have arbitrary number of > > phases, though it might be tricky to code (the way I envisioned it would > > involve a loop in storm, which is possible[1]), might have unintended > > consequences to guarantees (e.g. updating enrichments might not be able > to > > be applied in realtime) and could be tricky to reason about > > performance-wise. > > > > As it stands, the number of phases is a consequence of the topology > > itself. We do not currently have an architecture which would allow an > > arbitrary number of phases without changing the flux file itself. What > you > > can do, though, in a stellar enrichment is stack enrichments (e.g. depend > > on previous enrichments) because it's just a list of stellar statements. > > The consequence, of course, is that these statements get run within the > > same worker, which is unfortunate, but may be a stopgap workaround. > > > > *1. https://groups.google.com/forum/#!topic/storm-user/EjN1hU58Q_8 > > > > On Mon, Jan 9, 2017 at 10:48 AM, Otto Fowler > > wrote: > > > > > Maybe the naming of the phases is misleading? What if you could set up > > an > > > arbitrary number of stages, with defaults? > > > > > > > > > On January 8, 2017 at 16:25:01, Casey Stella (ceste...@gmail.com) > wrote: > > > > > > You could do the geo enrichment normally and do a stellar hbase > > enrichment > > > in the threat Intel phase. > > > > > > On Sun, Jan 8, 2017 at 16:22 Ryan Merriman > wrote: > > > > > > > Hbase enrichments and geo enrichments are done in parallel so I would > > > not > > > > expect this to work. You could do the Hbase enrichment as a threat > > Intel > > > > enrichment and that should work because enrichments and threat Intel > > are > > > > done in series. > > > > > > > > > > > > > > > > The ideal way would be to chain together Stellar enrichments but I > > don't > > > > think there is a geo enrichment function created yet. I think that > > > should > > > > be a Jira. I know someone is working on an update to
[GitHub] incubator-metron pull request #413: METRON-654 Create RPM Installer for Prof...
GitHub user nickwallen opened a pull request: https://github.com/apache/incubator-metron/pull/413 METRON-654 Create RPM Installer for Profiler ### [METRON-654](https://issues.apache.org/jira/browse/METRON-654) The Profiler currently must be installed using the Ansible deployment scripts. Prior to getting the Profiler integrated with the rest of the Ambari installation process, we need to package the Profiler in an RPM. ### Testing I tested this change by doing the following. (1) Build the RPMs by running the following command. ``` cd metron-deployment mvn clean package -Pbuild-rpms ``` (2) Stand-up a host running CentOS. I used the Bento base that is used for the Quick Dev environment. (3) Copy the RPMs to the host. ``` scp metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-*.rpm vagrant@node1:/tmp/ ``` (4) Install the RPMs. ``` [root@node1 ~]# rpm -ivh metron-*.rpm Preparing...### [100%] 1:metron-solr### [ 11%] 2:metron-profiler### [ 22%] 3:metron-pcap### [ 33%] 4:metron-parsers ### [ 44%] 5:metron-indexing### [ 56%] 6:metron-enrichment ### [ 67%] 7:metron-elasticsearch ### [ 78%] 8:metron-data-management ### [ 89%] 9:metron-common ### [100%] ``` (5) Validate that all of the Profiler's required files exist and are valid. ``` [root@node1 ~]# ls -l /usr/metron/0.3.0/config/profiler.properties -rw-r--r--. 1 root root 1257 Jan 3 14:59 /usr/metron/0.3.0/config/profiler.properties [root@node1 ~]# ls -l /usr/metron/0.3.0/bin/start_profiler_topology.sh -rwxr-xr-x. 1 root root 1080 Nov 22 22:36 /usr/metron/0.3.0/bin/start_profiler_topology.sh [root@node1 ~]# ls -l /usr/metron/0.3.0/flux/profiler/remote.yaml -rw-r--r--. 1 root root 3937 Jan 3 14:59 /usr/metron/0.3.0/flux/profiler/remote.yaml [root@node1 ~]# ls -l /usr/metron/0.3.0/lib/metron-profiler-0.3.0-uber.jar -rw-r--r--. 1 root root 62959781 Jan 9 18:26 /usr/metron/0.3.0/lib/metron-profiler-0.3.0-uber.jar ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/incubator-metron METRON-654 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/413.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #413 commit 615fa67a562edb6535a90f7f7314386f76da0341 Author: Nick Allen Date: 2017-01-09T19:15:08Z METRON-654 Create RPM Installer for Profiler --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
RE: Enrich enrichment
Adding new topologies adds more processing requirements to the system. It adds more topics (storage) and more producers and consumers to kafka (processing). I think what we need is a dependency of enrichments. Maybe we need to either derive the dependencies using the Stellar (potentially not that easy) or allow the enrichment to specify the order or enrichment calculations. This will allow users to calculate more enrichment in the same topology. Thanks Carolyn Sent from my Verizon, Samsung Galaxy smartphone Original message From: Nick Allen Date: 1/9/17 8:49 AM (GMT-08:00) To: dev@metron.incubator.apache.org Subject: Re: Enrich enrichment I agree that making it easy for the user to "enrich enrichments", as Dima put it, to an arbitrary depth, would be extremely useful for a lot of use cases. We've discussed the use case a little in the past in this thread [1]. Re-purposing the "threat intel" phase gives us something that is feasible today, but only to a "depth" of 2. We would also need to rename and redocument it so that users understand how they can leverage the two phases. This seems like a minimally viable option if we want to head down this road. The other extreme might involve inferring the topology needed based on the user's configuration. If the user needs 3 phases, then we build a topology that supports 3 phases. Under the covers instead of using Flux, we would use Storm's topology builder Java API to grok the configuration and build the topology(ies) that the user needs. I am not sure if we can infer this from the configuration as it exists today or if we would need to redefine the configuration somehow. Like I said this is "extreme", but could give the user more expressive and intuitive options. --- [1] http://mail-archives.apache.org/mod_mbox/incubator-metron-dev/201610.mbox/%3CCAHSJ8NwJUiyp3YO6NVE4tfLoSSkOc6QG%2BMsAJSSDu%2B-wfct_vw%40mail.gmail.com%3E On Mon, Jan 9, 2017 at 10:56 AM, Casey Stella wrote: > I think that would be a good feature to add to have arbitrary number of > phases, though it might be tricky to code (the way I envisioned it would > involve a loop in storm, which is possible[1]), might have unintended > consequences to guarantees (e.g. updating enrichments might not be able to > be applied in realtime) and could be tricky to reason about > performance-wise. > > As it stands, the number of phases is a consequence of the topology > itself. We do not currently have an architecture which would allow an > arbitrary number of phases without changing the flux file itself. What you > can do, though, in a stellar enrichment is stack enrichments (e.g. depend > on previous enrichments) because it's just a list of stellar statements. > The consequence, of course, is that these statements get run within the > same worker, which is unfortunate, but may be a stopgap workaround. > > *1. https://groups.google.com/forum/#!topic/storm-user/EjN1hU58Q_8 > > On Mon, Jan 9, 2017 at 10:48 AM, Otto Fowler > wrote: > > > Maybe the naming of the phases is misleading? What if you could set up > an > > arbitrary number of stages, with defaults? > > > > > > On January 8, 2017 at 16:25:01, Casey Stella (ceste...@gmail.com) wrote: > > > > You could do the geo enrichment normally and do a stellar hbase > enrichment > > in the threat Intel phase. > > > > On Sun, Jan 8, 2017 at 16:22 Ryan Merriman wrote: > > > > > Hbase enrichments and geo enrichments are done in parallel so I would > > not > > > expect this to work. You could do the Hbase enrichment as a threat > Intel > > > enrichment and that should work because enrichments and threat Intel > are > > > done in series. > > > > > > > > > > > > The ideal way would be to chain together Stellar enrichments but I > don't > > > think there is a geo enrichment function created yet. I think that > > should > > > be a Jira. I know someone is working on an update to how we do geo > > > enrichments so I will file a follow on Jira if it's not included in the > > > scope of that work. > > > > > > > > > > > > Ryan > > > > > > > > > > > > > On Jan 8, 2017, at 2:31 PM, Dima Kovalyov > > > wrote: > > > > > > > > > > > > > > Is it possible to enrich enrichment? > > > > > > > > > > > > > > For example I have IP address, I enrich it with geo and get City > name, > > > > > > > now I want to enrich City name with city crime level (assume I have > > that > > > > > > > data). But when I do that it just does not work. I specify enrichment > > > > > > > like that: > > > > > > >> { > > > > > > >> "index" : "msexchange", > > > > > > >> "batchSize" : 5, > > > > > > >> "enrichment" : { > > > > > > >> "fieldMap" : { > > > > > > >> "geo" : [ "destination_ip", "source_ip" ], > > > > > > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip.country" ], > > > > > > >> "hbaseEnrichment" : [ "enrichments:geo:destination_ip:country" ], > > > > > > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip:country" ] > > > > > > >> }, > > > >
[GitHub] incubator-metron issue #400: METRON-636: Capture memory and cpu details as a...
Github user mattf-horton commented on the issue: https://github.com/apache/incubator-metron/pull/400 Yes, it seems almost impossible on a VM. I'm not in favor of including ifconfig output. It doesn't tell you anything about bandwidth, and it produces a lot of info that I think is of negligible value for the purposes of this work item. I'd say you've got enough here, and should proceed with review process. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Kafka error when upgrading to Metron 0.3.0
Seems like it's just the unit tests that are failing for pcap backend. Was able to build successfully using new came with -DskipTests option along with the profile argument. Regards, Tyler Moore IT Specialist tyler.math...@yahoo.com 248-909-2769 > On Jan 9, 2017, at 02:29, Michael Miklavcic > wrote: > > Hi Tyler, I don't recall seeing any failures the last time I ran this. I > will take a look. > > On Jan 8, 2017 9:56 PM, "Tyler Moore" wrote: > > Michael, > > I am receiving error when trying to build with HPD profile, have you had > any problems with pcap-backend tests failing when building with hdp profile? > Errors and log files provided below: > > Results : > > Failed tests: > > PcapTopologyIntegrationTest.testTimestampInKey:152->testTopology:388->assertInOrder:542 > null > > PcapTopologyIntegrationTest.testTimestampInPacket:135->testTopology:388->assertInOrder:542 > null > > > > Tests run: 2, Failures: 2, Errors: 0, Skipped: 0 > > [INFO] > > [INFO] Reactor Summary: > [INFO] > [INFO] Metron . SUCCESS [ > 0.639 s] > [INFO] metron-analytics ... SUCCESS [ > 0.029 s] > [INFO] metron-maas-common . SUCCESS [ > 8.771 s] > [INFO] metron-platform SUCCESS [ > 0.057 s] > [INFO] metron-test-utilities .. SUCCESS [ > 0.976 s] > [INFO] metron-integration-test SUCCESS [ > 3.888 s] > [INFO] metron-maas-service SUCCESS [01:19 > min] > [INFO] metron-common .. SUCCESS [ > 41.718 s] > [INFO] metron-statistics .. SUCCESS [ > 30.713 s] > [INFO] metron-hbase ... SUCCESS [ > 37.286 s] > [INFO] metron-profiler-common . SUCCESS [ > 7.967 s] > [INFO] metron-profiler-client . SUCCESS [ > 54.088 s] > [INFO] metron-profiler SUCCESS [02:28 > min] > [INFO] metron-writer .. SUCCESS [ > 7.151 s] > [INFO] metron-enrichment .. SUCCESS [ > 57.076 s] > [INFO] metron-indexing SUCCESS [ > 7.028 s] > [INFO] metron-solr SUCCESS [ > 36.695 s] > [INFO] metron-pcap SUCCESS [ > 1.123 s] > [INFO] metron-parsers . SUCCESS [01:58 > min] > [INFO] metron-pcap-backend FAILURE [ > 49.928 s] > [INFO] metron-data-management . SKIPPED > [INFO] metron-api . SKIPPED > [INFO] metron-management .. SKIPPED > [INFO] elasticsearch-shaded ... SKIPPED > [INFO] metron-elasticsearch ... SKIPPED > [INFO] metron-deployment .. SKIPPED > [INFO] Metron Ambari Management Pack .. SKIPPED > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 11:31 min > [INFO] Finished at: 2017-01-08T22:23:59-05:00 > [INFO] Final Memory: 287M/6202M > [INFO] > > [ERROR] Failed to execute goal org.apache.maven.plugins: > maven-surefire-plugin:2.18:test (integration-tests) on project > metron-pcap-backend: There are test failures. > > > Regards, > > Tyler Moore > Software Engineer > Phone: 248-909-2769 <(248)%20909-2769> > Email: moore.ty...@goflyball.com > > > On Thu, Jan 5, 2017 at 5:21 PM, Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> Hey Tyler, >> >> Build Metron with the HDP profile to get the proper deps for this >> -PHDP-2.5.0.0 >> >> Hopefully that works for you. >> >> Best, >> Mike >> >> >>> On Thu, Jan 5, 2017 at 3:17 PM, Tyler Moore wrote: >>> >>> Hey all, >>> >>> Wondering if there is a solution to the "Offset lags for kafka not >>> supported for older versions. Please update kafka spout to latest >> version." >>> error seen when upgrading to metron 0.3.0? >>> >>> I know it is due to kafka-storm dependency that needs updated, but what >> is >>> the best way to go about that? Is it as simple at changing the version in >>> the metron pom.xml and provisioning? >>> >>> Regards, >>> >>> Tyler Moore >>> Software Engineer >>> Phone: 248-909-2769 >>> Email: moore.ty...@goflyball.com >>> >>
Re: Enrich enrichment
I agree that making it easy for the user to "enrich enrichments", as Dima put it, to an arbitrary depth, would be extremely useful for a lot of use cases. We've discussed the use case a little in the past in this thread [1]. Re-purposing the "threat intel" phase gives us something that is feasible today, but only to a "depth" of 2. We would also need to rename and redocument it so that users understand how they can leverage the two phases. This seems like a minimally viable option if we want to head down this road. The other extreme might involve inferring the topology needed based on the user's configuration. If the user needs 3 phases, then we build a topology that supports 3 phases. Under the covers instead of using Flux, we would use Storm's topology builder Java API to grok the configuration and build the topology(ies) that the user needs. I am not sure if we can infer this from the configuration as it exists today or if we would need to redefine the configuration somehow. Like I said this is "extreme", but could give the user more expressive and intuitive options. --- [1] http://mail-archives.apache.org/mod_mbox/incubator-metron-dev/201610.mbox/%3CCAHSJ8NwJUiyp3YO6NVE4tfLoSSkOc6QG%2BMsAJSSDu%2B-wfct_vw%40mail.gmail.com%3E On Mon, Jan 9, 2017 at 10:56 AM, Casey Stella wrote: > I think that would be a good feature to add to have arbitrary number of > phases, though it might be tricky to code (the way I envisioned it would > involve a loop in storm, which is possible[1]), might have unintended > consequences to guarantees (e.g. updating enrichments might not be able to > be applied in realtime) and could be tricky to reason about > performance-wise. > > As it stands, the number of phases is a consequence of the topology > itself. We do not currently have an architecture which would allow an > arbitrary number of phases without changing the flux file itself. What you > can do, though, in a stellar enrichment is stack enrichments (e.g. depend > on previous enrichments) because it's just a list of stellar statements. > The consequence, of course, is that these statements get run within the > same worker, which is unfortunate, but may be a stopgap workaround. > > *1. https://groups.google.com/forum/#!topic/storm-user/EjN1hU58Q_8 > > On Mon, Jan 9, 2017 at 10:48 AM, Otto Fowler > wrote: > > > Maybe the naming of the phases is misleading? What if you could set up > an > > arbitrary number of stages, with defaults? > > > > > > On January 8, 2017 at 16:25:01, Casey Stella (ceste...@gmail.com) wrote: > > > > You could do the geo enrichment normally and do a stellar hbase > enrichment > > in the threat Intel phase. > > > > On Sun, Jan 8, 2017 at 16:22 Ryan Merriman wrote: > > > > > Hbase enrichments and geo enrichments are done in parallel so I would > > not > > > expect this to work. You could do the Hbase enrichment as a threat > Intel > > > enrichment and that should work because enrichments and threat Intel > are > > > done in series. > > > > > > > > > > > > The ideal way would be to chain together Stellar enrichments but I > don't > > > think there is a geo enrichment function created yet. I think that > > should > > > be a Jira. I know someone is working on an update to how we do geo > > > enrichments so I will file a follow on Jira if it's not included in the > > > scope of that work. > > > > > > > > > > > > Ryan > > > > > > > > > > > > > On Jan 8, 2017, at 2:31 PM, Dima Kovalyov > > > wrote: > > > > > > > > > > > > > > Is it possible to enrich enrichment? > > > > > > > > > > > > > > For example I have IP address, I enrich it with geo and get City > name, > > > > > > > now I want to enrich City name with city crime level (assume I have > > that > > > > > > > data). But when I do that it just does not work. I specify enrichment > > > > > > > like that: > > > > > > >> { > > > > > > >> "index" : "msexchange", > > > > > > >> "batchSize" : 5, > > > > > > >> "enrichment" : { > > > > > > >> "fieldMap" : { > > > > > > >> "geo" : [ "destination_ip", "source_ip" ], > > > > > > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip.country" ], > > > > > > >> "hbaseEnrichment" : [ "enrichments:geo:destination_ip:country" ], > > > > > > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip:country" ] > > > > > > >> }, > > > > > > >> "fieldToTypeMap" : { > > > > > > >> "enrichments.geo.destination_ip.country" : [ "city_crime_level" ], > > > > > > >> "enrichments:geo:destination_ip:country" : [ "city_crime_level" ], > > > > > > >> "enrichments.geo.destination_ip:country" : [ "city_crime_level" ] > > > > > > >> }, > > > > > > >> "config" : { } > > > > > > >> }, > > > > > > >> "threatIntel" : { > > > > > > >> "fieldMap" : { }, > > > > > > >> "fieldToTypeMap" : { }, > > > > > > >> "config" : { }, > > > > > > >> "triageConfig" : { > > > > > > >> "riskLevelRules" : { }, > > > > > > >> "aggregator" : "MAX", > > > > > > >> "aggregationConfig" : { } > > > > > > >> } > > > > > > >> }, > > > > > > >> "co
Re: Enrich enrichment
You could do that, but at the moment, I believe the indexing topology is looking for its configs in the same place in zookeeper, so that'd get in the way of that. On Mon, Jan 9, 2017 at 11:08 AM, Otto Fowler wrote: > What about having multiple instances of the enrichment topology with > different configurations? Then you would just have the last one terminate > to indexing? Or something like that. > > > On January 9, 2017 at 10:56:38, Casey Stella (ceste...@gmail.com) wrote: > > I think that would be a good feature to add to have arbitrary number of > phases, though it might be tricky to code (the way I envisioned it would > involve a loop in storm, which is possible[1]), might have unintended > consequences to guarantees (e.g. updating enrichments might not be able to > be applied in realtime) and could be tricky to reason about > performance-wise. > > As it stands, the number of phases is a consequence of the topology > itself. We do not currently have an architecture which would allow an > arbitrary number of phases without changing the flux file itself. What you > can do, though, in a stellar enrichment is stack enrichments (e.g. depend > on previous enrichments) because it's just a list of stellar statements. > The consequence, of course, is that these statements get run within the > same worker, which is unfortunate, but may be a stopgap workaround. > > *1. https://groups.google.com/forum/#!topic/storm-user/EjN1hU58Q_8 > > On Mon, Jan 9, 2017 at 10:48 AM, Otto Fowler > wrote: > >> Maybe the naming of the phases is misleading? What if you could set up >> an arbitrary number of stages, with defaults? >> >> >> On January 8, 2017 at 16:25:01, Casey Stella (ceste...@gmail.com) wrote: >> >> You could do the geo enrichment normally and do a stellar hbase enrichment >> in the threat Intel phase. >> >> On Sun, Jan 8, 2017 at 16:22 Ryan Merriman wrote: >> >> > Hbase enrichments and geo enrichments are done in parallel so I would >> not >> > expect this to work. You could do the Hbase enrichment as a threat Intel >> > enrichment and that should work because enrichments and threat Intel are >> > done in series. >> > >> > >> > >> > The ideal way would be to chain together Stellar enrichments but I don't >> > think there is a geo enrichment function created yet. I think that >> should >> > be a Jira. I know someone is working on an update to how we do geo >> > enrichments so I will file a follow on Jira if it's not included in the >> > scope of that work. >> > >> > >> > >> > Ryan >> > >> > >> > >> > > On Jan 8, 2017, at 2:31 PM, Dima Kovalyov >> > wrote: >> > >> > > >> > >> > > Is it possible to enrich enrichment? >> > >> > > >> > >> > > For example I have IP address, I enrich it with geo and get City name, >> > >> > > now I want to enrich City name with city crime level (assume I have >> that >> > >> > > data). But when I do that it just does not work. I specify enrichment >> > >> > > like that: >> > >> > >> { >> > >> > >> "index" : "msexchange", >> > >> > >> "batchSize" : 5, >> > >> > >> "enrichment" : { >> > >> > >> "fieldMap" : { >> > >> > >> "geo" : [ "destination_ip", "source_ip" ], >> > >> > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip.country" ], >> > >> > >> "hbaseEnrichment" : [ "enrichments:geo:destination_ip:country" ], >> > >> > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip:country" ] >> > >> > >> }, >> > >> > >> "fieldToTypeMap" : { >> > >> > >> "enrichments.geo.destination_ip.country" : [ "city_crime_level" ], >> > >> > >> "enrichments:geo:destination_ip:country" : [ "city_crime_level" ], >> > >> > >> "enrichments.geo.destination_ip:country" : [ "city_crime_level" ] >> > >> > >> }, >> > >> > >> "config" : { } >> > >> > >> }, >> > >> > >> "threatIntel" : { >> > >> > >> "fieldMap" : { }, >> > >> > >> "fieldToTypeMap" : { }, >> > >> > >> "config" : { }, >> > >> > >> "triageConfig" : { >> > >> > >> "riskLevelRules" : { }, >> > >> > >> "aggregator" : "MAX", >> > >> > >> "aggregationConfig" : { } >> > >> > >> } >> > >> > >> }, >> > >> > >> "configuration" : { } >> > >> > >> } >> > >> > > I tried all the ways how enrichment field can be entered just to be >> sure >> > >> > > I do not mistype it. >> > >> > > >> > >> > > - Dima >> > >> > >> >> >
Re: Enrich enrichment
What about having multiple instances of the enrichment topology with different configurations? Then you would just have the last one terminate to indexing? Or something like that. On January 9, 2017 at 10:56:38, Casey Stella (ceste...@gmail.com) wrote: I think that would be a good feature to add to have arbitrary number of phases, though it might be tricky to code (the way I envisioned it would involve a loop in storm, which is possible[1]), might have unintended consequences to guarantees (e.g. updating enrichments might not be able to be applied in realtime) and could be tricky to reason about performance-wise. As it stands, the number of phases is a consequence of the topology itself. We do not currently have an architecture which would allow an arbitrary number of phases without changing the flux file itself. What you can do, though, in a stellar enrichment is stack enrichments (e.g. depend on previous enrichments) because it's just a list of stellar statements. The consequence, of course, is that these statements get run within the same worker, which is unfortunate, but may be a stopgap workaround. *1. https://groups.google.com/forum/#!topic/storm-user/EjN1hU58Q_8 On Mon, Jan 9, 2017 at 10:48 AM, Otto Fowler wrote: > Maybe the naming of the phases is misleading? What if you could set up an > arbitrary number of stages, with defaults? > > > On January 8, 2017 at 16:25:01, Casey Stella (ceste...@gmail.com) wrote: > > You could do the geo enrichment normally and do a stellar hbase enrichment > in the threat Intel phase. > > On Sun, Jan 8, 2017 at 16:22 Ryan Merriman wrote: > > > Hbase enrichments and geo enrichments are done in parallel so I would not > > expect this to work. You could do the Hbase enrichment as a threat Intel > > enrichment and that should work because enrichments and threat Intel are > > done in series. > > > > > > > > The ideal way would be to chain together Stellar enrichments but I don't > > think there is a geo enrichment function created yet. I think that should > > be a Jira. I know someone is working on an update to how we do geo > > enrichments so I will file a follow on Jira if it's not included in the > > scope of that work. > > > > > > > > Ryan > > > > > > > > > On Jan 8, 2017, at 2:31 PM, Dima Kovalyov > > wrote: > > > > > > > > > > Is it possible to enrich enrichment? > > > > > > > > > > For example I have IP address, I enrich it with geo and get City name, > > > > > now I want to enrich City name with city crime level (assume I have > that > > > > > data). But when I do that it just does not work. I specify enrichment > > > > > like that: > > > > >> { > > > > >> "index" : "msexchange", > > > > >> "batchSize" : 5, > > > > >> "enrichment" : { > > > > >> "fieldMap" : { > > > > >> "geo" : [ "destination_ip", "source_ip" ], > > > > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip.country" ], > > > > >> "hbaseEnrichment" : [ "enrichments:geo:destination_ip:country" ], > > > > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip:country" ] > > > > >> }, > > > > >> "fieldToTypeMap" : { > > > > >> "enrichments.geo.destination_ip.country" : [ "city_crime_level" ], > > > > >> "enrichments:geo:destination_ip:country" : [ "city_crime_level" ], > > > > >> "enrichments.geo.destination_ip:country" : [ "city_crime_level" ] > > > > >> }, > > > > >> "config" : { } > > > > >> }, > > > > >> "threatIntel" : { > > > > >> "fieldMap" : { }, > > > > >> "fieldToTypeMap" : { }, > > > > >> "config" : { }, > > > > >> "triageConfig" : { > > > > >> "riskLevelRules" : { }, > > > > >> "aggregator" : "MAX", > > > > >> "aggregationConfig" : { } > > > > >> } > > > > >> }, > > > > >> "configuration" : { } > > > > >> } > > > > > I tried all the ways how enrichment field can be entered just to be > sure > > > > > I do not mistype it. > > > > > > > > > > - Dima > > > > > >
Re: Enrich enrichment
I think that would be a good feature to add to have arbitrary number of phases, though it might be tricky to code (the way I envisioned it would involve a loop in storm, which is possible[1]), might have unintended consequences to guarantees (e.g. updating enrichments might not be able to be applied in realtime) and could be tricky to reason about performance-wise. As it stands, the number of phases is a consequence of the topology itself. We do not currently have an architecture which would allow an arbitrary number of phases without changing the flux file itself. What you can do, though, in a stellar enrichment is stack enrichments (e.g. depend on previous enrichments) because it's just a list of stellar statements. The consequence, of course, is that these statements get run within the same worker, which is unfortunate, but may be a stopgap workaround. *1. https://groups.google.com/forum/#!topic/storm-user/EjN1hU58Q_8 On Mon, Jan 9, 2017 at 10:48 AM, Otto Fowler wrote: > Maybe the naming of the phases is misleading? What if you could set up an > arbitrary number of stages, with defaults? > > > On January 8, 2017 at 16:25:01, Casey Stella (ceste...@gmail.com) wrote: > > You could do the geo enrichment normally and do a stellar hbase enrichment > in the threat Intel phase. > > On Sun, Jan 8, 2017 at 16:22 Ryan Merriman wrote: > > > Hbase enrichments and geo enrichments are done in parallel so I would > not > > expect this to work. You could do the Hbase enrichment as a threat Intel > > enrichment and that should work because enrichments and threat Intel are > > done in series. > > > > > > > > The ideal way would be to chain together Stellar enrichments but I don't > > think there is a geo enrichment function created yet. I think that > should > > be a Jira. I know someone is working on an update to how we do geo > > enrichments so I will file a follow on Jira if it's not included in the > > scope of that work. > > > > > > > > Ryan > > > > > > > > > On Jan 8, 2017, at 2:31 PM, Dima Kovalyov > > wrote: > > > > > > > > > > Is it possible to enrich enrichment? > > > > > > > > > > For example I have IP address, I enrich it with geo and get City name, > > > > > now I want to enrich City name with city crime level (assume I have > that > > > > > data). But when I do that it just does not work. I specify enrichment > > > > > like that: > > > > >> { > > > > >> "index" : "msexchange", > > > > >> "batchSize" : 5, > > > > >> "enrichment" : { > > > > >> "fieldMap" : { > > > > >> "geo" : [ "destination_ip", "source_ip" ], > > > > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip.country" ], > > > > >> "hbaseEnrichment" : [ "enrichments:geo:destination_ip:country" ], > > > > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip:country" ] > > > > >> }, > > > > >> "fieldToTypeMap" : { > > > > >> "enrichments.geo.destination_ip.country" : [ "city_crime_level" ], > > > > >> "enrichments:geo:destination_ip:country" : [ "city_crime_level" ], > > > > >> "enrichments.geo.destination_ip:country" : [ "city_crime_level" ] > > > > >> }, > > > > >> "config" : { } > > > > >> }, > > > > >> "threatIntel" : { > > > > >> "fieldMap" : { }, > > > > >> "fieldToTypeMap" : { }, > > > > >> "config" : { }, > > > > >> "triageConfig" : { > > > > >> "riskLevelRules" : { }, > > > > >> "aggregator" : "MAX", > > > > >> "aggregationConfig" : { } > > > > >> } > > > > >> }, > > > > >> "configuration" : { } > > > > >> } > > > > > I tried all the ways how enrichment field can be entered just to be > sure > > > > > I do not mistype it. > > > > > > > > > > - Dima > > > > > >
Re: Enrich enrichment
Maybe the naming of the phases is misleading? What if you could set up an arbitrary number of stages, with defaults? On January 8, 2017 at 16:25:01, Casey Stella (ceste...@gmail.com) wrote: You could do the geo enrichment normally and do a stellar hbase enrichment in the threat Intel phase. On Sun, Jan 8, 2017 at 16:22 Ryan Merriman wrote: > Hbase enrichments and geo enrichments are done in parallel so I would not > expect this to work. You could do the Hbase enrichment as a threat Intel > enrichment and that should work because enrichments and threat Intel are > done in series. > > > > The ideal way would be to chain together Stellar enrichments but I don't > think there is a geo enrichment function created yet. I think that should > be a Jira. I know someone is working on an update to how we do geo > enrichments so I will file a follow on Jira if it's not included in the > scope of that work. > > > > Ryan > > > > > On Jan 8, 2017, at 2:31 PM, Dima Kovalyov > wrote: > > > > > > Is it possible to enrich enrichment? > > > > > > For example I have IP address, I enrich it with geo and get City name, > > > now I want to enrich City name with city crime level (assume I have that > > > data). But when I do that it just does not work. I specify enrichment > > > like that: > > >> { > > >> "index" : "msexchange", > > >> "batchSize" : 5, > > >> "enrichment" : { > > >> "fieldMap" : { > > >> "geo" : [ "destination_ip", "source_ip" ], > > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip.country" ], > > >> "hbaseEnrichment" : [ "enrichments:geo:destination_ip:country" ], > > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip:country" ] > > >> }, > > >> "fieldToTypeMap" : { > > >> "enrichments.geo.destination_ip.country" : [ "city_crime_level" ], > > >> "enrichments:geo:destination_ip:country" : [ "city_crime_level" ], > > >> "enrichments.geo.destination_ip:country" : [ "city_crime_level" ] > > >> }, > > >> "config" : { } > > >> }, > > >> "threatIntel" : { > > >> "fieldMap" : { }, > > >> "fieldToTypeMap" : { }, > > >> "config" : { }, > > >> "triageConfig" : { > > >> "riskLevelRules" : { }, > > >> "aggregator" : "MAX", > > >> "aggregationConfig" : { } > > >> } > > >> }, > > >> "configuration" : { } > > >> } > > > I tried all the ways how enrichment field can be entered just to be sure > > > I do not mistype it. > > > > > > - Dima > >
[GitHub] incubator-metron pull request #407: METRON-643: Stellar function documentati...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/407 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #407: METRON-643: Stellar function documentation need...
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/407 @JonZeolla I agree; I like this approach. I will commit and update the `STATS_BIN` PR to conform to the new approach. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #400: METRON-636: Capture memory and cpu details as a...
Github user anandsubbu commented on the issue: https://github.com/apache/incubator-metron/pull/400 Thanks @mattf-horton and @JonZeolla, I have now included the output of `df -h` . Here is a sample output on Mac. ``` Total System Memory = 16384 MB Processor Model: Intel(R) Core(TM) i7-4870HQ CPU Processor Speed: 2.50GHz Total Physical Processors: 4 Total cores: 4 Disk information: /dev/disk1 465Gi 338Gi 127Gi73% 88564601 33279109 73% / ``` I did attempt to check for commands to capture the NIC card / interface information, but I have to agree with @mattf-horton that it is non-trivial and can vary widely depending on environments. I considered including the output of `ifconfig` at least, however I was not sure if a user's IP address would fall under sensitive information category--which some might not want having captured/shared in logs (or would they?) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #404: METRON-624: Updated Comparison/Equality Evaluat...
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/404 @jjmeyer0 Yeah, I tend to agree. I'll make a follow-on JIRA to add an `APPROX_EQUALS`. I feel better about it now that we have an `ABS` function. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: [VOTE] Reporting Issues Wiki
+1 (binding) On Fri, Jan 6, 2017 at 7:43 PM, Kyle Richardson wrote: > 0 (binding) > > I think it's good but it just feels a little cumbersome still. > > -Kyle > > > On Jan 6, 2017, at 7:53 AM, JJ Meyer wrote: > > > > +1 (non-binding) > > > > What do you think about changing `*DO NOT FILE A JIRA, DO NOT POST ON ANY > > OTHER BOARD` *to standard case, but use a confluence warning macro? The > all > > caps just makes me feel like I'm being yelled at :) > > > >> On Thu, Jan 5, 2017 at 8:10 PM, zeo...@gmail.com > wrote: > >> > >> +1 (non-binding) > >> > >>> On Thu, Jan 5, 2017, 8:30 PM Matt Foley wrote: > >>> > >>> +1 (non-binding) > >>> > >>> One typo is still in there: > > After discussion of the issue on the JIRA if it is clear that you > >>> found a bug then you should file a JIRA > >>> should be > > After discussion of the issue on the mailing list if it is clear > >>> that you found a bug then you should file a JIRA > >>> > >>> Cheers, > >>> --Matt > >>> > >>> > >>> On 1/5/17, 2:41 PM, "James Sirota" wrote: > >>> > >>>Based on feedback from the discuss thread. > >>> > >>>Please vote +1, -1, or 0. The vote will be open for 72 hours > >>> > >>> > >>> > >>> https://cwiki.apache.org/confluence/pages/viewpage. > >> action?pageId=67635199 > >>> > >>> > >>>All “I have found a bug” issues are considered developer-level > >>> issues. Please report all developer-level issues to > >>> dev@metron.incubator.apache.org. Examples of developer issues would > be: > >>>Project fails to compile or is failing unit or integration tests > >>>Project or individual components fail to install > >>>There are error messages or failures in my logs > >>>I need help with coding or extending a specific component > >>>etc... > >>>After discussion of the issue on the JIRA if it is clear that you > >>> found a bug then you should file a JIRA (unless you found a security > >>> vulnerability). Follow up on the mailing lists if you want advice with > >>> respect to workaround or a local fix. Our JIRA is located here. > >>> > >>>All “I have a problem” or "How do you use x" issues are usability > >>> issues. If you are an end-user of the product and have a comment or > >>> question then use u...@metron.incubator.apache.org. If you have a > >>> problem and a strong suspicion that you might have found a bug, please > >>> cross-reference dev@metron.incubator.apache.org as well > >>>I don't understand the UI, what does button x do? > >>>What should the output of function x be? > >>>It would be nice if I had feature x along with feature y > >>>etc... > >>> > >>>If you found a security-related issue, please report immediately to > >>> secur...@metron.incubator.apache.org. Please adhere to the following > >>> Apache policy found here. DO NOT FILE A JIRA, DO NOT POST ON ANY OTHER > >> BOARD > >>>I can get access to data I should not have access to > >>>I have privileges to do things I should not be allowed to do > >>>I found that this project is susceptible to an exploit > >>>etc... > >>> > >>>Please report issues related to the JIRA/Wiki to > >>> iss...@metron.incubator.apache.org > >>>I don't have access to create/assign JIRAs to myself > >>>I don't have visibility/access to certain JIRA featuers > >>>I can't create or view a wiki entry > >>>etc > >>> > >>> > >>>--- > >>>Thank you, > >>> > >>>James Sirota > >>>PPMC- Apache Metron (Incubating) > >>>jsirota AT apache DOT org > >>> > >>> > >>> > >>> > >>> -- > >> > >> Jon > >> > >> Sent from my mobile device > >> >