Re: Installation problem with Docker and processor that does not support virtualization
Hello, I didn't check the BIOS if I can enable it. Then, I see that yesterday has been updated the installation guideline for Metron 0.3.1 as follows: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548 Reading that, there is no mention to Docker. Maybe for my experiments (I do have to test some ML algo's) and improve my knowledge on this tool that installation should be enough. What do you think about? Thanks. Simone > > Il 7 giugno 2017 alle 23.32 "zeo...@gmail.com" ha > scritto: > > If your processor doesn't support virtualization right now I would suggest > looking into if it is simply disabled in your BIOS/UEFI (most processers > have supported this for 10+ years, excluding some processors of course). > Docker is integrated into the build process right now and is considered > mandatory (although you technically could work around it with some > effort). > > Assuming you are spinning up full-dev, vagrant should create a centos 6 VM > to run Metron in. > > Metron is the repository that you cloned from GitHub, downloaded from > Apache, etc. If you didn't do this, you will need to. Here is our last > release - http://metron.apache.org/documentation/#releases > > Hope that helps > > Jon > > On Wed, Jun 7, 2017, 4:00 PM wrote: > > > > > > Dear All, > > > > I'm installing Metron, following the instructions found here: > > > > > > https://github.com/apache/metron/tree/master/metron-deployment/vagrant/full-dev-platform > > > > Unfortunately, my processor does not support virtualization and I'm > > not > > able to launch Docker. > > > > Is there any workaround? > > > > I installed Vagrant on my OSX and I assumed to use Vagrant to > > create a VM > > with Ubuntu in which I would run Metron. Is it right? > > > > Another question about the instructions, I do not really understand > > where > > I get Metron. > > > > In this point: > > > >1. Deploy Metron > > > > cd metron-deployment/vagrant/full-dev-platform > > vagrant up > > > > I understood that I should have already downloaded Metron, but I > > don't. > > Where is Metron? > > > > Thank you. > > Simone > > > > -- > > > > > > Jon >
[GitHub] metron issue #580: METRON-942 [NO MERGE UNTIL METRON-777] Rest api and confi...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/580 Another question here - any parser started from the METRON-UI will not start the next time the service starts. This problem exists here as well, and should prob. be another jira --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 NOTE: I saw an error tonight in the UI that I don't quite understand about loading grok statements when I clicked on an inactive parser ( yaf ). I have to look at it tomorrow. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 I have written metron-extensions/metron-parser-extensions/ADDING_SYSTEM_PARSERS.md, and have updated the testing portion of the PR description. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r120786253
--- Diff:
bundles-lib/src/test/java/org/apache/metron/bundles/integration/components/MRComponent.java
---
@@ -0,0 +1,68 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles.integration.components;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.metron.bundles.integration.InMemoryComponent;
+
+import java.io.IOException;
+
+public class MRComponent implements InMemoryComponent {
--- End diff --
So, as mentioned above - the idea at the start was to create something that
make end up being a new shared library between NiFi and Metron down the road.
Now, I didn't hold my own feet to the fire on that, but there are some areas
where that thinking comes through. This is one. I did not want to create
dependencies between bundle-lib and any metron libs. So, where necessary I
created copies. I know this isn't great, and the original goal may just be
fantasy, but that is why.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r120785962 --- Diff: metron-extensions/metron-parser-extensions/metron-parser-bro-extension/metron-parser-bro/README.md --- @@ -0,0 +1,3 @@ +# bro Parser + +This is the bro parser --- End diff -- I believe that the bundle needs to be it's own thing, but I'm not 100% sure, since I never questioned it --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r120785888 --- Diff: metron-extensions/metron-parser-extensions/metron-parser-bro-extension/metron-parser-bro/README.md --- @@ -0,0 +1,3 @@ +# bro Parser + +This is the bro parser --- End diff -- The layout mirrors what the nifi nar archetypes produces, in that it creates a separate bundle/nar project from the code project. I added the assembly ( from metron's method ) as it's own project because it seemed more correct for me since it has to package both things. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r120785578 --- Diff: bundles-maven-plugin/README.md --- @@ -0,0 +1,89 @@ + +# Apache Metron Bundle Maven Plugin --- End diff -- Sure, I will fully admit to being hesitant to throw something into platform. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r120785493 --- Diff: bundles-maven-plugin/README.md --- @@ -0,0 +1,89 @@ + +# Apache Metron Bundle Maven Plugin --- End diff -- I don't disagree with this necessarily. A couple of thoughts as to why it is the way it is: 1. we may have archetypes that aren't extensions, so I didn't bury them there 2. at the start @mattf-horton and I talked about having bundles and the plugin actual be external to metron, with the goal being a shared system with NiFi, so I didn't fully integrate them that way. I am ok with moving them if that is what the consensus is. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r120785075
--- Diff:
metron-extensions/metron-parser-extensions/metron-parser-bro-extension/metron-parser-bro/src/main/config/zookeeper/enrichments/bro.json
---
@@ -0,0 +1,17 @@
+{
+"enrichment" : {
--- End diff --
This is seen more in METRON-942, because we actually install 3rd party
extensions. Which includes installing default configurations with the
extension configuration pushed into zookeeper.
What I would like to get to is after installing an extension, we can create
multiple instances of that extension, by cloning and editing the default
configurations that are installed with it.
I also track which parser instances come from which extensions,.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r120784845
--- Diff:
metron-extensions/metron-parser-extensions/metron-parser-bro-extension/metron-parser-bro/src/main/config/zookeeper/enrichments/bro.json
---
@@ -0,0 +1,17 @@
+{
+"enrichment" : {
--- End diff --
OK. This is important. A parser developer who is developing and
maintaining their own parsers outside the metron tree should not have to makes
changes to the metron code tree. Ever.
As an extension developer, I want my extension to be a manageable, singular
entity. The packaging should contain everything todo with my extension. I
would ask back, why when I create an extension would I **not** have all the
configurations and bits together?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r120784563 --- Diff: bundles-lib/pom.xml --- @@ -0,0 +1,179 @@ + + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> +4.0.0 + +org.apache.metron +Metron +0.4.0 + + +bundles-lib --- End diff -- In the future, we can include documentation, such that it is discoverable and loadable and other things ( like custom UI ) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r120784485 --- Diff: bundles-lib/pom.xml --- @@ -0,0 +1,179 @@ + +
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r120784388 --- Diff: bundles-lib/pom.xml --- @@ -0,0 +1,179 @@ + +
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r120784270 --- Diff: bundles-lib/pom.xml --- @@ -0,0 +1,179 @@ + + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> +4.0.0 + +org.apache.metron +Metron +0.4.0 + + +bundles-lib --- End diff -- metron-extensions/README.md has it spelled out. Basically, an extension is a 'special' form of the existing metron assemblies, which package up the configuration and patterns and uber jars. The difference ( besides parser relative pathing/target locations ) is that instead of an uber jar, an extension packages the executable code in a bundle. So - the extension is the sum of its loadable code ( bundle ) and it's configurations/patterns. At least for parsers. Each extension type will be bundle + ??? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #613: METRON-990: Clean up and organize flux properties
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/613 Still working on updating the integration tests with the new properties. Will have them fixed shortly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #613: METRON-990: Clean up and organize flux properties
GitHub user merrimanr opened a pull request: https://github.com/apache/metron/pull/613 METRON-990: Clean up and organize flux properties ## Contributor Comments This PR is mainly a refactor of the enrichment and indexing flux files along with their matching property files. The changes include: - moving important settings with hardcoded values in flux files to property files so that they are configurable through Ambari - removing old and unused properties - elasticsearch.properties file is now implemented as a jinja2 template in the mpack, matching the enrichment.properties implementation - global.json is now implemented as a jinja2 template in the mpack - properties are now organized in Ambari as separate tabs and sub sections, hopefully making them easier to find - changed a couple properties to use a dropdown widget in Ambari I wrote descriptions for new properties in a similar style as existing properties. I feel the descriptions are a little short, curious if others agree. I also stopped short of improving ALL our properties with better widgets than just a text box. I imagine people will have opinions on how to best present properties in Ambari but this is a start. This has been tested on full dev with the usual process. When reviewing, spin up full dev and navigate to the Metron service in Ambari. The Config section should look as described in the section above. The enrichment.properties and elasticsearch.properties should look much shorter and easier to read. One more thing. I added a rat exception for *.json.j2 since we already have an exception for *.json (no comments in json is the reason?). Let me know if that's wrong. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/merrimanr/incubator-metron METRON-990 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/613.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #613 commit 9cfa0181a95d0b2b21bd665530119e7b261962dc Author: merri
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 Thanks for the feedback @nickwallen. I'm finishing testing my 'how to add a new parser to metron proper' instructions at the moment. I will post them tonight and answer all your questions! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Installation problem with Docker and processor that does not support virtualization
If your processor doesn't support virtualization right now I would suggest looking into if it is simply disabled in your BIOS/UEFI (most processers have supported this for 10+ years, excluding some processors of course). Docker is integrated into the build process right now and is considered mandatory (although you technically could work around it with some effort). Assuming you are spinning up full-dev, vagrant should create a centos 6 VM to run Metron in. Metron is the repository that you cloned from GitHub, downloaded from Apache, etc. If you didn't do this, you will need to. Here is our last release - http://metron.apache.org/documentation/#releases Hope that helps Jon On Wed, Jun 7, 2017, 4:00 PM wrote: > Dear All, > > I'm installing Metron, following the instructions found here: > > > https://github.com/apache/metron/tree/master/metron-deployment/vagrant/full-dev-platform > > Unfortunately, my processor does not support virtualization and I'm not > able to launch Docker. > > Is there any workaround? > > I installed Vagrant on my OSX and I assumed to use Vagrant to create a VM > with Ubuntu in which I would run Metron. Is it right? > > Another question about the instructions, I do not really understand where > I get Metron. > > In this point: > > 2. Deploy Metron > > cd metron-deployment/vagrant/full-dev-platform > vagrant up > > I understood that I should have already downloaded Metron, but I don't. > Where is Metron? > > Thank you. > Simone > -- Jon
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r120747477 --- Diff: metron-extensions/README.md --- @@ -0,0 +1,51 @@ +# Metron Extensions --- End diff -- Your new extension mechanism seems like a core component for the platform. Should this live under `metron-platform` instead of being top-level? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r120743215 --- Diff: bundles-maven-plugin/README.md --- @@ -0,0 +1,89 @@ + +# Apache Metron Bundle Maven Plugin --- End diff -- The new top-level `metron-extensions` directory seems part of the core platform. Would it make sense for that to live under `metron-platform`? ``` /metron-platform -> /metron-extensions ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r120747187 --- Diff: bundles-maven-plugin/pom.xml --- @@ -0,0 +1,328 @@ + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> --- End diff -- The only reason that we need `bundles-lib`, `bundles-maven-plugin`, and `metron-maven-archetypes` is to implement the extension mechanism, right? Seems like it would make sense to have those live under `metron-extensions`, instead of being top-level projects. ``` /metron-extensions/bundles-lib /metron-extensions/bundles-maven-plugin /metron-extensions/metron-maven-archetypes ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r120740755
--- Diff:
bundles-lib/src/test/java/org/apache/metron/bundles/integration/components/MRComponent.java
---
@@ -0,0 +1,68 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles.integration.components;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.metron.bundles.integration.InMemoryComponent;
+
+import java.io.IOException;
+
+public class MRComponent implements InMemoryComponent {
--- End diff --
Why did we have to duplicate this from `metron-integration-test`. Was
there no way to reuse those?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r120738935 --- Diff: bundles-lib/pom.xml --- @@ -0,0 +1,179 @@ + + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> +4.0.0 + +org.apache.metron +Metron +0.4.0 + + +bundles-lib --- End diff -- What is the difference between a bundle and an extension? Is a *bundle* the terminology used in NARs? So we are effectively implementing an *extension* mechanism using *bundles*? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r120742937 --- Diff: bundles-maven-plugin/README.md --- @@ -0,0 +1,89 @@ + +# Apache Metron Bundle Maven Plugin --- End diff -- The new top-level projects; `bundles-lib`, `bundles-maven-plugin` and `metron-maven-archetypes` seem out-of-place to me. Would it make sense for these to live under `metron-extensions`? We only need this to support the extension mechanism, right? ``` /metron-extensions -> /bundles-lib -> /bundles-maven-plugin -> /metron-maven-archetypes ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r120745677 --- Diff: bundles-lib/pom.xml --- @@ -0,0 +1,179 @@ + +
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r120744702 --- Diff: metron-extensions/metron-parser-extensions/metron-parser-bro-extension/metron-parser-bro/README.md --- @@ -0,0 +1,3 @@ +# bro Parser + +This is the bro parser --- End diff -- What do each of the 3 projects for each parser do? Trying to understand why each parser extension can't just live in a single project. For example, why 3 projects for the Bro parser? ``` /metron-parser-bro-extension -> /metron-parser-bro -> /metron-parser-bro-assembly -> /metron-parser-bro-bundle ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r120746288
--- Diff:
bundles-lib/src/test/java/org/apache/metron/bundles/integration/components/MRComponent.java
---
@@ -0,0 +1,68 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles.integration.components;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.metron.bundles.integration.InMemoryComponent;
+
+import java.io.IOException;
+
+public class MRComponent implements InMemoryComponent {
--- End diff --
It seems like we have duplicated a number of classes from the
`metron-integration-test` project? Why did you have to do that? I'm sure
there was something that got in your way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r120741471
--- Diff:
metron-extensions/metron-parser-extensions/metron-parser-bro-extension/metron-parser-bro/src/main/config/zookeeper/enrichments/bro.json
---
@@ -0,0 +1,17 @@
+{
+"enrichment" : {
--- End diff --
Why package enrichments and indexing configuration with each parser? This
seems out-of-place to me. Just because I want to use the Bro parser, doesn't
mean that I would want these exact enrichments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
Installation problem with Docker and processor that does not support virtualization
Dear All, I'm installing Metron, following the instructions found here: https://github.com/apache/metron/tree/master/metron-deployment/vagrant/full-dev-platform Unfortunately, my processor does not support virtualization and I'm not able to launch Docker. Is there any workaround? I installed Vagrant on my OSX and I assumed to use Vagrant to create a VM with Ubuntu in which I would run Metron. Is it right? Another question about the instructions, I do not really understand where I get Metron. In this point: 2. Deploy Metron cd metron-deployment/vagrant/full-dev-platform vagrant up I understood that I should have already downloaded Metron, but I don't. Where is Metron? Thank you. Simone
High load on CentOS 7
Hi list, I've installed metron on CentOS 7 and when I start all services, the load is really high (as in 20+ on 4 and 8 CPUs). The same install on a CentOS 6 VM is fine... (load is 0.5 there). When I look at htop, the pid 27867 has 300%+ CPU% which is: storm 27867 328 3.2 6653844 1056312 ? Sl 11:53 8:52 /usr/jdk64/jdk1.8.0_77/bin/java -server -Xmx768m -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6700_JVM -Djava.library.path=/data1/hadoop/storm/supervisor/stormdist/enrichment-1-1496861469/resources/Linux-amd64:/data1/hadoop/storm/supervisor/stormdist/enrichment-1-1496861469/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dlogfile.name=worker.log -Dstorm.home=/usr/hdp/2.5.3.0-37/storm -Dworkers.artifacts=/var/log/storm/workers-artifacts -Dstorm.conf.file= -Dstorm.options= -Dstorm.log.dir=/var/log/storm -Djava.io.tmpdir=/data1/hadoop/storm/workers/44e84652-9940-430a-ae10-7ba5b1962a84/tmp -Dlogging.sensitivity=S3 -Dlog4j.configurationFile=/usr/hdp/2.5.3.0-37/storm/log4j2/worker.xml -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector -Dstorm.id=enrichment-1-1496861469 -Dworker.id=44e84652-9940-430a-ae10-7ba5b1962a84 -Dworker.port=6700 -cp /usr/hdp/2.5.3.0-37/storm/lib/asm-5.0.3.jar:/usr/hdp/2.5.3.0-37/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.5.3.0-37/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.5.3.0-37/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.5.3.0-37/storm/lib/objenesis-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.5.3.0-37/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.5.3.0-37/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.5.3.0-37/storm/lib/storm-core-1.0.1.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/storm/lib/storm-rename-hack-1.0.1.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/storm/lib/zookeeper.jar:/usr/hdp/2.5.3.0-37/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/current/storm-supervisor/conf:/data1/hadoop/storm/supervisor/ stormdist/enrichment-1-1496861469/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf org.apache.storm.daemon.worker enrichment-1-1496861469 9c90251f-54cc-42d7-8c4e-6f3bb2528752 6700 44e84652-9940-430a-ae10-7ba5b1962a84 Any idea what might cause this? Again, same install on CentOS 6, everything's fine...
[GitHub] metron issue #610: METRON-877 Extract core implementation and UDF support, c...
Github user mattf-horton commented on the issue: https://github.com/apache/metron/pull/610 > #468 's test plan Sounds good. I'll start working through it myself. Thanks for the suggestion. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #579: METRON-941 fix PaloAltoParser
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/579 Just the start. I got the previously existing test cases in a better shape, and @ctramnitz should be able to add on pretty easily. I do want to echo what @mattf-horton said about the quality of work and the length of the cycle. We definitely appreciate getting fixes like this, and I wanted to make sure we got tests in, not just for the quality of your work, but so that we avoid regressions in the future. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Help needed in assign a Task
Hi Raghu, this is appropriate for the ‘dev’ list rather than the ‘user’ list, since you’re interested in joining in development work – which is great, welcome! :-) To enable jira assignment, someone must add your name to the project contributor list. I’m sure our honored chairman has the karma to do this, but I think I’ve heard that any PMC member can also do it. Not being a PMC member myself, we’ll need input from others. Thanks, --Matt From: "zeo...@gmail.com" Reply-To: "u...@metron.apache.org" Date: Wednesday, June 7, 2017 at 8:02 AM To: "u...@metron.apache.org" Subject: Re: Help needed in assign a Task I'm unable to find you in the Assignee list. Jon On Wed, Jun 7, 2017 at 10:10 AM RaghuMitra Kandikonda wrote: Hi All, Can anyone help me in assigning METRON-988 to me?. My username is 'raghumitra'. -Raghu -- Jon
[GitHub] metron issue #579: METRON-941 fix PaloAltoParser
Github user mattf-horton commented on the issue: https://github.com/apache/metron/pull/579 First, a note to @ctramnitz : Christian, please be patient with the long cycle on this. It is not a reflection on your work, but rather a desire to make sure that work gets appropriate consideration. Your work is very much appreciated. We really do have a very strong bias in Apache toward test-driven development, and so we tend to ask that new work provide unit tests, even if those tests were lamentably lacking before the new work :-) Anyway, thanks to fast work by @justinleet , METRON-962 is now in, and the PaloAlto parser can be unit tested. Please rebase so it can be done. Since the one file you changed has not been touched by recent commits, the rebase should be painless. If you're not familiar with git rebase, ping me offline. If my understanding of Justin's PR is correct, you can then just layer on his bf8ce62 patch, and have a good start on testability. @justinleet is that correct? Was it your intent that this patch will fully provide the unit testing capability, or is it just the start? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: [GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
➢ Will MD files that are not README.md be pulled into site-book? Yes indeed. All md files are collected and used in the site-book. Md files named README are renamed to “index” and referenced by a friendly form of the directory name they are in. All other md files retain their name and location. To give an example, metron-deployment contains README.md and Kerberos-manual-setup.md. The LHS Table of Contents menu shows Deployment > which when selected presents the metron-deployment/index.html file (from the README), and the LHS menu expands to Deployment \/ Kerberos-manual-setup The “Kerberos-manual-setup”, if selected, shows Kerberos-manual-setup.html, from the corresponding md file. --Matt On 6/7/17, 7:01 AM, "ottobackwards" wrote: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 Will MD files that are not README.md be pulled into site-book? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #579: METRON-941 fix PaloAltoParser
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/579 https://github.com/ctramnitz/metron/pull/1 opened, if anyone cares. As noted, https://github.com/ctramnitz/metron/commit/bf8ce62cb5ed3846f288e4a7a606410ebbaf9d30 is the relevant commit. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #579: METRON-941 fix PaloAltoParser
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/579 I lied, it's a bit circular. It depends on your PR's fixes, so let's go back to the original strategy of making a PR against your branch. I'm more than happy to give up credit, since this is just moving where the test data lives and actually checking against it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #579: METRON-941 fix PaloAltoParser
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/579 Actually, I'll just open a separate ticket/PR. You'll still want to merge it in. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #579: METRON-941 fix PaloAltoParser
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/579 @ctramnitz Can you merge/rebase master into this PR? I'd like to make a PR against your branch, but it depends on METRON-962 being in. Basically I just pulled out the log files it loaded, and made it actually check the JSON field-by field (instead of just checking if results are not null) for the two test logs we already had (Traffic and threat on 60). At that point, you should have a pretty good basis for adding unit tests with the logs that you have. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 @justinleet merged in and resolved conflicts --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/530 @ottobackwards I'm good with that, Otto. The benefit doesn't outweigh the cost at this point. Thanks for clarifying. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 That being said, I fully understand that in hindsight we all could come up with ways to have done this differently --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 @nickwallen I did consider staging the PR's close to what you suggested at various points ( this is my 3rd re-write of this functionality ( Ansible + just packages, RPM + just packages, RPM + NAR/Bundle ). I was however concerned with submitting a small PR, which didn't break the PR master build/functional requirements, yet had enough context to be reviewable and worth reviewing. And writing a parallel deployment/rpm/ambari install did not seem like a good idea. My understanding was and is that each PR should be fully functional by the PR criteria. Thus I broke it up as I did between capabilities and user facing. From your breakup: 1) would have had to have been just the NAR stuff, out of context 2) **was** split out as METRON-942 in a sense. 3-4) cannot be done separately while maintaining a working metron If we were at the beginning, and had discussed the feature branch etc, I am sure we would progress that way and deal with the broken builds and have the reviewable chunks. But we are not. I will do whatever I can to address concerns and move this along, I am not sure that the effort to rework this and the dependent branches ( which would be quite a bit of work at this point ) is worth while. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user cestella commented on the issue: https://github.com/apache/metron/pull/530 Regarding creating a separate parser, I mean, you have two telemetries that are both JSON Maps, so you have 2 instances of the JSONMap parser topologies running with different names. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #610: METRON-877 Extract core implementation and UDF support, c...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/610 Ok, for testing this, I'd suggest we go through #468 's test plan. That touches every place we use Stellar atm, I think. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/530 I'm just poking my head in to say I'm also in favor of a feature branch for this work, as it will make my functional test much more straightforward. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 Thanks @cestella, I'll refactor with those steps. Although, I don't understand the create a second instance of a parser part. Can you elaborate? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/530 @ottobackwards I agree the feature branch approach is probably the best way to handle things like this. It naturally splits things into a smaller chunks, and it eliminates the need for all the end to end plumbing to be perfectly complete. What @cestella described as the smoke test is exactly what I was looking to validate. Thanks for articulating that an order of magnitude better than I was doing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/530 @ottobackwards METRON-962 is in and as you expected, the tests have to be deconflicted. I think you looked over those before, but I'm more than happy to help if you need anything. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #612: METRON-962: Configuration Based Unit Tests and Add...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/612 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user cestella commented on the issue: https://github.com/apache/metron/pull/530 So, the closest thing that I've come to as a comprehensive set of smoketests is the plan I created for #468 , but it's not nearly comprehensive enough ;) For this, I'd definitely validate: * Traditional smoketest that happens from fulldev with bro and yaf, etc. * Create a new parser and run it through * Create two instances of the same parser (one of the generic like csv or jsonmap) * Ensure the UI can spin up the squid parser Anything else that I missed? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/530 Maybe I am misunderstanding the functionality here, but I see a split looking like this. Of course, you know the work better than me, so maybe there is a slightly different way we need to tackle this. (1) PR 1: Introduce bundle/extension mechanism. We would have unit/integration tests for this functionality. Your manual test steps would describe step-by-step how to create a parser extension (meaning all the steps that your Maven plugin automates). (2) PR 2: Introduce the Maven plugin. You would have unit tests for the plugin along with a manual test plan describing how a user would go about using the plugin to create a parser extension. (3) PR 3: Migrate the parsers to use the bundle/extension mechanism. You would again have unit/integration tests. Your manual test steps would describe how to build and deploy the parsers using your extension mechanism. (4) PR 4: Updates to the Ambari MPack. This just automates some of the setup steps that were performed manually in the previous PRs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 ok - I'll add a Programming guide md or something then --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/530 @ottobackwards I believe it's all *.md files. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user cestella commented on the issue: https://github.com/apache/metron/pull/530 @ottobackwards they will be --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 Will MD files that are not README.md be pulled into site-book? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user cestella commented on the issue: https://github.com/apache/metron/pull/530 I tend to agree with @ottobackwards that this is split. For better or worse, we've preferred in this project to submit PRs to keep master fully functional. I don't know of a better split that would contribute a complete set of functionality. All that being said, you're right, going forward features like this might best be a feature branch where smaller PRs that do not necessarily leave things 100% working can be submitted against it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 Also, the PR **is** split. Between the system capability without new user facing functionality, and the user facing changes, the ability to install extensions etc by rest. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #606: METRON-980: Short circuit operations for Stellar
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/606#discussion_r120628075
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java
---
@@ -40,14 +40,22 @@
import static java.lang.String.format;
public class StellarCompiler extends StellarBaseListener {
- private static Token EXPRESSION_REFERENCE = new Token<>(null,
Object.class);
- private static Token LAMBDA_VARIABLES = new Token<>(null,
Object.class);
+ private static Token EXPRESSION_REFERENCE = new Token<>(null,
Object.class, null);
+ private static Token LAMBDA_VARIABLES = new Token<>(null,
Object.class, null);
private Expression expression;
private final ArithmeticEvaluator arithmeticEvaluator;
private final NumberLiteralEvaluator numberLiteralEvaluator;
private final ComparisonExpressionWithOperatorEvaluator
comparisonExpressionWithOperatorEvaluator;
+ public static class ShortCircuitFrame {}
+ public static class ShortCircuitOp {}
--- End diff --
Sorry, one of them can definitely be an interface. I made the change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 @justinleet I think this should have been ( and maybe can still be? ) one of the special branches that james mentioned. I don't see how something of this scope and breadth can be reviewed in non-functional pieces, much less understood. Extensions that aren't installed and running are not functional. The only thing in that line of thought that I could have cut back on is not breaking out all the extensions, like just doing asa and leaving the rest. But that would have **still** required full end to end plumbing, and would not have been that worth while as compared to just doing them all. If we want to move this to a 'whatever/epic' branch and pilot that process, I would be all for it. I will consolidate the REST API and the Management Stellar function ( METRON-942, METRON-947 ) and we can do the whole thing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/530 @cestella @mmiklavc Do we have a generic smoke test documented, e.g. from the various Storm upgrades or other component upgrades? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/530 @ottobackwards I definitely agree that we need more documentation around absolutely everything you listed. The main reason I want the outline for adding a new parser is just so I have a way to actually get a good handle on what's going on to be able to review it. It would be super helpful to even have a basic outline of how to do it. I assume @nickwallen feels the same way. For a first pass, I don't know that this needs to be absolutely point by point detailed, but at least enough for people to dig in and get started. I personally think it's reasonable to have the point by point detailed, nicely formatted version be a follow-on, but just something basic and (mostly) correct is what I personally want to see. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 @nickwallen WRT Test plan: Do we have a generic smoke test plan? If so, then I'll reference it. If not I will spell it out more clearly. There are, I think two aspects to this that I would like to detangle. From a 'how metron works' point of view, this change does not change what the system does in an observable way, it retains the status quo ante functionality. So that test plan is not different than another change that says "run up in full dev and verify that that topologies are running etc". That being said, I will list it out in more detail. I apologize for the vagueness.The other point of view, which I think you are referencing is "as a metron developer, who may have to add a new parser to metron proper". This **is** different from before, but was also not documented in the old architecture either, which is why I didn't take a stab at the dev guide in this PR. We also do not have guides for adding new ambari services, adding components to the rpm, creating new ambari scripts as well, all things I had to reverse and figure out to get this working, and we did not hold up those efforts to wait for them, so I did not think it was required. That is not to say that I don't agree such things should exist, and maybe should have been required then and be required now. I will certainly take a stab at it --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #606: METRON-980: Short circuit operations for Stellar
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/606#discussion_r120623833
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java
---
@@ -76,14 +92,80 @@ public Expression(Deque> tokenDeque) {
public Object apply(ExpressionState state) {
Deque> instanceDeque = new ArrayDeque<>();
- for(Iterator> it =
getTokenDeque().descendingIterator();it.hasNext();) {
-Token token = it.next();
-if(token.getUnderlyingType() == DeferredFunction.class) {
- DeferredFunction func = (DeferredFunction) token.getValue();
- func.apply(instanceDeque, state);
-}
-else {
- instanceDeque.push(token);
+ {
+boolean skipElse = false;
+Token token = null;
+for (Iterator> it = getTokenDeque().descendingIterator();
it.hasNext(); ) {
+ token = it.next();
+ //if we've skipped an else previously, then we need to skip the
deferred tokens associated with the else.
+ if(skipElse && token.getUnderlyingType() == ElseExpr.class) {
+while(it.hasNext()) {
+ token = it.next();
+ if(token.getUnderlyingType() == EndConditional.class) {
+break;
+ }
--- End diff --
Yep, I just added this case to the `StellarTest`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #606: METRON-980: Short circuit operations for Stellar
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/606#discussion_r120623722
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java
---
@@ -40,14 +40,22 @@
import static java.lang.String.format;
public class StellarCompiler extends StellarBaseListener {
- private static Token EXPRESSION_REFERENCE = new Token<>(null,
Object.class);
- private static Token LAMBDA_VARIABLES = new Token<>(null,
Object.class);
+ private static Token EXPRESSION_REFERENCE = new Token<>(null,
Object.class, null);
+ private static Token LAMBDA_VARIABLES = new Token<>(null,
Object.class, null);
private Expression expression;
private final ArithmeticEvaluator arithmeticEvaluator;
private final NumberLiteralEvaluator numberLiteralEvaluator;
private final ComparisonExpressionWithOperatorEvaluator
comparisonExpressionWithOperatorEvaluator;
+ public static class ShortCircuitFrame {}
+ public static class ShortCircuitOp {}
--- End diff --
We are actually instantiating them in this class. These are used to
indicate an instance of a short circuit frame on the stack. Unlike some of the
other marker classes, we actually want separate instances of these because you
could have nested frames.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/530 @nickwallen I agree in philosophy, but this has been a running PR for awhile that had plenty of dev list discussion around it before it was made. There's already follow-on work being done, and I'm worried that it's going to be an equally huge burden to split out everything at this point, but all on @ottobackwards . I think we should go ahead with this as one PR, and start a discuss thread laying out the difficulties we've had (both past and present) and make it well known that we strongly prefer PRs of large scope to come in as multiple, smaller PRs for the sake of easier, more comprehensive reviewing in the future. This may involve updating the dev guidelines to state this and that contributors should be aware that a request to split apart unwieldy PRs is a potential consequence. I don't think there is (or should be) a hard guideline, but we're going to have to adjust to avoid repeating these types of things in the future. I do however, think that we need a more formalized test plan, just for no other reason than that seems like the easiest way to dig into something of this size. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #606: METRON-980: Short circuit operations for Stellar
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/606#discussion_r120622230
--- Diff:
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/WindowProcessor.java
---
@@ -89,7 +89,7 @@ public void exitIdentifier(WindowParser.IdentifierContext
ctx) {
if(checkForException(ctx)) {
return;
}
-stack.push(new Token<>(ctx.getText().substring(1), String.class));
+stack.push(new Token<>(ctx.getText().substring(1), String.class,
null));
--- End diff --
Yeah, the polymorphic constructor was added after the fact. I can remove
these now. Good catch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron issue #612: METRON-962: Configuration Based Unit Tests and Add integr...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/612 +1 by inspection. This is good work; we inherited a lot of these from the initial opensoc import, so normalizing and reducing the dead weight is a valuable contribution. Thanks @justinleet --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #609: METRON-987: Allow stellar enrichments to be specif...
Github user cestella closed the pull request at: https://github.com/apache/metron/pull/609 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #609: METRON-987: Allow stellar enrichments to be specif...
GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/609
METRON-987: Allow stellar enrichments to be specified by a list as well as
a map
## Contributor Comments
Currently, stellar enrichments are specified by a map associating fields
with the stellar expressions associated with the fields. There is a significant
downside here in that you cannot update previously assigned fields. For
instance, the following cannot be represented currently:
```
"fieldMap": {
...
"stellar" : {
"config" : {
"hostname" : "if ENDS_WITH(hostname, '.') then CHOP(hostname)
else hostname",
"hostname" : "TO_LOWER(hostname)"
}
}
}
```
This would now be allowed thusly:
```
"fieldMap": {
...
"stellar" : {
"config" : [
"hostname := if ENDS_WITH(hostname, '.') then CHOP(hostname) else
hostname",
"hostname := TO_LOWER(hostname)"
]
}
}
```
A consequent of this deficiency is that we also cannot use temporary
variables and unset them after their use inside an enrichment group.
The proposed change is to allow users to use lists of strings representing
stellar expression assignments with the same syntax as the Stellar REPL. This
would be as an alternative to maps, but the map syntax would also be supported
for legacy.
## Test plan
* Follow the instructions located
[here](https://github.com/apache/metron/tree/master/metron-platform/metron-enrichment#example-enrichment-via-stellar)
to ensure no regressions
* Adjust the configuration for
`$METRON_HOME/config/zookeeper/enrichments/squid.json` to the following and run
data through the topologies again:
```
{
"enrichment" : {
"fieldMap": {
"stellar" : {
"config" : {
"numeric" : [
"foo := 1 + 1",
"grok := foo + 3"
]
,"ALL_CAPS" : "TO_UPPER(source.type)"
}
}
}
},
"threatIntel" : {
"fieldMap":{
"stellar" : {
"config" : [
"bar := TO_UPPER(source.type)"
]
}
},
"triageConfig" : {
}
}
}
```
* Ensure that each message has `ALL_CAPS`, `foo`, `grok` and `bar` fields.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root incubating-metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
Note:
Please ensure that once the PR is submitted, you check travis-ci for build
iss
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/530 > @nickwallen: Or did you submit this to get buy-in as a proof-of-concept, that we can then break down into multiple, smaller PRs, get reviewed and then merged into master? I feel like the second would be a good approach for this one. > @ottobackwards: The answer is 1. I did not submit until this set of functionality was done because it is completely end to end testable. I really think we should break this apart into multiple PRs. Each with their own documented test plan. I don't think we are going to get a review of sufficient quality without it. I am open to however you think it should be broken apart and can offer my own suggestion on breaking it apart should you need. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/530 > No. I did not write that all out. Do you feel that is required for this? There is no such guide presently. Can that be a follow on? Sorry, but I disagree. I really think you need to lay out a specific test plan for this. I feel like we don't have that. If you feel that the steps laid out in "the metron workflow that this enables" is your test plan, then please expand on that and document specific commands that need run for each step. I know it is painful, but it is important to have a repeatable, documented test plan. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #612: METRON-962: Configuration Based Unit Tests and Add integr...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/612 I'm actually just going to kill GrokAsaParser. We know BasicAsaParser is used. GrokAsaParser seems a bit questionable based on code changes and restructuring we've done, is less adaptable, and nobody has ever mentioned it. If someone feels that we need to keep it, let me know and I'll revert and deprecate it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #612: METRON-962: Configuration Based Unit Tests and Add integr...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/612 @kylerichardson What is the difference between the `BasicAsaParser` and the `GrokAsaParser`? Are you able to shed some light on why both exist? The motivation for the question is that minor changes had to be made to the GrokAsaParser to get unit tests (and possibly the parser itself working). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r120616605
--- Diff: bundles-lib/pom.xml ---
@@ -0,0 +1,206 @@
+
+
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+org.apache.metron
+Metron
+0.4.0
+
+
+bundles-lib
+jar
+
+
+
+The Apache Software License, Version 2.0
+http://www.apache.org/licenses/LICENSE-2.0.txt
+repo
+
+
+
+
+UTF-8
+
+
+
+org.apache.accumulo
+accumulo-start
+${global_accumulo_version}
+
+
+org.apache.hadoop
+hadoop-client
+
+
+commons-io
+commons-io
+
+
+
+
+org.atteo.classindex
+classindex
+${global_classindex_version}
+
+
+org.apache.hadoop
+hadoop-common
+${global_hadoop_version}
+
+
+servlet-api
+javax.servlet
+
+
+slf4j-log4j12
+org.slf4j
+
+
+log4j
+log4j
+
+
+
+
+org.apache.hadoop
+hadoop-hdfs
+${global_hadoop_version}
+
+
+servlet-api
+javax.servlet
+
+
+
+
+org.apache.hbase
+hbase-client
+${global_hbase_version}
+
+
+org.slf4j
+slf4j-log4j12
+
+
+log4j
+log4j
+
+
+
+
+
+
+org.slf4j
+slf4j-api
+${global_slf4j_version}
+
+
+org.slf4j
+slf4j-log4j12
+${global_slf4j_version}
+
+
+junit
+junit
+${global_junit_version}
+test
+
+
+org.powermock
+powermock-module-junit4
+1.6.6
+test
+
+
+org.apache.hadoop
+hadoop-common
+${global_hadoop_version}
+tests
+
+
+slf4j-log4j12
+org.slf4j
+
+
+
+
+org.apache.hadoop
+hadoop-hdfs
+${global_hadoop_version}
+tests
+
+
+org.apache.hadoop
+hadoop-client
+${global_hadoop_version}
+tests
+
+
+org.slf4j
+slf4j-log4j12
+
+
+
+
+
--- End diff --
It wasn't old when I did it ;) I'll take a look
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 @simonellistonball the main thing with the integration tests, is that due to the nature of the dependencies between the tests and the parsers, the integration test topologies, when loading the parsers by name, find them in the system classloader, as opposed to the bundle loading mechanism. To that end, I created a new module, with only an integration test and no dependencies that is sure to load from the bundle class loaders in the test : the bundle extension test mentioned above. again, without looking at the code, testing the system with these changes should be a "everything just works like it did before" operation, with the exception of metron-docker ( top level ) - which I am not sure how to bring forward since it does things differently. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Zeppelin required for Metron 0.4.0 install
I've installed cluster with Zeppelin and it works now, does not ask for the Zeppelin creds. - Dima On 06/07/2017 01:27 PM, Dima Kovalyov wrote: > Thank you Matt, > > Image: http://i.imgur.com/EIO7YBr.png > So far, I workaround it by specifying host and port of the remote > Zeppelin instance, but I am not sure if it was ever used. > > Can I just specify dummy host:port if I don't need Zeppelin for this > deployment? > > - Dima > > On 06/06/2017 02:18 AM, Matt Foley wrote: >> Dima, the Apache list server strips attachments for security and anti-spam >> reasons. >> You can post screenshots in Gist or Jira. >> >> On 6/5/17, 4:24 AM, "Dima Kovalyov" wrote: >> >> Thank you for prompt response Anand, >> >> I have attached screen.png one more time. >> >> Actions menu available after Metron mpack installation. But I need >> Zeppelin host and port specified on "Customize Services" step during >> cluster deployment. Also, on the step "Choose Services" there is no >> Zeppelin available in Amari 2.4.3. Am I the only one who encounters this? >> >> - Dima >> >> On 06/05/2017 03:41 PM, Anand Subramanian wrote: >> > Hello Dima, >> > >> > Looks like the attachment in your mail got excluded. You can install >> Zeppelin using the Actions -> Add Service from Ambari UI. Please note that >> you will also need to select Hive and Spark, since they are dependencies. >> > >> > Regards, >> > Anand >> > >> > >> > >> > >> > On 6/5/17, 3:29 PM, "Dima Kovalyov" wrote: >> > >> >> Hello, >> >> >> >> I've missed the point where zeppelin became a mandatory requirement >> for >> >> Metron installation. Attached screen.png. >> >> >> >> I am using Ambari 2.4.3 with Metron 0.4.0 and following article below >> >> for installation: >> >> >> https://cwiki.apache.org/confluence/display/METRON/Metron+with+HDP+2.5+bare-metal+install >> >> I don't have Zeppelin available in Ambari by default. Do I need to >> >> install it separately or should I use more recent Ambari version? >> >> >> >> Can I specify remote Zeppelin instead of installing new one? >> >> Please advise. >> >> >> >> Thank you. >> >> >> >> - Dima >> >> >> >> >
[GitHub] metron pull request #602: METRON-906: Rest service storm configuration does ...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/602#discussion_r120614845
--- Diff: metron-deployment/roles/ambari_config/vars/single_node_vm.yml ---
@@ -102,7 +102,7 @@ configurations:
required_configurations:
- metron-env:
- storm_rest_addr: "{{ groups.ambari_slave[0] }}:8744"
+ storm_rest_addr: "http://{{ groups.ambari_slave[0] }}:8744"
--- End diff --
@cestella Thoughts? @dlyle65535 is right that it's the definition of a
specific cluster. I'd argue that it's the user's responsibility to update
anything afterwards when they set up SSL (e.g. through Ambari). All I was
thinking of documenting was just to ensure that field gets updated, but I'm not
sure how useful that is anyway.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r120369783
--- Diff: bundles-lib/pom.xml ---
@@ -0,0 +1,206 @@
+
+
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+org.apache.metron
+Metron
+0.4.0
+
+
+bundles-lib
+jar
+
+
+
+The Apache Software License, Version 2.0
+http://www.apache.org/licenses/LICENSE-2.0.txt
+repo
+
+
+
+
+UTF-8
+
+
+
+org.apache.accumulo
+accumulo-start
+${global_accumulo_version}
+
+
+org.apache.hadoop
+hadoop-client
+
+
+commons-io
+commons-io
+
+
+
+
+org.atteo.classindex
+classindex
+${global_classindex_version}
+
+
+org.apache.hadoop
+hadoop-common
+${global_hadoop_version}
+
+
+servlet-api
+javax.servlet
+
+
+slf4j-log4j12
+org.slf4j
+
+
+log4j
+log4j
+
+
+
+
+org.apache.hadoop
+hadoop-hdfs
+${global_hadoop_version}
+
+
+servlet-api
+javax.servlet
+
+
+
+
+org.apache.hbase
+hbase-client
+${global_hbase_version}
+
+
+org.slf4j
+slf4j-log4j12
+
+
+log4j
+log4j
+
+
+
+
+
+
+org.slf4j
+slf4j-api
+${global_slf4j_version}
+
+
+org.slf4j
+slf4j-log4j12
+${global_slf4j_version}
+
+
+junit
+junit
+${global_junit_version}
+test
+
+
+org.powermock
+powermock-module-junit4
+1.6.6
+test
+
+
+org.apache.hadoop
+hadoop-common
+${global_hadoop_version}
+tests
+
+
+slf4j-log4j12
+org.slf4j
+
+
+
+
+org.apache.hadoop
+hadoop-hdfs
+${global_hadoop_version}
+tests
+
+
+org.apache.hadoop
+hadoop-client
+${global_hadoop_version}
+tests
+
+
+org.slf4j
+slf4j-log4j12
+
+
+
+
+
--- End diff --
This looks like it's copied from the old stuff, right? You should just be
able to dump the reporting section entirely and let it be inherited (and please
double check that works).
I'll need to dig in, but I suspect this type of stuff is true of the new
poms in general (e.g. rat already being inherited, checkstyle, etc.).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user simonellistonball commented on the issue: https://github.com/apache/metron/pull/530 Awesome, I guess this should be covered by the integration test suite as well, which has been kept reasonably up to date with the recent changes. Anything there you think might deserve a little tightening, or any specific risk areas you see that we should cover in integration-tests? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #612: METRON-962: Configuration Based Unit Tests and Add integr...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/612 +1 by inspection - @justinleet did you assign the jira over? I will have to port this to 777 when it hits. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Question about the customization of Metron with my machine learining algo.
Hello Casey, your explanations (and Matt's one with the other email) help me. By the way, if I could, I need more details. The origins of my questions are both conceptual (Metron is absolutely a new tool for me) and practical (e.g., I didn't find any guideline that explain where ML model should be stored to run with Metron. In which folder I mean.) * VM vs cluster: o I pointed out the need to use a cluster. Which is the main reason? Is it linked to the performance? I mean the processing resources needed to run Metron? o In this stage, in which I'm learning Metron making also some experiment I would install it into the VM. + Can I install Metron into Ubuntu 16.04 using vagrant? (implementation question) * VM: o Which Ubuntu do I have to use? 16.04? o Which version of Metron do I have to use? o Which version of NIFI do I have to install? o Is there any additional tool that I do have to install? * Model deployment: o I would use NIFI as tool to get data from my network. + Is it any recommendation? (implementation question) o Data collected with NIFI should be sent to Metron. Reading Metron architecture (https://cwiki.apache.org/confluence/display/METRON/Metron+Architecture), it seems possible. + I'm little confused about the data-flow at this point. You pointed out two caveats to parse data and then fill in the ML model. Can you please explain me something more? (conceptual/implementation question). o As my first test I would try a packet classifier ML model, but again after your two caveats in which you said that only the second method is support I don't understand if I could classify packet that comes from NIFI's probes. + Can you help me on this point? * REST API (linked also to Matt's email): o Ok, I'm not so string with this interface so my questions would be really basic. + Reading your example: https://gist.github.com/cestella/8dd83031b8898a732b6a5a60fce1b616 I understand that I should develop my ML model in python. + Can I reuse the file rest.py pointing to my new model? * About the steps to follow. I copy your indications in RED as follows and my question are in BLACK: o Anyway, so for you to use your own ML model, you'd do the following: 1. Ingest the sensor data source that you want to ingest into a kafka topic --> Can I use NIFI? Is it a transparent process for me or is there some code to be write? 2. Create or reuse one of the existing parsers that we support to convert the data from your data source --> I do not understand. Do you refer to Stellar? I don't undrestand what Stellar it is. 3. Create your model (see https://gist.github.com/cestella/8dd83031b8898a732b6a5a60fce1b616 as an example) 4. refer to your model from stellar 1. In the example I mentioned, we're doing that at https://github.com/apache/metron/tree/master/metron-analytics/metron-maas-service#adjust-configurations-for-squid-to-call-model 2. You might consider doing it in the enrichment topology, but to get you started, doing it as a field trasnformation as in the example should suffice * Dataworks summit: o You said that your speech is public, didn't it? + Do you know if I could follow it offline from somewhere link? * Blog: o Which is the blog that you are referring to? So, in summary I would test an ML network packets classification model. Most of my question are to understand where I should put my hands to have one VM that runs Metron. In this stage, as newbe for Metron I would use Metron as a tool, focusing on the ML model in Python. Thank you in advance for your useful answers. Best Regards, Simone > Il 6 giugno 2017 alle 19.43 Casey Stella ha scritto: > > So, first off, it's not a basic question at all and thanks for asking it. > I'm sure if it's not clear to you, then it's not clear to many and bears > some reinforcement and clarification. > > * Metron does indeed enable the deployment and use of machine > learning models on data ingested into Metron > * Metron runs atop Hadoop (storm + kafka + hdfs + hbase), so you > likely wouldn't run this successfully on a VM, but rather a cluster. We do > support running Metron for demonstration purposes and development purposes > inside a VM, but that's not a production configuration, I'd like to make > clear. > Models deployed via MaaS can be interacted with via Stellar on data > ingested into Metron under a couple caveats. There are two ways to ingest > data into Metron: > * Via a packet capture sensor (fastcapa) to Kafka to the pcap storm > topology, which writes directly to HDFS with no preamble or enrichment > * Via another, lower velocity sensor (e.g. bro for deep packet > inspection or yaf for flow data) which is routed to a parser topology, then > to enrichment and finally to indexing > We do not, at present, support interacting with models (or, indeed, any > enrichment) on raw packet data (the first case above). We do, however, > support it on the second usecase. The exampl
Re: Zeppelin required for Metron 0.4.0 install
Thank you Matt, Image: http://i.imgur.com/EIO7YBr.png So far, I workaround it by specifying host and port of the remote Zeppelin instance, but I am not sure if it was ever used. Can I just specify dummy host:port if I don't need Zeppelin for this deployment? - Dima On 06/06/2017 02:18 AM, Matt Foley wrote: > Dima, the Apache list server strips attachments for security and anti-spam > reasons. > You can post screenshots in Gist or Jira. > > On 6/5/17, 4:24 AM, "Dima Kovalyov" wrote: > > Thank you for prompt response Anand, > > I have attached screen.png one more time. > > Actions menu available after Metron mpack installation. But I need > Zeppelin host and port specified on "Customize Services" step during > cluster deployment. Also, on the step "Choose Services" there is no > Zeppelin available in Amari 2.4.3. Am I the only one who encounters this? > > - Dima > > On 06/05/2017 03:41 PM, Anand Subramanian wrote: > > Hello Dima, > > > > Looks like the attachment in your mail got excluded. You can install > Zeppelin using the Actions -> Add Service from Ambari UI. Please note that > you will also need to select Hive and Spark, since they are dependencies. > > > > Regards, > > Anand > > > > > > > > > > On 6/5/17, 3:29 PM, "Dima Kovalyov" wrote: > > > >> Hello, > >> > >> I've missed the point where zeppelin became a mandatory requirement for > >> Metron installation. Attached screen.png. > >> > >> I am using Ambari 2.4.3 with Metron 0.4.0 and following article below > >> for installation: > >> > https://cwiki.apache.org/confluence/display/METRON/Metron+with+HDP+2.5+bare-metal+install > >> I don't have Zeppelin available in Ambari by default. Do I need to > >> install it separately or should I use more recent Ambari version? > >> > >> Can I specify remote Zeppelin instead of installing new one? > >> Please advise. > >> > >> Thank you. > >> > >> - Dima > > > >
