[jira] [Commented] (METRON-1620) Fixes for forensic clustering use case example
[
https://issues.apache.org/jira/browse/METRON-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515226#comment-16515226
]
ASF GitHub Bot commented on METRON-1620:
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1065
**Testing**
You can run through the full use case, if desired. If you want the TL;DR
version to verify the template command, run the command in the README for
creating the ES template. Then do the following:
Make sure you have at least 1 other sensor with data, e.g. Bro. In full dev
you should be set, otherwise cat data from our unit tests
(https://github.com/apache/metron/blob/master/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
into the `bro` Kafka topic and make sure the bro topology is running.
e.g.
```
wget
https://github.com/apache/metron/raw/master/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput
-O ~/sample-bro.json
cat ~/sample-bro.json |
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list
$BROKERLIST --topic bro
```
Next, check the template was loaded correctly:
```
curl -XGET $ES_HOST'/_template/cowrie_index?pretty=true'
```
Then load the following into ES:
```
curl -XPUT $ES_HOST'/cowrie_index_1/cowrie_doc/1' -H 'Content-Type:
application/json' -d'
{
"eventid" : "cowrie.command.input",
"adapter:stellaradapter:end:ts" : "1529268179998",
"threatinteljoinbolt:joiner:ts" : "1529268180010",
"session" : "4c047bbc016c",
"threat:triage:rules:0:comment" : "Determine if a host is blacklisted",
"enrichmentsplitterbolt:splitter:begin:ts" : "1529268179997",
"enrichmentjoinbolt:joiner:ts" : "1529268180002",
"threat:triage:rules:0:name" : "Blacklisted Host",
"src_ip" : "94.51.110.74",
"source:type" : "cowrie",
"isError" : 0,
"original_string" :
"{\"src_ip\":\"94.51.110.74\",\"eventid\":\"cowrie.command.input\",\"input\":\"\\/bin\\/busybox
XUSRH\",\"system\":\"CowrieTelnetTransport,93,94.51.110.74\",\"isError\":0,\"session\":\"4c047bbc016c\",\"sensor\":\"a927e8b28666\",\"message\":\"CMD:
\\/bin\\/busybox XUSRH\",\"timestamp\":\"2017-09-17T04:06:40.419195Z\"}",
"threatintelsplitterbolt:splitter:end:ts" : "1529268180004",
"similarity_bin" : "166524",
"threat:triage:rules:0:score" : 10,
"timestamp" : 1505621619195,
"threat:triage:rules:0:reason" : "IP 94.51.110.74 is blacklisted",
"enrichmentsplitterbolt:splitter:end:ts" : "1529268179997",
"threat:triage:score" : 10.0,
"is_alert" : "true",
"adapter:stellaradapter:begin:ts" : "1529268179998",
"message" : "CMD: /bin/busybox XUSRH",
"input" : "/bin/busybox XUSRH",
"blacklisted" : true,
"system" : "CowrieTelnetTransport,93,94.51.110.74",
"threatintelsplitterbolt:splitter:begin:ts" : "1529268180004",
"guid" : "f4e441d2-74e7-4127-89c4-edcf8227f893",
"sensor" : "a927e8b28666",
"tlsh" :
"87A002C029850AFE3C890231B18B743C002C10825E5028A6DC8D00C1F213FC6FD31D0C"
}
'
```
Go to the Alerts UI and enter this in the search:
```
is_alert:true AND similarity_bin:166524
```
You should see the alert in the UI.
> Fixes for forensic clustering use case example
> --
>
> Key: METRON-1620
> URL: https://issues.apache.org/jira/browse/METRON-1620
> Project: Metron
> Issue Type: Bug
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
> ES mapping needed some adjustments. Change to dynamic template mapping so it
> will work for non-existent indexes yet to be created. Make work with ES 5.6.x
> data types.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1065: METRON-1620: Fixes for forensic clustering use case exam...
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1065
**Testing**
You can run through the full use case, if desired. If you want the TL;DR
version to verify the template command, run the command in the README for
creating the ES template. Then do the following:
Make sure you have at least 1 other sensor with data, e.g. Bro. In full dev
you should be set, otherwise cat data from our unit tests
(https://github.com/apache/metron/blob/master/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
into the `bro` Kafka topic and make sure the bro topology is running.
e.g.
```
wget
https://github.com/apache/metron/raw/master/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput
-O ~/sample-bro.json
cat ~/sample-bro.json |
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list
$BROKERLIST --topic bro
```
Next, check the template was loaded correctly:
```
curl -XGET $ES_HOST'/_template/cowrie_index?pretty=true'
```
Then load the following into ES:
```
curl -XPUT $ES_HOST'/cowrie_index_1/cowrie_doc/1' -H 'Content-Type:
application/json' -d'
{
"eventid" : "cowrie.command.input",
"adapter:stellaradapter:end:ts" : "1529268179998",
"threatinteljoinbolt:joiner:ts" : "1529268180010",
"session" : "4c047bbc016c",
"threat:triage:rules:0:comment" : "Determine if a host is blacklisted",
"enrichmentsplitterbolt:splitter:begin:ts" : "1529268179997",
"enrichmentjoinbolt:joiner:ts" : "1529268180002",
"threat:triage:rules:0:name" : "Blacklisted Host",
"src_ip" : "94.51.110.74",
"source:type" : "cowrie",
"isError" : 0,
"original_string" :
"{\"src_ip\":\"94.51.110.74\",\"eventid\":\"cowrie.command.input\",\"input\":\"\\/bin\\/busybox
XUSRH\",\"system\":\"CowrieTelnetTransport,93,94.51.110.74\",\"isError\":0,\"session\":\"4c047bbc016c\",\"sensor\":\"a927e8b28666\",\"message\":\"CMD:
\\/bin\\/busybox XUSRH\",\"timestamp\":\"2017-09-17T04:06:40.419195Z\"}",
"threatintelsplitterbolt:splitter:end:ts" : "1529268180004",
"similarity_bin" : "166524",
"threat:triage:rules:0:score" : 10,
"timestamp" : 1505621619195,
"threat:triage:rules:0:reason" : "IP 94.51.110.74 is blacklisted",
"enrichmentsplitterbolt:splitter:end:ts" : "1529268179997",
"threat:triage:score" : 10.0,
"is_alert" : "true",
"adapter:stellaradapter:begin:ts" : "1529268179998",
"message" : "CMD: /bin/busybox XUSRH",
"input" : "/bin/busybox XUSRH",
"blacklisted" : true,
"system" : "CowrieTelnetTransport,93,94.51.110.74",
"threatintelsplitterbolt:splitter:begin:ts" : "1529268180004",
"guid" : "f4e441d2-74e7-4127-89c4-edcf8227f893",
"sensor" : "a927e8b28666",
"tlsh" :
"87A002C029850AFE3C890231B18B743C002C10825E5028A6DC8D00C1F213FC6FD31D0C"
}
'
```
Go to the Alerts UI and enter this in the search:
```
is_alert:true AND similarity_bin:166524
```
You should see the alert in the UI.
---
[jira] [Commented] (METRON-1620) Fixes for forensic clustering use case example
[ https://issues.apache.org/jira/browse/METRON-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515223#comment-16515223 ] ASF GitHub Bot commented on METRON-1620: GitHub user mmiklavc opened a pull request: https://github.com/apache/metron/pull/1065 METRON-1620: Fixes for forensic clustering use case example ## Contributor Comments https://issues.apache.org/jira/browse/METRON-1620 Get the forensic hashing use case example working with ES 5.6.x. I tested this in a 10-node Amazon EC2 Metron cluster. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mmiklavc/metron update-forensic-clustering Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1065.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1065 commit 8a10f1ba497ce4156c0af63c6cd1dd1c82a10e02 Author: Michael Miklavcic Date: 2018-06-17T20:51:02Z Fixes for forensic clustering use case example. > Fixes for forensic clustering use case example > -- > > Key: METRON-1620 > URL: https://issues.apache.org/jira/browse/METRON-1620 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > > ES mapping needed some adjustments. Change to dynamic template mapping so it > will work for non-existent indexes yet to be created. Make work with ES 5.6.x > data types. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1620) Fixes for forensic clustering use case example
Michael Miklavcic created METRON-1620: - Summary: Fixes for forensic clustering use case example Key: METRON-1620 URL: https://issues.apache.org/jira/browse/METRON-1620 Project: Metron Issue Type: Bug Reporter: Michael Miklavcic Assignee: Michael Miklavcic ES mapping needed some adjustments. Change to dynamic template mapping so it will work for non-existent indexes yet to be created. Make work with ES 5.6.x data types. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1618) Stellar boolean expressions should treat missing variables as false
[ https://issues.apache.org/jira/browse/METRON-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515107#comment-16515107 ] ASF GitHub Bot commented on METRON-1618: Github user simonellistonball commented on the issue: https://github.com/apache/metron/pull/1063 Should the example target state not be: `is_alert := is_alert || geo_outlier` We should probably have a test for that, but I assume the null is falsey would apply on both sides of the boolean and that things like short cutting `is_thing && FUNC(thing)` would be unaffected (i.e if is_thing is null, FUNC would never run). Should we also include updates to the configs in the use cases folder (since these are kinda documentation) with this PR? > Stellar boolean expressions should treat missing variables as false > --- > > Key: METRON-1618 > URL: https://issues.apache.org/jira/browse/METRON-1618 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Priority: Major > > Currently, we treat missing variables as null in boolean expressions rather > than false. If we did adopted a more javascripty approach, stellar would be > easier to use and require fewer existence checks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1063: METRON-1618: Stellar boolean expressions should treat mi...
Github user simonellistonball commented on the issue: https://github.com/apache/metron/pull/1063 Should the example target state not be: `is_alert := is_alert || geo_outlier` We should probably have a test for that, but I assume the null is falsey would apply on both sides of the boolean and that things like short cutting `is_thing && FUNC(thing)` would be unaffected (i.e if is_thing is null, FUNC would never run). Should we also include updates to the configs in the use cases folder (since these are kinda documentation) with this PR? ---
[GitHub] metron issue #1063: METRON-1618: Stellar boolean expressions should treat mi...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1063 ```bash [Stellar]>>> foo := unknownvariable [Stellar]>>> foo [Stellar]>>> ``` This is not consistent. In my stellar assign PR, this is why I execute everything in stellar instead of part shell. ---
[jira] [Commented] (METRON-1618) Stellar boolean expressions should treat missing variables as false
[ https://issues.apache.org/jira/browse/METRON-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515092#comment-16515092 ] ASF GitHub Bot commented on METRON-1618: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1063 ```bash [Stellar]>>> foo := unknownvariable [Stellar]>>> foo [Stellar]>>> ``` This is not consistent. In my stellar assign PR, this is why I execute everything in stellar instead of part shell. > Stellar boolean expressions should treat missing variables as false > --- > > Key: METRON-1618 > URL: https://issues.apache.org/jira/browse/METRON-1618 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Priority: Major > > Currently, we treat missing variables as null in boolean expressions rather > than false. If we did adopted a more javascripty approach, stellar would be > easier to use and require fewer existence checks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1619) Stellar empty collections should be considered false in boolean expressions
[ https://issues.apache.org/jira/browse/METRON-1619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515090#comment-16515090 ] ASF GitHub Bot commented on METRON-1619: Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1064#discussion_r195931024 --- Diff: metron-stellar/stellar-common/README.md --- @@ -54,6 +54,12 @@ The Stellar language supports the following: * The ability to have parenthesis to make order of operations explicit * User defined functions, including Lambda expressions +### Boolean Expressions + +Similar to python and javascript, empty collections (e.g. `[]` and --- End diff -- Missing variables vs. NULL variables... this may be confusing. What this is saying is we support - boolean with true or false - variables that are present but explicitly null or empty - variables that are NOT present > Stellar empty collections should be considered false in boolean expressions > --- > > Key: METRON-1619 > URL: https://issues.apache.org/jira/browse/METRON-1619 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Priority: Major > > Similar to METRON-1618, we should follow the example of python and javascript > and make empty collections evaluate to false in boolean expressions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1619) Stellar empty collections should be considered false in boolean expressions
[ https://issues.apache.org/jira/browse/METRON-1619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515089#comment-16515089 ] ASF GitHub Bot commented on METRON-1619: Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1064#discussion_r195931103 --- Diff: metron-stellar/stellar-common/README.md --- @@ -54,6 +54,12 @@ The Stellar language supports the following: * The ability to have parenthesis to make order of operations explicit * User defined functions, including Lambda expressions +### Boolean Expressions + +Similar to python and javascript, empty collections (e.g. `[]` and --- End diff -- Or am i miss understanding the explicit null case? > Stellar empty collections should be considered false in boolean expressions > --- > > Key: METRON-1619 > URL: https://issues.apache.org/jira/browse/METRON-1619 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Priority: Major > > Similar to METRON-1618, we should follow the example of python and javascript > and make empty collections evaluate to false in boolean expressions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1064: METRON-1619: Stellar empty collections should be ...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1064#discussion_r195931103 --- Diff: metron-stellar/stellar-common/README.md --- @@ -54,6 +54,12 @@ The Stellar language supports the following: * The ability to have parenthesis to make order of operations explicit * User defined functions, including Lambda expressions +### Boolean Expressions + +Similar to python and javascript, empty collections (e.g. `[]` and --- End diff -- Or am i miss understanding the explicit null case? ---
[GitHub] metron pull request #1064: METRON-1619: Stellar empty collections should be ...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/1064#discussion_r195931024 --- Diff: metron-stellar/stellar-common/README.md --- @@ -54,6 +54,12 @@ The Stellar language supports the following: * The ability to have parenthesis to make order of operations explicit * User defined functions, including Lambda expressions +### Boolean Expressions + +Similar to python and javascript, empty collections (e.g. `[]` and --- End diff -- Missing variables vs. NULL variables... this may be confusing. What this is saying is we support - boolean with true or false - variables that are present but explicitly null or empty - variables that are NOT present ---
