[jira] [Commented] (METRON-1367) Stellar should optionally instrument function calls

2017-12-18 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/METRON-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296202#comment-16296202
 ] 

ASF GitHub Bot commented on METRON-1367:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/873
  
If there are any ideas on improving the tags implementation I would love to 
here them


> Stellar should optionally instrument function calls
> ---
>
> Key: METRON-1367
> URL: https://issues.apache.org/jira/browse/METRON-1367
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>
> It would be nice if stellar processor(s) could optionally instrument function 
> calls to test performance.
> The result would be some structure with the calls, the times and maybe some 
> context as a list or map



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (METRON-1372) Validate JIRA for Releases

2017-12-18 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/METRON-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296108#comment-16296108
 ] 

ASF GitHub Bot commented on METRON-1372:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/874
  
+1 worked great


> Validate JIRA for Releases
> --
>
> Key: METRON-1372
> URL: https://issues.apache.org/jira/browse/METRON-1372
> Project: Metron
>  Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Minor
>
> Create a script that allows the Release Manager to easily validate JIRA for 
> the next release.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (METRON-1372) Validate JIRA for Releases

2017-12-18 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/METRON-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295888#comment-16295888
 ] 

ASF GitHub Bot commented on METRON-1372:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/874#discussion_r157632089
  
--- Diff: build_utils/release-utils/validate-jira-for-release ---
@@ -0,0 +1,197 @@
+#!/bin/bash
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+# Finds all commits since the last release tag, then ensures that each
+# is marked 'Done' and that the fix version is set to the next release.
+#
+# For example, to validate JIRA for the 0.4.2 release, you would run the
+# following command.
+#
+# validate-jira-for-release --version=0.4.2 
--start=tags/apache-metron-0.4.1-release
+#
+# This will output a table containing each JIRA that was inspected along 
with
+# the fix version, status, and assignee.  If the fix version or status is 
incorrect
+# a link will be printed so that the JIRA can be manually fixed.  The JIRA
+# only needs to be fixed if a URL is shown.
+#
+#JIRASTATUS FIX VERSION ASSIGNEE  FIX
+# METRON-1345  Done   0.4.2 Michael Miklavcic
+# METRON-1349  DoneNext + 1 Nick Allen
https://issues.apache.org/jira/browse/METRON-1349
+# METRON-1343  Done Mohan 
https://issues.apache.org/jira/browse/METRON-1343
+#...
+#
+
+function help {
+  echo " "
+  echo "usage: ${0}"
+  echo "-v/--version=   The version of the next release. 
[Required]"
+  echo "-s/--start=   Defines the first commit to inspect. 
[Required]"
+  echo "-e/--end=   Defines the last commit to inspect. "
+  echo "-r/--repo= The Git repo to work from."
+  echo "-b/--branch= The branch to work from."
+  echo "-h/--helpUsage information."
+  echo " "
+  echo "example: "
+  echo "validate-jira-for-release --version=0.4.2 
--start=tags/apache-metron-0.4.1-release"
+  echo " "
+}
+
+# define default values
+END="HEAD"
+REPO="https://git-wip-us.apache.org/repos/asf/metron.git;
+BRANCH="master"
+
+# print help, if the user just runs this without any args
+if [ "$#" -eq 0 ]; then
+help
+exit 1
+fi
+
+# handle command line options
+for i in "$@"; do
+  case $i in
+#
+# VERSION: The release version to validate; the 'next' release.
+#
+#
+-v=*|--version=*)
+VERSION="${i#*=}"
+shift # past argument=value
+;;
+
+#
+# START: Defines the first commit to inspect
+#
+#   -s=tags/apache-metron-0.4.1-release
+#   --start=tags/apache-metron-0.4.1-release
+#
+-s=*|--start=*)
+START="${i#*=}"
+shift # past argument=value
+;;
+
+#
+# END: Defines the last commit to inspect
+#
+#   -e=HEAD
+#   --end=HEAD
+#
+-e=*|--end=*)
+END="${i#*=}"
+shift # past argument=value
+;;
+
+#
+# REPO: Define the Git repo to work from
+#
+#  -r=https://git-wip-us.apache.org/repos/asf/metron.git
+#  --repo=
+#
+-r=*|--repo=*)
+REPO="${i#*=}"
+shift # past argument=value
+;;
+
+#
+# BRANCH: The branch to work from.
+#
+#  -b=master
+#  --branch=master
+#
+-b=*|--branch=*)
+BRANCH="${i#*=}"
+shift # past argument with no value
+;;
+
+#
+# -h/--help
+#
+-h|--help)
+help
+exit 0
+shift # past argument with no value
+;;
+
+#
+# Unknown option
+#
+*)
+UNKNOWN_OPTION="${i#*=}"
+

[jira] [Commented] (METRON-1372) Validate JIRA for Releases

2017-12-18 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/METRON-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295284#comment-16295284
 ] 

ASF GitHub Bot commented on METRON-1372:


GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/874

METRON-1372 Validate JIRA for Releases

Adds a script that helps the Release Manager validate JIRA for a pending 
release.

Help is printed along with an example.
```
$ ./release-utils/validate-jira-for-release --help

usage: ./release-utils/validate-jira-for-release
-v/--version=   The version of the next release. [Required]
-s/--start=   Defines the first commit to inspect. [Required]
-e/--end=   Defines the last commit to inspect.
-r/--repo= The Git repo to work from.
-b/--branch= The branch to work from.
-h/--helpUsage information.

example:
validate-jira-for-release --version=0.4.2 
--start=tags/apache-metron-0.4.1-release
```

Validating the next release.
```
$ ./release-utils/validate-jira-for-release --version=0.4.2 
--start=tags/apache-metron-0.4.1-release
Cloning into 'metron-0.4.2'...
remote: Counting objects: 35046, done.
remote: Compressing objects: 100% (13698/13698), done.
remote: Total 35046 (delta 15702), reused 31650 (delta 12822)
Receiving objects: 100% (35046/35046), 53.06 MiB | 6.77 MiB/s, done.
Resolving deltas: 100% (15702/15702), done.
Switched to branch 'master'
Your branch is up to date with 'origin/master'.
Fetching origin
   JIRA  STATUS FIX VERSION   
ASSIGNEEFIX
METRON-1345Done  Michael 
Miklavcic  https://issues.apache.org/jira/browse/METRON-1345
METRON-1349DoneNext + 1 Nick 
Allen  https://issues.apache.org/jira/browse/METRON-1349
METRON-1343Done  
Mohan  https://issues.apache.org/jira/browse/METRON-1343
METRON-1306   To Do 
Unassigned  https://issues.apache.org/jira/browse/METRON-1306
METRON-1341DoneSimon Elliston 
Ball  https://issues.apache.org/jira/browse/METRON-1341
METRON-1313Done Jon 
Zeolla  https://issues.apache.org/jira/browse/METRON-1313
METRON-1346DoneOtto 
Fowler  https://issues.apache.org/jira/browse/METRON-1346
METRON-1336Done   0.4.2 Nick 
Allen
METRON-1335Done  Anand 
Subramanian  https://issues.apache.org/jira/browse/METRON-1335
METRON-1308Done Jon 
Zeolla  https://issues.apache.org/jira/browse/METRON-1308
METRON-1338Done   0.4.2 Nick 
Allen
METRON-1286   To Do   0.4.2 
Unassigned  https://issues.apache.org/jira/browse/METRON-1286
METRON-1334Done   0.4.2 Nick 
Allen
METRON-1277DoneOtto 
Fowler  https://issues.apache.org/jira/browse/METRON-1277
METRON-1239   To Do 
Unassigned  https://issues.apache.org/jira/browse/METRON-1239
METRON-1328Done  Anand 
Subramanian  https://issues.apache.org/jira/browse/METRON-1328
METRON-1333DoneOtto 
Fowler  https://issues.apache.org/jira/browse/METRON-1333
METRON-1252Done 
RaghuMitra  https://issues.apache.org/jira/browse/METRON-1252
METRON-1316   To DoNext + 1 
Unassigned  https://issues.apache.org/jira/browse/METRON-1316
METRON-1088Done Jon 
Zeolla  https://issues.apache.org/jira/browse/METRON-1088
METRON-1319   To Do  Ryan 
Merriman  https://issues.apache.org/jira/browse/METRON-1319
METRON-1321   To Do 
Unassigned  https://issues.apache.org/jira/browse/METRON-1321
METRON-1301Done   0.4.2 Nick 
Allen
METRON-1294   To Do  Ryan 
Merriman  https://issues.apache.org/jira/browse/METRON-1294
METRON-1291   To Do  Ryan 
Merriman  

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-12-18 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295255#comment-16295255
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r157540388
  
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -1,5 +1,14 @@
 # Elasticsearch in Metron
 
+## Table of Contents
+
+* [Introduction](#introduction)
+* [Properties](#properties)
+* [Upgrading to 5.6.2](#upgrading-to-562)
+* [Type Mappings](#type-mappings)
+* [Using Metron with Elasticsearch 
5.x](#using-metron-with-elasticsearch-5x)
--- End diff --

This link is broken, because the section name doesn't line up with the 
actual name


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-12-18 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295254#comment-16295254
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r157540768
  
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -33,7 +42,217 @@ For instance, an `es.date.format` of `.MM.dd.HH` 
would have the consequence
 roll hourly, whereas an `es.date.format` of `.MM.dd` would have the 
consequence that the indices would
 roll daily.
 
-## Using Metron with Elasticsearch 2.x
+## Upgrading to 5.6.2
+
+Users should be prepared to re-index when migrating from Elasticsearch 
2.3.3 to 5.6.2. There are a number of template changes, most notably around
+string type handling, that may cause issues when upgrading.
+

+[https://www.elastic.co/guide/en/elasticsearch/reference/5.6/setup-upgrade.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/setup-upgrade.html)
+
+Be aware that if you add a new string value and want to be able to filter 
and search on this value from the Alerts UI, you **must** add a mapping for 
that type to
+the appropriate Elasticsearch template. Below is more detail on how to 
choose the appropriate mapping type for your string value.
+
+## Type Mappings
+
+Type mappings have changed quite a bit from ES 2.x -> 5.x. Here is a brief 
rundown of the biggest changes. More detailed references from Elasticsearch
+are provided in the [Type Mapping References](#type-mapping-references) 
section below.
+* string fields replaced by text/keyword type
+* strings have new default mappings as follows
+
+```
+{
+  "type": "text",
+  "fields": {
+"keyword": {
+  "type": "keyword",
+  "ignore_above": 256
+}
+  }
+}
+```
+
+* There is no longer a `_timestamp` field that you can set "enabled" on. 
This field now causes an exception on templates.
+Replace with an application-created timestamp of "date" type.
+
+The semantics for string types have changed. In 2.x, you have the concept 
of index settings as either "analyzed" or "not_analyzed" which basically means 
"full text" and "keyword", respectively.
+Analyzed text basically means the indexer will split the text using a text 
analyzer thus allowing you to search on substrings within the original text. 
"New York" is split and indexed as two buckets,
+ "New" and "York", so you can search or query for aggregate counts for 
those terms independently and will match against the individual terms "New" or 
"York." "Keyword" means that the original text
+ will not be split/analyzed during indexing and instead treated as a whole 
unit, i.e. "New" or "York" will not match in searches against the document 
containing "New York", but searching on "New York"
+ as the full city name will. In 5.x language instead of using the "index" 
setting, you now set the "type" to either "text" for full text, or "keyword" 
for keywords.
+
+Below is a table depicting the changes to how String types are now handled.
+
+
+
+   sort, aggregate, or access values
+   ES 2.x
+   ES 5.x
+   Example
+
+
+   no
+   
+"my_property" : {
+  "type": "string",
+  "index": "analyzed"
+}
+
+   
+   
+"my_property" : {
+  "type": "text"
+}
+
+Additional defaults: "index": "true", "fielddata": "false"
+   
+   
+   "New York" handled via in-mem search as "New" and "York" 
buckets. No aggregation or sort.
+   
+
+
+   
+   yes
+   
+   
+"my_property": {
+  "type": "string",
+  "index": "analyzed"
+}
+
+   
+   
+"my_property": {
+  "type": "text",
+  "fielddata": "true"
+}
+
+   
+   
+   "New York" handled via in-mem search as "New" and "York" buckets. 
Can aggregate and sort.
+   
+
+
+   
+   yes
+   
+   
+"my_property": {
+  "type": "string",
+  "index": "not_analyzed"
+}
+
+   
+   
+"my_property" : {
+  "type": "keyword"
+}
+
+   
+   
+   "New York" searchable as single value. Can aggregate 
and sort. A search for "New" or "York" will not match against the whole value.
+   
+
+
+   
+   yes
+   
+   
+"my_property": {
+  "type": "string",
+  "index": "analyzed"
+}
+
+   
+   
+"my_property": {
+  "type": "text",
+  "fields": {
+"keyword": {
+  "type": "keyword",
+  "ignore_above": 256
+}
+  }
+}
+
+   
+   
+   "New York" 

[jira] [Created] (METRON-1372) Validate JIRA for Releases

2017-12-18 Thread Nick Allen (JIRA)
Nick Allen created METRON-1372:
--

 Summary: Validate JIRA for Releases
 Key: METRON-1372
 URL: https://issues.apache.org/jira/browse/METRON-1372
 Project: Metron
  Issue Type: Improvement
Reporter: Nick Allen
Assignee: Nick Allen
Priority: Minor


Create a script that allows the Release Manager to easily validate JIRA for the 
next release.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (METRON-1370) Create Full Dev Equivalent for Ubuntu

2017-12-18 Thread Nick Allen (JIRA)

[ 
https://issues.apache.org/jira/browse/METRON-1370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295144#comment-16295144
 ] 

Nick Allen commented on METRON-1370:


To create the Full Dev equivalent on Ubuntu, the Mpack would need to be able to 
deploy Metron on a Ubuntu platform.

> Create Full Dev Equivalent for Ubuntu
> -
>
> Key: METRON-1370
> URL: https://issues.apache.org/jira/browse/METRON-1370
> Project: Metron
>  Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>
> With METRON-1351, we will have installable packages to make the installation 
> of Metron on Ubuntu far simpler.  We need to create an equivalent version of 
> "Full Dev" that runs on Ubuntu.  This will allow us to easily ensure that the 
> packages, along with the rest of Metron, works when run on Ubuntu.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (METRON-1371) Enhance Mpack for Ubuntu Deployments

2017-12-18 Thread Nick Allen (JIRA)

[ 
https://issues.apache.org/jira/browse/METRON-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295143#comment-16295143
 ] 

Nick Allen commented on METRON-1371:


Need to have installable Ubuntu packages before the Mpack can be enhanced to 
support Ubuntu.  The Mpack would use the Ubuntu packages for installation.

> Enhance Mpack for Ubuntu Deployments
> 
>
> Key: METRON-1371
> URL: https://issues.apache.org/jira/browse/METRON-1371
> Project: Metron
>  Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>  Labels: Ubuntu
>
> The current Mpack supports installation on CentOS and RPM-based environments 
> only.  We should enhance the Mpack so that users can deploy on Ubuntu also.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (METRON-1371) Enhance Mpack for Ubuntu Deployments

2017-12-18 Thread Nick Allen (JIRA)
Nick Allen created METRON-1371:
--

 Summary: Enhance Mpack for Ubuntu Deployments
 Key: METRON-1371
 URL: https://issues.apache.org/jira/browse/METRON-1371
 Project: Metron
  Issue Type: Improvement
Reporter: Nick Allen
Assignee: Nick Allen


The current Mpack supports installation on CentOS and RPM-based environments 
only.  We should enhance the Mpack so that users can deploy on Ubuntu also.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (METRON-1370) Create Full Dev Equivalent for Ubuntu

2017-12-18 Thread Nick Allen (JIRA)
Nick Allen created METRON-1370:
--

 Summary: Create Full Dev Equivalent for Ubuntu
 Key: METRON-1370
 URL: https://issues.apache.org/jira/browse/METRON-1370
 Project: Metron
  Issue Type: Improvement
Reporter: Nick Allen
Assignee: Nick Allen


With METRON-1351, we will have installable packages to make the installation of 
Metron on Ubuntu far simpler.  We need to create an equivalent version of "Full 
Dev" that runs on Ubuntu.  This will allow us to easily ensure that the 
packages, along with the rest of Metron, works when run on Ubuntu.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Assigned] (METRON-1369) squid events are not seen in Alerts UI

2017-12-18 Thread Nick Allen (JIRA)

 [ 
https://issues.apache.org/jira/browse/METRON-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Allen reassigned METRON-1369:
--

Assignee: Nick Allen

> squid events are not seen in Alerts UI
> --
>
> Key: METRON-1369
> URL: https://issues.apache.org/jira/browse/METRON-1369
> Project: Metron
>  Issue Type: Bug
>Reporter: Anand Subramanian
>Assignee: Nick Allen
>
> On a metron cluster, I have added the squid telemetry by following the [Wiki 
> instructions|https://cwiki.apache.org/confluence/display/METRON/2016/05/02/Metron+Tutorial+-+Fundamentals+Part+4%3A+Pluggable+Threat+Intelligence].
> I am able to see the squid indices present in elasticsearch, but not able to 
> see them in the alerts UI. On further checking, I found out that this is due 
> to the missing nested "alert" field in the index mappings--which is required 
> for the Alerts UI to pick up the index.
> This was confirmed in the rest logs as well.
> {code}
> 17/12/18 10:39:40 ERROR dao.ElasticsearchRequestSubmitter: Shard search 
> failure [1/1]; reason=IllegalStateException: [nested] failed to find nested 
> object under path [alert], index=squid_index_2017.12.18.10, shard=0, 
> status=BAD_REQUEST, nodeId=NOjEpjy0RxCV1ltcqp4Erw
> [squid_index_2017.12.18.10/a7-5TaroSRCHMrJnCWkYww] QueryShardException[failed 
> to create query: {
>   "constant_score" : {
> "filter" : {
>   "bool" : {
> "must" : [
>   {
> "bool" : {
>   "should" : [
> {
>   "query_string" : {
> "query" : "*",
> "fields" : [ ],
> "use_dis_max" : true,
> "tie_breaker" : 0.0,
> "default_operator" : "or",
> "auto_generate_phrase_queries" : false,
> "max_determinized_states" : 1,
> "enable_position_increments" : true,
> "fuzziness" : "AUTO",
> "fuzzy_prefix_length" : 0,
> "fuzzy_max_expansions" : 50,
> "phrase_slop" : 0,
> "escape" : false,
> "split_on_whitespace" : true,
> "boost" : 1.0
>   }
> },
> {
>   "nested" : {
> "query" : {
>   "query_string" : {
> "query" : "*",
> "fields" : [ ],
> "use_dis_max" : true,
> "tie_breaker" : 0.0,
> "default_operator" : "or",
> "auto_generate_phrase_queries" : false,
> "max_determinized_states" : 1,
> "enable_position_increments" : true,
> "fuzziness" : "AUTO",
> "fuzzy_prefix_length" : 0,
> "fuzzy_max_expansions" : 50,
> "phrase_slop" : 0,
> "escape" : false,
> "split_on_whitespace" : true,
> "boost" : 1.0
>   }
> },
> "path" : "alert",
> "ignore_unmapped" : false,
> "score_mode" : "none",
> "boost" : 1.0
>   }
> }
>   ],
>   "disable_coord" : false,
>   "adjust_pure_negative" : true,
>   "boost" : 1.0
> }
>   },
>   {
> "bool" : {
>   "should" : [
> {
>   "term" : {
> "status" : {
>   "value" : "active",
>   "boost" : 1.0
> }
>   }
> },
> {
>   "bool" : {
> "must_not" : [
>   {
> "exists" : {
>   "field" : "status",
>   "boost" : 1.0
> }
>   }
> ],
> "disable_coord" : false,
> "adjust_pure_negative" : true,
> "boost" : 1.0
>   }
> }
>   ],
>   "disable_coord" : false,
>   "adjust_pure_negative" : true,
>   "boost" : 1.0
> }
>   }
> ],
> "must_not" : [
>   {
> "exists" : {
>   "field" : "metaalerts",
>   "boost" : 1.0
> }
>   }
> ],
> "disable_coord" : false,
> 

[jira] [Commented] (METRON-1364) Add an implementation of Robust PCA outlier detection

2017-12-18 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/METRON-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295043#comment-16295043
 ] 

ASF GitHub Bot commented on METRON-1364:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/870
  
How can we test this?


> Add an implementation of Robust PCA outlier detection
> -
>
> Key: METRON-1364
> URL: https://issues.apache.org/jira/browse/METRON-1364
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>
> With short circuiting in Stellar, we have the opportunity to delve into more 
> computationally intensive outlier detection techniques.  Generally these 
> would be executed only if simpler outlier detection techniques indicated an 
> outlier (e.g. statistical outlier tests).
> As the first one of these supported, I'd suggest a Robust PCA based technique 
> similar to Netflix's Surus.  See 
> https://medium.com/netflix-techblog/rad-outlier-detection-on-big-data-d6b0494371cc
>  and 
> https://metamarkets.com/2012/algorithmic-trendspotting-the-meaning-of-interesting/
>  for more detail.
> It should be noted that there are some caveats with this approach around 
> sparsity and orderedness.  
> Regarding sparsity,this outlier detection algorithm presumes dense output, 
> which is not the case for data spanning profiles (e.g. the profiler does not 
> write out data every period if no data was seen). To deal with this, I am 
> suggesting a modification to the profiler to allow PROFILE_GET to return a 
> default value.  That will be done in a separate JIRA.
> Regarding well-orderedness, this is an outlier detector for time series data, 
> so it is sensitive to order to a certain extent.  Given its computational 
> intensity, it is likely to be used with a sample of the data to shrink the 
> size of the data.  To that end, uniform sampling is not sensible here, but 
> rather a biased sample for recency.  Without this, you may get poor results 
> from this outlier detector.  This sampler should be done in a separate JIRA, 
> but I will ensure the infrastructure to add it is contributed in METRON-1350.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (METRON-1367) Stellar should optionally instrument function calls

2017-12-18 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/METRON-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16294936#comment-16294936
 ] 

ASF GitHub Bot commented on METRON-1367:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/873
  
One idea for a follow on : add a 'tag' or 'type' similar to log.INFO 
log.DEBUG to the records, so that you can filter the output.



> Stellar should optionally instrument function calls
> ---
>
> Key: METRON-1367
> URL: https://issues.apache.org/jira/browse/METRON-1367
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>
> It would be nice if stellar processor(s) could optionally instrument function 
> calls to test performance.
> The result would be some structure with the calls, the times and maybe some 
> context as a list or map



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (METRON-1369) squid events are not seen in Alerts UI

2017-12-18 Thread Anand Subramanian (JIRA)
Anand Subramanian created METRON-1369:
-

 Summary: squid events are not seen in Alerts UI
 Key: METRON-1369
 URL: https://issues.apache.org/jira/browse/METRON-1369
 Project: Metron
  Issue Type: Bug
Reporter: Anand Subramanian


On a metron cluster, I have added the squid telemetry by following the [Wiki 
instructions|https://cwiki.apache.org/confluence/display/METRON/2016/05/02/Metron+Tutorial+-+Fundamentals+Part+4%3A+Pluggable+Threat+Intelligence].

I am able to see the squid indices present in elasticsearch, but not able to 
see them in the alerts UI. On further checking, I found out that this is due to 
the missing nested "alert" field in the index mappings--which is required for 
the Alerts UI to pick up the index.

This was confirmed in the rest logs as well.

{code}
17/12/18 10:39:40 ERROR dao.ElasticsearchRequestSubmitter: Shard search failure 
[1/1]; reason=IllegalStateException: [nested] failed to find nested object 
under path [alert], index=squid_index_2017.12.18.10, shard=0, 
status=BAD_REQUEST, nodeId=NOjEpjy0RxCV1ltcqp4Erw
[squid_index_2017.12.18.10/a7-5TaroSRCHMrJnCWkYww] QueryShardException[failed 
to create query: {
  "constant_score" : {
"filter" : {
  "bool" : {
"must" : [
  {
"bool" : {
  "should" : [
{
  "query_string" : {
"query" : "*",
"fields" : [ ],
"use_dis_max" : true,
"tie_breaker" : 0.0,
"default_operator" : "or",
"auto_generate_phrase_queries" : false,
"max_determinized_states" : 1,
"enable_position_increments" : true,
"fuzziness" : "AUTO",
"fuzzy_prefix_length" : 0,
"fuzzy_max_expansions" : 50,
"phrase_slop" : 0,
"escape" : false,
"split_on_whitespace" : true,
"boost" : 1.0
  }
},
{
  "nested" : {
"query" : {
  "query_string" : {
"query" : "*",
"fields" : [ ],
"use_dis_max" : true,
"tie_breaker" : 0.0,
"default_operator" : "or",
"auto_generate_phrase_queries" : false,
"max_determinized_states" : 1,
"enable_position_increments" : true,
"fuzziness" : "AUTO",
"fuzzy_prefix_length" : 0,
"fuzzy_max_expansions" : 50,
"phrase_slop" : 0,
"escape" : false,
"split_on_whitespace" : true,
"boost" : 1.0
  }
},
"path" : "alert",
"ignore_unmapped" : false,
"score_mode" : "none",
"boost" : 1.0
  }
}
  ],
  "disable_coord" : false,
  "adjust_pure_negative" : true,
  "boost" : 1.0
}
  },
  {
"bool" : {
  "should" : [
{
  "term" : {
"status" : {
  "value" : "active",
  "boost" : 1.0
}
  }
},
{
  "bool" : {
"must_not" : [
  {
"exists" : {
  "field" : "status",
  "boost" : 1.0
}
  }
],
"disable_coord" : false,
"adjust_pure_negative" : true,
"boost" : 1.0
  }
}
  ],
  "disable_coord" : false,
  "adjust_pure_negative" : true,
  "boost" : 1.0
}
  }
],
"must_not" : [
  {
"exists" : {
  "field" : "metaalerts",
  "boost" : 1.0
}
  }
],
"disable_coord" : false,
"adjust_pure_negative" : true,
"boost" : 1.0
  }
},
"boost" : 1.0
  }
}]; nested: IllegalStateException[[nested] failed to find nested object under 
path [alert]];
at 
org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:329)
at 
org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:312)
at