[jira] [Commented] (METRON-1367) Stellar should optionally instrument function calls
[ https://issues.apache.org/jira/browse/METRON-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296202#comment-16296202 ] ASF GitHub Bot commented on METRON-1367: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/873 If there are any ideas on improving the tags implementation I would love to here them > Stellar should optionally instrument function calls > --- > > Key: METRON-1367 > URL: https://issues.apache.org/jira/browse/METRON-1367 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler > > It would be nice if stellar processor(s) could optionally instrument function > calls to test performance. > The result would be some structure with the calls, the times and maybe some > context as a list or map -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1372) Validate JIRA for Releases
[ https://issues.apache.org/jira/browse/METRON-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296108#comment-16296108 ] ASF GitHub Bot commented on METRON-1372: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/874 +1 worked great > Validate JIRA for Releases > -- > > Key: METRON-1372 > URL: https://issues.apache.org/jira/browse/METRON-1372 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Minor > > Create a script that allows the Release Manager to easily validate JIRA for > the next release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1372) Validate JIRA for Releases
[
https://issues.apache.org/jira/browse/METRON-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295888#comment-16295888
]
ASF GitHub Bot commented on METRON-1372:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/874#discussion_r157632089
--- Diff: build_utils/release-utils/validate-jira-for-release ---
@@ -0,0 +1,197 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Finds all commits since the last release tag, then ensures that each
+# is marked 'Done' and that the fix version is set to the next release.
+#
+# For example, to validate JIRA for the 0.4.2 release, you would run the
+# following command.
+#
+# validate-jira-for-release --version=0.4.2
--start=tags/apache-metron-0.4.1-release
+#
+# This will output a table containing each JIRA that was inspected along
with
+# the fix version, status, and assignee. If the fix version or status is
incorrect
+# a link will be printed so that the JIRA can be manually fixed. The JIRA
+# only needs to be fixed if a URL is shown.
+#
+#JIRASTATUS FIX VERSION ASSIGNEE FIX
+# METRON-1345 Done 0.4.2 Michael Miklavcic
+# METRON-1349 DoneNext + 1 Nick Allen
https://issues.apache.org/jira/browse/METRON-1349
+# METRON-1343 Done Mohan
https://issues.apache.org/jira/browse/METRON-1343
+#...
+#
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "-v/--version= The version of the next release.
[Required]"
+ echo "-s/--start= Defines the first commit to inspect.
[Required]"
+ echo "-e/--end= Defines the last commit to inspect. "
+ echo "-r/--repo= The Git repo to work from."
+ echo "-b/--branch= The branch to work from."
+ echo "-h/--helpUsage information."
+ echo " "
+ echo "example: "
+ echo "validate-jira-for-release --version=0.4.2
--start=tags/apache-metron-0.4.1-release"
+ echo " "
+}
+
+# define default values
+END="HEAD"
+REPO="https://git-wip-us.apache.org/repos/asf/metron.git;
+BRANCH="master"
+
+# print help, if the user just runs this without any args
+if [ "$#" -eq 0 ]; then
+help
+exit 1
+fi
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+#
+# VERSION: The release version to validate; the 'next' release.
+#
+#
+-v=*|--version=*)
+VERSION="${i#*=}"
+shift # past argument=value
+;;
+
+#
+# START: Defines the first commit to inspect
+#
+# -s=tags/apache-metron-0.4.1-release
+# --start=tags/apache-metron-0.4.1-release
+#
+-s=*|--start=*)
+START="${i#*=}"
+shift # past argument=value
+;;
+
+#
+# END: Defines the last commit to inspect
+#
+# -e=HEAD
+# --end=HEAD
+#
+-e=*|--end=*)
+END="${i#*=}"
+shift # past argument=value
+;;
+
+#
+# REPO: Define the Git repo to work from
+#
+# -r=https://git-wip-us.apache.org/repos/asf/metron.git
+# --repo=
+#
+-r=*|--repo=*)
+REPO="${i#*=}"
+shift # past argument=value
+;;
+
+#
+# BRANCH: The branch to work from.
+#
+# -b=master
+# --branch=master
+#
+-b=*|--branch=*)
+BRANCH="${i#*=}"
+shift # past argument with no value
+;;
+
+#
+# -h/--help
+#
+-h|--help)
+help
+exit 0
+shift # past argument with no value
+;;
+
+#
+# Unknown option
+#
+*)
+UNKNOWN_OPTION="${i#*=}"
+
[jira] [Commented] (METRON-1372) Validate JIRA for Releases
[ https://issues.apache.org/jira/browse/METRON-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295284#comment-16295284 ] ASF GitHub Bot commented on METRON-1372: GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/874 METRON-1372 Validate JIRA for Releases Adds a script that helps the Release Manager validate JIRA for a pending release. Help is printed along with an example. ``` $ ./release-utils/validate-jira-for-release --help usage: ./release-utils/validate-jira-for-release -v/--version= The version of the next release. [Required] -s/--start= Defines the first commit to inspect. [Required] -e/--end= Defines the last commit to inspect. -r/--repo= The Git repo to work from. -b/--branch= The branch to work from. -h/--helpUsage information. example: validate-jira-for-release --version=0.4.2 --start=tags/apache-metron-0.4.1-release ``` Validating the next release. ``` $ ./release-utils/validate-jira-for-release --version=0.4.2 --start=tags/apache-metron-0.4.1-release Cloning into 'metron-0.4.2'... remote: Counting objects: 35046, done. remote: Compressing objects: 100% (13698/13698), done. remote: Total 35046 (delta 15702), reused 31650 (delta 12822) Receiving objects: 100% (35046/35046), 53.06 MiB | 6.77 MiB/s, done. Resolving deltas: 100% (15702/15702), done. Switched to branch 'master' Your branch is up to date with 'origin/master'. Fetching origin JIRA STATUS FIX VERSION ASSIGNEEFIX METRON-1345Done Michael Miklavcic https://issues.apache.org/jira/browse/METRON-1345 METRON-1349DoneNext + 1 Nick Allen https://issues.apache.org/jira/browse/METRON-1349 METRON-1343Done Mohan https://issues.apache.org/jira/browse/METRON-1343 METRON-1306 To Do Unassigned https://issues.apache.org/jira/browse/METRON-1306 METRON-1341DoneSimon Elliston Ball https://issues.apache.org/jira/browse/METRON-1341 METRON-1313Done Jon Zeolla https://issues.apache.org/jira/browse/METRON-1313 METRON-1346DoneOtto Fowler https://issues.apache.org/jira/browse/METRON-1346 METRON-1336Done 0.4.2 Nick Allen METRON-1335Done Anand Subramanian https://issues.apache.org/jira/browse/METRON-1335 METRON-1308Done Jon Zeolla https://issues.apache.org/jira/browse/METRON-1308 METRON-1338Done 0.4.2 Nick Allen METRON-1286 To Do 0.4.2 Unassigned https://issues.apache.org/jira/browse/METRON-1286 METRON-1334Done 0.4.2 Nick Allen METRON-1277DoneOtto Fowler https://issues.apache.org/jira/browse/METRON-1277 METRON-1239 To Do Unassigned https://issues.apache.org/jira/browse/METRON-1239 METRON-1328Done Anand Subramanian https://issues.apache.org/jira/browse/METRON-1328 METRON-1333DoneOtto Fowler https://issues.apache.org/jira/browse/METRON-1333 METRON-1252Done RaghuMitra https://issues.apache.org/jira/browse/METRON-1252 METRON-1316 To DoNext + 1 Unassigned https://issues.apache.org/jira/browse/METRON-1316 METRON-1088Done Jon Zeolla https://issues.apache.org/jira/browse/METRON-1088 METRON-1319 To Do Ryan Merriman https://issues.apache.org/jira/browse/METRON-1319 METRON-1321 To Do Unassigned https://issues.apache.org/jira/browse/METRON-1321 METRON-1301Done 0.4.2 Nick Allen METRON-1294 To Do Ryan Merriman https://issues.apache.org/jira/browse/METRON-1294 METRON-1291 To Do Ryan Merriman
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295255#comment-16295255 ] ASF GitHub Bot commented on METRON-939: --- Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r157540388 --- Diff: metron-platform/metron-elasticsearch/README.md --- @@ -1,5 +1,14 @@ # Elasticsearch in Metron +## Table of Contents + +* [Introduction](#introduction) +* [Properties](#properties) +* [Upgrading to 5.6.2](#upgrading-to-562) +* [Type Mappings](#type-mappings) +* [Using Metron with Elasticsearch 5.x](#using-metron-with-elasticsearch-5x) --- End diff -- This link is broken, because the section name doesn't line up with the actual name > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > Labels: backwards-incompatible > Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - > Kibana.pdf > > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295254#comment-16295254
]
ASF GitHub Bot commented on METRON-939:
---
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r157540768
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -33,7 +42,217 @@ For instance, an `es.date.format` of `.MM.dd.HH`
would have the consequence
roll hourly, whereas an `es.date.format` of `.MM.dd` would have the
consequence that the indices would
roll daily.
-## Using Metron with Elasticsearch 2.x
+## Upgrading to 5.6.2
+
+Users should be prepared to re-index when migrating from Elasticsearch
2.3.3 to 5.6.2. There are a number of template changes, most notably around
+string type handling, that may cause issues when upgrading.
+
+[https://www.elastic.co/guide/en/elasticsearch/reference/5.6/setup-upgrade.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/setup-upgrade.html)
+
+Be aware that if you add a new string value and want to be able to filter
and search on this value from the Alerts UI, you **must** add a mapping for
that type to
+the appropriate Elasticsearch template. Below is more detail on how to
choose the appropriate mapping type for your string value.
+
+## Type Mappings
+
+Type mappings have changed quite a bit from ES 2.x -> 5.x. Here is a brief
rundown of the biggest changes. More detailed references from Elasticsearch
+are provided in the [Type Mapping References](#type-mapping-references)
section below.
+* string fields replaced by text/keyword type
+* strings have new default mappings as follows
+
+```
+{
+ "type": "text",
+ "fields": {
+"keyword": {
+ "type": "keyword",
+ "ignore_above": 256
+}
+ }
+}
+```
+
+* There is no longer a `_timestamp` field that you can set "enabled" on.
This field now causes an exception on templates.
+Replace with an application-created timestamp of "date" type.
+
+The semantics for string types have changed. In 2.x, you have the concept
of index settings as either "analyzed" or "not_analyzed" which basically means
"full text" and "keyword", respectively.
+Analyzed text basically means the indexer will split the text using a text
analyzer thus allowing you to search on substrings within the original text.
"New York" is split and indexed as two buckets,
+ "New" and "York", so you can search or query for aggregate counts for
those terms independently and will match against the individual terms "New" or
"York." "Keyword" means that the original text
+ will not be split/analyzed during indexing and instead treated as a whole
unit, i.e. "New" or "York" will not match in searches against the document
containing "New York", but searching on "New York"
+ as the full city name will. In 5.x language instead of using the "index"
setting, you now set the "type" to either "text" for full text, or "keyword"
for keywords.
+
+Below is a table depicting the changes to how String types are now handled.
+
+
+
+ sort, aggregate, or access values
+ ES 2.x
+ ES 5.x
+ Example
+
+
+ no
+
+"my_property" : {
+ "type": "string",
+ "index": "analyzed"
+}
+
+
+
+"my_property" : {
+ "type": "text"
+}
+
+Additional defaults: "index": "true", "fielddata": "false"
+
+
+ "New York" handled via in-mem search as "New" and "York"
buckets. No aggregation or sort.
+
+
+
+
+ yes
+
+
+"my_property": {
+ "type": "string",
+ "index": "analyzed"
+}
+
+
+
+"my_property": {
+ "type": "text",
+ "fielddata": "true"
+}
+
+
+
+ "New York" handled via in-mem search as "New" and "York" buckets.
Can aggregate and sort.
+
+
+
+
+ yes
+
+
+"my_property": {
+ "type": "string",
+ "index": "not_analyzed"
+}
+
+
+
+"my_property" : {
+ "type": "keyword"
+}
+
+
+
+ "New York" searchable as single value. Can aggregate
and sort. A search for "New" or "York" will not match against the whole value.
+
+
+
+
+ yes
+
+
+"my_property": {
+ "type": "string",
+ "index": "analyzed"
+}
+
+
+
+"my_property": {
+ "type": "text",
+ "fields": {
+"keyword": {
+ "type": "keyword",
+ "ignore_above": 256
+}
+ }
+}
+
+
+
+ "New York"
[jira] [Created] (METRON-1372) Validate JIRA for Releases
Nick Allen created METRON-1372: -- Summary: Validate JIRA for Releases Key: METRON-1372 URL: https://issues.apache.org/jira/browse/METRON-1372 Project: Metron Issue Type: Improvement Reporter: Nick Allen Assignee: Nick Allen Priority: Minor Create a script that allows the Release Manager to easily validate JIRA for the next release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1370) Create Full Dev Equivalent for Ubuntu
[ https://issues.apache.org/jira/browse/METRON-1370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295144#comment-16295144 ] Nick Allen commented on METRON-1370: To create the Full Dev equivalent on Ubuntu, the Mpack would need to be able to deploy Metron on a Ubuntu platform. > Create Full Dev Equivalent for Ubuntu > - > > Key: METRON-1370 > URL: https://issues.apache.org/jira/browse/METRON-1370 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > With METRON-1351, we will have installable packages to make the installation > of Metron on Ubuntu far simpler. We need to create an equivalent version of > "Full Dev" that runs on Ubuntu. This will allow us to easily ensure that the > packages, along with the rest of Metron, works when run on Ubuntu. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1371) Enhance Mpack for Ubuntu Deployments
[ https://issues.apache.org/jira/browse/METRON-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295143#comment-16295143 ] Nick Allen commented on METRON-1371: Need to have installable Ubuntu packages before the Mpack can be enhanced to support Ubuntu. The Mpack would use the Ubuntu packages for installation. > Enhance Mpack for Ubuntu Deployments > > > Key: METRON-1371 > URL: https://issues.apache.org/jira/browse/METRON-1371 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > Labels: Ubuntu > > The current Mpack supports installation on CentOS and RPM-based environments > only. We should enhance the Mpack so that users can deploy on Ubuntu also. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (METRON-1371) Enhance Mpack for Ubuntu Deployments
Nick Allen created METRON-1371: -- Summary: Enhance Mpack for Ubuntu Deployments Key: METRON-1371 URL: https://issues.apache.org/jira/browse/METRON-1371 Project: Metron Issue Type: Improvement Reporter: Nick Allen Assignee: Nick Allen The current Mpack supports installation on CentOS and RPM-based environments only. We should enhance the Mpack so that users can deploy on Ubuntu also. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (METRON-1370) Create Full Dev Equivalent for Ubuntu
Nick Allen created METRON-1370: -- Summary: Create Full Dev Equivalent for Ubuntu Key: METRON-1370 URL: https://issues.apache.org/jira/browse/METRON-1370 Project: Metron Issue Type: Improvement Reporter: Nick Allen Assignee: Nick Allen With METRON-1351, we will have installable packages to make the installation of Metron on Ubuntu far simpler. We need to create an equivalent version of "Full Dev" that runs on Ubuntu. This will allow us to easily ensure that the packages, along with the rest of Metron, works when run on Ubuntu. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (METRON-1369) squid events are not seen in Alerts UI
[
https://issues.apache.org/jira/browse/METRON-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen reassigned METRON-1369:
--
Assignee: Nick Allen
> squid events are not seen in Alerts UI
> --
>
> Key: METRON-1369
> URL: https://issues.apache.org/jira/browse/METRON-1369
> Project: Metron
> Issue Type: Bug
>Reporter: Anand Subramanian
>Assignee: Nick Allen
>
> On a metron cluster, I have added the squid telemetry by following the [Wiki
> instructions|https://cwiki.apache.org/confluence/display/METRON/2016/05/02/Metron+Tutorial+-+Fundamentals+Part+4%3A+Pluggable+Threat+Intelligence].
> I am able to see the squid indices present in elasticsearch, but not able to
> see them in the alerts UI. On further checking, I found out that this is due
> to the missing nested "alert" field in the index mappings--which is required
> for the Alerts UI to pick up the index.
> This was confirmed in the rest logs as well.
> {code}
> 17/12/18 10:39:40 ERROR dao.ElasticsearchRequestSubmitter: Shard search
> failure [1/1]; reason=IllegalStateException: [nested] failed to find nested
> object under path [alert], index=squid_index_2017.12.18.10, shard=0,
> status=BAD_REQUEST, nodeId=NOjEpjy0RxCV1ltcqp4Erw
> [squid_index_2017.12.18.10/a7-5TaroSRCHMrJnCWkYww] QueryShardException[failed
> to create query: {
> "constant_score" : {
> "filter" : {
> "bool" : {
> "must" : [
> {
> "bool" : {
> "should" : [
> {
> "query_string" : {
> "query" : "*",
> "fields" : [ ],
> "use_dis_max" : true,
> "tie_breaker" : 0.0,
> "default_operator" : "or",
> "auto_generate_phrase_queries" : false,
> "max_determinized_states" : 1,
> "enable_position_increments" : true,
> "fuzziness" : "AUTO",
> "fuzzy_prefix_length" : 0,
> "fuzzy_max_expansions" : 50,
> "phrase_slop" : 0,
> "escape" : false,
> "split_on_whitespace" : true,
> "boost" : 1.0
> }
> },
> {
> "nested" : {
> "query" : {
> "query_string" : {
> "query" : "*",
> "fields" : [ ],
> "use_dis_max" : true,
> "tie_breaker" : 0.0,
> "default_operator" : "or",
> "auto_generate_phrase_queries" : false,
> "max_determinized_states" : 1,
> "enable_position_increments" : true,
> "fuzziness" : "AUTO",
> "fuzzy_prefix_length" : 0,
> "fuzzy_max_expansions" : 50,
> "phrase_slop" : 0,
> "escape" : false,
> "split_on_whitespace" : true,
> "boost" : 1.0
> }
> },
> "path" : "alert",
> "ignore_unmapped" : false,
> "score_mode" : "none",
> "boost" : 1.0
> }
> }
> ],
> "disable_coord" : false,
> "adjust_pure_negative" : true,
> "boost" : 1.0
> }
> },
> {
> "bool" : {
> "should" : [
> {
> "term" : {
> "status" : {
> "value" : "active",
> "boost" : 1.0
> }
> }
> },
> {
> "bool" : {
> "must_not" : [
> {
> "exists" : {
> "field" : "status",
> "boost" : 1.0
> }
> }
> ],
> "disable_coord" : false,
> "adjust_pure_negative" : true,
> "boost" : 1.0
> }
> }
> ],
> "disable_coord" : false,
> "adjust_pure_negative" : true,
> "boost" : 1.0
> }
> }
> ],
> "must_not" : [
> {
> "exists" : {
> "field" : "metaalerts",
> "boost" : 1.0
> }
> }
> ],
> "disable_coord" : false,
>
[jira] [Commented] (METRON-1364) Add an implementation of Robust PCA outlier detection
[ https://issues.apache.org/jira/browse/METRON-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295043#comment-16295043 ] ASF GitHub Bot commented on METRON-1364: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/870 How can we test this? > Add an implementation of Robust PCA outlier detection > - > > Key: METRON-1364 > URL: https://issues.apache.org/jira/browse/METRON-1364 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella > > With short circuiting in Stellar, we have the opportunity to delve into more > computationally intensive outlier detection techniques. Generally these > would be executed only if simpler outlier detection techniques indicated an > outlier (e.g. statistical outlier tests). > As the first one of these supported, I'd suggest a Robust PCA based technique > similar to Netflix's Surus. See > https://medium.com/netflix-techblog/rad-outlier-detection-on-big-data-d6b0494371cc > and > https://metamarkets.com/2012/algorithmic-trendspotting-the-meaning-of-interesting/ > for more detail. > It should be noted that there are some caveats with this approach around > sparsity and orderedness. > Regarding sparsity,this outlier detection algorithm presumes dense output, > which is not the case for data spanning profiles (e.g. the profiler does not > write out data every period if no data was seen). To deal with this, I am > suggesting a modification to the profiler to allow PROFILE_GET to return a > default value. That will be done in a separate JIRA. > Regarding well-orderedness, this is an outlier detector for time series data, > so it is sensitive to order to a certain extent. Given its computational > intensity, it is likely to be used with a sample of the data to shrink the > size of the data. To that end, uniform sampling is not sensible here, but > rather a biased sample for recency. Without this, you may get poor results > from this outlier detector. This sampler should be done in a separate JIRA, > but I will ensure the infrastructure to add it is contributed in METRON-1350. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1367) Stellar should optionally instrument function calls
[ https://issues.apache.org/jira/browse/METRON-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16294936#comment-16294936 ] ASF GitHub Bot commented on METRON-1367: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/873 One idea for a follow on : add a 'tag' or 'type' similar to log.INFO log.DEBUG to the records, so that you can filter the output. > Stellar should optionally instrument function calls > --- > > Key: METRON-1367 > URL: https://issues.apache.org/jira/browse/METRON-1367 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler > > It would be nice if stellar processor(s) could optionally instrument function > calls to test performance. > The result would be some structure with the calls, the times and maybe some > context as a list or map -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (METRON-1369) squid events are not seen in Alerts UI
Anand Subramanian created METRON-1369:
-
Summary: squid events are not seen in Alerts UI
Key: METRON-1369
URL: https://issues.apache.org/jira/browse/METRON-1369
Project: Metron
Issue Type: Bug
Reporter: Anand Subramanian
On a metron cluster, I have added the squid telemetry by following the [Wiki
instructions|https://cwiki.apache.org/confluence/display/METRON/2016/05/02/Metron+Tutorial+-+Fundamentals+Part+4%3A+Pluggable+Threat+Intelligence].
I am able to see the squid indices present in elasticsearch, but not able to
see them in the alerts UI. On further checking, I found out that this is due to
the missing nested "alert" field in the index mappings--which is required for
the Alerts UI to pick up the index.
This was confirmed in the rest logs as well.
{code}
17/12/18 10:39:40 ERROR dao.ElasticsearchRequestSubmitter: Shard search failure
[1/1]; reason=IllegalStateException: [nested] failed to find nested object
under path [alert], index=squid_index_2017.12.18.10, shard=0,
status=BAD_REQUEST, nodeId=NOjEpjy0RxCV1ltcqp4Erw
[squid_index_2017.12.18.10/a7-5TaroSRCHMrJnCWkYww] QueryShardException[failed
to create query: {
"constant_score" : {
"filter" : {
"bool" : {
"must" : [
{
"bool" : {
"should" : [
{
"query_string" : {
"query" : "*",
"fields" : [ ],
"use_dis_max" : true,
"tie_breaker" : 0.0,
"default_operator" : "or",
"auto_generate_phrase_queries" : false,
"max_determinized_states" : 1,
"enable_position_increments" : true,
"fuzziness" : "AUTO",
"fuzzy_prefix_length" : 0,
"fuzzy_max_expansions" : 50,
"phrase_slop" : 0,
"escape" : false,
"split_on_whitespace" : true,
"boost" : 1.0
}
},
{
"nested" : {
"query" : {
"query_string" : {
"query" : "*",
"fields" : [ ],
"use_dis_max" : true,
"tie_breaker" : 0.0,
"default_operator" : "or",
"auto_generate_phrase_queries" : false,
"max_determinized_states" : 1,
"enable_position_increments" : true,
"fuzziness" : "AUTO",
"fuzzy_prefix_length" : 0,
"fuzzy_max_expansions" : 50,
"phrase_slop" : 0,
"escape" : false,
"split_on_whitespace" : true,
"boost" : 1.0
}
},
"path" : "alert",
"ignore_unmapped" : false,
"score_mode" : "none",
"boost" : 1.0
}
}
],
"disable_coord" : false,
"adjust_pure_negative" : true,
"boost" : 1.0
}
},
{
"bool" : {
"should" : [
{
"term" : {
"status" : {
"value" : "active",
"boost" : 1.0
}
}
},
{
"bool" : {
"must_not" : [
{
"exists" : {
"field" : "status",
"boost" : 1.0
}
}
],
"disable_coord" : false,
"adjust_pure_negative" : true,
"boost" : 1.0
}
}
],
"disable_coord" : false,
"adjust_pure_negative" : true,
"boost" : 1.0
}
}
],
"must_not" : [
{
"exists" : {
"field" : "metaalerts",
"boost" : 1.0
}
}
],
"disable_coord" : false,
"adjust_pure_negative" : true,
"boost" : 1.0
}
},
"boost" : 1.0
}
}]; nested: IllegalStateException[[nested] failed to find nested object under
path [alert]];
at
org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:329)
at
org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:312)
at
