[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252820#comment-16252820
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/840
  
I was looking at that today as well, and I think that should be a follow-on
considering how large the change is moving from 2.x to 5.x. The changes
I've implemented (and Casey and Nick) for 5.x should make moving to 6.x
much easier once we're ready for it, but we should probably stabilize
master on this before another upgrade. Also, we have a number of fixes to
meta alerts that I need to merge with this branch, which will also take
some time. There are also some enhancements to the dashboard that we're
tossing around as well.

On Nov 14, 2017 6:46 PM, "ottobackwards"  wrote:

> 6.0 was released, should we consider going to that while we are at it?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252814#comment-16252814
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/840
  
6.0 was released, should we consider going to that while we are at it?


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252699#comment-16252699
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/824#discussion_r150998084
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/HBaseDao.java
 ---
@@ -138,9 +142,17 @@ private Document getDocumentFromResult(Result result) 
throws IOException {
 Map.Entry entry= columns.lastEntry();
 Long ts = Bytes.toLong(entry.getKey());
 if(entry.getValue()!= null) {
-  Map json = JSONUtils.INSTANCE.load(new 
String(entry.getValue()), new TypeReference>() {
-  });
-  return new Document(json, Bytes.toString(result.getRow()), (String) 
json.get(SOURCE_TYPE), ts);
+  Map json = JSONUtils.INSTANCE.load(new 
String(entry.getValue()),
+  new TypeReference>() {});
+  ByteArrayInputStream baos = new 
ByteArrayInputStream(result.getRow());
--- End diff --

Ok, I submitted a 
[PR](https://github.com/merrimanr/incubator-metron/pull/34) against your branch 
that abstracts it out the key into an object and adds some explicit unit tests 
around it.


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252683#comment-16252683
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/824#discussion_r150995571
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/HBaseDao.java
 ---
@@ -138,9 +142,17 @@ private Document getDocumentFromResult(Result result) 
throws IOException {
 Map.Entry entry= columns.lastEntry();
 Long ts = Bytes.toLong(entry.getKey());
 if(entry.getValue()!= null) {
-  Map json = JSONUtils.INSTANCE.load(new 
String(entry.getValue()), new TypeReference>() {
-  });
-  return new Document(json, Bytes.toString(result.getRow()), (String) 
json.get(SOURCE_TYPE), ts);
+  Map json = JSONUtils.INSTANCE.load(new 
String(entry.getValue()),
+  new TypeReference>() {});
+  ByteArrayInputStream baos = new 
ByteArrayInputStream(result.getRow());
--- End diff --

I initially tried to do this but if felt sort of awkward since we need to 
return both the guid and sensorType.  Would it make sense to just return a 
Document with those fields set and set the other fields after that method is 
called?  Or we could accept the json and ts as inputs to that method too.


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252680#comment-16252680
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/824
  
The latest commit fixes the group by bug found by @iraghumitra and 
addresses the feedback 
[above](https://github.com/apache/metron/pull/824#pullrequestreview-76466048).

The group by fix was simple.  We are now just excluding the "metaalert" 
sensor type from group by queries.

Refactoring the HBaseDao was more involved.  One of the side effects of 
storing the sensorType in the row key is that now we require a map of guids to 
sensorTypes for an IndexDao.getAllLatest call.  These means the 
IndexDao.getAllLatest interface needed to change.  It made sense to just create 
a standard interface that can be used for looking up records since it is used 
in many different places and can be confusing as to what is needed to perform 
an operation on a document.  So this PR introduces a GetRequest that has the 
following structure:
```
{
  "guid": "some guid",
  "sensorType" : " some sensor type",
  "index": "some optional index, not required"
}
```

Since now keep the sensorType in the row key it is required for a look up.  
HBase still doesn't have any concept of an index so that is kept optional.  In 
the ElasticsearchDao this is handled by using the index of a GetRequest if 
present or looking it up in Elasticsearch if not.  Now it should be clear what 
is needed to get a document since the interface is more consistent across that 
DAO methods.

Because of the GetRequest change the testing process and REST interface has 
changed slightly.  The PR description has been updated to reflect the changes.  
I am still planning on doing a fresh vagrant build but this should be ready for 
testing.


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252677#comment-16252677
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/824#discussion_r150994812
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/HBaseDao.java
 ---
@@ -138,9 +142,17 @@ private Document getDocumentFromResult(Result result) 
throws IOException {
 Map.Entry entry= columns.lastEntry();
 Long ts = Bytes.toLong(entry.getKey());
 if(entry.getValue()!= null) {
-  Map json = JSONUtils.INSTANCE.load(new 
String(entry.getValue()), new TypeReference>() {
-  });
-  return new Document(json, Bytes.toString(result.getRow()), (String) 
json.get(SOURCE_TYPE), ts);
+  Map json = JSONUtils.INSTANCE.load(new 
String(entry.getValue()),
+  new TypeReference>() {});
+  ByteArrayInputStream baos = new 
ByteArrayInputStream(result.getRow());
--- End diff --

Can we pull this out into a `readRowKey()` method, like we have a 
`buildRowKey()`?


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252666#comment-16252666
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/824
  
@justinleet looking at your results I would expect the count to go down by 
2 instead of 1:  1 for the alert that was added to the meta alert and another 
for the meta alert itself since they shouldn't be included in group by queries. 
 I think the bug @iraghumitra reported is valid.


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251768#comment-16251768
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/824
  
@iraghumitra I'm unable to duplicate the grouping on the current code 
(although I admittedly ran through a pretty basic example). Here's what I did, 
so let me know if I missed something, or you have a particular query you made.

### Find an alert
```
/api/v1/search/search

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "fields": [
"guid"
  ],
  "from": 0,
  "indices": [
"snort"
  ],
  "query": "ip_dst_addr:192.168.66.121",
  "size": 1
}' 'http://node1:8082/api/v1/search/search'
```

In this case, I just grabbed one:
```
"d29925a7-da80-49d0-b7b5-0663380d526f":"snort_index_2017.11.14.15"
```

### Retrieve that alert from findOne to get something to group by
```
curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "guid": "d29925a7-da80-49d0-b7b5-0663380d526f",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```

I'll group, arbitrarily, with:
```
"ip_dst_addr": "192.168.66.121"
```

### Run a group by
```
/api/v1/search/group

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "groups": [
{
  "field": "ip_dst_addr"
}
  ],
  "indices": [
"snort",
"metaalert"
  ],
  "query":"ip_dst_addr:192.168.66.121"
}' 'http://node1:8082/api/v1/search/group'
```

Results in:
```
{
  "groupedBy": "ip_dst_addr",
  "groupResults": [
{
  "key": "192.168.66.121",
  "total": 1299,
  "score": null
}
  ]
}
```

### Create a metaalert with that GUID
```
/api/v1/metaalert/create

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "groups": [
"test"
  ],
  "guidToIndices": {
"adf65dbb-73fd-4347-a61d-a990166fbbb1":"snort_index_2017.11.14.15"
}
}' 'http://node1:8082/api/v1/metaalert/create'
```

It returns the guid, in this case:
```
715538d9-7f4f-4316-b149-a9659df06de4
```

### (Optionally) Look up the metaalert to double check it was created as 
expected.
```
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "guid": "715538d9-7f4f-4316-b149-a9659df06de4",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```

### Run the group by from before
```
/api/v1/search/group
curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "groups": [
{
  "field": "ip_dst_addr"
}
  ],
  "indices": [
"snort",
"metaalert"
  ],
  "query":"ip_dst_addr:192.168.66.121"
}' 'http://node1:8082/api/v1/search/group'
```

Validate that the count has gone down by one (since the metaalert doesn't 
return and the child alert is now hidden:
```
{
  "groupedBy": "ip_dst_addr",
  "groupResults": [
{
  "key": "3232252537",
  "total": 1298,
  "score": null
}
  ]
}
```


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251702#comment-16251702
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/824#discussion_r150892216
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/HBaseDao.java
 ---
@@ -135,8 +138,9 @@ private Document getDocumentFromResult(Result result) 
throws IOException {
 Map.Entry entry= columns.lastEntry();
 Long ts = Bytes.toLong(entry.getKey());
 if(entry.getValue()!= null) {
-  String json = new String(entry.getValue());
-  return new Document(json, Bytes.toString(result.getRow()), null, ts);
+  Map json = JSONUtils.INSTANCE.load(new 
String(entry.getValue()), new TypeReference>() {
+  });
+  return new Document(json, Bytes.toString(result.getRow()), (String) 
json.get(SOURCE_TYPE), ts);
--- End diff --

Yeah, let's go with this. It solves several problems nicely


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251688#comment-16251688
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/824#discussion_r150889867
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/HBaseDao.java
 ---
@@ -135,8 +138,9 @@ private Document getDocumentFromResult(Result result) 
throws IOException {
 Map.Entry entry= columns.lastEntry();
 Long ts = Bytes.toLong(entry.getKey());
 if(entry.getValue()!= null) {
-  String json = new String(entry.getValue());
-  return new Document(json, Bytes.toString(result.getRow()), null, ts);
+  Map json = JSONUtils.INSTANCE.load(new 
String(entry.getValue()), new TypeReference>() {
+  });
+  return new Document(json, Bytes.toString(result.getRow()), (String) 
json.get(SOURCE_TYPE), ts);
--- End diff --

I like this solution much better.


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251675#comment-16251675
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/824#discussion_r150887809
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/HBaseDao.java
 ---
@@ -135,8 +138,9 @@ private Document getDocumentFromResult(Result result) 
throws IOException {
 Map.Entry entry= columns.lastEntry();
 Long ts = Bytes.toLong(entry.getKey());
 if(entry.getValue()!= null) {
-  String json = new String(entry.getValue());
-  return new Document(json, Bytes.toString(result.getRow()), null, ts);
+  Map json = JSONUtils.INSTANCE.load(new 
String(entry.getValue()), new TypeReference>() {
+  });
+  return new Document(json, Bytes.toString(result.getRow()), (String) 
json.get(SOURCE_TYPE), ts);
--- End diff --

The problem with the "we're going to revert after ES 5" argument is that it 
is likely not going to be immediate.  Until then we have coupled HBaseDao and 
this feels like a kludge.  

If you don't like this solution, can I propose a compromise that will 
sidestep this entirely?  How about changing the hbase key in HBaseDao to pass 
along the sensor (guid + sensor type).  This has a couple of advantages:
* It requires no parsing of the document as it can retrieve the sensor type 
from the key
* It does not couple our field name conversion from ES with HBase in any way
* It's a small change

I think this neatly sidesteps the issue.  Thoughts?


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251596#comment-16251596
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/824#discussion_r150874132
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/HBaseDao.java
 ---
@@ -135,8 +138,9 @@ private Document getDocumentFromResult(Result result) 
throws IOException {
 Map.Entry entry= columns.lastEntry();
 Long ts = Bytes.toLong(entry.getKey());
 if(entry.getValue()!= null) {
-  String json = new String(entry.getValue());
-  return new Document(json, Bytes.toString(result.getRow()), null, ts);
+  Map json = JSONUtils.INSTANCE.load(new 
String(entry.getValue()), new TypeReference>() {
+  });
+  return new Document(json, Bytes.toString(result.getRow()), (String) 
json.get(SOURCE_TYPE), ts);
--- End diff --

That seems generally useful but it's a pretty significant change (several 
files and several lines of code) plus it adds another setting to our config.  
There is no unit test for IndexConfig so it also adds untested code (or would 
require writing a new test).

I prefer a simple 2-3 line change that we can easily revert when ES 5 
upgrade makes it in.  This should be a separate PR in my opinion.  This PR is 
already very large and complicated.


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251580#comment-16251580
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/824#discussion_r150872247
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/HBaseDao.java
 ---
@@ -135,8 +138,9 @@ private Document getDocumentFromResult(Result result) 
throws IOException {
 Map.Entry entry= columns.lastEntry();
 Long ts = Bytes.toLong(entry.getKey());
 if(entry.getValue()!= null) {
-  String json = new String(entry.getValue());
-  return new Document(json, Bytes.toString(result.getRow()), null, ts);
+  Map json = JSONUtils.INSTANCE.load(new 
String(entry.getValue()), new TypeReference>() {
+  });
+  return new Document(json, Bytes.toString(result.getRow()), (String) 
json.get(SOURCE_TYPE), ts);
--- End diff --

I agree with @cestella, I'd rather use the existing abstraction. Setting a 
field transformer is a cleaner way of doing things, in my opinion, than 
requiring the user to know know and input where all the GUIDs are coming from.

Another concern about ES5 is that PR may live for awhile, even after we're 
fairly confident in it (there's going to be a lot of testing involved there). 
Casey's PR is a pretty simple way of handling it (set a config, pass it in, use 
it basically the same way you set it up).


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251562#comment-16251562
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/824#discussion_r150868283
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/HBaseDao.java
 ---
@@ -135,8 +138,9 @@ private Document getDocumentFromResult(Result result) 
throws IOException {
 Map.Entry entry= columns.lastEntry();
 Long ts = Bytes.toLong(entry.getKey());
 if(entry.getValue()!= null) {
-  String json = new String(entry.getValue());
-  return new Document(json, Bytes.toString(result.getRow()), null, ts);
+  Map json = JSONUtils.INSTANCE.load(new 
String(entry.getValue()), new TypeReference>() {
+  });
+  return new Document(json, Bytes.toString(result.getRow()), (String) 
json.get(SOURCE_TYPE), ts);
--- End diff --

I will amend my constant briefly in that I don't like the first option 
there as the major flaw in all of these designs is coupling the ES logic within 
HBase logic.  I think given this is all going away after ES 5 (IMO), we should 
probably just use the existing abstractions used in the writers to handle field 
name transformations here where we need to transform field names.


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251557#comment-16251557
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/824#discussion_r150867558
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/HBaseDao.java
 ---
@@ -135,8 +138,9 @@ private Document getDocumentFromResult(Result result) 
throws IOException {
 Map.Entry entry= columns.lastEntry();
 Long ts = Bytes.toLong(entry.getKey());
 if(entry.getValue()!= null) {
-  String json = new String(entry.getValue());
-  return new Document(json, Bytes.toString(result.getRow()), null, ts);
+  Map json = JSONUtils.INSTANCE.load(new 
String(entry.getValue()), new TypeReference>() {
+  });
+  return new Document(json, Bytes.toString(result.getRow()), (String) 
json.get(SOURCE_TYPE), ts);
--- End diff --

I submitted a PR against this PR with my proposed solution of coupling the 
HBaseDao with the Elasticsearch specific field name transformation logic.  I 
propose that we do this:
* Migrate the FieldNameConverter as a field in the REST config
* Make the IndexDao take the FieldNameConverter in the AccessConfig
* Have the HBaseDao use the parameterized field name converter to transform 
`source.type` to the appropriate field.


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251549#comment-16251549
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/824#discussion_r150866003
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/HBaseDao.java
 ---
@@ -135,8 +138,9 @@ private Document getDocumentFromResult(Result result) 
throws IOException {
 Map.Entry entry= columns.lastEntry();
 Long ts = Bytes.toLong(entry.getKey());
 if(entry.getValue()!= null) {
-  String json = new String(entry.getValue());
-  return new Document(json, Bytes.toString(result.getRow()), null, ts);
+  Map json = JSONUtils.INSTANCE.load(new 
String(entry.getValue()), new TypeReference>() {
+  });
+  return new Document(json, Bytes.toString(result.getRow()), (String) 
json.get(SOURCE_TYPE), ts);
--- End diff --

Option 1 is not possible the way things are now.  This constant is already 
in an ES specific class in metron-elasticsearch but metron-elasticsearch 
depends on metron-indexing.  To get access to that constant we would need to 
add metron-elasticsearch as a dependency to metron-indexing thus creating a 
circular dependency.

Option 2 would be my least favorite option and I would rather just change 
the method signature of getAllLatest to include all guid/sensorType 
relationships.

Since we currently have a PR in review for the ES 5 upgrade that will allow 
us to just remove this constant, I don't feel like we should spend much time on 
it.  I would also argue that this was an issue long before this PR so anything 
more than a simple workaround should be a follow-on.


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251489#comment-16251489
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/824
  
@iraghumitra Do you have the specific metaalerts and calls that you made?  
I'm spinning this up again, but it'll be a bit before I can test something 
myself.


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251487#comment-16251487
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/824#discussion_r150854643
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/HBaseDao.java
 ---
@@ -135,8 +138,9 @@ private Document getDocumentFromResult(Result result) 
throws IOException {
 Map.Entry entry= columns.lastEntry();
 Long ts = Bytes.toLong(entry.getKey());
 if(entry.getValue()!= null) {
-  String json = new String(entry.getValue());
-  return new Document(json, Bytes.toString(result.getRow()), null, ts);
+  Map json = JSONUtils.INSTANCE.load(new 
String(entry.getValue()), new TypeReference>() {
+  });
+  return new Document(json, Bytes.toString(result.getRow()), (String) 
json.get(SOURCE_TYPE), ts);
--- End diff --

I would prefer to see one of two things happen here. Either we keep the 
constant in the ES specific classes (which is admittedly less than ideal, but 
it does limit the pollution of ES knowledge into HBase classes) and populate 
source type from there (basically moving the loading and source type population 
there).  Alternatively, we pass in a more general function that can be applied 
to the fields and configure and handle it appropriately.

I think the second one is probably more general useful to be able to do, 
but given the state of ES5 upgrade making this particular case obsolete, I'm 
amenable to doing the first option.

At bare minimum we should replace the '.'s with ':'s only if present.  Even 
if there's not a Solr implementation, I don't want HBaseDao tied to ES so 
directly.

@cestella Do you have a preference on implementation?  I know you'd had 
some comments earlier, but I don't want to put words in your mouth.


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (METRON-1307) Force install of java8 since java9 does not appear to work with the scripts

[Otto Fowler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otto Fowler reassigned METRON-1307:
---

Assignee: (was: Otto Fowler)

> Force install of java8 since java9 does not appear to work with the scripts
> ---
>
> Key: METRON-1307
> URL: https://issues.apache.org/jira/browse/METRON-1307
> Project: Metron
>  Issue Type: Bug
>Reporter: Brian Hurley
>Priority: Blocker
>
> Install script for dev instance does not work with Java9 (released in Sept 
> 2017 and pulled in by default), the change will force install of Java8 which 
> it does work with.
> - brew cask install vagrant virtualbox java docker
> + brew cask install vagrant virtualbox docker
> + brew cask install caskroom/versions/java8



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (METRON-1307) Force install of java8 since java9 does not appear to work with the scripts

[Otto Fowler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otto Fowler reassigned METRON-1307:
---

Assignee: Otto Fowler

> Force install of java8 since java9 does not appear to work with the scripts
> ---
>
> Key: METRON-1307
> URL: https://issues.apache.org/jira/browse/METRON-1307
> Project: Metron
>  Issue Type: Bug
>Reporter: Brian Hurley
>Assignee: Otto Fowler
>Priority: Blocker
>
> Install script for dev instance does not work with Java9 (released in Sept 
> 2017 and pulled in by default), the change will force install of Java8 which 
> it does work with.
> - brew cask install vagrant virtualbox java docker
> + brew cask install vagrant virtualbox docker
> + brew cask install caskroom/versions/java8



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1036) Stellar log function

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251460#comment-16251460
 ] 

Simon Elliston Ball commented on METRON-1036:
-

Yes. Looks like they got rolled up into that ticket. I’ll close this thanks for 
the cleanup Jon.

> Stellar log function
> 
>
> Key: METRON-1036
> URL: https://issues.apache.org/jira/browse/METRON-1036
> Project: Metron
>  Issue Type: Improvement
>Reporter: Simon Elliston Ball
>Priority: Critical
>
> Stellar does not currently have a natural log, or other base log function.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251406#comment-16251406
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/824
  
@merrimanr @justinleet tested the new API's they are working fine the only 
issue I see is 
>>When a meta alert is created the meta alert is returned as part of the 
group api which was not the case before.


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1036) Stellar log function

[Jon Zeolla (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251405#comment-16251405
 ] 

Jon Zeolla commented on METRON-1036:


Aren't these already implemented?

https://github.com/apache/metron/blob/b2375a1f1c66b19dc6a82cbcc8242741cdf65451/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/math/MathOperations.java#L30-L32
https://github.com/apache/metron/blob/cc84b6a50620447959c2ded4957a88ab329ba18c/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/MathFunctions.java#L51-L91


> Stellar log function
> 
>
> Key: METRON-1036
> URL: https://issues.apache.org/jira/browse/METRON-1036
> Project: Metron
>  Issue Type: Improvement
>Reporter: Simon Elliston Ball
>Priority: Critical
>
> Stellar does not currently have a natural log, or other base log function.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (METRON-813) Migrate metron-bro-plugin-kafka to be a bro package

[Jon Zeolla (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Zeolla updated METRON-813:
--
Summary: Migrate metron-bro-plugin-kafka to be a bro package  (was: Migrate 
bro-plugin-kafka to be a bro package)

> Migrate metron-bro-plugin-kafka to be a bro package
> ---
>
> Key: METRON-813
> URL: https://issues.apache.org/jira/browse/METRON-813
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> Per a 
> [discussion](https://lists.apache.org/thread.html/c92acd125dae05f0537d4505e0254dfa6382ca9f40edba7d2f4c6224@%3Cdev.metron.apache.org%3E)
>  on the dev mailing list, the kafka plugin should be hosted as a [bro 
> package](https://github.com/bro/packages) and mirrored to 
> https://github.com/apache/incubator-metron-bro-plugin-kafka.  The bro kafka 
> plugin should be installed, when necessary, using the bro package.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1303) Reorganize the metron-bro-plugin-kafka

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251322#comment-16251322
 ] 

ASF GitHub Bot commented on METRON-1303:


Github user asfgit closed the pull request at:

https://github.com/apache/metron-bro-plugin-kafka/pull/1


> Reorganize the metron-bro-plugin-kafka
> --
>
> Key: METRON-1303
> URL: https://issues.apache.org/jira/browse/METRON-1303
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> Currently the metron-bro-plugin-kafka plugin gets installed as Bro::Kafka, 
> which is somewhat confusing based on the way bro packages are currently 
> named/sorted[1].  Based on this, we should align the plugin namespace to run 
> under the owner of the repo, which is apache.
> 1:  https://github.com/bro/packages



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)