date:20180823

[GitHub] metron pull request #1172: METRON-1724: Date/time validation missing in PCAP...

2018-08-23 Thread ruffle1986

Github user ruffle1986 commented on a diff in the pull request:

https://github.com/apache/metron/pull/1172#discussion_r212204263
  
--- Diff: metron-interface/metron-alerts/src/app/pcap/model/pcap.request.ts 
---
@@ -17,13 +17,13 @@
  */
 
 export class PcapRequest {
-  startTimeMs: number = 0;
-  endTimeMs: number = 15;
-  ipSrcAddr: string = '';
-  ipSrcPort: number;
-  ipDstAddr: string = '';
-  ipDstPort: number;
-  protocol: string = '';
-  packetFilter: string = '';
-  includeReverse: boolean = false;
+  startTimeMs = 0;
+  endTimeMs = 15;
--- End diff --

It's not necessary to set these values here but I'm afraid the server 
relies on it as default values. So it might crash if we send it undefined 
values instead of magic numbers.


---

[jira] [Commented] (METRON-1724) Date/time validation missing in PCAP query

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589817#comment-16589817
 ] 

ASF GitHub Bot commented on METRON-1724:


Github user ruffle1986 commented on a diff in the pull request:

https://github.com/apache/metron/pull/1172#discussion_r212204263
  
--- Diff: metron-interface/metron-alerts/src/app/pcap/model/pcap.request.ts 
---
@@ -17,13 +17,13 @@
  */
 
 export class PcapRequest {
-  startTimeMs: number = 0;
-  endTimeMs: number = 15;
-  ipSrcAddr: string = '';
-  ipSrcPort: number;
-  ipDstAddr: string = '';
-  ipDstPort: number;
-  protocol: string = '';
-  packetFilter: string = '';
-  includeReverse: boolean = false;
+  startTimeMs = 0;
+  endTimeMs = 15;
--- End diff --

It's not necessary to set these values here but I'm afraid the server 
relies on it as default values. So it might crash if we send it undefined 
values instead of magic numbers.


> Date/time validation missing in PCAP query
> --
>
> Key: METRON-1724
> URL: https://issues.apache.org/jira/browse/METRON-1724
> Project: Metron
>  Issue Type: Bug
>Reporter: Tibor Meller
>Priority: Major
>
> Validation formula should be the following: 
>  From < To < current date/time
>  
> Validation messages:
> Selected date range is invalid. The "To" date must be later than the "From" 
> date and the "To" date cannot be in the future.
> Source IP address format is invalid. Use valid v4IP format, for example, 
> [192.168.0.1|http://192.168.0.1/].
> Source port is invalid. Port number must be within the range of 0-65535.
> Destination IP address format is invalid. Use valid v4IP format, for example, 
> [192.168.0.1|http://192.168.0.1/].
> Destination port is invalid. Port number must be within the range of 0-65535.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1172: METRON-1724: Date/time validation missing in PCAP...

2018-08-23 Thread ruffle1986

Github user ruffle1986 commented on a diff in the pull request:

https://github.com/apache/metron/pull/1172#discussion_r212205052
  
--- Diff: metron-interface/metron-alerts/src/app/pcap/model/pcap.request.ts 
---
@@ -17,13 +17,13 @@
  */
 
 export class PcapRequest {
-  startTimeMs: number = 0;
-  endTimeMs: number = 15;
-  ipSrcAddr: string = '';
-  ipSrcPort: number;
-  ipDstAddr: string = '';
-  ipDstPort: number;
-  protocol: string = '';
-  packetFilter: string = '';
-  includeReverse: boolean = false;
+  startTimeMs = 0;
+  endTimeMs = 15;
--- End diff --

but, again. it's the type of the request which is fine. we don't have to 
use the same values to set the state of the filter ui component. it can be a 
totally different type. When it comes to sending the request to the server (or 
getting a response from it), this is the proper place where we're supposed to 
deal with request types.


---

[jira] [Commented] (METRON-1724) Date/time validation missing in PCAP query

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589819#comment-16589819
 ] 

ASF GitHub Bot commented on METRON-1724:


Github user ruffle1986 commented on a diff in the pull request:

https://github.com/apache/metron/pull/1172#discussion_r212205052
  
--- Diff: metron-interface/metron-alerts/src/app/pcap/model/pcap.request.ts 
---
@@ -17,13 +17,13 @@
  */
 
 export class PcapRequest {
-  startTimeMs: number = 0;
-  endTimeMs: number = 15;
-  ipSrcAddr: string = '';
-  ipSrcPort: number;
-  ipDstAddr: string = '';
-  ipDstPort: number;
-  protocol: string = '';
-  packetFilter: string = '';
-  includeReverse: boolean = false;
+  startTimeMs = 0;
+  endTimeMs = 15;
--- End diff --

but, again. it's the type of the request which is fine. we don't have to 
use the same values to set the state of the filter ui component. it can be a 
totally different type. When it comes to sending the request to the server (or 
getting a response from it), this is the proper place where we're supposed to 
deal with request types.


> Date/time validation missing in PCAP query
> --
>
> Key: METRON-1724
> URL: https://issues.apache.org/jira/browse/METRON-1724
> Project: Metron
>  Issue Type: Bug
>Reporter: Tibor Meller
>Priority: Major
>
> Validation formula should be the following: 
>  From < To < current date/time
>  
> Validation messages:
> Selected date range is invalid. The "To" date must be later than the "From" 
> date and the "To" date cannot be in the future.
> Source IP address format is invalid. Use valid v4IP format, for example, 
> [192.168.0.1|http://192.168.0.1/].
> Source port is invalid. Port number must be within the range of 0-65535.
> Destination IP address format is invalid. Use valid v4IP format, for example, 
> [192.168.0.1|http://192.168.0.1/].
> Destination port is invalid. Port number must be within the range of 0-65535.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1172: METRON-1724: Date/time validation missing in PCAP...

2018-08-23 Thread ruffle1986

Github user ruffle1986 commented on a diff in the pull request:

https://github.com/apache/metron/pull/1172#discussion_r212205588
  
--- Diff: 
metron-interface/metron-alerts/src/app/pcap/pcap-filters/pcap-filters.component.ts
 ---
@@ -15,63 +15,116 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-import {Component, Input, Output, EventEmitter, OnInit, OnChanges, 
SimpleChanges} from '@angular/core';
+import {Component, Input, Output, EventEmitter, OnChanges, SimpleChanges} 
from '@angular/core';
+import { FormGroup, FormControl, Validators, ValidationErrors } from 
'@angular/forms';
+
 import * as moment from 'moment/moment';
 import { DEFAULT_TIMESTAMP_FORMAT } from '../../utils/constants';
 
 import { PcapRequest } from '../model/pcap.request';
 
+const DEFAULT_END_TIME = new Date();
--- End diff --

Yes, probably that would be the best because somehow, this request related 
stuff doesn't want to fit in with the filter component :D


---

[jira] [Commented] (METRON-1724) Date/time validation missing in PCAP query

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589824#comment-16589824
 ] 

ASF GitHub Bot commented on METRON-1724:


Github user ruffle1986 commented on a diff in the pull request:

https://github.com/apache/metron/pull/1172#discussion_r212205588
  
--- Diff: 
metron-interface/metron-alerts/src/app/pcap/pcap-filters/pcap-filters.component.ts
 ---
@@ -15,63 +15,116 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-import {Component, Input, Output, EventEmitter, OnInit, OnChanges, 
SimpleChanges} from '@angular/core';
+import {Component, Input, Output, EventEmitter, OnChanges, SimpleChanges} 
from '@angular/core';
+import { FormGroup, FormControl, Validators, ValidationErrors } from 
'@angular/forms';
+
 import * as moment from 'moment/moment';
 import { DEFAULT_TIMESTAMP_FORMAT } from '../../utils/constants';
 
 import { PcapRequest } from '../model/pcap.request';
 
+const DEFAULT_END_TIME = new Date();
--- End diff --

Yes, probably that would be the best because somehow, this request related 
stuff doesn't want to fit in with the filter component :D


> Date/time validation missing in PCAP query
> --
>
> Key: METRON-1724
> URL: https://issues.apache.org/jira/browse/METRON-1724
> Project: Metron
>  Issue Type: Bug
>Reporter: Tibor Meller
>Priority: Major
>
> Validation formula should be the following: 
>  From < To < current date/time
>  
> Validation messages:
> Selected date range is invalid. The "To" date must be later than the "From" 
> date and the "To" date cannot be in the future.
> Source IP address format is invalid. Use valid v4IP format, for example, 
> [192.168.0.1|http://192.168.0.1/].
> Source port is invalid. Port number must be within the range of 0-65535.
> Destination IP address format is invalid. Use valid v4IP format, for example, 
> [192.168.0.1|http://192.168.0.1/].
> Destination port is invalid. Port number must be within the range of 0-65535.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1111: METRON-1665 Host UIs on Spring Boot and add SSO authenti...

2018-08-23 Thread simonellistonball

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/
  
Thanks for the catches Ryan. 

My objective around metron-rest was to try and leave it as little changed 
as possible in this effort, since the ticket is more around the ui hosts, but I 
would agree that this could be cleaned up as you suggest. 

On the issue of Roles, I don't think we really make the best use of roles 
at present, and need a follow on to sort all that out. There is also some 
debate to be had about how roles should be used. A trivial (and in my view 
naive) view would be to map roles to LDAP groups. This is quite a common 
approach, but is also a wrong approach, Roles are logical groups of capability, 
not groups of user principals. I would like to see a follow on where we 
actually define and properly apply concepts of roles to access controlled 
objects in Metron, but in the short term we could definitely use the 
placeholder TODO here to add a naive mapping.


---

[jira] [Commented] (METRON-1665) Move hosting of Alerts and Config UIs from Nodejs to Spring Boot

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589896#comment-16589896
 ] 

ASF GitHub Bot commented on METRON-1665:


Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/
  
Thanks for the catches Ryan. 

My objective around metron-rest was to try and leave it as little changed 
as possible in this effort, since the ticket is more around the ui hosts, but I 
would agree that this could be cleaned up as you suggest. 

On the issue of Roles, I don't think we really make the best use of roles 
at present, and need a follow on to sort all that out. There is also some 
debate to be had about how roles should be used. A trivial (and in my view 
naive) view would be to map roles to LDAP groups. This is quite a common 
approach, but is also a wrong approach, Roles are logical groups of capability, 
not groups of user principals. I would like to see a follow on where we 
actually define and properly apply concepts of roles to access controlled 
objects in Metron, but in the short term we could definitely use the 
placeholder TODO here to add a naive mapping.


> Move hosting of Alerts and Config UIs from Nodejs to Spring Boot
> 
>
> Key: METRON-1665
> URL: https://issues.apache.org/jira/browse/METRON-1665
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Simon Elliston Ball
>Assignee: Simon Elliston Ball
>Priority: Major
>
> The current UIs are served up by very lightweight nodejs applications, which 
> serve the static bundle files produced by the angular build process, and 
> proxies the rest api.
> The proposal is to use a spring boot application, allowing us to harmonise 
> the security implementation across the UI static servers and the REST layer, 
> and to provide a routing platform for later microservices.
> The UIs currently proxy to the REST API to avoid CORS issues, this will be 
> achieved with Zuul.
> Spring Security will also be extended to use a Knox SSO authenticator. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1111: METRON-1665 Host UIs on Spring Boot and add SSO a...

2018-08-23 Thread simonellistonball

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/#discussion_r212225176
  
--- Diff: metron-interface/metron-ui-host/pom.xml ---
@@ -0,0 +1,142 @@
+
+
+http://maven.apache.org/POM/4.0.0";
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+metron-ui-host
+jar
+
+Metron Generic UI Host
+Spring Server to host config ui
+
+
+org.apache.metron
+metron-interface
+0.5.1
+
+
+
--- End diff --

Which properties do you mean? I can see a case for the very general ones 
for sure, but the spring cloud and jwt versions could very much differ from 
service to service. Should those live in the metron-ui-host and 
metron-ui-security project respectively you think? 


---

[jira] [Commented] (METRON-1665) Move hosting of Alerts and Config UIs from Nodejs to Spring Boot

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590090#comment-16590090
 ] 

ASF GitHub Bot commented on METRON-1665:


Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/#discussion_r212225176
  
--- Diff: metron-interface/metron-ui-host/pom.xml ---
@@ -0,0 +1,142 @@
+
+
+http://maven.apache.org/POM/4.0.0";
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+metron-ui-host
+jar
+
+Metron Generic UI Host
+Spring Server to host config ui
+
+
+org.apache.metron
+metron-interface
+0.5.1
+
+
+
--- End diff --

Which properties do you mean? I can see a case for the very general ones 
for sure, but the spring cloud and jwt versions could very much differ from 
service to service. Should those live in the metron-ui-host and 
metron-ui-security project respectively you think? 


> Move hosting of Alerts and Config UIs from Nodejs to Spring Boot
> 
>
> Key: METRON-1665
> URL: https://issues.apache.org/jira/browse/METRON-1665
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Simon Elliston Ball
>Assignee: Simon Elliston Ball
>Priority: Major
>
> The current UIs are served up by very lightweight nodejs applications, which 
> serve the static bundle files produced by the angular build process, and 
> proxies the rest api.
> The proposal is to use a spring boot application, allowing us to harmonise 
> the security implementation across the UI static servers and the REST layer, 
> and to provide a routing platform for later microservices.
> The UIs currently proxy to the REST API to avoid CORS issues, this will be 
> achieved with Zuul.
> Spring Security will also be extended to use a Knox SSO authenticator. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1111: METRON-1665 Host UIs on Spring Boot and add SSO a...

2018-08-23 Thread simonellistonball

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/#discussion_r212270804
  
--- Diff: metron-interface/metron-rest-client/bin/pom.xml ---
@@ -0,0 +1,52 @@
+
--- End diff --

Yes. Will remove.


---

[jira] [Commented] (METRON-1665) Move hosting of Alerts and Config UIs from Nodejs to Spring Boot

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590093#comment-16590093
 ] 

ASF GitHub Bot commented on METRON-1665:


Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/#discussion_r212270804
  
--- Diff: metron-interface/metron-rest-client/bin/pom.xml ---
@@ -0,0 +1,52 @@
+
--- End diff --

Yes. Will remove.


> Move hosting of Alerts and Config UIs from Nodejs to Spring Boot
> 
>
> Key: METRON-1665
> URL: https://issues.apache.org/jira/browse/METRON-1665
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Simon Elliston Ball
>Assignee: Simon Elliston Ball
>Priority: Major
>
> The current UIs are served up by very lightweight nodejs applications, which 
> serve the static bundle files produced by the angular build process, and 
> proxies the rest api.
> The proposal is to use a spring boot application, allowing us to harmonise 
> the security implementation across the UI static servers and the REST layer, 
> and to provide a routing platform for later microservices.
> The UIs currently proxy to the REST API to avoid CORS issues, this will be 
> achieved with Zuul.
> Spring Security will also be extended to use a Knox SSO authenticator. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1111: METRON-1665 Host UIs on Spring Boot and add SSO a...

2018-08-23 Thread simonellistonball

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/#discussion_r212275406
  
--- Diff: 
metron-interface/metron-alerts-host/src/main/scripts/metron-alerts.sh ---
@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+if [ -z "${METRON_SSL_PASSWORD}" ]; then
+echo "METRON_SSL_PASSWORD unset."
+fi
+
+METRON_VERSION=${project.version}
+METRON_HOME="${METRON_HOME:-/usr/metron/${METRON_VERSION}}"
+METRON_SYSCONFIG="${METRON_SYSCONFIG:-/etc/default/metron}"
+
+echo "METRON_VERSION=${METRON_VERSION}"
+echo "METRON_HOME=${METRON_HOME}"
+echo "METRON_SYSCONFIG=${METRON_SYSCONFIG}"
+
+if [ -f "$METRON_SYSCONFIG" ]; then
+echo "METRON_SYSCONFIG=${METRON_SYSCONFIG}"
+set -a
+. "$METRON_SYSCONFIG"
+fi
+
+echo "METRON_SPRING_PROFILES_ACTIVE=${METRON_SPRING_PROFILES_ACTIVE}"
+
+METRON_CONFIG_LOCATION=" 
--spring.config.location=classpath:/application.yml,$METRON_HOME/config/alerts_ui.yml"
+echo "METRON_CONFIG_LOCATION=${METRON_CONFIG_LOCATION}"
+METRON_SPRING_OPTIONS+=${METRON_CONFIG_LOCATION}
--- End diff --

They're actually in different environments, so didn't feel the need to 
change the convention. These scripts take all their environment from Ambari 
launchers in a subsequent PR, rather than the /etc/defaults approach, so there 
is no actual collision. I can see an argument for changing them all for 
clarity, but could also see the argument for consistency in the context of 
potentially growing number of services like this. 


---

[jira] [Commented] (METRON-1665) Move hosting of Alerts and Config UIs from Nodejs to Spring Boot

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590116#comment-16590116
 ] 

ASF GitHub Bot commented on METRON-1665:


Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/#discussion_r212275406
  
--- Diff: 
metron-interface/metron-alerts-host/src/main/scripts/metron-alerts.sh ---
@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+if [ -z "${METRON_SSL_PASSWORD}" ]; then
+echo "METRON_SSL_PASSWORD unset."
+fi
+
+METRON_VERSION=${project.version}
+METRON_HOME="${METRON_HOME:-/usr/metron/${METRON_VERSION}}"
+METRON_SYSCONFIG="${METRON_SYSCONFIG:-/etc/default/metron}"
+
+echo "METRON_VERSION=${METRON_VERSION}"
+echo "METRON_HOME=${METRON_HOME}"
+echo "METRON_SYSCONFIG=${METRON_SYSCONFIG}"
+
+if [ -f "$METRON_SYSCONFIG" ]; then
+echo "METRON_SYSCONFIG=${METRON_SYSCONFIG}"
+set -a
+. "$METRON_SYSCONFIG"
+fi
+
+echo "METRON_SPRING_PROFILES_ACTIVE=${METRON_SPRING_PROFILES_ACTIVE}"
+
+METRON_CONFIG_LOCATION=" 
--spring.config.location=classpath:/application.yml,$METRON_HOME/config/alerts_ui.yml"
+echo "METRON_CONFIG_LOCATION=${METRON_CONFIG_LOCATION}"
+METRON_SPRING_OPTIONS+=${METRON_CONFIG_LOCATION}
--- End diff --

They're actually in different environments, so didn't feel the need to 
change the convention. These scripts take all their environment from Ambari 
launchers in a subsequent PR, rather than the /etc/defaults approach, so there 
is no actual collision. I can see an argument for changing them all for 
clarity, but could also see the argument for consistency in the context of 
potentially growing number of services like this. 


> Move hosting of Alerts and Config UIs from Nodejs to Spring Boot
> 
>
> Key: METRON-1665
> URL: https://issues.apache.org/jira/browse/METRON-1665
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Simon Elliston Ball
>Assignee: Simon Elliston Ball
>Priority: Major
>
> The current UIs are served up by very lightweight nodejs applications, which 
> serve the static bundle files produced by the angular build process, and 
> proxies the rest api.
> The proposal is to use a spring boot application, allowing us to harmonise 
> the security implementation across the UI static servers and the REST layer, 
> and to provide a routing platform for later microservices.
> The UIs currently proxy to the REST API to avoid CORS issues, this will be 
> achieved with Zuul.
> Spring Security will also be extended to use a Knox SSO authenticator. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1014: METRON-1563 : Base Stellar assign for feature branch

2018-08-23 Thread JonZeolla

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/1014
  
I wonder if it's good enough just to keep `:=` and `=` around, but not have 
that affect the format of `++`, `+=`, etc.


---

[jira] [Commented] (METRON-1563) Initial Feature Branch Work

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590166#comment-16590166
 ] 

ASF GitHub Bot commented on METRON-1563:


Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/1014
  
I wonder if it's good enough just to keep `:=` and `=` around, but not have 
that affect the format of `++`, `+=`, etc.


> Initial Feature Branch Work
> ---
>
> Key: METRON-1563
> URL: https://issues.apache.org/jira/browse/METRON-1563
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1172: METRON-1724: Date/time validation missing in PCAP query

2018-08-23 Thread tiborm

Github user tiborm commented on the issue:

https://github.com/apache/metron/pull/1172
  
@sardell @ruffle1986 I made the following changes based on your feedback:

- default values for the timestamps removed
- DEFAULT_START_TIME and DEFAULT_END_TIME moved to constants.ts 
(theoretically these values are reusable)
- transform functions moved to utils.ts to make the filter component cleaner

I think this is cleaner and more aligned with the other parts of the code.


---

[jira] [Commented] (METRON-1724) Date/time validation missing in PCAP query

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590192#comment-16590192
 ] 

ASF GitHub Bot commented on METRON-1724:


Github user tiborm commented on the issue:

https://github.com/apache/metron/pull/1172
  
@sardell @ruffle1986 I made the following changes based on your feedback:

- default values for the timestamps removed
- DEFAULT_START_TIME and DEFAULT_END_TIME moved to constants.ts 
(theoretically these values are reusable)
- transform functions moved to utils.ts to make the filter component cleaner

I think this is cleaner and more aligned with the other parts of the code.


> Date/time validation missing in PCAP query
> --
>
> Key: METRON-1724
> URL: https://issues.apache.org/jira/browse/METRON-1724
> Project: Metron
>  Issue Type: Bug
>Reporter: Tibor Meller
>Priority: Major
>
> Validation formula should be the following: 
>  From < To < current date/time
>  
> Validation messages:
> Selected date range is invalid. The "To" date must be later than the "From" 
> date and the "To" date cannot be in the future.
> Source IP address format is invalid. Use valid v4IP format, for example, 
> [192.168.0.1|http://192.168.0.1/].
> Source port is invalid. Port number must be within the range of 0-65535.
> Destination IP address format is invalid. Use valid v4IP format, for example, 
> [192.168.0.1|http://192.168.0.1/].
> Destination port is invalid. Port number must be within the range of 0-65535.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1743) CEF test confusing URL

2018-08-23 Thread Jon Zeolla (JIRA)

Jon Zeolla created METRON-1743:
--

 Summary: CEF test confusing URL
 Key: METRON-1743
 URL: https://issues.apache.org/jira/browse/METRON-1743
 Project: Metron
  Issue Type: Sub-task
Reporter: Jon Zeolla


A confusing test URL is used 
here




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1744) CEF testPaloAltoCEF test using a confusing variable name

2018-08-23 Thread Jon Zeolla (JIRA)

Jon Zeolla created METRON-1744:
--

 Summary: CEF testPaloAltoCEF test using a confusing variable name
 Key: METRON-1744
 URL: https://issues.apache.org/jira/browse/METRON-1744
 Project: Metron
  Issue Type: Sub-task
Reporter: Jon Zeolla


A confusing test URL 
here



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1745) CEF runTest test isn't properly leveraging an explicit json sample

2018-08-23 Thread Jon Zeolla (JIRA)

Jon Zeolla created METRON-1745:
--

 Summary: CEF runTest test isn't properly leveraging an explicit 
json sample
 Key: METRON-1745
 URL: https://issues.apache.org/jira/browse/METRON-1745
 Project: Metron
  Issue Type: Sub-task
Reporter: Jon Zeolla


https://github.com/apache/metron/blob/9fdccba371c1f1e0dcb79e00a7207a934b79b64c/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/cef/CEFParserTest.java#L225-L228



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1742) Improve CEF log management throughout

2018-08-23 Thread Jon Zeolla (JIRA)

Jon Zeolla created METRON-1742:
--

 Summary: Improve CEF log management throughout
 Key: METRON-1742
 URL: https://issues.apache.org/jira/browse/METRON-1742
 Project: Metron
  Issue Type: Improvement
Reporter: Jon Zeolla


There are numerous potential improvements to the CEF parser.

* A confusing test URL 
here
* A noop test 
here
* An unclear test based on the prior bullet 
here
* A lack of an ES template 
here
* A lack of integration tests 
here



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-1745) CEF runTest test isn't properly leveraging an explicit json sample

2018-08-23 Thread Jon Zeolla (JIRA)



 [ 
https://issues.apache.org/jira/browse/METRON-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Zeolla updated METRON-1745:
---
Description: 
https://github.com/apache/metron/blob/9fdccba371c1f1e0dcb79e00a7207a934b79b64c/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/cef/CEFParserTest.java#L225-L228

This also makes 
this
 confusing, as it is properly passed but unused.

  
was:https://github.com/apache/metron/blob/9fdccba371c1f1e0dcb79e00a7207a934b79b64c/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/cef/CEFParserTest.java#L225-L228


> CEF runTest test isn't properly leveraging an explicit json sample
> --
>
> Key: METRON-1745
> URL: https://issues.apache.org/jira/browse/METRON-1745
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Jon Zeolla
>Priority: Minor
>
> https://github.com/apache/metron/blob/9fdccba371c1f1e0dcb79e00a7207a934b79b64c/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/cef/CEFParserTest.java#L225-L228
> This also makes 
> this
>  confusing, as it is properly passed but unused.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1747) CEF has no integration tests

2018-08-23 Thread Jon Zeolla (JIRA)

Jon Zeolla created METRON-1747:
--

 Summary: CEF has no integration tests
 Key: METRON-1747
 URL: https://issues.apache.org/jira/browse/METRON-1747
 Project: Metron
  Issue Type: Sub-task
Reporter: Jon Zeolla


A lack of integration tests is shown 
here



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1746) CEF lacks an ES template

2018-08-23 Thread Jon Zeolla (JIRA)

Jon Zeolla created METRON-1746:
--

 Summary: CEF lacks an ES template
 Key: METRON-1746
 URL: https://issues.apache.org/jira/browse/METRON-1746
 Project: Metron
  Issue Type: Sub-task
Reporter: Jon Zeolla


An ES template should exist 
here

If you only pass in CEF data, the alerts UI will not be able to display 
anything, and ES will return a 500 saying "Fielddata is disabled on text fields 
by default."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-1742) Improve CEF log management throughout

2018-08-23 Thread Jon Zeolla (JIRA)



 [ 
https://issues.apache.org/jira/browse/METRON-1742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Zeolla updated METRON-1742:
---
Priority: Minor  (was: Major)

> Improve CEF log management throughout
> -
>
> Key: METRON-1742
> URL: https://issues.apache.org/jira/browse/METRON-1742
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Priority: Minor
>
> There are numerous potential improvements to the CEF parser.
> * A confusing test URL 
> here
> * A noop test 
> here
> * An unclear test based on the prior bullet 
> here
> * A lack of an ES template 
> here
> * A lack of integration tests 
> here



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1172: METRON-1724: Date/time validation missing in PCAP...

2018-08-23 Thread merrimanr

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/1172#discussion_r212341861
  
--- Diff: 
metron-interface/metron-alerts/src/app/pcap/pcap-filters/pcap-filters.component.spec.ts
 ---
@@ -179,19 +172,6 @@ describe('PcapFiltersComponent', () => {
 component.onSubmit();
   });
 
-  it('Port fields should be removed from request when set to empty', () => 
{
--- End diff --

Why was this test removed?


---

[jira] [Commented] (METRON-1724) Date/time validation missing in PCAP query

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590339#comment-16590339
 ] 

ASF GitHub Bot commented on METRON-1724:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/1172#discussion_r212341861
  
--- Diff: 
metron-interface/metron-alerts/src/app/pcap/pcap-filters/pcap-filters.component.spec.ts
 ---
@@ -179,19 +172,6 @@ describe('PcapFiltersComponent', () => {
 component.onSubmit();
   });
 
-  it('Port fields should be removed from request when set to empty', () => 
{
--- End diff --

Why was this test removed?


> Date/time validation missing in PCAP query
> --
>
> Key: METRON-1724
> URL: https://issues.apache.org/jira/browse/METRON-1724
> Project: Metron
>  Issue Type: Bug
>Reporter: Tibor Meller
>Priority: Major
>
> Validation formula should be the following: 
>  From < To < current date/time
>  
> Validation messages:
> Selected date range is invalid. The "To" date must be later than the "From" 
> date and the "To" date cannot be in the future.
> Source IP address format is invalid. Use valid v4IP format, for example, 
> [192.168.0.1|http://192.168.0.1/].
> Source port is invalid. Port number must be within the range of 0-65535.
> Destination IP address format is invalid. Use valid v4IP format, for example, 
> [192.168.0.1|http://192.168.0.1/].
> Destination port is invalid. Port number must be within the range of 0-65535.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1172: METRON-1724: Date/time validation missing in PCAP...

2018-08-23 Thread merrimanr

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/1172#discussion_r212342518
  
--- Diff: 
metron-interface/metron-alerts/src/app/pcap/pcap-packet-line/pcap-packet-line.component.spec.ts
 ---
@@ -143,38 +132,4 @@ describe('PcapPacketLineComponent', () => {
   "protos": null 
 });
   });
-
-  it('should extract UDP ipSrcPort fields', () => {
--- End diff --

Why were these tests removed?


---

[jira] [Commented] (METRON-1724) Date/time validation missing in PCAP query

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590341#comment-16590341
 ] 

ASF GitHub Bot commented on METRON-1724:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/1172#discussion_r212342518
  
--- Diff: 
metron-interface/metron-alerts/src/app/pcap/pcap-packet-line/pcap-packet-line.component.spec.ts
 ---
@@ -143,38 +132,4 @@ describe('PcapPacketLineComponent', () => {
   "protos": null 
 });
   });
-
-  it('should extract UDP ipSrcPort fields', () => {
--- End diff --

Why were these tests removed?


> Date/time validation missing in PCAP query
> --
>
> Key: METRON-1724
> URL: https://issues.apache.org/jira/browse/METRON-1724
> Project: Metron
>  Issue Type: Bug
>Reporter: Tibor Meller
>Priority: Major
>
> Validation formula should be the following: 
>  From < To < current date/time
>  
> Validation messages:
> Selected date range is invalid. The "To" date must be later than the "From" 
> date and the "To" date cannot be in the future.
> Source IP address format is invalid. Use valid v4IP format, for example, 
> [192.168.0.1|http://192.168.0.1/].
> Source port is invalid. Port number must be within the range of 0-65535.
> Destination IP address format is invalid. Use valid v4IP format, for example, 
> [192.168.0.1|http://192.168.0.1/].
> Destination port is invalid. Port number must be within the range of 0-65535.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1171: METRON-1740 make parser support CONFIG and SYSTEM log ty...

2018-08-23 Thread nickwallen

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1171
  
Thanks @liuy-tnz! 

Does anyone else have a panos firewall that they can test this against?


---

[jira] [Commented] (METRON-1740) Improve Palo Alto parser to handle CONFIG and SYSTEM syslog messages

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590393#comment-16590393
 ] 

ASF GitHub Bot commented on METRON-1740:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1171
  
Thanks @liuy-tnz! 

Does anyone else have a panos firewall that they can test this against?


> Improve Palo Alto parser to handle CONFIG and SYSTEM syslog messages
> 
>
> Key: METRON-1740
> URL: https://issues.apache.org/jira/browse/METRON-1740
> Project: Metron
>  Issue Type: Improvement
>Reporter: Yi Liu
>Priority: Major
>
> As a Metron's user (security analyst)
> I would like Metron's Palo Alto parser be able to parse CONFIG and SYSTEM 
> PanOS syslog messages
> so that I can know what, when how the system configuration has been changed 
> and how the system has been running. 
>  
> The current PaloAlto parser (BasicPaloAltoFirewallParser) only supports 
> THREAT and TRAFFIC log messages. The task is to extend it to support CONFIG 
> and SYSTEM log messages. The supported PanOS versions are 6.1, 7.0 and 8.0.
> The sample of CONFIG log (PanOS 7.0)
> {code:java}
> 1,2017/08/11 11:23:36,,CONFIG,0,0,2017/08/11 
> 11:23:36,192.168.14.162,,edit,admin,Web,Succeeded, vsys  vsys4 rule X 
> rules  dev-to-dev-ext-http-https,1336,0x0,0,0,0,0,,dev-something200-01
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1724) Date/time validation missing in PCAP query

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590434#comment-16590434
 ] 

ASF GitHub Bot commented on METRON-1724:


Github user tiborm commented on a diff in the pull request:

https://github.com/apache/metron/pull/1172#discussion_r212369764
  
--- Diff: 
metron-interface/metron-alerts/src/app/pcap/pcap-packet-line/pcap-packet-line.component.spec.ts
 ---
@@ -143,38 +132,4 @@ describe('PcapPacketLineComponent', () => {
   "protos": null 
 });
   });
-
-  it('should extract UDP ipSrcPort fields', () => {
--- End diff --

Merge issue. I checked these and the related code change too. Got no idea 
how could I missed these. Fixing.


> Date/time validation missing in PCAP query
> --
>
> Key: METRON-1724
> URL: https://issues.apache.org/jira/browse/METRON-1724
> Project: Metron
>  Issue Type: Bug
>Reporter: Tibor Meller
>Priority: Major
>
> Validation formula should be the following: 
>  From < To < current date/time
>  
> Validation messages:
> Selected date range is invalid. The "To" date must be later than the "From" 
> date and the "To" date cannot be in the future.
> Source IP address format is invalid. Use valid v4IP format, for example, 
> [192.168.0.1|http://192.168.0.1/].
> Source port is invalid. Port number must be within the range of 0-65535.
> Destination IP address format is invalid. Use valid v4IP format, for example, 
> [192.168.0.1|http://192.168.0.1/].
> Destination port is invalid. Port number must be within the range of 0-65535.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1172: METRON-1724: Date/time validation missing in PCAP...

2018-08-23 Thread tiborm

Github user tiborm commented on a diff in the pull request:

https://github.com/apache/metron/pull/1172#discussion_r212369764
  
--- Diff: 
metron-interface/metron-alerts/src/app/pcap/pcap-packet-line/pcap-packet-line.component.spec.ts
 ---
@@ -143,38 +132,4 @@ describe('PcapPacketLineComponent', () => {
   "protos": null 
 });
   });
-
-  it('should extract UDP ipSrcPort fields', () => {
--- End diff --

Merge issue. I checked these and the related code change too. Got no idea 
how could I missed these. Fixing.


---

[jira] [Commented] (METRON-1746) CEF lacks an ES template

2018-08-23 Thread Simon Elliston Ball (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590449#comment-16590449
 ] 

Simon Elliston Ball commented on METRON-1746:
-

One thing to think about here is that it really can't have a fixed template. In 
the CEF parser we unroll CEF's custom field label mechanism into field names, 
so we do not know the name of the all the fields the parser will emit without 
parsing the data. As such, you could generate a template, but not write and 
commit one, unless you just want to create a default one for the CEF core 
fields, and have anything else just be dynamic.

> CEF lacks an ES template
> 
>
> Key: METRON-1746
> URL: https://issues.apache.org/jira/browse/METRON-1746
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Jon Zeolla
>Priority: Blocker
>
> An ES template should exist 
> here
> If you only pass in CEF data, the alerts UI will not be able to display 
> anything, and ES will return a 500 saying "Fielddata is disabled on text 
> fields by default."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1746) CEF lacks an ES template

2018-08-23 Thread Otto Fowler (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590479#comment-16590479
 ] 

Otto Fowler commented on METRON-1746:
-

I think that is a good point [~simonellistonball], but would still be more 
valuable to the user than nothing.

> CEF lacks an ES template
> 
>
> Key: METRON-1746
> URL: https://issues.apache.org/jira/browse/METRON-1746
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Jon Zeolla
>Priority: Blocker
>
> An ES template should exist 
> here
> If you only pass in CEF data, the alerts UI will not be able to display 
> anything, and ES will return a 500 saying "Fielddata is disabled on text 
> fields by default."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1172: METRON-1724: Date/time validation missing in PCAP...

2018-08-23 Thread tiborm

Github user tiborm commented on a diff in the pull request:

https://github.com/apache/metron/pull/1172#discussion_r212383664
  
--- Diff: 
metron-interface/metron-alerts/src/app/pcap/pcap-filters/pcap-filters.component.spec.ts
 ---
@@ -179,19 +172,6 @@ describe('PcapFiltersComponent', () => {
 component.onSubmit();
   });
 
-  it('Port fields should be removed from request when set to empty', () => 
{
--- End diff --

It was intentional. We changed back the port type to string so this 
behavior not needed anymore.


---

[jira] [Commented] (METRON-1724) Date/time validation missing in PCAP query

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590529#comment-16590529
 ] 

ASF GitHub Bot commented on METRON-1724:


Github user tiborm commented on a diff in the pull request:

https://github.com/apache/metron/pull/1172#discussion_r212383664
  
--- Diff: 
metron-interface/metron-alerts/src/app/pcap/pcap-filters/pcap-filters.component.spec.ts
 ---
@@ -179,19 +172,6 @@ describe('PcapFiltersComponent', () => {
 component.onSubmit();
   });
 
-  it('Port fields should be removed from request when set to empty', () => 
{
--- End diff --

It was intentional. We changed back the port type to string so this 
behavior not needed anymore.


> Date/time validation missing in PCAP query
> --
>
> Key: METRON-1724
> URL: https://issues.apache.org/jira/browse/METRON-1724
> Project: Metron
>  Issue Type: Bug
>Reporter: Tibor Meller
>Priority: Major
>
> Validation formula should be the following: 
>  From < To < current date/time
>  
> Validation messages:
> Selected date range is invalid. The "To" date must be later than the "From" 
> date and the "To" date cannot be in the future.
> Source IP address format is invalid. Use valid v4IP format, for example, 
> [192.168.0.1|http://192.168.0.1/].
> Source port is invalid. Port number must be within the range of 0-65535.
> Destination IP address format is invalid. Use valid v4IP format, for example, 
> [192.168.0.1|http://192.168.0.1/].
> Destination port is invalid. Port number must be within the range of 0-65535.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1172: METRON-1724: Date/time validation missing in PCAP query

2018-08-23 Thread tiborm

Github user tiborm commented on the issue:

https://github.com/apache/metron/pull/1172
  
As a result of the followup discussion about the placing of the transform 
functions, I moved them back to the filter component.


---

[jira] [Commented] (METRON-1724) Date/time validation missing in PCAP query

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590532#comment-16590532
 ] 

ASF GitHub Bot commented on METRON-1724:


Github user tiborm commented on the issue:

https://github.com/apache/metron/pull/1172
  
As a result of the followup discussion about the placing of the transform 
functions, I moved them back to the filter component.


> Date/time validation missing in PCAP query
> --
>
> Key: METRON-1724
> URL: https://issues.apache.org/jira/browse/METRON-1724
> Project: Metron
>  Issue Type: Bug
>Reporter: Tibor Meller
>Priority: Major
>
> Validation formula should be the following: 
>  From < To < current date/time
>  
> Validation messages:
> Selected date range is invalid. The "To" date must be later than the "From" 
> date and the "To" date cannot be in the future.
> Source IP address format is invalid. Use valid v4IP format, for example, 
> [192.168.0.1|http://192.168.0.1/].
> Source port is invalid. Port number must be within the range of 0-65535.
> Destination IP address format is invalid. Use valid v4IP format, for example, 
> [192.168.0.1|http://192.168.0.1/].
> Destination port is invalid. Port number must be within the range of 0-65535.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (METRON-1743) CEF test confusing URL

2018-08-23 Thread Jon Zeolla (JIRA)



 [ 
https://issues.apache.org/jira/browse/METRON-1743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Zeolla reassigned METRON-1743:
--

Assignee: Jon Zeolla

> CEF test confusing URL
> --
>
> Key: METRON-1743
> URL: https://issues.apache.org/jira/browse/METRON-1743
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>Priority: Trivial
>
> A confusing test URL is used 
> here



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-1743) CEF test confusing URL

2018-08-23 Thread Jon Zeolla (JIRA)



 [ 
https://issues.apache.org/jira/browse/METRON-1743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Zeolla updated METRON-1743:
---
Description: 
https://github.com/apache/metron/blob/9fdccba371c1f1e0dcb79e00a7207a934b79b64c/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/cef/CEFParserTest.java#L225-L228
  (was: A confusing test URL is used 
here
)

> CEF test confusing URL
> --
>
> Key: METRON-1743
> URL: https://issues.apache.org/jira/browse/METRON-1743
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>Priority: Trivial
>
> https://github.com/apache/metron/blob/9fdccba371c1f1e0dcb79e00a7207a934b79b64c/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/cef/CEFParserTest.java#L225-L228



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-1743) CEF testPaloAltoCEF test using a confusing variable name

2018-08-23 Thread Jon Zeolla (JIRA)



 [ 
https://issues.apache.org/jira/browse/METRON-1743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Zeolla updated METRON-1743:
---
Summary: CEF testPaloAltoCEF test using a confusing variable name  (was: 
CEF test runTest doesn't consider targetJson)

> CEF testPaloAltoCEF test using a confusing variable name
> 
>
> Key: METRON-1743
> URL: https://issues.apache.org/jira/browse/METRON-1743
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>Priority: Trivial
>
> https://github.com/apache/metron/blob/9fdccba371c1f1e0dcb79e00a7207a934b79b64c/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/cef/CEFParserTest.java#L225-L228



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-1743) CEF test runTest doesn't consider targetJson

2018-08-23 Thread Jon Zeolla (JIRA)



 [ 
https://issues.apache.org/jira/browse/METRON-1743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Zeolla updated METRON-1743:
---
Summary: CEF test runTest doesn't consider targetJson  (was: CEF test 
confusing URL)

> CEF test runTest doesn't consider targetJson
> 
>
> Key: METRON-1743
> URL: https://issues.apache.org/jira/browse/METRON-1743
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>Priority: Trivial
>
> https://github.com/apache/metron/blob/9fdccba371c1f1e0dcb79e00a7207a934b79b64c/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/cef/CEFParserTest.java#L225-L228



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-1744) CEF test runTest doesn't consider targetJson

2018-08-23 Thread Jon Zeolla (JIRA)



 [ 
https://issues.apache.org/jira/browse/METRON-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Zeolla updated METRON-1744:
---
Description: 
https://github.com/apache/metron/blob/9fdccba371c1f1e0dcb79e00a7207a934b79b64c/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/cef/CEFParserTest.java#L225-L228
  (was: A confusing test URL 
here)

> CEF test runTest doesn't consider targetJson
> 
>
> Key: METRON-1744
> URL: https://issues.apache.org/jira/browse/METRON-1744
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Jon Zeolla
>Priority: Major
>
> https://github.com/apache/metron/blob/9fdccba371c1f1e0dcb79e00a7207a934b79b64c/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/cef/CEFParserTest.java#L225-L228



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-1744) CEF test runTest doesn't consider targetJson

2018-08-23 Thread Jon Zeolla (JIRA)



 [ 
https://issues.apache.org/jira/browse/METRON-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Zeolla updated METRON-1744:
---
Summary: CEF test runTest doesn't consider targetJson  (was: CEF 
testPaloAltoCEF test using a confusing variable name)

> CEF test runTest doesn't consider targetJson
> 
>
> Key: METRON-1744
> URL: https://issues.apache.org/jira/browse/METRON-1744
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Jon Zeolla
>Priority: Major
>
> A confusing test URL 
> here



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-1743) CEF testPaloAltoCEF test using a confusing variable name

2018-08-23 Thread Jon Zeolla (JIRA)



 [ 
https://issues.apache.org/jira/browse/METRON-1743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Zeolla updated METRON-1743:
---
Description: A confusing test URL 
here
  (was: 
https://github.com/apache/metron/blob/9fdccba371c1f1e0dcb79e00a7207a934b79b64c/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/cef/CEFParserTest.java#L225-L228)

> CEF testPaloAltoCEF test using a confusing variable name
> 
>
> Key: METRON-1743
> URL: https://issues.apache.org/jira/browse/METRON-1743
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>Priority: Trivial
>
> A confusing test URL 
> here



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1743) CEF testPaloAltoCEF test using a confusing variable name

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590552#comment-16590552
 ] 

ASF GitHub Bot commented on METRON-1743:


GitHub user JonZeolla opened a pull request:

https://github.com/apache/metron/pull/1173

METRON-1743: CEF test confusing URL

## Contributor Comments
Trivial variable rename.


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [X] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JonZeolla/metron METRON-1743

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1173.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1173


commit 165617be95101fa536facbdfeb73f7c1b6abc3ca
Author: Jon Zeolla 
Date:   2018-08-23T17:09:14Z

METRON-1743: CEF test confusing URL




> CEF testPaloAltoCEF test using a confusing variable name
> 
>
> Key: METRON-1743
> URL: https://issues.apache.org/jira/browse/METRON-1743
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>Priority: Trivial
>
> A confusing test URL 
> here



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1173: METRON-1743: CEF test confusing URL

2018-08-23 Thread JonZeolla

GitHub user JonZeolla opened a pull request:

https://github.com/apache/metron/pull/1173

METRON-1743: CEF test confusing URL

## Contributor Comments
Trivial variable rename.


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [X] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JonZeolla/metron METRON-1743

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1173.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1173


commit 165617be95101fa536facbdfeb73f7c1b6abc3ca
Author: Jon Zeolla 
Date:   2018-08-23T17:09:14Z

METRON-1743: CEF test confusing URL




---

[GitHub] metron issue #1172: METRON-1724: Date/time validation missing in PCAP query

2018-08-23 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/1172
  
I found one small bug.  Here's how to reproduce:

1. Set the FROM date to be greater than the TO date.  A validation message 
should appear as expected.
2. Set the TO date to be in the future.  Validation message still displays 
as expected and both inputs are marked with red.
3. Change the FROM date to be less than the TO date and not in the future.  
The FROM date is still marked in red when it should not be.
4. Change the TO date to not be in the future but greater than the FROM 
date.  The TO data is no longer marked in red but the validation message is 
still displayed.  At this point there should be no validation messages.


---

[jira] [Commented] (METRON-1724) Date/time validation missing in PCAP query

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590564#comment-16590564
 ] 

ASF GitHub Bot commented on METRON-1724:


Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/1172
  
I found one small bug.  Here's how to reproduce:

1. Set the FROM date to be greater than the TO date.  A validation message 
should appear as expected.
2. Set the TO date to be in the future.  Validation message still displays 
as expected and both inputs are marked with red.
3. Change the FROM date to be less than the TO date and not in the future.  
The FROM date is still marked in red when it should not be.
4. Change the TO date to not be in the future but greater than the FROM 
date.  The TO data is no longer marked in red but the validation message is 
still displayed.  At this point there should be no validation messages.


> Date/time validation missing in PCAP query
> --
>
> Key: METRON-1724
> URL: https://issues.apache.org/jira/browse/METRON-1724
> Project: Metron
>  Issue Type: Bug
>Reporter: Tibor Meller
>Priority: Major
>
> Validation formula should be the following: 
>  From < To < current date/time
>  
> Validation messages:
> Selected date range is invalid. The "To" date must be later than the "From" 
> date and the "To" date cannot be in the future.
> Source IP address format is invalid. Use valid v4IP format, for example, 
> [192.168.0.1|http://192.168.0.1/].
> Source port is invalid. Port number must be within the range of 0-65535.
> Destination IP address format is invalid. Use valid v4IP format, for example, 
> [192.168.0.1|http://192.168.0.1/].
> Destination port is invalid. Port number must be within the range of 0-65535.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1748) Improve Storm Profiler Integration Test

2018-08-23 Thread Nick Allen (JIRA)

Nick Allen created METRON-1748:
--

 Summary: Improve Storm Profiler Integration Test
 Key: METRON-1748
 URL: https://issues.apache.org/jira/browse/METRON-1748
 Project: Metron
  Issue Type: Bug
Reporter: Nick Allen
Assignee: Nick Allen






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-1748) Improve Storm Profiler Integration Test

2018-08-23 Thread Nick Allen (JIRA)



 [ 
https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Allen updated METRON-1748:
---
Description: We should use the Profiler Client, like PROFILE_GET, to 
validate the output of the Storm Profiler Integration Test.  This is better 
validation that things are working end-to-end.

> Improve Storm Profiler Integration Test
> ---
>
> Key: METRON-1748
> URL: https://issues.apache.org/jira/browse/METRON-1748
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> We should use the Profiler Client, like PROFILE_GET, to validate the output 
> of the Storm Profiler Integration Test.  This is better validation that 
> things are working end-to-end.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1174: METRON-1748 Improve Storm Profiler Integration Te...

2018-08-23 Thread nickwallen

GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/1174

METRON-1748 Improve Storm Profiler Integration Test

Improved the Storm Profiler integration tests based on improvements I made 
for the Spark Profiler feature branch.

* Validate the output of the Profiler using the client library; 
`PROFILE_GET`.  This is better validation that things are working end-to-end.

* Using more telemetry messages to validate event time processing.

* Added logging around cache maintenence.

* Added logging of the tuple windows that are passed from Storm's 
WindowManager.  This can help debug issues where the time lag and window size 
are incorrectly set.

## Pull Request Checklist

- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron METRON-1748

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1174.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1174


commit 56b70f666dd75a2861d17a23b7ed307124ba1f74
Author: Nick Allen 
Date:   2018-08-17T17:13:10Z

METRON-1748 Improve Storm Profiler Integration Test




---

[jira] [Commented] (METRON-1748) Improve Storm Profiler Integration Test

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590666#comment-16590666
 ] 

ASF GitHub Bot commented on METRON-1748:


GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/1174

METRON-1748 Improve Storm Profiler Integration Test

Improved the Storm Profiler integration tests based on improvements I made 
for the Spark Profiler feature branch.

* Validate the output of the Profiler using the client library; 
`PROFILE_GET`.  This is better validation that things are working end-to-end.

* Using more telemetry messages to validate event time processing.

* Added logging around cache maintenence.

* Added logging of the tuple windows that are passed from Storm's 
WindowManager.  This can help debug issues where the time lag and window size 
are incorrectly set.

## Pull Request Checklist

- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron METRON-1748

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1174.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1174


commit 56b70f666dd75a2861d17a23b7ed307124ba1f74
Author: Nick Allen 
Date:   2018-08-17T17:13:10Z

METRON-1748 Improve Storm Profiler Integration Test




> Improve Storm Profiler Integration Test
> ---
>
> Key: METRON-1748
> URL: https://issues.apache.org/jira/browse/METRON-1748
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> We should use the Profiler Client, like PROFILE_GET, to validate the output 
> of the Storm Profiler Integration Test.  This is better validation that 
> things are working end-to-end.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1707) Port Profiler to Spark

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590835#comment-16590835
 ] 

ASF GitHub Bot commented on METRON-1707:


Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/1150
  
I was able to understand everything by following 
BatchProfilerIntegrationTest.  Looks like a great start to me. +1


> Port Profiler to Spark
> --
>
> Key: METRON-1707
> URL: https://issues.apache.org/jira/browse/METRON-1707
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Create a port of the Profiler that runs in Spark.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1150: METRON-1707 Port Profiler to Spark [Feature Branch]

2018-08-23 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/1150
  
I was able to understand everything by following 
BatchProfilerIntegrationTest.  Looks like a great start to me. +1


---

[jira] [Commented] (METRON-1016) METRON Demo System as separate install option

2018-08-23 Thread Simon Elliston Ball (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590840#comment-16590840
 ] 

Simon Elliston Ball commented on METRON-1016:
-

I agree that this would be a great step. Perhaps the demos belong in a more 
structured version on the use-cases folder, or in contrib. I have a repo I use 
of configs and scripts that have some basic demos in a standard-ish form which 
was meant to fill this gap 
[https://github.com/simonellistonball/metron-field-demos] 

I'm not sure it would benefit from an extra Mpack or Ambari service, since 
concepts like services in Ambari seem heavy for a Metron demo, but a set of 
scripts to install makes sense. We could probably also do with stripping out 
some of the default parsers into a separate section along with this, keeping 
the core Metron lightweight with the generics (Grok, Java, CSV, JSON) and 
pulling the device ones out into a separate location. That may go beyond the 
scope of this exercise but does feel related.

For the full dev load, I would say that an ansible role running post blueprint 
install would be a good full dev solution. 

> METRON Demo System as separate install option
> -
>
> Key: METRON-1016
> URL: https://issues.apache.org/jira/browse/METRON-1016
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Priority: Major
>
> Right now, the demo system ( bro, snort, yaf ) are part of the default 
> installation, and the default configurations required for those systems ( ES, 
> Enrichment ) are in the main code base.  For example bro's default enrichment 
> is the enrichment that assumes the demo setup.
> This is not really correct.  The demo system should be a separately 
> installable feature of metron, and it's configurations should be managed 
> outside the defaults for the parsers etc.
> This would allow separate development of the demo without affecting others.
> Either checkbox or a separate ambari service?
> What should the default enrichment be?
> How to have vagrant install the service



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1016) METRON Demo System as separate install option

2018-08-23 Thread Otto Fowler (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590850#comment-16590850
 ] 

Otto Fowler commented on METRON-1016:
-

Well, this would govern the install of snort, bro and yaf data generation, the 
dashboard for s b y, and the sensors.  So, ambari++

> METRON Demo System as separate install option
> -
>
> Key: METRON-1016
> URL: https://issues.apache.org/jira/browse/METRON-1016
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Priority: Major
>
> Right now, the demo system ( bro, snort, yaf ) are part of the default 
> installation, and the default configurations required for those systems ( ES, 
> Enrichment ) are in the main code base.  For example bro's default enrichment 
> is the enrichment that assumes the demo setup.
> This is not really correct.  The demo system should be a separately 
> installable feature of metron, and it's configurations should be managed 
> outside the defaults for the parsers etc.
> This would allow separate development of the demo without affecting others.
> Either checkbox or a separate ambari service?
> What should the default enrichment be?
> How to have vagrant install the service



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1174: METRON-1748 Improve Storm Profiler Integration Test

2018-08-23 Thread nickwallen

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1174
  
I am seeing sporadic integration test failures from 
`ProfilerIntegrationTest.testProcessingTimeWithTimeToLiveFlush` with this 
change.  I need to track that down before this gets merged.


---

[jira] [Commented] (METRON-1748) Improve Storm Profiler Integration Test

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590854#comment-16590854
 ] 

ASF GitHub Bot commented on METRON-1748:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1174
  
I am seeing sporadic integration test failures from 
`ProfilerIntegrationTest.testProcessingTimeWithTimeToLiveFlush` with this 
change.  I need to track that down before this gets merged.


> Improve Storm Profiler Integration Test
> ---
>
> Key: METRON-1748
> URL: https://issues.apache.org/jira/browse/METRON-1748
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> We should use the Profiler Client, like PROFILE_GET, to validate the output 
> of the Storm Profiler Integration Test.  This is better validation that 
> things are working end-to-end.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1150: METRON-1707 Port Profiler to Spark [Feature Branc...

2018-08-23 Thread nickwallen

Github user nickwallen closed the pull request at:

https://github.com/apache/metron/pull/1150


---

[GitHub] metron issue #1150: METRON-1707 Port Profiler to Spark [Feature Branch]

2018-08-23 Thread nickwallen

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1150
  
Thanks for the review guys!  I merged this one into the feature branch.  

For those reviewers wanting further punishment, #1161 is the next PR to 
review for this feature branch.


---

[jira] [Commented] (METRON-1707) Port Profiler to Spark

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590862#comment-16590862
 ] 

ASF GitHub Bot commented on METRON-1707:


Github user nickwallen closed the pull request at:

https://github.com/apache/metron/pull/1150


> Port Profiler to Spark
> --
>
> Key: METRON-1707
> URL: https://issues.apache.org/jira/browse/METRON-1707
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Create a port of the Profiler that runs in Spark.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1707) Port Profiler to Spark

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590861#comment-16590861
 ] 

ASF GitHub Bot commented on METRON-1707:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1150
  
Thanks for the review guys!  I merged this one into the feature branch.  

For those reviewers wanting further punishment, #1161 is the next PR to 
review for this feature branch.


> Port Profiler to Spark
> --
>
> Key: METRON-1707
> URL: https://issues.apache.org/jira/browse/METRON-1707
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Create a port of the Profiler that runs in Spark.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1171: METRON-1740 make parser support CONFIG and SYSTEM log ty...

2018-08-23 Thread JonZeolla

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/1171
  
I can.  Testing steps would help me get this done tomorrow otherwise I'm 
not sure when I'll get a chance to.


---

[jira] [Commented] (METRON-1740) Improve Palo Alto parser to handle CONFIG and SYSTEM syslog messages

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590960#comment-16590960
 ] 

ASF GitHub Bot commented on METRON-1740:


Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/1171
  
I can.  Testing steps would help me get this done tomorrow otherwise I'm 
not sure when I'll get a chance to.


> Improve Palo Alto parser to handle CONFIG and SYSTEM syslog messages
> 
>
> Key: METRON-1740
> URL: https://issues.apache.org/jira/browse/METRON-1740
> Project: Metron
>  Issue Type: Improvement
>Reporter: Yi Liu
>Priority: Major
>
> As a Metron's user (security analyst)
> I would like Metron's Palo Alto parser be able to parse CONFIG and SYSTEM 
> PanOS syslog messages
> so that I can know what, when how the system configuration has been changed 
> and how the system has been running. 
>  
> The current PaloAlto parser (BasicPaloAltoFirewallParser) only supports 
> THREAT and TRAFFIC log messages. The task is to extend it to support CONFIG 
> and SYSTEM log messages. The supported PanOS versions are 6.1, 7.0 and 8.0.
> The sample of CONFIG log (PanOS 7.0)
> {code:java}
> 1,2017/08/11 11:23:36,,CONFIG,0,0,2017/08/11 
> 11:23:36,192.168.14.162,,edit,admin,Web,Succeeded, vsys  vsys4 rule X 
> rules  dev-to-dev-ext-http-https,1336,0x0,0,0,0,0,,dev-something200-01
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1171: METRON-1740 make parser support CONFIG and SYSTEM log ty...

2018-08-23 Thread liuy-tnz

Github user liuy-tnz commented on the issue:

https://github.com/apache/metron/pull/1171
  
@JonZeolla Thank you first. Let me try to list the testing steps as I 
haven't done it before. 

Reproduce steps
1. setup the panos firewall (v6.1, v7.0 or v8.0)
2. setup Metron with NO my changes
3. Copy log messages generated by the firewall to the landing Kafka topic 
producer
4. Verify the outcome JSON string in the parser Kafka topic consumer. 

Expected result: all type messages (CONFIG, SYSTEM, THREAT and TRAFFIC) 
shall be parsed successfully
Actual result: An exception generated when parsing CONFIG and SYSTEM 
messages. 
Reason: not support CONFIG and SYSTEM message

Please contact me if you need any help. 
Thanks






---

[jira] [Commented] (METRON-1740) Improve Palo Alto parser to handle CONFIG and SYSTEM syslog messages

2018-08-23 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591019#comment-16591019
 ] 

ASF GitHub Bot commented on METRON-1740:


Github user liuy-tnz commented on the issue:

https://github.com/apache/metron/pull/1171
  
@JonZeolla Thank you first. Let me try to list the testing steps as I 
haven't done it before. 

Reproduce steps
1. setup the panos firewall (v6.1, v7.0 or v8.0)
2. setup Metron with NO my changes
3. Copy log messages generated by the firewall to the landing Kafka topic 
producer
4. Verify the outcome JSON string in the parser Kafka topic consumer. 

Expected result: all type messages (CONFIG, SYSTEM, THREAT and TRAFFIC) 
shall be parsed successfully
Actual result: An exception generated when parsing CONFIG and SYSTEM 
messages. 
Reason: not support CONFIG and SYSTEM message

Please contact me if you need any help. 
Thanks






> Improve Palo Alto parser to handle CONFIG and SYSTEM syslog messages
> 
>
> Key: METRON-1740
> URL: https://issues.apache.org/jira/browse/METRON-1740
> Project: Metron
>  Issue Type: Improvement
>Reporter: Yi Liu
>Priority: Major
>
> As a Metron's user (security analyst)
> I would like Metron's Palo Alto parser be able to parse CONFIG and SYSTEM 
> PanOS syslog messages
> so that I can know what, when how the system configuration has been changed 
> and how the system has been running. 
>  
> The current PaloAlto parser (BasicPaloAltoFirewallParser) only supports 
> THREAT and TRAFFIC log messages. The task is to extend it to support CONFIG 
> and SYSTEM log messages. The supported PanOS versions are 6.1, 7.0 and 8.0.
> The sample of CONFIG log (PanOS 7.0)
> {code:java}
> 1,2017/08/11 11:23:36,,CONFIG,0,0,2017/08/11 
> 11:23:36,192.168.14.162,,edit,admin,Web,Succeeded, vsys  vsys4 rule X 
> rules  dev-to-dev-ext-http-https,1336,0x0,0,0,0,0,,dev-something200-01
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

69 matches

Mail list logo