[jira] [Commented] (METRON-1563) Initial Feature Branch Work
[ https://issues.apache.org/jira/browse/METRON-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680781#comment-16680781 ] ASF GitHub Bot commented on METRON-1563: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1014 This is in, not sure why it isn't closed > Initial Feature Branch Work > --- > > Key: METRON-1563 > URL: https://issues.apache.org/jira/browse/METRON-1563 > Project: Metron > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1740) Improve Palo Alto parser to handle CONFIG and SYSTEM syslog messages
[
https://issues.apache.org/jira/browse/METRON-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680786#comment-16680786
]
ASF GitHub Bot commented on METRON-1740:
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1171
@nickwallen can we get a +1 on this? @JonZeolla you are a +1 but not
explicitly?
> Improve Palo Alto parser to handle CONFIG and SYSTEM syslog messages
>
>
> Key: METRON-1740
> URL: https://issues.apache.org/jira/browse/METRON-1740
> Project: Metron
> Issue Type: Improvement
>Reporter: Yi Liu
>Priority: Major
>
> As a Metron's user (security analyst)
> I would like Metron's Palo Alto parser be able to parse CONFIG and SYSTEM
> PanOS syslog messages
> so that I can know what, when how the system configuration has been changed
> and how the system has been running.
>
> The current PaloAlto parser (BasicPaloAltoFirewallParser) only supports
> THREAT and TRAFFIC log messages. The task is to extend it to support CONFIG
> and SYSTEM log messages. The supported PanOS versions are 6.1, 7.0 and 8.0.
> The sample of CONFIG log (PanOS 7.0)
> {code:java}
> 1,2017/08/11 11:23:36,,CONFIG,0,0,2017/08/11
> 11:23:36,192.168.14.162,,edit,admin,Web,Succeeded, vsys vsys4 rule X
> rules dev-to-dev-ext-http-https,1336,0x0,0,0,0,0,,dev-something200-01
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1171: METRON-1740 make parser support CONFIG and SYSTEM log ty...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1171 @nickwallen can we get a +1 on this? @JonZeolla you are a +1 but not explicitly? ---
[jira] [Commented] (METRON-1847) Create reusable script with functions from prepare-commit
[ https://issues.apache.org/jira/browse/METRON-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680784#comment-16680784 ] ASF GitHub Bot commented on METRON-1847: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1248 > Create reusable script with functions from prepare-commit > - > > Key: METRON-1847 > URL: https://issues.apache.org/jira/browse/METRON-1847 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > this will allow code reuse if and when we create more scripts -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1248: METRON-1847 break out functions from prepare-comm...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1248 ---
[jira] [Commented] (METRON-1563) Initial Feature Branch Work
[ https://issues.apache.org/jira/browse/METRON-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680782#comment-16680782 ] ASF GitHub Bot commented on METRON-1563: Github user ottobackwards closed the pull request at: https://github.com/apache/metron/pull/1014 > Initial Feature Branch Work > --- > > Key: METRON-1563 > URL: https://issues.apache.org/jira/browse/METRON-1563 > Project: Metron > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1014: METRON-1563 : Base Stellar assign for feature bra...
Github user ottobackwards closed the pull request at: https://github.com/apache/metron/pull/1014 ---
[GitHub] metron issue #1014: METRON-1563 : Base Stellar assign for feature branch
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1014 This is in, not sure why it isn't closed ---
[jira] [Commented] (METRON-1834) Migrate Elasticsearch from TransportClient to new Java REST API
[
https://issues.apache.org/jira/browse/METRON-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680537#comment-16680537
]
ASF GitHub Bot commented on METRON-1834:
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/1242
> The last remaining bit before I merge this will be sharing some results
around the performance/regression testing performed by @anandsubbu.
I completed a set of performance comparison tests between the REST and
Transport clients. Here's the update.
### Note on the Testing Done
* The performance regression comparison tests are run on a multi-node
VM-based environment by:
a) Running a set of tests using the ES REST Client
b) Swap out the REST client with the Transport client, and re-run the same
set of tests.
### Observations
1) The indexing rate numbers with the REST client is similar to that of the
Transport client.
2) I had noticed storm rebalance error messages for the kafkaSpout for both
REST and Transport client runs, which required to adjust the values of
`request.timeout.ms` and `session.timeout.ms` in the ES flux file under
'kafkaProps'. The errors were resolved after making these changes.
3) For the REST client, I observed listener timeout exceptions with the
default settings. I had to increase max retry and socket timeout values in
global config as follows, in order to resolve the exceptions.
```
"es.client.settings" : {
"max.retry.timeout.millis" : "6",
"socket.timeout.millis" : "6"
}
```
4) For both cases, the kafkaSpout complete latency was seen to be on the
higher side (~30 secs)
5) Lastly, I noticed that there were a number of failed tuples seen in the
kafkaSpout of the RA Indexing topology. I am continuing to dig more into this,
but since this was seen for both the REST and Transport client tests, I am
suspecting it could be an anomaly with the test environment, or a different
issue common to both REST/Transport clients.
Based on the above, it is seen that there is no immediate regression in
performance b/w the REST and Transport client.
+1, great work @mmiklavc
> Migrate Elasticsearch from TransportClient to new Java REST API
> ---
>
> Key: METRON-1834
> URL: https://issues.apache.org/jira/browse/METRON-1834
> Project: Metron
> Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1242: METRON-1834: Migrate Elasticsearch from TransportClient ...
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/1242
> The last remaining bit before I merge this will be sharing some results
around the performance/regression testing performed by @anandsubbu.
I completed a set of performance comparison tests between the REST and
Transport clients. Here's the update.
### Note on the Testing Done
* The performance regression comparison tests are run on a multi-node
VM-based environment by:
a) Running a set of tests using the ES REST Client
b) Swap out the REST client with the Transport client, and re-run the same
set of tests.
### Observations
1) The indexing rate numbers with the REST client is similar to that of the
Transport client.
2) I had noticed storm rebalance error messages for the kafkaSpout for both
REST and Transport client runs, which required to adjust the values of
`request.timeout.ms` and `session.timeout.ms` in the ES flux file under
'kafkaProps'. The errors were resolved after making these changes.
3) For the REST client, I observed listener timeout exceptions with the
default settings. I had to increase max retry and socket timeout values in
global config as follows, in order to resolve the exceptions.
```
"es.client.settings" : {
"max.retry.timeout.millis" : "6",
"socket.timeout.millis" : "6"
}
```
4) For both cases, the kafkaSpout complete latency was seen to be on the
higher side (~30 secs)
5) Lastly, I noticed that there were a number of failed tuples seen in the
kafkaSpout of the RA Indexing topology. I am continuing to dig more into this,
but since this was seen for both the REST and Transport client tests, I am
suspecting it could be an anomaly with the test environment, or a different
issue common to both REST/Transport clients.
Based on the above, it is seen that there is no immediate regression in
performance b/w the REST and Transport client.
+1, great work @mmiklavc
---
[jira] [Commented] (METRON-1850) Stellar REST function
[ https://issues.apache.org/jira/browse/METRON-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680529#comment-16680529 ] ASF GitHub Bot commented on METRON-1850: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1250 > Stellar REST function > - > > Key: METRON-1850 > URL: https://issues.apache.org/jira/browse/METRON-1850 > Project: Metron > Issue Type: New Feature >Reporter: Ryan Merriman >Priority: Major > > It would be useful to be able to enrich messages with Stellar using 3rd party > (or internal) REST services. At a minimum this function would: > * Stellar function available to GET from an HTTP API > * Optional parameters for basic auth (user/password) which generate correct > Authorization header > * Function returns null value for errors, connection failures etc and logs > error > * Function must provide and use pooled connection objects at the process > level > * Function must send Accept: application/json header > * A global setting must be available to set a proxy for all API calls, and > if present the proxy must be used. > * Proxy authentication must also be supported using basic auth. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1250: METRON-1850: Stellar REST function
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1250 ---
[jira] [Commented] (METRON-1861) REST fails to start when LDAP enabled and 'Active Spring profiles' config is empty
[
https://issues.apache.org/jira/browse/METRON-1861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679809#comment-16679809
]
ASF GitHub Bot commented on METRON-1861:
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/1256
> REST fails to start when LDAP enabled and 'Active Spring profiles' config is
> empty
> --
>
> Key: METRON-1861
> URL: https://issues.apache.org/jira/browse/METRON-1861
> Project: Metron
> Issue Type: Bug
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>Priority: Major
>
> When 'LDAP enabled' (in Configs -> Metron -> Security tab) is set to 'On' and
> the 'Active Spring profiles' (in Configs -> Metron -> REST tab) is blank,
> REST service fails to start with the following error:
>
> {code:java}
> 18/11/08 08:10:02 ERROR boot.SpringApplication: Application run failed
> java.lang.IllegalArgumentException: Invalid profile []: must contain text
> at
> org.springframework.core.env.AbstractEnvironment.validateProfile(AbstractEnvironment.java:365)
> at
> org.springframework.core.env.AbstractEnvironment.setActiveProfiles(AbstractEnvironment.java:256)
> at
> org.springframework.core.env.AbstractEnvironment.doGetActiveProfiles(AbstractEnvironment.java:242)
> at
> org.springframework.core.env.AbstractEnvironment.getActiveProfiles(AbstractEnvironment.java:226)
> at
> org.springframework.boot.SpringApplication.configureProfiles(SpringApplication.java:522)
> at
> org.springframework.boot.SpringApplication.configureEnvironment(SpringApplication.java:479)
> at
> org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:357)
> at org.springframework.boot.SpringApplication.run(SpringApplication.java:317)
> at
> org.springframework.boot.SpringApplication.run(SpringApplication.java:1255)
> at
> org.springframework.boot.SpringApplication.run(SpringApplication.java:1243)
> at
> org.apache.metron.rest.MetronRestApplication.main(MetronRestApplication.java:36){code}
>
> This is because the 'metron_spring_profiles_active' is set to the value
> ',ldap' which is not valid.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1304) Allow metron-bro-plugin-kafka to include or exclude logs
[ https://issues.apache.org/jira/browse/METRON-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679813#comment-16679813 ] ASF GitHub Bot commented on METRON-1304: Github user nickwallen commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/2 @JonZeolla Can you add the latest testing instructions in the description? This seems to have evolved over time and am not sure what is needed at this point. > Allow metron-bro-plugin-kafka to include or exclude logs > > > Key: METRON-1304 > URL: https://issues.apache.org/jira/browse/METRON-1304 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Major > > Right now, you must specify which logs you want to send to kafka via > metron-bro-plugin-kafka. This would allow the additional feature of > excluding certain logs, and sending everything else. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1304) Allow metron-bro-plugin-kafka to include or exclude logs
[ https://issues.apache.org/jira/browse/METRON-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679967#comment-16679967 ] ASF GitHub Bot commented on METRON-1304: Github user JonZeolla commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/2 https://github.com/apache/metron-bro-plugin-kafka/pull/2#issuecomment-429502420 is still accurate for testing. I'll edit the initial comment to add it. > Allow metron-bro-plugin-kafka to include or exclude logs > > > Key: METRON-1304 > URL: https://issues.apache.org/jira/browse/METRON-1304 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Major > > Right now, you must specify which logs you want to send to kafka via > metron-bro-plugin-kafka. This would allow the additional feature of > excluding certain logs, and sending everything else. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1861) REST fails to start when LDAP enabled and 'Active Spring profiles' config is empty
[
https://issues.apache.org/jira/browse/METRON-1861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anand Subramanian updated METRON-1861:
--
Description:
When 'LDAP enabled' (in Configs -> Metron -> Security tab) is set to 'On' and
the 'Active Spring profiles' (in Configs -> Metron -> REST tab) is blank, REST
service fails to start with the following :
{code:java}
18/11/08 08:10:02 boot.SpringApplication: Application run
java.lang.IllegalArgumentException: Invalid profile []: must contain text
at
org.springframework.core.env.AbstractEnvironment.validateProfile(AbstractEnvironment.java:365)
at
org.springframework.core.env.AbstractEnvironment.setActiveProfiles(AbstractEnvironment.java:256)
at
org.springframework.core.env.AbstractEnvironment.doGetActiveProfiles(AbstractEnvironment.java:242)
at
org.springframework.core.env.AbstractEnvironment.getActiveProfiles(AbstractEnvironment.java:226)
at
org.springframework.boot.SpringApplication.configureProfiles(SpringApplication.java:522)
at
org.springframework.boot.SpringApplication.configureEnvironment(SpringApplication.java:479)
at
org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:357)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:317)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1255)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1243)
at
org.apache.metron.rest.MetronRestApplication.main(MetronRestApplication.java:36){code}
This is because the 'metron_spring_profiles_active' is set to the value ',ldap'
which is not valid.
was:
When 'LDAP enabled' (in Configs -> Metron -> Security tab) is set to 'On' and
the 'Active Spring profiles' (in Configs -> Metron -> REST tab) is blank, REST
service fails to start with the following error:
{code:java}
18/11/08 08:10:02 ERROR boot.SpringApplication: Application run failed
java.lang.IllegalArgumentException: Invalid profile []: must contain text
at
org.springframework.core.env.AbstractEnvironment.validateProfile(AbstractEnvironment.java:365)
at
org.springframework.core.env.AbstractEnvironment.setActiveProfiles(AbstractEnvironment.java:256)
at
org.springframework.core.env.AbstractEnvironment.doGetActiveProfiles(AbstractEnvironment.java:242)
at
org.springframework.core.env.AbstractEnvironment.getActiveProfiles(AbstractEnvironment.java:226)
at
org.springframework.boot.SpringApplication.configureProfiles(SpringApplication.java:522)
at
org.springframework.boot.SpringApplication.configureEnvironment(SpringApplication.java:479)
at
org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:357)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:317)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1255)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1243)
at
org.apache.metron.rest.MetronRestApplication.main(MetronRestApplication.java:36){code}
This is because the 'metron_spring_profiles_active' is set to the value ',ldap'
which is not valid.
> REST fails to start when LDAP enabled and 'Active Spring profiles' config is
> empty
> --
>
> Key: METRON-1861
> URL: https://issues.apache.org/jira/browse/METRON-1861
> Project: Metron
> Issue Type: Bug
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>Priority: Major
>
> When 'LDAP enabled' (in Configs -> Metron -> Security tab) is set to 'On' and
> the 'Active Spring profiles' (in Configs -> Metron -> REST tab) is blank,
> REST service fails to start with the following :
>
> {code:java}
> 18/11/08 08:10:02 boot.SpringApplication: Application run
> java.lang.IllegalArgumentException: Invalid profile []: must contain text
> at
> org.springframework.core.env.AbstractEnvironment.validateProfile(AbstractEnvironment.java:365)
> at
> org.springframework.core.env.AbstractEnvironment.setActiveProfiles(AbstractEnvironment.java:256)
> at
> org.springframework.core.env.AbstractEnvironment.doGetActiveProfiles(AbstractEnvironment.java:242)
> at
> org.springframework.core.env.AbstractEnvironment.getActiveProfiles(AbstractEnvironment.java:226)
> at
> org.springframework.boot.SpringApplication.configureProfiles(SpringApplication.java:522)
> at
> org.springframework.boot.SpringApplication.configureEnvironment(SpringApplication.java:479)
> at
> org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:357)
> at org.springframework.boot.SpringApplication.run(SpringApplication.java:317)
> at
> org.springframework.boot.SpringApplication.run(SpringApplication.java:1255)
> at
>
[jira] [Commented] (METRON-1304) Allow metron-bro-plugin-kafka to include or exclude logs
[ https://issues.apache.org/jira/browse/METRON-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679836#comment-16679836 ] ASF GitHub Bot commented on METRON-1304: Github user nickwallen commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/2 @JonZeolla I just want to confirm, all of this is related only to include/exclude functionality? There seems to be more here, but maybe I am misunderstanding on first glance. > Allow metron-bro-plugin-kafka to include or exclude logs > > > Key: METRON-1304 > URL: https://issues.apache.org/jira/browse/METRON-1304 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Major > > Right now, you must specify which logs you want to send to kafka via > metron-bro-plugin-kafka. This would allow the additional feature of > excluding certain logs, and sending everything else. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1850) Stellar REST function
[ https://issues.apache.org/jira/browse/METRON-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680453#comment-16680453 ] ASF GitHub Bot commented on METRON-1850: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1250 +1, thanks! > Stellar REST function > - > > Key: METRON-1850 > URL: https://issues.apache.org/jira/browse/METRON-1850 > Project: Metron > Issue Type: New Feature >Reporter: Ryan Merriman >Priority: Major > > It would be useful to be able to enrich messages with Stellar using 3rd party > (or internal) REST services. At a minimum this function would: > * Stellar function available to GET from an HTTP API > * Optional parameters for basic auth (user/password) which generate correct > Authorization header > * Function returns null value for errors, connection failures etc and logs > error > * Function must provide and use pooled connection objects at the process > level > * Function must send Accept: application/json header > * A global setting must be available to set a proxy for all API calls, and > if present the proxy must be used. > * Proxy authentication must also be supported using basic auth. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1250: METRON-1850: Stellar REST function
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1250 +1, thanks! ---
[jira] [Created] (METRON-1864) Stellar date format test fails after daylight saving
Otto Fowler created METRON-1864: --- Summary: Stellar date format test fails after daylight saving Key: METRON-1864 URL: https://issues.apache.org/jira/browse/METRON-1864 Project: Metron Issue Type: Bug Reporter: Otto Fowler Assignee: Otto Fowler Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.62 sec - in org.apache.metron.stellar.dsl.functions.FunctionalFunctionsTest Results : Failed tests: DateFunctionsTest.testDateFormatDefault:239 null -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1834) Migrate Elasticsearch from TransportClient to new Java REST API
[ https://issues.apache.org/jira/browse/METRON-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679197#comment-16679197 ] ASF GitHub Bot commented on METRON-1834: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1242 Thanks @nickwallen. I pushed one more commit to make a note of this Jira in Upgrading.md. I verified it looks correct in the site-book and checked it off on the Jira tasks as well. I guess just a quick look (it's like 5 lines) when you get a moment would be great. The last remaining bit before I merge this will be sharing some results around the performance/regression testing performed by @anandsubbu. > Migrate Elasticsearch from TransportClient to new Java REST API > --- > > Key: METRON-1834 > URL: https://issues.apache.org/jira/browse/METRON-1834 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1862) Cant Load Metron Bro Plugin
Mustafa Akmal created METRON-1862: - Summary: Cant Load Metron Bro Plugin Key: METRON-1862 URL: https://issues.apache.org/jira/browse/METRON-1862 Project: Metron Issue Type: Bug Affects Versions: 0.4.1 Reporter: Mustafa Akmal I received the following error when running 'broctl deploy' : can't find packages/metron-bro-plugin-kafka/Apache/Kafka This happens when I load the metron bro plugin by making changes to 'local.bro' by inserting the following line "[@load|https://github.com/load] packages/metron-bro-plugin-kafka/Apache/Kafka" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1858) BasicFireEyeParser check style cleanup and optimization
[ https://issues.apache.org/jira/browse/METRON-1858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680315#comment-16680315 ] ASF GitHub Bot commented on METRON-1858: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1255 > BasicFireEyeParser check style cleanup and optimization > --- > > Key: METRON-1858 > URL: https://issues.apache.org/jira/browse/METRON-1858 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > Labels: patch-pending > > Compiling regex's every parse isn't not very performant and the code doesn't > pass checkstyle. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1864) Stellar date format test fails after daylight saving
[ https://issues.apache.org/jira/browse/METRON-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680012#comment-16680012 ] ASF GitHub Bot commented on METRON-1864: GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/1258 METRON-1864 fix issue where daylight savings breaks tests Thanks to @justinleet for the fix. I would have use a totally awesome but overdone regex. ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [-] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [-] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [-] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [-] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [-] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron stellar-datefunction-tz-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1258.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1258 commit b89068d2a52e5e6ceb7c925d6ca3e14ee8fc15f6 Author: Otto Fowler Date: 2018-11-08T16:12:40Z fix issue where daylight savings breaks tests > Stellar date format test fails after daylight saving > > > Key: METRON-1864 > URL: https://issues.apache.org/jira/browse/METRON-1864 > Project: Metron > Issue Type: Bug >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.62 sec - > in org.apache.metron.stellar.dsl.functions.FunctionalFunctionsTest > Results : > Failed tests: > DateFunctionsTest.testDateFormatDefault:239 null -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1858) BasicFireEyeParser check style cleanup and optimization
[ https://issues.apache.org/jira/browse/METRON-1858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679845#comment-16679845 ] ASF GitHub Bot commented on METRON-1858: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1255 Right @justinleet, I forgot about that. > BasicFireEyeParser check style cleanup and optimization > --- > > Key: METRON-1858 > URL: https://issues.apache.org/jira/browse/METRON-1858 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > Labels: patch-pending > > Compiling regex's every parse isn't not very performant and the code doesn't > pass checkstyle. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1749) Update Angular to latest release in Management UI
[ https://issues.apache.org/jira/browse/METRON-1749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679800#comment-16679800 ] ASF GitHub Bot commented on METRON-1749: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1217 @tiborm Great, thanks for running through that effort. We should definitely get that automated as part of the build. @sardell +1, it's great to see this stuff getting properly updated > Update Angular to latest release in Management UI > - > > Key: METRON-1749 > URL: https://issues.apache.org/jira/browse/METRON-1749 > Project: Metron > Issue Type: Improvement >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > > Currently, the Management UI is on Angular v2. Not only has there been many > updates and improvements to Angular, that release is no longer supported by > the Angular team. This means critical fixes and security patches are no > longer being done. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1861) REST fails to start when LDAP enabled and 'Active Spring profiles' config is empty
[
https://issues.apache.org/jira/browse/METRON-1861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anand Subramanian updated METRON-1861:
--
Description:
When 'LDAP enabled' (in Configs -> Metron -> Security tab) is set to 'On' and
the 'Active Spring profiles' (in Configs -> Metron -> REST tab) is blank, REST
service fails to start with the following error:
{code:java}
18/11/08 08:10:02 ERROR boot.SpringApplication: Application run failed
java.lang.IllegalArgumentException: Invalid profile []: must contain text
at
org.springframework.core.env.AbstractEnvironment.validateProfile(AbstractEnvironment.java:365)
at
org.springframework.core.env.AbstractEnvironment.setActiveProfiles(AbstractEnvironment.java:256)
at
org.springframework.core.env.AbstractEnvironment.doGetActiveProfiles(AbstractEnvironment.java:242)
at
org.springframework.core.env.AbstractEnvironment.getActiveProfiles(AbstractEnvironment.java:226)
at
org.springframework.boot.SpringApplication.configureProfiles(SpringApplication.java:522)
at
org.springframework.boot.SpringApplication.configureEnvironment(SpringApplication.java:479)
at
org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:357)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:317)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1255)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1243)
at
org.apache.metron.rest.MetronRestApplication.main(MetronRestApplication.java:36){code}
This is because the 'metron_spring_profiles_active' is set to the value ',ldap'
which is not valid.
> REST fails to start when LDAP enabled and 'Active Spring profiles' config is
> empty
> --
>
> Key: METRON-1861
> URL: https://issues.apache.org/jira/browse/METRON-1861
> Project: Metron
> Issue Type: Bug
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>Priority: Major
>
> When 'LDAP enabled' (in Configs -> Metron -> Security tab) is set to 'On' and
> the 'Active Spring profiles' (in Configs -> Metron -> REST tab) is blank,
> REST service fails to start with the following error:
>
> {code:java}
> 18/11/08 08:10:02 ERROR boot.SpringApplication: Application run failed
> java.lang.IllegalArgumentException: Invalid profile []: must contain text
> at
> org.springframework.core.env.AbstractEnvironment.validateProfile(AbstractEnvironment.java:365)
> at
> org.springframework.core.env.AbstractEnvironment.setActiveProfiles(AbstractEnvironment.java:256)
> at
> org.springframework.core.env.AbstractEnvironment.doGetActiveProfiles(AbstractEnvironment.java:242)
> at
> org.springframework.core.env.AbstractEnvironment.getActiveProfiles(AbstractEnvironment.java:226)
> at
> org.springframework.boot.SpringApplication.configureProfiles(SpringApplication.java:522)
> at
> org.springframework.boot.SpringApplication.configureEnvironment(SpringApplication.java:479)
> at
> org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:357)
> at org.springframework.boot.SpringApplication.run(SpringApplication.java:317)
> at
> org.springframework.boot.SpringApplication.run(SpringApplication.java:1255)
> at
> org.springframework.boot.SpringApplication.run(SpringApplication.java:1243)
> at
> org.apache.metron.rest.MetronRestApplication.main(MetronRestApplication.java:36){code}
>
> This is because the 'metron_spring_profiles_active' is set to the value
> ',ldap' which is not valid.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1861) REST fails to start when LDAP enabled and 'Active Spring profiles' config is empty
[
https://issues.apache.org/jira/browse/METRON-1861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679801#comment-16679801
]
ASF GitHub Bot commented on METRON-1861:
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/1256
+1, good catch.
> REST fails to start when LDAP enabled and 'Active Spring profiles' config is
> empty
> --
>
> Key: METRON-1861
> URL: https://issues.apache.org/jira/browse/METRON-1861
> Project: Metron
> Issue Type: Bug
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>Priority: Major
>
> When 'LDAP enabled' (in Configs -> Metron -> Security tab) is set to 'On' and
> the 'Active Spring profiles' (in Configs -> Metron -> REST tab) is blank,
> REST service fails to start with the following error:
>
> {code:java}
> 18/11/08 08:10:02 ERROR boot.SpringApplication: Application run failed
> java.lang.IllegalArgumentException: Invalid profile []: must contain text
> at
> org.springframework.core.env.AbstractEnvironment.validateProfile(AbstractEnvironment.java:365)
> at
> org.springframework.core.env.AbstractEnvironment.setActiveProfiles(AbstractEnvironment.java:256)
> at
> org.springframework.core.env.AbstractEnvironment.doGetActiveProfiles(AbstractEnvironment.java:242)
> at
> org.springframework.core.env.AbstractEnvironment.getActiveProfiles(AbstractEnvironment.java:226)
> at
> org.springframework.boot.SpringApplication.configureProfiles(SpringApplication.java:522)
> at
> org.springframework.boot.SpringApplication.configureEnvironment(SpringApplication.java:479)
> at
> org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:357)
> at org.springframework.boot.SpringApplication.run(SpringApplication.java:317)
> at
> org.springframework.boot.SpringApplication.run(SpringApplication.java:1255)
> at
> org.springframework.boot.SpringApplication.run(SpringApplication.java:1243)
> at
> org.apache.metron.rest.MetronRestApplication.main(MetronRestApplication.java:36){code}
>
> This is because the 'metron_spring_profiles_active' is set to the value
> ',ldap' which is not valid.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1834) Migrate Elasticsearch from TransportClient to new Java REST API
[ https://issues.apache.org/jira/browse/METRON-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679855#comment-16679855 ] ASF GitHub Bot commented on METRON-1834: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1242 @mmiklavc +1 remains. Addition to upgrade doc looks good. A user may just have to translate from old properties to new. > Migrate Elasticsearch from TransportClient to new Java REST API > --- > > Key: METRON-1834 > URL: https://issues.apache.org/jira/browse/METRON-1834 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1255: METRON-1858 Fix Fireeye formatting and some optim...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1255 ---
[jira] [Commented] (METRON-1864) Stellar date format test fails after daylight saving
[ https://issues.apache.org/jira/browse/METRON-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680132#comment-16680132 ] ASF GitHub Bot commented on METRON-1864: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1258 +1 > Stellar date format test fails after daylight saving > > > Key: METRON-1864 > URL: https://issues.apache.org/jira/browse/METRON-1864 > Project: Metron > Issue Type: Bug >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.62 sec - > in org.apache.metron.stellar.dsl.functions.FunctionalFunctionsTest > Results : > Failed tests: > DateFunctionsTest.testDateFormatDefault:239 null -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1864) Stellar date format test fails after daylight saving
[ https://issues.apache.org/jira/browse/METRON-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680215#comment-16680215 ] ASF GitHub Bot commented on METRON-1864: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1258 > Stellar date format test fails after daylight saving > > > Key: METRON-1864 > URL: https://issues.apache.org/jira/browse/METRON-1864 > Project: Metron > Issue Type: Bug >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.62 sec - > in org.apache.metron.stellar.dsl.functions.FunctionalFunctionsTest > Results : > Failed tests: > DateFunctionsTest.testDateFormatDefault:239 null -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1258: METRON-1864 fix issue where daylight savings brea...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1258 ---
[jira] [Commented] (METRON-1815) Separate metron-parsers into metron-parsers-common and metron-parsers-storm
[ https://issues.apache.org/jira/browse/METRON-1815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679872#comment-16679872 ] ASF GitHub Bot commented on METRON-1815: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1249 Yes @justinleet exactly that. Sorry I didn't explain it more betterer > Separate metron-parsers into metron-parsers-common and metron-parsers-storm > --- > > Key: METRON-1815 > URL: https://issues.apache.org/jira/browse/METRON-1815 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > In order to expose our parsers to 3rd party components (e.g. the discussions > on NiFi and potentially other platforms like Spark), we should > separate the storm-bits into its own project. The metron-parsers-common > project should contain only parser-oriented code, whereas the > metron-parsers-storm project should contain the storm specific code > (e.g. the parser bolt). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1864) Stellar date format test fails after daylight saving
[ https://issues.apache.org/jira/browse/METRON-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680102#comment-16680102 ] ASF GitHub Bot commented on METRON-1864: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1258 @ottobackwards I'm good with that > Stellar date format test fails after daylight saving > > > Key: METRON-1864 > URL: https://issues.apache.org/jira/browse/METRON-1864 > Project: Metron > Issue Type: Bug >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.62 sec - > in org.apache.metron.stellar.dsl.functions.FunctionalFunctionsTest > Results : > Failed tests: > DateFunctionsTest.testDateFormatDefault:239 null -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1864) Stellar date format test fails after daylight saving
[ https://issues.apache.org/jira/browse/METRON-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680036#comment-16680036 ] ASF GitHub Bot commented on METRON-1864: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1258 Thanks @ottobackwards and @justinleet! Did you mean to completely remove the assertions for that test? It seems like we would still want to validate the results somehow. Also, since the context for this PR came from discussions in Slack that don't show up on the dev list, can you add a comment about the motivation/origin for this fix? > Stellar date format test fails after daylight saving > > > Key: METRON-1864 > URL: https://issues.apache.org/jira/browse/METRON-1864 > Project: Metron > Issue Type: Bug >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.62 sec - > in org.apache.metron.stellar.dsl.functions.FunctionalFunctionsTest > Results : > Failed tests: > DateFunctionsTest.testDateFormatDefault:239 null -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1749) Update Angular to latest release in Management UI
[ https://issues.apache.org/jira/browse/METRON-1749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679824#comment-16679824 ] ASF GitHub Bot commented on METRON-1749: Github user tiborm commented on the issue: https://github.com/apache/metron/pull/1217 @justinleet Thanks! Fully agree. Created a followup ticket to track Travis integration of license checking. https://issues.apache.org/jira/browse/METRON-1863 > Update Angular to latest release in Management UI > - > > Key: METRON-1749 > URL: https://issues.apache.org/jira/browse/METRON-1749 > Project: Metron > Issue Type: Improvement >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > > Currently, the Management UI is on Angular v2. Not only has there been many > updates and improvements to Angular, that release is no longer supported by > the Angular team. This means critical fixes and security patches are no > longer being done. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1815) Separate metron-parsers into metron-parsers-common and metron-parsers-storm
[ https://issues.apache.org/jira/browse/METRON-1815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679823#comment-16679823 ] ASF GitHub Bot commented on METRON-1815: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1249 @ottobackwards I was sick the past couple days, so sorry about the delay in responding. Hopefully, what we're saying is, for this PR, refactor this to ``` └── metron-parsing ├── metron-parsers ├── metron-parsers-common └── metron-parsing-storm ``` Where metron-parsers-common contains the various utils, plus the building block parsers (e.g. Grok). Whenever another platform gets added, it would go in "metron-parsing-" and they can create "metron-parsing-common" if needed (because right now it would be empty). Sound like a plan? > Separate metron-parsers into metron-parsers-common and metron-parsers-storm > --- > > Key: METRON-1815 > URL: https://issues.apache.org/jira/browse/METRON-1815 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > In order to expose our parsers to 3rd party components (e.g. the discussions > on NiFi and potentially other platforms like Spark), we should > separate the storm-bits into its own project. The metron-parsers-common > project should contain only parser-oriented code, whereas the > metron-parsers-storm project should contain the storm specific code > (e.g. the parser bolt). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1858) BasicFireEyeParser check style cleanup and optimization
[ https://issues.apache.org/jira/browse/METRON-1858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679815#comment-16679815 ] ASF GitHub Bot commented on METRON-1858: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1255 Sidenote, the functional review as a whole is much more straightforward if you turn off whitespace changes in the diff settings. I think that's why @ottobackwards has the disclaimer about reviewing by change. > BasicFireEyeParser check style cleanup and optimization > --- > > Key: METRON-1858 > URL: https://issues.apache.org/jira/browse/METRON-1858 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > Labels: patch-pending > > Compiling regex's every parse isn't not very performant and the code doesn't > pass checkstyle. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1258: METRON-1864 fix issue where daylight savings breaks test...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1258 @ottobackwards I'm good with that ---
[jira] [Commented] (METRON-1850) Stellar REST function
[ https://issues.apache.org/jira/browse/METRON-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680025#comment-16680025 ] ASF GitHub Bot commented on METRON-1850: Github user merrimanr closed the pull request at: https://github.com/apache/metron/pull/1250 > Stellar REST function > - > > Key: METRON-1850 > URL: https://issues.apache.org/jira/browse/METRON-1850 > Project: Metron > Issue Type: New Feature >Reporter: Ryan Merriman >Priority: Major > > It would be useful to be able to enrich messages with Stellar using 3rd party > (or internal) REST services. At a minimum this function would: > * Stellar function available to GET from an HTTP API > * Optional parameters for basic auth (user/password) which generate correct > Authorization header > * Function returns null value for errors, connection failures etc and logs > error > * Function must provide and use pooled connection objects at the process > level > * Function must send Accept: application/json header > * A global setting must be available to set a proxy for all API calls, and > if present the proxy must be used. > * Proxy authentication must also be supported using basic auth. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1850) Stellar REST function
[ https://issues.apache.org/jira/browse/METRON-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680031#comment-16680031 ] ASF GitHub Bot commented on METRON-1850: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1250 @merrimanr Pending a successful Travis run, I'm +1 by inspection. Thanks! > Stellar REST function > - > > Key: METRON-1850 > URL: https://issues.apache.org/jira/browse/METRON-1850 > Project: Metron > Issue Type: New Feature >Reporter: Ryan Merriman >Priority: Major > > It would be useful to be able to enrich messages with Stellar using 3rd party > (or internal) REST services. At a minimum this function would: > * Stellar function available to GET from an HTTP API > * Optional parameters for basic auth (user/password) which generate correct > Authorization header > * Function returns null value for errors, connection failures etc and logs > error > * Function must provide and use pooled connection objects at the process > level > * Function must send Accept: application/json header > * A global setting must be available to set a proxy for all API calls, and > if present the proxy must be used. > * Proxy authentication must also be supported using basic auth. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1304) Allow metron-bro-plugin-kafka to include or exclude logs
[ https://issues.apache.org/jira/browse/METRON-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679990#comment-16679990 ] ASF GitHub Bot commented on METRON-1304: Github user JonZeolla commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/2 In looking through the PR, this is mostly related to the include/exclude functionality yes, but you are right that it may look like there are some other things included because: * As a part of updating the documentation for these new features I attempted to make things more uniform in format (which included some text reordering), and I added details regarding what a successful `bro-pkg` install looks like (before this PR that was broken/undocumented) * This also fixes the previously broken btest which was a 1 line change to `tests/Baseline/kafka.show-plugin/output` * There are some minor documentation changes such as changing suggested priority to -10, or adding multiple brokers to the docs in response to a [review comment](https://github.com/apache/metron-bro-plugin-kafka/pull/2/files#r224996990). > Allow metron-bro-plugin-kafka to include or exclude logs > > > Key: METRON-1304 > URL: https://issues.apache.org/jira/browse/METRON-1304 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Major > > Right now, you must specify which logs you want to send to kafka via > metron-bro-plugin-kafka. This would allow the additional feature of > excluding certain logs, and sending everything else. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1864) Stellar date format test fails after daylight saving
[ https://issues.apache.org/jira/browse/METRON-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680093#comment-16680093 ] ASF GitHub Bot commented on METRON-1864: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1258 I think that the test logically tests what we are concerned about, that the format of the produced string matches the desired format. Testing that the value is explicitly x,y,z isn't testing the format, it is testing something else. I propose, with a mind to getting master and pr's building again, that I comment as requested but not change the test, and that any other ideas for improvement and or expansion of the tests be put to a jira for later work. > Stellar date format test fails after daylight saving > > > Key: METRON-1864 > URL: https://issues.apache.org/jira/browse/METRON-1864 > Project: Metron > Issue Type: Bug >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.62 sec - > in org.apache.metron.stellar.dsl.functions.FunctionalFunctionsTest > Results : > Failed tests: > DateFunctionsTest.testDateFormatDefault:239 null -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1864) Stellar date format test fails after daylight saving
[ https://issues.apache.org/jira/browse/METRON-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680071#comment-16680071 ] ASF GitHub Bot commented on METRON-1864: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1258 @mmiklavc Right now the function just tests that the result is properly formatted (which is really all the original test did, but it just checked the string matched a timezone). This will throw an exception if it's not. The problem with validating further on this test is that it's essentially against "now", which means that we don't know exactly what the date is, other than what we get back. Anything we choose would be slightly off. We could probably refactor things around to make it possible to inject a given timestamp for a no arg run of the statement, but I'm not convinced that needs to happen to resolve the current unit test problem. > Stellar date format test fails after daylight saving > > > Key: METRON-1864 > URL: https://issues.apache.org/jira/browse/METRON-1864 > Project: Metron > Issue Type: Bug >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.62 sec - > in org.apache.metron.stellar.dsl.functions.FunctionalFunctionsTest > Results : > Failed tests: > DateFunctionsTest.testDateFormatDefault:239 null -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1864) Stellar date format test fails after daylight saving
[ https://issues.apache.org/jira/browse/METRON-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680080#comment-16680080 ] ASF GitHub Bot commented on METRON-1864: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1258 @justinleet We do use mock clocks in other parts of the platform, but for the sake of this fix I think this is fine. @ottobackwards Can we just throw a comment in the test so it's obvious that this is intentionally without assertions here? > Stellar date format test fails after daylight saving > > > Key: METRON-1864 > URL: https://issues.apache.org/jira/browse/METRON-1864 > Project: Metron > Issue Type: Bug >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.62 sec - > in org.apache.metron.stellar.dsl.functions.FunctionalFunctionsTest > Results : > Failed tests: > DateFunctionsTest.testDateFormatDefault:239 null -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1850) Stellar REST function
[
https://issues.apache.org/jira/browse/METRON-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680026#comment-16680026
]
ASF GitHub Bot commented on METRON-1850:
GitHub user merrimanr reopened a pull request:
https://github.com/apache/metron/pull/1250
METRON-1850: Stellar REST function
## Contributor Comments
This PR adds a Stellar REST function that can be used to enrich messages
with data from 3rd party REST services. This function leverages the Apache
HttpComponents library for Http requests.
The function call follows this format: `REST_GET(rest uri, optional rest
settings)`
There are a handful of settings including basic authentication credentials,
proxy support, and timeouts. These settings are included in the `RestConfig`
class. Any of these settings can be defined in the global config under the
`stellar.rest.settings` property and will override any default values. The
global config settings can also be overridden by passing in a config object to
the expression. This will allow support for multiple REST services that may
have different requirements.
Responses are expected to be in JSON format. Successful requests will
return a MAP object in Stellar. Errors will be logged and null will be
returned by default. There are ways to override this behavior by configuring a
list of acceptable status codes and/or values to be returned on error or empty
responses.
Considering this will be used on streaming data, how we handle timeouts is
important. This function exposes HttpClient timeout settings but those are not
enough to keep unwanted latency from being introduced. A hard timeout is
implemented in the function to abort a request if the timeout is exceeded. The
idea is to guarantee the total request time will not exceed a configured value.
### Changes Included
- HttpClient capability added to Stellar
- HttpClient setup added to the various bolts and Stellar REPL
- Utility added for setting up a pooling HttpClient (and possibly other
types of clients in the future)
- Configuration mechanism added for Stellar REST function including
settings for authentication, proxy support, timeouts and other settings
- Function implementation and appropriate unit/integration tests (both unit
tests and integration tests are included)
### Testing
There are several different ways to test this feature and I would encourage
reviewers to get creative and look for cases I may not have thought of. For my
testing, I used an online Http service that provides simple endpoints for
simulating different use cases: `http://httpbin.org/#/`. Feel free to try
your own or use this one.
I tested this in full dev using the Stellar REPL and the parser and
enrichment topologies. First you need to perform a couple setup steps:
1. Spin up full dev and ensure everything comes up and data is being indexed
2. Ssh to full dev and install the Squid proxy server:
```
yum -y install squid
```
3. Create a password file that Squid can use for basic authentication
```
yum -y install httpd-tools
touch /etc/squid/passwd && chown squid /etc/squid/passwd
htpasswd /etc/squid/passwd user # (Will prompt for a password)
```
4. Configure Squid for basic authentication by adding these lines to
`/etc/squid/squid.conf`, under the lines with `acl Safe_ports*`:
```
auth_param basic program /usr/lib64/squid/ncsa_auth /etc/squid/passwd
auth_param basic children 5
auth_param basic realm Squid Basic Authentication
auth_param basic credentialsttl 2 hours
acl auth_users proxy_auth REQUIRED
http_access allow auth_users
```
5. Start Squid and verify it is working correctly:
```
service squid restart
curl --proxy-user user:password -x node1:3128 http://www.google.com/
```
6. Next create password files in HDFS:
```
su hdfs
cd ~
echo -n 'passwd' > basicPassword.txt
hdfs dfs -put basicPassword.txt /apps/metron
echo -n 'password' > proxyPassword.txt
hdfs dfs -put proxyPassword.txt /apps/metron
exit
```
To test with the Stellar REPL, follow these steps:
1. Start the Stellar REPL and verify the `REST_GET` function is available:
```
/usr/metron/0.6.1/bin/stellar --zookeeper node1:2181
[Stellar]>>> %functions REST
REST_GET
```
2. Test a simple get request:
```
[Stellar]>>> REST_GET('http://httpbin.org/get')
{args={}, headers={Accept=application/json, Accept-Encoding=gzip,deflate,
Cache-Control=max-age=259200, Connection=close, Host=httpbin.org,
User-Agent=Apache-HttpClient/4.3.2 (java 1.5)}, origin=127.0.0.1,
136.62.241.236, url=http://httpbin.org/get}
```
3. Test a get request with basic authentication:
```
[Stellar]>>> config :=
[GitHub] metron issue #1258: METRON-1864 fix issue where daylight savings breaks test...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1258 I think that the test logically tests what we are concerned about, that the format of the produced string matches the desired format. Testing that the value is explicitly x,y,z isn't testing the format, it is testing something else. I propose, with a mind to getting master and pr's building again, that I comment as requested but not change the test, and that any other ideas for improvement and or expansion of the tests be put to a jira for later work. ---
[jira] [Commented] (METRON-1862) Cant Load Metron Bro Plugin
[ https://issues.apache.org/jira/browse/METRON-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679635#comment-16679635 ] ASF GitHub Bot commented on METRON-1862: GitHub user mustafakmal opened a pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/15 METRON-1862 ## Contributor Comments The previous Load path was "@load packages/metron-bro-plugin-kafka/Apache/Kafka". However, in the new version the path to load the metron bro plugin is "packages/metron-bro-plugin-kafka/scripts/Apache/Kafka" When I loaded the plugin from the old path that is "packages/metron-bro-plugin-kafka/Apache/Kafka" I received the following error when running 'broctl deploy' : can't find packages/metron-bro-plugin-kafka/Apache/Kafka Therefore to correctly load the plugin, ensure your load path is correct like this: @load /metron-bro-plugin-kafka/scripts/Apache/Kafka ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron's Bro kafka writer plugin. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed via: ``` bro-pkg test $GITHUB_USERNAME/metron-bro-plugin-kafka --version $BRANCH ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Apache Metron's [Vagrant full-dev environment](https://github.com/apache/metron/tree/master/metron-deployment/development/centos6) or the equivalent? You can merge this pull request into a Git repository by running: $ git pull https://github.com/mustafakmal/metron-bro-plugin-kafka master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron-bro-plugin-kafka/pull/15.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #15 commit 186467d76db2ef46ba2040626c9f4072803dfdb6 Author: Mustafa Akmal Date: 2018-11-07T09:27:44Z Updated The Path To Load Metron Bro Plugin The previous Load path was "@load packages/metron-bro-plugin-kafka/Apache/Kafka". However, in the new version the path to load the metron bro plugin is "packages/metron-bro-plugin-kafka/scripts/Apache/Kafka" When I loaded the plugin from the old path that is "packages/metron-bro-plugin-kafka/Apache/Kafka" I received the following error when running 'broctl deploy' : can't find packages/metron-bro-plugin-kafka/Apache/Kafka Therefore to correctly load the plugin, ensure your load path is correct like this: @load /metron-bro-plugin-kafka/scripts/Apache/Kafka > Cant Load Metron Bro Plugin > --- > > Key: METRON-1862 > URL: https://issues.apache.org/jira/browse/METRON-1862 > Project: Metron > Issue Type: Bug >Affects Versions: 0.4.1 >Reporter: Mustafa Akmal >Priority: Blocker > Labels: Bro, metron, plugin > > I received the following error when running 'broctl deploy' : can't find > packages/metron-bro-plugin-kafka/Apache/Kafka > This happens when I load the metron bro plugin by making changes to > 'local.bro' by inserting the following line "[@load|https://github.com/load] > packages/metron-bro-plugin-kafka/Apache/Kafka" > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1836) Finish updating README documentation including migration guide
[ https://issues.apache.org/jira/browse/METRON-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679195#comment-16679195 ] Michael Miklavcic commented on METRON-1836: --- Also added note in Upgrading.md > Finish updating README documentation including migration guide > -- > > Key: METRON-1836 > URL: https://issues.apache.org/jira/browse/METRON-1836 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1850) Stellar REST function
[ https://issues.apache.org/jira/browse/METRON-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680019#comment-16680019 ] ASF GitHub Bot commented on METRON-1850: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1250 Travis failure does not appear related here > Stellar REST function > - > > Key: METRON-1850 > URL: https://issues.apache.org/jira/browse/METRON-1850 > Project: Metron > Issue Type: New Feature >Reporter: Ryan Merriman >Priority: Major > > It would be useful to be able to enrich messages with Stellar using 3rd party > (or internal) REST services. At a minimum this function would: > * Stellar function available to GET from an HTTP API > * Optional parameters for basic auth (user/password) which generate correct > Authorization header > * Function returns null value for errors, connection failures etc and logs > error > * Function must provide and use pooled connection objects at the process > level > * Function must send Accept: application/json header > * A global setting must be available to set a proxy for all API calls, and > if present the proxy must be used. > * Proxy authentication must also be supported using basic auth. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1836) Finish updating README documentation including migration guide
[ https://issues.apache.org/jira/browse/METRON-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679181#comment-16679181 ] Michael Miklavcic commented on METRON-1836: --- Added to metron-elasticsearch > Finish updating README documentation including migration guide > -- > > Key: METRON-1836 > URL: https://issues.apache.org/jira/browse/METRON-1836 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1861) REST fails to start when LDAP enabled and 'Active Spring profiles' config is empty
[
https://issues.apache.org/jira/browse/METRON-1861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679495#comment-16679495
]
ASF GitHub Bot commented on METRON-1861:
GitHub user anandsubbu opened a pull request:
https://github.com/apache/metron/pull/1256
METRON-1861: REST fails to start when LDAP enabled and 'Active Spring
profiles' config is empty
## Contributor Comments
Set 'Active Spring profiles' to just `ldap`, when
`metron_spring_profiles_active` is blank.
### Testing Steps
1. Set 'LDAP Enabled' to On and set 'Active Spring profiles' as blank
2. Restart REST service and validate that it starts up fine.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/anandsubbu/incubator-metron METRON-1861
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1256.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1256
commit ad781b6da1b03524ea63c318ebe68c47758c5b64
Author: Anand Subramanian
Date: 2018-11-08T09:24:45Z
Handle case when metron_spring_profiles_active is empty
> REST fails to start when LDAP enabled and 'Active Spring profiles' config is
> empty
> --
>
> Key: METRON-1861
> URL: https://issues.apache.org/jira/browse/METRON-1861
> Project: Metron
> Issue Type: Bug
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>Priority: Major
>
> When 'LDAP enabled' (in Configs -> Metron -> Security tab) is set to 'On' and
> the 'Active Spring profiles' (in Configs -> Metron -> REST tab) is blank,
> REST service fails to start with the following error:
>
> {code:java}
> 18/11/08 08:10:02 ERROR boot.SpringApplication: Application run failed
> java.lang.IllegalArgumentException: Invalid profile []: must contain text
> at
>
[jira] [Created] (METRON-1861) REST fails to start when LDAP enabled and 'Active Spring profiles' config is empty
Anand Subramanian created METRON-1861: - Summary: REST fails to start when LDAP enabled and 'Active Spring profiles' config is empty Key: METRON-1861 URL: https://issues.apache.org/jira/browse/METRON-1861 Project: Metron Issue Type: Bug Reporter: Anand Subramanian Assignee: Anand Subramanian -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1258: METRON-1864 fix issue where daylight savings breaks test...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1258 @justinleet We do use mock clocks in other parts of the platform, but for the sake of this fix I think this is fine. @ottobackwards Can we just throw a comment in the test so it's obvious that this is intentionally without assertions here? ---
[GitHub] metron issue #1258: METRON-1864 fix issue where daylight savings breaks test...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1258 @mmiklavc Right now the function just tests that the result is properly formatted (which is really all the original test did, but it just checked the string matched a timezone). This will throw an exception if it's not. The problem with validating further on this test is that it's essentially against "now", which means that we don't know exactly what the date is, other than what we get back. Anything we choose would be slightly off. We could probably refactor things around to make it possible to inject a given timestamp for a no arg run of the statement, but I'm not convinced that needs to happen to resolve the current unit test problem. ---
[GitHub] metron issue #1258: METRON-1864 fix issue where daylight savings breaks test...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1258 Thanks @ottobackwards and @justinleet! Did you mean to completely remove the assertions for that test? It seems like we would still want to validate the results somehow. Also, since the context for this PR came from discussions in Slack that don't show up on the dev list, can you add a comment about the motivation/origin for this fix? ---
[GitHub] metron issue #1250: METRON-1850: Stellar REST function
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1250 @merrimanr Pending a successful Travis run, I'm +1 by inspection. Thanks! ---
[GitHub] metron pull request #1250: METRON-1850: Stellar REST function
Github user merrimanr closed the pull request at: https://github.com/apache/metron/pull/1250 ---
[GitHub] metron pull request #1250: METRON-1850: Stellar REST function
GitHub user merrimanr reopened a pull request:
https://github.com/apache/metron/pull/1250
METRON-1850: Stellar REST function
## Contributor Comments
This PR adds a Stellar REST function that can be used to enrich messages
with data from 3rd party REST services. This function leverages the Apache
HttpComponents library for Http requests.
The function call follows this format: `REST_GET(rest uri, optional rest
settings)`
There are a handful of settings including basic authentication credentials,
proxy support, and timeouts. These settings are included in the `RestConfig`
class. Any of these settings can be defined in the global config under the
`stellar.rest.settings` property and will override any default values. The
global config settings can also be overridden by passing in a config object to
the expression. This will allow support for multiple REST services that may
have different requirements.
Responses are expected to be in JSON format. Successful requests will
return a MAP object in Stellar. Errors will be logged and null will be
returned by default. There are ways to override this behavior by configuring a
list of acceptable status codes and/or values to be returned on error or empty
responses.
Considering this will be used on streaming data, how we handle timeouts is
important. This function exposes HttpClient timeout settings but those are not
enough to keep unwanted latency from being introduced. A hard timeout is
implemented in the function to abort a request if the timeout is exceeded. The
idea is to guarantee the total request time will not exceed a configured value.
### Changes Included
- HttpClient capability added to Stellar
- HttpClient setup added to the various bolts and Stellar REPL
- Utility added for setting up a pooling HttpClient (and possibly other
types of clients in the future)
- Configuration mechanism added for Stellar REST function including
settings for authentication, proxy support, timeouts and other settings
- Function implementation and appropriate unit/integration tests (both unit
tests and integration tests are included)
### Testing
There are several different ways to test this feature and I would encourage
reviewers to get creative and look for cases I may not have thought of. For my
testing, I used an online Http service that provides simple endpoints for
simulating different use cases: `http://httpbin.org/#/`. Feel free to try
your own or use this one.
I tested this in full dev using the Stellar REPL and the parser and
enrichment topologies. First you need to perform a couple setup steps:
1. Spin up full dev and ensure everything comes up and data is being indexed
2. Ssh to full dev and install the Squid proxy server:
```
yum -y install squid
```
3. Create a password file that Squid can use for basic authentication
```
yum -y install httpd-tools
touch /etc/squid/passwd && chown squid /etc/squid/passwd
htpasswd /etc/squid/passwd user # (Will prompt for a password)
```
4. Configure Squid for basic authentication by adding these lines to
`/etc/squid/squid.conf`, under the lines with `acl Safe_ports*`:
```
auth_param basic program /usr/lib64/squid/ncsa_auth /etc/squid/passwd
auth_param basic children 5
auth_param basic realm Squid Basic Authentication
auth_param basic credentialsttl 2 hours
acl auth_users proxy_auth REQUIRED
http_access allow auth_users
```
5. Start Squid and verify it is working correctly:
```
service squid restart
curl --proxy-user user:password -x node1:3128 http://www.google.com/
```
6. Next create password files in HDFS:
```
su hdfs
cd ~
echo -n 'passwd' > basicPassword.txt
hdfs dfs -put basicPassword.txt /apps/metron
echo -n 'password' > proxyPassword.txt
hdfs dfs -put proxyPassword.txt /apps/metron
exit
```
To test with the Stellar REPL, follow these steps:
1. Start the Stellar REPL and verify the `REST_GET` function is available:
```
/usr/metron/0.6.1/bin/stellar --zookeeper node1:2181
[Stellar]>>> %functions REST
REST_GET
```
2. Test a simple get request:
```
[Stellar]>>> REST_GET('http://httpbin.org/get')
{args={}, headers={Accept=application/json, Accept-Encoding=gzip,deflate,
Cache-Control=max-age=259200, Connection=close, Host=httpbin.org,
User-Agent=Apache-HttpClient/4.3.2 (java 1.5)}, origin=127.0.0.1,
136.62.241.236, url=http://httpbin.org/get}
```
3. Test a get request with basic authentication:
```
[Stellar]>>> config :=
{'basic.auth.user':'user','basic.auth.password.path':'/apps/metron/basicPassword.txt'}
{basic.auth.user=user,
basic.auth.password.path=/apps/metron/basicPassword.txt}
[Stellar]>>> REST_GET('http://httpbin.org/basic-auth/user/passwd',
[GitHub] metron issue #1250: METRON-1850: Stellar REST function
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1250 Travis failure does not appear related here ---
[GitHub] metron pull request #1258: METRON-1864 fix issue where daylight savings brea...
GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/1258 METRON-1864 fix issue where daylight savings breaks tests Thanks to @justinleet for the fix. I would have use a totally awesome but overdone regex. ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [-] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [-] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [-] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [-] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [-] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron stellar-datefunction-tz-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1258.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1258 commit b89068d2a52e5e6ceb7c925d6ca3e14ee8fc15f6 Author: Otto Fowler Date: 2018-11-08T16:12:40Z fix issue where daylight savings breaks tests ---
[GitHub] metron-bro-plugin-kafka issue #2: METRON-1304: Allow metron-bro-plugin-kafka...
Github user JonZeolla commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/2 In looking through the PR, this is mostly related to the include/exclude functionality yes, but you are right that it may look like there are some other things included because: * As a part of updating the documentation for these new features I attempted to make things more uniform in format (which included some text reordering), and I added details regarding what a successful `bro-pkg` install looks like (before this PR that was broken/undocumented) * This also fixes the previously broken btest which was a 1 line change to `tests/Baseline/kafka.show-plugin/output` * There are some minor documentation changes such as changing suggested priority to -10, or adding multiple brokers to the docs in response to a [review comment](https://github.com/apache/metron-bro-plugin-kafka/pull/2/files#r224996990). ---
[GitHub] metron-bro-plugin-kafka issue #2: METRON-1304: Allow metron-bro-plugin-kafka...
Github user JonZeolla commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/2 https://github.com/apache/metron-bro-plugin-kafka/pull/2#issuecomment-429502420 is still accurate for testing. I'll edit the initial comment to add it. ---
[GitHub] metron issue #1249: METRON-1815: Separate metron-parsers into metron-parsers...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1249 Yes @justinleet exactly that. Sorry I didn't explain it more betterer ---
[GitHub] metron issue #1242: METRON-1834: Migrate Elasticsearch from TransportClient ...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1242 @mmiklavc +1 remains. Addition to upgrade doc looks good. A user may just have to translate from old properties to new. ---
[GitHub] metron issue #1255: METRON-1858 Fix Fireeye formatting and some optimization
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1255 Right @justinleet, I forgot about that. ---
[GitHub] metron-bro-plugin-kafka issue #2: METRON-1304: Allow metron-bro-plugin-kafka...
Github user nickwallen commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/2 @JonZeolla I just want to confirm, all of this is related only to include/exclude functionality? There seems to be more here, but maybe I am misunderstanding on first glance. ---
[GitHub] metron issue #1217: METRON-1749: Update Angular to latest release in Managem...
Github user tiborm commented on the issue: https://github.com/apache/metron/pull/1217 @justinleet Thanks! Fully agree. Created a followup ticket to track Travis integration of license checking. https://issues.apache.org/jira/browse/METRON-1863 ---
[GitHub] metron issue #1249: METRON-1815: Separate metron-parsers into metron-parsers...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1249 @ottobackwards I was sick the past couple days, so sorry about the delay in responding. Hopefully, what we're saying is, for this PR, refactor this to ``` âââ metron-parsing âââ metron-parsers âââ metron-parsers-common âââ metron-parsing-storm ``` Where metron-parsers-common contains the various utils, plus the building block parsers (e.g. Grok). Whenever another platform gets added, it would go in "metron-parsing-" and they can create "metron-parsing-common" if needed (because right now it would be empty). Sound like a plan? ---
[GitHub] metron issue #1255: METRON-1858 Fix Fireeye formatting and some optimization
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1255 +1, this is a lot cleaner, and moving the pattern out is definitely good. ---
[GitHub] metron-bro-plugin-kafka issue #2: METRON-1304: Allow metron-bro-plugin-kafka...
Github user nickwallen commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/2 @JonZeolla Can you add the latest testing instructions in the description? This seems to have evolved over time and am not sure what is needed at this point. ---
[GitHub] metron pull request #1256: METRON-1861: REST fails to start when LDAP enable...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1256 ---
[GitHub] metron issue #1256: METRON-1861: REST fails to start when LDAP enabled and '...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1256 +1, good catch. ---
[GitHub] metron issue #1217: METRON-1749: Update Angular to latest release in Managem...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1217 @tiborm Great, thanks for running through that effort. We should definitely get that automated as part of the build. @sardell +1, it's great to see this stuff getting properly updated ---
[GitHub] metron-bro-plugin-kafka pull request #15: METRON-1862
GitHub user mustafakmal opened a pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/15 METRON-1862 ## Contributor Comments The previous Load path was "@load packages/metron-bro-plugin-kafka/Apache/Kafka". However, in the new version the path to load the metron bro plugin is "packages/metron-bro-plugin-kafka/scripts/Apache/Kafka" When I loaded the plugin from the old path that is "packages/metron-bro-plugin-kafka/Apache/Kafka" I received the following error when running 'broctl deploy' : can't find packages/metron-bro-plugin-kafka/Apache/Kafka Therefore to correctly load the plugin, ensure your load path is correct like this: @load /metron-bro-plugin-kafka/scripts/Apache/Kafka ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron's Bro kafka writer plugin. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed via: ``` bro-pkg test $GITHUB_USERNAME/metron-bro-plugin-kafka --version $BRANCH ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Apache Metron's [Vagrant full-dev environment](https://github.com/apache/metron/tree/master/metron-deployment/development/centos6) or the equivalent? You can merge this pull request into a Git repository by running: $ git pull https://github.com/mustafakmal/metron-bro-plugin-kafka master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron-bro-plugin-kafka/pull/15.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #15 commit 186467d76db2ef46ba2040626c9f4072803dfdb6 Author: Mustafa Akmal Date: 2018-11-07T09:27:44Z Updated The Path To Load Metron Bro Plugin The previous Load path was "@load packages/metron-bro-plugin-kafka/Apache/Kafka". However, in the new version the path to load the metron bro plugin is "packages/metron-bro-plugin-kafka/scripts/Apache/Kafka" When I loaded the plugin from the old path that is "packages/metron-bro-plugin-kafka/Apache/Kafka" I received the following error when running 'broctl deploy' : can't find packages/metron-bro-plugin-kafka/Apache/Kafka Therefore to correctly load the plugin, ensure your load path is correct like this: @load /metron-bro-plugin-kafka/scripts/Apache/Kafka ---
[GitHub] metron pull request #1256: METRON-1861: REST fails to start when LDAP enable...
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/1256 METRON-1861: REST fails to start when LDAP enabled and 'Active Spring profiles' config is empty ## Contributor Comments Set 'Active Spring profiles' to just `ldap`, when `metron_spring_profiles_active` is blank. ### Testing Steps 1. Set 'LDAP Enabled' to On and set 'Active Spring profiles' as blank 2. Restart REST service and validate that it starts up fine. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1861 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1256.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1256 commit ad781b6da1b03524ea63c318ebe68c47758c5b64 Author: Anand Subramanian Date: 2018-11-08T09:24:45Z Handle case when metron_spring_profiles_active is empty ---
