[jira] [Commented] (METRON-2195) Add defensive log level checks when constructing logs is expensive

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894738#comment-16894738
 ] 

Simon Elliston Ball commented on METRON-2195:
-

[https://logging.apache.org/log4j/2.x/performance.html]

See the section on asynchronous parameter evaluation. That said, method calls 
may not be lazy evaluated in that case judging by Mike's results. Has anyone 
ever tried running this code through an actual profiler?

Perhaps we do have to revert to doing what the library should be doing for us 
(long live proper functional programming and async programming, and soon may it 
come to our corner of java). While we're at it, should we look at an async 
logger on the backend?

> Add defensive log level checks when constructing logs is expensive
> --
>
> Key: METRON-2195
> URL: https://issues.apache.org/jira/browse/METRON-2195
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Priority: Critical
>
> There are instances where we log, and output strings for json objects and 
> other things that are quite expensive.
> These are done regardless of the log level being enabled and can increase 
> performance significantly:
> https://gist.github.com/mmiklavc/7fd6af13bfa0ca05d9b3f4e7806c8d77
> https://github.com/apache/metron/blob/master/metron-platform/metron-writer/metron-writer-storm/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java#L127
> We need to find places where this happens, and employ the best practice check 
> for the log level being enabled ( best practice if your parameters or log 
> message construction takes a lot of time ).



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (METRON-2195) Add defensive log level checks when constructing logs is expensive

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894674#comment-16894674
 ] 

Simon Elliston Ball commented on METRON-2195:
-

Looks like that’s already fine to me. The log4j parameters pattern means those 
calls shouldn’t be executed. Are you seeing the call to message.toJsonString 
being made with log level less than trace? Or is this about finding other 
instances that don’t follow the example you give here?

> Add defensive log level checks when constructing logs is expensive
> --
>
> Key: METRON-2195
> URL: https://issues.apache.org/jira/browse/METRON-2195
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Priority: Critical
>
> There are instances where we log, and output strings for json objects and 
> other things that are quite expensive.
> These are done regardless of the log level being enabled and can increase 
> performance significantly:
> https://gist.github.com/mmiklavc/7fd6af13bfa0ca05d9b3f4e7806c8d77
> https://github.com/apache/metron/blob/master/metron-platform/metron-writer/metron-writer-storm/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java#L127
> We need to find places where this happens, and employ the best practice check 
> for the log level being enabled ( best practice if your parameters or log 
> message construction takes a lot of time ).



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (METRON-2178) Can two user access different data in one index at ElasticSearch after Apache Metron Indexing?

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881113#comment-16881113
 ] 

Simon Elliston Ball commented on METRON-2178:
-

This question really belongs on [the 
us...@metron.apache.org|mailto:the%c2%a0us...@metron.apache.org] list. What you 
need to do is use the tenant field from your key (I'm assuming you are using 
the mergeMetadata or readMetadata methods) to build the index pattern 
expression in your indexing config to route differently based on the tenant 
data.

On a side note, you should consolidate your NiFi flows and use the attributed 
expression language there to avoid creating too many NiFi processors (you only 
need one PublishKafka)

> Can two user access different data in one index at ElasticSearch after Apache 
> Metron Indexing?
> --
>
> Key: METRON-2178
> URL: https://issues.apache.org/jira/browse/METRON-2178
> Project: Metron
>  Issue Type: Task
>Affects Versions: 0.7.1
>Reporter: Rendiyono Wahyu Saputro
>Priority: Minor
>  Labels: beginner, features
> Attachments: rendi.png
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Good Morning,
>  Hi everyone
> Right now, i use two ( 2 ) TailFile on Squid Log in Apache Nifi. Each one, 
> connected to separated two ( 2 ) PutKafka with different Kafka Key but using 
> one Topic name.
> And the end, Apache Metron save indices to ElasticSearch as one ( 1 ) index.
> The question is, how can different user ( User A and User B ) to access 
> different Squid Log ( using metadata / Kafka Key ) in one ( 1 ) index.
> I want to accomplish this, because this is much efficient to parser different 
> data source using one ( 1 ) sensor. Can you help me or give a guidance about 
> this ?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-2157) To better abstract the streaming infrastructure, use Apache Beam.

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864124#comment-16864124
 ] 

Simon Elliston Ball commented on METRON-2157:
-

My one worry about it would be that it's a potentially more complex abstraction 
to tune, given the metron focus on high throughput. The beam programming model 
might fit our current use cases, but is a least common denominator. That said 
there has been a lot of work recently to abstract storm out, so beam could be a 
viable engine to run our business logic on. The challenge would then be to 
figure out the optimal runtime underneath that. I suspect this would benefit 
from a discuss thread on the mail lists.

> To better abstract the streaming infrastructure, use Apache Beam.
> -
>
> Key: METRON-2157
> URL: https://issues.apache.org/jira/browse/METRON-2157
> Project: Metron
>  Issue Type: Improvement
>Reporter: Phillip Henry
>Priority: Major
>
> Apache Beam is "[a]n advanced unified programming model" that can "Implement 
> batch and streaming data processing jobs that run on any execution engine" 
> (according to their [docs|[https://beam.apache.org/]). 
> If Metron were to use such an abstraction, there could be greater portability 
> between different offerings of streaming infrastructure (Spark, Flink etc).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-2118) Parser for LEEF data

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-2118:
---

 Summary: Parser for LEEF data
 Key: METRON-2118
 URL: https://issues.apache.org/jira/browse/METRON-2118
 Project: Metron
  Issue Type: Bug
Reporter: Simon Elliston Ball
Assignee: Simon Elliston Ball


LEEF is a popular format mainly in environments using Qradar as a SIEM. It 
would be useful for Metron to be able to consume data from Qradar, and related 
collectors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-2105) Enhance BasicAsaParser to add patterns dynamically

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-2105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832508#comment-16832508
 ] 

Simon Elliston Ball commented on METRON-2105:
-

I really like this idea. The grok parser is probably ok as a short term 
solution, maybe filtering out the messages the ASA parser doesn't cover right 
now. 

[~ottobackwards] The ciscotag field is used to determine which ASA grok pattern 
is necessary for the message, so is sort of a router to the right pattern. 
Given the variety of different message types on an ASA source, you can't just 
parse all of it with a single grok (or at least not with a sane pattern!), 
hence the tag.

> Enhance BasicAsaParser to add patterns dynamically
> --
>
> Key: METRON-2105
> URL: https://issues.apache.org/jira/browse/METRON-2105
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.7.0, 0.7.1
>Reporter: Pieter Baele
>Priority: Critical
>
> The current implementation of the ASA parser uses a static patternMap. As a 
> result, the parser needs recompilation when patterns are added. 
> An enhancement would be dyamic reload capabilities based on a mapping file 
> using ciscotag and grok patterns loaded using hdfs.
> This can be especially important regarding the number of patterns that can be 
> introduced by 3-th party commiters as the list of possible messages is quite 
> long [https://www.cisco.com/c/en/us/td/docs/security/asa/syslog/b_syslog.html]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (METRON-1923) Default Kafka consumer group for parser topology is "_parser"

[Simon Elliston Ball (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Elliston Ball reassigned METRON-1923:
---

Assignee: Simon Elliston Ball

> Default Kafka consumer group for parser topology is "_parser"
> 
>
> Key: METRON-1923
> URL: https://issues.apache.org/jira/browse/METRON-1923
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Stefan Kupstaitis-Dunkler
>Assignee: Simon Elliston Ball
>Priority: Minor
>
> The default Kafka consumer group for parser topology is "_parser".
> This will prevent users from reading from the same Kafka topic per default.
> Workaround at the moment is to configure "group.id" manually in the spout 
> config.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1923) Default Kafka consumer group for parser topology is "_parser"

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710661#comment-16710661
 ] 

Simon Elliston Ball commented on METRON-1923:
-

This would probably be better defaulted to the name of the parser, rather than 
the topic as the default. Should be a simple fix in 
[https://github.com/apache/metron/blob/877b510146456f2eed6eb12c35b1124c35b00aa5/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyBuilder.java#L208]
 to use the parser name instead of the topic.

> Default Kafka consumer group for parser topology is "_parser"
> 
>
> Key: METRON-1923
> URL: https://issues.apache.org/jira/browse/METRON-1923
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Stefan Kupstaitis-Dunkler
>Assignee: Simon Elliston Ball
>Priority: Minor
>
> The default Kafka consumer group for parser topology is "_parser".
> This will prevent users from reading from the same Kafka topic per default.
> Workaround at the moment is to configure "group.id" manually in the spout 
> config.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (METRON-1756) REST tests should use Embedded LDAP in metron-security

[Simon Elliston Ball (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Elliston Ball reassigned METRON-1756:
---

Assignee: Ryan Merriman

> REST tests should use Embedded LDAP in metron-security
> --
>
> Key: METRON-1756
> URL: https://issues.apache.org/jira/browse/METRON-1756
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> Tests in metron-rest are still using JDBC-based authentication.  These tests 
> (and all security-related tests) should use the testing infrastructure 
> introduced in METRON-1665.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1779) Front-end authentication simulation for development environments

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1779:
---

 Summary: Front-end authentication simulation for development 
environments
 Key: METRON-1779
 URL: https://issues.apache.org/jira/browse/METRON-1779
 Project: Metron
  Issue Type: Sub-task
Reporter: Simon Elliston Ball
Assignee: Simon Elliston Ball


When working on the UI components, it is common to use ng serve on a local 
machine pointing the proxy conf to a full-dev hosted instance of metron-rest. 
In order to make this work we need a development profile with test users for 
the full-dev instance, and an injector to simulate authentication in the 
absence of an SSO setup. 

This could be simulated with Basic auth injection against the test LDAP 
credentials as part of a development environment configuration on the front-end.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1754) Add users to role mapping for ldap based Metron REST

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1754:
---

 Summary: Add users to role mapping for ldap based Metron REST
 Key: METRON-1754
 URL: https://issues.apache.org/jira/browse/METRON-1754
 Project: Metron
  Issue Type: Sub-task
Reporter: Simon Elliston Ball


We need a means of mapping users to roles in Spring Security for Metron REST. 
This should be done via the new LDAP group lookups, and be similar to the way 
Hadoop groups are mapped for consistency.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1016) METRON Demo System as separate install option

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590840#comment-16590840
 ] 

Simon Elliston Ball commented on METRON-1016:
-

I agree that this would be a great step. Perhaps the demos belong in a more 
structured version on the use-cases folder, or in contrib. I have a repo I use 
of configs and scripts that have some basic demos in a standard-ish form which 
was meant to fill this gap 
[https://github.com/simonellistonball/metron-field-demos] 

I'm not sure it would benefit from an extra Mpack or Ambari service, since 
concepts like services in Ambari seem heavy for a Metron demo, but a set of 
scripts to install makes sense. We could probably also do with stripping out 
some of the default parsers into a separate section along with this, keeping 
the core Metron lightweight with the generics (Grok, Java, CSV, JSON) and 
pulling the device ones out into a separate location. That may go beyond the 
scope of this exercise but does feel related.

For the full dev load, I would say that an ansible role running post blueprint 
install would be a good full dev solution. 

> METRON Demo System as separate install option
> -
>
> Key: METRON-1016
> URL: https://issues.apache.org/jira/browse/METRON-1016
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Priority: Major
>
> Right now, the demo system ( bro, snort, yaf ) are part of the default 
> installation, and the default configurations required for those systems ( ES, 
> Enrichment ) are in the main code base.  For example bro's default enrichment 
> is the enrichment that assumes the demo setup.
> This is not really correct.  The demo system should be a separately 
> installable feature of metron, and it's configurations should be managed 
> outside the defaults for the parsers etc.
> This would allow separate development of the demo without affecting others.
> Either checkbox or a separate ambari service?
> What should the default enrichment be?
> How to have vagrant install the service



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1746) CEF lacks an ES template

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590449#comment-16590449
 ] 

Simon Elliston Ball commented on METRON-1746:
-

One thing to think about here is that it really can't have a fixed template. In 
the CEF parser we unroll CEF's custom field label mechanism into field names, 
so we do not know the name of the all the fields the parser will emit without 
parsing the data. As such, you could generate a template, but not write and 
commit one, unless you just want to create a default one for the CEF core 
fields, and have anything else just be dynamic.

> CEF lacks an ES template
> 
>
> Key: METRON-1746
> URL: https://issues.apache.org/jira/browse/METRON-1746
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Jon Zeolla
>Priority: Blocker
>
> An ES template should exist 
> here
> If you only pass in CEF data, the alerts UI will not be able to display 
> anything, and ES will return a 500 saying "Fielddata is disabled on text 
> fields by default."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1688) Pass through of CSRF protection for proxied api calls

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1688:
---

 Summary: Pass through of CSRF protection for proxied api calls
 Key: METRON-1688
 URL: https://issues.apache.org/jira/browse/METRON-1688
 Project: Metron
  Issue Type: Sub-task
Reporter: Simon Elliston Ball


With the UIs hosted through a proxy, and the potential for multiple backends, 
ownership of CSRF protection headers is unclear, and also impossible to pass 
through due to conflicts.

We should use the front-end host to protect and proxy the CSRF protection to 
allow the backend to be picky about its level of CSRF protection.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1453) Create a Generic Syslog Base Parser Capability

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550989#comment-16550989
 ] 

Simon Elliston Ball commented on METRON-1453:
-

Agreed, just a passing thought.

> Create a Generic Syslog Base Parser Capability
> --
>
> Key: METRON-1453
> URL: https://issues.apache.org/jira/browse/METRON-1453
> Project: Metron
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> We have several parsers now, with many imaginable that are based on syslog, 
> where the format is SYSLOG HEADER MESSAGE.
> With message being in a different format.  It would be great is we
> had a way to generically handle syslog headers, such that ANY parser data 
> could come over syslog.
> Either you could have a custom parser, or configure CSV or JSON such that 
> they could be the payload, such that you can handle JSON over syslog by 
> configuration only.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1453) Create a Generic Syslog Base Parser Capability

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550955#comment-16550955
 ] 

Simon Elliston Ball commented on METRON-1453:
-

A syslog parser with good date handling (maybe out existing utils rules???) and 
support for all the many wild types of syslog would be awesome, and the 
chaining goes on to solve a lot of the rest (what on earth is this device 
inside the syslog).

+1 for all this.

Should we go as far as to have our own socket listeners? Or is that really 
something for NiFi? 

> Create a Generic Syslog Base Parser Capability
> --
>
> Key: METRON-1453
> URL: https://issues.apache.org/jira/browse/METRON-1453
> Project: Metron
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> We have several parsers now, with many imaginable that are based on syslog, 
> where the format is SYSLOG HEADER MESSAGE.
> With message being in a different format.  It would be great is we
> had a way to generically handle syslog headers, such that ANY parser data 
> could come over syslog.
> Either you could have a custom parser, or configure CSV or JSON such that 
> they could be the payload, such that you can handle JSON over syslog by 
> configuration only.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1453) Create a Generic Syslog Base Parser Capability

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550847#comment-16550847
 ] 

Simon Elliston Ball commented on METRON-1453:
-

[~ottobackwards] would you see this a parser that handles both types of syslog, 
and sorts out the date debacle that is the missing year in syslog, which could 
then work with [~cestella]'s chaining stuff to apply Grok, CEF, csv yada yada 
whatever format parser on the output of this?

> Create a Generic Syslog Base Parser Capability
> --
>
> Key: METRON-1453
> URL: https://issues.apache.org/jira/browse/METRON-1453
> Project: Metron
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> We have several parsers now, with many imaginable that are based on syslog, 
> where the format is SYSLOG HEADER MESSAGE.
> With message being in a different format.  It would be great is we
> had a way to generically handle syslog headers, such that ANY parser data 
> could come over syslog.
> Either you could have a custom parser, or configure CSV or JSON such that 
> they could be the payload, such that you can handle JSON over syslog by 
> configuration only.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (METRON-1666) Mpack updates for LDAP, SSO config and removal of JPA

[Simon Elliston Ball (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Elliston Ball reassigned METRON-1666:
---

Assignee: Simon Elliston Ball

> Mpack updates for LDAP, SSO config and removal of JPA
> -
>
> Key: METRON-1666
> URL: https://issues.apache.org/jira/browse/METRON-1666
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Simon Elliston Ball
>Assignee: Simon Elliston Ball
>Priority: Major
>
> Add config files, templates and params to setup SSO and LDAP authentication.
> Remove the redundant config elements from JPA.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (METRON-1664) Add Knox to Blueprints for full dev

[Simon Elliston Ball (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Elliston Ball reassigned METRON-1664:
---

  Assignee: Simon Elliston Ball
Resolution: (was: Done)

> Add Knox to Blueprints for full dev
> ---
>
> Key: METRON-1664
> URL: https://issues.apache.org/jira/browse/METRON-1664
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Simon Elliston Ball
>Assignee: Simon Elliston Ball
>Priority: Major
>
> Knox needs to be included in full dev for testing, it also needs proper 
> certificates to enable full dev blueprints to extract the public key for sso 
> verification.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (METRON-1665) Move hosting of Alerts and Config UIs from Nodejs to Spring Boot

[Simon Elliston Ball (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Elliston Ball reassigned METRON-1665:
---

Assignee: Simon Elliston Ball

> Move hosting of Alerts and Config UIs from Nodejs to Spring Boot
> 
>
> Key: METRON-1665
> URL: https://issues.apache.org/jira/browse/METRON-1665
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Simon Elliston Ball
>Assignee: Simon Elliston Ball
>Priority: Major
>
> The current UIs are served up by very lightweight nodejs applications, which 
> serve the static bundle files produced by the angular build process, and 
> proxies the rest api.
> The proposal is to use a spring boot application, allowing us to harmonise 
> the security implementation across the UI static servers and the REST layer, 
> and to provide a routing platform for later microservices.
> The UIs currently proxy to the REST API to avoid CORS issues, this will be 
> achieved with Zuul.
> Spring Security will also be extended to use a Knox SSO authenticator. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (METRON-1667) Remove the JPA and MySQL elements

[Simon Elliston Ball (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Elliston Ball reassigned METRON-1667:
---

Assignee: Simon Elliston Ball

> Remove the JPA and MySQL elements
> -
>
> Key: METRON-1667
> URL: https://issues.apache.org/jira/browse/METRON-1667
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Simon Elliston Ball
>Assignee: Simon Elliston Ball
>Priority: Major
>
> Once authentication has been moved out of JPA and MySQL, we will no longer 
> need MySQL, eclipselink or any of the spring-data-jpa pieces. We should 
> remove these dependencies to reduce our attack surface.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (METRON-1668) Remove login services and screens from UIs

[Simon Elliston Ball (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Elliston Ball reassigned METRON-1668:
---

Assignee: Simon Elliston Ball

> Remove login services and screens from UIs
> --
>
> Key: METRON-1668
> URL: https://issues.apache.org/jira/browse/METRON-1668
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Simon Elliston Ball
>Assignee: Simon Elliston Ball
>Priority: Major
>
> Once we have integrated SSO, there is no need for the front-end apps to 
> present a login screen, we should remove these from the load footprint.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1677) UUIDv4 GUID is not Lucene friendly

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547609#comment-16547609
 ] 

Simon Elliston Ball commented on METRON-1677:
-

This is an excellent point on the performance side. Guids still need to be 
generated by metron topologies to ensure consistency with hdfs based stored (eg 
reindexing scenarios and consistency of the hbase based update log for 
mutation). However uuid1 may make more sense. The interesting element will be 
how we encode that. Binary encoding would be optimal, but we will need to 
consider the implications for Json friendly encoding of the binary uuid1 and 
other touchpoints for uuid use. This could be a pretty broad PR touching the 
DAO, REST and UI layers as well as the ingest pipeline.

> UUIDv4 GUID is not Lucene friendly
> --
>
> Key: METRON-1677
> URL: https://issues.apache.org/jira/browse/METRON-1677
> Project: Metron
>  Issue Type: Bug
>Reporter: Ali Nazemian
>Priority: Major
>
> Using UUIDv4 by UUID.randomUUID() in Java is not Lucene friendly and impacts 
> Elasticsearch and Solr indexing/search performance and makes it unpredictable 
> sometimes.
> http://blog.mikemccandless.com/2014/05/choosing-fast-unique-identifier-uuid.html
> Moreover, specifying doc id at the client side will impact indexing 
> throughput due to enabling Elasticsearch deduplication policy and changing 
> insert to upsert. Hence, indexing throughput can be increased by providing an 
> ability to disable ID generation at the client side. Currently, the way ID is 
> generated can be overwritten at the config level by replacing Metron default 
> guid via Stellar, but it is not possible to disable it completely to let 
> Elasticsearch decide what ID can be used for the corresponding document.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1670) Stellar WEEK_OF_YEAR test is locale sensitive

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1670:
---

 Summary: Stellar WEEK_OF_YEAR test is locale sensitive
 Key: METRON-1670
 URL: https://issues.apache.org/jira/browse/METRON-1670
 Project: Metron
  Issue Type: Bug
Affects Versions: 0.5.0
Reporter: Simon Elliston Ball


The Stellar WEEK_OF_YEAR(epoch) function is sensitive to the locale of the 
machine it is running on. The tests in DateFunctionsTest are not, this leads to 
test failures on machine locales that differ in their first day of week 
definition or days in first week definition.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-1666) Mpack updates for LDAP, SSO config and removal of JPA

[Simon Elliston Ball (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Elliston Ball updated METRON-1666:

Description: 
Add config files, templates and params to setup SSO and LDAP authentication.

Remove the redundant config elements from JPA.

  was:Add config files, templates and params to setup SSO and LDAP 
authentication

Summary: Mpack updates for LDAP, SSO config and removal of JPA  (was: 
Add mpack settings to configure spring security)

> Mpack updates for LDAP, SSO config and removal of JPA
> -
>
> Key: METRON-1666
> URL: https://issues.apache.org/jira/browse/METRON-1666
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Simon Elliston Ball
>Priority: Major
>
> Add config files, templates and params to setup SSO and LDAP authentication.
> Remove the redundant config elements from JPA.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-1665) Move hosting of Alerts and Config UIs from Nodejs to Spring Boot

[Simon Elliston Ball (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Elliston Ball updated METRON-1665:

Description: 
The current UIs are served up by very lightweight nodejs applications, which 
serve the static bundle files produced by the angular build process, and 
proxies the rest api.

The proposal is to use a spring boot application, allowing us to harmonise the 
security implementation across the UI static servers and the REST layer, and to 
provide a routing platform for later microservices.

The UIs currently proxy to the REST API to avoid CORS issues, this will be 
achieved with Zuul.

Spring Security will also be extended to use a Knox SSO authenticator. 

  was:
The current UIs are served up by very lightweight nodejs applications, which 
serve the static bundle files produced by the angular build process, and 
proxies the rest api.

The proposal is to use a spring boot application, allowing us to harmonise the 
security implementation across the UI static servers and the REST layer, and to


> Move hosting of Alerts and Config UIs from Nodejs to Spring Boot
> 
>
> Key: METRON-1665
> URL: https://issues.apache.org/jira/browse/METRON-1665
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Simon Elliston Ball
>Priority: Major
>
> The current UIs are served up by very lightweight nodejs applications, which 
> serve the static bundle files produced by the angular build process, and 
> proxies the rest api.
> The proposal is to use a spring boot application, allowing us to harmonise 
> the security implementation across the UI static servers and the REST layer, 
> and to provide a routing platform for later microservices.
> The UIs currently proxy to the REST API to avoid CORS issues, this will be 
> achieved with Zuul.
> Spring Security will also be extended to use a Knox SSO authenticator. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-1669) Create a Knox Service as SSL gateway

[Simon Elliston Ball (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Elliston Ball updated METRON-1669:

Description: Knox also provides an SSL gateway which allows us to proxy 
both the UIs and the REST services through an SSL reverse proxy sharing a 
domain. This allows us to avoid additional proxying from the UIs to REST which 
we currently do to avoid CORS restrictions. Creating a package of Knox service 
definitions would allow all UIs to be proxies for SSL and SSO on a single 
endpoint (Knox)  (was: Once we have integrated SSO, there is no need for the 
front-end apps to present a login screen, we should remove these from the load 
footprint.)
Summary: Create a Knox Service as SSL gateway  (was: Remove login 
services and screens from UIs)

> Create a Knox Service as SSL gateway
> 
>
> Key: METRON-1669
> URL: https://issues.apache.org/jira/browse/METRON-1669
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Simon Elliston Ball
>Priority: Major
>
> Knox also provides an SSL gateway which allows us to proxy both the UIs and 
> the REST services through an SSL reverse proxy sharing a domain. This allows 
> us to avoid additional proxying from the UIs to REST which we currently do to 
> avoid CORS restrictions. Creating a package of Knox service definitions would 
> allow all UIs to be proxies for SSL and SSO on a single endpoint (Knox)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1668) Remove login services and screens from UIs

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1668:
---

 Summary: Remove login services and screens from UIs
 Key: METRON-1668
 URL: https://issues.apache.org/jira/browse/METRON-1668
 Project: Metron
  Issue Type: Sub-task
Reporter: Simon Elliston Ball


Once we have integrated SSO, there is no need for the front-end apps to present 
a login screen, we should remove these from the load footprint.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1669) Remove login services and screens from UIs

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1669:
---

 Summary: Remove login services and screens from UIs
 Key: METRON-1669
 URL: https://issues.apache.org/jira/browse/METRON-1669
 Project: Metron
  Issue Type: Sub-task
Reporter: Simon Elliston Ball


Once we have integrated SSO, there is no need for the front-end apps to present 
a login screen, we should remove these from the load footprint.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1667) Remove the JPA and MySQL elements

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1667:
---

 Summary: Remove the JPA and MySQL elements
 Key: METRON-1667
 URL: https://issues.apache.org/jira/browse/METRON-1667
 Project: Metron
  Issue Type: Sub-task
Reporter: Simon Elliston Ball


Once authentication has been moved out of JPA and MySQL, we will no longer need 
MySQL, eclipselink or any of the spring-data-jpa pieces. We should remove these 
dependencies to reduce our attack surface.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1666) Add mpack settings to configure spring security

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1666:
---

 Summary: Add mpack settings to configure spring security
 Key: METRON-1666
 URL: https://issues.apache.org/jira/browse/METRON-1666
 Project: Metron
  Issue Type: Sub-task
Reporter: Simon Elliston Ball


Add config files, templates and params to setup SSO and LDAP authentication



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-1665) Move hosting of Alerts and Config UIs from Nodejs to Spring Boot

[Simon Elliston Ball (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Elliston Ball updated METRON-1665:

Summary: Move hosting of Alerts and Config UIs from Nodejs to Spring Boot  
(was: Moving hosting of Alerts and Config UIs from Nodejs to Spring Boot)

> Move hosting of Alerts and Config UIs from Nodejs to Spring Boot
> 
>
> Key: METRON-1665
> URL: https://issues.apache.org/jira/browse/METRON-1665
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Simon Elliston Ball
>Priority: Major
>
> The current UIs are served up by very lightweight nodejs applications, which 
> serve the static bundle files produced by the angular build process, and 
> proxies the rest api.
> The proposal is to use a spring boot application, allowing us to harmonise 
> the security implementation across the UI static servers and the REST layer, 
> and to



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-1665) Moving hosting of Alerts and Config UIs from Nodejs to Spring Boot

[Simon Elliston Ball (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Elliston Ball updated METRON-1665:

Issue Type: Sub-task  (was: Bug)
Parent: METRON-1663

> Moving hosting of Alerts and Config UIs from Nodejs to Spring Boot
> --
>
> Key: METRON-1665
> URL: https://issues.apache.org/jira/browse/METRON-1665
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Simon Elliston Ball
>Priority: Major
>
> The current UIs are served up by very lightweight nodejs applications, which 
> serve the static bundle files produced by the angular build process, and 
> proxies the rest api.
> The proposal is to use a spring boot application, allowing us to harmonise 
> the security implementation across the UI static servers and the REST layer, 
> and to



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1665) Moving hosting of Alerts and Config UIs from Nodejs to Spring Boot

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1665:
---

 Summary: Moving hosting of Alerts and Config UIs from Nodejs to 
Spring Boot
 Key: METRON-1665
 URL: https://issues.apache.org/jira/browse/METRON-1665
 Project: Metron
  Issue Type: Bug
Reporter: Simon Elliston Ball


The current UIs are served up by very lightweight nodejs applications, which 
serve the static bundle files produced by the angular build process, and 
proxies the rest api.

The proposal is to use a spring boot application, allowing us to harmonise the 
security implementation across the UI static servers and the REST layer, and to



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1664) Add Knox to Blueprints for full dev

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1664:
---

 Summary: Add Knox to Blueprints for full dev
 Key: METRON-1664
 URL: https://issues.apache.org/jira/browse/METRON-1664
 Project: Metron
  Issue Type: Sub-task
Reporter: Simon Elliston Ball


Knox needs to be included in full dev for testing, it also needs proper 
certificates to enable full dev blueprints to extract the public key for sso 
verification.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1663) Single Sign On for Metron UIs and REST

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1663:
---

 Summary: Single Sign On for Metron UIs and REST
 Key: METRON-1663
 URL: https://issues.apache.org/jira/browse/METRON-1663
 Project: Metron
  Issue Type: Improvement
Reporter: Simon Elliston Ball
Assignee: Simon Elliston Ball


At present we store user credentials un-encrypted in a mysql database behind 
the rest service.

We should move towards integration with existing user stores (LDAP) and 
applying secure SSO to each of the Metron user facing components. 

Apache Knox provides SSO services aggregating and integrating other IDPs, and 
SSO authentication for a number of the other components in our stack. We should 
integrate our authentication with Knox.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1662) PCAP UI: Downloading PDML

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540153#comment-16540153
 ] 

Simon Elliston Ball commented on METRON-1662:
-

Is PDML a realistic format for download to an end user? I would have thought 
that PCAP format would be better for an end user download.

> PCAP UI: Downloading PDML
> -
>
> Key: METRON-1662
> URL: https://issues.apache.org/jira/browse/METRON-1662
> Project: Metron
>  Issue Type: New Feature
>Reporter: Tibor Meller
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1639) Grok Parser does not handle missing year well in syslog rfc3164 timestamps

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1639:
---

 Summary: Grok Parser does not handle missing year well in syslog 
rfc3164 timestamps
 Key: METRON-1639
 URL: https://issues.apache.org/jira/browse/METRON-1639
 Project: Metron
  Issue Type: Improvement
Affects Versions: 0.5.0
Reporter: Simon Elliston Ball
Assignee: Simon Elliston Ball


The grok parser does not handle timestamp fields in rfc3164 format well, since 
the format omits a year from the date, the year defaults to 1970. We should 
either switch this to default year to current, or create a "dateFormat" config 
option "syslog" which runs the SyslogUtils parser used in other parsers on the 
captured fields for the field specified in "timestampField" config.

This capability should also reflect the timezone for the sensor, which is not 
currently applied to Grok parsing but is honoured in parsers like 
BasicASAParser. Note that it is not universally applied across all parsers, but 
probably should be.

"Mmm dd hh:mm:ss" is the canonical date format in rfc3164, with options to 
include a timezone and year. We currently handle this and variants found in the 
wild in
SyslogUtils::parseTimestampToEpochMillis, which also accounts for timezone 
based on a Clock parameter. This function assumes that any date more than 4 
days in the future is in the past, which seems acceptable and consistent for 
our purposes and covers the possibility of year end discrepancies. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1615) Fill in some endpoint details from Ambari info

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1615:
---

 Summary: Fill in some endpoint details from Ambari info
 Key: METRON-1615
 URL: https://issues.apache.org/jira/browse/METRON-1615
 Project: Metron
  Issue Type: Improvement
Affects Versions: 0.5.0
Reporter: Simon Elliston Ball


When installing a metron cluster, users are required to fill in values that 
Ambari should already know: 
 * location of storm ui
 * location of elastic search
 * location of zeppelin notebook server

While it is useful to be able to override the ES location if you're using a 
non-metron managed ES, the others are absolutely a function of the Metron 
cluster, whose locations are known to Ambari, so users should not be expected 
to fill them in.

If the ES hosts settings is left blank, it should be calculated from info in 
the ES mpack.

The kibana service in the elastic mpack should also automatically set the url 
for ES masters.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1583) issue regarding cisco asa logs

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494581#comment-16494581
 ] 

Simon Elliston Ball commented on METRON-1583:
-

Would it be possible to share the full log error message from the storm logs, 
rather than the screenshot?

> issue regarding cisco asa logs
> --
>
> Key: METRON-1583
> URL: https://issues.apache.org/jira/browse/METRON-1583
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: manisha tank
>Priority: Major
> Fix For: 0.4.2
>
> Attachments: cisco_asa_logs_error.png
>
>
> I am trying to ingest cisco asa logs but I am facing some issue .
> I have created log pattern below
> CISCO_TAGGED_SYSLOG ^%\{SYSLOGTIMESTAMP} %\{SYSLOGHOST:sysloghost} 
> <%\{POSINT:syslog_pri}>%\{CISCOTIMESTAMP}?: %%\{CISCOTAG:ciscotag}: 
> %\{GREEDYDATA:message}
> CISCOTIMESTAMP %\{MONTH} +%\{MONTHDAY}(?: %\{YEAR})? %\{TIME}
> CISCOTAG [A-Z0-9]+-%\{INT}-(?:[A-Z0-9_]+)
> sample logs
> Oct 25 02:14:52 172.20.4.5 <163>Oct 24 2017 21:29:23: %ASA-3-304006: URL 
> Server 172.19.83.105 not responding
> Oct 25 02:14:51 198.6.1.2 <164>Oct 24 2017 21:28:47: %ASA-4-410001: Dropped 
> UDP DNS reply from outside:198.6.1.2/53 to inside:172.20.220.87/63887; packet 
> length 932 bytes exceeds configured limit of 512 bytes
> Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:34: %ASA-4-733100: [ 
> Scanning] drop rate-1 exceeded. Current burst rate is 46 per second, max 
> configured rate is 10; Current average rate is 103 per second, max configured 
> rate is 5; Cumulative total count is 62196
> Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:21: %ASA-4-733100: [    
> SYSLOG  514] drop rate-1 exceeded. Current burst rate is 31 per second, max 
> configured rate is 40; Current average rate is 119 per second, max configured 
> rate is 20; Cumulative total count is 71776
>  
> Oct 25 02:14:52 192.168.19.7 <164>Oct 24 2017 21:29:29: %ASA-4-419002: 
> Duplicate TCP SYN from inside:192.168.19.7/64266 to outside:192.168.10.10/257 
> with different initial sequence number
>  
> PFA error facing while inegesting cisco asa logs
>  
> !cisco_asa_logs_error.png!  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1538) Don't use GUIDS for Elastic document id, but autogenerated ID's for performance

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449941#comment-16449941
 ] 

Simon Elliston Ball commented on METRON-1538:
-

Absolutely. In general I agree with your thoughts, just not sure we want to 
make the user make the choice and add to our already complex grid of 
co-dependent tuning variables.

> Don't use GUIDS for Elastic document id, but autogenerated ID's for 
> performance
> ---
>
> Key: METRON-1538
> URL: https://issues.apache.org/jira/browse/METRON-1538
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.4.3
>Reporter: Ward Bekker
>Priority: Major
>  Labels: performance
>
> Metron currently uses GUIDS for ES document Ids, this goes against the best 
> practice:
> "When indexing a document that has an explicit id, Elasticsearch needs to 
> check whether a document with the same id already exists within the same 
> shard, which is a costly operation and gets even more costly as the index 
> grows. By using auto-generated ids, Elasticsearch can skip this check, which 
> makes indexing faster."
> [https://www.elastic.co/guide/en/elasticsearch/reference/master/tune-for-indexing-]speed.html#_use_auto_generated_ids



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1538) Don't use GUIDS for Elastic document id, but autogenerated ID's for performance

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449914#comment-16449914
 ] 

Simon Elliston Ball commented on METRON-1538:
-

In ES defence, it makes sense to do the uniqueness check, because it can't know 
the guid you give it is a Guid, or that it's not already seen that one. ES 
would have to check even if we did say it was a guid because we can't guarantee 
we're not duplicate indexing or updating an existing doc, so fair play. 

I'm not sure this is something that would be an env by env thing to be honest. 
I know my argument was about a performance tradeoff, but it does feel like 
something we should be opinionated about. I would lean towards having our guid 
just be a field, and letting elastic make up its own ids, which we would pay no 
attention to. 

> Don't use GUIDS for Elastic document id, but autogenerated ID's for 
> performance
> ---
>
> Key: METRON-1538
> URL: https://issues.apache.org/jira/browse/METRON-1538
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.4.3
>Reporter: Ward Bekker
>Priority: Major
>  Labels: performance
>
> Metron currently uses GUIDS for ES document Ids, this goes against the best 
> practice:
> "When indexing a document that has an explicit id, Elasticsearch needs to 
> check whether a document with the same id already exists within the same 
> shard, which is a costly operation and gets even more costly as the index 
> grows. By using auto-generated ids, Elasticsearch can skip this check, which 
> makes indexing faster."
> [https://www.elastic.co/guide/en/elasticsearch/reference/master/tune-for-indexing-]speed.html#_use_auto_generated_ids



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1538) Don't use GUIDS for Elastic document id, but autogenerated ID's for performance

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449783#comment-16449783
 ] 

Simon Elliston Ball commented on METRON-1538:
-

This is not necessarily a good idea. We use guids for message amendments, and 
meta alerting. The guids therefore need to be generated in the Metron pipeline, 
otherwise guids will not match in ES and HDFS indices, leading to total data 
mis-match. 

What we could do is to have a useless id created in ES in addition to the guid. 
This would add storage overhead, and hurt performance on meta alerts and search 
lookup, but may not be as significant in the ingest impact of 
non-auto-generated keys. Arguably this scenario risks data corruption in ES 
unless we perform the same uniqueness checks anyway, but that may be something 
that can be resolved, or accepted as a small front-end event duplication risk 
in short term indices.

> Don't use GUIDS for Elastic document id, but autogenerated ID's for 
> performance
> ---
>
> Key: METRON-1538
> URL: https://issues.apache.org/jira/browse/METRON-1538
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.4.3
>Reporter: Ward Bekker
>Priority: Major
>  Labels: performance
>
> Metron currently uses GUIDS for ES document Ids, this goes against the best 
> practice:
> "When indexing a document that has an explicit id, Elasticsearch needs to 
> check whether a document with the same id already exists within the same 
> shard, which is a costly operation and gets even more costly as the index 
> grows. By using auto-generated ids, Elasticsearch can skip this check, which 
> makes indexing faster."
> [https://www.elastic.co/guide/en/elasticsearch/reference/master/tune-for-indexing-]speed.html#_use_auto_generated_ids



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1537) Vagrant Up becomes unsuccessfull after metron Build is successfull

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449559#comment-16449559
 ] 

Simon Elliston Ball commented on METRON-1537:
-

Right, nested VMs are highly unlikely to work. Try without the multi-layer 
nesting of virtualisation and your system will be much happier.

> Vagrant Up becomes unsuccessfull after metron Build is successfull
> --
>
> Key: METRON-1537
> URL: https://issues.apache.org/jira/browse/METRON-1537
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.3.1
> Environment: Ubuntu Vm
>Reporter: YAKESHRAJ M
>Priority: Major
> Attachments: metron (2).jpg, metron (3).jpg
>
>
> On giving vagrant up command to start the metron, at first an error occurs 
> stating" Time out occurs while waiting for the machine to boot" and on trying 
> vagrant up command again it gives out an error stating" There was an error 
> while executing 'VBoxManage', a CLI used by vagrant for controlling 
> VirtualBox" so i'm not able to proceed with metron thereafter , need  a 
> solution to fix it.. tried upgrading virtualbox, running virtualbox-dkms.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1537) Vagrant Up becomes unsuccessfull after metron Build is successfull

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447995#comment-16447995
 ] 

Simon Elliston Ball commented on METRON-1537:
-

It looks like your machine, most likely your processor, but possibly just your 
kernel build, is not compatible with VirtualBox. You'll need to be able to run 
virtual box and vagrant before you can use this Metron install method. I would 
recommend using one of the cloud options as a preference if your hardware can't 
do virtualization. 

> Vagrant Up becomes unsuccessfull after metron Build is successfull
> --
>
> Key: METRON-1537
> URL: https://issues.apache.org/jira/browse/METRON-1537
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.3.1
> Environment: Ubuntu Vm
>Reporter: YAKESHRAJ M
>Priority: Major
> Attachments: metron (2).jpg, metron (3).jpg
>
>
> On giving vagrant up command to start the metron, at first an error occurs 
> stating" Time out occurs while waiting for the machine to boot" and on trying 
> vagrant up command again it gives out an error stating" There was an error 
> while executing 'VBoxManage', a CLI used by vagrant for controlling 
> VirtualBox" so i'm not able to proceed with metron thereafter , need  a 
> solution to fix it.. tried upgrading virtualbox, running virtualbox-dkms.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1534) Typosquat Detection via Bloom filters overlaps

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1534:
---

 Summary: Typosquat Detection via Bloom filters overlaps
 Key: METRON-1534
 URL: https://issues.apache.org/jira/browse/METRON-1534
 Project: Metron
  Issue Type: Bug
Affects Versions: 0.4.3
Reporter: Simon Elliston Ball


The typosquat detection use case overpopulates the bloom filter. 

For example, using the alexa 10k set, cnn.com, or bbc.co.uk  are both detected 
as typosquats. While legitimate in themselves, they appear in dns twists of 
other legitimate domains. (e.g. bbc is a typosquat for rbc).

This problem is further accentuated by a longer set of legitimate domains such 
as the alexa 1m.

The bloom filter additions need to be be prevented for values which are 
included in the raw 'good' source. This is hard to do in a space and compute 
performant way with the current implementation, given the need to effectively 
join the full input set (smallish) with the generated set (very large).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1517) GeoLite discontinuation notice

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437184#comment-16437184
 ] 

Simon Elliston Ball commented on METRON-1517:
-

Seems maxmind have changed their mind on that...

*"Note:* After careful consideration, taking into account customer feedback, we 
have decided against removing latitude and longitude coordinates from the 
GeoLite2 databases. We are in the process of reviewing coordinates used in all 
of our GeoLite2 and GeoIP databases to ensure there is no risk of misuse."

- https://dev.maxmind.com/geoip/geoip2/geolite2/

> GeoLite discontinuation notice
> --
>
> Key: METRON-1517
> URL: https://issues.apache.org/jira/browse/METRON-1517
> Project: Metron
>  Issue Type: Bug
>Reporter: Laurens Vets
>Priority: Major
>
> I believe the geolite db used by Metron will be discontinued:
> https://support.maxmind.com/geolite-legacy-discontinuation-notice/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1523) Add issues list links to metron.apache.org community page

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435525#comment-16435525
 ] 

Simon Elliston Ball commented on METRON-1523:
-

Looks like this has been done in METRON-1510, but not published.

> Add issues list links to metron.apache.org community page
> -
>
> Key: METRON-1523
> URL: https://issues.apache.org/jira/browse/METRON-1523
> Project: Metron
>  Issue Type: Task
>Reporter: Simon Elliston Ball
>Assignee: Simon Elliston Ball
>Priority: Trivial
>
> Now we've split out issues the links should be added to our website.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1523) Add issues list links to metron.apache.org community page

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1523:
---

 Summary: Add issues list links to metron.apache.org community page
 Key: METRON-1523
 URL: https://issues.apache.org/jira/browse/METRON-1523
 Project: Metron
  Issue Type: Task
Reporter: Simon Elliston Ball
Assignee: Simon Elliston Ball


Now we've split out issues the links should be added to our website.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1507) Add support for Vagrant Cachier plugin if present

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1507:
---

 Summary: Add support for Vagrant Cachier plugin if present
 Key: METRON-1507
 URL: https://issues.apache.org/jira/browse/METRON-1507
 Project: Metron
  Issue Type: Improvement
Reporter: Simon Elliston Ball


Add configuration in development vagrant files to support vagrant cachier 
([http://fgrehm.viewdocs.io/vagrant-cachier/)] to reduce development machine 
install times. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1293) NiFi Indexer(?) capability

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289538#comment-16289538
 ] 

Simon Elliston Ball commented on METRON-1293:
-

So you would imagine something that picks up data from the indexing topic in 
Kafka, and sends it to NiFi site-to-site. Personally I don't see any benefit in 
that vs just having NiFi ConsumeKafka pointed at the indexing topic.

> NiFi Indexer(?) capability
> --
>
> Key: METRON-1293
> URL: https://issues.apache.org/jira/browse/METRON-1293
> Project: Metron
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Priority: Trivial
>
> Either through an extension ( if and when they come to indexing ) or through 
> a normal module support for indexing to nifi input ports would allow for 
> flexible capabilities for systems where nifi is used.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1293) NiFi Indexer(?) capability

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289486#comment-16289486
 ] 

Simon Elliston Ball commented on METRON-1293:
-

I see the thinking there, and there is some benefit in directly supporting 
site-to-site protocol in terms of load balancing direct from Metron to NiFi 
without the intermediate step of load balancing across kafka consumers, but I'm 
not sure I would recommend writing direct into NiFi from a resilience point of 
view, given that the NiFi content repository is not distributed. 

Still, once we've split out indexing to make it more extensible, this would be 
a good indexing option (that I would never recommend to anyone in production)

> NiFi Indexer(?) capability
> --
>
> Key: METRON-1293
> URL: https://issues.apache.org/jira/browse/METRON-1293
> Project: Metron
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Priority: Trivial
>
> Either through an extension ( if and when they come to indexing ) or through 
> a normal module support for indexing to nifi input ports would allow for 
> flexible capabilities for systems where nifi is used.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (METRON-1341) Projection FieldTransformation

[Simon Elliston Ball (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Elliston Ball updated METRON-1341:

Summary: Projection FieldTransformation  (was: Projection 
FieldTansformation)

> Projection FieldTransformation
> --
>
> Key: METRON-1341
> URL: https://issues.apache.org/jira/browse/METRON-1341
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.4.2
>Reporter: Simon Elliston Ball
>Assignee: Simon Elliston Ball
>  Labels: newbie
>
> It would be useful to have a projection transformation for Parsers which 
> could use configured to limit the fields output from the parser.
> The configuration would look like this:
> {code:java}
> {
>  "fieldTransformations": [
>{
>  "transformation": "STELLAR",
>  "config": [
>"ipSrc = TRIM(raw_ip_src)"
>"ip_src_addr := ipSrc"
>  ]
>},
> {
>  "transformation": "SELECT",
>  "output" : [ "ip_src_addr", "ip_dst_addr", "message"]
>}
>  ]
> }
> {code}
> This would lead to only the fields in the output definition of the SELECT 
> transformation being put into the outbound message.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (METRON-1341) Projection FieldTansformation

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1341:
---

 Summary: Projection FieldTansformation
 Key: METRON-1341
 URL: https://issues.apache.org/jira/browse/METRON-1341
 Project: Metron
  Issue Type: Improvement
Affects Versions: 0.4.2
Reporter: Simon Elliston Ball
Assignee: Simon Elliston Ball


It would be useful to have a projection transformation for Parsers which could 
use configured to limit the fields output from the parser.

The configuration would look like this:
{code:java}

{
 "fieldTransformations": [
   {
 "transformation": "STELLAR",
 "config": [
   "ipSrc = TRIM(raw_ip_src)"
   "ip_src_addr := ipSrc"
 ]
   },
{
 "transformation": "SELECT",
 "output" : [ "ip_src_addr", "ip_dst_addr", "message"]
   }
 ]
}
{code}

This would lead to only the fields in the output definition of the SELECT 
transformation being put into the outbound message.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1036) Stellar log function

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251460#comment-16251460
 ] 

Simon Elliston Ball commented on METRON-1036:
-

Yes. Looks like they got rolled up into that ticket. I’ll close this thanks for 
the cleanup Jon.

> Stellar log function
> 
>
> Key: METRON-1036
> URL: https://issues.apache.org/jira/browse/METRON-1036
> Project: Metron
>  Issue Type: Improvement
>Reporter: Simon Elliston Ball
>Priority: Critical
>
> Stellar does not currently have a natural log, or other base log function.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1238) Enrichment topology start at EARLIEST again after de-activating topology

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197007#comment-16197007
 ] 

Simon Elliston Ball commented on METRON-1238:
-

While this is potentially an issue in Kafka / Storm, I suspect the likely cause 
is that you need to, for some reason reset kafka offsets when processing has 
lagged to far. 

Instead of reverting to EARLIEST, you can use my very hacky tool 
https://github.com/simonellistonball/kafka-utils to manipulate offsets for the 
relevant consumer group. Note this will be superseded by a much cleaner version 
in Kafka 0.11. 

See 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling
 
and https://issues.apache.org/jira/browse/KAFKA-4743 

for details on the proper way.

> Enrichment topology start at EARLIEST again after de-activating topology
> 
>
> Key: METRON-1238
> URL: https://issues.apache.org/jira/browse/METRON-1238
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
> Environment: 0.4.1.3.0.0
>Reporter: Jasper Knulst
>
> When the enrichment topology is started with the firstPollStrategy EARLIEST 
> and then de-activate the topology from the Storm UI, the kafkaspout begins 
> consuming again from EARLIEST when the topology is activated again.
> To reproduce:
> -Start enrichment with firstPollOffsetStrategy set to EARLIEST
> -De-activate topology (Storm UI) after a while (check topology offset)
> -Wait
> -Activate topology again 
> -Check topology offset
> I think the Kafkaspouts are reinstantiated when you do this and the spout 
> config is hanging somewhere in its original state



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1217) ALERT-UI should call out limitation of searchable indices in the readme

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185660#comment-16185660
 ] 

Simon Elliston Ball commented on METRON-1217:
-

I don't thing this is a real limitation once we've dealt with METRON-1216 
(which is a pretty simple fix), so we can probably do without this ticket.

> ALERT-UI should call out limitation of searchable indices in the readme
> ---
>
> Key: METRON-1217
> URL: https://issues.apache.org/jira/browse/METRON-1217
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>
> If the ALERT-UI can only search a subset of available sensors, it should be 
> documented in the readme.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-685) Scores in Threat Triage should be a Stellar Statement

[Simon Elliston Ball (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177663#comment-16177663
 ] 

Simon Elliston Ball commented on METRON-685:


I would agree that it should be limited to numbers, and possibly in a given 
range eg. [-100, 100]. Not sure about integers though. There's a lot to be said 
for accepting real numbers rather than just integers in this use case (which is 
roughly speaking multiply the literal score by some scaling factor. Forcing 
integers would lose precision around the aggregation in a weird way. 

> Scores in Threat Triage should be a Stellar Statement
> -
>
> Key: METRON-685
> URL: https://issues.apache.org/jira/browse/METRON-685
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.3.0
>Reporter: Simon Elliston Ball
>
> When writing threat triage rules I would like the score for a rule to be 
> determined by a stellar statement, rather than a fixed number triggered by a 
> boolean stellar statement.
> For example: 
> {code}
> "triageConfig" : {
>  "riskLevelRules" : [
>{
>  "name" : "Abnormal Value",
>  "comment" : "FORMAT('For %s; the value %s exceeds threshold of %d',
> hostname, value, value_threshold)"
>  "rule" : "SOME_STELLAR_FUNCTION(value) > value_threshold",
>  "score" : "SOME_STELLAR_FUNCTION(value)"
>}
>  ],
>  "aggregator" : "MAX"
> }
> {code}
> Note that in this scenario it would also be beneficial to cache part of the 
> statement to avoid likely duplication between rule and score evaluation. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (METRON-1140) Profiler MPack does not handle creation of kafka for hbase acls in a kerberised cluster

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1140:
---

 Summary: Profiler MPack does not handle creation of kafka for 
hbase acls in a kerberised cluster
 Key: METRON-1140
 URL: https://issues.apache.org/jira/browse/METRON-1140
 Project: Metron
  Issue Type: Bug
Affects Versions: 0.4.1
Reporter: Simon Elliston Ball


The new Mpack launches the profiler topology on a kerberised cluster, but the 
topologies fail to work because they are missing ACL for kafka and hbase. e.g. 
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to 
access group: profiler

The MPack should support kerberos environments and create the relevant acls and 
permissions for the profiler topology.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (METRON-1139) MPack for the profiler breaks stack advisor

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1139:
---

 Summary: MPack for the profiler breaks stack advisor
 Key: METRON-1139
 URL: https://issues.apache.org/jira/browse/METRON-1139
 Project: Metron
  Issue Type: Bug
Affects Versions: 0.4.1
Reporter: Simon Elliston Ball


The mpack additions for the profiler work fine, but prevent subsequent 
components installing with a stack advisor error (missing metronProfileHost). 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (METRON-1037) Stellar power function

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1037:
---

 Summary: Stellar power function
 Key: METRON-1037
 URL: https://issues.apache.org/jira/browse/METRON-1037
 Project: Metron
  Issue Type: Improvement
Affects Versions: 0.4.0
Reporter: Simon Elliston Ball


Stellar does not currently have a power function. We should have a function of 
the form POWER(number,power) named for consistency with SQL and Excel. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (METRON-1036) Stellar log function

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1036:
---

 Summary: Stellar log function
 Key: METRON-1036
 URL: https://issues.apache.org/jira/browse/METRON-1036
 Project: Metron
  Issue Type: Improvement
Reporter: Simon Elliston Ball
Priority: Critical


Stellar does not currently have a natural log, or other base log function.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (METRON-1035) Documentation for rules triage aggregation does not include SUM

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1035:
---

 Summary: Documentation for rules triage aggregation does not 
include SUM
 Key: METRON-1035
 URL: https://issues.apache.org/jira/browse/METRON-1035
 Project: Metron
  Issue Type: Bug
Reporter: Simon Elliston Ball
Priority: Trivial


The docs in the enrichment project does not indicate that SUM is an allowed 
aggregation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (METRON-1033) Profiler example uses incorrect units for expires

[Simon Elliston Ball (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Elliston Ball updated METRON-1033:

Description: The profiler documentation states that expires is in ms, but 
examples show (correctly) days.   (was: The profiler example in the README use 
days as a unit, but the value is expected in ms. )

> Profiler example uses incorrect units for expires
> -
>
> Key: METRON-1033
> URL: https://issues.apache.org/jira/browse/METRON-1033
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Simon Elliston Ball
>Priority: Trivial
>  Labels: newbie
>
> The profiler documentation states that expires is in ms, but examples show 
> (correctly) days. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (METRON-1033) Profiler example uses incorrect units for expires

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1033:
---

 Summary: Profiler example uses incorrect units for expires
 Key: METRON-1033
 URL: https://issues.apache.org/jira/browse/METRON-1033
 Project: Metron
  Issue Type: Bug
Affects Versions: 0.4.0
Reporter: Simon Elliston Ball
Priority: Trivial


The profiler example in the README use days as a unit, but the value is 
expected in ms. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (METRON-906) Rest service storm configuration does not allow for proper URLs

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-906:
--

 Summary: Rest service storm configuration does not allow for 
proper URLs
 Key: METRON-906
 URL: https://issues.apache.org/jira/browse/METRON-906
 Project: Metron
  Issue Type: Bug
Affects Versions: 0.3.1
Reporter: Simon Elliston Ball


The storm.ui.url  field does not accept a URL, but instead accepts a hostname, 
port and path, without the schema. The code then prepends http://  This is 
somewhat confusing, and prevents us later moving to support ssl access to the 
api. The field should retain its name as url, but accept an actual url.

Note this is a breaking config change, and will required documentation.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)