Re: Metron's Future / Alternatives

[Simon Elliston Ball]

We’re also replacing much of the functionality of Metron with a series of
Apache Flink based streaming components in a number of installations. It
makes for a composable approach, and borrows from elements of the Metron
architecture, while using more efficient formats like Apache Avro instead
of JSON to reduce Kafka consumption and increase performance. Using Flink
also allows for more efficient aggregation and sql based rules.

It’s much more of a custom solution than the generic project Metron took
on, but seems to be working well for many of the same log related use cases.

Simon

On Wed, 3 Feb 2021 at 00:08, Alex Scammon  wrote:

> Hey there Jack,
>
> We were also disappointed that Metron was shuttered.  But we've seen it as
> an opportunity to continue an internal project which builds on top of
> Metron.  Our goal is to make our project open source as a potential
> successor to Metron.
>
> We're maybe a month or two away from making it public, but we'd love some
> eyes on it before we take that step if you're interested in taking a look.
>
> Since it builds on Metron, a lot of the core architecture remains
> familiar.  Java, Kafka, Storm, etc  Hopefully, that presents a familiar
> ecosystem for folks who are currently using Metron.  For improvements, we
> focused on:
>
>- Ensuring that simple configuration mistakes don't bring down the
>whole pipeline
>- A git-based approval workflow for rules updates (approvals and an
>audit trail are important for us)
>- An improved, modern-looking UI in Angular
>- Easier installation steps
>
> Let me know if you're interested in discussing more -- I'd be interested
> to hear whether there are any particulars about the models you're running
> that we should take into consideration.
>
> Cheers,
>
> Alex Scammon
> Head of Open Source Development
> G-Research
> gresearch.co.uk
>
>
> On Tue, Feb 2, 2021 at 1:18 AM Jack Roberts  wrote:
>
>> Hi Metron community,
>>
>>
>>
>> I recently started to explore Metron as part of a research project I’m
>> involved in, but I’ve just seen the unfortunate news that the project is
>> being “moved to the Attic”. I’d be very grateful if anyone could help to
>> clarify the following:
>>
>>
>>
>>- Is there likely to be any continued development of Metron outside
>>of Apache?
>>- Are there any alternatives to Metron that people in this community
>>would recommend? In particular, we’re looking for something open source
>>that we can deploy ourselves, and with the functionality to
>>straightforwardly integrate our own machine learning models for anomaly
>>detection/similar.
>>
>>
>>
>> Many thanks and best wishes,
>>
>> Jack
>> The Alan Turing Institute is a limited liability company, registered in
>> England with registered number 09512457 with registered offices at British
>> Library, 96 Euston Road, London, England, NW1 2DB
>> <https://www.google.com/maps/search/96+Euston+Road,+London,+England,+NW1+2DB?entry=gmail=g>.
>> We are also a charity registered in England with charity number 1162533.
>> DISCLAIMER: Although we have taken reasonable precautions to ensure the
>> completeness and accuracy of this e-mail, transmission cannot be guaranteed
>> to be secure or error-free as information could be intercepted, corrupted,
>> lost, destroyed, arrive late or be incomplete. If you receive a suspicious
>> or unexpected email from us, or purporting to have been sent on our behalf,
>> particularly containing different bank details, please do not reply to the
>> email, click on any links, open any attachments, or comply with any
>> instructions contained within it. Our Transparency Notice found here -
>> https://www.turing.ac.uk/transparency-notice sets out how and why we
>> collect, store, use and share your personal data and it explains your
>> rights and how to raise concerns with us.
>>
> --
--
simon elliston ball
@sireb

Re: How to customise the streams visible in metron ui

[Simon Elliston Ball]

Yes, the easiest way is to use the source type filter, or search on
specific source types.

Simon

On Sun, 17 May 2020 at 12:13, Hema malini  wrote:

> Hi all,
>
> In the Metron UI, is it possible to show only particular streams or
> devices for users.
>
> Thanks and Regards,
> Hema
>
-- 
--
simon elliston ball
@sireb

Re: 3rd party stellar functions

[Simon Elliston Ball]

Did you sync the global settings to zk, and ensure you started stellar with
the zk set correctly? Any errors logged?

On Fri, 21 Feb 2020 at 10:55, Gonçalo Pedras  wrote:

> Hi,
>
> I was following this tutorial on custom Stellar functions
> https://metron.apache.org/current-book/metron-stellar/stellar-common/3rdPartyStellar.html
> , and it just doesn’t work. I changed the Global configs, uploaded the
> “jar” file to HDFS, and then ran the stellar environment to test it. No
> errors at all, but the function isn’t even listed when I do “%functions”.
>
>
>
> Thanks
>


-- 
--
simon elliston ball
@sireb

Re: Having more than one use case on a Metron instance

[Simon Elliston Ball]

Not at present, no, but you can just open multiple instances. Saved
searches are also per user, so that might provide a workaround.

Simon

On Wed, 19 Feb 2020 at 16:23, Euan Hope  wrote:

> Thanks so much for the quick feedback. I will put this forward to the
> client.
>
> To gain a sense of what is possible, is there possibly a way to configure
> more tabs in the Alerts UI (for example, there is the PCAP tab available in
> our UI)?
>
> Or possibly as another alternative, is it possible to configure different
> Alerts UI for different users. Say for example that user A can only access
> Alerts UI A, user B can only access Alerts UI B?
>
> Thanks again for your input, it is very much appreciated.
>
> On Wed, Feb 19, 2020 at 5:59 PM Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
>> I would suggest using saved searches, which also remember the selected
>> columns.
>>
>> Simon
>>
>> On Wed, 19 Feb 2020 at 15:51, Euan Hope  wrote:
>>
>>> Hi again Metron community.
>>>
>>> Sorry to post another question in such quick succession.
>>>
>>> Our client has asked us to implement another use case on the Metron
>>> instance we have set up for them. This new use case uses similar data to
>>> the original use case but the threat triage rules for scoring the records
>>> are very different.
>>>
>>> The request was to have another tab in the Alerts UI so that the
>>> different SOC analysts could use different screens for the different use
>>> cases.
>>>
>>> Is there any way to configure this? And if not, does anyone in the
>>> community have suggestions on how to approach this?
>>>
>>> Thanks in advance for the help.
>>>
>> --
>> --
>> simon elliston ball
>> @sireb
>>
> --
--
simon elliston ball
@sireb

Re: Having more than one use case on a Metron instance

[Simon Elliston Ball]

I would suggest using saved searches, which also remember the selected
columns.

Simon

On Wed, 19 Feb 2020 at 15:51, Euan Hope  wrote:

> Hi again Metron community.
>
> Sorry to post another question in such quick succession.
>
> Our client has asked us to implement another use case on the Metron
> instance we have set up for them. This new use case uses similar data to
> the original use case but the threat triage rules for scoring the records
> are very different.
>
> The request was to have another tab in the Alerts UI so that the different
> SOC analysts could use different screens for the different use cases.
>
> Is there any way to configure this? And if not, does anyone in the
> community have suggestions on how to approach this?
>
> Thanks in advance for the help.
>
-- 
--
simon elliston ball
@sireb

Re: Using something other than colons in field names?

[Simon Elliston Ball]

The colons were originally required to avoid poor performance in sub
documents in elastic search 2.x. It’s really a legacy thing now, and the
NOOP should be considered the best path going forward.

Simon

On Mon, 3 Feb 2020 at 12:41, Yerex, Tom  wrote:

> Thank you Vladimir.
>
> Before I go diving into making a lot of changes from the default, does
> anyone happen to recall why the colon was selected as the default? I'm
> curious if it works better for analysis in HDFS or Zeppelin?
>
> Cheers,
>
> Tom.
>
> On 2020-02-02, 8:53 PM, "Vladimir Mikhailov" <
> v.mikhai...@content-media.ru> wrote:
>
> Hi
>
> There is a parameter "fieldNameConverter" in the parser indexing
> configuration:
>
> fieldNameConverter
>
> "Defines how field names are transformed before being written to the
> index. Only applicable to elasticsearch.
>
> Defaults to DEDOT. Acceptable values are DEDOT that replaces all '.'
> with ':' or NOOP that does not change the field names."
>
>
> https://github.com/apache/metron/blob/master/metron-platform/metron-indexing/metron-indexing-common/README.md#sensor-indexing-configuration
>
> Usage example:
>
> "elasticsearch": {
> "batchSize": 100,
> "enabled": true,
> "index": "myindex",
> "fieldNameConverter": "NOOP"
> },
>
> On 2020/02/01 00:00:04, "Yerex, Tom"  wrote:
> > Good afternoon,
> >
> >
> >
> > Our Metron installation uses colons in the field names. For example,
> geo ip enriched data appears as “enrichments:geo:ip_dst_addr:country”.
> Under Kibana (and from what I read Banana), the colon cannot be properly
> escaped for use with Timelion.
> >
> >
> >
> > My question: has anyone figured out a way to escape colons in their
> query or another work around in general? Is there a setting somewhere that
> can be used to change the default from a colon to a period or another
> character?
> >
> >
> >
> > Thank you,
> >
> >
> >
> > Tom.
> >
> >
> >
> >
>
> --
--
simon elliston ball
@sireb

Re: Mysterious Metron UI screenshot

[Simon Elliston Ball]

Those were old mockups of potential alerts ui designs showing roadmap
thoughts (as explained in the webinar they are from). They were not built.

Simon

On Wed, 8 Jan 2020 at 21:07, Dima Kovalyov  wrote:

> Hello, Metron community,
>
> Here are two screenshots from Slideshare:
>
> https://www.slideshare.net/hortonworks/combating-phishing-attacks-how-big-data-helps-detect-impersonators
> Screen1
> <https://image.slidesharecdn.com/webinarcybersecuritybigdataphishinghortonworksapr20171-170427220538/95/combating-phishing-attacks-how-big-data-helps-detect-impersonators-27-1024.jpg>
>  and
> Screen2
> <https://image.slidesharecdn.com/webinarcybersecuritybigdataphishinghortonworksapr20171-170427220538/95/combating-phishing-attacks-how-big-data-helps-detect-impersonators-26-1024.jpg>
>
> It looks like Alerts UI but it was early in development at a times
> (screens are dated 2016). Is it Metron Management UI + Kibana iframes?
> Can anyone shed more light on how these screens were created?
>
> Thank you.
>
> p.s. Can you please invite me to the slack channel?
>
> - Dima
>
> --
--
simon elliston ball
@sireb

Re: How can i send batch of data to MaaS

[Simon Elliston Ball]

If you’re looking to send sequences to an LSTM model, you are probably
looking for the profiler, which can assemble sequential features such as
those that would go into an LSTM. You would then use the triage output
method from the profiler to pass a stream of batches to MaaS.

Simon

On Tue, 10 Dec 2019 at 16:16, Hema malini  wrote:

> Thanks Otto for the confirmation.
>
> On Tue, 10 Dec, 2019, 8:46 PM Otto Fowler, 
> wrote:
>
>> As Metron is a streaming system, it doesn’t send batches as part of
>> normal in flow operation. MAAS is called through stellar, which operates on
>> a per message basis.
>>
>> The batching we *do* have is at the termination of the stream, at the
>> indexing where we batch writes out of the pipeline. This won’t help you
>> with stellar however.
>>
>>
>>
>>
>> On December 10, 2019 at 09:39:27, Hema malini (nhemamalin...@gmail.com)
>> wrote:
>>
>> Hi,
>>
>> Is there any way to pass a batch of data to Metron MaaS. We have some
>> models like LSTM, which requires data to be aggregated and passed to model
>> .Can you please suggest whether is it possible.
>>
>> Thanks and Regards,
>> Hema
>>
>> --
--
simon elliston ball
@sireb

Re: ingesting syslog and asa log into metron

[Simon Elliston Ball]

If you find any missing patterns you should also consider contributing them
back to the open source project.

https://metron.apache.org/current-book/CONTRIBUTING.html

Simon

On Mon, 25 Nov 2019 at 15:00, Hema malini  wrote:

> After enabling the parsers, kindly check for the patterns missed out and
> add grok patterns based on the log messages.
>
>
>
> On Mon, 25 Nov, 2019, 7:44 PM Simon Elliston Ball, <
> si...@simonellistonball.com> wrote:
>
>> Use the nifi listen syslog processor to push Asa logs into a Kafka topic,
>> then the metron asa parser to get that into your metron flow.
>>
>> Simon
>>
>> On Mon, 25 Nov 2019 at 14:12, updates on tube 
>> wrote:
>>
>>> hey guys first I really appreciate your urgent replies on my previous
>>> posts >>
>>> and for now, I went to ask how can I ingest Syslog and asa log into
>>> apache metron using nifi?
>>>
>> --
>> --
>> simon elliston ball
>> @sireb
>>
> --
--
simon elliston ball
@sireb

Re: ingesting syslog and asa log into metron

[Simon Elliston Ball]

Use the nifi listen syslog processor to push Asa logs into a Kafka topic,
then the metron asa parser to get that into your metron flow.

Simon

On Mon, 25 Nov 2019 at 14:12, updates on tube 
wrote:

> hey guys first I really appreciate your urgent replies on my previous
> posts >>
> and for now, I went to ask how can I ingest Syslog and asa log into apache
> metron using nifi?
>
-- 
--
simon elliston ball
@sireb

Re: Score not being issued by ThreatIntel Enrichment

[Simon Elliston Ball]

The threat intel rules will only be run to create a score if the is_alert
field is present in the alert message. You can use the enrichments stage to
set this based on detections / threat intel / enrichment sources etc. If
that field is set true, then you should see your scoring rules run.

Simon

On Thu, 21 Nov 2019 at 16:10, Gonçalo Pedras  wrote:

> Hi,
>
> I’ve deployed Metron alongside the current Ambari version using the Metron
> HDP3.1 support provided by a branch in the GitHub project.
>
>
>
> Fast forward, I’m testing Metron:
>
> 1.   I’ve deployed a custom CSV parser with 3 fields ( 2 dummy fields
> and a IP field). The parser works fine.
>
> 2.   Created a custom template for my sensor with the required fields
> (guid, ip_src_addr, ip_dst_addr, …) for Elasticsearch for the pattern
> indexes. Works fine, even Metron can recognize the indexes.
>
> 3.   Created a custom Threat Intel source (extractor  enrichment
> config JSON files, and the CSV content file). Also works fine, I’ve tested
> it using Stellar with ENRICHMENT_GET function, returning the content I
> wrote in the CSV file.
>
> 4.   Configured Threat Triage for the sensor with the rule
> “ip_src_addr == ‘’” and the score of 5.
> Doesn’t work… The data in the Elasticsearch’s index is still being issued
> without the threat score.
>
>
>
> The enrichment config of the threat intel source:
>
> {
>
>  "zkQuorum" : ":",
>
>  "sensorToFieldList": {
>
>"xcsvtest": {
>
>"type": "THREAT_INTEL",
>
>"fieldToEnrichmentTypes": {
>
>"ip_src_addr" : ["testList"]
>
>}
>
>}
>
>  }
>
> }
>
>
>
> My enrichment configuration:
>
>
>
> {
>
> "enrichment": {
>
>"fieldMap": {
>
>"geo": [
>
>
> "ip_src_addr"
>
>]
>
>},
>
>"fieldToTypeMap": {},
>
>"config": {}
>
> },
>
> "threatIntel": {
>
>"fieldMap": {},
>
>"fieldToTypeMap": {
>
>"ip_src_addr": [
>
>"testList"
>
>]
>
>},
>
>"config": {},
>
>"triageConfig": {
>
>"riskLevelRules": [
>
>{
>
>
> "name": "All_threat",
>
>
> "comment": "",
>
>
> "rule": "ip_src_addr == ‘8.8.8.8’ ",
>
>
> "reason": null,
>
>
> "score": "5"
>
>}
>
>],
>
>"aggregator": "MAX",
>
>"aggregationConfig": {}
>
>}
>
> },
>
> "configuration": {}
>
> }
>
>
>
>
>
>
>
> Appreciate any help.
>
> Thanks
>


-- 
--
simon elliston ball
@sireb

Re: Enable optional fields in csv parser

[Simon Elliston Ball]

A better way of doing this would be to use the fieldTransformation setting
and the REMOVE method to get rid of the extraneous fields. Docs are
included at
https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html#

That way you don’t need a separate preprocessing step.

Simon

On Sat, 16 Nov 2019 at 16:09, Hema malini  wrote:

> Thanks ..will do preprocessing of data..
>
> On Sat, 16 Nov, 2019, 9:25 PM Otto Fowler, 
> wrote:
>
>> No, there is no way to do this currently.
>>
>> The parser parses the line into and array of strings that must match the
>> size of the columns.
>>
>> The underlying opencsv parser does not support this either.  You may have
>> to do some normalization work on your data if you need to account for this.
>>
>>
>>
>>
>> On November 16, 2019 at 08:49:36, Hema malini (nhemamalin...@gmail.com)
>> wrote:
>>
>> Hi all,
>>
>> Is there any way to mark some columns as optional in column mapping in
>> CSV parser.
>>
>> Thanks and Regards,
>> Hema
>>
>> --
--
simon elliston ball
@sireb

Re: CSV parser

[Simon Elliston Ball]

Perhaps you could post your config? You should have a dictionary in it
called columns which maps column name to index.

Simon

On Tue, 12 Nov 2019 at 16:05, Hema malini  wrote:

> Yes. I uploaded as mentioned in the document.
>
> On Tue, 12 Nov, 2019, 9:31 PM Simon Elliston Ball, <
> si...@simonellistonball.com> wrote:
>
>> Did you upload your configs to zookeeper?
>>
>> On Tue, 12 Nov 2019 at 16:00, Hema malini 
>> wrote:
>>
>>> I referred the document - .
>>> https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html..
>>> I modified the file $METRON_HOME/config/zookeeper/parsers/csv.json. i am
>>> getting still column metadata not defined.please let me know what I am
>>> missing in this.
>>>
>>> On Tue, 12 Nov, 2019, 9:23 PM Simon Elliston Ball, <
>>> si...@simonellistonball.com> wrote:
>>>
>>>> You modify the column data in the parser config. I suggest checking the
>>>> docs for the csv parser.
>>>>
>>>> On Tue, 12 Nov 2019 at 15:51, Hema malini 
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I enabled CSV parser and created Kafka topic for CSV. Where i need to
>>>>> configure column metadata. Similarly for json parser where i should enable
>>>>> the json key ,value. Do i need to modify the parser class and redeploy the
>>>>> jar again.
>>>>>
>>>>> Thanks and regards,
>>>>> Hema
>>>>>
>>>> --
>>>> --
>>>> simon elliston ball
>>>> @sireb
>>>>
>>> --
>> --
>> simon elliston ball
>> @sireb
>>
> --
--
simon elliston ball
@sireb

Re: CSV parser

[Simon Elliston Ball]

Did you upload your configs to zookeeper?

On Tue, 12 Nov 2019 at 16:00, Hema malini  wrote:

> I referred the document - .
> https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html..
> I modified the file $METRON_HOME/config/zookeeper/parsers/csv.json. i am
> getting still column metadata not defined.please let me know what I am
> missing in this.
>
> On Tue, 12 Nov, 2019, 9:23 PM Simon Elliston Ball, <
> si...@simonellistonball.com> wrote:
>
>> You modify the column data in the parser config. I suggest checking the
>> docs for the csv parser.
>>
>> On Tue, 12 Nov 2019 at 15:51, Hema malini 
>> wrote:
>>
>>> Hi,
>>>
>>> I enabled CSV parser and created Kafka topic for CSV. Where i need to
>>> configure column metadata. Similarly for json parser where i should enable
>>> the json key ,value. Do i need to modify the parser class and redeploy the
>>> jar again.
>>>
>>> Thanks and regards,
>>> Hema
>>>
>> --
>> --
>> simon elliston ball
>> @sireb
>>
> --
--
simon elliston ball
@sireb

Re: Metron parser for firewall

[Simon Elliston Ball]

Grok cannot easily parse asa on it’s own, which is why there is a separate
parser class for asa. Consider using they parser class in your
configuration (you’ll find details on that in the parser documentation). If
there are messages that are not covered in ten existing map and patterns,
you should consider submitting a PR.

Simon

On Fri, 8 Nov 2019 at 07:58, Hema malini  wrote:

> Hi ,
>
> How can i enable that parser? Also the grok patterns in that is missing
> few more sys log firewall messages. Do i have to add that as additional
> parser or can i use grok patterns.
>
> Thanks and Regards,
> Hema
>
> On Fri, 8 Nov, 2019, 8:32 PM Simon Elliston Ball, <
> si...@simonellistonball.com> wrote:
>
>> There is a Cisco ASA parser built into metron. I suggest using that.
>>
>> Simon
>>
>> On Fri, 8 Nov 2019 at 04:50, Hema malini  wrote:
>>
>>> Hi,
>>> Any parser available for firewall logs for Metron. I am trying to
>>> integrate ciscoasa firewall logs with Metron.
>>>
>>> Thanks and regards,
>>> Hema
>>>
>> --
>> --
>> simon elliston ball
>> @sireb
>>
> --
--
simon elliston ball
@sireb

Re: Apache Metron production deployment

[Simon Elliston Ball]

I would recommend against using the AWS deploy method on the github. It’s not 
really that well maintained, and the Ambari method is definitely the preferred 
at present, but then I tend to use a distro to install, or full dev if it’s 
just for local testing.

Simon

> On 29 Oct 2019, at 13:35, Eric Jacksch  wrote:
> 
> I thought I may have just missed something, but one of my customer's
> DevOps team worked on it for three days and couldn't get it going
> either.
> 
>> On Tue, 29 Oct 2019 at 09:32, Marcus Persson  wrote:
>> 
>> then its just not me that have had problems... BUt my problem was best way 
>> of running it on CentOS..
>> 
>>> On 2019/10/29 13:22:08, Eric Jacksch  wrote:
>>> We unfortunately gave up after trying several approaches to getting
>>> Metron running in AWS. I'm disappointed -- I think Metron has huge
>>> potential.
>>> 
>>> I suspect those who are using it have established development systems
>>> and that there are some undocumented prerequisites. If anyone on the
>>> Metron team has time, just try to deploy it in AWS using a freshly
>>> spun up EC2 instance as your build/deploy machine and the issues will
>>> rapidly become evident.
>>> 
>>> Regards,
>>> Eric
>>> 
>>> On Tue, 29 Oct 2019 at 08:54,  wrote:
 
 Hello,
 
 How are you using Metron in a production environment?
 I have checked around and my conclusion is that the Ambari-solution
 should not be used in production environment and just for poc/testing.
 
 I want to run Metron with Hadoop on CentOS 7 or 8, If you have other
 recomendation I can change OS.
 
 
 Thanks alot in advance!
 
 Best Regards
 Marcus
>>> 
>>> 
>>> 
>>> --
>>> Eric Jacksch, CPP, CISM, CISSP
>>> e...@jacksch.com
>>> Twitter: @EricJacksch
>>> https://SecurityShelf.com
>>> 
> 
> 
> 
> -- 
> Eric Jacksch, CPP, CISM, CISSP
> e...@jacksch.com
> Twitter: @EricJacksch
> https://SecurityShelf.com

Re: Threat Intel hailataxii

[Simon Elliston Ball]

Looks to me like your discovery server is not working properly, hence the 
failure message. This could be a temporary connectivity issue, but if it’s 
repeatable I would look into your opentaxii config. 

Simon 

> On 29 Oct 2019, at 13:23, Thiago Rahal Disposti  
> wrote:
> 
> 
> Anyone knows what's going on with the Hail a Taxii server?
> 
> I getting a service temporarily unavailable response for more than 3 weeks 
> now.
> 
> 
> 
> 
> 
> Thanks,
> Thiago Rahal

Re: Apache Metron production deployment

[Simon Elliston Ball]

Everyone I know of running metron at scale in production uses the ambari
based install method through a distribution, running on centos 6 on HDP
2.6.5 and for the new feature branch centos 7 on top of HDP 3.1.4.

Simon

On Tue, 29 Oct 2019 at 12:54,  wrote:

> Hello,
>
> How are you using Metron in a production environment?
> I have checked around and my conclusion is that the Ambari-solution
> should not be used in production environment and just for poc/testing.
>
> I want to run Metron with Hadoop on CentOS 7 or 8, If you have other
> recomendation I can change OS.
>
>
> Thanks alot in advance!
>
> Best Regards
> Marcus
>
-- 
--
simon elliston ball
@sireb

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Simon Elliston Ball]

Not sure it’s in the scope of the project to handle the HDP upgrade as
well, I would scope it to metron config only, and point at the extensive
upgrade capability of Ambari, rather than us trying to recreate the way the
distribution works.

Simon

On Tue, 27 Aug 2019 at 22:23, Otto Fowler  wrote:

> If anyone can think of the things that need to be backed up, please
> comment the jira.
>
>
>
>
> On August 27, 2019 at 17:07:20, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> Good idea METRON–2239 [blocker].
>
>
>
> On August 27, 2019 at 16:30:13, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> You could always submit a Jira :)
>
> On Tue, 27 Aug 2019 at 21:27, Otto Fowler  wrote:
>
>> You are right, that is much better than backup_metron_configs.sh.
>>
>>
>>
>> On August 27, 2019 at 16:05:38, Simon Elliston Ball (
>> si...@simonellistonball.com) wrote:
>>
>> You can do this with zk_load_configs and Ambari’s blueprint api, so we
>> kinda already do.
>>
>> Simon
>>
>> On Tue, 27 Aug 2019 at 20:19, Otto Fowler 
>> wrote:
>>
>>> Maybe we need some automated method to backup configurations and restore
>>> them.
>>>
>>>
>>>
>>> On August 27, 2019 at 14:46:58, Michael Miklavcic (
>>> michael.miklav...@gmail.com) wrote:
>>>
>>> > Once you back up your metron configs, the same configs that worked on
>>> the previous version will continue to work on the version running on HDP
>>> 3.x.  If there is any discrepancy between the two or additional settings
>>> will be required, those will be documented in the release notes.  From the
>>> Metron perspective, this upgrade would be no different than simply
>>> upgrading to the new Metron version.
>>>
>>> This upgrade cannot be performed the same way we've done it in the past.
>>> A number of platform upgrades, including OS, are required:
>>>
>>>1. Requires the OS to be updated on all nodes because there are no
>>>Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
>>>2. The final new HBase code will not run on HDP 2.6
>>>3. The MPack changes for new Ambari are not backwards compatible
>>>4. YARN and HDFS/MR are also at risk to be backwards incompatible
>>>
>>>
>>> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
>>> michael.miklav...@gmail.com> wrote:
>>>
>>>> Adding the dev list back into the thread (a reply-all was missed).
>>>>
>>>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota 
>>>> wrote:
>>>>
>>>>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>>>>> everyone will likely need to migrate by end of year.  Doing this platform
>>>>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>>>>> looks like so they have time to plan and upgrade at their own pace.
>>>>> Feature-wise, the Metron application itself will be unchanged.  It is
>>>>> merely the platform underneath that is changing.  HDP itself can be
>>>>> upgraded per instructions here:
>>>>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>>>>
>>>>> Once you back up your metron configs, the same configs that worked on
>>>>> the previous version will continue to work on the version running on HDP
>>>>> 3.x.  If there is any discrepancy between the two or additional settings
>>>>> will be required, those will be documented in the release notes.  From the
>>>>> Metron perspective, this upgrade would be no different than simply
>>>>> upgrading to the new Metron version.
>>>>>
>>>>> James
>>>>>
>>>>>
>>>>> 27.08.2019, 07:09, "Simon Elliston Ball" >>>> >:
>>>>>
>>>>> Something worth noting here is that HDP 2.6.5 is quite old and
>>>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of 
>>>>> a
>>>>> large number of users who require this upgrade ASAP, and in fact an aware
>>>>> of zero users who wish to remain on HDP 2.
>>>>>
>>>>> Perhaps those users who want to stay on the old platform can stick
>>>>> their hands up and raise concerns, but this move will likely have to 
>>>>> happen
>>>>> very soon.
>>>>

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Simon Elliston Ball]

You could always submit a Jira :)

On Tue, 27 Aug 2019 at 21:27, Otto Fowler  wrote:

> You are right, that is much better than backup_metron_configs.sh.
>
>
>
>
> On August 27, 2019 at 16:05:38, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> You can do this with zk_load_configs and Ambari’s blueprint api, so we
> kinda already do.
>
> Simon
>
> On Tue, 27 Aug 2019 at 20:19, Otto Fowler  wrote:
>
>> Maybe we need some automated method to backup configurations and restore
>> them.
>>
>>
>>
>> On August 27, 2019 at 14:46:58, Michael Miklavcic (
>> michael.miklav...@gmail.com) wrote:
>>
>> > Once you back up your metron configs, the same configs that worked on
>> the previous version will continue to work on the version running on HDP
>> 3.x.  If there is any discrepancy between the two or additional settings
>> will be required, those will be documented in the release notes.  From the
>> Metron perspective, this upgrade would be no different than simply
>> upgrading to the new Metron version.
>>
>> This upgrade cannot be performed the same way we've done it in the past.
>> A number of platform upgrades, including OS, are required:
>>
>>1. Requires the OS to be updated on all nodes because there are no
>>Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
>>2. The final new HBase code will not run on HDP 2.6
>>3. The MPack changes for new Ambari are not backwards compatible
>>4. YARN and HDFS/MR are also at risk to be backwards incompatible
>>
>>
>> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>>
>>> Adding the dev list back into the thread (a reply-all was missed).
>>>
>>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota 
>>> wrote:
>>>
>>>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>>>> everyone will likely need to migrate by end of year.  Doing this platform
>>>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>>>> looks like so they have time to plan and upgrade at their own pace.
>>>> Feature-wise, the Metron application itself will be unchanged.  It is
>>>> merely the platform underneath that is changing.  HDP itself can be
>>>> upgraded per instructions here:
>>>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>>>
>>>> Once you back up your metron configs, the same configs that worked on
>>>> the previous version will continue to work on the version running on HDP
>>>> 3.x.  If there is any discrepancy between the two or additional settings
>>>> will be required, those will be documented in the release notes.  From the
>>>> Metron perspective, this upgrade would be no different than simply
>>>> upgrading to the new Metron version.
>>>>
>>>> James
>>>>
>>>>
>>>> 27.08.2019, 07:09, "Simon Elliston Ball" :
>>>>
>>>> Something worth noting here is that HDP 2.6.5 is quite old and
>>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>>>> large number of users who require this upgrade ASAP, and in fact an aware
>>>> of zero users who wish to remain on HDP 2.
>>>>
>>>> Perhaps those users who want to stay on the old platform can stick
>>>> their hands up and raise concerns, but this move will likely have to happen
>>>> very soon.
>>>>
>>>> Simon
>>>>
>>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler 
>>>> wrote:
>>>>
>>>> Although we had the discussion, and some great ideas where passed
>>>> around, I do not believe we came to some kind of consensus on what 1.0
>>>> should look like. So that discussion would have to be picked up again so
>>>> that we could know where we are at, and make it an actual thing if we were
>>>> going to make this a 1.0 release.
>>>>
>>>> I believe that the issues raised in that discussion gating 1.0 are
>>>> still largely applicable, including upgrade.
>>>>
>>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
>>>> supporting 3.1 work and releases. So every user and deployment we currently
>>>> have will feel real pain, have to slay real dragons to move forward with
>>>> metron.
>>>>
>>

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Simon Elliston Ball]

You can do this with zk_load_configs and Ambari’s blueprint api, so we
kinda already do.

Simon

On Tue, 27 Aug 2019 at 20:19, Otto Fowler  wrote:

> Maybe we need some automated method to backup configurations and restore
> them.
>
>
>
>
> On August 27, 2019 at 14:46:58, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> > Once you back up your metron configs, the same configs that worked on
> the previous version will continue to work on the version running on HDP
> 3.x.  If there is any discrepancy between the two or additional settings
> will be required, those will be documented in the release notes.  From the
> Metron perspective, this upgrade would be no different than simply
> upgrading to the new Metron version.
>
> This upgrade cannot be performed the same way we've done it in the past. A
> number of platform upgrades, including OS, are required:
>
>1. Requires the OS to be updated on all nodes because there are no
>Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
>2. The final new HBase code will not run on HDP 2.6
>3. The MPack changes for new Ambari are not backwards compatible
>4. YARN and HDFS/MR are also at risk to be backwards incompatible
>
>
> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> Adding the dev list back into the thread (a reply-all was missed).
>>
>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota  wrote:
>>
>>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>>> everyone will likely need to migrate by end of year.  Doing this platform
>>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>>> looks like so they have time to plan and upgrade at their own pace.
>>> Feature-wise, the Metron application itself will be unchanged.  It is
>>> merely the platform underneath that is changing.  HDP itself can be
>>> upgraded per instructions here:
>>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>>
>>> Once you back up your metron configs, the same configs that worked on
>>> the previous version will continue to work on the version running on HDP
>>> 3.x.  If there is any discrepancy between the two or additional settings
>>> will be required, those will be documented in the release notes.  From the
>>> Metron perspective, this upgrade would be no different than simply
>>> upgrading to the new Metron version.
>>>
>>> James
>>>
>>>
>>> 27.08.2019, 07:09, "Simon Elliston Ball" :
>>>
>>> Something worth noting here is that HDP 2.6.5 is quite old and
>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>>> large number of users who require this upgrade ASAP, and in fact an aware
>>> of zero users who wish to remain on HDP 2.
>>>
>>> Perhaps those users who want to stay on the old platform can stick their
>>> hands up and raise concerns, but this move will likely have to happen very
>>> soon.
>>>
>>> Simon
>>>
>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler 
>>> wrote:
>>>
>>> Although we had the discussion, and some great ideas where passed
>>> around, I do not believe we came to some kind of consensus on what 1.0
>>> should look like. So that discussion would have to be picked up again so
>>> that we could know where we are at, and make it an actual thing if we were
>>> going to make this a 1.0 release.
>>>
>>> I believe that the issues raised in that discussion gating 1.0 are still
>>> largely applicable, including upgrade.
>>>
>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
>>> supporting 3.1 work and releases. So every user and deployment we currently
>>> have will feel real pain, have to slay real dragons to move forward with
>>> metron.
>>>
>>> With regards to support for older versions, the “backward capability”
>>> that has been mentioned, I would not say that I have any specific plan for
>>> that in mind. What I would say rather, is that I believe that we must be
>>> explicit, setting expectations correctly and clearly with regards to our
>>> intent while demonstrating that we have thought through the situation. That
>>> discussion has not happened, at least I do not believe that the prior dev
>>> thread really handles it in context.
>>>
>>> Depending on the upgrade situation for going to 3.1, it may be tha

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Simon Elliston Ball]

Something worth noting here is that HDP 2.6.5 is quite old and approaching
EoL rapidly, so the issue of upgrade is urgent. I am aware of a large
number of users who require this upgrade ASAP, and in fact an aware of zero
users who wish to remain on HDP 2.

Perhaps those users who want to stay on the old platform can stick their
hands up and raise concerns, but this move will likely have to happen very
soon.

Simon

On Tue, 27 Aug 2019 at 15:04, Otto Fowler  wrote:

> Although we had the discussion, and some great ideas where passed around,
> I do not believe we came to some kind of consensus on what 1.0 should look
> like. So that discussion would have to be picked up again so that we could
> know where we are at, and make it an actual thing if we were going to make
> this a 1.0 release.
>
> I believe that the issues raised in that discussion gating 1.0 are still
> largely applicable, including upgrade.
>
> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
> supporting 3.1 work and releases. So every user and deployment we currently
> have will feel real pain, have to slay real dragons to move forward with
> metron.
>
> With regards to support for older versions, the “backward capability” that
> has been mentioned, I would not say that I have any specific plan for that
> in mind. What I would say rather, is that I believe that we must be
> explicit, setting expectations correctly and clearly with regards to our
> intent while demonstrating that we have thought through the situation. That
> discussion has not happened, at least I do not believe that the prior dev
> thread really handles it in context.
>
> Depending on the upgrade situation for going to 3.1, it may be that a dual
> stream of releases or fixes or new features to the extent that we can do it
> will greatly reduce the pain for folks, or make it viable to stick with
> metron until they can upgrade.
>
> The issue of what metron *is* features wise may be another one we want to
> take up at some point. The idea being can we separate the metron
> _integration parts from the metron core functionality such that we can work
> on them separately and thus support multiple platforms through
> integrations/applications. Of course definition of metron’s value beyond
> integration, and what those features and application boundaries are would
> be necessary.
>
>
>
>
> On August 26, 2019 at 18:52:57, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> Hi devs and users,
>
> Some questions were asked in the Slack channel about our ongoing
> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original
> Hadoop upgrade discuss thread can be found here
> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
>
> The major issue and impact with upgrading the Hadoop platform is that
> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x.
> Here is a sampling of core components we depend on that we know of so
> far that are not backwards compatible:
>
>1. The Core OS - we currently base our builds and test deployment off
>of artifacts pulled from HDP. The latest rev of HDP no longer ships RPMs
>for Centos 6, which means we need to upgrade to Centos 7
>2. Ambari
>3. HBase
>
> This differs from individual components we've upgraded in the past in that
> our code could still be deployed on the old as well as new version of the
> component in a backwards compatible way. Based on semantic versioning, I
> don't know if we can introduce this level of change in a point release,
> which is the reason for kicking off this discussion. In the past, users and
> developers in the community have suggested that they are -1 on a 1.x
> release that does not provide an upgrade
> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
> .
>
> Is there a way we can avoid a 1.x release? If we do need 1.x, do we still
> see upgrades as a gating function? The main issue is that this has the
> potential to drag out the upgrade and further couple it with other
> features. And with Storm 1.x being eol'ed, I'm not sure this is something
> we can wait much longer for. I'll think on this and send out my own
> thoughts once folks have had a chance to review.
>
> Best,
> Mike Miklavcic
> Apache Metron, PMC, committer
>
>
> --
--
simon elliston ball
@sireb

Re: Profiler Examples Not working

[Simon Elliston Ball]

Looks like you did not initialise a profiler. Checkout profiler init, and
the early part of the examples.

On Mon, 22 Jul 2019 at 11:04, Farrukh Naveed Anjum 
wrote:

> [Stellar]>>> msg :=
> '{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10","bytes_in":"234"}'
>
> {"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10","bytes_in":"234"}
> [Stellar]>>> PROFILER_APPLY(msg, profiler)
> [!] Unable to parse: PROFILER_APPLY(msg, profiler) due to: null with
> relevant variables
> msg={"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10","bytes_in":"234"},profiler=missing
> org.apache.metron.stellar.dsl.ParseException: Unable to parse:
> PROFILER_APPLY(msg, profiler) due to: null with relevant variables
> msg={"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10","bytes_in":"234"},profiler=missing
> at
> org.apache.metron.stellar.common.BaseStellarProcessor.createException(BaseStellarProcessor.java:173)
> at
> org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:154)
> at
> org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.executeStellar(DefaultStellarShellExecutor.java:407)
> at
> org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:257)
> at
> org.apache.metron.stellar.common.shell.cli.StellarShell.execute(StellarShell.java:359)
> at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at
> org.apache.metron.profiler.repl.ProfilerFunctions$ProfilerApply.apply(ProfilerFunctions.java:140)
> at
> org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:664)
> at
> org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:259)
> at
> org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151)
> ... 7 more
>
>
> --
> *Best Regards*
> Farrukh Naveed Anjum
> *M:* +92 321 5083954 (WhatsApp Enabled)
> *W:* https://www.farrukh.cc/
>
-- 
--
simon elliston ball
@sireb

Re: [ask] create profile for profiler with multiple fields on foreach

[Simon Elliston Ball]

I suspect what you mean is a concatenation of the source and dest to get
the pairs as a foreach key, so you actually want something like: "foreach":
"ip_src_addr + '-' + ip_dst_addr"

Simon

On Wed, 17 Jul 2019 at 17:01, Anil Donthireddy 
wrote:

> Hi,
>
>
>
> The expression provided in foreach statement is not valid. It should
> resolve to single object string.
>
>
>
> Thanks,
>
> Anil.
>
>
>
> *From:* Youzha [mailto:yuza.ras...@gmail.com]
> *Sent:* Wednesday, July 17, 2019 9:17 PM
> *To:* user@metron.apache.org
> *Subject:* [ask] create profile for profiler with multiple fields on
> foreach
>
>
>
> Hi,
>
>
>
> it is possible to use multiple fields inside foreach on profiler ?
>
>
>
> i’ve try using AND like below but it failed. pls help
>
>
>
> {
>
>   "profiles": [
>
> {
>
>   "profile": "hello-world",
>
>   "onlyif":  "exists(ip_src_addr) AND exists(ip_dst_addr)",
>
>   "foreach": "ip_src_addr AND ip_dst_addr,
>
>   "init":{ "count": "0" },
>
>   "update":  { "count": "count + 1" },
>
>   "result":  "count"
>
> }
>
>   ]
>
> }
>
>

-- 
--
simon elliston ball
@sireb

Re: batch indexing in JSON format

[Simon Elliston Ball]

Most users will have a batch process converting the JSON short term output into 
ORC or Parquet files, often adding them to hive tables at the same time. I 
usually do this with a spark job run every hour, or even every 15mins or less 
in some cases for high throughput environments. Anecdotally, I’ve found ORC 
compresses slightly better for most Metron data than parquet, but the 
difference is marginal. 

The reason for this is that HDFS writer was built of the goal of getting data 
persisted in HDFS as soon as possible, so writing a columnar format would 
introduce latency to the streaming process. I suspect that a dev list 
discussion on schema management and alternative output formats will be 
forthcoming. To handle that with a sensible approach to schema migration is not 
trivial, but certainly desirable.

Simon

> On 15 Jul 2019, at 13:25,  
>  wrote:
> 
> Hello all,
>  
> I have a question regarding batch indexing. As as I can see, data are stored 
> in json format in hdfs. Nevertheless, this uses a lot of storage because of 
> json verbosity, enrichment,.. Is there any way to use parquet for example? I 
> guess it’s possible to do it the day after, I mean you read the json and with 
> spark you save as another format, but is it possible to choose the format at 
> the batch indexing configuration level?
>  
> Thanks a lot
>  
> Stéphane
>  
>  

Re: flatfile_summarizer

[Simon Elliston Ball]

Another workaround might be to specify a never occurring character as the
separator and sticking with CSV.

Simon

On Wed, 10 Jul 2019 at 11:26, Michael Miklavcic 
wrote:

> Hi David,
>
> In this case you would probably want to write your own extractor by
> implementing the following interface and setting it as your extractor
> implementation -
> https://github.com/apache/metron/blob/5d3e73ab95adf0c8f49c3f821975740e365df91a/metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/extractor/Extractor.java
>
> Reference -
> https://github.com/apache/metron/blob/f43035c02ef01f07ff382bbf136eb1bada727fbb/metron-platform/metron-data-management/README.md#extractor-framework
>
> Best,
> Mike
>
>
> On Fri, Jul 5, 2019 at 12:40 PM David Auclair 
> wrote:
>
>> I’m trying to generate a serialized object using the flatfile_summarizer
>> and I’m having some difficulty…
>>
>>
>>
>> I’m trying to take a list of RegEx’s in a text file (one regex per line),
>> and load it with the following extractor:
>>
>>
>>
>> {
>>
>>   "config" : {
>>
>> "columns" : {
>>
>>   "regex" : 0
>>
>> },
>>
>> "value_filter" : "LENGTH(regex) > 0",
>>
>> "state_init" : "SET_INIT()",
>>
>> "state_update" : {
>>
>>   "state" : "SET_ADD(state,regex)"
>>
>> },
>>
>> "state_merge" : "SET_MERGE(states)",
>>
>> "separator" : ","
>>
>>   },
>>
>>   "extractor" : "CSV"
>>
>> }
>>
>>
>>
>> Running the tool, as follows:
>>
>> /usr/hcp/current/metron/bin/flatfile_summarizer.sh -i ./regex.txt -o
>> regex.ser -e regex_extractor.json
>>
>>
>>
>> I end up with the following error message:
>>
>> Exception in thread "main" java.lang.NullPointerException
>>
>> at
>> org.apache.metron.dataloads.nonbulk.flatfile.writer.LocalWriter.write(LocalWriter.java:45)
>>
>> at
>> org.apache.metron.dataloads.nonbulk.flatfile.writer.Writers.write(Writers.java:54)
>>
>> at
>> org.apache.metron.dataloads.nonbulk.flatfile.writer.Writer.write(Writer.java:30)
>>
>> at
>> org.apache.metron.dataloads.nonbulk.flatfile.importer.LocalSummarizer.importData(LocalSummarizer.java:136)
>>
>> at
>> org.apache.metron.dataloads.nonbulk.flatfile.SimpleFlatFileSummarizer.main(SimpleFlatFileSummarizer.java:51)
>>
>> at
>> org.apache.metron.dataloads.nonbulk.flatfile.SimpleFlatFileSummarizer.main(SimpleFlatFileSummarizer.java:38)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:498)
>>
>> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
>>
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
>>
>>
>>
>> Am I doing something wrong?  Also, is there a better alternative to the
>> “CSV” extractor?  I’m ideally looking to load the entire line, regardless
>> of any specific characters (regex may contain commas for example).
>>
>>
>>
>> Thanks in advance,
>>
>> David Auclair
>>
>>
>>
>

-- 
--
simon elliston ball
@sireb

Re: TASK [bro : Download bro] - fatal: [node1]: FAILED!

[Simon Elliston Ball]

even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
>> PURPOSE.
>> --
>> Compiler is C++11 compliant
>> --
>> Linux ub1604in2017 4.4.0-148-generic #174-Ubuntu SMP Tue May 7 12:20:14
>> UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
>> --
>> Total System Memory = 15994.3 MB
>> Processor Model: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
>> Processor Speed: 3361.617 MHz
>> Processor Speed: 3448.476 MHz
>> Processor Speed: 3323.898 MHz
>> Processor Speed: 3400.664 MHz
>> Processor Speed: 3359.625 MHz
>> Processor Speed: 3283.125 MHz
>> Processor Speed: 3382.335 MHz
>> Processor Speed: 3275.023 MHz
>> Total Physical Processors: 8
>> Total cores: 32
>> Disk information:
>> /dev/sdb195G   52G   38G  58% /
>
>
>
> *Best regards!*
> Pablo de Azevedo
>


-- 
--
simon elliston ball
@sireb

Re: [ask] detect unsual login duration

[Simon Elliston Ball]

You could pull that out in a report in zeppelin easily enough, but to do it
real-time we would need to add some sort of state table and a triggered
check of that state, unless you say wanted to alert only on logout (I’m
assuming you don’t want to wait for the logout, but alerts after some fixed
duration or better some anomalous duration?)

Is that the sort of use case?

Simon

On Thu, 16 May 2019 at 03:59, tkg_cangkul  wrote:

> Hi,
>
> Does metron support to do detection an unusual login duration?
>
> For example.
> IP A login for 3 days without logout. then metron will give some alert
> to us.
>
> If this possible, how to do that?
> Pls help.
>
>
> Best Regards,
>
> Tkg_cangkul
>
> --
--
simon elliston ball
@sireb

Re: Issue when trying to load JSON

[Simon Elliston Ball]

jar:1.1.0.2.6.5.1050-37]
>
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> [storm-core-1.1.0.2.6.5.1050-37.jar:1.1.0.2.6.5.1050-37]
>
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> [storm-core-1.1.0.2.6.5.1050-37.jar:1.1.0.2.6.5.1050-37]
>
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
> [storm-core-1.1.0.2.6.5.1050-37.jar:1.1.0.2.6.5.1050-37]
>
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> [storm-core-1.1.0.2.6.5.1050-37.jar:1.1.0.2.6.5.1050-37]
>
> at
> org.apache.storm.daemon.executor$fn__10195$fn__10208$fn__10263.invoke(executor.clj:855)
> [storm-core-1.1.0.2.6.5.1050-37.jar:1.1.0.2.6.5.1050-37]
>
> at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:484)
> [storm-core-1.1.0.2.6.5.1050-37.jar:1.1.0.2.6.5.1050-37]
>
>     at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
>
>
>
>
>
> How can I debug this?
>
>
>
> Thanks
>
>
>
> Stéphane
>
> _
>
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu 
> ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
> electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou 
> falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged 
> information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete 
> this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been 
> modified, changed or falsified.
> Thank you.
>
>

-- 
--
simon elliston ball
@sireb

Re: Question about "parser_invalid"

[Simon Elliston Ball]

Timestamp in Metron is always a unix epoch to avoid things like timezone issues.

In this case, you can resolve this using a field transformation at the parsing 
stage, with the TO_EPOCH_TIMESTAMP function. Some custom parsers already do 
this, but for those that don’t, a simple bit of stellar will clean it up.

Simon

> On 10 Apr 2019, at 07:34,  
>  wrote:
> 
> Hello everybody,
>  
> Don’t worry, I won’t ask you to debug my Grok statement J
>  
> By the way, I’m facing the following situation: I have in my “error_index” 
> Elastic index some documents with a raw_message field that shows that the 
> origin message was parsed (see screenshot) and contains in addition an 
> “original_string” which is the raw message:
> 
>  
> What is wrong here? Why does it go to error_index?
>  
> Thanks,
>  
> Stéphane
> _
> 
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu 
> ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
> electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou 
> falsifie. Merci.
> 
> This message and its attachments may contain confidential or privileged 
> information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete 
> this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been 
> modified, changed or falsified.
> Thank you.

Re: Metron concept

[Simon Elliston Ball]

One thing worth noting is that group.id is essentially a client identifier, so 
if you specify one that matches another consumer (such as Metron topologies) 
then they will interfere, and you are likely to balance across your console and 
the actual Metron processes, so generally when watching a Kafka topic for 
debugging you should let Kafka choose a random group.id. You should not have to 
specify partition either if you want the console to show all messages.

Simon 

> On 9 Apr 2019, at 11:17,  
>  wrote:
> 
> Hello,
>  
> I haven’t sorted out yet this issue, but I think I’ve narrowed it. Actually, 
> after many tests with Kafka console-consumer and basic Python scripts, I 
> realize that I can only consume messages when I specify the partition number 
> and not the group.id. This is of course not what storm tries to do, it tries 
> do dynamically fetch the right partition and commit the offset.
>  
> My Kafka cluster is a fresh one installed with Hortonworks data platform with 
> 3 brokers. I can’t find any kind of option around this behavior. Moreover, we 
> regularly use Kafka for some other purpose with our docker images and have 
> never faced issues like this…
>  
> Any idea?
>  
> Thanks,
>  
> Stéphane
>  
>  
> From: DAVY Stephane OBS/CSO 
> Sent: Monday, April 08, 2019 17:56
> To: user@metron.apache.org
> Subject: RE: Metron concept
>  
> Well, I realize that the console-consumer works with the—zookeeper option, 
> which is the “old consumer”, while it doesn’t work when I specify 
> –bootstrap-server, which is the “new consumer” way. So, it looks like a Kafka 
> issue…
>  
>  
> From: DAVY Stephane OBS/CSO 
> Sent: Monday, April 08, 2019 16:45
> To: 'user@metron.apache.org'
> Subject: RE: Metron concept
>  
> Hello Simon,
>  
> I send just one line at a time, and the line has been validated in the Metron 
> UI. I see no message in the topology logs. I switched to DEBUG mode, and I 
> can see the following sequence again and again:
>  
> 2019-04-08 16:35:50.463 o.a.k.c.c.i.AbstractCoordinator 
> Thread-14-kafkaSpout-executor[4 4] [DEBUG] Sending coordinator request for 
> group forti1_parser to broker r-petya:6667 (id: 1011 rack: /default-rack)
> 2019-04-08 16:35:50.463 o.a.k.c.c.i.AbstractCoordinator 
> Thread-14-kafkaSpout-executor[4 4] [DEBUG] Received group coordinator 
> response ClientResponse(receivedTimeMs=1554734150463, disconnected=false, 
> request=ClientRequest(expectResponse=true, 
> callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@35437dce,
>  
> request=RequestSend(header={api_key=10,api_version=0,correlation_id=61518,client_id=consumer-1},
>  body={group_id=forti1_parser}), createdTimeMs=1554734150463, 
> sendTimeMs=1554734150463), 
> responseBody={error_code=15,coordinator={node_id=-1,host=,port=-1}})
> 2019-04-08 16:35:50.562 o.a.k.c.NetworkClient Thread-14-kafkaSpout-executor[4 
> 4] [DEBUG] Sending metadata request {topics=[forti1]} to node 1011
> 2019-04-08 16:35:50.562 o.a.k.c.Metadata Thread-14-kafkaSpout-executor[4 4] 
> [DEBUG] Updated cluster metadata version 30761 to Cluster(nodes = 
> [r-petya:6667 (id: 1011 rack: /default-rack), r-jigsaw:6667 (id: 1012 rack: 
> /default-rack), r-wannacry.rd.francetelecom.fr:6667 (id: 1010 rack: 
> /default-rack)], partitions = [Partition(topic = forti1, partition = 0, 
> leader = 1012, replicas = [1012,], isr = [1012,]])
>  
>  
> Is it normal to have 
> “responseBody={error_code=15,coordinator={node_id=-1,host=,port=-1}})” in the 
> response?
>  
> Thanks,
>  
> Stéphane
>  
>  
> From: Simon Elliston Ball [mailto:si...@simonellistonball.com] 
> Sent: Monday, April 08, 2019 16:29
> To: user@metron.apache.org
> Subject: Re: Metron concept
>  
> Are you seeing events on the enrichments topic, and if so, are they getting 
> to indexing? Any messages in the storm logs for these topologies?
>  
> Are you also certain the parser is correct, and there are no invalid or error 
> messages being sent to the error index?
>  
> Simon 
>  
> On Mon, 8 Apr 2019 at 15:26,  wrote:
> Hello Nick,
>  
> Thanks for your answer. I went through this post and see that all my events 
> should go in Elastic, which is what I want, but which it isn’t what I get L
>  
> I have setup the following basic setup:
> -  New telemetry with grok parser (validated in UI with sample) and a 
> kafka topic => the topic didn’t exist before, and it is created automatically 
> as I can see with the kafka-topics.sh CLI utility
> 
> -  A simple Nifi flow to push data in this topic => I can see some 
> data in the topic with the kafka-console-consumer.sh CLI utility.
> 
>  
> But I ha

Re: Metron concept

[Simon Elliston Ball]

Are you seeing events on the enrichments topic, and if so, are they getting
to indexing? Any messages in the storm logs for these topologies?

Are you also certain the parser is correct, and there are no invalid or
error messages being sent to the error index?

Simon

On Mon, 8 Apr 2019 at 15:26,  wrote:

> Hello Nick,
>
>
>
> Thanks for your answer. I went through this post and see that all my
> events should go in Elastic, which is what I want, but which it isn’t what
> I get L
>
>
>
> I have setup the following basic setup:
>
> -  New telemetry with grok parser (validated in UI with sample)
> and a kafka topic => the topic didn’t exist before, and it is created
> automatically as I can see with the kafka-topics.sh CLI utility
>
> -  A simple Nifi flow to push data in this topic => I can see
> some data in the topic with the kafka-console-consumer.sh CLI utility.
>
>
>
> But I have the feeling that my topology never consume Kafka messages. The
> Storm UI shows “0” figure nearly everywhere in my topology, and the Elastic
> index is not created (_cat/indices). I see also nothing in the “indexing”
> Kafka topic.
>
>
>
> But I see no error message, I don’t really know how to go on…
>
>
>
> Does anybody have a suggestion for me? I guess I’m not the first one with
> kind of issue but I cannot find any case close to mine.
>
>
>
>
>
>
>
> *From:* Nick Allen [mailto:n...@nickallen.org]
> *Sent:* Monday, April 08, 2019 15:17
> *To:* user@metron.apache.org
> *Subject:* Re: Metron concept
>
>
>
> All events are indexed by default.
>
>
>
> See if this guide helps you any.
> https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source
>
>
>
> On Mon, Apr 8, 2019 at 2:49 AM  wrote:
>
> Hello all,
>
>
>
> There is one my point that isn’t clear for me. When sending data into
> Metron, are all the events all indexed sent to Elastic and / or HDFS, or
> only the events that trigger a triage rule?
>
>
>
> For now I’m trying to send some FW logs in Metron, I feed a Kafka topic
> with Nifi, I can see that the topic has data thanks to Kafka CLI, but
> nothing more happens after I’ve configured a new source from UI management…
>
>
>
> Stéphane
>
> _
>
>
>
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc
>
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu 
> ce message par erreur, veuillez le signaler
>
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
> electroniques etant susceptibles d'alteration,
>
> Orange decline toute responsabilite si ce message a ete altere, deforme ou 
> falsifie. Merci.
>
>
>
> This message and its attachments may contain confidential or privileged 
> information that may be protected by law;
>
> they should not be distributed, used or copied without authorisation.
>
> If you have received this email in error, please notify the sender and delete 
> this message and its attachments.
>
> As emails may be altered, Orange is not liable for messages that have been 
> modified, changed or falsified.
>
> Thank you.
>
> _
>
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu 
> ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
> electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou 
> falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged 
> information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete 
> this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been 
> modified, changed or falsified.
> Thank you.
>
> --
--
simon elliston ball
@sireb

Re: Metron-REST is always stopping

[Simon Elliston Ball]

Did you check to see if it was listening? Sometimes this can misreport in 
ambari if you have an incorrect version of the python requests library 
installed.

Simon

> On 4 Apr 2019, at 22:47,  
>  wrote:
> 
> Hello all,
>  
> I’ve installed Metron last week and everything was working correctly. I’m 
> currently playing with and trying to understand how it works. After a few 
> hours spent on the Management GUI, I started to have some disconnections and 
> finally I’m no longer able to login. I can see that actually the metron rest 
> is stopped. Nevertheless, I unable to start it anymore. It is first reported 
> as “started” in Ambari, and then goes to “stopped” roughly one minute later.
>  
> Regarding the logs themselves:
> -  The last lines of /var/log/metron/metron-rest.log are:
>  
> 5359 [Atlas Notifier 0] WARN  o.a.k.c.p.ProducerConfig - The configuration 
> 'zookeeper.sync.time.ms' was supplied but isn't a known config.
> 5359 [Atlas Notifier 0] WARN  o.a.k.c.p.ProducerConfig - The configuration 
> 'session.timeout.ms' was supplied but isn't a known config.
> 5359 [Atlas Notifier 0] WARN  o.a.k.c.p.ProducerConfig - The configuration 
> 'auto.offset.reset' was supplied but isn't a known config.
> 5360 [Atlas Notifier 0] INFO  o.a.k.c.u.AppInfoParser - Kafka version : 
> 1.0.0.2.6.5.1050-37
> 5360 [Atlas Notifier 0] INFO  o.a.k.c.u.AppInfoParser - Kafka commitId : 
> 2ff1ddae17fb8503
>  
> -  The last lines in /var/log/ambari-agent/ambary-agent.log are:
> INFO 2019-04-04 16:41:57,905 RecoveryManager.py:255 - METRON_REST needs 
> recovery, desired = STARTED, and current = INSTALLED.
> INFO 2019-04-04 16:42:04,790 RecoveryManager.py:255 - METRON_REST needs 
> recovery, desired = STARTED, and current = INSTALLED.
> INFO 2019-04-04 16:42:15,629 RecoveryManager.py:255 - METRON_REST needs 
> recovery, desired = STARTED, and current = INSTALLED.
> INFO 2019-04-04 16:42:15,640 Controller.py:410 - Adding recovery command 
> START for component METRON_REST
>  
>  
> Is there any other good place to find some logs?
>  
> Please note that:
> -  Mariadb is up and running
> -  Filesystems are not full
> -  All the other Hortonworks services are up and running
>  
>  
> It actually  started to go weird when I stopped the bro, snort, yaf,… sensors 
> which I currently don’t need.
>  
> Thanks for your help,
>  
> Stéphane
>  
> _
> 
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu 
> ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
> electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou 
> falsifie. Merci.
> 
> This message and its attachments may contain confidential or privileged 
> information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete 
> this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been 
> modified, changed or falsified.
> Thank you.

Re: Metron logs for parser

[Simon Elliston Ball]

Metron runs as a series of storm topologies, so you find the logs in storm. 
Your best route to get there is via the storm ui from ambari.

Simon

> On 25 Mar 2019, at 23:53, Meenakshi.S  
> wrote:
> 
> Hi ,
>  
> I installed Metron using Ambari Server .
> I was able to see the logs for Metron-rest alone .
>  
> I am not able to see any logs in other Metron modules like parsers etc..
>  
> Can you please guide me on getting the logs for other modules also .
>  
>  
> Regards,
> Meenakshi

Re: Use case question

[Simon Elliston Ball]

Hi Sanket,

This is certainly an interesting case. Metron is deliberately designed for
flexibility in terms of ingest and schema, so that non-network data sources
and use cases can be accommodated. The one caveat I would suggest is that
the Metron pipeline is designed for analytics and detection, but not
necessarily for the kind of guaranteed latency you might need for something
like a web application experience. While it is streaming and realtime by
nature, it can in some circumstances take a second or so to get a message
from end to end, particularly if you have a lot of detection or models
running, so it's not ideal as part of an interactive process. That said,
for the actual detection of fraud, and strange behaviour patterns on your
website, it would be a great fit.

Hope that helps,
Simon

On Mon, 4 Mar 2019 at 02:04, Hammad  wrote:

> Following!!
>
> On Mon, Mar 4, 2019 at 2:29 PM Sanket Sharma 
> wrote:
>
>>
>>
>> Hi,
>>
>> I've been looking at metron for a few days now and I have a unique use -
>> thought of asking the experts if it makes sense to use metron in this
>> scenario.
>>
>> My understanding of the project so far is that its a framework built for
>> analyzing cybersecurity threats. This includes analyzing IP packets,
>> network traffics, URLs etc to calculate risk scores etc. The framework also
>> enables data scientists to build and test their models. There are data
>> collection plugins that collect data from variety of sources, stream it
>> over kafka and makes them available for use by various models.
>>
>> Now, we have a customer facing portal where customers login, submit all
>> kinds of orders and transactions. We were looking at ways to analyze fraud
>> that originates from our portal and I stumbled upon Metron. While we can
>> definitely use Metron for analyzing source traffic, but would it be a good
>> idea to use Metron to analyze the actual transactions themselves? I do
>> understand that we will have to build our models etc. but given that all
>> the heavy lifting is already done, I'm tempted to try Metron for this use
>> case (instead of re-inventing the wheel).
>>
>> Is this possible/recommended? Or would you recommend using Metron
>> strictly for network related analysis?
>>
>> Best Regards,
>> Sanket
>>
>

-- 
--
simon elliston ball
@sireb

Re: Help regarding Parser Configuration

[Simon Elliston Ball]

You might like to look into parser chaining for this: 
https://metron.apache.org/current-book/metron-platform/metron-parsers/ParserChaining.html

Simon 

> On 20 Feb 2019, at 16:47, Farrukh Naveed Anjum  
> wrote:
> 
> Yes, I am using BRO Parser, Can I sub divide the message field
> 
>> On Wed, Feb 20, 2019 at 7:39 PM Otto Fowler  wrote:
>> Can you print what the fields are after parsing?  These are the fields that 
>> you will be able to use Stellar on, to possibly extract your info.
>> Are you using the Bro parser?
>> 
>> 
>>> On February 20, 2019 at 02:14:17, Farrukh Naveed Anjum 
>>> (anjum.farr...@gmail.com) wrote:
>>> 
>>> Hi,
>>> I wanted to know how can I define and extract a field in parser from 
>>> messages. With If It Exists like option
>>> 
>>> For example. I am using Bro Syslog. Following is a sample data
>>> 
>>> 
>>> SYSLOG | severity:ERR uid:C5oe7F5SYMWqfVKKj id.orig_p:514
>>> id.resp_p:514 proto:udp id.orig_h:10.2.2.1 message:Feb 20 12:11:18
>>> suricata[72950]: [1:2000538:8] ET SCAN NMAP -sA (1)
>>> [Classification:
>>> Attempted Information Leak] [Priority: 2] {TCP} 74.125.133.189:443 -> 
>>> 10.2.2.202:52012 facility:LOCAL5
>>> ts:1550646678.442785 id.resp_h:172.16.4.18 
>>> 
>>> From Message Field, I want to extract Classification, Priority and TCP From 
>>> -> To IPs.
>>> 
>>> Can I make some kind of configurations in Bro Parser to get this 
>>> information Back As
>>> 
>>> Classification 
>>> Priority 
>>> TCP From 
>>> TCP To 
>>> 
>>> Any guidance will be great help.
>>> 
>>> 
>>> 
>>>  
>>> 
>>> --
>>> With Regards
>>> Farrukh Naveed Anjum
> 
> 
> -- 
> With Regards
> Farrukh Naveed Anjum

Re: Graphs based on Metron or PCAP data

[Simon Elliston Ball]

Graph enables a number of interesting use cases, and it really depends on what 
you’re after as to which tech makes sense. 

Spark graphx is a strong contender for analytics of things like betweenness and 
community linkage on HDFS indexed data. That would tend to be batch and through 
something like zeppelin. The very latest zeppelin also supports a network 
visualisation method which gives a graph like visual option.

For more interactive, streaming graph and alerting on graph an actual graph 
database makes more sense. I’ve seen some work done around Metron stacks with 
janusgraph, which leans on solr and Hbase so avoids adding too much complexity. 
Janus is not an apache project, but should be includable. At present I’ve only 
seen that used in Metron based distributions rather than Metron core.

Simon 


> On 2 Jan 2019, at 11:59, Otto Fowler  wrote:
> 
> Pieter,
> Can you create a jira with your use case?  It is important to capture.  We 
> have some outstanding jira’s around graph support.
> 
> 
>> On January 2, 2019 at 04:40:23, Stefan Kupstaitis-Dunkler 
>> (stefan@gmail.com) wrote:
>> 
>> Hi Pieter,
>> 
>>  
>> 
>> Happy new year!
>> 
>>  
>> 
>> I believe that always depends on a lot of factors and applies to any kind of 
>> visualization problem with big amounts of data:
>> 
>> How fast do you need the visualisations available?
>> How up-to-date do they need to be?
>> How complex?
>> How beautiful/custom modified?
>> How familiar are you with these frameworks? (could be a reason not to use a 
>> lib if they are otherwise equal in capabilities)
>>  
>> 
>> It sounds like you want to create a simple histogram across the full history 
>> of stored data. So I’ll throw in another option, that is commonly used for 
>> such use cases:
>> 
>> Zeppelin notebook:
>> Access data stored in HDFS via Hive.
>> A bit of preparation in Hive is required (and can be scheduled), e.g. 
>> creating external tables and converting data into a more efficient format, 
>> such as ORC.
>>  
>> 
>> Best,
>> 
>> Stefan
>> 
>>  
>> 
>> From: Pieter Baele 
>> Reply-To: "user@metron.apache.org" 
>> Date: Wednesday, 2. January 2019 at 07:50
>> To: "user@metron.apache.org" 
>> Subject: Graphs based on Metron or PCAP data
>> 
>>  
>> 
>> Hi,
>> 
>>  
>> 
>> (and good New Year to all as well!)
>> 
>>  
>> 
>> What would you consider as the easiest approach to create a Graph based 
>> primarly on ip_dst and ip_src adresses and the number (of connections) of 
>> those?
>> 
>>  
>> 
>> I was thinking:
>> 
>> - graph functionality in Elastic stack, but limited (ex only recent data in 
>> 1 index?)
>> 
>> - interfacing with Neo4J
>> 
>> - GraphX using Spark?
>> 
>> - using R on data stored in HDFS?
>> 
>> - using Python: plotly? Pandas?
>> 
>>  
>> 
>>  
>> 
>>  
>> 
>> Sincerely
>> 
>> Pieter

Re: Raw Message Strategy "Envelope"

[Simon Elliston Ball]

The envelope strategy controls how the parser views it's incoming data. In
other words, if the incoming data is json, should it treat one field as is
it were the original message, or treat the whole json as original.

To your example, the syslog parser would produce JSON, probably with a
bunch of syslog header fields and a field called something like "message".
(Note that these syslog wrappers may be useful or significant so you
probably don't want to just throw them away. This means:

A: (syslog wrapped) -> JSON (with message fields) -> Parser (P) for /This
is the .*message/
B: (raw) -> Parser (P) for /This is the .*message/

So you will need a syslog kafka topic, a syslog_parsed (json) kafka topic,
and a raw_message topic, along with two copies of P, one with envelope, one
without.

A better answer would be
A: (syslog wrapped) -> JSON (with message fields) -
B: (raw) ->  NoOp parser (e.g. Grok GREEDYDATA:message) which wraps into
metron JSON (with message fields)

now both those outputs can go into the same logical input topic for the
common envelope strategy parser:
Parser (P) for /This is the .*message/

It may seem counter intuitive to wrap the raw with an extra parser, but
this means you will end up with one Parser P set to anchor things like
enrichment and indexing config off further down the line, instead of two.

Simon




On Mon, 3 Dec 2018 at 18:32, Stefan Kupstaitis-Dunkler 
wrote:

> Hi,
>
> just out of interest: what is/should be the expected behaviour of the raw
> message strategy "ENVELOPE"?
>
>
>- Should a parser with this strategy only accept message that were
>already pre-processed by another parser?
>- Or should parser like this accept both? Direct ingests as well as
>ingests that are chained from a previous parser?
>
>
> Imagine you have 2 different log sources. One adds a syslog header the
> other doesn't.
>
> Example message from source 1: "<86>Dec 3 18:25:10 my.hostname.com This
> is the message"
> Example message from source 2: "This is the other message".
>
> Assumption is, that both "This is the message" and "This is the other
> message" can be parsed using the same pattern.
>
> Would I/Should I need to use 3 Kafka topics  (1 for the syslog parser, 1
> for the chained parser and another identical for the direct ingestion) or 2
> Kafka topics (1 for the syslog parser, 1 for both, the enveloped/chained
> source and the "default" source).
>
> Appreciate your thoughts and comments.
>
> Best,
> Stefan
> --
> Stefan Kupstaitis-Dunkler
> https://datahovel.com/
> https://www.meetup.com/Hadoop-User-Group-Vienna/
> https://twitter.com/StefanDunkler
>


-- 
--
simon elliston ball
@sireb

Re: Error deploying Metron 0.3.1 single Node

[Simon Elliston Ball]

Are you looking to install a dev build? If not and you just want to use the 
system, you may be better off with a pre-built distribution. 

Simon 

> On 30 Nov 2018, at 12:48, Babak Abbaschian  wrote:
> 
> It’s two weeks that I’m trying to install metron 0.6.1, but I end up with an 
> error with mpm failing to install some dependencies. 
> From the other side everything in Metron’s documentation is too old, CentOs 
> 6, Ubuntu 14, Ansible 2.0.0.2 etc. And at the same time we need NodeJs 
> 9(accompanied with NPM 5) with NPM 6(accompanies NodeJs 10).
> So I thought instead of this amount of patchwork try the one with all same 
> age components, but it failed as well. :(
> 
> 
> Sent from Yahoo Mail for iPhone
> 
> On Thursday, November 29, 2018, 10:09 PM, Laurens Vets  
> wrote:
> 
> I would suggest to try with a newer version (0.6.0), 0.3.1 is very old.
> 
>> On 2018-11-29 6:20 p.m., Babak Abbaschian wrote:
> 
> Followed this link: 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548
> With the following info:
> 
> Metron 0.3.1
> --
> * (detached from origin/Metron_0.3.1)
> --
> commit 7abd7e8a231c6cbe9ee4ab23a5df1e97344f5212
> Author: justinleet 
> Date:   Thu Feb 23 10:40:14 2017 -0500
> 
> METRON-734 Builds failing because of MaxMind DB transitive dependency 
> (justi
> --
> --
> ansible 2.0.0.2
>   config file = /etc/ansible/ansible.cfg
>   configured module search path = Default w/o overrides
> --
> Vagrant 2.2.0
> --
> Python 2.7.12
> --
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 
> 2015-11-10T11:41:47-05:00)
> Maven home: /usr/local/apache-maven/apache-maven-3.3.9
> Java version: 1.8.0_191, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-8-oracle/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "4.4.0-139-generic", arch: "amd64", family: "unix"
> --
> Linux upctv 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct 31 10:55:11 UTC 
> 2018 x86_64 x86_64 x86_64 GNU/Linux
> --
> Total System Memory = 15968.6 MB
> Processor Model: Intel(R) Core(TM) i5-6600K CPU @ 3.50GHz
> Processor Speed: 3899.902 MHz
> Total Physical Processors: 4
> Total cores: 16
> Disk information:
> /dev/sda3   269G   33G  223G  13% /
> /dev/sda1   659G  453G  206G  69% /media/ubuntu/DropBox
> **
> 
> And got this error: 
> 
> TASK [kibana : Install the Metron Dashboard] 
> ***
> fatal: [node1]: FAILED! => {"changed": true, "cmd": "elasticdump 
> --output=http://node1:9200/.kibana --input=/tmp/kibana-index.json", "delta": 
> "0:00:00.069235", "end": "2018-11-30 01:39:46.092021", "failed": true, "rc": 
> 1, "start": "2018-11-30 01:39:46.022786", "stderr": 
> "/usr/lib/node_modules/elasticdump/elasticdump.js:3\nconst {EventEmitter} = 
> require('events')\n  ^\n\nSyntaxError: Unexpected token {\nat 
> exports.runInThisContext (vm.js:53:16)\nat Module._compile 
> (module.js:373:25)\nat Object.Module._extensions..js (module.js:416:10)\n 
>at Module.load (module.js:343:32)\nat Function.Module._load 
> (module.js:300:12)\nat Module.require (module.js:353:17)\nat require 
> (internal/module.js:12:17)\nat Object. 
> (/usr/lib/node_modules/elasticdump/bin/elasticdump:6:19)\nat 
> Module._compile (module.js:409:26)\nat Object.Module._extensions..js 
> (module.js:416:10)", "stdout": "", "stdout_lines": [], "warnings": []}
> 
> 
> 
> 
> 
> 

Re: Issue with BasicIseParser

[Simon Elliston Ball]

Sounds like a perfect opportunity to contribute a fix, or a test case for
the broken log types. I would suggest raising a JIRA, and even a PR.

Simon

On Thu, 1 Nov 2018 at 14:35, Muhammed Irshad  wrote:

> Hi ,
>
> Seems string escaping is not handled in built in ISE parser. I am getting
> wired output for some of the log from cisco ise collected via splunk. The
> same issue is there for the test logs as well. PFA input string and output
> json. Same issue is there for the unit test case messages as well.
>
> --
> Muhammed Irshad K T
> Senior Software Engineer
> +919447946359
> irshadkt@gmail.com
> Skype : muhammed.irshad.k.t
>


-- 
--
simon elliston ball
@sireb

Re: https access to Metron Alert UI

[Simon Elliston Ball]

Metron doesn’t fully support this yet out of the box, but you can hack it up by 
changing the templates for the spring yaml config. 

More commonly, put it behind a reverse proxy for the ssl. There was talk about 
integrating that with Knox for ssl proxying, but that’s on pause now. 

Simon 

Sent from my iPhone

> On 30 Sep 2018, at 02:18, Charles Lo  wrote:
> 
> Hello,
> Is https access to Metron Alert UI (via port 4201) supported?
> If so, how do I configure it?
> Thank you.
>  

Re: Metron Not Reading From Kafka?

[Simon Elliston Ball]

It might be worth looking in the error and invalid topics, if you have any
validation, or your parser is not producing proper timestamps (that's what
I usually forget to check!) you may be getting messages routed to the error
index. Are you indexing topologies picking any of this up? Is there
anything in the ES error index?

The other thing worth doing is upgrading 0.4.3 is a very very old version
and a lot of changes have come in since then.

Simon

On 17 August 2018 at 16:07, David McGinnis 
wrote:

> All,
>
> We have a Metron 0.4.3 installation on an HDP cluster which has a sensor
> set up to read from a Kafka topic and write the data out to Elasticsearch.
> Data is being inserted into the Kafka topic, and we can read that through
> Kafka console consumer, but the system is not reporting any data coming
> through. The Storm spout says no data has been processed, and the index
> hasn't even been created in Elastic, despite running for nearly a month
> now.
>
> We've searched the worker logs for Storm, and the only error that comes up
> is a (we think) unrelated error about not being able to find the jmxmetrics
> JAR file. Metron reports that the topic is found, and does not tell us that
> the topic is not emitting, so we suspect it sees the data in there.
>
> Do you all have any ideas on where we can look to determine the cause of
> this issue, or things to try?
>
> Thanks!
>
> --
> David McGinnis
> Staff Hadoop Consultant | Avalon Consulting, LLC
> <http://www.avalonconsult.com/>M: (513) 439-0082
> LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc> | Google+
> <http://www.google.com/+AvalonConsultingLLC> | Twitter
> <https://twitter.com/avalonconsult>
> 
> -
> This message (including any attachments) contains confidential information
> intended for a specific individual and purpose, and is protected by law.
> If
> you are not the intended recipient, you should delete this message. Any
> disclosure, copying, or distribution of this message, or the taking of any
> action based on it, is strictly prohibited.
>



-- 
--
simon elliston ball
@sireb

Re: CEF Parser not Indexing data via Nifi (SysLogs)

[Simon Elliston Ball]

What you need to do is NOT ParseCEF in NiFi. Metron should handle be CEF 
parsing. 

Just use NiFi to do the listen syslog (no need to parse in NiFi) then SplitText 
to get one line of CEF per kafka message (if your syslog is batching, this may 
not be necessary. Set up a sensor in Metron using the CEF parser and you should 
be fine. 

Simon 


> On 20 Jul 2018, at 09:39, Srikanth Nagarajan  wrote:
> 
> Hi Farrukh,
> 
> You can try using the Grok Parser and search for regular expression pattern 
> for your log.  You can customize the regex to meet your needs.   
> 
> https://cwiki.apache.org/confluence/display/METRON/2016/04/25/Metron+Tutorial+-+Fundamentals+Part+1%3A+Creating+a+New+Telemetry
> 
> Look at Step-5 on how to create a regex for grok parser. Grok parser also 
> allows to validate the fields.
> 
> Good luck !
> 
> Thanks
> Srikanth
> 
>> On July 20, 2018 at 4:23 AM Farrukh Naveed Anjum  
>> wrote: 
>> 
>> Hi,
>> 
>> I am trying to index the Syslog using CEF Parser with Nifi.
>> 
>> It does not give any error though, transport data to kafa without indexing 
>> it. It keepg giving FAILED in Spout.
>> 
>> I believe indexing Syslog are most basic usecase for all. But metron fails 
>> to do it with each in standard format.
>> 
>> I tried bro for it. But even it keeps giving PARSER Error.
>> 
>> Any help ? Fast will be apperciated.
>> 
>> 
>> 
>> 
>> -- 
>> With Regards 
>> Farrukh Naveed Anjum
> 
> __
> 
> Srikanth Nagarajan 
> Principal
> 
> Gandiva Networks Inc
> 
> 732.690.1884 Mobile
> 
> s...@gandivanetworks.com
> 
> www.gandivanetworks.com
> 
> Please consider the environment before printing this. NOTICE: The information 
> contained in this e-mail message is intended for addressee(s) only. If you 
> have received this message in error please notify the sender.

Re: How to delete the original message field once the message parsed?

[Simon Elliston Ball]

Agreed. I think of the hdfs batch store as the throw away nothing store, and 
the lucene real-time store as more of an index or cache which does not have to 
be quite so complete, where we could definitely optimise down some of the 
fields.

Simon

> On 26 Jun 2018, at 04:57, Otto Fowler  wrote:
> 
> Also, theoretically, ‘not throwing anything away’ allows future 
> processing/reprocessing of data to gain new insights.  It is not uncommon 
> from the SEIM’s that I’ve seen to store the raw log information for the 
> reasons Simon states for example.
> 
> 
> So all these things that Simon and James have mentioned are true, and are the 
> why from a capabilities perspective.
> 
> That doesn’t invalidate your very practical point Michel however, and it is 
> important to understand field issues as people put Metron into use.  If these 
> features are not being used, or don’t exist yet (replay) can someone not tune
> them down for their scenario with some understanding of the tradeoffs?
> 
> I don’t think there is currently a way to do this, but it is worth having a 
> discussion on the issue.
> 
> 
>> On June 25, 2018 at 20:04:16, Simon Elliston Ball 
>> (si...@simonellistonball.com) wrote:
>> 
>> Very sorry... posted on the wrong thread...
>> 
>> The original string serves purposes well beyond debugging. Many users will 
>> need to be able to prove provenance to the raw logs in order to prove or 
>> prosecute an attack from an internal threat, or provide evidence to law 
>> enforcement or an external threat. As such, the original string is 
>> important. 
>> 
>> It also provides a valuable source for the free text search where parsing 
>> has not extracted all the necessary tokens for a hunt use case, so it can be 
>> a valuable field to have in Elastic or Solr for text rather than keyword 
>> indexing.
>> 
>> That said, it may make sense to remove a heavy weight processing and storage 
>> field like this from the lucene store. We have been talking for a while 
>> about filtering some of the data out of the realtime index, and preserving 
>> full copies in the batch index, which could meet the forensic use cases 
>> above, and would make it a matter of user choice. That would probably be 
>> configured through indexing config to filter fields.
>> 
>> Simon
>> 
>> 
>>> On 25 June 2018 at 23:49, Michel Sumbul  wrote:
>>> Hi James,
>>> 
>>> Will it not be interesting, to have an option to remove that field just 
>>> before indexing? This save storage space/Cost in HDFS and ES?
>>> For example, during development/debugging you keep that field and when 
>>> everything is ready for prod, you check a box to remove that field before 
>>> indexing?
>>> 
>>> Michel
>>> 
>>> 2018-06-25 23:37 GMT+01:00 James Sirota :
>>>> Hi Michael, the original_string is there for a reason. It's an immutable 
>>>> field that preserves the original message. While enrichments are added, 
>>>> various parts of the message are parsed out, changed, filtered out, 
>>>> ocncantenated, etc., you can always recover the original message from the 
>>>> original string.
>>>>  
>>>> Thanks,
>>>> James
>>>> 
>>>> 
>>>> 25.06.2018, 15:18, "Michel Sumbul" :
>>>>> Hello,
>>>>> 
>>>>> Is there a way to avoid to keep the field "original message", once the 
>>>>> message have been parsed?
>>>>> The objectif is to reduce the size of the message to store in HDFS, ES 
>>>>> and the traffic between storm/kafka. 
>>>>> Currently, we have all the fields + the original message which means that 
>>>>> we are going to used 2 time more space to store an information.
>>>>> 
>>>>> Thanks for the help,
>>>>> Michel
>>>> 
>>>> 
>>>> --- 
>>>> Thank you,
>>>>  
>>>> James Sirota
>>>> PMC- Apache Metron
>>>> jsirota AT apache DOT org
>>>> 
>>> 
>> 
>> 
>> 
>> --
>> --
>> simon elliston ball
>> @sireb

Re: Alerts Not Being Generated?

[Simon Elliston Ball]

Hi David,

One quick thing just in case, is_alert, not is_alarm. 

That said that should not affect what’s in the alerts ui. You should see data 
from your geo source as well (whatever you called it). It is possible there may 
be a problem with your elastic template. You might be interested in 
https://github.com/simonellistonball/metron-field-demos/blob/master/geo/es.json 

 which is based on the use case. Note that there is a field in there:  { alert: 
{ type: nested } } this is necessary for the Alerts UI and specifically the 
meta alerts capability. 

Note that you may also need to reload your alerts ui, and possibly restart the 
REST service to pickup new index types in the alerts ui, there may be issues 
with caching.

Simon


> On 1 Mar 2018, at 15:46, David McGinnis  wrote:
> 
> All,
> 
> I am following the instructions located here for creating a parser which 
> detects user logins distant from their recent logins, and raising alarms: 
> https://github.com/apache/metron/tree/master/use-cases/geographic_login_outliers
>  
> .
>  I have been able to successfully see the data show up in Kibana, including 
> the is_alarm field, which shows true when distant logins are reported, and 
> null or empty otherwise (I believe this is the correct behavior?).
> 
> The issue I'm having is that none of these distant logins are reported in the 
> Alarms UI. I have made the condition the same as the one I'm using for 
> is_alarm, and also used conditions that should always be true, but the only 
> alarms that show up are alarms from some sample Bro data that I can pass 
> through the system and see alerts for. 
> 
> Any ideas for how I can get alarms to show up correctly in the UI, or where 
> else I can check? I am not very familiar with the process of going from 
> enrichments to alerts UI at this point.
> 
> Thanks!
> 
> -- 
> David McGinnis
> Staff Hadoop Consultant | Avalon Consulting, LLC
>  M: (513) 439-0082
> LinkedIn  | Google+ 
>  | Twitter 
> 
> -
> This message (including any attachments) contains confidential information 
> intended for a specific individual and purpose, and is protected by law. If 
> you are not the intended recipient, you should delete this message. Any 
> disclosure, copying, or distribution of this message, or the taking of any 
> action based on it, is strictly prohibited.

Re: Best Metron version for development

[Simon Elliston Ball]

The full dev platform may be the easiest to test things like that on. It can be 
a little brittle if you’re running it in limited RAM, but it also has things 
like the sensor-stubs, which provides an easy means to fake up some input 
traffic. That may be useful for your development and testing. 

Simon

> On 15 Feb 2018, at 18:58, Helder Reia  wrote:
> 
> I want to implement some algorithms, in order to perform cluster and 
> classification over some data, that is why I asked if the better version is 
> the latest one or one of the older ones
> 
> Thank you,
> 
> 2018-02-15 18:53 GMT+00:00 Laurens Vets  >:
> I'm not sure I understand the question completely, but my guess would be the 
> latest release, i.e. 0.4.2?
> 
> On 2018-02-15 10:19, Helder Reia wrote:
> 
>> Hi, I am trying to build a intrusion detection system and I was thinking on 
>> using Apache Metron, but I have a question: which is the best version to 
>> development? I ask this because I will need to implement some 
>> cluster-classify algorithms. Also which guide should I take to install 
>> Apache Metron?
>>  
>> Thank you for your help
>> 
>>  
>> 
>>Sem vírus. www.avast.com 
>> 
> 
> 
> 
> -- 
> Helder Reia
> ALF-AL TM
> 
> 

Re: Error when trying to install Apache Metron CentOS7

[Simon Elliston Ball]

To be honest, rather than messing about with grub for this, I would follow the 
alternative route outlines in the wiki page.

To be even more honest, I wouldn’t use that method from the wiki and would 
probably go with something like the full dev VM platform if you’re looking to 
do development work. If you’re looking for production scale, a proper mpack 
based install on an existing Ambari managed cluster (see elsewhere for docs on 
how to set one of these up with a hadoop distribution) is probably better. 

Simon

> On 14 Feb 2018, at 13:07, Helder Reia  wrote:
> 
> Hi all,
> I am following the installation guide for installing Metron 0.4.1 with HDP 
> 2.5 on CentOS 7 and I am having trouble after I change this line and reboot:
> 
> # Change the line:
> GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv =cl/root 
> rd.lvm.lv =cl/swap rhgb quiet"
> # To:
> GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv =cl/root 
> rd.lvm.lv =cl/swap rhgb quiet transparent_hugepage=never"
> # Afterwards, run:
> grub2-mkconfig -o /boot/grub2/grub.cfg
> 
> Basically after I reboot, the OS won't start again, gives a fatal error
> 
> I am installing it on a VM with CentOS7, I don't know if that is the problem, 
> but I wanted to install it here first before I install on some nodes.
> 
> 
> Thank you all for the help,
> -- 
> Helder Reia
> ALF-AL TM
> 
> 

Re: Stellar post-parsing transformation conditional statement

[Simon Elliston Ball]

You either want a MAP_GET in your IF or a match statement in there I expect. 
See the match statement at 
https://github.com/apache/metron/blob/master/metron-stellar/stellar-common/README.md
 under core functions (it’s relatively new)

Simon 

Sent from my iPhone

> On 9 Feb 2018, at 03:55, Ali Nazemian  wrote:
> 
> Hi All,
> 
> I was wondering how we can address if statement in the config section to have 
> a different mapping in certain conditions. The following syntax is not 
> acceptable.
> 
> {
>   "parserClassName": "org.apache.metron.parsers.asa.BasicAsaParser",
>   "filterClassName": null,
>   "sensorTopic": "test-asa",
>   "writerClassName": null,
>   "errorWriterClassName": null,
>   "invalidWriterClassName": null,
>   "parserConfig": {},
>   "fieldTransformations": [
>   {
>   "input": [],
>   "output": [
>   "x",
>   "y",
>   "z",
>   ],
>   "transformation": "STELLAR",
>   "config": {
> IF "ip_src_port" == 39296 THEN
>   "x": "something",
>   "y": "something else"
> 
> ELSE
>   "y": "something",
>   "z": "something else"
>   }
>   }
>   ]
> }
> 
> Regards,
> Ali

Re: CentOS and Ubuntu

[Simon Elliston Ball]

Not particularly. The centos builds seem to be used by more people on dev, 
probably because they’ve been around for longer, and so are arguably more 
tested. The area where it’s most likely to be relevant is in the install of 
repos for ES and potentially the fastcapa pcap probe (don’t quote me on that 
though, I don’t know if anyone has run that on Ubuntu yet), but other than that 
they’re pretty similar these days. 

Simon

Sent from my iPhone

> On 7 Feb 2018, at 13:00, Helder Reia  wrote:
> 
> Hey everyone!
> I am new to Apache Metron and I don't know much about this! Are there any 
> differences on using CentOS or Ubuntu ? I am used to work with Ubuntu but I 
> can look for CentOS if it is easier to use / has advantages !
> 
> Thank you for your help!
> 
> -- 
> Helder Reia
> ALF-AL TM
> 
> 

Re: elasticsearch template question.

[Simon Elliston Ball]

Hi Laurens, 

In Metron all fields tend to get flattened into an un-nested structure of keys 
and values. Some of the keys do represent a flattened tree structure (for 
example our standard enrichment fields). The reason for this is essentially 
ingest speed for nested documents in lucene based indices like Elastic and 
Solr. So, we never used, nor need nested template, and tend to just use the ‘:’ 
separated fields to define the hierarchy.

Is there a particular use case you need the nesting for? 

Simon

> On 7 Feb 2018, at 01:26, Laurens Vets  wrote:
> 
> I hope there's an elasticsearch expert on the mailing list :D
> 
> I have a field called "responseElements:subnets" which can either contain:
> 
> {
>  "subnetIdentifier": "subnet-abcdefgh",
>  "subnetStatus": "Active",
>  "subnetAvailabilityZone": {
>"name": "us-west-2c"
>  }
> },
> {
>  "subnetIdentifier": "subnet-12345678",
>  "subnetStatus": "Active",
>  "subnetAvailabilityZone": {
>"name": "us-west-2b"
>  }
> }
> 
> or:
> 
> subnet-abcdefgh, subnet-12345678, subnet-a1b2c3d4
> 
> Any idea how I can map this in my template?
> 
> For the first case, I got:
> 
> "responseElements:subnets": {
>  "type": "nested",
>  "properties": {
>"subnetIdentifier": { "type": "string"  },
>"subnetStatus": { "type": "string"  },
>"subnetAvailabilityZone": {
>  "type": "nested",
>  "properties": {
>"name": { "type": "string" } } } } }
> 
> But how can I map the 2nd case?

Re: Define a function that can be used in Stellar

[Simon Elliston Ball]

I forgot we added OBJECT_GET. How does the caching work on that? 

Simn

> On 2 Feb 2018, at 14:33, Nick Allen <n...@nickallen.org> wrote:
> 
> There are many functions that use the global configuration.  For example, 
> GET_GEO in org.apache.metron.enrichment.stellar.GeoEnrichmentFunctions.  
> There might be a better example, but that is one is staring at me at the 
> moment.
>   
> There is an OBJECT_GET function defined in 
> org.apache.metron.enrichment.stellar.ObjectGet that was purpose-built to 
> retrieve files from HDFS.  If you wanted to retrieve a configuration from 
> HDFS that would be a good example (if you can't just use that functions 
> directly).
> 
> On Fri, Feb 2, 2018 at 8:50 AM Ali Nazemian <alinazem...@gmail.com 
> <mailto:alinazem...@gmail.com>> wrote:
> Is there any Stellar function already been implemented in Metron that has a 
> config file associated with it? I am trying to get an idea of how it works.
> 
> On 3 Feb. 2018 00:44, "Simon Elliston Ball" <si...@simonellistonball.com 
> <mailto:si...@simonellistonball.com>> wrote:
> Depends how you write the function class, but most likely, yes. Hence global 
> config option. 
> 
> Simon
> 
>> On 2 Feb 2018, at 13:42, Ali Nazemian <alinazem...@gmail.com 
>> <mailto:alinazem...@gmail.com>> wrote:
>> 
>> Does it mean every time the function gets called it will load the config, 
>> but if I use the global one it will only read it one time and it will be 
>> available in memory?
>> 
>> On 2 Feb. 2018 21:53, "Simon Elliston Ball" <si...@simonellistonball.com 
>> <mailto:si...@simonellistonball.com>> wrote:
>> Shouldn’t be. The one this I would point out though is that you don’t 
>> necessarily know which supervisor you will be running from, so pulling from 
>> HDFS would make sense. That said, the performance implications are probably 
>> not great. A good option here would be to have the config available in the 
>> global config for example and refer to that, since most instances of stellar 
>> apply global config to their context. 
>> 
>> Simon
>> 
>> 
>>> On 2 Feb 2018, at 07:14, Ali Nazemian <alinazem...@gmail.com 
>>> <mailto:alinazem...@gmail.com>> wrote:
>>> 
>>> Will be any problem if the Stellar function we want to implement need to 
>>> load an external config file?
>>> 
>>> Cheers,
>>> Ali
>>> 
>>> On Thu, Jan 18, 2018 at 4:58 PM, Ali Nazemian <alinazem...@gmail.com 
>>> <mailto:alinazem...@gmail.com>> wrote:
>>> Thanks, All.
>>> 
>>> Yes, Nick. It is highly related to our use case and the way that we are 
>>> going to enrich events with assets and vulnerability properties. It is not 
>>> a general case at all.
>>> 
>>> Cheers,
>>> Ali
>>> 
>>> On Thu, Jan 18, 2018 at 5:43 AM, Matt Foley <ma...@apache.org 
>>> <mailto:ma...@apache.org>> wrote:
>>> Besides the example code Simon mentioned at 
>>> https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example
>>>  
>>> <https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example>
>>>  ,
>>> there is some documentation at 
>>> http://metron.apache.org/current-book/metron-stellar/stellar-common/3rdPartyStellar.html
>>>  
>>> <http://metron.apache.org/current-book/metron-stellar/stellar-common/3rdPartyStellar.html>
>>>  
>>> 
>>> From: Nick Allen <n...@nickallen.org <mailto:n...@nickallen.org>>
>>> Reply-To: "user@metron.apache.org <mailto:user@metron.apache.org>" 
>>> <user@metron.apache.org <mailto:user@metron.apache.org>>
>>> Date: Wednesday, January 17, 2018 at 4:46 AM
>>> To: "user@metron.apache.org <mailto:user@metron.apache.org>" 
>>> <user@metron.apache.org <mailto:user@metron.apache.org>>
>>> Subject: Re: Define a function that can be used in Stellar
>>> 
>>>  
>>> 
>>>  
>>> 
>>>  
>>> 
>>> If something we have already does not fit the bill, I would recommend 
>>> creating that function in Java.   Since you described it as "a bit complex" 
>>> and "the logic would be complicated" I don't see any value in defining 
>>> something like this in Stellar with named functions.
>>> 
>>>  
>>> 
>>> Best
>>> 
>>>  
>>> 
>>&g

Re: Apache Metron functions implementation

[Simon Elliston Ball]

Hi Helder, 

It is very much possible, and very easy to create your own functions and models 
on top of Metron. 

There are two main ways in which you would do this, depending on the type of 
use case you’re looking at. 

Metron uses a language called Stellar as part of the enrichment stage (and 
elsewhere) to implement a number of algorithms which can then be composed in 
configuration. You can also extend this language to implement your own 
algorithms in the real time stream 
(https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example
 

 gives a toy example, also checkout some of the source for the more interesting 
stellar functions in 
https://github.com/apache/metron/tree/master/metron-analytics/metron-statistics 
.
 

If your algorithms tend more towards the traditional ML approach, using for 
example Spark, python, or R, then the Model as a Service extension points might 
be more useful. This allows you to run arbitrary micro-service type model 
inference, or scoring, and plug that into he Metron real-time stream 
(https://github.com/apache/metron/tree/master/metron-analytics/metron-maas-service
 

 provides more information and a worked example of how you would plug in an 
example python based model).  

I would also suggest taking a look at some of the recent custom use-cases we 
have included in the project to get some starters: 
https://github.com/apache/metron/tree/master/use-cases 
.

I hope that helps, and wish you the best of luck with your project. Also, do 
let the community know what you’re working on, and I’m sure we will be more 
than happy to provide any help and assistance we can. Looking forward to seeing 
what you come up with, and welcome to Metron. 

Simon

> On 2 Feb 2018, at 12:11, Helder Reia  wrote:
> 
> Hello,
> I am a student currently finishing my master degree and for my final work I 
> am proposing to make a security analytics tool. I will want to make it on 
> Apache Metron framework but I have some questions:
> - Is it possible to implement my own functions ? ( I will want to have 
> clustering and classification algorithms )
> - If so, can you give me helpon how to implement those algorithms?
> 
> Thank you for the help !
> 
> -- 
> Helder Reia
> ALF-AL TM
> 
> 

Re: Define a function that can be used in Stellar

[Simon Elliston Ball]

Shouldn’t be. The one this I would point out though is that you don’t 
necessarily know which supervisor you will be running from, so pulling from 
HDFS would make sense. That said, the performance implications are probably not 
great. A good option here would be to have the config available in the global 
config for example and refer to that, since most instances of stellar apply 
global config to their context. 

Simon


> On 2 Feb 2018, at 07:14, Ali Nazemian <alinazem...@gmail.com> wrote:
> 
> Will be any problem if the Stellar function we want to implement need to load 
> an external config file?
> 
> Cheers,
> Ali
> 
> On Thu, Jan 18, 2018 at 4:58 PM, Ali Nazemian <alinazem...@gmail.com 
> <mailto:alinazem...@gmail.com>> wrote:
> Thanks, All.
> 
> Yes, Nick. It is highly related to our use case and the way that we are going 
> to enrich events with assets and vulnerability properties. It is not a 
> general case at all.
> 
> Cheers,
> Ali
> 
> On Thu, Jan 18, 2018 at 5:43 AM, Matt Foley <ma...@apache.org 
> <mailto:ma...@apache.org>> wrote:
> Besides the example code Simon mentioned at 
> https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example
>  
> <https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example>
>  ,
> there is some documentation at 
> http://metron.apache.org/current-book/metron-stellar/stellar-common/3rdPartyStellar.html
>  
> <http://metron.apache.org/current-book/metron-stellar/stellar-common/3rdPartyStellar.html>
>  
> 
> From: Nick Allen <n...@nickallen.org <mailto:n...@nickallen.org>>
> Reply-To: "user@metron.apache.org <mailto:user@metron.apache.org>" 
> <user@metron.apache.org <mailto:user@metron.apache.org>>
> Date: Wednesday, January 17, 2018 at 4:46 AM
> To: "user@metron.apache.org <mailto:user@metron.apache.org>" 
> <user@metron.apache.org <mailto:user@metron.apache.org>>
> Subject: Re: Define a function that can be used in Stellar
> 
>  
> 
>  
> 
>  
> 
> If something we have already does not fit the bill, I would recommend 
> creating that function in Java.   Since you described it as "a bit complex" 
> and "the logic would be complicated" I don't see any value in defining 
> something like this in Stellar with named functions.
> 
>  
> 
> Best
> 
>  
> 
>  
> 
>  
> 
>  
> 
> On Wed, Jan 17, 2018 at 7:38 AM Simon Elliston Ball 
> <si...@simonellistonball.com <mailto:si...@simonellistonball.com>> wrote:
> 
> Have you looked at the recent TLSH functions in Stellar? We already have that 
> for similarity preserving hashes.
> 
>  
> 
> Simon
> 
>  
> 
> 
> On 17 Jan 2018, at 12:35, Ali Nazemian <alinazem...@gmail.com 
> <mailto:alinazem...@gmail.com>> wrote:
> 
> It is a bit complex. We want to create a function that accepts a list of 
> arguments for an asset and generate an asset identifier that can be used as a 
> row_key for the enrichment store. The logic would be complicated, though. We 
> may need to include some sort of similarity aware hash function as a part of 
> this custom function.
> 
>  
> 
> On Wed, Jan 17, 2018 at 10:32 PM, Nick Allen <n...@nickallen.org 
> <mailto:n...@nickallen.org>> wrote:
> 
> Ali - Can you describe the logic that you are trying to perform? That would 
> be useful as a use case to help drive a discussion around creating named 
> functions in Stellar.
> 
>  
> 
>  
> 
>  
> 
>  
> 
> On Wed, Jan 17, 2018 at 6:29 AM Ali Nazemian <alinazem...@gmail.com 
> <mailto:alinazem...@gmail.com>> wrote:
> 
> Thanks, Simon. We have already got a script to deal with classpath management 
> for the parsers. We should be able to use it for this extension as well.
> 
>  
> 
> Yeah, I agree. It will be much easier to define functions on the fly and use 
> them afterwards. It could be defined as Lambda or custom function. 
> 
>  
> 
> Regards,
> 
> Ali
> 
>  
> 
>  
> 
>  
> 
> On Wed, Jan 17, 2018 at 9:42 PM, Simon Elliston Ball 
> <si...@simonellistonball.com <mailto:si...@simonellistonball.com>> wrote:
> 
> https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example
>  
> <https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example>
>  gives good details on how to add a stellar function.
> 
>  
> 
> Stellar will pick up an annotated function on its class path, so to add 
> function there is no need to rebuild metron module, but you do need y

Re: HBase enrichment vs Stellar enrichment for HBase look up

[Simon Elliston Ball]

There shouldn’t be. Both run through the same kind of bolt-side caching, so you 
should be able to use the Stellar version, and in fact that’s the general 
direction the project is heading. We haven’t quite deprecated the plain HBase 
Bolt… but Stellar is definitely the preferred option. 

Simon

> On 2 Feb 2018, at 07:10, Ali Nazemian  wrote:
> 
> Hi All,
> 
> Is there any performance difference between HBase enrichment and Stellar 
> enrichment? We have an HBase enrichment that we need to have a customised key 
> for it. HBase enrichment doesn't give us the full flexibility of using any 
> logic for a Key generation, so I was wondering whether there will be any 
> performance difference if we try to proceed with Stellar enrichment and look 
> up HBase based on our logic or not?
> 
> Regards,
> Ali

Re: Indexing Bolt Error

[Simon Elliston Ball]

Yes, configure your indexing. 
https://metron.apache.org/current-book/metron-platform/metron-indexing/index.html
 


Note it’s a warning, not an error, that default values are being used because 
you do not have a specific configuration entry for the snort indexing.

Simon

> On 24 Jan 2018, at 08:31, Farrukh Naveed Anjum  
> wrote:
> 
> Any Idea how to fix this up ?
> 
> On Wed, Jan 24, 2018 at 1:27 PM, Farrukh Naveed Anjum 
> > wrote:
> Hi,
> 
> I am getting this error while starting up squid parsing again. Upon 
> restarting Apache Strom Indexing Bolt is showing up this error any idea how 
> can I fix this ?
> 
> java.lang.Exception: WARNING: Default and (likely) unoptimized writer config 
> used for hdfs writer and sensor snort
>   at 
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:234)
>  [stormjar.jar:?]
>   at 
> org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) 
> [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> 
> -- 
> With Regards
> Farrukh Naveed Anjum
> 
> 
> 
> -- 
> With Regards
> Farrukh Naveed Anjum

Re: Some Metron Alerts UI questions

[Simon Elliston Ball]

Hi Laurens, 

A few quick answers inline…

Simon

> On 20 Jan 2018, at 00:37, Laurens Vets  wrote:
> 
> Hi list,
> 
> I have some general Alerts UI questions/comments/remarks, I hope you don't 
> mind :) I'm using the UI that's part of Metron 0.4.2. These apply to my 
> specific use case, so I might be completely wrong in how I use the UI…

Comment and feedback are always welcome!

> 
> - When you're talking about 'alerts', from what I can see in the UI, that's 
> synonymous with just events in elasticsearch right? Wouldn't it make more 
> sense to treat alerts as events where "is_alert" == True?
> 

At present the search does not exclude non-alerts… it’s maybe a little odd to 
call it the alerts view right now, but right now it’s the only way to see 
everything, so this should probably separate out into an ‘everything’ hunting 
focused view and a alerts only view.

The reasons I kinda like the current approach is that it’s good for picking up 
things that have become alerts because they’re in threat intel for example, 
along with things clustered against them by something like the new TLSH 
functions, which makes it easier to combine known alerts with un-detected 
events in a meta alert.

> - It seems that everything I do in the UI is only stored locally? See 
> https://github.com/apache/metron/tree/master/metron-interface/metron-alerts. 
> Can this made persistent for multiple people?

Yep. A lot of the preferences, saved searched, column layouts etc, are stored 
in local storage by the browser right now. We need a REST endpoint and to 
figure out how to store them (against user / against a group / global??? 
thoughts?) server side. A lot of the mechanism to do that is in, it’s just not 
quite done done because of those open questions I expect. 

> 
> - How can I change the content "Filters" on the left of the UI?

You wait for https://github.com/apache/metron/pull/853 
 to land. 

> 
> - How do I create a MetaAlert?

You can create a meta-alert from a grouped set of alerts, use the grouping 
buttons at the top and you’ll find a merge alert. Slightly odd process at the 
moment true, but a button to create a meta-alert from all the selected, or all 
the visible alerts on the results page might be a good addition, what do you 
think?

Very quick video of the current method here: https://youtu.be/JkFeNKTOd38

> 
> - What's the plan regarding notifying someone when alerts triggers?

Currently there is no external notification, but the answer here would likely 
be to consume the indexing topic in kafka and integrate to an enterprise alarm 
or monitoring system (alerting and alarms is a massive topic which probably 
deserves its own project beyond metron and I’ve seen people use all sorts of 
things for this, usually some big enterprisey thing mandated by IT).

Re: SysLog using CEF Parser (RSysLogs)

[Simon Elliston Ball]

Are there any errors in the logs for the indexing bolt? I would expect the 
errors are probably at the elastic ingest point, and probably caused by an 
incorrect elastic template for the CEF data. 

Simon

> On 22 Jan 2018, at 08:24, Farrukh Naveed Anjum  
> wrote:
> 
> Yes its Strom Indexing Bolt that is halting it. Any one working on CEF Parser 
> (Can Syslog work with it like RSyslog). We are stuck at that point.
> 
> Please see the above error and suggest
> 
> On Mon, Jan 22, 2018 at 1:10 PM, Gaurav Bapat  > wrote:
> Hi,
> 
> Even I am stuck with the same, and dont know how to solve the issue.
> 
> Looks like this is a parsing error
> 
> On 22 January 2018 at 13:00, Farrukh Naveed Anjum  > wrote:
> Hi,
> 
> I am trying to Ingest syslog using CEF Parser it is not creating any Elastic 
> Search Index based on. 
> 
> Any suggestion how can I achieve it ?
> 
> 
> 
> 
> -- 
> With Regards
> Farrukh Naveed Anjum
> 
> 
> 
> 
> -- 
> With Regards
> Farrukh Naveed Anjum

Re: Define a function that can be used in Stellar

[Simon Elliston Ball]

Have you looked at the recent TLSH functions in Stellar? We already have that 
for similarity preserving hashes.

Simon

> On 17 Jan 2018, at 12:35, Ali Nazemian <alinazem...@gmail.com> wrote:
> 
> It is a bit complex. We want to create a function that accepts a list of 
> arguments for an asset and generate an asset identifier that can be used as a 
> row_key for the enrichment store. The logic would be complicated, though. We 
> may need to include some sort of similarity aware hash function as a part of 
> this custom function.
> 
>> On Wed, Jan 17, 2018 at 10:32 PM, Nick Allen <n...@nickallen.org> wrote:
>> Ali - Can you describe the logic that you are trying to perform? That would 
>> be useful as a use case to help drive a discussion around creating named 
>> functions in Stellar.
>> 
>> 
>> 
>> 
>>> On Wed, Jan 17, 2018 at 6:29 AM Ali Nazemian <alinazem...@gmail.com> wrote:
>>> Thanks, Simon. We have already got a script to deal with classpath 
>>> management for the parsers. We should be able to use it for this extension 
>>> as well.
>>> 
>>> Yeah, I agree. It will be much easier to define functions on the fly and 
>>> use them afterwards. It could be defined as Lambda or custom function. 
>>> 
>>> Regards,
>>> Ali
>>> 
>>> 
>>> 
>>>> On Wed, Jan 17, 2018 at 9:42 PM, Simon Elliston Ball 
>>>> <si...@simonellistonball.com> wrote:
>>>> https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example
>>>>  gives good details on how to add a stellar function.
>>>> 
>>>> Stellar will pick up an annotated function on its class path, so to add 
>>>> function there is no need to rebuild metron module, but you do need your 
>>>> modules on the classpath, and, pending 777, to deal with things like class 
>>>> path clash in your dependencies. 
>>>> 
>>>> Another idea worth discussion on the dev list is probably the notion of 
>>>> defining stellar functions in stellar, which would be a much simpler 
>>>> solution than custom java functions if you can already express you logic 
>>>> in stellar. 
>>>> 
>>>> Simon
>>>> 
>>>> 
>>>>> On 17 Jan 2018, at 10:37, Ali Nazemian <alinazem...@gmail.com> wrote:
>>>>> 
>>>>> Hi Simon,
>>>>> 
>>>>> Yes, that is exactly what we are looking for. Is there any example 
>>>>> regarding adding a Stellar function in Java? Hopefully, we don't need to 
>>>>> rebuild the corresponding modules for this?
>>>>> 
>>>>> Cheers,
>>>>> Ali
>>>>> 
>>>>>> On Wed, Jan 17, 2018 at 8:40 PM, Simon Elliston Ball 
>>>>>> <si...@simonellistonball.com> wrote:
>>>>>> At present you can certainly create custom stellar functions in Java. 
>>>>>> I’m guessing however that what you’re looking to do is create a kind of 
>>>>>> function that combines a number of stellar functions to avoid 
>>>>>> repetition, or to ensure consistency of certain parameters for example. 
>>>>>> Is that what you’re looking for? Maybe some sort of syntax to create a 
>>>>>> named stellar function similar to the way we create lambdas?
>>>>>> 
>>>>>> Simon
>>>>>> 
>>>>>> > On 17 Jan 2018, at 07:25, Ali Nazemian <alinazem...@gmail.com> wrote:
>>>>>> >
>>>>>> > Hi all,
>>>>>> >
>>>>>> > Is there any way that we can define a function that can be used rather 
>>>>>> > than duplicating a logic multiple times?
>>>>>> >
>>>>>> > Cheers,
>>>>>> > Ali
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> A.Nazemian
>>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> A.Nazemian
> 
> 
> 
> -- 
> A.Nazemian

Re: Define a function that can be used in Stellar

[Simon Elliston Ball]

https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example
 
<https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example>
 gives good details on how to add a stellar function.

Stellar will pick up an annotated function on its class path, so to add 
function there is no need to rebuild metron module, but you do need your 
modules on the classpath, and, pending 777, to deal with things like class path 
clash in your dependencies. 

Another idea worth discussion on the dev list is probably the notion of 
defining stellar functions in stellar, which would be a much simpler solution 
than custom java functions if you can already express you logic in stellar. 

Simon

> On 17 Jan 2018, at 10:37, Ali Nazemian <alinazem...@gmail.com> wrote:
> 
> Hi Simon,
> 
> Yes, that is exactly what we are looking for. Is there any example regarding 
> adding a Stellar function in Java? Hopefully, we don't need to rebuild the 
> corresponding modules for this?
> 
> Cheers,
> Ali
> 
> On Wed, Jan 17, 2018 at 8:40 PM, Simon Elliston Ball 
> <si...@simonellistonball.com <mailto:si...@simonellistonball.com>> wrote:
> At present you can certainly create custom stellar functions in Java. I’m 
> guessing however that what you’re looking to do is create a kind of function 
> that combines a number of stellar functions to avoid repetition, or to ensure 
> consistency of certain parameters for example. Is that what you’re looking 
> for? Maybe some sort of syntax to create a named stellar function similar to 
> the way we create lambdas?
> 
> Simon
> 
> > On 17 Jan 2018, at 07:25, Ali Nazemian <alinazem...@gmail.com 
> > <mailto:alinazem...@gmail.com>> wrote:
> >
> > Hi all,
> >
> > Is there any way that we can define a function that can be used rather than 
> > duplicating a logic multiple times?
> >
> > Cheers,
> > Ali
> 
> 
> 
> 
> -- 
> A.Nazemian

Re: Metron Reference Application (Profiling Your Streams Fails)

[Simon Elliston Ball]

Looks like a docs typo on the wiki: 

What you need is CONFIG_PUT(“PROFILER”, profilerConfig)

Simon

> On 15 Jan 2018, at 10:45, Farrukh Naveed Anjum  
> wrote:
> 
> Can you help on this ?
> 
> On Mon, Jan 15, 2018 at 3:42 PM, Farrukh Naveed Anjum 
>  wrote:
> Any Idea of getting ride of this problem ?
> 
> On Mon, Jan 15, 2018 at 3:38 PM, Farrukh Naveed Anjum 
>  wrote:
> Hi,
> 
> I am trying to setup up Metron Reference Application (SQUID) Example given on 
> main metron website. 
> 
> I am facing a problem during (Profiling Your Streams ) Config_Put Step
> 
> 
> CONFIG_PUT(profilerConfig)
> 
> It causes to display following error
> 
> [!] No enum constant 
> org.apache.metron.common.configuration.ConfigurationType.{
>   "profiles": [
> {
>   "profile": "squid-miss",
>   "foreach": "ip_src_addr",
>   "onlyif": "source.type == 'squid' and action == 'TCP_MISS'",
>   "update": {
> "m": "STATS_ADD(m, 1)"
>   },
>   "result": "m"
> },
> {
>   "profile": "url-length",
>   "foreach": "ip_src_addr",
>   "onlyif": "source.type == 'squid'",
>   "update": {
> "m": "STATS_ADD(m, LENGTH(url))"
>   },
>   "result": "m"
> }
>   ]
> }
> java.lang.IllegalArgumentException: No enum constant 
> org.apache.metron.common.configuration.ConfigurationType.{
>   "profiles": [
> {
>   "profile": "squid-miss",
>   "foreach": "ip_src_addr",
>   "onlyif": "source.type == 'squid' and action == 'TCP_MISS'",
>   "update": {
> "m": "STATS_ADD(m, 1)"
>   },
>   "result": "m"
> },
> {
>   "profile": "url-length",
>   "foreach": "ip_src_addr",
>   "onlyif": "source.type == 'squid'",
>   "update": {
> "m": "STATS_ADD(m, LENGTH(url))"
>   },
>   "result": "m"
> }
>   ]
> }
> at java.lang.Enum.valueOf(Enum.java:238)
> at 
> org.apache.metron.common.configuration.ConfigurationType.valueOf(ConfigurationType.java:31)
> at 
> org.apache.metron.management.ConfigurationFunctions$ConfigPut.apply(ConfigurationFunctions.java:269)
> at 
> org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:556)
> at 
> org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:160)
> at 
> org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:152)
> at 
> org.apache.metron.stellar.common.shell.StellarExecutor.execute(StellarExecutor.java:287)
> at 
> org.apache.metron.stellar.common.shell.StellarShell.handleStellar(StellarShell.java:270)
> at 
> org.apache.metron.stellar.common.shell.StellarShell.execute(StellarShell.java:409)
> at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 
> 
> How can I solve this problem
> 
> -- 
> With Regards
> Farrukh Naveed Anjum
> 
> 
> 
> -- 
> With Regards
> Farrukh Naveed Anjum
> 
> 
> 
> -- 
> With Regards
> Farrukh Naveed Anjum

Re: Metron Rest Kerberos -- Kafka topic ACL

[Simon Elliston Ball]

The ansible roles and playbooks included with Metron install Ambari to handle 
the setup of the Metron and the Hadoop, Kafka etc. components, so yes. 

> On 10 Jan 2018, at 03:18, varsha mordi  wrote:
> 
> Can Ambari UI work with Ansible?
> 
> On Wed, Jan 10, 2018 at 3:46 PM, Mohan Venkateshaiah 
> > 
> wrote:
> Srikanth,
> 
>  
> 
> There is no way you can list all topics to particular user, there is PR for 
> adding REST endpoints to provide required ACL to topic.
> 
>  
> 
> Thanks
> 
> Mohan DV
> 
>  
> 
> From: prakash r >
> Reply-To: "user@metron.apache.org " 
> >
> Date: Wednesday, January 10, 2018 at 7:50 AM
> To: "user@metron.apache.org " 
> >
> Subject: Metron Rest Kerberos -- Kafka topic ACL
> 
>  
> 
> Hello,
> 
>  
> 
> We have kerberosed Hadoop Cluster.
> 
>  
> 
> Metron is trying to access all the Kafka topics (ir-respective of Kafka 
> topics which needed for Metron)
> 
>  
> 
> Since it does not have access to all topics, in UI Kafka related infos are 
> not displayed.
> 
>  
> 
> For Ex :
> 
>  
> 
> Kafka has some other topics like (checking123) Metron need authorization for 
> those topic as well.
> 
>  
> 
> 2018-01-10T11:17:39.576 DEBUG 
> [org.springframework.web.servlet.mvc.method.annotation.HttpEntityMethodProcessor]
>  - Written [{timestamp=Wed Jan 10 11:17:39 AEDT 2018, status=500, 
> error=Internal Server Error, 
> exception=org.apache.kafka.common.errors.TopicAuthorizationException, 
> message=Not authorized to access topics: [checking123], 
> path=/api/v1/kafka/topic/snort}] as "application/json" using 
> [org.springframework.http.converter.json.MappingJackson2HttpMessageConverter@ab327c]
> 
>  
> 
>  
> 
> Can Metron Rest restrict access only to those topics which is needed for the 
> same, thanks
> 
> 
> Regards,
> 
> Prakash R
> 
> 
> 
> 
> -- 
> Thanks & Regards,
> Varsha Mordi
> Prodevans Technologies LLP.
> M: +91 9637109734  | L: +91 80 64533365 | www.prodevans.com 
> 
> 

Re: Metron Version

[Simon Elliston Ball]

Are the logs you’re sending with syslog in CEF format? You will note that the 
CEF sensor uses the CEF parser, which means unless your logs are in CEF format, 
they will fail to parse and be dropped into the error index (worth checking the 
error index in kibana via the Metron Error Dashboard. That will likely tell you 
why things aren’t parsing. 

The most likely scenario is that you are sending something non-CEF on the 
syslog feed, in which case you will need something like a Grok parser. I 
suggest reading through the Squid example in the documentation on how to do 
this. 

Simon

> On 4 Jan 2018, at 18:49, Gaurav Bapat  wrote:
> 
> They are syslogs and my topic name is cef, I get one parsed logs out of 1000+ 
> and I want to do analytics using Spark but I cant find a way out.

Re: metron vs ossec

[Simon Elliston Ball]

In many ways it’s a matter of scale. OSSIM is a kind of lite version of 
AlienVault, and used by them. I’ve seen people move from an OSSIM architecture 
to Metron specifically to get better scaling, things like PCAP capabilities 
etc. but also retain the OSSEC agents to handle endpoint and scanning use 
cases, which they then feed into Metron. In these cases it was mostly about 
scalability and flexibility to extend, as well as manageability of multi-tenant 
environments. 

In functional terms, Metron also emphasises behaviour profiling and machine 
learning, whereas OSSIM is a more traditional rules-centric way of looking at 
security and log monitoring. 

Hope that helps you understand the difference a little better,
Simon

> On 21 Dec 2017, at 12:22, moshe jarusalem  wrote:
> 
> Jon thanks for the information.
> 
> I am indeed trying to learn both of them just wanted to get expert ideas. 
> 
> OSSEC is also supported by OSSIM which is somewhat like metron. I  would like 
> to hear  ideas which may make metron better alternative and or composite 
> usage.
> 
> Regards,
> 
> 
> On Thu, Dec 21, 2017 at 2:39 PM, zeo...@gmail.com  > wrote:
> Yes, I run both in my environment and they are both security products but 
> that's about where the similarities end.  Ossec is a host based solution that 
> monitors local activity with it's tree based rules engine, Metron is a 
> distributed solution that handles large sets of data from many sources and a 
> lot more.  A possible connection between the two may be that ossec 
> logs/alerts could be fed into Metron for enrichment, triage, alerting, and 
> analysis.
> 
> I would recommend either reading the documentation for both of them in more 
> detail, or spinning them both up to get a better handle on the differences.
> 
> Jon
> 
> 
> On Thu, Dec 21, 2017, 00:34 moshe jarusalem  > wrote:
> Hi All, 
> I have come across OSSEC project and find it similar to metron. I am confused 
> a bit. 
> is anyone aware of Ossec and give some comparisons?
> 
> Regards,
> -- 
> Jon
> 
> 

Re: machine learning libraries supported

[Simon Elliston Ball]

Spark’s ML models are primarily batch in their nature. There is talk about 
incorporating things like naive bayes and streaming kmeans to structured 
streaming (which will require some schema work in metron to make sense). These 
are still open issues not seeing a lot of progress in the spark community. 

The most common mistake I’ve seen using spark streaming with ML in the cyber 
world is people thinking that the FP Growth association rules models can be 
online-learnt, because there exists a class of streaming FP Growth models. The 
Spark implementations of FP Growth however, rely on Batch (mathematically!) and 
while technically can be run on the micro-batches Spark streaming provides, are 
not actually meaningful. Just because your model runs and gives you an output, 
doesn’t mean it’s mathematically defensible to do so. 

All that said...

Streaming inference makes some sense in spark, but that’s probably better 
handled through MaaS in Metron, which will generalise to spark and other 
libraries, and absolutely, use the spark models and the ML pipelining to 
perform inference in a spark job run with parallel instances in MaaS. Note that 
the reason for this is primality that Spark is a data parallel engine, where as 
Metron MaaS applies task parallelism, in order to reduce latency. 

To the point of a good example of python / spark / MaaS / Metron, I would 
recommend taking a look at Casey’s blog at 
https://hortonworks.com/blog/model-service-modern-streaming-data-science-apache-metron/
 
<https://hortonworks.com/blog/model-service-modern-streaming-data-science-apache-metron/>
 which is a walk though on score ad python scikit-learn model in MaaS. For the 
spark piece, I’ve seen a number of examples based on these same principals, 
using the spark classes for scoring based on saved models produced by a batch 
trainer. Apologies, I don’t have any readily publishable examples of the whole 
thing, but may work something synthetic up if it would be useful. 

Simon

> On 7 Dec 2017, at 13:09, Martin Andreoni <mar...@gta.ufrj.br> wrote:
> 
> Hello Simon,
> 
> thanks for the information.
> 
> However, why do u affirm that the streaming models are not well suited?
> 
>> You could as some have suggested use spark streaming, but to be honest, the 
>> spark ML models are not well suited to streaming use cases
> Is there a performance problem or how would you justify that phrase? 
> 
> thanks
> 
> Le 07/12/2017 à 13:55, Simon Elliston Ball a écrit :
>> I would recommend starting out with something like Spark, but the short 
>> answer is that anything that will run inside a yarn container, so the answer 
>> is most ML libraries. 
>> 
>> Using Spark to train models on the historical store is a good bet, and then 
>> using the trained models with model as a service.
>> 
>> See 
>> https://github.com/apache/metron/tree/master/metron-analytics/metron-maas-service
>>  
>> <https://github.com/apache/metron/tree/master/metron-analytics/metron-maas-service>
>>  for information on models and some sample boilerplate for deploying your 
>> own python based models. 
>> 
>> You could as some have suggested use spark streaming, but to be honest, the 
>> spark ML models are not well suited to streaming use cases, and you would be 
>> very much breaking the metron flow rather than benefitting from elements 
>> like MaaS (you’d basically be building a 100% custom side project, which 
>> would be fine, but you’re missing a lot of the benefits of Metron that way). 
>> If you do go down that route I would strong recommend having the output of 
>> your streaming jobs feed back into a Metron sensor. To be honest though, 
>> you’re much better off training in batch and scoring / inferring via the 
>> Model as a Service approach. 
>> 
>> Simon
>> 
>> 
>>> On 6 Dec 2017, at 07:45, moshe jarusalem <tuu...@gmail.com 
>>> <mailto:tuu...@gmail.com>> wrote:
>>> 
>>> Hi All,
>>> Would you please suggest some documentation about machine learning 
>>> libraries can be used in metron architecture? and how ? any examples 
>>> appretiated.
>>> 
>>> regards,
>>> 
>> 
> 
> -- 
> Martin Andreoni
> PhD. Candidate at GTA/LIP6
> 
> UFRJ/UPMC
> 
> www.gta.ufrj.br/~martin <http://www.gta.ufrj.br/%7Emartin>

Re: machine learning libraries supported

[Simon Elliston Ball]

I would recommend starting out with something like Spark, but the short answer 
is that anything that will run inside a yarn container, so the answer is most 
ML libraries. 

Using Spark to train models on the historical store is a good bet, and then 
using the trained models with model as a service.

See 
https://github.com/apache/metron/tree/master/metron-analytics/metron-maas-service
 

 for information on models and some sample boilerplate for deploying your own 
python based models. 

You could as some have suggested use spark streaming, but to be honest, the 
spark ML models are not well suited to streaming use cases, and you would be 
very much breaking the metron flow rather than benefitting from elements like 
MaaS (you’d basically be building a 100% custom side project, which would be 
fine, but you’re missing a lot of the benefits of Metron that way). If you do 
go down that route I would strong recommend having the output of your streaming 
jobs feed back into a Metron sensor. To be honest though, you’re much better 
off training in batch and scoring / inferring via the Model as a Service 
approach. 

Simon


> On 6 Dec 2017, at 07:45, moshe jarusalem  wrote:
> 
> Hi All,
> Would you please suggest some documentation about machine learning libraries 
> can be used in metron architecture? and how ? any examples appretiated.
> 
> regards,
> 

Re: Basic analysis

[Simon Elliston Ball]

Agreed… for the users list I would just say use the Install Notebooks action, 
and look at the squid example on the wiki, but since it was you who asked for 
links, Otto, I went a bit dev list ;)

Simon

> On 6 Dec 2017, at 14:33, Otto Fowler <ottobackwa...@gmail.com> wrote:
> 
> The issue is the requirement for people on the user list to go to the source.
> 
> 
> On December 6, 2017 at 09:16:39, Simon Elliston Ball 
> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote:
> 
>> No problem, I’ll grant you it’s not in the most intuitive part of the source 
>> tree to go digging in, but you can also get to the zeppelin bits via the 
>> actions button on the Metron config section (Install Notebooks)
>> 
>> If anyone has any good ideas (or code!) for sample zeppelin notebooks that 
>> would be useful, you can add them to a specific instance of the platform via 
>> the config/zeppelin/metron location and run the action again I believe, and 
>> this would be a great place for more security people to contribute sample 
>> run books for example. There are also efforts by commercial support 
>> providers I believe to add more samples of both dashboards and use cases.
>> 
>> Simon
>> 
>>> On 6 Dec 2017, at 14:12, Otto Fowler <ottobackwa...@gmail.com 
>>> <mailto:ottobackwa...@gmail.com>> wrote:
>>> 
>>> Thanks Simon
>>> 
>>> 
>>> On December 6, 2017 at 09:11:50, Simon Elliston Ball 
>>> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote:
>>> 
>>>> In product… Install Zeppelin Notebooks, and the samples including 
>>>> notebooks at 
>>>> https://github.com/apache/metron/tree/master/metron-platform/metron-indexing/src/main/config/zeppelin/metron
>>>>  
>>>> <https://github.com/apache/metron/tree/master/metron-platform/metron-indexing/src/main/config/zeppelin/metron>
>>>> 
>>>> as of course there are similar Kibana dashboards included, which are 
>>>> examples of custom visualisation of metron data, there is also the run 
>>>> book for visualising squid data in kibana on the docs wiki 
>>>> https://cwiki.apache.org/confluence/display/METRON/Enhancing+Metron+Dashboard
>>>>  
>>>> <https://cwiki.apache.org/confluence/display/METRON/Enhancing+Metron+Dashboard>
>>>> 
>>>> Should at least get us started. 
>>>> 
>>>> Simon
>>>> 
>>>>> On 6 Dec 2017, at 14:00, Otto Fowler <ottobackwa...@gmail.com 
>>>>> <mailto:ottobackwa...@gmail.com>> wrote:
>>>>> 
>>>>> Links?
>>>>> 
>>>>> 
>>>>> On December 6, 2017 at 08:18:23, Simon Elliston Ball 
>>>>> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote:
>>>>> 
>>>>>> We do already have a number of example of exactly this, but sure if 
>>>>>> someone feels like adding to those that would be great. 
>>>>>> 
>>>>>> Simon
>>>>>> 
>>>>>>> On 6 Dec 2017, at 13:14, Otto Fowler <ottobackwa...@gmail.com 
>>>>>>> <mailto:ottobackwa...@gmail.com>> wrote:
>>>>>>> 
>>>>>>> Maybe a Jira logged for an ‘example’ notebook for this would be 
>>>>>>> appropriate as well?
>>>>>>> 
>>>>>>> 
>>>>>>> On December 6, 2017 at 07:06:30, Simon Elliston Ball 
>>>>>>> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) 
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Yes. Consider a zeppelin notebook, or kibana dashboard for this.  
>>>>>>>> 
>>>>>>>> If you want to use these values for detection, consider building a 
>>>>>>>> profile based on the stats objects (see the profiler section of the 
>>>>>>>> documentation under analytics. 
>>>>>>>> 
>>>>>>>> Simon 
>>>>>>>> 
>>>>>>>> > On 6 Dec 2017, at 07:42, Syed Hammad Tahir <mscs16...@itu.edu.pk 
>>>>>>>> > <mailto:mscs16...@itu.edu.pk>> wrote: 
>>>>>>>> >  
>>>>>>>> > Hi, 
>>>>>>>> >  
>>>>>>>> > Can I setup custom visualization to show lets say the peak netrwork 
>>>>>>>> > usage traffic in a certain time? 
>>>>>>>> >  
>>>>>>>> > Regards.

Re: Basic analysis

[Simon Elliston Ball]

No problem, I’ll grant you it’s not in the most intuitive part of the source 
tree to go digging in, but you can also get to the zeppelin bits via the 
actions button on the Metron config section (Install Notebooks)

If anyone has any good ideas (or code!) for sample zeppelin notebooks that 
would be useful, you can add them to a specific instance of the platform via 
the config/zeppelin/metron location and run the action again I believe, and 
this would be a great place for more security people to contribute sample run 
books for example. There are also efforts by commercial support providers I 
believe to add more samples of both dashboards and use cases.

Simon

> On 6 Dec 2017, at 14:12, Otto Fowler <ottobackwa...@gmail.com> wrote:
> 
> Thanks Simon
> 
> 
> On December 6, 2017 at 09:11:50, Simon Elliston Ball 
> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote:
> 
>> In product… Install Zeppelin Notebooks, and the samples including notebooks 
>> at 
>> https://github.com/apache/metron/tree/master/metron-platform/metron-indexing/src/main/config/zeppelin/metron
>>  
>> <https://github.com/apache/metron/tree/master/metron-platform/metron-indexing/src/main/config/zeppelin/metron>
>> 
>> as of course there are similar Kibana dashboards included, which are 
>> examples of custom visualisation of metron data, there is also the run book 
>> for visualising squid data in kibana on the docs wiki 
>> https://cwiki.apache.org/confluence/display/METRON/Enhancing+Metron+Dashboard
>>  
>> <https://cwiki.apache.org/confluence/display/METRON/Enhancing+Metron+Dashboard>
>> 
>> Should at least get us started. 
>> 
>> Simon
>> 
>>> On 6 Dec 2017, at 14:00, Otto Fowler <ottobackwa...@gmail.com 
>>> <mailto:ottobackwa...@gmail.com>> wrote:
>>> 
>>> Links?
>>> 
>>> 
>>> On December 6, 2017 at 08:18:23, Simon Elliston Ball 
>>> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote:
>>> 
>>>> We do already have a number of example of exactly this, but sure if 
>>>> someone feels like adding to those that would be great. 
>>>> 
>>>> Simon
>>>> 
>>>>> On 6 Dec 2017, at 13:14, Otto Fowler <ottobackwa...@gmail.com 
>>>>> <mailto:ottobackwa...@gmail.com>> wrote:
>>>>> 
>>>>> Maybe a Jira logged for an ‘example’ notebook for this would be 
>>>>> appropriate as well?
>>>>> 
>>>>> 
>>>>> On December 6, 2017 at 07:06:30, Simon Elliston Ball 
>>>>> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote:
>>>>> 
>>>>>> Yes. Consider a zeppelin notebook, or kibana dashboard for this.  
>>>>>> 
>>>>>> If you want to use these values for detection, consider building a 
>>>>>> profile based on the stats objects (see the profiler section of the 
>>>>>> documentation under analytics. 
>>>>>> 
>>>>>> Simon 
>>>>>> 
>>>>>> > On 6 Dec 2017, at 07:42, Syed Hammad Tahir <mscs16...@itu.edu.pk 
>>>>>> > <mailto:mscs16...@itu.edu.pk>> wrote: 
>>>>>> >  
>>>>>> > Hi, 
>>>>>> >  
>>>>>> > Can I setup custom visualization to show lets say the peak netrwork 
>>>>>> > usage traffic in a certain time? 
>>>>>> >  
>>>>>> > Regards.

Re: Basic analysis

[Simon Elliston Ball]

In product… Install Zeppelin Notebooks, and the samples including notebooks at 
https://github.com/apache/metron/tree/master/metron-platform/metron-indexing/src/main/config/zeppelin/metron
 
<https://github.com/apache/metron/tree/master/metron-platform/metron-indexing/src/main/config/zeppelin/metron>

as of course there are similar Kibana dashboards included, which are examples 
of custom visualisation of metron data, there is also the run book for 
visualising squid data in kibana on the docs wiki 
https://cwiki.apache.org/confluence/display/METRON/Enhancing+Metron+Dashboard 
<https://cwiki.apache.org/confluence/display/METRON/Enhancing+Metron+Dashboard>

Should at least get us started. 

Simon

> On 6 Dec 2017, at 14:00, Otto Fowler <ottobackwa...@gmail.com> wrote:
> 
> Links?
> 
> 
> On December 6, 2017 at 08:18:23, Simon Elliston Ball 
> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote:
> 
>> We do already have a number of example of exactly this, but sure if someone 
>> feels like adding to those that would be great. 
>> 
>> Simon
>> 
>>> On 6 Dec 2017, at 13:14, Otto Fowler <ottobackwa...@gmail.com 
>>> <mailto:ottobackwa...@gmail.com>> wrote:
>>> 
>>> Maybe a Jira logged for an ‘example’ notebook for this would be appropriate 
>>> as well?
>>> 
>>> 
>>> On December 6, 2017 at 07:06:30, Simon Elliston Ball 
>>> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote:
>>> 
>>>> Yes. Consider a zeppelin notebook, or kibana dashboard for this.  
>>>> 
>>>> If you want to use these values for detection, consider building a profile 
>>>> based on the stats objects (see the profiler section of the documentation 
>>>> under analytics. 
>>>> 
>>>> Simon 
>>>> 
>>>> > On 6 Dec 2017, at 07:42, Syed Hammad Tahir <mscs16...@itu.edu.pk 
>>>> > <mailto:mscs16...@itu.edu.pk>> wrote: 
>>>> >  
>>>> > Hi, 
>>>> >  
>>>> > Can I setup custom visualization to show lets say the peak netrwork 
>>>> > usage traffic in a certain time? 
>>>> >  
>>>> > Regards.

Re: Basic analysis

[Simon Elliston Ball]

Yes. Consider a zeppelin notebook, or kibana dashboard for this. 

If you want to use these values for detection, consider building a profile 
based on the stats objects (see the profiler section of the documentation under 
analytics.

Simon

> On 6 Dec 2017, at 07:42, Syed Hammad Tahir  wrote:
> 
> Hi,
> 
> Can I setup custom visualization to show lets say the peak netrwork usage 
> traffic in a certain time?
> 
> Regards.

Re: Not able to run metron.

[Simon Elliston Ball]

Just FYI, that’s a little outdated, Ubuntu builds are now included in the 
hortonworks distribution, but if you want to build from source the advice there 
very much applies. 

> On 22 Nov 2017, at 13:19, Otto Fowler <ottobackwa...@gmail.com> wrote:
> 
> Btw:  If you have ubuntu clusters, have you seen : 
> https://community.hortonworks.com/articles/88843/manually-installing-apache-metron-on-ubuntu-1404.html
>  
> <https://community.hortonworks.com/articles/88843/manually-installing-apache-metron-on-ubuntu-1404.html>
> 
> On November 22, 2017 at 08:17:41, Otto Fowler (ottobackwa...@gmail.com 
> <mailto:ottobackwa...@gmail.com>) wrote:
> 
>> I build on mac, and have :
>> 
>> --
>> node
>> v6.10.2
>> --
>> npm
>> 3.10.10
>> 
>> for my node versions.
>> 
>> 
>> On November 22, 2017 at 08:05:02, Simon Elliston Ball 
>> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote:
>> 
>>> Sorry, you’re right, you do need ansible. Make sure the version is EXACTLY 
>>> the version in the docs. 
>>> 
>>> Simon
>>> 
>>>> On 22 Nov 2017, at 13:03, Otto Fowler <ottobackwa...@gmail.com 
>>>> <mailto:ottobackwa...@gmail.com>> wrote:
>>>> 
>>>> You DO need ansible for full_dev deployment.
>>>> You do need Docker installed and running
>>>> 
>>>> 
>>>> 
>>>> On November 22, 2017 at 07:51:47, Pawel Bialasiewicz (pa...@evengx.com 
>>>> <mailto:pa...@evengx.com>) wrote:
>>>> 
>>>>> Here is the output:
>>>>> 
>>>>> platform-info.sh
>>>>> Metron 0.4.2
>>>>> --
>>>>> * master
>>>>> --
>>>>> commit 8022f2c8c4e9018a15a4f04d0a66f8bc0ea653c3
>>>>> Author: merrimanr <merrim...@gmail.com <mailto:merrim...@gmail.com>>
>>>>> Date:   Tue Nov 21 13:46:35 2017 -0600
>>>>> 
>>>>> METRON-1319 Column Metadata REST service should use default indices 
>>>>> on empty input (merrimanr) closes apache/metron#843
>>>>> --
>>>>>  metron-deployment/vagrant/full-dev-platform/Vagrantfile | 2 +-
>>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>> --
>>>>> ansible 2.0.0.2
>>>>>   config file = /etc/ansible/ansible.cfg
>>>>>   configured module search path = Default w/o overrides
>>>>> --
>>>>> Vagrant 2.0.1
>>>>> --
>>>>> Python 2.7.12
>>>>> --
>>>>> Apache Maven 3.3.9
>>>>> Maven home: /usr/share/maven
>>>>> Java version: 1.8.0_151, vendor: Oracle Corporation
>>>>> Java home: /usr/lib/jvm/java-8-openjdk-amd64/jre
>>>>> Default locale: en_US, platform encoding: UTF-8
>>>>> OS name: "linux", version: "4.4.0-87-generic", arch: "amd64", family: 
>>>>> "unix"
>>>>> --
>>>>> Docker version 1.13.1, build 092cba3
>>>>> --
>>>>> node
>>>>> v8.9.1
>>>>> --
>>>>> npm
>>>>> 5.5.1
>>>>> --
>>>>> Linux visor 4.4.0-87-generic #110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017 
>>>>> x86_64 x86_64 x86_64 GNU/Linux
>>>>> --
>>>>> Total System Memory = 15996.9 MB
>>>>> Processor Model: Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz
>>>>> Processor Speed: 3427.968 MHz
>>>>> Processor Speed: 3443.437 MHz
>>>>> Processor Speed: 3351.210 MHz
>>>>> Processor Speed: 3438.046 MHz
>>>>> Total Physical Processors: 4
>>>>> Total cores: 16
>>>>> Disk information:
>>>>> /dev/sda1   219G   20G  189G  10% /
>>>>> This CPU appears to support virtualization
>>>>> 
>>>>> 
>>>>> On Wed, Nov 22, 2017 at 1:50 PM, zeo...@gmail.com 
>>>>> <mailto:zeo...@gmail.com> <zeo...@gmail.com <mailto:zeo...@gmail.com>> 
>>>>> wrote:
>>>>> You will need docker to be installed.  In theory this should work across 
>>>>> any *nix distro, but the instructions provided are most thoroughly tested 
>>>>> on macOS endpoints so I always like to clarify.  I know in the past I had 
>>>>> some bumps with spinning it up on CentOS just due to lack of testi

Re: Not able to run metron.

[Simon Elliston Ball]

Sorry, you’re right, you do need ansible. Make sure the version is EXACTLY the 
version in the docs. 

Simon

> On 22 Nov 2017, at 13:03, Otto Fowler  wrote:
> 
> You DO need ansible for full_dev deployment.
> You do need Docker installed and running
> 
> 
> 
> On November 22, 2017 at 07:51:47, Pawel Bialasiewicz (pa...@evengx.com 
> ) wrote:
> 
>> Here is the output:
>> 
>> platform-info.sh
>> Metron 0.4.2
>> --
>> * master
>> --
>> commit 8022f2c8c4e9018a15a4f04d0a66f8bc0ea653c3
>> Author: merrimanr >
>> Date:   Tue Nov 21 13:46:35 2017 -0600
>> 
>> METRON-1319 Column Metadata REST service should use default indices on 
>> empty input (merrimanr) closes apache/metron#843
>> --
>>  metron-deployment/vagrant/full-dev-platform/Vagrantfile | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>> --
>> ansible 2.0.0.2
>>   config file = /etc/ansible/ansible.cfg
>>   configured module search path = Default w/o overrides
>> --
>> Vagrant 2.0.1
>> --
>> Python 2.7.12
>> --
>> Apache Maven 3.3.9
>> Maven home: /usr/share/maven
>> Java version: 1.8.0_151, vendor: Oracle Corporation
>> Java home: /usr/lib/jvm/java-8-openjdk-amd64/jre
>> Default locale: en_US, platform encoding: UTF-8
>> OS name: "linux", version: "4.4.0-87-generic", arch: "amd64", family: "unix"
>> --
>> Docker version 1.13.1, build 092cba3
>> --
>> node
>> v8.9.1
>> --
>> npm
>> 5.5.1
>> --
>> Linux visor 4.4.0-87-generic #110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017 
>> x86_64 x86_64 x86_64 GNU/Linux
>> --
>> Total System Memory = 15996.9 MB
>> Processor Model: Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz
>> Processor Speed: 3427.968 MHz
>> Processor Speed: 3443.437 MHz
>> Processor Speed: 3351.210 MHz
>> Processor Speed: 3438.046 MHz
>> Total Physical Processors: 4
>> Total cores: 16
>> Disk information:
>> /dev/sda1   219G   20G  189G  10% /
>> This CPU appears to support virtualization
>> 
>> 
>> On Wed, Nov 22, 2017 at 1:50 PM, zeo...@gmail.com  
>> > wrote:
>> You will need docker to be installed.  In theory this should work across any 
>> *nix distro, but the instructions provided are most thoroughly tested on 
>> macOS endpoints so I always like to clarify.  I know in the past I had some 
>> bumps with spinning it up on CentOS just due to lack of testing.
>> 
>> Can you run this[1] and report back with the output?  Thanks,
>> 
>> 1:  
>> https://github.com/apache/metron/blob/master/metron-deployment/scripts/platform-info.sh
>>  
>> 
>> 
>> Jon
>> 
>> On Wed, Nov 22, 2017 at 7:21 AM Pawel Bialasiewicz > > wrote:
>> Thank you for the answer Zeolla!
>> 
>> As for now lets focus on the vagrant spin up.
>> 
>> Correct me if I'm wrong: the vagrant full-dev install should work on any 
>> linux distro that has: Vagrant, Virtualbox, Ansible, git, Maven, 
>> vagrant-hostmanager. 
>> 
>> I'm have all of those things installed on a fresh install of Ubuntu 16 LTS. 
>> And it still crashes with the output that I included in the Issue 1327.
>> 
>> Pawel 
>> 
>> 
>> 
>> On Wed, Nov 22, 2017 at 1:11 PM, zeo...@gmail.com  
>> > wrote:
>> It looks like you have two issues - one related to having java_home unset, 
>> and one with an old version of npm.
>> 
>> I would suggest focusing on the Vagrant spin-up, as it is the easiest to get 
>> running.  Are you running this on a CentOS 6 machine, or are you referring 
>> to the full-dev VM's CentOS 6 OS?  Also, you have noted that this affects 
>> version 0.4.0, but the latest release is 0.4.1, is that accurate?  Have you 
>> retried the `vagrant up` after your npm upgrade and a npm` cache clean`?
>> 
>> Jon
>> 
>> On Wed, Nov 22, 2017 at 7:01 AM Pawel Bialasiewicz > > wrote:
>> Hi,
>> 
>> I have been trying to run Metron in many ways and all of the deployment 
>> methods failed:
>> 
>> 1) Bare metal –-> [Issue 1320]
>> 2) AWS –-> [Issue 1318]
>> 3) Vagrant –-> [Issue 1327]
>> 
>> So currently I'm out of options. All the deployments methods failed.
>> 
>> Is the all of documentation outdated, or maybe I should use a more specific 
>> branch(other then master)?
>> 
>> Can somebody confirm that it is currently possible to build metron-config? 
>> It fails for me in every deployment.
>> 
>> Any help would be appreciated.  
>> --
>> Jon
>> 
>> 
>> --
>> Jon
>> 

Re: Not able to run metron.

[Simon Elliston Ball]

You shouldn’t need ansible for the full-dev build, but you will need maven, 
docker and an up-to-date nodejs and npm package to do the actual build. I would 
recommend against using the OS provided nodejs and go with the packages from 
nodesource instead. 

The full-dev build is also the best starting point if you’re looking for an 
environment to test extensions or contributions to the platform, though I would 
make sure you give it plenty of RAM (16GB is a good starting point for the VM I 
find, so you may need to adjust the Vagrantfile).

In a ‘real’ environment, the best way to install is through the Mpack method on 
an existing Ambari install. I would not recommend using ansible at all. It is 
extremely sensitive to ansible version number and very brittle as a result 
because ansible apis keep changing from build to build. 

Simon

> On 22 Nov 2017, at 12:21, Pawel Bialasiewicz  wrote:
> 
> Thank you for the answer Zeolla!
> 
> As for now lets focus on the vagrant spin up.
> 
> Correct me if I'm wrong: the vagrant full-dev install should work on any 
> linux distro that has: Vagrant, Virtualbox, Ansible, git, Maven, 
> vagrant-hostmanager. 
> 
> I'm have all of those things installed on a fresh install of Ubuntu 16 LTS. 
> And it still crashes with the output that I included in the Issue 1327.
> 
> Pawel 
> 
> 
> 
> On Wed, Nov 22, 2017 at 1:11 PM, zeo...@gmail.com  > wrote:
> It looks like you have two issues - one related to having java_home unset, 
> and one with an old version of npm.
> 
> I would suggest focusing on the Vagrant spin-up, as it is the easiest to get 
> running.  Are you running this on a CentOS 6 machine, or are you referring to 
> the full-dev VM's CentOS 6 OS?  Also, you have noted that this affects 
> version 0.4.0, but the latest release is 0.4.1, is that accurate?  Have you 
> retried the `vagrant up` after your npm upgrade and a npm` cache clean`?
> 
> Jon
> 
> On Wed, Nov 22, 2017 at 7:01 AM Pawel Bialasiewicz  > wrote:
> Hi,
> 
> I have been trying to run Metron in many ways and all of the deployment 
> methods failed:
> 
> 1) Bare metal –-> [Issue 1320]
> 2) AWS –-> [Issue 1318]
> 3) Vagrant –-> [Issue 1327]
> 
> So currently I'm out of options. All the deployments methods failed.
> 
> Is the all of documentation outdated, or maybe I should use a more specific 
> branch(other then master)?
> 
> Can somebody confirm that it is currently possible to build metron-config? It 
> fails for me in every deployment.
> 
> Any help would be appreciated.  
> -- 
> Jon
> 
> 

Re: Snort enrichment issue

[Simon Elliston Ball]

Did you setup and load the geo enrichment database? 
https://metron.apache.org/current-book/metron-platform/metron-data-management/index.html#GeoLite2_Loader
 


Also, we can’t really see the error from screenshots, please send log entries. 

Simon

> On 17 Nov 2017, at 07:11, Syed Hammad Tahir  wrote:
> 
> Hi all, I am starting it again. Last one got a bit messy
> 
> Ok, Now I have started everything again from scratch (redeployed single node 
> based ambari metron cluster with ansibleSkipTags = 'quick-dev') and now when 
> I execute this command: 
> 
> shuf -n 10 snort.out | sed -e "s/[^,]\+ ,/`date 
> +'%m\/%d\/%y-%H:%M:%S'`.00 ,/g" | 
> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list 
> node1:6667 --topic snort
> 
> (format of ths command was taken from: 
> https://github.com/apache/metron/blob/master/metron-deployment/roles/sensor-stubs/templates/start-snort-stub
>  
> )
> 
> I get this under enrichment storm topology :
> 
> 
> 
> 
> 
> I have come this far, please help me push these dummy preformatted snort logs 
> into kibana dashboard.
> 
> Regards.
> 

Re: Kibana Error

[Simon Elliston Ball]

Ok, this is an elastic problem which prevents it shutting down. Find the 
elastic processes, kill them, and start it up again.


> On 25 Oct 2017, at 13:15, Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote:
> 
> Just gave the command but its stuck here. I restart it earleir via ambari 
> after changing heapsize. Now doing it via console
> 
> 
> 
> On Wed, Oct 25, 2017 at 5:13 PM, Simon Elliston Ball 
> <si...@simonellistonball.com <mailto:si...@simonellistonball.com>> wrote:
> That just shows running, not health. The problem is that it is not 
> responding. I assume you have tried restarting elastic. 
> 
>> On 25 Oct 2017, at 13:12, Syed Hammad Tahir <mscs16...@itu.edu.pk 
>> <mailto:mscs16...@itu.edu.pk>> wrote:
>> 
>> It shows healthy
>> 
>> 
>> But when I click in any quick link it shows this
>> 
>> 
>> 
>> On Wed, Oct 25, 2017 at 5:07 PM, Simon Elliston Ball 
>> <si...@simonellistonball.com <mailto:si...@simonellistonball.com>> wrote:
>> Did you check the elastic service was running and healthy with the health 
>> checks. Try a few of the quick links from the elastic section in ambari.
>> 
>>> On 25 Oct 2017, at 13:05, Syed Hammad Tahir <mscs16...@itu.edu.pk 
>>> <mailto:mscs16...@itu.edu.pk>> wrote:
>>> 
>>> I have increased size to 2048mb. Still seeing it
>>> 
>>> 
>>> 
>>> On Wed, Oct 25, 2017 at 3:45 PM, Simon Elliston Ball 
>>> <si...@simonellistonball.com <mailto:si...@simonellistonball.com>> wrote:
>>> I strongly suggest you spend some time learning about elastic search and 
>>> some of the basic components. This is not a bug, it’s that elastic is down. 
>>> The default heap (use the ambari search in the elastic section) is probably 
>>> set too low. The default is 128m. Change this to more, probably more like 
>>> 2048m.
>>> 
>>> Essential background reading for metron is an understanding of elastic 
>>> search, kafka, hadoop (hdfs in particular) and Linux. Our docs will assume 
>>> you have at least some familiarity with those technologies.
>>> 
>>> Simon 
>>> 
>>> On 25 Oct 2017, at 11:40, Syed Hammad Tahir <mscs16...@itu.edu.pk 
>>> <mailto:mscs16...@itu.edu.pk>> wrote:
>>> 
>>>> Sorry, I didnt understand. Which baremetal guide should I look into? And I 
>>>> googled it and found no help. Please help me guys, there are bigger issues 
>>>> at hand and I cant afford to waste much time on this problem :( 
>>>> 
>>>> On Wed, Oct 25, 2017 at 1:31 PM, Farrukh Naveed Anjum 
>>>> <anjum.farr...@gmail.com <mailto:anjum.farr...@gmail.com>> wrote:
>>>> Its a bug reported in metron, 
>>>> 
>>>> Look into barematel guide, Turn Red to green Cluster google it.
>>>> 
>>>> On Oct 25, 2017 1:21 PM, "Syed Hammad Tahir" <mscs16...@itu.edu.pk 
>>>> <mailto:mscs16...@itu.edu.pk>> wrote:
>>>> SHould I do it from here? If yes then please guide me how to
>>>> 
>>>> 
>>>> 
>>>> On Wed, Oct 25, 2017 at 1:17 PM, Simon Elliston Ball 
>>>> <si...@simonellistonball.com <mailto:si...@simonellistonball.com>> wrote:
>>>> Your elastic search instance has died. Try given it more heap size in the 
>>>> elastic section on ambari.
>>>> 
>>>> 
>>>> > On 25 Oct 2017, at 09:16, Syed Hammad Tahir <mscs16...@itu.edu.pk 
>>>> > <mailto:mscs16...@itu.edu.pk>> wrote:
>>>> >
>>>> > When I try to open node1:5000 I see this.
>>>> >
>>>> > 
>>>> >
>>>> > What could be the problem and its solution?
>>>> 
>>>> 
>>>> 
>>> 
>> 
>> 
> 
> 

Re: Kibana Error

[Simon Elliston Ball]

That just shows running, not health. The problem is that it is not responding. 
I assume you have tried restarting elastic. 

> On 25 Oct 2017, at 13:12, Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote:
> 
> It shows healthy
> 
> 
> But when I click in any quick link it shows this
> 
> 
> 
> On Wed, Oct 25, 2017 at 5:07 PM, Simon Elliston Ball 
> <si...@simonellistonball.com <mailto:si...@simonellistonball.com>> wrote:
> Did you check the elastic service was running and healthy with the health 
> checks. Try a few of the quick links from the elastic section in ambari.
> 
>> On 25 Oct 2017, at 13:05, Syed Hammad Tahir <mscs16...@itu.edu.pk 
>> <mailto:mscs16...@itu.edu.pk>> wrote:
>> 
>> I have increased size to 2048mb. Still seeing it
>> 
>> 
>> 
>> On Wed, Oct 25, 2017 at 3:45 PM, Simon Elliston Ball 
>> <si...@simonellistonball.com <mailto:si...@simonellistonball.com>> wrote:
>> I strongly suggest you spend some time learning about elastic search and 
>> some of the basic components. This is not a bug, it’s that elastic is down. 
>> The default heap (use the ambari search in the elastic section) is probably 
>> set too low. The default is 128m. Change this to more, probably more like 
>> 2048m.
>> 
>> Essential background reading for metron is an understanding of elastic 
>> search, kafka, hadoop (hdfs in particular) and Linux. Our docs will assume 
>> you have at least some familiarity with those technologies.
>> 
>> Simon 
>> 
>> On 25 Oct 2017, at 11:40, Syed Hammad Tahir <mscs16...@itu.edu.pk 
>> <mailto:mscs16...@itu.edu.pk>> wrote:
>> 
>>> Sorry, I didnt understand. Which baremetal guide should I look into? And I 
>>> googled it and found no help. Please help me guys, there are bigger issues 
>>> at hand and I cant afford to waste much time on this problem :( 
>>> 
>>> On Wed, Oct 25, 2017 at 1:31 PM, Farrukh Naveed Anjum 
>>> <anjum.farr...@gmail.com <mailto:anjum.farr...@gmail.com>> wrote:
>>> Its a bug reported in metron, 
>>> 
>>> Look into barematel guide, Turn Red to green Cluster google it.
>>> 
>>> On Oct 25, 2017 1:21 PM, "Syed Hammad Tahir" <mscs16...@itu.edu.pk 
>>> <mailto:mscs16...@itu.edu.pk>> wrote:
>>> SHould I do it from here? If yes then please guide me how to
>>> 
>>> 
>>> 
>>> On Wed, Oct 25, 2017 at 1:17 PM, Simon Elliston Ball 
>>> <si...@simonellistonball.com <mailto:si...@simonellistonball.com>> wrote:
>>> Your elastic search instance has died. Try given it more heap size in the 
>>> elastic section on ambari.
>>> 
>>> 
>>> > On 25 Oct 2017, at 09:16, Syed Hammad Tahir <mscs16...@itu.edu.pk 
>>> > <mailto:mscs16...@itu.edu.pk>> wrote:
>>> >
>>> > When I try to open node1:5000 I see this.
>>> >
>>> > 
>>> >
>>> > What could be the problem and its solution?
>>> 
>>> 
>>> 
>> 
> 
> 

Re: Kibana Error

[Simon Elliston Ball]

Did you check the elastic service was running and healthy with the health 
checks. Try a few of the quick links from the elastic section in ambari.

> On 25 Oct 2017, at 13:05, Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote:
> 
> I have increased size to 2048mb. Still seeing it
> 
> 
> 
> On Wed, Oct 25, 2017 at 3:45 PM, Simon Elliston Ball 
> <si...@simonellistonball.com <mailto:si...@simonellistonball.com>> wrote:
> I strongly suggest you spend some time learning about elastic search and some 
> of the basic components. This is not a bug, it’s that elastic is down. The 
> default heap (use the ambari search in the elastic section) is probably set 
> too low. The default is 128m. Change this to more, probably more like 2048m.
> 
> Essential background reading for metron is an understanding of elastic 
> search, kafka, hadoop (hdfs in particular) and Linux. Our docs will assume 
> you have at least some familiarity with those technologies.
> 
> Simon 
> 
> On 25 Oct 2017, at 11:40, Syed Hammad Tahir <mscs16...@itu.edu.pk 
> <mailto:mscs16...@itu.edu.pk>> wrote:
> 
>> Sorry, I didnt understand. Which baremetal guide should I look into? And I 
>> googled it and found no help. Please help me guys, there are bigger issues 
>> at hand and I cant afford to waste much time on this problem :( 
>> 
>> On Wed, Oct 25, 2017 at 1:31 PM, Farrukh Naveed Anjum 
>> <anjum.farr...@gmail.com <mailto:anjum.farr...@gmail.com>> wrote:
>> Its a bug reported in metron, 
>> 
>> Look into barematel guide, Turn Red to green Cluster google it.
>> 
>> On Oct 25, 2017 1:21 PM, "Syed Hammad Tahir" <mscs16...@itu.edu.pk 
>> <mailto:mscs16...@itu.edu.pk>> wrote:
>> SHould I do it from here? If yes then please guide me how to
>> 
>> 
>> 
>> On Wed, Oct 25, 2017 at 1:17 PM, Simon Elliston Ball 
>> <si...@simonellistonball.com <mailto:si...@simonellistonball.com>> wrote:
>> Your elastic search instance has died. Try given it more heap size in the 
>> elastic section on ambari.
>> 
>> 
>> > On 25 Oct 2017, at 09:16, Syed Hammad Tahir <mscs16...@itu.edu.pk 
>> > <mailto:mscs16...@itu.edu.pk>> wrote:
>> >
>> > When I try to open node1:5000 I see this.
>> >
>> > 
>> >
>> > What could be the problem and its solution?
>> 
>> 
>> 
> 

Re: Kibana Error

[Simon Elliston Ball]

I strongly suggest you spend some time learning about elastic search and some 
of the basic components. This is not a bug, it’s that elastic is down. The 
default heap (use the ambari search in the elastic section) is probably set too 
low. The default is 128m. Change this to more, probably more like 2048m.

Essential background reading for metron is an understanding of elastic search, 
kafka, hadoop (hdfs in particular) and Linux. Our docs will assume you have at 
least some familiarity with those technologies.

Simon 

> On 25 Oct 2017, at 11:40, Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote:
> 
> Sorry, I didnt understand. Which baremetal guide should I look into? And I 
> googled it and found no help. Please help me guys, there are bigger issues at 
> hand and I cant afford to waste much time on this problem :( 
> 
>> On Wed, Oct 25, 2017 at 1:31 PM, Farrukh Naveed Anjum 
>> <anjum.farr...@gmail.com> wrote:
>> Its a bug reported in metron, 
>> 
>> Look into barematel guide, Turn Red to green Cluster google it.
>> 
>>> On Oct 25, 2017 1:21 PM, "Syed Hammad Tahir" <mscs16...@itu.edu.pk> wrote:
>>> SHould I do it from here? If yes then please guide me how to
>>> 
>>> 
>>> 
>>>> On Wed, Oct 25, 2017 at 1:17 PM, Simon Elliston Ball 
>>>> <si...@simonellistonball.com> wrote:
>>>> Your elastic search instance has died. Try given it more heap size in the 
>>>> elastic section on ambari.
>>>> 
>>>> 
>>>> > On 25 Oct 2017, at 09:16, Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote:
>>>> >
>>>> > When I try to open node1:5000 I see this.
>>>> >
>>>> > 
>>>> >
>>>> > What could be the problem and its solution?
>>>> 
>>> 
> 

Re: SysLog Parser in Metron

[Simon Elliston Ball]

Short answer: grok parsers. 

Longer answer: syslog is more a transport, not just a log format, so it 
encapsulates a wide variety of data sources. Your best bet is probably to use 
NiFi to listen for syslog from a remote host (ListenSyslog) and then route each 
application in the syslog to a different kafka topic. That way you have kafka 
topics for each type of data you care about eg sshd, login, cups... whatever. 
From there it’s easiest to use a grok parser in metron to pull out the fields. 
There are many prebuilt patterns for the common services around on the web.

Simon 

> On 25 Oct 2017, at 05:55, Farrukh Naveed Anjum  
> wrote:
> 
> Hi,
> 
> How can I get syslog in metron any help (pattern / parser). Kindly help ?
> 
> -- 
> With Regards
> Farrukh Naveed Anjum

Re: multiple pattern grok parser in 1 file

[Simon Elliston Ball]

My bad, the pattern surpasses names of capture groups.

AUTHLOG1 %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
%{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} password for 
%{USERNAME:username} from %{IP:ip} %{GREEDYDATA}
AUTHLOG2 %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
%{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} closed for user 
%{USERNAME:username}

AUTHLOG (%{AUTHLOG1}|%{AUTHLOG2})

should work… though to be honest, your patterns look a little unusual. You seem 
to have logs with a timestamp in epoch at the front, which is a very weird way 
to setup syslog, so the issue might be that your patterns flat out don’t match 
the logs. 

Simon


> On 23 Oct 2017, at 10:36, tkg_cangkul <yuza.ras...@gmail.com> wrote:
> 
> Hi Simon,
> 
> I've tried your suggestion but i have an error msg like below :
> 
> 
> 
> On 23/10/17 16:22, Simon Elliston Ball wrote:
>> That is not valid grok. Pattern names should be unique in the grok. 
>> 
>> What you probably mean is something like:
>> 
>> AUTHLOG1 %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
>> %{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} password for 
>> %{USERNAME:username} from %{IP:ip} %{GREEDYDATA}
>> AUTHLOG2 %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
>> %{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} closed for user 
>> %{USERNAME:username}
>> AUTHLOG (?:%{AUTHLOG1}|%{AUTHLOG2})
>> 
>> Simon
>> 
>> 
>>> On 23 Oct 2017, at 08:53, tkg_cangkul <yuza.ras...@gmail.com>
>>>  wrote:
>>> 
>>> FYI,
>>> 
>>> i've trying to using Grok parser metron with multiple pattern in single 
>>> file but it doesn't work. this is my sample grok pattern on 
>>> /apps/metron/patterns/authlog :
>>> 
>>> AUTHLOG %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
>>> %{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} password for 
>>> %{USERNAME:username} from %{IP:ip} %{GREEDYDATA}
>>> AUTHLOG %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
>>> %{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} closed for user 
>>> %{USERNAME:username}
>>> 
>>> When the sensor started, the second grok pattern doesn't work. Only first 
>>> pattern works.
>>> There is an error message like this on storm logs:
>>> 
>>> Caused by: java.lang.RuntimeException: Grok statement produced a null 
>>> message.
>>> 
>>> 
>>> On 23/10/17 10:49, tkg_cangkul wrote:
>>> 
>>>> Hi Wasim, 
>>>> 
>>>> thx for your reply.
>>>> So it means i should use logstash parser for metron?
>>>> Is there any documentation about use logstash parser for metron?
>>>> I didn't found any documentation about that on metron. 
>>>> i just find logstash basic parser but there is no documentation about that.
>>>> 
>>>> 
>>>> 
>>>> On 23/10/17 10:33, Wasim Halani wrote:
>>>> 
>>>>> Hi Youzha,
>>>>> 
>>>>> It should be possible to add multiple patterns in a single config file. 
>>>>> For reference, you can check out the use of multiple patterns in a repo I 
>>>>> maintain [1].
>>>>> You would find the patterns in [2] useful for your use-case.
>>>>> 
>>>>> However, do note that there is a cost to every grok failure [3] - so you 
>>>>> need to ensure that your most common event patterns are at the top of the 
>>>>> list.
>>>>> 
>>>>> As a side-note, if you have any logstash parsers which are not available 
>>>>> in the repo, please feel to submit a PR to [4] 
>>>>> 
>>>>> 
>>>>> [1] 
>>>>> https://bitbucket.org/networkintelligence/logstash-configs/raw/aae3d61bb6c53beb0678536e2e9b33d7996e2960/cisco-asa.conf
>>>>> 
>>>>> [2] 
>>>>> https://bitbucket.org/networkintelligence/logstash-configs/raw/aae3d61bb6c53beb0678536e2e9b33d7996e2960/linux-system.conf
>>>>> 
>>>>> [3] 
>>>>> https://www.elastic.co/blog/do-you-grok-grok
>>>>> 
>>>>> [4] 
>>>>> https://bitbucket.org/networkintelligence/logstash-configs/
>>>>> 
>>>>> 
>>>>> Regards,
>>>>> ---
>>>>> Wasim Halani
>>>>> 
>>>>> http://twitter.com/washalsec
>>>>> http://securitythoughts.wordpress.com
>>>>> 
>>>>> --
>>>>> To keep silent when you can say something wise and useful is as bad as 
>>>>> keeping on propagating foolish and unwise thoughts. -- Imam Ali (p.b.u.h.)
>>>>> 
>>>>> On Mon, Oct 23, 2017 at 8:08 AM, Youzha 
>>>>> <yuza.ras...@gmail.com>
>>>>>  wrote:
>>>>> Hi, is that possible to using multiple pattern grok parser ini 1 pattern 
>>>>> file?
>>>>> i’m trying to parsing authlog file in /var/log/secure into metron. the 
>>>>> problem is there are different structures of logs inside /var/log/secure. 
>>>>> any suggest for this pls?
>>>>> 
>>>>> 
>>>>> Best Regards,
>>>>> 
>>>>> 
>>>>> 
> 

Re: multiple pattern grok parser in 1 file

[Simon Elliston Ball]

That is not valid grok. Pattern names should be unique in the grok. 

What you probably mean is something like:

AUTHLOG1 %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
%{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} password for 
%{USERNAME:username} from %{IP:ip} %{GREEDYDATA}
AUTHLOG2 %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
%{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} closed for user 
%{USERNAME:username}
AUTHLOG (?:%{AUTHLOG1}|%{AUTHLOG2})

Simon

> On 23 Oct 2017, at 08:53, tkg_cangkul  wrote:
> 
> FYI,
> 
> i've trying to using Grok parser metron with multiple pattern in single file 
> but it doesn't work. this is my sample grok pattern on 
> /apps/metron/patterns/authlog :
> 
> AUTHLOG %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
> %{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} password for 
> %{USERNAME:username} from %{IP:ip} %{GREEDYDATA}
> AUTHLOG %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
> %{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} closed for user 
> %{USERNAME:username}
> 
> When the sensor started, the second grok pattern doesn't work. Only first 
> pattern works.
> There is an error message like this on storm logs:
> 
> Caused by: java.lang.RuntimeException: Grok statement produced a null message.
> 
> 
> On 23/10/17 10:49, tkg_cangkul wrote:
>> Hi Wasim, 
>> 
>> thx for your reply.
>> So it means i should use logstash parser for metron?
>> Is there any documentation about use logstash parser for metron?
>> I didn't found any documentation about that on metron. 
>> i just find logstash basic parser but there is no documentation about that.
>> 
>> 
>> 
>> On 23/10/17 10:33, Wasim Halani wrote:
>>> Hi Youzha,
>>> 
>>> It should be possible to add multiple patterns in a single config file. For 
>>> reference, you can check out the use of multiple patterns in a repo I 
>>> maintain [1].
>>> You would find the patterns in [2] useful for your use-case.
>>> 
>>> However, do note that there is a cost to every grok failure [3] - so you 
>>> need to ensure that your most common event patterns are at the top of the 
>>> list.
>>> 
>>> As a side-note, if you have any logstash parsers which are not available in 
>>> the repo, please feel to submit a PR to [4] 
>>> 
>>> 
>>> [1] 
>>> https://bitbucket.org/networkintelligence/logstash-configs/raw/aae3d61bb6c53beb0678536e2e9b33d7996e2960/cisco-asa.conf
>>> [2] 
>>> https://bitbucket.org/networkintelligence/logstash-configs/raw/aae3d61bb6c53beb0678536e2e9b33d7996e2960/linux-system.conf
>>> [3] https://www.elastic.co/blog/do-you-grok-grok
>>> [4] https://bitbucket.org/networkintelligence/logstash-configs/
>>> 
>>> Regards,
>>> ---
>>> Wasim Halani
>>> http://twitter.com/washalsec
>>> http://securitythoughts.wordpress.com
>>> --
>>> To keep silent when you can say something wise and useful is as bad as 
>>> keeping on propagating foolish and unwise thoughts. -- Imam Ali (p.b.u.h.)
>>> 
>>> On Mon, Oct 23, 2017 at 8:08 AM, Youzha  wrote:
>>> Hi, is that possible to using multiple pattern grok parser ini 1 pattern 
>>> file?
>>> i’m trying to parsing authlog file in /var/log/secure into metron. the 
>>> problem is there are different structures of logs inside /var/log/secure. 
>>> any suggest for this pls?
>>> 
>>> 
>>> Best Regards,
>>> 
>>> 
>> 
> 

Re: event correlation on metron

[Simon Elliston Ball]

Best bet there is to create a new sensor config using the grok parser type. So 
you would for example have a kafka topic called host_dhcp and a sensor called 
host_dhcp with the relevant grok pattern. 

Simon 

> On 17 Oct 2017, at 19:19, Youzha <yuza.ras...@gmail.com> wrote:
> 
> that’s what i mean.
> what sensor that i need if i want to do this case? 
> especially when i wanna parse some host logs into metron enrichment and 
> indexing
> 
>> On Wed, 18 Oct 2017 at 01.03 Simon Elliston Ball 
>> <si...@simonellistonball.com> wrote:
>> What you want to do in this setting is just TailFile, the just push to 
>> Kafka. The grok piece is more efficiently handled in the Metron grok parser.
>> 
>> Push to a kafka topic named for your sensor, then setup a sensor (a parser 
>> topology to do the grok parsing and any transformation you need). Each 
>> sensor gets its own parser topology.
>> 
>> Simon 
>> 
>> 
>>> On 17 Oct 2017, at 19:00, Youzha <yuza.ras...@gmail.com> wrote:
>>> 
>>> after nifi procces :
>>> 
>>> TAILFILE -> TRANSFORM_TO_GROK -> PUSH_KAFKA
>>> 
>>> what metron topology that i can use to procces the data in kafka? so it can 
>>> be enrichment by metron. i’ve check the article about adding new telemetry 
>>> source with squid, there is a squid topology that will ingest from the 
>>> squid topic in kafka and then put on enrichment kafka topic. 
>>> so how about my use case above? is there any topology that i can use?
>>> 
>>>> On Wed, 18 Oct 2017 at 00.30 Otto Fowler <ottobackwa...@gmail.com> wrote:
>>>> So, 
>>>> There are several options parsing the data and enriching.
>>>> 
>>>> 1.  A native parser ( java ), which you have noticed is not there
>>>> 2.  An instance of the GROK parser, with GROK rules that parser the input
>>>> 3.  If it is CSV an instance of the CSV parser
>>>> 4.  If it is JSON an instance of the JSONMap parser
>>>> 
>>>> If these cannot be applied to your file then your options are:
>>>> 
>>>> 1.  Write or open a jira for a native parser
>>>> 2. find a way to transform your data to one of the above formats, so you 
>>>> can use those parsers.  This again is where nifi can help.  Something like:
>>>> 
>>>> 
>>>> [nifi]
>>>> 
>>>> TAILFILE -> TRANSFORM_TO_JSON -> PUSH_KAFKA
>>>> 
>>>> where TRANSFORM_TO_JSON is a script processor or something built in 
>>>> depending on your format.
>>>> 
>>>> 
>>>> 
>>>>> On October 17, 2017 at 13:16:05, Youzha (yuza.ras...@gmail.com) wrote:
>>>>> 
>>>>> Hi Lauren thx for your reply,
>>>>> 
>>>>> yeah your suggestion absolutely right. i was able to ingest the logs to 
>>>>> kafka. but how metron can enrich and index all of it? i think there are 
>>>>> only  bro, snort, yaf, snort, pcap, websphere topology storm on metron 
>>>>> for parsers. so, how metron can read the logs telemetry and proccess it 
>>>>> so i can use it to event correlation
>>>>> 
>>>>>> On Tue, 17 Oct 2017 at 23.11 Laurens Vets <laur...@daemon.be> wrote:
>>>>>> Hi Youzha,
>>>>>> 
>>>>>> Either check how the snort logs on the full dev installation are 
>>>>>> ingested (I believe it's with a script) or check the Apache NiFi project 
>>>>>> which makes it very easy to read logs from almost any format and ingest 
>>>>>> them to Metron via Kafka.
>>>>>> 
>>>>>>> On 2017-10-17 08:53, Youzha wrote:
>>>>>>> 
>>>>>>> is it possible to ingest other logs like /var/log/secure for example to 
>>>>>>> be new telemetry on metron? i've seen the metron architecture on the 
>>>>>>> website like picture below. host logs, email, av, etc can be telemetry 
>>>>>>> event buffer on metron. if this possible, could you give me some 
>>>>>>> suggestion how to do it ?
>>>>>>>  
>>>>>>> 
>>>>>>>> On Tue, 17 Oct 2017 at 21.00 Nick Allen <n...@nickallen.org> wrote:
>>>>>>>> If you want to look at failed login attempts for each user over time, 
>>>>>>>> then the Profiler might be a good solution.  Your profile will de

Re: event correlation on metron

[Simon Elliston Ball]

What you want to do in this setting is just TailFile, the just push to Kafka. 
The grok piece is more efficiently handled in the Metron grok parser.

Push to a kafka topic named for your sensor, then setup a sensor (a parser 
topology to do the grok parsing and any transformation you need). Each sensor 
gets its own parser topology.

Simon 

> On 17 Oct 2017, at 19:00, Youzha  wrote:
> 
> after nifi procces :
> 
> TAILFILE -> TRANSFORM_TO_GROK -> PUSH_KAFKA
> 
> what metron topology that i can use to procces the data in kafka? so it can 
> be enrichment by metron. i’ve check the article about adding new telemetry 
> source with squid, there is a squid topology that will ingest from the squid 
> topic in kafka and then put on enrichment kafka topic. 
> so how about my use case above? is there any topology that i can use?
> 
>> On Wed, 18 Oct 2017 at 00.30 Otto Fowler  wrote:
>> So, 
>> There are several options parsing the data and enriching.
>> 
>> 1.  A native parser ( java ), which you have noticed is not there
>> 2.  An instance of the GROK parser, with GROK rules that parser the input
>> 3.  If it is CSV an instance of the CSV parser
>> 4.  If it is JSON an instance of the JSONMap parser
>> 
>> If these cannot be applied to your file then your options are:
>> 
>> 1.  Write or open a jira for a native parser
>> 2. find a way to transform your data to one of the above formats, so you can 
>> use those parsers.  This again is where nifi can help.  Something like:
>> 
>> 
>> [nifi]
>> 
>> TAILFILE -> TRANSFORM_TO_JSON -> PUSH_KAFKA
>> 
>> where TRANSFORM_TO_JSON is a script processor or something built in 
>> depending on your format.
>> 
>> 
>> 
>>> On October 17, 2017 at 13:16:05, Youzha (yuza.ras...@gmail.com) wrote:
>>> 
>>> Hi Lauren thx for your reply,
>>> 
>>> yeah your suggestion absolutely right. i was able to ingest the logs to 
>>> kafka. but how metron can enrich and index all of it? i think there are 
>>> only  bro, snort, yaf, snort, pcap, websphere topology storm on metron for 
>>> parsers. so, how metron can read the logs telemetry and proccess it so i 
>>> can use it to event correlation
>>> 
 On Tue, 17 Oct 2017 at 23.11 Laurens Vets  wrote:
 Hi Youzha,
 
 Either check how the snort logs on the full dev installation are ingested 
 (I believe it's with a script) or check the Apache NiFi project which 
 makes it very easy to read logs from almost any format and ingest them to 
 Metron via Kafka.
 
> On 2017-10-17 08:53, Youzha wrote:
> 
> is it possible to ingest other logs like /var/log/secure for example to 
> be new telemetry on metron? i've seen the metron architecture on the 
> website like picture below. host logs, email, av, etc can be telemetry 
> event buffer on metron. if this possible, could you give me some 
> suggestion how to do it ?
>  
> 
>> On Tue, 17 Oct 2017 at 21.00 Nick Allen  wrote:
>> If you want to look at failed login attempts for each user over time, 
>> then the Profiler might be a good solution.  Your profile will depend on 
>> the fields available in your telemetry, but it would look something like 
>> this, as an example.
>>  
>> {
>>   "profile": "failed-logins",
>>   "foreach": "user.name",
>>   "onlyif": "source.type == 'activedirectory' and event.type == 
>> 'failed_login'"
>>   "init": { "count": 0 },
>>   "update": { "count" : "count + 1" },
>>   "result": "count"
>> }
>>  
>> You can find an introduction and more information on using the Profiler 
>> below.
>> * 
>> https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
>> * https://www.slideshare.net/secret/GFBf2RTXBG35PB
>>  
>> Best of luck
>> 
>>> On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul  
>>> wrote:
>>> for example,
>>> 
>>> i wanna try to correlate between logs.
>>> how many times user A have login failed and how many times user A have 
>>> login succeed. include detail IP, timestamp etc.
>>> is this possible to do with metron?
>>> 
>>> 
>>> 
>>> 
 On 17/10/17 02:56, James Sirota wrote:
 What specifically are you looking to correlate?  Can you talk a little 
 more about your use case?
 
 16.10.2017, 02:23, "tkg_cangkul" :
> hi,
> 
> anyone could explain me about event correlation using apache metron?
> does metron support event correlation?
> 
> Pls Advice
 ---
 Thank you,
 
 James Sirota
 PMC- Apache Metron
 jsirota AT apache DOT org
 

Re: Metron Error in Barematel Installation

[Simon Elliston Ball]

If you can run again with -X and post the debug output somewhere, we should be 
able to figure out where the dependency that’s failing is. 

Simon

> On 16 Oct 2017, at 12:30, Farrukh Naveed Anjum <anjum.farr...@gmail.com> 
> wrote:
> 
> Used that too but, It seems like as you said some dependency got updated... 
> and its is breaking it.
> 
> On Mon, Oct 16, 2017 at 4:25 PM, Simon Elliston Ball 
> <si...@simonellistonball.com <mailto:si...@simonellistonball.com>> wrote:
> This looks like an error in the frontend build. Sometimes this is transient 
> (problems downloading npm packages) so a retry may help. However, we really 
> should be looking at pinning the dependency versions, as this can also be 
> caused by third-party npm packages being updated in the wild and breaking 
> backward compatibility. 
> 
> Btw: an easier way to avoid rat errors in the build is to just use 
> "-Drat.skip=true" instead of "-Dlicense.skip=true 
> -Drat.numUnapprovedLicenses=100"
> 
> Simon
> 
> 
>> On 16 Oct 2017, at 12:20, Farrukh Naveed Anjum <anjum.farr...@gmail.com 
>> <mailto:anjum.farr...@gmail.com>> wrote:
>> 
>> Hi,
>> 
>> I am experience error during build. Mean while following 0.4.1 Installation. 
>> Any help will be appericated
>> 
>> cd metron
>> mvn clean package -DskipTests=true -Dlicense.skip=true 
>> -Drat.numUnapprovedLicenses=100  -T 2C -P HDP-2.5.0.0,mpack
>> 
>> https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.1+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST
>>  
>> <https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.1+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST>
>> 
>> 
>> Log output
>> 0 info it worked if it ends with ok
>> 1 verbose cli [ '/root/metron/metron-interface/metron-config/node/node',
>> 1 verbose cli   
>> '/root/metron/metron-interface/metron-config/node/node_modules/npm/bin/npm-cli.js',
>> 1 verbose cli   'run',
>> 1 verbose cli   'build' ]
>> 2 info using npm@3.8.9
>> 3 info using node@v6.2.0
>> 4 verbose run-script [ 'prebuild', 'build', 'postbuild' ]
>> 5 info lifecycle metron-management-ui@0.4.1~prebuild: 
>> metron-management-ui@0.4.1
>> 6 silly lifecycle metron-management-ui@0.4.1~prebuild: no script for 
>> prebuild, continuing
>> 7 info lifecycle metron-management-ui@0.4.1~build: metron-management-ui@0.4.1
>> 8 verbose lifecycle metron-management-ui@0.4.1~build: unsafe-perm in 
>> lifecycle true
>> 9 verbose lifecycle metron-management-ui@0.4.1~build: PATH: 
>> /root/metron/metron-interface/metron-config/node/node_modules/npm/bin/node-gyp-bin:/root/metron/metron-interface/metron-config/node_modules/.bin:/root/metron/metron-interface/metron-config/node:/root/metron/metron-interface/metron-config/node:/opt/apache-maven-3.3.9/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:
>> 10 verbose lifecycle metron-management-ui@0.4.1~build: CWD: 
>> /root/metron/metron-interface/metron-config
>> 11 silly lifecycle metron-management-ui@0.4.1~build: Args: [ '-c', 
>> './node_modules/angular-cli/bin/ng build -prod' ]
>> 12 silly lifecycle metron-management-ui@0.4.1~build: Returned: code: 1  
>> signal: null
>> 13 info lifecycle metron-management-ui@0.4.1~build: Failed to exec build 
>> script
>> 14 verbose stack Error: metron-management-ui@0.4.1 build: 
>> `./node_modules/angular-cli/bin/ng build -prod`
>> 14 verbose stack Exit status 1
>> 14 verbose stack at EventEmitter. 
>> (/root/metron/metron-interface/metron-config/node/node_modules/npm/lib/utils/lifecycle.js:245:16)
>> 14 verbose stack at emitTwo (events.js:106:13)
>> 14 verbose stack at EventEmitter.emit (events.js:191:7)
>> 14 verbose stack at ChildProcess. 
>> (/root/metron/metron-interface/metron-config/node/node_modules/npm/lib/utils/spawn.js:24:14)
>> 14 verbose stack at emitTwo (events.js:106:13)
>> 14 verbose stack at ChildProcess.emit (events.js:191:7)
>> 14 verbose stack at maybeClose (internal/child_process.js:850:16)
>> 14 verbose stack at Process.ChildProcess._handle.onexit 
>> (internal/child_process.js:215:5)
>> 15 verbose pkgid metron-management-ui@0.4.1
>> 16 verbose cwd /root/metron/metron-interface/metron-config
>> 17 error Linux 3.10.0-327.el7.x86_64
>> 18 error argv "/root/metron/metron-interface/metron-config/node/node" 
>> "/root/metron/metron-interface/metron-config/node/node_modules/npm/bin/npm-cli.js"
>>  "run" "build"
>> 19 error node v6.2.0
>&g

Re: Metron Error in Barematel Installation

[Simon Elliston Ball]

This looks like an error in the frontend build. Sometimes this is transient 
(problems downloading npm packages) so a retry may help. However, we really 
should be looking at pinning the dependency versions, as this can also be 
caused by third-party npm packages being updated in the wild and breaking 
backward compatibility. 

Btw: an easier way to avoid rat errors in the build is to just use 
"-Drat.skip=true" instead of "-Dlicense.skip=true 
-Drat.numUnapprovedLicenses=100"

Simon


> On 16 Oct 2017, at 12:20, Farrukh Naveed Anjum  
> wrote:
> 
> Hi,
> 
> I am experience error during build. Mean while following 0.4.1 Installation. 
> Any help will be appericated
> 
> cd metron
> mvn clean package -DskipTests=true -Dlicense.skip=true 
> -Drat.numUnapprovedLicenses=100  -T 2C -P HDP-2.5.0.0,mpack
> 
> https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.1+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST
>  
> 
> 
> 
> Log output
> 0 info it worked if it ends with ok
> 1 verbose cli [ '/root/metron/metron-interface/metron-config/node/node',
> 1 verbose cli   
> '/root/metron/metron-interface/metron-config/node/node_modules/npm/bin/npm-cli.js',
> 1 verbose cli   'run',
> 1 verbose cli   'build' ]
> 2 info using npm@3.8.9
> 3 info using node@v6.2.0
> 4 verbose run-script [ 'prebuild', 'build', 'postbuild' ]
> 5 info lifecycle metron-management-ui@0.4.1~prebuild: 
> metron-management-ui@0.4.1
> 6 silly lifecycle metron-management-ui@0.4.1~prebuild: no script for 
> prebuild, continuing
> 7 info lifecycle metron-management-ui@0.4.1~build: metron-management-ui@0.4.1
> 8 verbose lifecycle metron-management-ui@0.4.1~build: unsafe-perm in 
> lifecycle true
> 9 verbose lifecycle metron-management-ui@0.4.1~build: PATH: 
> /root/metron/metron-interface/metron-config/node/node_modules/npm/bin/node-gyp-bin:/root/metron/metron-interface/metron-config/node_modules/.bin:/root/metron/metron-interface/metron-config/node:/root/metron/metron-interface/metron-config/node:/opt/apache-maven-3.3.9/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:
> 10 verbose lifecycle metron-management-ui@0.4.1~build: CWD: 
> /root/metron/metron-interface/metron-config
> 11 silly lifecycle metron-management-ui@0.4.1~build: Args: [ '-c', 
> './node_modules/angular-cli/bin/ng build -prod' ]
> 12 silly lifecycle metron-management-ui@0.4.1~build: Returned: code: 1  
> signal: null
> 13 info lifecycle metron-management-ui@0.4.1~build: Failed to exec build 
> script
> 14 verbose stack Error: metron-management-ui@0.4.1 build: 
> `./node_modules/angular-cli/bin/ng build -prod`
> 14 verbose stack Exit status 1
> 14 verbose stack at EventEmitter. 
> (/root/metron/metron-interface/metron-config/node/node_modules/npm/lib/utils/lifecycle.js:245:16)
> 14 verbose stack at emitTwo (events.js:106:13)
> 14 verbose stack at EventEmitter.emit (events.js:191:7)
> 14 verbose stack at ChildProcess. 
> (/root/metron/metron-interface/metron-config/node/node_modules/npm/lib/utils/spawn.js:24:14)
> 14 verbose stack at emitTwo (events.js:106:13)
> 14 verbose stack at ChildProcess.emit (events.js:191:7)
> 14 verbose stack at maybeClose (internal/child_process.js:850:16)
> 14 verbose stack at Process.ChildProcess._handle.onexit 
> (internal/child_process.js:215:5)
> 15 verbose pkgid metron-management-ui@0.4.1
> 16 verbose cwd /root/metron/metron-interface/metron-config
> 17 error Linux 3.10.0-327.el7.x86_64
> 18 error argv "/root/metron/metron-interface/metron-config/node/node" 
> "/root/metron/metron-interface/metron-config/node/node_modules/npm/bin/npm-cli.js"
>  "run" "build"
> 19 error node v6.2.0
> 20 error npm  v3.8.9
> 21 error code ELIFECYCLE
> 22 error metron-management-ui@0.4.1 build: `./node_modules/angular-cli/bin/ng 
> build -prod`
> 22 error Exit status 1
> 23 error Failed at the metron-management-ui@0.4.1 build script 
> './node_modules/angular-cli/bin/ng build -prod'.
> 23 error Make sure you have the latest version of node.js and npm installed.
> 23 error If you do, this is most likely a problem with the 
> metron-management-ui package,
> 23 error not with npm itself.
> 23 error Tell the author that this fails on your system:
> 23 error ./node_modules/angular-cli/bin/ng build -prod
> 23 error You can get information on how to open an issue for this project 
> with:
> 23 error npm bugs metron-management-ui
> 23 error Or if that isn't available, you can get their info via:
> 23 error npm owner ls metron-management-ui
> 23 error There is likely additional logging output above.
> 24 verbose exit [ 1, true ]
> 
> -- 
> With Regards
> Farrukh Naveed Anjum

Re: Initial Testing

[Simon Elliston Ball]

Syed, I would strongly suggest you go through the Squid based tutorial to get 
an idea of how enrichment and indexing works. See: 
https://cwiki.apache.org/confluence/display/METRON/Metron+Reference+Application 


> On 5 Oct 2017, at 09:13, Syed Hammad Tahir  wrote:
> 
> Thanks for the information. Can I get any tutorial or guide on that 
> enrichment and labelling phase in metron?
> 
> On Thu, Oct 5, 2017 at 1:05 PM, Umesh Kaushik  > wrote:
> Yes, after passing your data from enrichment and labelling phase you can 
> further take it do data modelling phase where you can use python kind of 
> language to apply different modelling techniques on your data.
> 
> Cheers,
> Umesh Kaushik
> 9620023458
> 
> Sent from mobile device, kindly ignore the typographical errors.
> 
> On 05-Oct-2017 10:55 AM, "Syed Hammad Tahir"  > wrote:
> Hi,
> 
> Lets say I have dumped snort data. Can I apply some machine learning on it in 
> metron?
> 
> On Thu, Oct 5, 2017 at 12:54 AM, James Sirota  > wrote:
> 1 - It us up to you to install and configure snort however you want. Metron 
> simply consumes the Snort telemetry, but is not opinionated about how you 
> setup your sensors. I would recommend starting with the community rule set: 
> https://www.snort.org/faq/what-are-community-rules 
> 
>  
> 2 - Again, this is outside of scope of Metron. You can view this video to get 
> you started: https://www.youtube.com/watch?v=RUmYojxy3Xw 
> 
>  
> 3 - Metron is not a network mapping tool (although support for graph 
> databases is not too far in the future). Today, the best way to generate a 
> network map (graph) is by using kibana. I would refer you to the following 
> article: https://www.elastic.co/products/x-pack/graph 
> 
>  
> 4 - The snort generated data would be indexed in Elasticsearch and/or stored 
> on HDFS, depending on how you configured the system
>  
> Thanks,
> James
> 
> 
> 04.10.2017, 03:23, "Syed Hammad Tahir"  >:
>> Hi all,
>> 
>> Now that I have installed metron (single node installation on ubuntu 
>> machine), I want to do some initial testing on snort data. I have a few 
>> questions regarding this:
>> 
>> 1- In how many configurations can I use snort with metron (for ex packet 
>> capture in sniffing mode etc)?
>> 
>> 2- How can I change the rules in snort
>> 
>> 3- Can I map the network using metron?
>> 
>> 4- Is snort generated data stored somewhere?
>> 
>> KIndly also give me some tutorial to follow for better understanding.
>> Regards.
>> 
>> 
> 
> 
> --- 
> Thank you,
>  
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
> 
> 
> 

Re: Metron Alerts UI, no alerts

[Simon Elliston Ball]

Right now, you can't. I believe we should be taking the lost of index prefixes 
we use in the ui from the index config via the rest api, we can pull the names 
from each sensor index config and use that as the prefix in the ui. That way we 
pickup any new index automatically.

Simon 

> On 28 Sep 2017, at 20:04, Otto Fowler  wrote:
> 
> How would you add a new sensor in?  Like squid if you were doing the tutorial?
> 
> 
>> On September 28, 2017 at 14:52:11, RaghuMitra Kandikonda 
>> (raghumitra@gmail.com) wrote:
>> 
>> Alerts UI shows all the records in the indexes for the following 
>> sensors 'websphere', 'snort', 'asa', 'bro', 'yaf'. It does not show 
>> records under .kibana as they are not the alerts generated by the 
>> system. Usually the index names for the sensors would have a sensor 
>> name prefix followed by timestamp Ex: snort_index_2017.09.28.18 
>> 
>> -Raghu 
>> 
>> On Thu, Sep 28, 2017 at 11:08 PM, Laurens Vets  wrote: 
>> > Hello, 
>> > 
>> > I've got the Alerts UI up and running. However, I do not see any alerts. I 
>> > can see events in Kibana with "is_alert" set to "true" and with a score as 
>> > well, but they do not show up in the Alerts UI. 
>> > 
>> > How and where does the Alerts UI get actual alerts? 

Re: Not seeing any Metron alerts.

[Simon Elliston Ball]

Probably, though there are things (unlikely things) you can do to templates 
that would prevent that. 

> On 26 Sep 2017, at 17:25, Laurens Vets <laur...@daemon.be> wrote:
> 
> Why would I need to update my ES template? I should see the field (possibly 
> with the wrong type) anyways in the event after I refreshed the fields in 
> Kibana right?
> 
> On 2017-09-26 09:16, Simon Elliston Ball wrote:
> 
>> There should be, though you may need to update your templates in ES if 
>> you've got any custom templates there, and make sure you refresh the fields 
>> in kibana's index config. 
>>  
>> Simon
>>  
>> 
>>> On 26 Sep 2017, at 17:13, Laurens Vets <laur...@daemon.be 
>>> <mailto:laur...@daemon.be>> wrote:
>>> 
>>> After setting is_alert to true, this field is now shown in my event in 
>>> Kibana. I would expect there also to be a field "threat:triage:level" in 
>>> those same events (if my rules work?)
>>> 
>>> On 2017-09-25 16:46, zeo...@gmail.com <mailto:zeo...@gmail.com> wrote:
>>> 
>>> I was quickly reading through this on my mobile device so sorry if I'm off 
>>> base here, but it may be because threat.triage.level is changed to 
>>> threat:triage:level just before indexing due to the inability to use a 
>>> period in keys on older versions of ES.  Not sure exactly what you mean by 
>>> you don't get a threat.triage.level field.
>>> 
>>> Jon
>>> 
>>> 
>>> On Mon, Sep 25, 2017, 19:34 Laurens Vets <laur...@daemon.be 
>>> <mailto:laur...@daemon.be>> wrote:
>>> Next problem:
>>> 
>>> I'm setting the "is_alert" field to true. It shows up in Kibana, but I
>>> don't get a threat.triage.level field which means that either my
>>> riskLevelRules rules don't trigger or something else goes wrong.
>>> 
>>> How and where can I look for additional information on why my rules
>>> might not be working? (Metron UI accepts my JSON without issues)
>>> 
>>> On 2017-09-25 13:39, Laurens Vets wrote:
>>> > Thanks!
>>> >
>>> > On 2017-09-25 13:16, Simon Elliston Ball wrote:
>>> >> The second statement overwrites the first, but also uses the previous
>>> >> value.
>>> >>
>>> >> Technically that is an or. Note this construct is designed to allow
>>> >> multiple different trigger conditions to make is_alert true, hence the
>>> >> second one being is_alert := is_alert || something_else.
>>> >>
>>> >> && is bitwise and
>>> >> || is bitwise or
>>> >>
>>> >> Simon
>>> >>
>>> >>> On 25 Sep 2017, at 21:12, Laurens Vets <laur...@daemon.be 
>>> >>> <mailto:laur...@daemon.be>> wrote:
>>> >>>
>>> >>> Thanks! Followup question, the below is_alert 'rules' in the snippet
>>> >>> from
>>> >>> http://apache.website-solution.net/metron/0.4.1/site-book/use-cases/geographic_login_outliers/index.html
>>> >>>  
>>> >>> <http://apache.website-solution.net/metron/0.4.1/site-book/use-cases/geographic_login_outliers/index.html>,
>>> >>> are those an AND or OR?
>>> >>>
>>> >>>  "threatIntel": {
>>> >>>"fieldMap": {
>>> >>>  "stellar" : {
>>> >>>"config" : [
>>> >>>  "geo_distance_distr:= STATS_MERGE( PROFILE_GET(
>>> >>> 'geo_distribution_from_centroid', 'global', PROFILE_FIXED( 2,
>>> >>> 'MINUTES')))",
>>> >>>  "dist_median := STATS_PERCENTILE(geo_distance_distr, 50.0)",
>>> >>>  "dist_sd := STATS_SD(geo_distance_distr)",
>>> >>>  "geo_outlier := ABS(dist_median - geo_distance) >=
>>> >>> 5*dist_sd",
>>> >>>  "is_alert := exists(is_alert) && is_alert",
>>> >>>  "is_alert := is_alert || (geo_outlier != null && geo_outlier
>>> >>> == true)",
>>> >>>  "geo_distance_distr := null"
>>> >>>]
>>> >>>  }
>>> >>>},
>>> >>>
>>> >>> For instance, can the 2nd is_alert line overwrite the value as

Re: Not seeing any Metron alerts.

[Simon Elliston Ball]

the _score field is actually an elastic search matching score field, and is not 
relevant to metron. You should see the scores in the threat:triage:score field. 
However, your rules will only be run if the telemetry has is_alert set true, so 
you should ensure that the enrichment phase sets is_alert: true somewhere for 
alerts you want to go to triage? 

Simon

> On 25 Sep 2017, at 18:46, Laurens Vets  wrote:
> 
> I have the following configuration:
> 
> "threatIntel": {
>  "fieldMap": {},
>"fieldToTypeMap": {},
>"config": {},
>"triageConfig": {
>  "riskLevelRules": [
>{
>  "name": "Rule1",
>  "comment": "Checks whatever 1.",
>  "rule": "test == \"false\"",
>  "score": 20,
>  "reason": null
>},
>{
>  "name": "Rule1",
>  "comment": "Checks whatever 2.",
>  "rule": "test2 == \"False\"",
>  "score": 20,
>  "reason": null
>},
>{
>  "name": "Rule3",
>  "comment": "Checks whatever 2.",
>  "rule": "test3 == \"No\"",
>  "score": 20,
>  "reason": null
>}
>  ],
>  "aggregator": "SUM",
>  "aggregationConfig": {}
>}
> },
> 
> I have no additional configuration in enrichment besides filling a specific 
> with true or false based on a Stellar expression.
> 
> I expected that when events would match my above rules, the _score field 
> would be filled in. That does not seem to be the case.
> 
> Does anyone know what I might be missing?

Re: Unable to add the hosts

[Simon Elliston Ball]

The list says it wants one host per line, you have given it comma separated. 

> On 25 Sep 2017, at 09:31, kotipalli venkatesh 
>  wrote:
> 
> 
> Hi All,
> 
> Please help on the below error, Target host, we added nodes and import the 
> id_rsa file on the main node. and click the ok button but confirm host status 
> is failed.
> 
> please give suggestion on the below error. 
> 
> 
> 
> 
> 
> 
> 
> 

Re: 192.168.138.158 address in yaf index

[Simon Elliston Ball]

That sounds like an address from the standard example.pcap used to demo metron 
capability. In a real deployment you should not run pcap-replay which is what 
inserts this demo data.

Simon 

> On 21 Sep 2017, at 00:29, Frank Horsfall  
> wrote:
> 
> Morning all,
>  
> I have several logs showing an address of 192.168.138.158 as ip_src_addr and 
> 192.168.138.2 as ip_dst_addr.
>  
> My internal network does not have the 192.168.0.0/24 range which leads me to 
> believe that somewhere there is a test record with the data.
>  
> Would anybody know where I might be able to find it?
>  
> Frank
>  

Re: Clearing of data to start over

[Simon Elliston Ball]

Multiple Kafka brokers will help a lot. The wizard allows you too add more by 
using the plus symbol next to Kafka on the master selection screen. After the 
fact you can add more with the add service button on the hosts screen in ambari.

When adding brokers, don't forget to also alter your topics to have more 
partitions to make use of those brokers. Out of the box the default is a pretty 
useless 1. You should have at least as many partitions as you have disk 
spindles for kafka.

For pulling data from remote sites into metron I would suggest something like 
apache NiFi, using NiFi site to site to a NiFi collocated with your metron. 
That would then just write to kafka. So you can think of NiFi as being a bit 
like an agent or a forwarder. 

Good luck!

Simon 

Sent from my iPhone

> On 7 Sep 2017, at 04:01, Frank Horsfall  
> wrote:
> 
> I'm on a role with questions.
> 
> I'm curious to see if I can relieve processing pressure by adding a new vm. 
> 
> Would you know how I would go about it?
> 
> Also
> I would like to pull data from sources instead of have the sources push data 
> to my site. Have you come across this scenario?
> 
> F
> 
> 
> 
> Sent from my Bell Samsung device over Canada's largest network.
> 
> 
>  Original message 
> From: Frank Horsfall 
> Date: 2017-09-06 10:51 PM (GMT-05:00)
> To: user@metron.apache.org
> Subject: Re: Clearing of data to start over
> 
> Also 
> 
> Laurens you recommended to make 3 Kafka brokers but the install wizard would 
> not let me. 
> 
> As a result my node1 is the only broker currently.  Would this cause a 
> bottleneck?
> 
> If so is there a method to install and configures the 2 additional brokers 
> post initial install?
> 
> kindest regards 
> 
> Frank 
> 
> 
> 
> Sent from my Bell Samsung device over Canada's largest network.
> 
> 
>  Original message 
> From: Frank Horsfall 
> Date: 2017-09-06 10:38 PM (GMT-05:00)
> To: user@metron.apache.org
> Subject: Re: Clearing of data to start over
> 
> Thanks Laurens and Nick.
> 
> I want to let the queues run over night to give us some possible insights 
> into heap sizes etc.
> 
> I currently have 3 vms configured each with 8 cores  500 gigs of data 
> capacity  and 30 gigs of memory.
> 
> Elasticsearch has been configured with 10 gigs xmx.
> 
> I've set storm worker childopts at 7 gigs for now so it takes a while to max 
> out and generate heap errors.
> 
> I deleted approx 6 million events and shut off the data generating apps.
> 
> The idea is to see how much will be processed overnight.
> 
> One thing that has me puzzled is why my bro app isn't emitting events. I 
> double checked my config based on what's recommended but nothing is coming 
> through. A mystery. lol
> 
> 
> Also I kept some notes during the whole process and want to share them if you 
> are interested.  let me know
> 
> Frank
> 
> 
> 
> 
> 
> 
> 
> 
> Sent from my Bell Samsung device over Canada's largest network.
> 
> 
>  Original message 
> From: Laurens Vets 
> Date: 2017-09-06 6:17 PM (GMT-05:00)
> To: user@metron.apache.org
> Cc: Frank Horsfall 
> Subject: Re: Clearing of data to start over
> 
> Hi Frank,
> 
> If you all your queues (Kafka/Storm) are empty, the following should work:
> 
> - Deleting your elasticsearch indices: curl -X DELETE 
> 'http://localhost:9200/snort_index_*', curl -X DELETE 
> 'http://localhost:9200/yaf_index_*', etc...
> 
> - Deleting your Hadoop data:
> 
> Become the hdfs user: sudo su - hdfs
> Show what's been indexed in Hadoop: hdfs dfs -ls 
> /apps/metron/indexing/indexed/ 
> Output should show the following probably:
> /apps/metron/indexing/indexed/error
> /apps/metron/indexing/indexed/snort
> /apps/metron/indexing/indexed/yaf
> ...
> 
> You can remove these with:
> hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/
> hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/snort/
> 
> Or the individial files with
> 
> hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/FILENAME
> 
> 
>> On 2017-09-06 13:59, Frank Horsfall wrote:
>> 
>> Hello all,
>> 
>> I have installed a 3 node system using the bare metal Centos 7 guideline.
>> 
>>  
>> 
>> https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST
>> 
>>  
>> 
>> It has taken me a while to have all components working properly and I left 
>> the yaf,bro,snort apps running so quite a lot of data has been generated.  
>> Currently, I have almost 18 million events identified in Kibana. 16+ million 
>> are yaf based, and 2+ million are snort  …. 190 events are my new squid 
>> telemetry,  J.   It looks like it still has a while to go before it catches 
>> up to current day.   I recently shutdown the apps.
>> 
>>  
>> 
>>  
>> 
>> My questions are:
>> 
>>  
>> 
>> 

Re: Threat triage rules using stellar geo enrichment

[Simon Elliston Ball]

A much better way of doing this is to run the geo enrichment as part of the 
regular enrichment process and then just use the output field for the rule. 
Your config already does this, so your rule is in effect running the same 
enrichment twice. Just use enrichments.geo.ip_dst_addr.country != ‘US’ for a 
significantly simpler and more performant rule.

Simon


> On 8 Aug 2017, at 14:47, Anand Subramanian  
> wrote:
> 
> Thank you, Casey. That worked!
> 
> Regards,
> Anand
> 
> From: Casey Stella >
> Reply-To: "user@metron.apache.org " 
> >
> Date: Tuesday, August 8, 2017 at 7:12 PM
> To: "user@metron.apache.org " 
> >
> Subject: Re: Threat triage rules using stellar geo enrichment
> 
> I think you want:
> GEO_GET( ip_dst_addr, ['country']) != 'US'
> 
> 
> On Tue, Aug 8, 2017 at 7:29 AM, Anand Subramanian 
> > wrote:
> Hello All,
> 
> I am trying to write a triage rule where I would like to set the alert score 
> based on Geo enrichment output, as follows. 
> 
> $ cat $METRON_HOME/config/zookeeper/enrichments/snort.json
> {
>   "enrichment" : {
> "fieldMap":
>   {
>   "geo": ["ip_dst_addr", "ip_src_addr"],
>   "host": ["host"]
> }
>   },
>   "threatIntel" : {
> "fieldMap":
>   {
>   "hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
> },
> "fieldToTypeMap":
>   {
>   "ip_src_addr" : ["malicious_ip"],
>   "ip_dst_addr" : ["malicious_ip"]
> },
> "triageConfig" : {
>   "riskLevelRules" : [
> {
>   "name" : "Rule 1",
>   "rule" : "not(IN_SUBNET(ip_dst_addr, '192.168.0.0/24') 
> )",
>   "score" : 10
> },
> {
>   "name" : "Rule 2",
>   "rule" : "not(GEO_GET(ip_dst_addr, '[country]'), 'US')",
>   "score" : 20
> }
>   ],
>   "aggregator" : "MAX"
> }
>   }
> }
> 
> But I am getting the following error when trying to push the configuration 
> into zookeeper:
> 
> Exception in thread "main" java.lang.RuntimeException: Unable to load {
>   "enrichment" : {
> "fieldMap":
>   {
>   "geo": ["ip_dst_addr", "ip_src_addr"],
>   "host": ["host"]
> }
> 
> at 
> org.apache.metron.common.configuration.ConfigurationType.lambda$static$2(ConfigurationType.java:54)
> at 
> org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:93)
> at 
> org.apache.metron.common.configuration.ConfigurationsUtils.writeSensorEnrichmentConfigToZookeeper(ConfigurationsUtils.java:123)
> at 
> org.apache.metron.common.configuration.ConfigurationsUtils.uploadConfigsToZookeeper(ConfigurationsUtils.java:265)
> at 
> org.apache.metron.common.configuration.ConfigurationsUtils.uploadConfigsToZookeeper(ConfigurationsUtils.java:226)
> at 
> org.apache.metron.common.cli.ConfigurationManager.push(ConfigurationManager.java:155)
> at 
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:170)
> at 
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:161)
> at 
> org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:198)
> Caused by: org.apache.metron.jackson.databind.JsonMappingException: N/A
>  at [Source: {
> 
> }
> ; line: 31, column: 7] (through reference chain: 
> org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig["threatIntel"]->org.apache.metron.common.configuration.enrichment.threatintel.ThreatIntelConfig["triageConfig"]->org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig["riskLevelRules"])
> at 
> org.apache.metron.jackson.databind.JsonMappingException.from(JsonMappingException.java:262)
> at 
> org.apache.metron.jackson.databind.deser.SettableBeanProperty._throwAsIOE(SettableBeanProperty.java:537)
> at 
> org.apache.metron.jackson.databind.deser.SettableBeanProperty._throwAsIOE(SettableBeanProperty.java:518)
> at 
> org.apache.metron.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:99)
> at 
> org.apache.metron.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:260)
> at 
> org.apache.metron.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:125)
> at 
> org.apache.metron.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490)
> at 
> org.apache.metron.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:95)
> at 
> org.apache.metron.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:260)
> at 
> org.apache.metron.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:125)
> at 
> 

Re: How to change Elasticsearch indexing policy

[Simon Elliston Ball]

You could change the index data format. One word of caution here though; the 
last time I saw this done it caused huge problems with locking on ingest 
against people running queries on the current day’s data and tended to knock 
recent relevant indexes out of disk cache at the OS level. It might look like 
it will help a bit for ingest initially, but with load on the end user side, 
it’s probably going to kill your disks at any reasonable scale. 

Simon

> On 14 Jul 2017, at 10:31, Ali Nazemian  wrote:
> 
> Hi,
> 
> I am investigating different tuning aspects, and I was wondering how I can 
> change the policy of Elasticsearch indexing. Currently, as a default 
> behaviour, events are stored in separate indices hourly. How can I change 
> this behaviour? Is this a hard-coded design or I can change it through 
> configurations?
> 
> Cheers,
> Ali

Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance

[Simon Elliston Ball]

In your config for the individual profile though, you expire the content every 
30ms (per the zip file)

Simon



> On 12 Jul 2017, at 12:20, Krishna Dhanekula <krishna.dhanek...@sstech.us> 
> wrote:
> 
> Thanks Simon for replying.
>  
> This is my ‘profiler.properties’ file content. TTL is 30 minutes and for 
> every 30 seconds I am flushing to Hbase.
>  
> profiler.workers=1
> profiler.executors=0
> profiler.input.topic=indexing
> profiler.period.duration=1
> profiler.period.duration.units=MINUTES
> profiler.ttl=30
> profiler.ttl.units=MINUTES
> profiler.hbase.salt.divisor=1000
> profiler.hbase.table=profiler
> profiler.hbase.column.family=P
> profiler.hbase.batch=1
> profiler.hbase.flush.interval.seconds=30
>  
> # Kafka #
>  
> kafka.zk=10.10.110.184:2181
> kafka.broker=10.10.110.184:6667
> kafka.start=WHERE_I_LEFT_OFF
>  
>  
>  
> Regards,
>  
> Balakrishna
>  
>  
> From: Simon Elliston Ball [mailto:si...@simonellistonball.com] 
> Sent: Wednesday, July 12, 2017 3:28 PM
> To: user@metron.apache.org
> Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase 
> Instance
>  
> Looks like you’ve set the profile to purge (expires) every 30 ms, and your 
> period is set to 30 minutes, so the data is being expired long before it has 
> a change to write. 
>  
> Simon
>  
>  
> On 12 Jul 2017, at 06:17, Krishna Dhanekula <krishna.dhanek...@sstech.us 
> <mailto:krishna.dhanek...@sstech.us>> wrote:
>  
> I have an problem where profiled data is not storing to Hbase instance:
>  
> Installed metron profiler in ‘HDP-2.5.5.0-157’ with the help of this link:
>  
> https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html
>  
> <https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html>
>  
> Attached my profiler.properties and profiler.json files, remote.yaml and 
> worker.log files.
>  
> Once I place the following message into kafka topic 'indexing'
>  
> [{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
>  
> It's picked by splitterBolt and passed till HBaseBolt. But data is not 
> getting stored to Hbase.
>  
> Attached metron-artifacts.zip
>  
> Attached storm-ui screenshot.storm-ui-screen.png
>  
> Please suggest what am I missing here.
>  
> Thanks in advance.
>  
> Attachments:
> storm-ui-screen.png (17.4 kB)
> metron-artifacts.zip (36.6 kB)
>  
>  
> Regards,
> Balakrishna
> 

Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance

[Simon Elliston Ball]

Looks like you’ve set the profile to purge (expires) every 30 ms, and your 
period is set to 30 minutes, so the data is being expired long before it has a 
change to write. 

Simon


> On 12 Jul 2017, at 06:17, Krishna Dhanekula  
> wrote:
> 
> I have an problem where profiled data is not storing to Hbase instance:
>  
> Installed metron profiler in ‘HDP-2.5.5.0-157’ with the help of this link:
>  
> https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html
>  
> 
>  
> Attached my profiler.properties and profiler.json files, remote.yaml and 
> worker.log files.
>  
> Once I place the following message into kafka topic 'indexing'
>  
> [{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
>  
> It's picked by splitterBolt and passed till HBaseBolt. But data is not 
> getting stored to Hbase.
>  
> Attached metron-artifacts.zip
>  
> Attached storm-ui screenshot.storm-ui-screen.png
>  
> Please suggest what am I missing here.
>  
> Thanks in advance.
>  
> Attachments:
> storm-ui-screen.png (17.4 kB)
> metron-artifacts.zip (36.6 kB)
>  
>  
> Regards,
> Balakrishna
> 

Re: Metron in-memory enrichment

[Simon Elliston Ball]

Surely the caching should make this effectively an in memory lookup. Does the 
stellar enrichment function not use the same clientside caching as the Hbase 
bolt?

Simon 

> On 19 Jun 2017, at 06:21, Casey Stella  wrote:
> 
> In order to do that, the easiest thing to do is to create a stellar function 
> to load and do in-memory lookups.
> 
>> On Sun, Jun 18, 2017 at 11:48 PM, Ali Nazemian  wrote:
>> Hi all,
>> 
>> We are using Metron HBase enrichment for a few use cases, but we have 
>> noticed the achievable throughput is not very great. I was wondering whether 
>> there is a way to load the external enrichment data in-memory and use it 
>> with normal Stellar enrichments. In our use cases, the number of rows in the 
>> external enrichments that we are dealing with is less than a 100k and it is 
>> a static list, so it is feasible to load them in-memory and use that for the 
>> enrichment. However, I am not sure how that would be achievable from the 
>> Metron capabilities.
>> 
>> Regards,
>> Ali
> 

Re: Question on Windows event log ingest and parse

[Simon Elliston Ball]

And just to check… you have the pattern definition you previously sent in 
/patterns/winlogbeat (file) on HDFS.

It looks like the most likely problem from your config is that you have two 
parserConfig elements. I suspect the second is over-riding the first, and hence 
you are losing the grokPath config, if you move the dc2tz element into the 
first parserConfig, you should be good.

As an aside from a quick look at your pattern, it looks like it may be easier 
to use the JSONMapParser for this particular sensor. 

Simon

> On 4 May 2017, at 01:28, ed d <ragdel...@hotmail.com> wrote:
> 
> Correction, deploying the Storm topology is this:
> 
> /usr/metron/$METRON_VERSION/bin/start_parser_topology.sh -z `hostname 
> -f`:2181 -k `hostname -f`:6667 -s winlogbeat
> 
> 
> 
> 
> 
> From: Simon Elliston Ball <si...@simonellistonball.com>
> Sent: Wednesday, May 3, 2017 5:59 PM
> To: user@metron.apache.org
> Subject: Re: Question on Windows event log ingest and parse
>  
> Hi Ed, 
> 
> Sounds like a really nice piece of work to get pushed into the core… how 
> would you feel about taking that grok parser and formalising it into the core 
> of Metron (happy to help there by the way).
> 
> On the actual issue, is sounds like it’s likely to be something to do with 
> conversion of the timestamp format to the unixtime used in Metron. We can 
> look at that. Did you see any log messages in the storm logs from the 
> topology that died? 
> 
> Simon
> 
> 
>> On 3 May 2017, at 22:34, ed d <ragdel...@hotmail.com 
>> <mailto:ragdel...@hotmail.com>> wrote:
>> 
>> Metron version – 0.4.0
>> Single node install, bare metal install
>> No significant changes to base install besides maintenance mode on 
>> elasticsearch mpack and manual configuration.
>>  
>> I have a Windows 2012 server running AD, AD LDS, DNS, and DHCP. I installed 
>> Winlogbeat <https://www.elastic.co/downloads/beats/winlogbeat>5.3.2 64 bit 
>> onto the server. It was configured to push logs to the Elasticsearch on my 
>> Metron install, and it works great. No issues.
>>  
>> I modified the Winlogbeat configuration to push logs directly to Kafka as I 
>> want to enrich the logs. I followed this guide 
>> <https://www.elastic.co/guide/en/beats/winlogbeat/master/kafka-output.html>.
>>  
>> I can see logs coming into the Kafka topic, so I built a Grok parser to 
>> slice and dice. It seems to work fine on Grok Constructor 
>> <http://grokconstructor.appspot.com/do/match> and Grok Debugger 
>> <https://grokdebug.herokuapp.com/>, but when I load it into Metron as a 
>> parser, it kills the Storm topology. It seems to be sticking on the 
>> timestamp, which is ISO_8601 <https://en.wikipedia.org/wiki/ISO_8601> format 
>> (2017-05-03T21:04:33Z).
>>  
>> My question to the group, before troubleshooting my install, is to see if 
>> anyone else has had success ingesting and parsing Windows event logs?
>>  
>> Does anyone pull Windows log into Kafka, Nifi, or other with the intent to 
>> enrich the elements of the log? And if yes, what have you found to be most 
>> useful?
>>  
>> FYI here is my Grok parser for reference:
>>  
>> timestamp"\:"%{TIMESTAMP_ISO8601:timestamp}","beat"\:\{"hostname"\:%{QUOTEDSTRING:hostname},"name"\:%{QUOTEDSTRING:name},"version"\:%{QUOTEDSTRING:beat_version}\},"computer_name"\:%{QUOTEDSTRING:computer_name},"event_data"\:\{("AuthenticationPackageName"\:%{QUOTEDSTRING:AuthenticationPackageName},?)?("ImpersonationLevel"\:%{QUOTEDSTRING:ImpersonationLevel},?)?("FailureReason"\:%{QUOTEDSTRING:FailureReason},?)?("IpAddress"\:"%{IP:ip_src_addr}",?)?("IpPort"\:%{QUOTEDSTRING:IpPort},?)?("KeyLength"\:%{QUOTEDSTRING:KeyLength},?)?("LmPackageName"\:%{QUOTEDSTRING:LmPackageName},?)?("LogonGuid"\:%{QUOTEDSTRING:LogonGuid},?)?("LogonProcessName"\:%{QUOTEDSTRING:LogonProcessName},?)?("LogonType"\:%{QUOTEDSTRING:LogonType},?)?("PrivilegeList"\:%{QUOTEDSTRING:PrivilegeList},?)?("ProcessId"\:%{QUOTEDSTRING:ProcessId},?)?("ProcessName"\:%{QUOTEDSTRING:ProcessName},?)?("PackageName"\:%{QUOTEDSTRING:PackageName},?)?("Status"\:%{QUOTEDSTRING:Status},?)?("SubStatus"\:%{QUOTEDSTRING:SubStatus},?)?("SubjectDomainName"\:%{QUOTEDSTRING:SubjectDomainName},?)?("SubjectLogonId"\:%{QUOTEDSTRING:SubjectLogonId},?)?("SubjectUserName"\:%{QUOTEDSTRING:SubjectUserName},?)?("SubjectUserSid"\:%{QUOTEDSTRING:SubjectUserSid},?